Between Regulatory Rocks and Hard Places: Quantifying Tolerance for Pharmaceutical Risks

(The following was presented during the Second Plenary Session, “Drug Safety and Risk-Benefit Decision-Making,” at the ISPOR 13th Annual International Meeting, May 6, 2008, Toronto, ON, Canada)

One of the reasons for the current interest in benefit-risk assessment is the recurrence of embarrassing regulatory situations. A recent example involving synthetic blood products was widely reported in the media. Sixteen studies found that synthetic blood products were associated with tripling the risk of heart attacks and increasing the chance of dying by 30%. Researchers publicly questioned the competence of the Food and Drug Administration bureaucrats who allowed additional testing of these products to continue. An FDA official interviewed said that agency experts had carefully scrutinized the evidence and were satisfied that the potential benefits outweighed the risks of further testing and pointed out the huge unmet need for synthetic blood. A successful blood substitute could save thousands of lives of trauma patients (Washington Post, Tuesday, April 29, 2008).

FDA recently has begun to elicit the help of industry and academics in quantifying benefit-risk tradeoffs in ways that might assist and inform regulatory decisions and also help the agency explain decisions involving benefit-risk tradeoffs to the health care community and the general public. There are a number of quantitative approaches that are being explored but this article will describe two of these methods: incremental net benefits and patient benefit-risk tradeoff preferences.

Figure 1a. The Implicit Regulatory Benefit-Risk Threshold	Figure 1b. Patient Benefit-Risk Thresholds

Suppose we had data on a large number of FDA decisions involving benefits and risks where both the benefits and risks were directly comparable. Of course, such data do not exist; however, if they did, we could identify the benefit- risk combinations for drugs that were approved and drugs that were not approved. The clustering of these outcomes would reveal the implicit regulatory benefit-risk threshold that FDA actually is using. Figure 1a shows how this threshold splits the benefit-risk space into two regions, the area where drugs are approvable and the area where drugs are not approvable. Such information would be particularly useful to sponsors of new pharmaceuticals.

Now suppose that we could estimate a patient benefit-risk threshold and compare it to the FDA threshold. Threshold 0A indicates that patients largely agree with FDA; that is, all the drugs that patients want approved are approved and all the drugs that patients do not want approved are not approved. On the other hand, benefit-risk threshold 0B indicates that that patients are more risk averse than FDA, in which case some drugs are approved that patients do not want approved. It is possible, of course, that patients' benefit-risk threshold is 0C, and indicating patients are more tolerant of risks than FDA regulators. In this case FDA does not approve some drugs that patients want approved.

The idea of a patient benefit-risk threshold is useful even in the absence of enough information to specify FDA's regulatory threshold. Suppose we have good evidence on the incremental effectiveness and incremental risk of a new pharmaceutical. Should we approve this drug or not? My purpose is not to suggest a decision rule to FDA, but it may be possible to determine whether patients would want access to the drug or not. Point BR indicates the benefitrisk combination for the drug in question relative to patients' benefit-risk tradeoff curve. The tradeoff curve indicates the threshold below which perceived benefits outweigh perceived risks. It also indicates the minimum benefit a treatment would have to offer patients to induce them to accept the observed treatment risk. This benefit is labeled the minimum acceptable benefit in Figure 2. In this example the benefit is considerably larger than the minimum necessary to get patients to accept the risk to which they actually would be exposed. The net gain is shown as the net effectiveness benefit; that is, the amount of benefit in excess of the minimum necessary you would have to offer patients.

Figure 2. Four Quantitative Benefit-Risk Measures

We can identify similar constructs in the risk dimension of the diagram. Given the actual effectiveness of this drug, what is the maximum acceptable risk patients would tolerate before they would reject treatment? Point BR lies below the benefit-risk tradeoff curve in that dimension as well. Thus approving the drug also provides a net safety benefit to patients because its risks are less than the maximum that patients are willing to accept.

The recurring question of how much of what kind of risk is acceptable arose when VioxxÆ was withdrawn from the market a few years ago. Post-marketing data revealed an elevated level of fatal and non-fatal myocardial infarctions relative to naproxen. The reason that rofecoxib initially was approved was that hundreds of cases per 10,000 patient years of dyspepsia, GI bleeds, and GI perforations would be avoided by patients who used rofecoxib to control pain instead of naproxen. Withdrawing approval meant that regulators were willing to accept thousands of cases of milder adverse events to avoid a relatively small number of heart attacks.

Lynd and colleagues have quantified this tradeoff in an unpublished study. The incremental changes in health utilities are 0.059 for rofecoxib and 0.056 for naproxen, respectively. Weighting the outcome gains and losses by the corresponding utilities results in a net gain in quality-adjusted life years of about 0.001. That is the equivalent of about one additional day of good health for patients on a rofecoxib regime instead of naproxen. Event-simulation indicates there is a 98.3 percent chance that the incremental net benefit is greater than zero. Setting aside other important considerations, if all drugs were approved whose incremental net benefit were greater than zero, then on average patients should be better off.

Incremental net benefits are a logical extension of existing epidemiology and health-economics methods. The approach combines epidemiological data with health-utility data to determine the net-benefit threshold. The problem with this approach is that it often combines good epidemiology with health-utilities of dubious quality. When estimated net benefits are as small as they are in the rofecoxib study, the result is likely to be sensitive to measurement error in the assumed utility values.

There are additional grounds for skepticism about using health utilities to compare treatment benefits and risks. Even if QALYs are useful in health-economic studies and for reimbursement decision making, there are good reasons why QALYs are inappropriate for benefit-risk evaluations. The most serious problem is that health utilities provide outcome weights, not measures of risk tolerance. QALYs as conventionally measured require that we assume that patients and FDA decision makers are risk neutral. That is, we have to assume that patients and regulators do not buy insurance, because anyone who buys insurance is averse to risk. They are willing to pay more in insurance premiums than the expected payments they will receive from the insurance company to avoid having the bear the risk of a large loss.

Figure 3 shows how we can elicit patient preferences that are relevant for benefit- risk evaluations. Patients are asked to make choices among hypothetical treatments. Each treatment description includes specified features such as efficacy, mild to moderate side effects, and serious adverse-event risks. The combinations of treatments that subjects evaluate are based on an experimental design with known statistical properties. Analysis of the pattern of choices observed under these controlled stimuli reveals the implicit preference weights subjects used to guide their choices. Thus we obtain estimates of the relative subjective importance of clinically relevant benefits and harms that also account for patients' aversion to bearing risk.

Figure 3. Example Benefit-Risk Tradeoff Question

Researchers have developed such preference-elicitation methods, variously called conjoint analysis, discrete-choice experiments, or stated-choice surveys, over the last 30 years for a wide range of applications. However, we have relatively little experience in using these methods to elicit patients' benefit-risk tradeoff preferences. The biggest challenge to obtaining valid and reliable preference data involves coping with patients' innumeracy about low-probability risks. We help patients evaluate risk magnitudes by showing them absolute risks, relative risks, comparisons with more familiar risks of similar size, number of patients out of 1000 who would be affected, and a graphical depiction involving a 1000-square grid.

Figure 4 shows an example of an actual estimated benefit-risk tradeoff curve for Crohn's disease [1]. The risk is the risk of dying from progressive multifocal leukoencephalopathy (PML) within ten years of beginning treatment. As expected, patients are willing to accept higher PML risks for better treatment benefits. This study was one of two patient benefit-risk tradeoff studies that were submitted to FDA advisory committees in the last year. The same survey was administered not only to patients, but also to parents of Crohn's children and to physicians with some interesting results. Adult patients generally are more tolerant of risks than parents and both of them are more tolerant of risks than physicians. For example, for the lowest level of benefit -that is, an improvement from mild Crohn's symptoms to remission - physicians were unwilling to accept any additional treatment risk, but parents and patients are [2].

Figure 4. Crohn's Disease Benefit-Risk Tradeoff Curve

We also have administered a benefit-risk tradeoff survey to 2,000 Americans over the age of 60. Subjects evaluated hypothetical treatments that would alter Alzheimer's disease (AD) progression. The tradeoff involved modifying the course of the disease over a seven-year period but at a risk of dying from stroke in the first year of treatment. Subjects were asked whether they were willing to accept tradeoffs between clinically relevant changes in disease progression and clinically relevant stroke mortality risks, not the instantaneous painless death that health-utility subjects evaluate. For a treatment that would halt disease progressions at the mild stage, our subjects, on average, were willing to accept an almost one in three chance of fatal stroke in the first year of treatment [3].

Data on stated-preferences for benefit-risk tradeoffs support a number of useful calculations [4-5]. Table 1 illustrates these possibilities using the AD estimates. The best outcome offered survey subjects was mild cognitive impairment. We can calculate the mild-year equivalent (MYE) for any diseaseprogression profile. We do not obtain simple health-utility values that can be added across durations and disease stages. Rather, preferences are nonlinear and the importance of particular stage duration depends realistically on the durations of other stages in the progression profile. In line (1) of Table 1, the quality equivalence for 7 years with mild AD is simply 7 MYEs. The no-treatment profile is equivalent to 4.9 MYEs. Thus the incremental benefit in line (2) is 2.1 MYEs. The maximum acceptable risk of stroke mortality is 0.31 in line (3). The minimum acceptable number needed to harm is simply the inverse of the maximum acceptable risk, which is 3.2 for the 7-year mild profile. Number needed to harm is often calculated; however, but without knowing what the threshold value is, it is hard to know how to interpret a given result. Preference data provide this missing information. For an assumed actual risk of 0.05, the net safety benefit is 0.26, measured as the difference between maximum acceptable and actual risk. For the same assumed 0.05 risk, the minimum acceptable benefit is 0.2 MYEs, which gives a net efficacy benefit of 1.9 MYEs for the 7-year mild profile.

Table 1. Alzheimer's Benefit-Risk Tradeoff Measures

An Institute of Medicine report [6] found that “in both the pre-approval and the post-marketing setting, the risk-benefit analysis that currently goes into FDA decisions appears to be ad hoc, informal, and qualitative”. Having better evidence on patients' risk tolerance may be useful in responding to such criticisms. Patients currently do have a voice in the regulatory process. A selfselected group of patients often appear at the end of an advisory committee meeting. Some tell stories about how a drug hurt them and some tell stories about how a drug helped them. The stories are often emotional and heartrending. It is unclear how such anecdotal evidence affects advisory-panel recommendations. FDA has expressed an interest in quantifying such evidence in representative samples of patients to determine patients' overall tolerance for treatment risks associated with new pharmaceuticals*. It is important to emphasize that these methods do not solve familiar problems of uncertainty in the evidence, particularly the lack of reliable data on safety. However, we have begun to think about how to incorporate uncertainty in eliciting patients' benefit-risk preferences.

Ultimately regulatory decisions involve value judgments. Such decisions are made on behalf of the rest of us about how much treatment risk we are willing to accept as a society. These judgments often are very difficult and no quantitative method will provide a decision rule to guide decision makers. Nevertheless, better evidence about the preferences of the ultimate stakeholders, the patients themselves, may be a useful complement to traditional forms of evidence used to inform these difficult decisions.

*Note: This refers to remarks made by Robert Powell PharmD, at the Second Plenary Session of the ISPOR 13th Annual International Meeting, May 6, 2008, Toronto, ON, Canada.

REFERENCES

[1] Johnson FR, Özdemir S, Mansfield CA, et al. Crohn's disease patients' benefit-risk preferences: serious adverse event risks versus treatment efficacy. Gastroenterology 2007;133:769-79.

[2] Johnson FR, Hauber AB, Özdemir S, et al. Are physicians less tolerant of treatment risks than patients? - Benefit-risk preferences in Crohn's disease management. 2008; under review.

[3] Hauber AB, Johnson FR, Mohamed AF, et al. The importance of modifying the course of Alzheimer's disease: older Americans' risk-benefit preferences for new treatments. Alzheimer Disease and Associated Disorders 2008; in press..

[4] Hauber AB, Johnson FR, Özdemir S. Using conjoint analysis to estimate healthy-year equivalents for acute conditions: An application to vasomotor symptoms. Value Health. 2008; in press

[5] Hauber AB, Johnson FR, Mohamed AF, et al. Preferences for modifying the course of alzheimer's disease: Risk-benefit tradeoffs and healthy-year equivalents. 2008; under review.

[6] Institute of Medicine (U.S.). Committee on the Assessment of the US Drug Safety System, Alina Baciu, Kathleen Stratton, Sheila P. Burke, Editors. The Future of Drug Safety: promoting and protecting the health of the public. 2007.

  Issues Index | 2008 Issues Index