Between Regulatory Rocks and Hard Places:
Quantifying Tolerance for Pharmaceutical Risks
(The following was presented during the Second Plenary Session, “Drug
Safety and Risk-Benefit Decision-Making,” at the ISPOR 13th Annual
International Meeting, May 6, 2008, Toronto, ON, Canada)
One of the reasons for the current interest in benefit-risk assessment is the
recurrence of embarrassing regulatory situations. A recent example involving
synthetic blood products was widely reported in the media. Sixteen studies
found that synthetic blood products were associated with tripling the risk of
heart attacks and increasing the chance of dying by 30%. Researchers publicly
questioned the competence of the Food and Drug Administration bureaucrats
who allowed additional testing of these products to continue. An FDA official
interviewed said that agency experts had carefully scrutinized the evidence
and were satisfied that the potential benefits outweighed the risks of further
testing and pointed out the huge unmet need for synthetic blood. A successful
blood substitute could save thousands of lives of trauma patients
(Washington Post, Tuesday, April 29, 2008).
FDA recently has begun to elicit the help of industry and academics in quantifying
benefit-risk tradeoffs in ways that might assist and inform regulatory decisions
and also help the agency explain decisions involving benefit-risk tradeoffs to the
health care community and the general public. There are a number of quantitative
approaches that are being explored but this article will describe two
of these methods: incremental net benefits and patient benefit-risk tradeoff
|Figure 1a. The Implicit Regulatory Benefit-Risk Threshold
||Figure 1b. Patient Benefit-Risk Thresholds
Suppose we had data on a large number of FDA decisions involving benefits
and risks where both the benefits and risks were directly comparable. Of
course, such data do not exist; however, if they did, we could identify the benefit-
risk combinations for drugs that were approved and drugs that were not
approved. The clustering of these outcomes would reveal the implicit regulatory
benefit-risk threshold that FDA actually is using. Figure 1a shows how this
threshold splits the benefit-risk space into two regions, the area where drugs
are approvable and the area where drugs are not approvable. Such information
would be particularly useful to sponsors of new pharmaceuticals.
Now suppose that we could estimate a patient benefit-risk threshold and compare
it to the FDA threshold. Threshold 0A indicates that patients largely agree
with FDA; that is, all the drugs that patients want approved are approved and
all the drugs that patients do not want approved are not approved. On the other
hand, benefit-risk threshold 0B indicates that that patients are more risk averse
than FDA, in which case some drugs are approved that patients do not want
approved. It is possible, of course, that patients' benefit-risk threshold is 0C,
and indicating patients are more tolerant of risks than FDA regulators. In this
case FDA does not approve some drugs that patients want approved.
The idea of a patient benefit-risk threshold is useful even in the absence of
enough information to specify FDA's regulatory threshold. Suppose we have
good evidence on the incremental effectiveness and incremental risk of a new
pharmaceutical. Should we approve this drug or not? My purpose is not to
suggest a decision rule to FDA, but it may be possible to determine whether
patients would want access to the drug or not. Point BR indicates the benefitrisk
combination for the drug in question relative to patients' benefit-risk tradeoff
curve. The tradeoff curve indicates the threshold below which perceived
benefits outweigh perceived risks. It also indicates the minimum benefit a treatment
would have to offer patients to induce them to accept the observed treatment
risk. This benefit is labeled the minimum acceptable benefit in Figure 2.
In this example the benefit is considerably larger than the minimum necessary
to get patients to accept the risk to which they actually would be exposed. The
net gain is shown as the net effectiveness benefit; that is, the amount of benefit
in excess of the minimum necessary you would have to offer patients.
|Figure 2. Four Quantitative Benefit-Risk Measures
We can identify similar constructs in the risk dimension of the diagram. Given
the actual effectiveness of this drug, what is the maximum acceptable risk
patients would tolerate before they would reject treatment? Point BR lies below
the benefit-risk tradeoff curve in that dimension as well. Thus approving the
drug also provides a net safety benefit to patients because its risks are less
than the maximum that patients are willing to accept.
The recurring question of how much of what kind of risk is acceptable arose
when VioxxĆ was withdrawn from the market a few years ago. Post-marketing
data revealed an elevated level of fatal and non-fatal myocardial infarctions
relative to naproxen. The reason that rofecoxib initially was approved was that
hundreds of cases per 10,000 patient years of dyspepsia, GI bleeds, and GI
perforations would be avoided by patients who used rofecoxib to control pain
instead of naproxen. Withdrawing approval meant that regulators were willing
to accept thousands of cases of milder adverse events to avoid a relatively
small number of heart attacks.
Lynd and colleagues have quantified this tradeoff in an unpublished study. The
incremental changes in health utilities are 0.059 for rofecoxib and 0.056 for
naproxen, respectively. Weighting the outcome gains and losses by the corresponding
utilities results in a net gain in quality-adjusted life years of about
0.001. That is the equivalent of about one additional day of good health for
patients on a rofecoxib regime instead of naproxen. Event-simulation indicates
there is a 98.3 percent chance that the incremental net benefit is greater than
zero. Setting aside other important considerations, if all drugs were approved
whose incremental net benefit were greater than zero, then on average patients
should be better off.
Incremental net benefits are a logical extension of existing epidemiology and
health-economics methods. The approach combines epidemiological data
with health-utility data to determine the net-benefit threshold. The problem with
this approach is that it often combines good epidemiology with health-utilities
of dubious quality. When estimated net benefits are as small as they are in the
rofecoxib study, the result is likely to be sensitive to measurement error in the
assumed utility values.
There are additional grounds for skepticism about using health utilities to compare
treatment benefits and risks. Even if QALYs are useful in health-economic
studies and for reimbursement decision making, there are good reasons why
QALYs are inappropriate for benefit-risk evaluations. The most serious problem
is that health utilities provide outcome weights, not measures of risk tolerance.
QALYs as conventionally measured require that we assume that patients and
FDA decision makers are risk neutral. That is, we have to assume that patients
and regulators do not buy insurance, because anyone who buys insurance is
averse to risk. They are willing to pay more in insurance premiums than the
expected payments they will receive from the insurance company to avoid having
the bear the risk of a large loss.
Figure 3 shows how we can elicit patient preferences that are relevant for benefit-
risk evaluations. Patients are asked to make choices among hypothetical
treatments. Each treatment description includes specified features such as
efficacy, mild to moderate side effects, and serious adverse-event risks. The
combinations of treatments that subjects evaluate are based on an experimental
design with known statistical properties. Analysis of the pattern of choices
observed under these controlled stimuli reveals the implicit preference weights
subjects used to guide their choices. Thus we obtain estimates of the relative
subjective importance of clinically relevant benefits and harms that also
account for patients' aversion to bearing risk.
|Figure 3. Example Benefit-Risk Tradeoff Question
Researchers have developed such preference-elicitation methods, variously
called conjoint analysis, discrete-choice experiments, or stated-choice surveys,
over the last 30 years for a wide range of applications. However, we have relatively
little experience in using these methods to elicit patients' benefit-risk
tradeoff preferences. The biggest challenge to obtaining valid and reliable preference
data involves coping with patients' innumeracy about low-probability
risks. We help patients evaluate risk magnitudes by showing them absolute
risks, relative risks, comparisons with more familiar risks of similar size, number
of patients out of 1000 who would be affected, and a graphical depiction
involving a 1000-square grid.
Figure 4 shows an example of an actual estimated benefit-risk tradeoff curve
for Crohn's disease . The risk is the risk of dying from progressive multifocal
leukoencephalopathy (PML) within ten years of beginning treatment. As
expected, patients are willing to accept higher PML risks for better treatment
benefits. This study was one of two patient benefit-risk tradeoff studies that
were submitted to FDA advisory committees in the last year. The same survey
was administered not only to patients, but also to parents of Crohn's children
and to physicians with some interesting results. Adult patients generally are
more tolerant of risks than parents and both of them are more tolerant of risks
than physicians. For example, for the lowest level of benefit -that is, an improvement
from mild Crohn's symptoms to remission - physicians were unwilling to
accept any additional treatment risk, but parents and patients are .
|Figure 4. Crohn's Disease Benefit-Risk Tradeoff Curve
We also have administered a benefit-risk tradeoff survey to 2,000 Americans
over the age of 60. Subjects evaluated hypothetical treatments that would alter
Alzheimer's disease (AD) progression. The tradeoff involved modifying the
course of the disease over a seven-year period but at a risk of dying from stroke
in the first year of treatment. Subjects were asked whether they were willing to
accept tradeoffs between clinically relevant changes in disease progression and
clinically relevant stroke mortality risks, not the instantaneous painless death
that health-utility subjects evaluate. For a treatment that would halt disease progressions
at the mild stage, our subjects, on average, were willing to accept an
almost one in three chance of fatal stroke in the first year of treatment .
Data on stated-preferences for benefit-risk tradeoffs support a number of useful
calculations [4-5]. Table 1 illustrates these possibilities using the AD estimates.
The best outcome offered survey subjects was mild cognitive impairment.
We can calculate the mild-year equivalent (MYE) for any diseaseprogression
profile. We do not obtain simple health-utility values that can be
added across durations and disease stages. Rather, preferences are nonlinear
and the importance of particular stage duration depends realistically on the
durations of other stages in the progression profile. In line (1) of Table 1, the
quality equivalence for 7 years with mild AD is simply 7 MYEs. The no-treatment
profile is equivalent to 4.9 MYEs. Thus the incremental benefit in line (2)
is 2.1 MYEs. The maximum acceptable risk of stroke mortality is 0.31 in line
(3). The minimum acceptable number needed to harm is simply the inverse of
the maximum acceptable risk, which is 3.2 for the 7-year mild profile. Number
needed to harm is often calculated; however, but without knowing what the
threshold value is, it is hard to know how to interpret a given result. Preference
data provide this missing information. For an assumed actual risk of 0.05, the net safety benefit is 0.26, measured as the difference between maximum
acceptable and actual risk. For the same assumed 0.05 risk, the minimum
acceptable benefit is 0.2 MYEs, which gives a net efficacy benefit of 1.9 MYEs
for the 7-year mild profile.
|Table 1. Alzheimer's Benefit-Risk Tradeoff Measures
An Institute of Medicine report  found that “in both the pre-approval and the
post-marketing setting, the risk-benefit analysis that currently goes into FDA
decisions appears to be ad hoc, informal, and qualitative”. Having better
evidence on patients' risk tolerance may be useful in responding to such
criticisms. Patients currently do have a voice in the regulatory process. A selfselected
group of patients often appear at the end of an advisory committee
meeting. Some tell stories about how a drug hurt them and some tell stories
about how a drug helped them. The stories are often emotional and heartrending.
It is unclear how such anecdotal evidence affects advisory-panel
recommendations. FDA has expressed an interest in quantifying such
evidence in representative samples of patients to determine patients' overall
tolerance for treatment risks associated with new pharmaceuticals*.
It is important to emphasize that these methods do not solve familiar problems
of uncertainty in the evidence, particularly the lack of reliable data on safety.
However, we have begun to think about how to incorporate uncertainty in
eliciting patients' benefit-risk preferences.
Ultimately regulatory decisions involve value judgments. Such decisions are
made on behalf of the rest of us about how much treatment risk we are willing
to accept as a society. These judgments often are very difficult and no quantitative
method will provide a decision rule to guide decision makers.
Nevertheless, better evidence about the preferences of the ultimate stakeholders,
the patients themselves, may be a useful complement to traditional forms
of evidence used to inform these difficult decisions.
*Note: This refers to remarks made by Robert Powell PharmD, at the Second
Plenary Session of the ISPOR 13th Annual International Meeting, May 6, 2008,
Toronto, ON, Canada.
 Johnson FR, Özdemir S, Mansfield CA, et al. Crohn's disease patients' benefit-risk preferences:
serious adverse event risks versus treatment efficacy. Gastroenterology 2007;133:769-79.
 Johnson FR, Hauber AB, Özdemir S, et al. Are physicians less tolerant of treatment risks than
patients? - Benefit-risk preferences in Crohn's disease management. 2008; under review.
 Hauber AB, Johnson FR, Mohamed AF, et al. The importance of modifying the course of
Alzheimer's disease: older Americans' risk-benefit preferences for new treatments. Alzheimer
Disease and Associated Disorders 2008; in press..
 Hauber AB, Johnson FR, Özdemir S. Using conjoint analysis to estimate healthy-year equivalents
for acute conditions: An application to vasomotor symptoms. Value Health. 2008; in press
 Hauber AB, Johnson FR, Mohamed AF, et al. Preferences for modifying the course of
alzheimer's disease: Risk-benefit tradeoffs and healthy-year equivalents. 2008; under review.
 Institute of Medicine (U.S.). Committee on the Assessment of the US Drug Safety System, Alina
Baciu, Kathleen Stratton, Sheila P. Burke, Editors. The Future of Drug Safety: promoting and protecting
the health of the public. 2007.