|
Issue II
METHODOLOGICAL ISSUES IN CONDUCTING PHARMACOECONOMIC EVALUATION - MODELING STUDIES
Goal
Identify key contentious methodology issues in conducting health care pharmacoeconomic evaluations - Modeling Studies
Specific Objectives
- Identify and prioritize the key issues associated with pharmacoeconomic modeling studies
- Identify a plan of action to resolve these issues
- Recommend next steps
Co-chairs
- Joel Hay PhD, Associate Professor, Department of Pharmaceutical Economics & Policy, University of
Southern California
- Joseph Jackson PhD, Executive Director, Outcomes Research, Bristol Myers Squibb
- Panelists:
- Bryan Luce PhD MBA, Senior Research Leader & CEO of MEDTAP International
- Jerry Avorn MD, Associate Professor, Harvard Medical School
- Talat Ashraf MD MS, Vice President, PPD Pharmaco & Informatics
Background and Context
The primary purpose of modeling is to inform the decision-making process (Medical Decision Making 473; Clinical
Decision Analysis 474). One considerable benefit of model formalization is that the uncertainties and assumptions in
this process are made explicit and transparent.
To estimate costs and outcomes, existing data is frequently insufficient to allow optimal health care decision-making.
Each type of data - retrospective, prospective, meta-analysis, expert opinion - has inherent strengths and weaknesses.
Good modeling practice incorporates the best available evidence from all possible sources into a set of explicit parameters.
Although randomized Clinical studies (RCT) are the gold standard for clinical research, they are not always the
best source of pharmacoeconomic and outcomes data. RCT-based data collection is often too costly, too time-consuming,
or otherwise not feasible. Sometimes modeling is the only accessible means to inform the clinical and healthcare
decision-making process (Gold 471). Although useful for determining efficacy, data from RCTs have significant limitations
that sharply reduce their usefulness for measuring the clinical outcomes and economic consequences of drug use in
actual populations, including:
- limited duration of follow-up, often only weeks or months;
- exclusion or under-representation of many types of patients, especially the vulnerable;
- sample sizes too small to detect infrequent events;
- atypical treatment settings, providers and subjects which may influence compliance, event rates and costs;
- no assessment of health care utilization in routine care.
Mathematical modeling allows a rational and scientific approach to overcoming these inherent limitations of RCTs,
using the best available evidence.
Problem Statement
Currently there are two major obstacles confronting modeling methodology currently: a) how to optimize the production
of useful information for health economic decision-makers, and b) how to encourage its acceptance and use of information.
Issues
There are seven key issues (areas of controversy) in modeling methodology
- Standardization
- Making choices
- Methodological development
- Extending Clinical studies and data issues
- Effectiveness measures
- Model validation
- Peer review
1. Standardization
Comparability is the essence that determines the preference of one intervention among alternatives; differences in
cost-effectiveness should reflect true differences and not unnecessary differences in method. This panel is not the first
to recognize the need for consensus on a set of standards that will promote comparability of studies.
When resources are limited, how are they allocated to programs important to the respective segments of society?
The Panel on Cost-effectiveness in Health and Medicine (Weinstein 113) recommended cost effectiveness analysis
(CEA) from a societal perspective for policy decision-making on healthcare resource allocation. They recommended a
standardized reference case analysis across all CEAs regardless of the intervention or outcome to provide the
methodological uniformity that supports comparability.
Besides health, real-world decisions include other considerations such as access to services, helping the most
vulnerable, and other values impacted by health decisions. Economic assessment is only one of the tools decision-makers must
use, and the information it provides must be weighed within the context of these other criteria. Values outside of health
care, which often influence choices about health services, cannot be quantified in CEA. Cost-benefit analysis (CBA),
cost-effectiveness analysis (CEA) and cost-consequence analysis (CCA) are complementary and the use of one does
not preclude the use of others in a study. Although Quality-adjusted Life Years (QALYs) have the advantage that they
measure changes in quality as well as quantity of life, as currently defined, they do not reflect perfectly everything about
health that matters to people, and perhaps never can.
The panel on Cost-Effectiveness in Health and Medicine (Weinstein 113) made recommendations concerning items
of intervention and outcome to be included in the numerator and the items for the denominator for a reference case
scenario. Most are based on reasonable facts, but some are arbitrarily chosen and recommended for a reference case
to maintain consistency across studies. At present, there is still no standard guide to good modeling practices that can
be used as a teaching or reference tool. No clear taxonomy of modeling techniques has been documented, and there are
no standardized presentation formats.
2. Making Choices
Beyond standardization, in each study a range of choices are made to fit the model to the research question.
Wherever choices are made, conservative values of all parameters should be chosen, and the base case should represent the
most plausible assumptions.
In deciding on perspective, societal perspective is preferable, and the societal perspective includes all relevant cost
and outcomes consequences. Certain options in decision-making will be cost-effective from societal perspective, and not
from the patient's perspective. Resource allocation decisions are based on cost-effectiveness evaluated at a specific
level. Decisions made at a higher level will affect the resource availability at lower levels. For example, at the societal
(governmental) level, policy decision on resource allocation is made based on the largest proportion of the public affected.
When individual perspective is examined, that segment of the population afflicted with a condition evaluated as secondary
for resource allocation purposes would need to seek resources elsewhere and their cost-effectiveness model would have
to take this into account. Many healthcare providers find the societal perspective irrelevant for their purposes; a great deal
of controversy continues concerning the use of a narrower perspective and whether it should only be presented
accompanied by the societal perspective.
Choice of the costs of an intervention from the governmental or societal perspective will take into account the
actual wholesale price (AWP) or discounted wholesale price (DWP). From the patient perspective there is a question of
which price to use. Are cost and price the same? Should the actual price paid or the discounted retail price, the
brand-name drug prices or generic drug prices be used? Each decision should be transparent and based on sound rationale.
Choice of assumptions should be realistic, reflecting available data. No model perfectly represents reality; its validity
rests on whether its assumptions are reasonable in light of the needs and purposes of the decision-maker and whether
after close examination its implications make sense. In making discounting decisions, both costs and benefits should be
done at the same rate, a standard of care should be used as the appropriate comparator, and the time horizon should be
the duration of time a drug can be expected to meaningfully impact the patient's health.
3. Methodological Development
A limitation of decision tree models is that they are not well suited to represent recurrent events over time (Evans
340; Beck 475). In chronic diseases, outcome events such as complications of the disease or its treatment, recurrence
of disease, and mortality, are confounded frequently during a lifetime, with probabilities that change with time, age,
and health status. Rather than model each event as a separate branch of a complex decision tree, health economic
modeling methodology has room for maturation and refinement to allow more efficient mathematical representations of such
events. Current alternatives in development include state-transition models, difference equation, deterministic models,
and stochastic models or discrete event simulations (O'Brien 476).
4. Extending Clinical studies and Data Issues
Some degree of modeling is usually necessary to assess clinical outcomes and economic consequences beyond
the necessarily parameters of a clinical trial, and modeling represents the only appropriate analytic approach to
estimate health care utilization, practice patterns and other costs associated with observations across defined geographic areas
or treatment settings, such as from country to country, health management organization to fee-for-service, or government
to private (Drummond 379).
Many cost and outcomes distributions violate standard normality assumptions, and outliers can have a major impact
on results. There are substantial problems with aggregation bias when costs and outcomes are averaged or combined
for large groups such as Disease Related Group (DRG) reimbursement levels or average length of stay. As far as
possible, data should be analyzed at the individual level for both costs and outcomes. Many still question whether RCT data
should be made available to support individual-level analyses.
Analysis of data from all study subjects is necessary to support interpretation of clinical trial data for
pharmacoeconomic modeling. However, although intent-to-treat analysis is important, it is not necessarily the only way to analyze RCT
data for modeling.
5. Effectiveness Measures
Several other issues arise in the estimation of effectiveness for modeling methodology: specification of survival
parameters; use of disease-specific or total mortality data; modeling patient characteristics; using models to vary
program parameters; use of modeling to address lead-time and length biases; estimating uncertainties. The techniques that exist
to deal with these issues are serviceable, but have not yet achieved state-of-the-art status or standardization.
Parameter uncertainty is generally handled on a qualitative basis with either univariate or multivariate sensitivity
analysis or max-min analysis, or quantified using statistical approaches such as the Delta method, joint confidence
intervals, bootstrapped estimates, or Monte Carlo simulation. No proven method exists to validate structure uncertainty in a
model due to either the parameter values assigned or to the mathematical form in which the parameter values are
combined, except to compute C/E ratio estimates for each alternative structural assumption and examine appropriateness of
the results. Even process uncertainty is an unknown. Would any two analysts follow the same model, or if the same
problem is posed to an analyst a second time (without awareness of the first result), would the same model be followed?
While it is generally agreed that proper application of multivariate sensitivity analysis is necessary, there is
ongoing controversy over its value.
6. Model Validation
As a mathematical device, and as a potentially important component of healthcare decision-making, credibility of
a pharmacoeconomic model rests on its validity. Besides an estimate of the range of uncertainty of its parameters,
each model should be shown to demonstrate face validity and predictive validity. Wherever possible, models should be
validated against other data sets.
7. Peer review of models
To ensure the quality and enhance the acceptability of pharmacoeconomic modeling, all models should undergo
systematic peer review before presentation. This could be a standardized audit of the structure, process and validity of the
model and would ensure that all salient model results are transparent. A technical peer review would necessitate passing
an electronic copy of the computer model to the reviewer(s) and raises questions about the handling of proprietary
property and confidentiality.
Recommendations
The following recommendations address the seven issues identified:
- Working towards general acceptance that modeling of both costs and effectiveness is a valid and often
essential method to inform health care decision-making will be necessary to establish modeling as an invaluable
healthcare decision-making tool.
- Because the usefulness of modeling studies is necessarily based on comparability, it is important to assemble
a consensus of opinion on standardized practices and policies.
- Once standardization has been achieved, a reference text of these practices should be prepared and disseminated.
- Pharmacoeconomic claims based on these generally accepted modeling approaches should be permitted by
regulatory agencies, and should always include transparency and appropriate disclaimers such as: "This economic
analysis is based on assumptions and simulations concerning the efficacy of [drug name] that meet FDA criteria for claims
of efficacy." Any model that relies on assumptions about a drug's efficacy that are not based on data from RCTs
must prominently disclose such limitation in any promotion.
- We recommend that as an independent professional association of pharmacoeconomic and outcomes
researchers, (ISPOR) take the initiative of assembling a balanced international panel of thought-leaders and end-users in the
field of modeling to develop a package of generally accepted modeling practices, building published upon previously work.
- Once these practices have been documented, the goal of ISPOR should be to encourage all stakeholders,
professional societies, manufacturing associations, journals, government agencies, regulatory agencies, payers and
health care providers, to accept these as standards and to endorse their use.
Summary
Mathematical modeling is a potentially invaluable tool to assist the health economic decision-making process. It serves
a unique methodological function. However, its practical value is currently limited by:
- Insufficient standardization;
- meager documentation of practices and policies;
- no systematic quality surveillance;
- a low level of acceptance by regulatory agencies and end users.
It is to be hoped that by supporting the development of standard practices, policy consensus, and a peer review
process the use and acceptability of health economic modeling will be potentiated.
Advisory Reports Main Page
|
