Goal

Identify key contentious methodology issues in conducting health care pharmacoeconomic evaluations ­ clinical studies

Specific Objectives

• Identify and prioritize the key issues associated with including pharmacoeconomic and outcomes research projects in clinical studies
• Identify a plan of action to resolve these issues
• Recommend next steps

Panel Co-chairs

• Margaret Healey PhD, Director of Clinical Research at the Institute for Research & Education, HealthSystems Minneapolis
• Patricia Deverka MD MS, Medical Director, Hastings Healthcare Group

Panelists

• Steven Fox MD, Center for Outcomes & Effectiveness Research, Agency for Healthcare Policy & Research
• Kathleen Gondek PhD, Director of Health Care Economics & Managed Care at Boehringer Ingelheim
• Barbara Edelman Lewis PhD, Vice President, Focus Managed Research Inc.
• Eva Lydick PhD, Director of Epidemiology & Applied Health Economics, SmithKline Beecham,
• Gurkipal Singh MD, Clinical Instructor from Stanford University Medical Center
• Robert Temple MD, Associate Director of Medical Policy, Food & Drug Administration
• Jeff Trotter MBA, President, ClinTrials Ovation

Background and Context

The pros and cons of various clinical study designs for economic evaluations have been described elsewhere (O'Brien 409; Haycox 39; Drummond 380; Drummond 379). The weaknesses of controlled Clinical studies that incorporate pharmacoeconomic evaluation include questionable choices of comparator therapies, inadequate sample sizes, limited duration of patient follow-up, the inability to separate protocol-driven costs from actual costs of care, and choice of outcome measures that may not be relevant to standard or usual care. These weaknesses threaten generalization of the results from controlled clinical studies. However, such trials have become the standard for establishing the efficacy and safety of new therapies. Study methods used to demonstrate cost-effectiveness often fall short of satisfying regulatory standards. Conversely, study methods used to demonstrate drug efficacy, such as randomized controlled trials (randomized Clinical studies), have been criticized for not adequately addressing cost-effectiveness issues in applied settings.

Consumers of pharmacoeconomics (in particular managed care decision-makers) are increasingly demanding evidence of the value of new interventions to inform formulary and treatment guideline decision-making. For the purpose of this paper, value is defined as outcome as a function of cost, but the relative importance of "cost" depends on the research question and the perspective of the target audience. Pharmacoeconomic data generated from randomized trials has the advantage of interpretability, statistical rigor, better control of bias and well-established and accepted methodology. Limiting pharmacoeconomic data to that derived from randomized Clinical studies, however, would limit the body of available information resulting in decisions based primarily on price and not value. In addition, relying solely on randomized clinical studies may limit the validity and generalization of the economic impact of a treatment in real world practice. Alternatively, results from studies other than randomized Clinical studies may be difficult to interpret and contain bias. Better understanding of the appropriate use of the different methods will increase the usefulness and validity of resulting pharmacoeconomic data.

Problem Statement

There is considerable overlap between acceptable methodologies for demonstrating the efficacy of drugs and those sanctioned by health economists for addressing questions of value, but the degree of overlap is a matter of fervent debate. Pharmacoeconomic methods used to assess cost alone or other measures of value often fall short of regulatory standards. Conversely, study methods used to demonstrate drug efficacy, such as randomized Clinical studies, are insufficient for addressing the question of value in other applied settings. To overcome the limitations of using clinical studies data for health economic evaluations, researchers, decision-makers, policy-makers and consumers should be well-versed in the appropriate use of clinical studies (both experimental and observational).

Issues

Four key issues were identified to be addressed in this document:

1. When should randomized clinical studies be the primary approach to assessing questions of value?
2. What modifications to randomized clinical studies would improve their usefulness as tools for health economic decision-making?
3. When should observational studies be used to assess questions of value?
4. What modifications to observational studies would improve their usefulness as tools for health economic decision-making?

1. When should randomized clinical studies be the primary approach to assessing questions of value?

Pharmacoeconomic data can be derived from randomized clinical studies, but these are not always first choice for a variety of reasons, both practical and scientific. These studies should be used as the primary approach only when certain criteria are met.

1. Are the potential results of sufficient importance to justify the expense of a randomized clinical trial? Not every question requires a randomized clinical trial to arrive at a credible answer. The "cost-effectiveness" of the study should be evaluated.
2. Is there reasonable likelihood of obtaining outcomes within a relatively short timeframe, or are intermediate endpoints acceptable? Principally because of their cost, randomized Clinical studies are best suited to studying acute outcomes or intermediate endpoints with well-established links to long-term outcomes. When intermediate outcome variables are used, the data supporting the level of certainty in the link between the intermediate variable and final outcomes needs to be disclosed.
3. Is there consensus about appropriate comparator(s)?
4. Are effect sizes predicted to be small (e.g., relative risk less than 2.0)?
5. Is randomization an acceptable strategy for the patients and the investigators?
6. Will compliance issues remain independent of the study outcomes?

2. What modifications to randomized clinical studies would improve their usefulness as tools for health economic decision-making?

To increase the overlap between acceptable randomized clinical studies and acceptable health economic assessments, rigorous application of good scientific and clinical practice principles are the foundation (Spilker 481). More specific recommendations include the following.

1. Conduct more randomized clinical studies within the settings where decisions will be made, such as within managed care organizations.
2. Adopt more naturalistic study designs including imitating usual care as much as possible, to increase generalizability of the results.
3. Minimize or liberalize randomized clinical trial inclusion and exclusion criteria.
4. Use active comparators rather than placebos, which are not usually considered relevant by decision-maker. When less robust measures of effectiveness are anticipated, as in antidepressant trials, a placebo comparator may also be needed to increase credibility of a cost-effectiveness assessment.
5. Test multiple doses of comparator.
6. Without compromising safety, reduce the burden of adverse event reporting.
7. If there is adequate power in sub-populations, design studies to answer secondary questions using nested designs.
8. Provide more variance information around economic variables to enhance the usefulness of data to decision-makers.
9. Design randomized clinical studies to capture more resource utilization data.

3. When should observational studies be used to assess questions of value?

Discussion here is limited to those observational studies that make use of primary data sources. Observational studies should be considered for use as a primary approach only under certain conditions. When the cost-effectiveness of performing a randomized Clinical studies questionable, adequate data may be derived from an observational study. In fact, the unacceptable cost of a randomized clinical trial may be created by some of the following conditions.

When the primary outcome is a rare event.

• When multiple comparators are considered desirable.
• When the primary outcome has a long term horizon.
• When the objective of a study is to evaluate a single intervention, such as the cost of treatment before and after introduction of a new therapy.
• When the intervention in question has been available for a long time.
• When conducting a randomized clinical trial is ethically questionable, such as when the magnitude of effect is very large or in the case of breakthrough treatments of life-threatening diseases.
• When randomization is not an acceptable option for patients, such as surgery versus pharmacotherapy or placebo-controlled trials in life-threatening illnesses.
• When an assessment of practice patterns is part of the evaluation.

4. What modifications to observational studies would improve their usefulness as tools for health economic decision-making?

While good clinical research practices obviously apply to the conduct of observational studies as well as to randomized clinical studies, observational studies are more frequently criticized for methodological inadequacies. Following good research practices such as those provided in the International Society for PharmacoEpidemiology recommendations (ISPE 470) are again the foundation. Specifically, the study should be hypothesis-driven, with the research questions specified at the outset, as well as the study design and methodology, analysis plan and how the results will be disseminated. As well, studies should be replicated in several settings to reinforce validity of the findings, and to determine their generalizability.

Recommendations

The following recommendations address the four issues identified:

1. The definition of usual care requires development. As trials adopt more naturalistic designs, the use of "usual care" as comparator will likely increase and comparability will be necessary.
2. New methods to account for protocol-related costs should be developed, especially for studies where these costs are not balanced across study groups.
3. Alternative methods to intent-to-treat analyses in usual care trials should be explored. Although the intent-to-treat analysis is important, there may be other research questions important to decision-makers that would benefit from alternate or additional analysis.
4. Problems of pooling economic data must be addressed, due to significant differences in clinical practice across settings and sites, especially in international studies.
5. Establish the range for "acceptable" levels of certainty of study results to inform value-based decision-making.
6. There is a need for improvement of statistical methods for adjusting for selection bias, for example, instrumental variables and propensity scores.
7. Better methods can be used to estimate variance around the components of value, such as bootstrapping and resampling techniques.
8. Systematic comparison should be made of randomized Clinical studies and observational studies on the same interventions.
9. Other disciplines (such as psychology, sociology, marketing research) should be delved for new approaches to methodology, particularly in the areas of data collection, analyses and instrumentation.
10. Use large, simple trials such as early Phase IV studies, to measure resource utilization.
11. Develop better methods to measure direct medical costs not routinely captured, such as nursing time and telephone care (OMERACT panel).
12. Develop better methods to measure relevant indirect costs such as caregiver time or lost productivity (ARAMIS).
13. As electronic medical records evolve and expand, encourage inclusion of standardized outcome measures.

The timeliness of these recommendations will be relative to the evolution of the research (user) with respect to pharmacoeconomic measurement and application.

Summary

There is agreement that useful pharmacoeconomic data can be derived from both randomized Clinical studies and observational studies. The process of randomization does lend some credibility to studies, although this may not always be perceived as necessary by the end user. The most important factors for a researcher to consider in designing a clinical trial are knowing the precise question(s) that the study is required to answer and knowing the informational needs of the target audience. Over the long term, it is recommended that effort be directed towards more consensus and a greater degree of standardization of definitions and methodologies for assessing health economic questions through clinical studies.

Contact ISPOR @ info@ispor.org  |  View Legal Disclaimer
©2010 International Society for Pharmacoeconomics and Outcomes Research. All rights reserved under International and Pan-American Copyright Conventions. 
Website design by Eagle Systems USA, Inc.