Goal

Identify key contentious methodology issues in conducting health care pharmacoeconomic evaluations ­ using retrospective studies and claims data

Specific Objectives

• Identify and prioritize the key issues associated with pharmacoeconomic and outcomes research studies using retrospective and claims data
• Identify a plan of action to resolve these issues
• Recommend next steps

Co-chairs

• Renee Goldberg Arnold PharmD, President, Pharmacon International
• James G. Kotsanos MD MS, Director, Division of Global Health Economic Research, Eli Lilly and Company

Panelists

• Brenda Motheral RPh, PhD MBA, Associate Professor, University of Arizona
• Scott Ramsey MD PhD, Center for Cost & Outcomes Research, University of Washington, Seattle, WA
• William Crown PhD, Economist, MEDSTAT,
• Katherine Puder DrPh MS, President of QCORE LLC
• Mark Hornbrook PhD, Program Director, Kaiser Permanente
• Alan Wright MD MPH, VP & Chief Medical Officer, Advanced Paradigm Inc.
• Michael Murray PharmD MPH, Regenstrief Institute

Background and Context

Pharmacoeconomic analyses can be conducted within the context of clinical studies or using retrospective databases. Randomized Clinical studies (RCTs) and observational studies, specifically retrospective database analyses, answer different questions. For example, most randomized Clinical studies are designed to measure efficacy, not effectiveness. Existing databases can provide effectiveness and "real world" data. Cost-effectiveness analyses utilizing retrospective databases can provide real-time, relevant, and comprehensive decision support.

Retrospective analyses are relatively inexpensive to perform and can be done quickly. They are reflective of specific populations, that cannot be easily studied using RCTs, and for which data may be difficult to obtain. Relative to randomized controlled trials, retrospective studies tend to cover more realistic time frames, and are not constrained by the limitations of a set trial period. The perspective of a particular organization's experience can be obtained, and large samples can be surveyed. Usually, the sample sizes of retrospective databases are much larger than RCTs, enhancing their statistical power to detect important differences in outcomes. Retrospective databases encompass a wealth of variables, and analyses of these data can be used for benchmarking purposes, and for capturing real-world prescribing patterns. Notwithstanding the advantages of retrospective database analyses, there are challenges that face these analyses when used for health economic evaluations.

Problem Statement

Health care decision-makers require rapid access to information. The evidence that assists decision-makers to draw conclusions often has not been available. Both RCTs, and retrospective methods using existing databases, provide such information, and typically answer different questions. Most RCTs are designed to measure efficacy, not effectiveness. "Real world" data can be provided by database studies. The validity of retrospective analyses is often questioned due to the potential for selection bias, confounding factors, sponsorship, data quality and privacy issues. In addition, since there may be a time lag in the availability of information about new therapies and its incorporation into and availability from databases, the timeliness of the economic evaluations resulting from a retrospective analysis can be questioned.

Issues

Eight key issues were identified which are addressed in this document:

• What research questions can be answered by retrospective analyses?
• What data sources are available to answer these questions?
• How is cost-effectiveness measured using automated databases?
• How can data quality within a database be evaluated?
• What types of statistical methods can be utilized to control for treatment effects?
• What potential types of bias exist in retrospective database analyses?
• What alternative methods for assessing selection bias are available?
• How can transparency be ensured in retrospective database analyses?

1. What research questions can be answered by retrospective database analyses?

The types of economic studies that can be conducted using automated databases include cost- consequence, cost-effectiveness, and cost-of-illness analyses. Currently, cost-utility analyses are rarely addressed due to the lack of utility data captured in databases.

Of paramount importance to utilizing retrospective database analysis for health economic evaluations is the careful crafting of the research question. This question must be derived at the outset from the perspective of the appropriate parties, and such perspective may include society, the provider, the payer or the patient.

Crafting the question is critical to the success of a project since it drives all other aspects of the research project. It dictates:

• how the literature review is performed;
• the study objectives;
• the definitions of health outcomes and variables to be studied;
• the study design, including sample size;
• the time frame to be evaluated;
• data source(s), validation and analyses; and
• the budget.

2. What data sources are available to answer the question?

The types of data sources that can be utilized for health economic evaluations based on retrospective analysis include:

• electronic medical records (integrated modules of pharmacy, laboratory, and clinical databases),
• claims data such as that from managed care databases, the Health Care Finance Administration (HCFA), the Department of Defense, Veterans Affairs, self-insured employers, pharmacy benefit managers (PBMs);
• encounter data such as that from a staff/group model of health maintenance organizations (HMOs);
• expert opinion;
• results of published literature, such as meta-analyses;
• patient registries; and
• national survey databases, such as the Medical Expenditure Panel Survey (MEPS), formerly the National Medical Expenditure Survey (NMES).

3. How is cost-effectiveness measured using automated databases?

Cost-effectiveness studies require comparison of two or more competing therapeutic options. It is possible to measure cost-effectiveness using automated databases. First, one must determine if the particular database is appropriate to answer the question. Evaluation depends on the disease in question and availability of the outcome measures of effectiveness. For example, it would be inappropriate to select a hospital discharge database to evaluate lithotripsy use in kidney stones if the procedure is performed on an outpatient basis. Databases containing both outcome and cost information are required to perform cost-effectiveness analyses, including, for example, administrative claims as well as laboratory and prescription data. In some cases, linkage of databases might be appropriate to address some research questions. An example of database linkage for estimating cost of care for patients with cancer has been published (Potosky et al 666). Outcome measures can include, for example, diabetic ketoacidosis episodes or amputations avoided, which are coded using the International Classification of Disease (ICD-9) or Current Procedural Terminology (CPT-4) codes. Intermediate outcome measures can also be used, such as the percentage reduction of low-density lipoprotein-cholesterol rather than occurrence of a major cardiovascular event. Currently, databases do not usually contain quality of life (QoL) or utility data. However, QoL data are beginning to be collected, such as in the Patient Outcomes Research Teams (PORT) databases, in selected managed care databases, and in the managed care Health Plan Employer Data and Information Set (HEDIS) that captures data from questionnaires like the SF-36.

4. How can data quality within a database be evaluated?

Assessing the accuracy and completeness of the database (The MEDSTAT Group 94 665; Iezzoni 359) is integral to a research study based on automated database information. Data checks must be performed regularly and consistently. Issues to be considered include the following:

• Are there missing data elements, and if so, what percentage is missing?
• Is the cohort continuously tracked over the period of interest?
• Is it possible to trace services and diagnoses across healthcare settings (e.g. hospitals, nursing homes, and clinics)?
• Were all relevant diagnoses and procedure codes accurately recorded for the visit or episode of care?
• Are data recorded uniformly using widely accepted recording standards?
• Was the accuracy of the diagnostic and utilization records verified with chart reviews or benchmarking?
• Were logical consistency checks performed, such as searching the database for illogical matches, e.g. those between "hysterectomies" and "males"?
• Are there unique identifiers for each family member?
• Were events recorded when they actually occurred?
• Is population-denominator data available? That is, does the database contain enrollment information that enables the identification of individuals without healthcare utilization?

5. What types of statistical methods can be utilized to control for treatment effects?

With retrospective database studies in particular, it is important to control for as many confounding variables as possible. A need exists for statistical methods to control for these effects and implies that multivariate analysis should be performed whenever possible.

6. What potential types of bias exist in retrospective database analyses?

The potential areas for bias (Sackett 655) within retrospective database analyses form a long list, including, but not limited to the following examples.

• Selection bias. Individuals are not typically randomly assigned to health plans or treatments. Estimates of the effects and costs can be biased due to a correlation between unobserved factors associated with treatment selection and outcomes, such as baseline health status. Sample selection bias is often referred to by other names. For example, vintage bias, which is due to variation in physician training and practice styles or in availability of technologies resulting in confounding measures of costs and outcomes, is a specific type of selection bias.
• Bias from censoring of data. Bias can be introduced when the length of time that individuals are observed is correlated with their outcomes. For example, some studies may impose a minimum eligibility period that leads to the omission of subjects with short-term eligibility. If the reasons for the failure to meet the minimum eligibility criteria are correlated with utilization patterns, such as a death, then biased conclusion will result. Similarly, if the length of the observation period varies for individuals, then bias may be introduced because of failure to observe utilization that occurred after the observation period ended (e.g., There is less opportunity to observe service utilization patterns for patients treated with relatively new drugs.)
• Measurement error bias. There are numerous sources of error in measurement of data that can introduce bias. For example, specialists may tend to code diagnoses with more specificity than general practitioners. Similarly, fee-for-service providers may have an incentive to document diagnoses and services relative to at-risk providers who may have an incentive to minimize the burden of documentation. Recall bias is yet another form of measurement error. Recall bias refers to the tendency, on the part of respondents, to recall service utilization that occurred more recently with greater accuracy than service utilization that occurred in the more distant past.
• Misspecification bias. As with measurement error, there are a variety of forms of misspecification errors that can lead to biased estimates. These include omitted variables, incorrect function form, and using single equation models when a multi-equation model is more appropriate.
• Investigator bias/ obsolescence bias. In addition to the sources of bias just mentioned, there are sources of bias that may be introduced which are not directly a function of retrospective data themselves. These include investigator bias which arises when a researcher interprets findings in the context of preconceived viewpoints or adopts a study design (e.g., exclusion criteria) that biases the study results in the direction of the researcher's preconceived viewpoints. Investigator bias is often unintentional. Similarly, obsolescence bias may occur because medical technology used during the periods covered by a retrospective database is obsolete by the time the study is conducted.

Finally, bias is not the only statistical problem that may undermine the validity of inferences drawn from retrospective studies. Other problems include the correlation of error terms among respondents (autocorrelation), non-constant variance of error terms among respondents (heteroscedasticity), and high correlation among explanatory variables (multicollinearity).

7. What alternative methods for assessing selection bias are available?

Perhaps the most fundamental area in which bias occurs in a retrospective database analysis is during selection of study subjects. Whereas RCTs reduce sample selection bias through randomization by evenly distributing subjects among treatment arms, retrospective database analyses are non-randomized. Non-randomized studies that attempt to evaluate treatment outcomes have been widely criticized (CCOHTA Guidelines 94 96; Anderson 368) because unobserved variables might correlate with both treatment selection and outcomes. Such a correlation can result in erroneous inferences about the magnitude and statistical significance of treatment effects.

Alternative methods for assessing selection bias include using propensity scores, instrumental variables (IVs) and sample selection models; however, these methods may fail to fully control for selection bias.

1. Propensity score analysis has received growing attention as a methodology for reducing the bias due to inherent differences between treatment groups that go unobserved (Rosenbaum 84 659; Robins 92 660; Drake 95 358). Although the propensity score approach is non-parametric, Angrist (Angrist 97 66) has recently shown that propensity score analysis may be more closely related to sample selection models than previously believed.

2. The use of Instrumental Variables (IVs), in recent papers by McClellan (McClellan 94 662; McClellan 95 663), has been proposed to control for the confounding effects of unobserved variables. Instrumental variables are widely used by researchers to correct for a variety of statistical problems, most notably, simultaneous equation bias and errors in measurement (Kennedy 92 658; Green 93).

3. Sample Selection Models attempt to control the bias introduced by unobserved variables in treatment selection, which are also correlated with the outcome variable of interest. Sample selection models have seen wide use in the econometrics literature to study labor supply decisions, and to model the effectiveness of job training programs, housing programs, welfare experiments, and many others (Heckman 95 664). Very recently, these models have begun to find application in the health economics literature (Dowd 96 657; Hylan 97 656). Selection models are a special case of IVs and may require parameter estimates.

8. How can transparency be ensured in retrospective database analyses?

A great deal of consensus exists among the guidelines about transparency of assumptions and methods (Mullins 98 426). Full disclosure and detailed methodology when reporting study results of health economic evaluations is recommended by the Uniform Requirements for Manuscripts Submitted to Biomedical Journals (Uniform Requirements 393), the Task Force on Principles for Economic Analysis of Health Care Technology (Task Force 479), and the Pharmaceutical Research and Manufacturers of America's Methodological and Conduct Principles for Pharmacoeconomic Research (PhRMA principles 423).

One final point is that data privacy has become an increased focus of attention. The International Society of Pharmacoeconomics and Outcomes Research (ISPOR) has recently adopted the data privacy recommendations developed by the International Society for Pharmacoepidemiology (ISPE Data Privacy).

Recommendations and Next Steps

The credibility of retrospective database analyses in health economic evaluations must be enhanced by good research design, multiple and varied checks on data quality and attention to areas of potential bias in a given study. The following recommendations address the eight issues identified.

1. It is recommended that retrospective database analysis studies begin with a clear question and design, based on the International Society for Pharmacoepidemiology guidelines for good epidemiological practices (ISPE Guidelines 470).
2. Privacy of individuals must be ensured at all times in retrospective database analyses.
3. Techniques that exist to address shortcomings of retrospective data sets should be used.
4. Multivariate models should be subjected to extensive specification testing.
5. Examining age- or gender-adjusted utilization rates and annual per capita expense by payer, health plan, geographic region and country is suggested.
6. Administrative databases, frequently used for retrospective pharmacoeconomic studies, need to be augmented to include more clinical information (e.g., the results of lab tests, not just an indicator that a lab test was conducted).
7. Bias is a threat to the validity of inferences drawn from any retrospective and claims database analysis. When bias exists, either real or perceived, standard measures should be established to deal with all areas of potential bias. Ways to address specific biases include:
   • Selection Bias: modeling (e.g., Heckman approach, fixed-effects and random effects models), or propensity scores;
   • Measurement Error and Bias: modeling (e.g., bootstrap estimation), data imputation;
   • Mis-specification Bias and other violations of model assumptions: conduct and report specification error tests, select appropriate model and error distribution, or adjust observations to remove the specification error(s);
   • Investigator Bias: disclosure of conflict of interest, analysis assumptions, and study sponsor for investigator(s) and key staff;
   • Obsolescence Bias: use expert opinion to identify key innovative technologies and model them; look at time trends in key utilization parameters; estimate models for separate time periods;
   • Vintage Bias re: Human & Physical Capital: modeling; link with other data sources such as area resource file (HCFA);

Summary

Retrospective databases can be useful tools for health economic evaluations. They offer large populations using real-world information within rapid and realistic treatment periods, and can answer questions related to cost-effectiveness. Nevertheless, pharmacoeconomic studies based upon retrospective databases face a variety of threats to the validity of the inferences drawn from them. The credibility of retrospective database analyses in health economic evaluations must be enhanced by careful study design, elimination of bias, and reporting the results of these studies in a clear, believable, and transparent fashion.

Contact ISPOR @ info@ispor.org  |  View Legal Disclaimer
©2008 International Society for Pharmacoeconomics and Outcomes Research. All rights reserved under International and Pan-American Copyright Conventions. 
Website design by Eagle Systems USA, Inc.