The following term, and many others, can be found in HEALTH CARE COST, QUALITY, AND OUTCOMES: ISPOR BOOK OF TERMS edited by Marc L. Berger MD, Kerstin Bingefors PhD, MSc, Edwin Hedblom PharmD, Chris L. Pashos PhD, and George Torrance PhD.

Pharmacoeconomics

Brief Definition
Pharmacoeconomics is the scientific discipline that evaluates the clinical, economic and humanistic aspects of pharmaceutical products, services, and programs, as well as other health care interventions to provide health care decision makers, providers and patients with valuable information for optimal outcomes and the allocation of health care resources. Pharmacoeconomics incorporates health economics, clinical evaluations, risk analysis, technology assessment, and health-related quality of life, epidemiology, decision sciences and health services research in the examination of drugs, medical devices, diagnostics, biotechnology, surgery, disease-prevention services.

Explanation
Pharmacoeconomics is a collection of descriptive and analytic techniques for evaluating pharmaceutical interventions in the health care system. Pharmacoeconomic techniques include cost minimization, cost effectiveness, cost utility, cost benefit, cost of illness, cost consequence, and any other economic analytic technique that provides valuable information to health care decision makers for the allocation of scarce resources. Pharmacoeconomics is often referred to as health economics (see terms: Cost-Benefit Analysis, Cost-Consequence Analysis, Cost-Effectiveness Analysis, Cost-Minimization Analysis, Cost-Utility Analysis, Health Economics) or health outcomes research, especially when it includes nonpharmaceutical courses of therapy or preventive strategies, such as surgical interventions or screening techniques.

Value and Use
Several potential uses for pharmacoeconomic analyses are in pharmaceutical reimbursement, price negotiations, formulary discussions, clinical practice guideline developments, and communications to prescribing physicians. In recent years, pharmacoeconomics has grown rapidly because its core subject, cost-effectiveness analysis, is easy to apply and has powerful applications for health care decision making in both the public and private sectors. Recent research from the Tufts Center for the Study of Drug Development suggests that the demand for pharmacoeconomic analyses conducted by the pharmaceutical industry is likely to grow substantially from the present spending (average 1% of pharmaceutical research and development cost) in the near future. In Australia and Canada, the acceptance of new chemical entities in the national or provincial formularies depends on pharmacoeconomic studies. Different countries have different approaches to making pricing and reimbursement decisions using formal pharmacoeconomic evaluations. In a recent study based on nine European countries (Finland, France, Germany, Norway, Austria, The Netherlands, Portugal, Spain, and the United Kingdom), one-third of all respondents (government agencies, physicians, hospital pharmacists, hospital managers, sickness funds, and the pharmaceutical industry) across all countries stated that they have used results from health economics studies for decision making (Hoffman 2000). In the United States, pharmacoeconomic analyses are not required for the submission of a new drug to the Food and Drug Administration (FDA). However, the potential value of pharmacoeconomics in drug coverage decisions by private and public health plans is promising. In a recent survey, 88% of the 24 managed care decision makers from 15 companies across the United States indicated that pharmacoeconomic information is useful. Managed care plans are increasingly requiring or encouraging the submission of pharmacoeconomic information (sometimes in “dossier” format) by pharmaceutical companies seeking coverage of or reimbursement for new therapies.

Issues
The credibility of pharmacoeconomics lies in developing studies in accordance with generally applicable standards of analysis and interpretation. Then users can translate pharmacoeconomic research findings into practices to ensure that decision makers allocate scarce health care resources wisely, fairly, and efficiently. Several groups in the United States (an expert panel commissioned by the United States Public Health Service, Center for Disease Control and Prevention; and the Division of Drug Marketing, Advertising, and Communication [DDMAC] of the FDA) have developed guidelines for proper conduct of pharmacoeconomic studies. The Academy of Managed Care Pharmacy has developed guidelines for formulary submission (AMCP 2002). This guideline contains extensive information on which pharmacoeconomic data should be included for consideration of drug coverage decisions. Individual countries, including Australia, Canada, Italy, Spain, The Netherlands, Switzerland, Germany, France, and the United Kingdom have developed their own sets of guidelines. The Pharmaceutical Research and Manufacturers of America (PhRMA) also developed a set of voluntary principles to guide industry members in conducting pharmacoeconomic studies that minimize bias and ensure transparency. The future success of pharmacoeconomic analyses relies on the continued accumulation and dissemination of robust information that can be utilized by different users under different circumstances.

Bibliography

1. Academy of Managed Care Pharmacy (AMCP). Format for Formulary Submissions Version 2.0. October 2002. http://www.fmcpnet.org/data/resource/formatv20.pdf. Accessed August 27, 2003.
2. Fry RN, Avey SG, Sullivan SD. The academy of managed care pharmacy format for formulary submissions: An evolving standard—A foundation for managed care pharmacy task force report. Value Health 2003;6:505–521.
3. Grizzle AJ, Olson BM, Motheral BR, et al. Therapeutic value: who decides? Pharmaceutical Executive. 2000;84–90.
4. Hoffman C, Graf von der Schulenburg JM. The influence of economic evaluation studies on decision making: a European survey. The EUROMET Group Health Policy. 2000;52:179–192.
5. Pashos CL, Klein EG, Wanke LA. ISPOR LEXICON First Edition. Princeton: International Society for Pharmacoeconomics and Outcomes Research; 1998.

The following term, and many others, can be found in HEALTH CARE COST, QUALITY, AND OUTCOMES: ISPOR BOOK OF TERMS edited by Marc L. Berger MD, Kerstin Bingefors PhD, MSc, Edwin Hedblom PharmD, Chris L. Pashos PhD, and George Torrance PhD.

Outcomes Research

RELATED TERMS

• ECHO Model
• Consequences
• Clinical Intermediary
• Clinical Outcomes
• Humanistic Intermediary
• Humanistic outcomes
• Costs
• Economic outcomes
• Treatment modifiers

Brief Definition
Outcomes research is the scientific discipline that evaluates the effect of health care interventions on patient health status, often using the ECHO model involving economic, clinical, or humanistic outcomes. Outcomes research is generally based on the conceptual framework that evaluation of treatment alternatives involves the simultaneous assessment of multiple types of outcomes that are often disease-related.

Explanation
A diagram of outcomes research–the ECHO model (economic, clinical, and humanistic outcomes) is shown below. Outcomes research–the ECHO model includes the following terminology.

• Consequences: A general term used to refer to any effect related to the alternatives modeled (see term: Cost-Consequence Analysis).
• Clinical intermediary: Measurements of a patient’s physical or biomedical status used as a surrogate for or to infer the degree of disease (e.g., blood pressure, forced expiratory volume).
• Clinical outcomes: Medical events that occur as a result of disease or treatment (e.g., stroke, disability, hospitalization) (see terms: Clinical Trial, Epidemiology, Pharmacoepidemiology).
• Humanistic intermediary: Factors that affect the formation of patients’ opinions about the effects of disease or treatment on their lives and wellbeing (e.g., values, norms, perceptions) (see terms: Health-Related Quality of Life, Patient-Reported Outcomes).
• Humanistic outcomes: Patient self-assessment of the impact of disease or treatment on their lives and well-being (e.g., satisfaction, quality of life) (see terms: Health-Related Quality of Life, Patient-Reported Outcomes).
• Costs: Direct medical, direct nonmedical, and indirect costs associated with the treatment alternatives evaluated (see term: Cost – Health Economics).
• Economic outcomes: Direct and indirect costs compared to consequences of medical treatment alternatives typically expressed as ratios of cost to consequence (e.g., cost minimization, cost-effectiveness, cost utility, and cost-benefit ratios) (see terms: Cost-Benefit Analysis, Cost-Effectiveness Analysis, Cost- Minimization Analysis, Cost-Utility Analysis).
• Treatment modifiers: Factors that may alter intermediaries or outcomes associated with treatment alternatives (e.g., side effects, compliance) (see term: Compliance).

Figure reprinted from CM Kozma, CE Reeder, RM Schulz. Economic, clinical, and humanistic outcomes: A planning model for pharmacoeconomic research. Clin Ther. 1993;15:1121–32.

In outcomes research, there is a distinction between outcomes (i.e., end results) and intermediaries (e.g., surrogate endpoints). It is an important distinction because it forces reflection on differences between “end results” and intermediaries. For example, blood pressure is a clinical intermediary in the treatment of hypertension while stroke or myocardial infarctions are clinical outcomes (i.e., end results). Distinction between intermediaries and outcomes is also important because it has implications for measurement such as reliability, validity, and duration of a study. While most researchers believe that there is a strong relationship between blood pressure and stroke, intermediaries for other diseases might not be as well established. The timing of data collection and size of sample required is also affected by choice of outcome or endpoint. Studying stroke rates requires much longer periods of time and much larger samples than does measuring blood pressure. The distinction between outcomes and intermediaries may also vary depending on the perspective taken for the analysis. For example, if a pharmacist is implementing a counseling program, compliance would be viewed as an “end result” from the pharmacist’s perspective. However, from the perspective of a health care organization studying stroke, compliance may be viewed as an intermediary while the outcome of interest is survival.

When planning outcomes research, a societal perspective should be used for identification of consequences, even if this perspective is later restricted for a particular application. During the outcomes research planning process, specification of the disease or condition and selection of relevant alternatives for comparison is important. All consequences related to the treatment alternatives should then be listed for the patient populations of interest.

Value and Use
The outcomes research ECHO model represents only one of many available outcomes research models. The outcomes research ECHO model has been used extensively for teaching medical professionals about designing and reviewing outcomes research studies. The value is in using it prior to development of a study or evaluation of a body of literature. A priori specification of the factors that are important to the evaluator can lead to better decision making.

Bibliography

1. CM Kozma, CE Reeder, RM Schulz. Economic, clinical, and humanistic outcomes: A planning model for pharmacoeconomic research. Clin Ther. 1993;15:1121–1132.

The following term, and many others, can be found in HEALTH CARE COST, QUALITY, AND OUTCOMES: ISPOR BOOK OF TERMS edited by Marc L. Berger MD, Kerstin Bingefors PhD, MSc, Edwin Hedblom PharmD, Chris L. Pashos PhD, and George Torrance PhD.

Health Technology Assessment (HTA)

RELATED TERMS

• International Network of Health Technology Assessment Agencies (INAHTA)

Brief Definition
Health technology assessment (HTA) is a form of policy research that examines short- and long-term consequences of the application of a health care technology.

Explanation
The goal of HTA is to provide policymakers with information on policy alternatives. Health care technology within the concept of HTA is defined broadly as consisting of

• Drugs (e.g., aspirin, antibiotics, beta-blockers);
• Biologics (e.g., vaccines, blood products, biotechnology-derived substances);
• Devices, equipment, supplies (e.g., cardiac pacemaker, CT scanner, surgical gloves);
• Medical and surgical procedures (e.g., acupuncture, cancer chemotherapy, cesarean section);
• Support systems (e.g., drug formulary, clinical laboratory, patient record system); and
• Organizational, delivery, and managerial systems (e.g., emergency medical system, immunization program, disease management program, health care payment system).

For any given technology, properties and impacts assessed may include technical properties (this is particularly germane for sophisticated equipment), evidence of safety, efficacy (including patient-reported outcomes), real-world effectiveness, cost, and cost-effectiveness as well as estimated social, legal, ethical, and political impacts. Thus, HTA is conceived as being much broader than is typically true of health and economic outcomes research of a health care technology.

HTA is commonly performed at a national or multisystem level by governmental, quasi-governmental, or nonprofit organized groups and individuals to inform health care policies or decisions, such as to:

• Support decisions by industry about technology development and marketing,
• Advise regulatory agencies about allowing the marketing or use of a technology,
• Advise health plans and other payers concerning coverage and payment for a technology,
• Advise clinicians and patients about the appropriate use of a technology,
• Help managers of hospitals and other providers make decisions about acquiring a technology, and/or
• Support decisions by financial groups (e.g., venture capitalists) about investment in new technologies and companies.

Value and Use In the broad sense depicted above, HTA can be (and is at times) employed by any person or groups. However, it was originally conceived as an adjunct to governmental policy decision making. This was exemplified by the US Congress’s former Office of Technology Assessment, credited by many with having introduced HTA in a formal sense to United States society and indeed the world. Today, governments around the world have created internal agencies authorized to include a broad HTA function. The International Network of Health Technology Assessment Agencies (INAHTA) is an association of these governmental agencies. The respective governments look to these agencies to guide them in policy decisions such as those enumerated above.

There are also private HTA agencies, especially in the United States. Examples are Blue Cross Blue Shield’s Technology Evaluation Center (TEC), ECRI, HAYES Inc, and the University HealthSystem Consortium. Although these private groups tend to practice a narrower brand of HTA than their public counterparts, their reports are viewed by their respective constituents as useful guidance for technology adoption and reimbursement decision making.

Issues
A key issue associated with HTA, at least as it has been practiced at the governmental level (which has been its main focus worldwide) has been the lack of collaborative relationships between the innovators, producers, and advocates of health technology on the one hand and the governmental policymakers, on the other hand, who are the consumers of HTA and whose charge is to decide on societal adoption, use, and reimbursement of health care technologies.

Bibliography

Banta HD, Luce BR. Health Care Technology and its Assessment: An International Perspective. Oxford, UK: Oxford University Press, 1993.

The following term, and many others, can be found in HEALTH CARE COST, QUALITY, AND OUTCOMES: ISPOR BOOK OF TERMS edited by Marc L. Berger MD, Kerstin Bingefors PhD, MSc, Edwin Hedblom PharmD, Chris L. Pashos PhD, and George Torrance PhD.

Evidence-Based Medicine

RELATED TERMS

• Cochrane Collaboration
• Centre for Reviews and Dissemination
• Evidence-Based Practice Centers

Brief Definition:
Evidence-based medicine has been defined as an approach to health care practice in which the clinician is aware of the evidence in support of his or her clinical practice and of the strength of that evidence and is then able to apply that knowledge in clinical practice. Evidence-based medicine, therefore, consists of clinical expertise and patient preferences combined with critical appraisal of clinical research, with the goal of providing optimal individual patient care. Optimal care thus takes into account patient outcomes and the relative efficiencies among competing alternatives, as demonstrated in the medical literature. This approach to patient care demands that the clinician’s expertise and the appraisal of the clinical evidence base be current and up to date.

Explanation
Historically, clinical care has been provided based on clinical expertise and some notional review of the clinical literature. Today, there is an ever increasing burden on clinicians to keep up with relevant literature and advances in their respective fields as both the body of medical literature and clinical advances continue to expand and accelerate. Additionally, health care delivery systems are under pressure to provide state-of-the-art medical interventions, but in the most efficient manner. To help balance these twin burdens, evidence-based medicine (EBM) is being increasingly pursued as a rational approach to patient care.

EBM requires that clinicians have access to critical, unbiased reviews of currently available evidence. One especially visible example of early efforts to review literature in a systematic way is the Cochrane Collaboration. This effort was begun in response to the suggestion by a British epidemiologist, Archie Cochrane, that critical, systematic and up-to-date reviews of all relevant randomized controlled trials in health care were needed to fully realize the potential of medical research for patient care. Cochrane Centres across the world engage in the production of Cochrane reviews according to a specific methodology. Many other research bodies conduct rigorous systematic reviews, among which, for example, are the Centre for Reviews and Dissemination (CRD) at the University of York in the United Kingdom and Evidence-Based Practice Centers (EPCs) funded by the Agency for Health care Research and Quality (AHRQ) in the United States.

Value and Use
The need for and value of EBM is increasingly being recognized. It is appearing more frequently in medical school curricula and postgraduate workshops, and medical journals are now devoted to EBM. The academic courses include teaching critical appraisal of the literature so that clinicians become savvier regarding individual published studies and reviews. Also, since the late 1980s, there has been a large increase in the number of written clinical practice guidelines produced that use the principles of systematic reviews together with expert clinical opinion to help inform patient-level decision making. Like EBM as used in clinical practice, clinical practice guidelines take into account the knowledge gained through experience of clinicians and typically reflect local or regional practice. Systematic reviews focus less on experiential evidence and rely instead on evaluation of clinical trials and aggregation of their results to make conclusions concerning the evidence. The goal of EBM is to enable physicians to obtain and take unbiased sources of information into the patient encounter and use them in the clinical decision-making process. It is a basic tenet of EBM that patient specific characteristics and preferences be explicitly considered.

Issues
Although the goal of EBM reviews and guidelines is to be timely, current, and unbiased, EBM in actuality is also sometimes criticized for not having those characteristics. Sometimes, reviews and guidelines are criticized for not being timely and not reflecting current knowledge. As well, controversy exists as to what clinical literature deserves to be included in the reviews and depended upon for the guidelines. For example, some argue that only randomized controlled trials (RCTs) should be included, while others maintain that such trials suffer from external generalizability and other issues and can be supplemented and complemented by a variety of outcomes studies, especially in the absence of trial data. Similarly, some maintain that only indexed literature has the quality to be included, while others maintain that the “gray” (nonindexed) literature may be included if it meets predetermined criteria for inclusion. Of course, that increases the intensity and scope of EBM reviews, and their concomitant expense. Overall, however, it is critical that such reviews be of the highest quality. If they are not, the field risks being perceived as simply another cover for decision makers merely trying to reduce or contain costs with inadequate evidence. This concern highlights the importance of an unbiased treatment of the evidence base. Impartial reviewers and adherence to standard procedures can help mitigate these concerns. It should also be noted that the most expensive treatment may in fact be the most efficient in improving patient outcomes.

Bibliography

1. Guyatt GH, Haynes BR, Jaeschke RZ, et al. For the Evidence-Based Medicine Working Group. Users’ guides to the medical literature: XXV. Evidence-based medicine: principles for applying the users’ guides to patient care. JAMA. 2000;284:1290–1296.
2. Sackett DL, Rosenberg WMC, Gray JAM, et al. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312:71–72.
3. Sackett DL, Straus SE, Richardson WS, et al; Meyer HS, ed. Evidence-Based Medicine: How to Practice and Teach. General Practice. 1995;45(8398):506.

The following term, and many others, can be found in HEALTH CARE COST, QUALITY, AND OUTCOMES: ISPOR BOOK OF TERMS edited by Marc L. Berger MD, Kerstin Bingefors PhD, MSc, Edwin Hedblom PharmD, Chris L. Pashos PhD, and George Torrance PhD.

Drug Formulary

RELATED TERMS

• Drug Formulary System
• Closed Formulary
• Open-Preferred Formulary
• Tiered Copayment
• Open-Passive Formulary
• Formulary Management

Brief Definition
A drug formulary is a continually updated list of medications that is preferred for use by a health system and will be dispensed through participating pharmacies to covered persons. The formulary is the product of the drug formulary system.

A drug formulary system is an ongoing process whereby a health care organization, through its physicians, pharmacists, and other health care professionals, establishes policies on the use of drug products and therapies, and identifies drug products and therapies that are the most medically appropriate and costeffective to best serve the health interests of a given patient population.

Explanation
Health management organizations have used the formulary process since the 1970s. The formulary is typically the product of the organization’s pharmacy and therapeutics committee review and evaluation process that seeks to guide physician prescribing through the inclusion of only selected products from the various drug classes. Drugs are usually chosen for formulary inclusion based upon an evaluation of published literature around the efficacy, safety, and tolerability of a drug compared with others (if applicable) within the same class. A pricing contract is then negotiated with the product manufacturer and factored into the decision. A formulary is typically classified as closed (or partially closed), open-preferred or open-passive. Closed formulary indicates that one or more classes of drugs are closed to only certain products. Only drugs included on the formulary in a closed class are covered by the drug benefit. Open-preferred formulary indicates an incentive-based system to encourage the use of preferred agents within the class. These incentives include lower copayments for the patient, academic detailing, and requests (usually in the form of a letter) to the physician to reconsider the choice of nonformulary drugs. Under this style of formulary, a tiered copayment approach is often employed. Under the tiered copayment system, the patient pays less for generic and progressively more for brand name and nonformulary drugs. A tiered formulary typically contains three to five tiers. An open-passive formulary indicates that few incentives are provided to encourage or change prescribing nonformulary drugs.

Formulary system decisions are based on scientific and economic considerations that achieve appropriate, safe, and cost-effective drug therapy. A formulary often contains information about available formulations and dosing but does not usually contain recommendations about the therapeutic use of the drugs.

Value and Use
When structured appropriately, the formulary can be a powerful tool in directing physicians to prescribe the most efficacious and cost-effective drugs to patients. The formulary management process is dynamic, with the formulary under constant review and revision as new products come onto the market and new data surface on existing drugs. A well designed and well-managed pharmacy and therapeutics committee can serve as an excellent source of expertise and guidance for the prescribing physician.

Moreover, several organizations have begun building outcomes criteria into the product review and evaluation process. These criteria can include: effectiveness (versus efficacy), quality of life, patient satisfaction, total cost of care (versus simply acquisition cost of the product), and workplace productivity.

Issues
Many physicians see the formulary process as restrictive and cost-driven. In addition, because most physicians participate in multiple health plan provider networks, they must deal with the differing formulary of each of the entities. This can often lead to confusion, errors, and inefficiencies within the physician’s practice that can undermine the physician’s willingness to follow the formulary. It is only with the highest degree of standards and communications that these obstacles can be overcome.

Bibliography

1. Academy of Managed Care Pharmacy. A Format for the Submission of Clinical and Economic Evaluation Data in Support of Formulary Consideration by Managed Health Care Systems in the United States. Alexandria, Va: Academy of Managed Care Pharmacy; 2000.
2. The Coalition Working Group: Cahill JA, Fry R, Cranston JW, Zellmer WA, et al. Principles of a Sound Drug Formulary System. Formulary Management-Endorsed Document. Bethesda, Md: ASHP; 2000.
3. Goldberg RB. Managing the pharmacy benefit: the formulary system. J Manag Care Pharm. 1997;3:565–573.
4. Regence Washington Health Pharmacy Services. Guidelines for the Submission of Clinical and Economic Data Supporting Formulary Consideration. Seattle, Wash: Regence Washington Health; 1997.

The following term, and many others, can be found in HEALTH CARE COST, QUALITY, AND OUTCOMES: ISPOR BOOK OF TERMS edited by Marc L. Berger MD, Kerstin Bingefors PhD, MSc, Edwin Hedblom PharmD, Chris L. Pashos PhD, and George Torrance PhD.

Patient-Reported Outcomes (PRO)

Brief Definition
Patient-reported outcomes (PRO) is an umbrella term that includes outcome data reported directly by the patient. It is one source of data that may be used to describe a patient’s condition and response to treatment. It includes such outcomes as global impressions, functional status, well-being, symptoms, healthrelated quality of life (HRQL), satisfaction with treatment, and treatment adherence (see terms: Compliance, Health-Related Quality of Life).

Explanation
PRO is a term that has recently come into use. It was first proposed February 2001 at a meeting of the PRO Harmonization Group (a working group composed of members of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), International Society for Quality of Life Research (ISOQOL), the Pharmaceutical Research and Manufacturers of America Health Outcomes Committee (PhRMA-HOC), and the European Regulatory Issues on Quality of Life Assessment (ERIQA). While there has been a large increase in interest in understanding the patient’s perspective of disease and treatment, the lack of a clear framework and agreed-upon terminology often has resulted in communication difficulties among researchers and with regulatory authorities. One approach to conceptualizing data collected in clinical trials is to consider the source of the data. There are several potential sources of data to evaluate the safety and efficacy of a new drug:

• Patient-reported outcomes (e.g., global impression, functional status, HRQL, symptoms)
• Caregiver-reported outcomes (e.g., dependency, functional status)
• Clinician-reported outcomes (e.g., global impressions, observations, tests of function)
• Physiological outcomes (e.g., FEV1, HbA1c, tumor size)

Each source serves as an umbrella term for the different types of data that may be provided by each source. The different sources have been shown to provide unique information regarding the efficacy of a therapy (e.g., Surarez- Almazor 2001; Rothman et al. 1991). The PRO component of the framework is based on the following assumptions:

• The patient’s subjective experience provides a unique and valuable contribution to the drug development process.
• The information provided by the patient is inherently subjective.
• Scientific methods for assessing subjective outcomes (e.g., psychometrics and utility measurement) are well developed and provide the basis for PRO assessment.
• The design of PRO studies should follow the rules defined for other types of clinical trials (e.g., clear specification of hypotheses, standard methods of analysis and interpretation).

Value and Use
The proposed framework recognizes the unique value of data from each source including patients. The appropriateness of using data from each source is context specific; that is, in some cases a PRO may be the best or only source of data while in other cases clinician-reported or multiple sources of outcome data may be most appropriate. This framework also encourages researchers to specify the type of PRO being assessed (e.g., symptoms, HRQL, functional status). This specificity is expected to facilitate communication among researchers and regulatory authorities.

Issues
This framework for clarifying patient outcomes assessment is proposed for use within the context of the drug development and regulatory process. Appropriateness of use beyond this context must be evaluated separately. It is recognized that PRO assessment is an evolving field; thus the proposed framework is intended to foster further research and enhance communication rather than be prescriptive.

Bibliography

1. Acquadro C, Berzon R, Dubois D, et al. Incorporating the patient’s perspective into drug development and communication: An ad hoc task force report of the Patient-Reported Outcomes (PRO) Harmonization Group meeting of the Food and Drug Administration, February 16, 2001. Value Health. 2003;6:522–531.
2. Rothman ML, Hedrick SC, Bulcroft KA, et al. The validity of proxy-generated scores as measures of patient health status. Med Care. 1991;29:115–124.
3. Surarez-Almazor ME, Conner-Spady B, Kendall CJ, et al. Lack of congruence in the ratings of patients’ health status by patients and their physicians. Med Decis Making. 2001;21:113–121.

The following term, and many others, can be found in HEALTH CARE COST, QUALITY, AND OUTCOMES: ISPOR BOOK OF TERMS edited by Marc L. Berger MD, Kerstin Bingefors PhD, MSc, Edwin Hedblom PharmD, Chris L. Pashos PhD, and George Torrance PhD.

Real-World Data

Brief Definition:
The ISPOR Real-World Data Task Force considered “real-world data” as data used for clinical, coverage, and payment decision-making that are not collected in conventional randomized controlled trials (RCTs).  Real-world data could be characterized in a number of different ways, e.g., by type of outcome, by location in a hierarchy of evidence, or by type of data source.  The Task Force examined all three of these dimensions.  Real-world data issues apply globally to all countries as well as to all types of interventions, including drugs, devices, procedures, and health programs.

Explanation
Decision makers rely on multiple sources of real world data and integrate or synthesize them in some fashion.  While RCTs remain the gold standard for demonstrating clinical efficacy in restricted trial setting, other designs contribute to the evidence base.  In some situations, real-world data may provide clear advantage for understanding outcomes of treatment, for example, for patients excluded from trials, patients in actual clinical practice settings (vs. research settings), and patients whose treatment is not determined by protocol.

One dimension of the real-world data is type of outcome:  clinical, economic, or patient-reported outcomes/quality of life. Clinical outcomes include biological measures of morbidity (e.g., symptoms, acute events, side effects) and mortality.  Clinical outcomes include both surrogate (intermediate) and long-term measures.  Economic outcomes include estimates of medical and non-medical resource utilization and their associated costs.  Such data are used to project the expected cost of an intervention in the real world—e.g., in the numerator of a cost-effectiveness ratio.  Patient-reported outcome (PRO) is the term adopted by the FDA to encompass any report coming directly from patients about a health condition and its treatment, including symptoms, functional status, health-related quality of life, treatment satisfaction, preference and adherence.

A second dimension of real-world data is in terms of evidence hierarchies.  A number of groups have developed evidence hierarchies over the years that reflect the primacy of data from RCTs, and grade other types of evidence by the rigor of the research design.  For example, the hierarchy adopted by AHRQ grades evidence in order from most to least rigorous as follows:  1) systematic reviews and meta-analyses of RCTs; 2) non-randomized intervention studies; 3) observational studies; 4) non-experimental studies; 5) expert opinion. Strict use of evidence hierarchies may not account for the methodological quality of studies or may fail to reflect the overall strength of the evidence base.  Some recent grading systems combine judgments about evidence quality with judgments about the usefulness of the intervention.

A third dimension operationally categorizes real-world data by type of data source:  1) supplements to traditional registration RCTs; 2) large simple trials (also called practical clinical trials); 3) registries; 4) administrative data; 5) health surveys; and 6) electronic health records. To provide additional data alongside standard registration RCTs, researchers often gather information on variables such as PRO/quality of life, and medical resource use or costs.  Large simple trials (also called practical or pragmatic clinical trials) involve prospective, randomized assignment but aimed at larger more diverse real world population.  Practical clinical trials have the important strength of randomization, which minimizes bias in the estimation of treatment effects.  These trials are by necessity larger than conventional RCTs.  For this reason, they are more likely to have sufficient power to capture significant differences in key outcomes of interest, such as hospitalizations.  Registries are prospective, observational cohort studies of patients who have a particular disease and/or are receiving a particular treatment or intervention.  They can be used for understanding natural history, assessing or monitoring real-world safety and effectiveness, assessing quality of care and provider performance, and determining value or reimbursement levels.  Registries involve prospective data collection of clinical, economic and PRO information.  They typically include a larger and more diverse group of patients than what is generally studied in phase III RCTs; therefore, they may better reflect real-world patients, management practices, and outcomes.  Claims databases are administrative data (typically retrospective or real time, if possible) collected primarily for reimbursement, but containing some clinical diagnosis and procedure use with detailed information on charges.  These databases lend themselves to retrospective longitudinal and cross-sectional analyses of clinical and economic outcomes at patient, group, or population levels.  Such analyses can be performed at low overall cost and in a short period of time.  Between the sheer size of claims databases and the use of statistics, researchers can identify outcomes of patients with rare events, assess economic impact of various interventions, and gain insight into possible association between intervention and outcomes. Health surveys are designed to collect descriptions of health status and well being, health care utilization, treatment patterns, and health care expenditures from patients, providers, or individuals in the general population.  They typically collect information on representative individuals in the target population, whether patients, physicians or general population.  Surveys use design features that are as methodologically rigorous as those used in RCTs, for example, the complex sample survey designs.  Electronic health records (and other future technologies capturing real-time clinical treatment and outcomes) may well be important sources for real-world data in the future that will likely to be more generally available in clinical settings throughout the world.  EHRs contain more detailed information including disease specific symptoms at the personal level. 

Value and Use

Among the benefits of real-world data are that they can provide:

• Estimates of effectiveness rather than efficacy in a variety of typical practice settings;
• Comparison of multiple alternative interventions (e.g. older vs. newer drugs) or clinical strategies to inform optimal therapy choices beyond placebo comparators;
• Estimates of long-term (and rare) clinical benefits and harms from interventions;
• Examination of clinical outcomes in a diverse study population that reflects the range and distribution of patients observed in clinical practice;
• Results on a broader range of outcomes (e.g., quality of life and symptoms) than are traditionally collected in RCTs (i.e., mortality and major morbidity);
• Data on resource use for the costing of health care services and economic evaluation;
• Information on how a product is dosed and applied in clinical practice and on levels of compliance and adherence to therapy;
• Data in situations where it is not possible to conduct an RCT (i.e. narcotic abuse). 
• Substantiation of data collected in more controlled settings;
• Data in circumstances where there is an urgency to provide reimbursement for some therapies because it is the only therapy available and may be life-saving;
• Interim evidence--in the absence of RCT data--upon which preliminary decisions can be made; 
• Data on the net clinical, economic, and PRO impacts following implementation of coverage or payment policies or other health management programs (e.g., the kind of data CMS expects to collect under its coverage with evidence development policy).

Real-world data also have some important limitations.  For all non-randomized data, the most significant concern is the potential for bias.  Retrospective or prospective observational or database studies do not meet the methodological rigor of RCTs, despite the availability of sophisticated statistical approaches to adjust for selection bias in observational data (covariate adjustment, propensity scores, instrumental variables etc.).  Observational studies need to be evaluated rigorously to identify sources of bias and confounding prior to estimating direction and magnitude of health outcomes.  Observational or database studies may also require substantial resources.

Issues
Level of The complexity of data collection underscores the fact that the level of evidence required in any circumstance will relate to the question at hand.  It is important to recognize the variable quality of all data (whether prospective or retrospective, or experimental or observational).  The extent to which data provide good or bad evidence depends on the research design, the quality of the information collected, and how the data are used.  The optimal solution will depend on the circumstances.  Decisions typically rely on multiple sources, and are best thought of as conditional, to be revisited as additional evidence is generated.
In terms of data collection, it is important that efforts follow well-established research practices.  These include posing well-defined questions, specifying timeframes for the duration of data collection, conducting periodic monitoring to ensure quality and responsiveness to research questions, and limiting sample sizes to the minimum necessary.  They should also ensure that informed consent and human subject protections are in place.

Good process in using real-world data is an important issue. Observers point to several conditions for establishing good process, including transparency (the decision and the rationale for making them must be publicly accessible) and relevance (there must be a reasonable explanation for a decision’s rationale).  They should also be fair in the sense that real-world data will be used in similar fashion across technologies, or if situations demand a different rationale, the circumstances or principles would be known.  Decisions should not be “bureaucratically arbitrary,” or based on reasons that people do not view as meaningful or just. Processes should also allow opportunity for stakeholder participation.

Evidence costs money.  Inevitably, there are questions about whether resources devoted to the endeavor are worthwhile.  There is a need to prioritize decisions about real-world data such that the benefits of collecting additional information are expected to outweigh the costs.  The tool of “value-of-information analysis” offers a formal approach to deciding when and what types of data to collect.

Combining and integrating real-world data requires models.  As defined by the ISPOR Task Force on Modeling:  “Models synthesize evidence on health consequences and costs from many different sources, including data from clinical trials, observational studies, insurance claim databases, case registries, public health statistics, and preference surveys.”  Such bioclinical cost-effectiveness models and analyses are the primary vehicle for combining RCT and real-world data to support coverage and reimbursement decision-making.

There is an important need for ongoing stakeholder dialog on all of these issues, and a central policy question is the appropriate role of the public sector in producing and judging evidence.  Who collects and evaluates real-world data, who pays for these efforts, and what magnitude of resources is provided are key questions for policy makers worldwide.

Finally, there remains much work to be done to produce methodological and practical guidance on how best to collect and use the different types of real-world for different types of issues.

Bibliography

1. Garrison LP and Neumann PJ (co-chairs); Erickson P, Marshall D, Mullins CD.  Using real world data for coverage and payment decisions:  the ISPOR real world data task force report.  Value Health 2007; 10(5):  326-335.
2. Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA 2003; 290(12):1624-1632.
3. Weinstein MC, O'Brien B, Hornberger J et al. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices--Modeling Studies. Value Health 2003; 6(1):9-17.