WORKSHOPS - SESSION I Monday, May 17, 2010: 3:00 PM-4:00 PM
Clinical Outcomes Research
W1: COMPARATIVE EFFECTIVENESS RESEARCH FOR PERSONALIZED MEDICINE
Discussion Leaders: James Signorovitch PhD, Manager, Analysis Group, Inc., Boston, MA, USA; Eric Q. Wu PhD, Vice President, Analysis Group, Inc., Boston, MA, USA; Lee-Jen Wei PhD, Professor, Biostatistics, Harvard University, Boston, MA, USA
PURPOSE: Comparative effectiveness research (CER) has traditionally focused on the average effects of health interventions across broad populations. While this may seem to put CER at odds with the personalization of medical care, both research efforts share the goal of improving health outcomes through treatment optimization, and apparent conflicts stem primarily from differences in data sources and statistical methods. The purpose of this workshop is to review advances in research methodology that can bridge apparent gaps between CER and personalized medicine.
DESCRIPTION: The workshop will explore two problems at the interface of CER and personalized medicine though real-world examples. First, we will show how to identify patient subgroups that benefit most from one treatment option vs. another using either randomized trials or observational data. Since patients can have heterogeneous responses to alternative therapies, even treatments with limited clinical or economic benefits for the average patient can provide valuable benefits for certain subgroups of patients. Identifying these subgroups, which may be defined by clinical and molecular markers, will facilitate individually optimized care and will maximize the collective value of alternative treatment options. We will describe traditional approaches to detecting important subgroups, and compare and contrast with recently-developed statistical methods that provide increased power while avoiding multiple post-hoc comparisons. Secondly, we will show how to identify patient subgroups that benefit most from an added prognostic test (e.g., measurement of a new biomarker) that may be invasive, risky or costly. For treatments and tests, we will give an example of how the personalization of care can substantially improve population average effectiveness. The audience will be invited to participate though discussion of real-world examples and of the assumptions underlying the compared research methods.
Economic Outcomes Research
W2: CHALLENGES IN DESIGNING, CONDUCTING AND ANALYSING MULTINATIONAL ECONOMIC CLINICAL TRIALS
Discussion Leaders: Michael Drummond PhD, Professor of Health Economics, Centre for Health Economics, University of York, York, UK; Andrea Manca PhD, MSc, Senior Research Fellow, Centre for Health Economics, University of York, York, UK; Arie Barlev PhD, Senior Manager, Global Health Economics, Amgen, Thousand Oaks, CA, USA
PURPOSE: To discuss the challenges in designing, conducting and analysing multinational economic clinical trials and to offer some solutions.
DESCRIPTION: Designing, conducting and analysing economic studies to inform reimbursement decisions in different jurisdictions poses considerable challenges. The recent ISPOR Good Research Practices Task Force on Economic Data Transferability makes several suggestions on how relevant analysis can be conducted, either through data collected alongside clinical trials, or by using decision-analytic modelling. However, there are growing challenges in implementing these suggestions, now that multinational trials are increasingly being conducted in a wider range of jurisdictions, many of which are outside the main markets for which reimbursement dossiers are required. Therefore, this workshop will address several current issues in the design, conduct and analysis of economic studies conducted alongside multinational trials. Issues to be addressed will include: how should countries and clinical centres be selected for inclusion in studies; what background information should be collected about the countries/centres; what level of detail should there be in economic data capture; how does one select the type of statistical model for analysing the data; does it make sense to group countries and on what basis; how can the outputs of the analyses be used to produce country-specific cost-effectiveness ratios and other estimates useful for decision-analytic models? The workshop participants will draw on the discussion leaders’ personal experience, the published literature and on analyses they have previously conducted. The overall aim will be to offer a practical way forward to those considering collecting economic data alongside multinational clinical trials in the future.
W3: COST OF ILLNESS STUDIES: A FRESH LOOK AT ONE OF THE FOUNDATIONS OF HEALTH POLICY AND OUTCOMES RESEARCH
Discussion Leaders: Tami Mark PhD, MBA, Director, Thomson Reuters, Washington, DC, USA; Dan Huse MA, VP, Outcomes Research, Thomson Reuters, Cambridge, MA, USA; Joel W. Hay PhD, Associate Professor, Pharmaceutical Economics & Policy, University of Southern California, Los Angeles, CA, USA
PURPOSE: To provide a comprehensive and thoughtful presentation of the types of costs of illness (COI) studies, the different approaches to develop COI estimates, and the challenges, advantages and disadvantages inherent in different approaches.
DESCRIPTION: Cost of illness (COI) information is commonly the foundation of values statements for medical inventions. Both private and public sector entities typically start discussions about a disease by a stating how much the disease costs. The Federal government is considering developing disease specific health accounts. Despite the relevance of COI studies, the methods employed are not often critically reviewed and discussed. We will begin by describing applications of COI to health care decision-making and policy-making. We will then discuss the different meanings of COI including the cost to “treat a disease” the cost of people “with a given disease” and the cost of “treating a disease and its sequelae.” We will describe two main COI approaches: (1) the “top down” approach used in the earliest U.S. COI studies involving allocation of spending from the National Health Expenditure Accounts, (2) the “bottom up” approach that uses diagnostic information on micro-level data sets such as insurance claims data and the Medical Expenditure Panel Survey (MEPS). The advantages and disadvantages of various methods and data sources (e.g., claims versus MEPs or other public data sets) will be highlighted. Common challenges and issues such as allocating costs to secondary diagnoses and prescription medications, different payer perspectives, and the distinctions between incidence-based and prevalence-based COI approaches will be presented. We will delineate the approaches, challenges, and pitfalls of studies that attempt to estimate the cost of a disease and its sequelae, contrasting methods that allocate costs by specifying sequelae and using attributable risk fractions to approaches that attribute costs through propensity or regression modeling controlling for unrelated costs. Examples will be used throughout the workshop and discussion based on participants' experiences will be encouraged.
Health Care Policy Development Using Outcomes Research
W4: THE 2009 UPDATE OF THE AMCP FORMAT FOR FORMULARY COVERAGE AND REIMBURSEMENT DECISIONS IN THE US
Discussion Leaders: Richard Fry RPh, Director of Programs, Foundation for Managed Care Pharmacy, Alexandria, VA, USA; Sean Sullivan PhD, RPh, MS, Professor and Director, Pharmaceutical Outcomes Research & Policy Program, Department of Pharmacy, University of Washington, Seattle, WA, USA; Pete Penna PharmD, President, Formulary Resources, LLC, Mercer Island, WA, USA; Iris Tam PharmD, Director, Institutional Medical Communications, Genentech, South San Francisco, CA, USA
PURPOSE: The purpose of the workshop is to provide, in detail, revised evidentiary requirements contained in Version 3.0 of the U.S. Academy of Managed Care Pharmacy’s (AMCP) Format for Formulary Submissions and to discuss revisions and key issues regarding use of the Format that users and pharmaceutical manufacturers submitted in response to a general solicitation to revise the Format.
DESCRIPTION: AMCP first published its Format for Formulary Submissions in October 2000. AMCP Leadership and its members were motivated to develop these Guidelines by a growing need to ensure that increased use of medications, biopharmaceuticals, and vaccine products was appropriate and that newer products would bring added clinical and economic value to US payers. Since first publication of the Format, it has attracted considerable attention. As adoption of the Format has spread, manufacturers have begun to standardize the framework within which they present evidence of clinical and economic benefit. Since publication of the Format, AMCP and The Foundation for Managed Care Pharmacy (FMCP) have continuously sought input from pharmaceutical manufacturers and health system pharmacists through various venues in order to improve and clarify the process. Version 3.0 is the most recent attempt to address user’s comments and concerns. Methods: In January 2009, AMCP members and other users were solicited to suggest revisions for the evidence requirements and guidance document that comprises the Format. A transparent and stakeholder driven process for revising the guidelines was followed and a final revision was issued in October 2009. Results: Comments were received from over 25 organizations representing US payers, the pharmaceutical industry and contract research organizations. A multitude of changes to the Format were made, including: reducing redundancy, clarifying clinical evidence requirements, specifying an approach to present economic evidence in a more meaningful manner, and improving the efficiency and relevance of the written dossier.
Patient-Reported Outcomes/PREFERENCE-BASED Research
W5: REGULATORY EVALUATION OF PATIENT-REPORTED OUTCOME MEASURES (PROS) USED IN CLINICAL TRIALS
Discussion Leaders: Laurie B. Burke RPh, MPH, Director, Study Endpoints and Label Development, Center for Drug Evaluation and Research, Food and Drug Administration, Silver Spring, MD, USA; June Cai MD, Endpoints Reviewer, Study Endpoints and Label Development, Center for Drug Evaluation and Research, Food and Drug Administration, Silver Spring, MD, USA; Ann Marie Trentacosti MD, Endpoints Reviewer, Study Endpoints and Label Development, Center for Drug Evaluation and Research, Food and Drug Administration, Silver Spring, MD, USA
PURPOSE: To describe 1. FDA's qualification review of PRO instruments, 2. the critical components for FDA's PRO instrument qualification review and 3. best practices for interpretation of PRO clinical trial results.
DESCRIPTION: Increasingly, patient-reported outcome measures are used as efficacy endpoints in medical product clinical trials. Proposed medical product labeling claims determine the adequacy of PRO measures and endpoints within a specific clinical trial. PROs are held to the same rigorous standards as those applied to other clinical endpoints. PRO labeling claims must be supported by substantial scientific evidence and must not be false or misleading in any way. FDA’s current thinking on the use of PROs to support labeling claims is found in the FDA final guidance for industry “Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims” published December 9, 2009. This session will provide an overview of that guidance and will highlight how the final guidance differs from the 2006 draft guidance of the same name. This session will also describe FDA's qualification review of PRO instruments including the critical components necessary for that qualification review. This session will conclude with a description of best practices for interpretation of PRO clinical trial results.
Use of Real World Data
W7: HOW TO USE ADMINISTRATIVE CLAIMS DATABASES TO ASSESS LONG TERM CLINICAL BENEFIT OF MEDICATION INITIATION FROM AN EVOLUTIONARY PERSPECTIVE: TRADITIONAL ANALYTICAL APPROACHES VS. NOVEL USE OF TIME SERIES ANALYSIS
Discussion Leaders: Peter Sun MD, PhD, Vice President, Kailo Research Group, Indianapolis, IN, USA; Joanne R. Chang PhD, MD, Vice President, Evidence Based Medicine, Novartis Pharmaceuticals Corporation, East Hanover, NJ, USA; Jie Zhang PhD, Director, Novartis Pharmaceuticals Corporation, East Hanover, NJ, USA
PURPOSE: 1. Explore common issues of traditional analytical approaches in analyzing long term treatment effect among patients with a chronic disease, 2. Present a new evolutionary analytical approach that use interrupted time series analysis to accommodate the evolutionary nature of disease and treatment, 3. Demonstrate the novel use of interrupted time series analysis in assessing the long term clinical benefit of a treatment for a chronic disease through a real world case study, 4. Enable audience to improve the quality of long term treatment effect studies through innovatively applying interrupted time series analysis in comparative outcomes research among patients with chronic diseases.
DESCRIPTION: Treatment effect studies in outcomes research usually use cross-cohort comparison within the same period, or same-cohort comparison across two periods. These traditional analytical approaches often fail to adequately capture the evolutionary nature of disease progression and treatment effect, therefore, can result into under- or over-estimation of treatment effect when disease severity and treatment effect evolve over time. This workshop will introduce a new evolutionary approach that uses interrupted time series analysis to assess the long term effect of a treatment for a chronic disease. Topics covered in this workshop include a brief conceptual framework of the evolutionary analytical approach, in-depth introduction of interrupted time series analysis, detailed steps of applying the interrupted time series analysis in comparative outcomes research using administrative claims data, and a real world case study that will demonstrate the novel use of interrupted time series analysis in assessing long term clinical benefit of a medication initiation among patients with a chronic disease. After the real world case study, an interactive discussion or exercise will ensue. At the end of the workshop, audience should be able to apply interrupted time series to assess long term treatment effect among patients with an irreversible chronic disease.
WORKSHOPS - SESSION II Monday, May 17, 2010: 4:15 PM-5:15 PM
Clinical Outcomes Research
W8: MODEL ASSUMPTIONS IN MIXED TREATMENT COMPARISONS (MTCS): WHY THEY MATTER AND WHY YOU SHOULD CARE
Discussion Leaders: Rachael Fleurence PhD, Senior Research Scientist, Center for Health Economics and Science Policy, United BioSource Corporation, Bethesda, MD, USA; Kyle Fahrbach PhD, Senior Biostatistician, United BioSource Corporation, Lexington, MA, USA; David Vanness PhD, Assistant Professor, Population Health Sciences, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA
PURPOSE: To assist outcomes researchers in understanding the assumptions underlying mixed treatment comparison (MTC) models and how they impact results.
DESCRIPTION: MTCs are an increasingly important meta-analytic method for comparative effectiveness research (CER), as demonstrated both by increasing numbers of published studies and a growing reliance on MTCs as part of the technology appraisal process in various countries, including the U.K. and Canada. MTCs are also gaining ground in the US where their role in CER enterprise is being evaluated. Given the importance and novelty of these methods, an understanding of the critical assumptions that are being made in the statistical models is essential to outcomes researchers and decision-makers alike. We will first briefly present the theoretical basics of MTC analysis and describe how the commonly-used Bayesian hierarchical approach differs from classical meta-analysis. The audience will be given a worksheet to compare methodological differences between MTC and classical meta-analysis using the practical example of biologic treatments for rheumatoid arthritis (RA). In the second part of the workshop, we will explore some of the critical assumptions made in MTC analyses, including use of fixed, random or unconstrained effects models for placebo response rates and treatment effects. We will use published studies and the practical example of biologics in RA to show how these assumptions can critically affect results. We will also briefly discuss heterogeneity among included studies and approaches to identifying and coping with it. Health care decision makers and outcomes researchers should have an understanding of the meaning and impact of MTC modeling assumptions in order to both appropriately conduct and critically appraise MTC analyses. The workshop is not geared towards statisticians and this material will be presented for outcomes researchers with little knowledge of MTCs or meta-analytical techniques.
Economic Outcomes Research
W9: THRESHOLD ANALYSIS TO CHARACTERIZE PARAMETER UNCERTAINTY IN COST-EFFECTIVENESS ANALYSES WITH AMBIGUOUS RESULTS
Discussion Leaders: Amy O'Sullivan PhD, Director, Health Economics & Outcomes Research, i3 Innovus, Medford, MA, USA; Milton C. Weinstein PhD, Henry J Kaiser Professor of Health Policy & Management, Harvard School of Public Health, Boston, MA, USA
PURPOSE: Current recommendations for presenting uncertainty in cost-effectiveness analysis (CEA) include deterministic and probabilistic sensitivity analyses. This workshop will describe the use of threshold analysis as an alternative way of characterizing the effects of uncertain parameters on the results of a CEA.
DESCRIPTION: Deterministic and probabilistic sensitivity analysis (PSA) have become standard methods for handling parameter uncertainty in CEAs. PSA, which is recommended by many international decision-making bodies, involves assigning distributions around mean values of model parameters and conducting a Monte Carlo simulation. This allows the effects of uncertainty around multiple parameter values to be assessed simultaneously. However, there are instances in which the added value of conducting a PSA may be in question -- for example, if results from deterministic sensitivity analyses on 2-3 key parameters reveal considerable ambiguity in the results (e.g., results range from dominated to dominant). In this case, threshold analyses can be used to depict, for the relevant parameters, the ranges over which the intervention of interest would be cost-effective according to specified cost-effectiveness criteria. This approach may be particularly useful for decision-makers who have access to values of the relevant parameters for their target population. It may be more useful than cost-effectiveness acceptability curves, which reveal the probabilities that one or another strategy is cost-effective, but not what parameters the choice depends on. In this workshop, we will explore the use of threshold analysis and how it can be used to characterize parameter uncertainty. This will be demonstrated via a case study in which various graphical representations of threshold analysis results will be presented. Challenges faced in presenting results of threshold analyses and PSAs to a broad audience will be discussed, and workshop participants will be encouraged to share their experiences.
W10: WHAT CAN RESOURCE ALLOCATION IN SITUATIONS OF EXTREME SCARCITY TELL US ABOUT THE ROLE OF THE QALY IN PRIORITY SETTING?
Discussion Leaders: Peter Kolominsky-Rabas PhD, MBA, Scientific Director, Interdisziplinäres Zentrum für Public Health (IZPH) der Universität Erlangen-Nürnberg, Erlangen, Germany; Georgia Mitsi PhD, MBA, MSc, Manager, United BioSource Corporation, Lexington, MA, USA; J. Jaime Caro MD, Senior Vice President of Health Economics, United BioSource Corporation, Lexington, MA, USA
PURPOSE: When demand for health care services massively exceeds supply (e.g., during a hurricane, earthquake massive terrorist attack) it becomes necessary to prioritize allocation of available resources. This prioritization is fraught with moral quandaries and emotional responses but it is important to provide guidance in advance to help rescuers and providers act consistently and ethically. This has been the task of a US government task force, where the question of using the QALY as the arbiter was bitterly debated. This workshop will review the task force discussions to address two issues: What are the ethical approaches to resource allocation in this setting? What lessons do they hold for priority setting in less extreme situations, such as routine reimbursement decisions?
DESCRIPTION: The context examined by the task force (an improvised terrorist nuclear attack) will be reviewed and the proposed approaches for priority setting will be presented, including the idea of using QALY gains. The ethical implications of each will be described, especially in relation to denying treatment to individuals based on specific considerations (e.g., life expectancy and expected quality of life). The tension between the egalitarian position and the utilitarian one will be discussed and the final proposal based on need will be presented. The workshop will discuss how a need based algorithm would apply at different levels of scarcity, while recognizing the gap in the perspectives of the individual practitioners and the system-wide decision makers. The learnings from the extreme context will be juxtaposed with the current use of the QALY to prioritize across the health care system in more routine situations.
Health Care Policy Development Using Outcomes Research
W11: YOUR ROLE IN INTERPRETING AND APPROPRIATELY COMMUNICATING CER RESULTS IN A HIGHLY CHARGED ENVIRONMENT
Discussion Leaders: Robert Dubois PhD, MD, Chief Medical Officer, Cerner LifeSciences, Beverly Hills, CA, USA; Jean Slutsky PA, MSPH, Director, Center for Outcomes and Evidence, Agency for Healthcare Research and Quality, Rockville, MD, USA; Lester Paul MD, MS, Vice President, Clinical & Scientific Affairs, National Pharmaceutical Council, Reston, VA, USA; Brian Sweet RPh, MBA, Chief Pharmacy Officer, WellPoint, Grand Island, NY, USA
PURPOSE: 1) to present a new approach to examining forthcoming comparative effectiveness research (CER), and 2) to interactively explore how key stakeholders may interpret and communicate that research in a way that is most impactful for target audiences
DESCRIPTION: With the current national focus on improving our health care system, CER has received substantial attention. The recent highly charged controversy about the USPSTF mammography guidelines showed that stakeholders interpret and communicate findings in varying ways based upon their unique perspective and personal interest in the results. This workshop will first present a new approach to evaluating these forthcoming reports that is built around 3 consecutive steps (1: Consider for whom the findings are applicable; 2: Consider whether any aspects of the study design might affect the results; 3: Consider whether the findings may change with new research) and 5 associated checklists. Next, three stakeholders (AHRQ, health plan, manufacturer) will describe how they differ in the key elements that they seek in CER reports, and how these elements inform their assessment. Each stakeholder will have distinct views on the level of evidence required and the extent of study generalizability. Payers may have stricter views than those wishing to gain access to the assessed technologies. Understanding how those stakeholders might interpret and communicate that information will be important. In the final section, workshop participants will be encouraged to react to the approaches and provide methodologic or policy recommendations for improvement. Upon completion of the workshop, attendees should: 1) learn a structured approach to examining CER reports, 2) understand how stakeholders are likely to differ in their responses to them, and 3) enhance their own ability to interpret and communicate CER.
Patient-Reported Outcomes/PREFERENCE-BASED Research
W12: INTERPRETATION OF PRO TRIAL RESULTS TO SUPPORT FDA LABELING CLAIMS
Discussion Leaders: Kathleen W. Wyrwich PhD, Senior Research Leader, Center for Health Outcomes Research, United BioSource Corporation, Bethesda, MD, USA; Laurie B. Burke RPh, MPH, Director, Study Endpoints and Label Development, Center for Drug Evaluation and Research, Food and Drug Administration, Silver Spring, MD, USA; Ann Marie Trentacosti MD, Endpoints Reviewer, Study Endpoints and Label Development, Center for Drug Evaluation and Research, Food and Drug Administration, Silver Spring, MD, USA
PURPOSE: Although the 2009 FDA Guidance for Industry Patient-Reported Outcomes (PRO) Measures: Use in Medical Product Development to Support Labeling Claims (hereafter referred to as the 2009 PRO Guidance) excluded the term minimum important difference (MID), interpretation of PRO change scores remains a crucial issue for understanding the effect of treatment. The objective of this workshop is to discuss methods for interpreting PRO clinical trial results, focusing on the responder definition and the cumulative distribution function.
DESCRIPTION: Appropriate clinical trial design and results, including statistically significant PRO differences between treatment and control groups, are necessary for a PRO labeling claims. However, it is also important to provide an interpretation of PRO changes over time to facilitate the evaluation of efficacy and the communication of PRO results to regulators and ultimately patients, physicians, and providers. The 2009 PRO Guidance discusses two methods for interpretation. The first method classifies the change of each trial subject within each treatment and control group via an a priori responder definition, which is a minimum change threshold for identifying meaningful PRO improvement in an individual patient and is specific to the targeted population and other trial design characteristics. The second method is the presentation of data in the form of cumulative distribution function response curves for treatment and control groups. With this diagram, the percentage of patients in each treatment group achieving a spectrum of responder thresholds can be easily compared, alleviating the need for the selection of a specific responder definition to interpret the PRO changes in each group. During this workshop, we will discuss why PRO interpretation is important, methods for ascertaining the responder definition, the value and use of cumulative distribution functions, and communication of the interpretation methods and results in PRO evidence dossiers and clinical study reports to support labeling claims.
Use of Real World Data
W13: PROJECTING DISEASE PREVALENCE: THE IMPACT OF METHODOLOGY– EXAMPLES IN ATRIAL FIBRILLATION
Discussion Leaders: Kathy L. Schulman MA, Principal, Outcomes Research Solutions, Inc., Bolton, MA, USA; Stephen S. Johnston MA, Research Leader, Outcomes Research, Thomson Reuters, Washington, DC, USA; Jay Lin PhD, MBA, Director, Evidence Based Medicine, sanofi-aventis U.S., Bridgewater, NJ, USA; Dan Huse MA, Vice President, Outcomes Research, Thomson Reuters, Cambridge, MA, USA
PURPOSE: The objective of this workshop will be to educate the audience on the use of secondary data sources to estimate current and future disease prevalence and the sensitivity of such estimates to various methodologies and assumptions.
DESCRIPTION: Secondary data sources are increasingly used to estimate disease prevalence in the US but published estimates can vary widely, depending on the population observed and the methodology employed. Some studies rely on cross-sectional snapshots as the basis for projection while other employ prevalence trends or use measures of incident disease in combination with recursive models. Estimates may be standardized to different populations at different points in time, making comparison difficult. Estimates may also need to account for trends in the prevalence of risk factors which in turn influence future incidence and prevalence of the disease in question. Quantifying these second-order effects presents analytical challenges and additional sources of uncertainty in the estimates. This workshop will use a case study to illustrate the sensitivity of prevalence estimates to differences in study design and various statistical and non-statistical assumptions. Additional comparisons will be drawn from published studies estimating the prevalence of atrial fibrillation. The workshop will review the following: 1) ecological vs. cohort designs 2) considerations in using cross-sectional snapshots 3) considerations in trending 4) estimations of disease incidence and 5) recursive models. Workshop participants will be encouraged to offer their perspectives and recommendations.
W14: POST-APPROVAL RESEARCH: ALTERNATIVE STUDY DESIGNS AND OPERATIONAL CONSIDERATIONS
Discussion Leaders: Cynthia Verst PharmD, MS, Senior Vice President, Late Phase Research, i3 Innovus, Cold Spring, KY, USA; Michael Drummond PhD, Professor of Health Economics, Centre for Health Economics, University of York, York, UK; Myoung Kim PhD, MA, MBA, Director, Health Economics & Outcomes Research, Ortho-McNeil Janssen Scientific Affairs, Raritan, NJ, USA; Frank Lobeck PharmD, Vice President, Americas, Late Phase Research, i3 Innovus, Dexter, MI, USA
PURPOSE: The objectives of this workshop are: 1) to describe the key features, strengths and weaknesses of different study designs for post-launch research; 2) to outline how to select study designs based on methodological and practical considerations; and 3) to enable the audience to select study designs based on these criteria, as well as help the audience to consider important operational options when implementing studies.
DESCRIPTION: In the past, most post-launch studies have focused on adverse event monitoring. Several payers/reimbursement agencies are now making recommendations for post-launch research. These post-launch studies offer the opportunity to collect data that cannot be collected prior to launch, including effectiveness and economic data. Although pre-approval studies follow regulatory processes with standard study designs such as randomized controlled trials (RCTs) and strict guidance on implementation, post-approval research offers more flexibility in possible study designs and study implementation. This workshop will identify and discuss the strengths and weaknesses of different study designs for post-launch research, including pragmatic RCTs, observational studies, registries, chart reviews, database analysis and models. This workshop will demonstrate the methodological factors (e.g. timeframe, generalizability, randomization, rare outcomes, long-term data) and practical factors (e.g. outcomes collected directly from patients, timing of results, budget) that the audience should consider when selecting between the different study designs. Audience participation will be facilitated through a case study and discussion of audience-generated issues and solutions. The case study will present a scenario based on product requirements, existing studies, timing and general budget. The audience will consider the following operational factors when making decisions on study implementation: study design, protocol complexity, number of outcomes measured, Good Clinical Practice requirements, investigator experience, patient recruitment and retention, number of sites and patients, site training and start-up, clinical monitoring, data collection and management, and data analysis.
WORKSHOPS - SESSION III Tuesday, May 18, 2010: 4:00 PM-5:00 PM
Clinical Outcomes Research
W15: SELECTING COMPARATORS AND OUTCOMES FOR COMPARATIVE EFFECTIVENESS STUDIES IN ONCOLOGY
Discussion Leaders: C. Daniel Mullins PhD, Professor, Pharmaceutical Health Services Research Department, University of Maryland, School of Pharmacy, Baltimore, MD, USA; Russell W. Montgomery MHS, Project Manager, Center for Medical Technology Policy, Baltimore, MD, USA
PURPOSE: The purpose of the workshop is to provide guidance and facilitate discussion on the selection of comparators and outcomes for oncology comparative effectiveness studies, based on a set of guidelines developed by a diverse multi-stakeholder group of clinical and health policy decision makers.
DESCRIPTION: Substantial uncertainty exists about the benefits and harms of many oncology drugs, creating important challenges for clinical and health policy decision makers, including patients, providers, payers, and policymakers. The Center for Medical Technology Policy (CMTP) conducted a six-month initiative to develop recommendations on methods and strategies to improve the validity, relevance, and consistency of clinical research designed to assess the comparative effectiveness and value of oncology drugs. To accomplish the goal, CMTP worked with more than 60 experts and stakeholders in an in-person meeting and through a series of interviews to reach consensus on better ways to more frequently and consistently design, fund, and implement prospective clinical studies of new indications for oncology drugs. This effort resulted in specific recommendations and principles to serve as an evidentiary framework for CER study design for oncology drugs, including substantial work on identifying comparators and outcomes that meet the evidence needs of stakeholders. This workshop will present the consensus-based recommendations on comparators and outcomes, as well as real-world examples of high-quality CER studies in oncology. The discussion leaders are the two leaders of the initiative and have planned and conducted all meetings and interviews.
Economic Outcomes Research
W16: IDENTIFYING AND PROJECTING PRODUCT VALUE THROUGH EARLY-PHASE CLINICAL-ECONOMIC MODELING
Discussion Leaders: Ipek Özer-Stillman MS, Director, Abt Bio-Pharma Solutions, Inc., Lexington, MA, USA; Bjorn Bolinder MBA, Head, Primary Care, Global Health Outcomes, Merck & Co., Kenilworth, NJ, USA; J. Jaime Caro MDCM, FRCPC, FACP, Senior Vice President of Health Economics, United BioSource Corporation, Lexington, MA, USA
PURPOSE: The purpose of this workshop is to explore the design and development of early-phase clinical-economic models, and how they can be used to identify and project pharmacoeconomic value messages.
DESCRIPTION: Only recently has early-phase pharmacoeconomic modeling emerged as a functional tool in product value messaging. Early phase models help facilitate the efficient allocation of development resources, inform future pricing considerations, and identify key barriers and drivers related to coverage and reimbursement. Optimal processes for designing and developing early-phase clinical-economic models have yet to be determined. As opposed to previous presentations on this topic, this workshop will focus on steps to design an integrated approach for global early-phase modeling and how to use early-phase models to support a successful outcomes research strategy. Topics of discussion in the workshop will include the variety of modeling analyses that can facilitate decisions related to pricing, such as threshold pricing, break-even pricing. Baseline benchmarking for potential risk-sharing schemes also will be discussed. Workshop leaders will use examples to reveal the advantages and limitations of early-phase models and their role in development of later-phase modeling. Finally, case studies will guide a discussion about ways to incorporate model-generated value messages into commercialization plans. Throughout the workshop, participants will be encouraged to share their own experiences.
W17: STRUCTURAL EQUATION TECHNIQUES IN COMPARATIVE EFFECTIVENESS RESEARCH: ESTIMATION OF TREATMENT EFFECT USING OBSERVATIONAL DATA
Discussion Leaders: William Crown PhD, President, i3 Innovus, Waltham, MA, USA; Xin Ye PhD, MS, Associate Director, Health Economics & Reimbursement, Ethicon, a Johnson & Johnson Company, Somerville, NJ, USA; Henry J. Henk PhD, Director, HEOR, i3 Innovus, Eden Prairie, MN, USA
PURPOSE: The purpose of this workshop is to 1) provide a brief review of the interpretation of treatment effects captured through typical analytic methods (e.g., OLS regression models) for analyses of observational data and 2) introduce structural equation and decomposition methods for CER analyses of observational data.
DESCRIPTION: Some estimates suggest that as much as one-third of all health care expenditures generate no clinical benefit (Fisher, Wennberg, Stukel, et al, 2003). Hence, with the goal to stop treating patients with therapies that generate little, or no, clinical benefit it is not surprising that comparative effectiveness research has received considerable attention in the health reform discussions currently taking place in the United States. A variety of methods may be used to support CER including systematic reviews and meta analyses, head-to-head clinical trials, registries, and analyses of retrospective claims data, medical records, and survey data. However, it is reasonable to expect, with the emphasis of CER on the real world effectiveness and safety of treatments, that increased emphasis will be placed on observational research methods (e.g., analysis of observational data). The purpose of this workshop is to 1) provide a brief review of the interpretation of treatment effects captured through typical analytic methods (e.g., OLS regression models) for analyses of observational data and 2) introduce structural equation and decomposition methods for CER analyses of observational data. Although these methods have not been commonly used for outcomes research, they offer the opportunity to extract significantly more information regarding treatment effects than the typical analytic methods that rely on the use of a dummy variable to estimate treatment effect. Methodological and all other issues associated with the use of these methods will be discussed. Audience participation will be encouraged through the use of a real-world example.
Health Care Policy Development Using Outcomes Research
W18: ENGAGING STAKEHOLDERS TO ENHANCE PERFORMANCE OF COMPARATIVE EFFECTIVENESS RESEARCH
Discussion Leaders: Jean Slutsky PA, MSPH, Director, Center for Outcomes and Evidence, Agency for Healthcare Research and Quality, Rockville, MD, USA; Newell Mcelwee PharmD, Executive Director, US Outcomes Research, Merck & Co., Inc., North Wales, PA, USA; Eric Wall MD, MPH, Senior Medical Director, Qualis Health, Seattle, WA, USA; Jennifer Bright MPA, Executive Director, Society for Healthcare Epidemiology of America, Alexandria, VA, USA
PURPOSE: The purpose of this workshop is to discuss the engagement of stakeholders to enhance comparative effectiveness research, specifically utilizing the model employed by the Effective Health Care Program.
DESCRIPTION: Traditionally, stakeholder involvement in research has meant communicating questions or problems at the beginning and communicating results at the end (Burger, Gochfeld & Pletnikoff, 2009). While this model may be more efficient for the research enterprise, priorities, methods, and results do not necessarily address the needs of the end-user. Stakeholder engagement is a core function of the Agency for Healthcare Research and Quality's Effective Health Care (EHC) Program, and an emerging and debatable issue in outcomes research. The model used by the EHC Program is driven by stakeholders to ensure that research is relevant, processes are transparent, and the results are valued and useable. In the EHC Program, Stakeholders play a central role in identifying research priorities, crafting key research questions, consulting on research methods, and translating and disseminating research products. The model presents many challenges, but arguably enhances research and makes it more relevant to decision-makers. Although there are numerous approaches to stakeholder engagement in research, this workshop will focus on the approach utilized by the EHC Program. Panelists will discuss the benefits and challenges of engaging stakeholders in 1) topic generation, priority setting and program development (Dr. Eric Wall), 2) developing research reports and methods (Dr. Newell McElwee), and 3) the dissemination and uptake of research products (Jennifer Bright). The panel will discuss the importance of interactive relationships between investigators and stakeholders in order to encourage reciprocal learning and knowledge generation to improve health outcomes. The audience will be asked to provide input on each of the topic areas addressed by panelists.
Patient-Reported Outcomes/PREFERENCE-BASED Research
W19: STRATEGIES AND METHODS FOR OPTIMIZING THE USE OF OUTCOMES REPORTED BY PATIENTS ALONG THE HEALTH CARE DECISION MAKING CONTINUUM—FROM REGULATORS, TO HEALTH TECHNOLOGY ASSESSORS, TO PAYERS
Discussion Leaders: Asha Harreendran PhD, Senior Research Scientist, Health Care Analytics, United BioSource Corporation, London, UK; William Lenderking PhD, Senior Research Leader, Health Care Analytics, United BioSource Corporation, Lexington, MA, USA; Lou Garrison PhD, Professor, Pharmaceutical Outcomes Research & Policy Program, Department of Pharmacy, University of Washington, Seattle, WA, USA
PURPOSE: The workshop will aim to discuss strategies that pharmacoeconomic and outcomes researchers can use to enhance methods for using patient-reported outcomes (PROS) and other patient-provided data, to collect evidence of the impact of the benefits and harms of interventions in the major phases of healthcare decision making.
DESCRIPTION: The patient perspective on the value of innovative medicines has become increasingly important as consumers become more educated and active in their own healthcare. Health care decision makers - regulators, health technology assessors, and payers—have differing methodological requirements for measuring value from the patient’s point of view. Researchers responsible for developing and implementing strategies to demonstrate patient relevant outcomes need to consider how best to match methods for evidence collection to the differing requirements of decision makers. The first discussant will present evidence and methods required to evaluate benefit and risk of health technologies for market authorization, "coverage"/market access decisions and reimbursement decisions. The second discussant will share specific examples of studies used to collect the evidence, to highlight the use of methods required by these decision makers. Participants will then be given a hypothetical product profile and workshop discussants will facilitate sub-group discussion of potential value statements and optimal methods for collecting appropriate evidence to meet requirements of specific decision makers. The workshop will conclude with a presentation highlighting specific considerations for developing and selecting patient reported outcomes to support patient relevant value statements for the full range of health care decision makers.
Use of Real World Data
W20: DEVELOPING TOOLS FOR CONDUCTING OBSERVATIONAL DATABASE RESEARCH ACROSS A NETWORK OF REAL-WORLD DATA SOURCES
Discussion Leaders: Abraham G. Hartzema MSPH, PhD, FISPE, Professor and Eminent Scholar Perry A Foote Chair in Health Outcomes Research & Pharmacoeconomics, Department of Epidemiology and Biostatistics College of Public Health and Health, University of Florida, Gainesville, FL, USA; Vinit Nair MS, RPh, Director, Miami-Humana Health Services Research, Miami-Humana Health Services Research Center, Humana Inc, Miami, FL, USA; Greg Hess MD, MBA, MSc, Vice President for Health Economics / Outcomes Research & Chief Medical Officer, SDI Health, Plymouth Meeting , PA, USA; Patrick B. Ryan MEng, Statistical and Quantitative Sciences, Research & Development, GlaxoSmithKline, NC, NC, USA
PURPOSE: To allow researchers to share perspectives on strategies for maximizing the use of a network of real-world observational databases, including developing standardized tools and processes for summarizing data, applying analysis methods and integrating results into healthcare decision-making processes.
DESCRIPTION: Real world observational data (both administrative claims and electronic health records) have provided fertile ground for conducting targeted studies to evaluate relationships between medical treatment and health outcomes. Several recent efforts have attempted to establish networks of disparate data sources to facilitate more extensive use of the data for comparative effectiveness research, quality of care, and pharmacoepidemiology. The Observational Medical Outcomes Partnership (OMOP, http://omop.fnih.org) is a public-private partnership that has established a research community to conduct methodological research to inform the appropriate use of observational data for systematic monitoring of the effects of medicines. Topics covered in this workshop include a conceptual framework for organizing observational data to enable research across a network of data sources; the development of standardized procedures for creating descriptive data characteristic summaries of observation data; review of processes and tools for defining Health Outcomes of Interest; assessment of analysis techniques for estimating drug-outcome associations; and conclude with a discussion of opportunities and challenges for adapting active drug safety surveillance strategies for outcomes research and comparative effectiveness evaluations.
W21: PROSPECTIVE OBSERVATIONAL STUDIES: AVOIDING DESIGN PITFALLS USING PROCESS MAPS, DECISION-TREE METHODS, AND ELECTRONIC MEDICAL RECORDS
Discussion Leaders: Kelly Hollis MBA, Senior Director Survey Research, RTI Health Solutions, Research Triangle Park, NC, USA; Margaret Mordin MS, Senior Director Regulatory and Health Outcomes Strategy, RTI Health Solutions, Ann Arbor, MI, USA; Pete Bechtel, President and CEO, eCast Corporation, Raleigh, NC, USA; Lori McLeod PhD, Head, Psychometrics, Patient-Reported Outcomes, RTI Health Solutions, Research Triangle Park, NC, USA
PURPOSE: The objective of this workshop is to identify key decision points and outline critical steps necessary for designing a successful prospective observational study.
DESCRIPTION: Collection of data in prospective observational studies is often required to gather real-world evidence of product value and safety not available through other sources. With increased focus on comparative effectiveness and limited health care budgets, the need for documented product value is more critical than ever. Because of internal and external pressures to initiate and complete these studies quickly, many studies begin without adequate discussion of the purpose, intended audience (e.g., regulators, prescribers, payers, patients) for results/messages, and other design aspects of the study. In this workshop, we will provide an overview of the major design considerations necessary for a successful prospective study. Using process mapping and decision-tree methodology, we will provide a guide to help research teams optimize their design discussions prior to protocol development. As an illustration, we will discuss a real-life scenario starting with the study purpose and will include the participants in a discussion surrounding key design considerations such as the target population, endpoints, patient follow-up, and feasibility. As part of the feasibility assessment, we will describe the use of emerging technologies, such as electronic medical records and natural language processing, which can offer alternative ways to assess protocol design in a cost-effective manner prior to study start. Additional examples will provide a backdrop for active participant discussion regarding successes and failures. This interactive session will revolve around exchange of information and participant experiences about study design and how to avoid pitfalls using the guidance provided in the workshop.
WORKSHOPS - SESSION IV Tuesday, May 18, 2010: 5:15 PM-6:15 PM
Clinical Outcomes Research
W22: PUBMED®/MEDLINE®/NLM: THE NEW AND IMPROVED LITERATURE SEARCH
Discussion Leaders: Kristine Ogden, Research Manager, Global Health Economics & Outcomes Research, Lifecycle Sciences Group, ICON Clinical Research, San Francisco, CA, USA; Kimberly Miller PhD, Associate Director, Global Health Economics and Outcomes Research, Lifecycle Sciences Group, ICON Clinical Research, San Francisco, CA, USA; Alison Aldrich MSI, MPH, NN/LM PNR Technology Outreach Coordinator, Health Sciences Library, University of Washington, Seattle, WA, USA; Ione Auston MLS, Librarian, National Information Center on Health Services Research (NICHSR), National Library of Medicine (NLM), Bethesda, MD, USA
PURPOSE: The National Library of Medicine (NLM) provides free online access to PubMed®, a database that houses one of the largest and most comprehensive collections of citations for internationally-published biomedical literature. Most PubMed® citations have MEDLINE® status and have been reviewed and indexed by the NLM with Medical Subject Headings (MeSH). Although PubMed® offers a fairly simple search interface, its intricacies are not readily apparent to even the most experienced user. This workshop aims to provide an overview of the essential features of PubMed® that will enable participants to conduct literature searches more efficiently and better understand and assess the results of searches conducted by others.
DESCRIPTION: Outcomes researchers turn to published literature to gather evidence in comparative assessments, aid clinical decision making, select modeling parameters, identify resources to include in meta analyses, and for other reasons. Almost always, a structured approach, with a clear methodology to identify and select articles is needed to enable a thorough review and critical literature evaluation. PubMed® offers tools that assist with both systematic research and casual reviews for medical literature. The workshop leaders will review practical strategies for building searches and managing results in PubMed®. This discussion will include an overview of MeSH and review the pros and cons of constructing PubMed® searches with MeSH terms. There will be a demonstration of PubMed® filters and queries that allow researchers to easily narrow search results based on additional criteria or run pre-defined search strategies with a particular focus (e.g., limited to systematic reviews, a disease subset or a clinical study category). Finally, the leaders will introduce participants to specific queries developed by the National Information Center on Health Services Research and Health Care Technology (NICHSR) that are tailored to the outcomes and/or comparative effectiveness researcher.
Economic Outcomes Research
W23: SENSITIVITY ANALYSIS IN COST-EFFECTIVENESS STUDIES: FROM GUIDELINES TO PRACTICE
Discussion Leaders: Rahul Jain PhD, Assistant Professor, Department of Clinical and Administrative Pharmacy, University of Georgia, College of Pharmacy, Athens, GA, USA; Eberechukwu Onukwugha PhD, MS, Assistant Professor, Pharmaceutical Health Services Research Department, University of Maryland, School of Pharmacy, Baltimore, MD, USA; Michael J. Grabner PhD, Postdoctoral Fellow, Pharmaceutical Health Services Research Department, University of Maryland, School of Pharmacy, Baltimore, MD, USA
PURPOSE: This workshop has three objectives: (1) To describe the different sources of uncertainty in cost-effectiveness analysis (CEA); (2) To illustrate differences in addressing uncertainty between published guidelines and current practice, based on a literature review; (3) To provide practical advice to researchers on the conduct and reporting of sensitivity analysis (SA) in a checklist format.
DESCRIPTION: CEAs are one of the most popular forms of economic evaluation and rely on a number of ‘testable’ assumptions. Changes in these assumptions can lead to changes in results, introducing uncertainty. SA formalizes different methods to express and evaluate this uncertainty. The workshop will begin by presenting a brief theoretical overview of the three different sources of uncertainty in a CEA (methodological, structural, and parametric). The audience will be challenged to classify a list of variables and scenarios according to these three sources of uncertainty. Next, we will describe the results from a literature review conducted by the presenters based on 406 CEA articles. We will discuss (patterns over time in) the evaluation of each source of uncertainty in CEAs based on the findings from the literature review and subsequent multivariable analyses. The concept of ‘robustness’ within the context of CEA also will be discussed. In the final part of the workshop we will present two detailed examples of CEAs and interact with the audience to identify the different kinds of uncertainty that might be relevant in these cases as well as how to address them. We will conclude the workshop by providing some practical advice for researchers in the field using a checklist developed by the presenters.
W24: WHEN THERE IS NO HEAD-TO-HEAD TRIAL: USING A PARTS APPROACH
Discussion Leaders: Zeba Khan PhD, Vice President, Pricing and Market Access, Celgene, Summit, NJ, USA; J. Jaime Caro MD, Senior Vice President of Health Economics, United BioSource Corporation, Lexington, MA, USA
PURPOSE: Decision makers increasingly demand comparison of newer products’ effectiveness with alternatives but head-to-head randomized clinical trials (RCT) are rarely done. Such RCTs are very expensive and don’t provide required evidence immediately. Meta-analytic techniques like mixed treatment comparisons can help but cannot fully address differences in populations, protocols etc. An alternative is to reproduce the product’s phase III RCT(s) in a simulation and use this to estimate what would have occurred if there was a head-to-head trial. PARTS (Phantom Arm Randomized Trial Simulation) is a viable, timely, cost-effective methodology to supply the required information. In this workshop, we will present this approach and its advantages and limitations.
DESCRIPTION: Patient-level data from the product’s RCTs are used to develop a set of predictive equations reflecting what occurred in the trial. These equations must be detailed, precise, and typically time-dependent, often using parametric failure-time analyses. Impact of the missing comparator(s) is derived from their own trials using a back-and-forth calibration algorithm. This inserts terms in the equations reflecting comparator’s effects. The calibrated equations are used to construct a detailed simulation of the RCT, including rolling enrollment, randomization, discontinuation, assessment of outcomes etc. The actual patients from the RCT are fed into the simulation and those randomized to placebo are reassigned to comparator. PARTS is used to generate the clinical endpoints for comparators as if they had been part of the original RCT. These results can be used in assessments of risk-benefit and economic evaluations. Examples that have been submitted to FDA, EMEA and various HTAs will be discussed.
Health Care Policy Development Using Outcomes Research
W25: MAKING A CASE FOR VALUE: DIFFERENCES IN DOSSIER DEVELOPMENT AND VALUE COMMUNICATION FOR PHARMACEUTICALS VERSUS DIAGNOSTICS AND MEDICAL DEVICES
Discussion Leaders: Eric Faulkner MPH, Senior Director, RTI Health Solutions, Research Triangle Park, NC, USA; Shahnaz Khan MPH, Senior Director, RHOS, RTI Health Solutions, Research Triangle Park, NC, USA; John Watkins RPh, MPH, BCPS, Pharmacy Manager, Formulary Development, Premera Blue Cross, Mountlake Terrace, WA, USA; Vincent Polkus, Global Marketing Reimbursement Manager, GE Healthcare, Waukesha, WI, USA
PURPOSE: The objectives for this workshop are to evaluate similarities and differences associated with submission of evidence characterizing the value of drugs, medical devices and diagnostics to health decision makers. The workshop will also serve as a forum for discussion and generation of ideas for moving towards a standard format for dossier development for devices/diagnostics that is practical for these industries and also meets health technology assessor and payer requirements.
DESCRIPTION: Clinical and economic evidence in support of a pharmaceutical product is often accomplished using standard formats such as the AMCP Format for Formulary submissions in the US and the NICE guidance for manufacturers in the UK. Although these formats work well for pharmaceutical submissions, there are not comparable guidelines for diagnostic/device submissions and despite prior attempts no accepted format has yet emerged. Similar to pharmaceutical manufacturers, medical device and diagnostics manufacturers face increasing pressure to demonstrate value for their products. What is the best format for preparing this information for review by healthcare decision makers? What information would health decision makers ideally like to see? Can the AMCP Format in the US be used as a starting point for submission of medical devices/diagnostics and which format sections are not an ideal fit given significant differences in product development, adoption and reimbursement? How should medical technology dossiers address issues like off-label use, product feature and outcomes comparisons, iterative product development, influence of technology users on outcomes, and substantive differences in the processes for clinical and economic evidence development? The worskhop will focus on the unique issues related to dossier development for devices and diagnostics compared to drugs. The session will include an interactive discussion on ideas for developing dossiers that takes into account clinical and economic evidence development considerations, technology attributes, and value communication issues relevant to different technology types.
W26: MARKET ACCESS IN THE LATIN AMERICAN CONTEXT: HOW FAR WILL THE STANDARD HEALTH ECONOMIC TOOLKIT CARRY YOU?
Discussion Leaders: Joanna Campbell PhD, Senior Project Manager, Health Economics & Outcomes, i3Innovus, Medford, MA, USA; Gabriella Tannus Branco de Araújo MBA, Health Economics Director, AxiaBio Consulting, Sao Paulo, Brazil; Ximena Burbano MD, International Research Director, Zilonis, Boca Raton, FL, USA; David Thompson PhD, Senior Vice President, Health Economics & Strategic Consulting, i3 Innovus, Medford, MA, USA
PURPOSE: The purpose of this workshop is to discuss differences in market access requirements in Latin America compared with the US and Europe, and the implications for global health economic programs designed to support products across the three regions.
DESCRIPTION: The use of pharmacoeconomic data is a standard part of formulary submissions and price negotiations in Europe and the US. The formalization of pharmacoeconomic requirements is a more recent development in Latin America: Brazil, Mexico and Colombia have recently released guidelines, while other countries have guidelines in development. There are important differences between the US, European and Latin American health care systems that influence the development of appropriate global pharmacoeconomic programs. In the US, guidelines exist within the AMCP Format, but submission and price negotiations are largely performed on a payer-by-payer basis with priority often given to budgetary impact analyses. In the UK, rigorous pharmacoeconomic guidelines have been developed by NICE, with an emphasis on cost-utility analyses. In Brazil and Colombia, in contrast, newly released pharmacoeconomic guidelines from the Ministries of Health are intended to guide submissions to both the government and federally-regulated commercial health plans in Brazil and to the universal Mandatory Health Plan in Colombia, with emphasis on both cost-effectiveness and budgetary impact analyses. Brazil and Colombia are also poised to make wider use of innovative arrangements, more commonly used in Europe than the US, such as risk-sharing agreements and government-allocated funds for patients otherwise denied coverage for higher-cost drugs. Case studies based on the leaders’ experiences implementing health economic strategies in the US, Brazil, Colombia, and the UK will illustrate commonalities and differences in approaches. In true workshop fashion, participants will be asked to discuss their own experiences preparing pharmacoeconomic submissions for the Latin American region.
Patient-Reported Outcomes/PREFERENCE-BASED Research
W27: USING A CATALOGUE OF EQ-5D SCORES TO MODEL QALYS FOR COST-EFFECTIVENESS ANALYSES
Discussion Leaders: Patrick Sullivan PhD, MA, Associate Professor, Regis University, Rueckert-Hartman College for Health Professions, Denver, CO, USA; Mark J. Sculpher PhD, Health Economics and Director of the Programme on Economic Evaluation and Health Technology Assessment, Centre for Health Economics, The University of York, York, UK; Julia F. Slejko, PhD Candidate, University of Colorado Denver, Aurora, CO, USA; Vahram H. Ghushchyan PhD, Professional Research Assistant, University of Colorado Denver, Aurora, CO, USA
PURPOSE: To demonstrate how EQ-5D scores from a publicly-available catalogue of chronic condition scores and correlated uncertainty parameters can be used to calculate QALYs for cost-effectiveness analyses.
DESCRIPTION: Guidelines for many countries recommend the use of QALYs as the gold standard. The National Institute for Health and Clinical Excellence (NICE) in England and Wales has expressed a preference for the EQ-5D in developing QALYs for cost-effectiveness analyses. In addition, analysts typically ignore the uncertainty associated with the tariff estimation algorithm. However, directly eliciting EQ-5D scores for specific disease states can be cumbersome and expensive. Often, researchers attempt to find utility scores from the published literature to populate their models, but in many cases these estimates are not available or inappropriate. This workshop will begin with an overview of the recent development of a catalogue of several hundred EQ-5D scores for a wide variety of chronic ICD-9 and CCC codes that can be used to estimate QALYs. The catalogue currently provides preference scores for the U.K. and U.S. national populations. In addition, the presenters will discuss recent research producing uncertainty intervals from the underlying tariff algorithm. We will discuss how to correctly interpret the EQ-5D marginal disutilities from the catalogue and how to adjust for multiple comorbidities. How to incorporate novel research addressing the uncertainty associated with tariff valuation into probabilistic sensitivity analysis will also be discussed. Next, we will demonstrate how to use the existing publicly-available EQ-5D scores from this catalogue to calculate QALYs for cost-effectiveness analyses from the U.K. and the U.S. perspectives. Specific examples of how to utilize the chronic condition EQ-5D scores to calculate QALYs for cost-effectiveness models will be provided. Finally, workshop participants will be encouraged to discuss the general use of the catalogue scores as well as develop questions related to their own specific applications.
Use of Real World Data
W28: REGISTRIES AND EVIDENCE DEVELOPMENT
Discussion Leaders: Richard Gliklich MD, President and CEO, Outcome, Cambridge, MA, USA; Jean Slutsky PA, MSPH, Director, Center for Outcomes and Evidence, Agency for Healthcare Research and Quality, Rockville, MD, USA
PURPOSE: Discuss the evolving role of patient registries for evidence development including strengths and limitations; present activities by AHRQ in the area of registries and prospective observational studies; discuss the design, operation, and evaluation of registries, including new methods and challenges.
DESCRIPTION: Recent U.S. government initiatives including the 2007 Food and Drug Administration Amendments Act (FDAAA) and the 2009 American Recovery and Reinvestment Act (ARRA) have pushed the expanded use of registries and observational studies as part of the evidence base for evaluating the outcomes of care associated with a range of healthcare decisions. As the use of registries increases, researchers are increasingly faced with new issues and challenges with developing high quality programs. Organizations such as the Agency for Healthcare Research and Quality (AHRQ) have been tasked with initiatives to leverage or expand the available evidence base to address the varied needs of stakeholders and decision makers as well as to assist in development of appropriate methods to utilize these approaches. Discussion leaders will discuss the evolving role of registries and the potential for registries in evidence development. The goals of several federal initiatives in the area of patient registries and prospective observational studies will be described. Key elements in the design, operation, and evaluation of patient registries will be presented, including several case examples. Attendees will be encouraged to participate in discussion on the role for registries in evidence development and decision making; utilizing and designing registries for specific goals; applying good practice approaches to create high-quality registries; and addressing emerging challenges and concerns.
WORKSHOPS - SESSION V Wednesday, May 19, 2010: 1:45 PM-2:45 PM
Clinical Outcomes Research
W29: HOW TO SELECT APPROPRIATE CAUSAL EFFECT ESTIMATION METHODS IN NON-EXPERIMENTAL COMPARATIVE OUTCOMES RESEARCH: INSTRUMENTAL VARIABLE, PROPENSITY SCORE, MARGINAL STRUCTURE MODEL, DOUBLY ROBUST ESTIMATION, OR OTHERS?
Discussion Leaders: Peter Sun MD, PhD, Vice President, Kailo Research Group, Indianapolis, IN, USA; Boxiong Tang MD, PhD, Senior Director, Health Economics and Outcomes Research, Pfizer Inc, New York, NY, USA; Yang Zhao PhD, Senior Research Scientist, Global Health Outcomes, Eli Lilly and Company, Indianapolis, IN, USA
PURPOSE: 1. Explore common issues that lead to inappropriate use of causal effect estimation methods in comparative outcomes and pharmacoeconomics research. 2. Discuss and demonstrate different criteria or tools to select appropriate causal effect estimation methods, such as instrumental variable, propensity score, marginal structure model and doubly robust estimation. 3. Enable audience to discern and avoid common pitfalls that lead to the inappropriate selection of causal effect estimation methods.
DESCRIPTION: Over the last 10 years, the number of publications using causal effect estimation methods in comparative outcomes research has grown significantly. At the same time, inappropriate selection of individual causal effect estimation methods has become widespread and may lead to directional inconsistency of the same treatment effect across different studies. Different from previous workshops on causal effect estimation method, this workshop not only explores the common issues that may cause inappropriate selection of causal effect estimation methods, but also provides useful criteria and/or tools to help select the appropriate methods and improve the quality of comparative effectiveness research. Topics covered in this workshop include a brief conceptual framework of different causal effect estimation methods (i.e., instrumental variable, propensity score method, marginal structural model, and doubly robust estimation), common issues that may lead to inappropriate selection of various causal effect estimation methods, tools or techniques that can be used in method selections. A real world case study that compares the pros and cons of different causal effect estimation methods will be given, and interactive discussion or exercise will be encouraged. At the end of the workshop, the audience should be able to discern, avoid and solve the most common issues that often lead to the inappropriate choice of causal effect estimation methods.
Economic Outcomes Research
W30: ENRICHING BUDGET IMPACT MODELS FOR US PAYERS WITH RELEVANT CLINICAL OUTCOMES
Discussion Leaders: Ipek Özer-Stillman MS, Director, Abt Bio-Pharma Solutions, Lexington, MA, USA; Bjorn Bolinder MBA, Head, Primary Care, Global Health Outcomes, Merck & Co., Kenilworth, NJ, USA; John D. Whalen, Research Associate II, United BioSource Corporation, Lexington, MA, USA
PURPOSE: The purpose of this workshop is to introduce a comprehensive approach to address the value-messaging needs of pharmaceutical industry stakeholders in the U.S. and to communicate product value effectively through use of clinically-enriched budget impact models.
DESCRIPTION: Despite rumors to the contrary, public (and many private) payers in the US consider more than just financial "bottom line" results when making coverage and reimbursement decisions. Clinical outcomes also are important and, when taken together with financial or economic outcomes, help to communicate the "value for money" proposition. Budget impact models often are designed to communicate only the financial consequences of formulary additions and substitutions. By so doing, opportunities to effectively communicate product value can be missed. In this workshop, participants will learn how health economic evidence expressed in budget impact models can be enriched using clinical outcomes metrics to leverage product value and support market access. Discussion leaders will explain how to design and develop such models using an integrated approach that accounts for market realities and incorporates differing payers' perspectives for multi-market segments in the US. Examples will be used to present the benefits of these models for aligning internal stakeholders and for communicating evidence to decision makers.
Health Care Policy Development Using Outcomes Research
W31: OPTIMIZING STRATEGIC DIRECTION WITH SYSTEMATIC VALUE DEVELOPMENT
Discussion Leaders: Karen A. Joy, Director, Health Economics and Outcomes Research, IMS Health, Falls Church, VA, USA; Sonya J. Lewis RPh, MBA, Pharmacy Director, BCBS of Colorado/Nevada, Lafayette, CO, USA
PURPOSE: The purpose of this workshop is to demonstrate how to systematically develop the value of a product, beginning in early development to create fertile ground for achieving market access goals as a product transitions from clinical development to launch.
DESCRIPTION: Successful product launch depends on a number of factors, including how well a product’s value was cultivated from the early stages of development. We will present a systematic framework for developing and demonstrating product value that includes an “environmental assessment” that encompasses the perspectives and needs of developers’ internal and external stakeholders. The process begins with an understanding of the burden of illness, unmet needs, competitive landscape, pricing and reimbursement constraints, and future healthcare trends. Results from this review are then grounded in primary research with payers and medical providers. The end result of this process is a strategic plan that includes a timetable for health economic and outcomes studies that allow pharmaceutical companies to collect relevant and useful data, as well as straightforward ways to transform that data into actionable information for internal audiences and persuasive tactics to communicate value to external audiences. Examples from multiple therapeutic areas representing different value challenges will be presented. We will also include examples of effective communication of value to various stakeholders, including payers, medical providers, employers, and patients. We will invite the audience to participate in a value development exercise based on a composite case study.
Patient-Reported Outcomes/PREFERENCE-BASED Research
W6: DEFINING A RESPONDER: IMPLEMENTING THE PATIENT-REPORTED OUTCOME (PRO) GUIDANCE RECOMMENDATIONS
Discussion Leaders: Lori McLeod PhD, Head, Psychometrics, Patient-Reported Outcomes, RTI Health Solutions, Research Triangle Park, NC, USA; Ron D. Hays PhD, Professor of Medicine, Division of General Internal Medicine and Health Services Research, University of California at Los Angeles, Los Angeles, CA, USA; Susan Martin MSPH, Senior Director, Patient-Reported Outcomes, RTI Health Solutions, Ann Arbor, MI, USA; Sheri Fehnel PhD, Vice President, Patient-Reported Outcomes, RTI Health Solutions, Research Triangle Park, NC, USA
PURPOSE: The objective of this workshop is to describe the necessary steps to define the threshold of change required on a PRO instrument to identify a responder. Multiple approaches to define this threshold will be compared.
DESCRIPTION: In December 2009, the FDA released their final guidance for industry, Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims (US Department of Health and Human Services, 2009). This guidance includes general recommendations for deriving a responder definition. The recommendations specify that a responder threshold should be defined a priori using anchor-based methods. This threshold should then be used to guide interpretation of treatment benefits based on the PRO instrument. Distribution-based methods are discussed as providing support for the anchor-based threshold. In this workshop, we will begin with an overview of the conceptual difference between identifying a minimal important difference and defining a responder threshold. Then, we will provide recommendations on defining a responder. We will discuss the statistical significance of individual change and selection of appropriate anchors, as well as distribution-based approaches. Finally, we will discuss the use of cumulative distribution functions and how to apply this tool to provide further support for a responder threshold or to provide interpretation of PRO score changes in lieu of a specific threshold. We will conclude with an interactive discussion of specific applications, including potential next steps when the anchor-based threshold is not supported by the other approaches and comparison of cumulative distribution functions from different trials in support of the same endpoint.
W32: EXPANDING THE NUMBER OF ATTRIBUTES IN YOUR CONJOINT STUDY: USE OF LINKED-LATENT CONJOINT ANALYSIS
Discussion Leaders: David R. Walker PhD, Associate Director, Global Health Economics, Renal Division, Baxter Healthcare Corporation, McGaw Park, IL, USA; Jason Cole PhD, Director, Health Economics and Outcomes Research, Covance Market Access Services, San Diego, CA, USA
PURPOSE: Given the importance of conjoint surveys in understanding patients’ treatment preferences, pricing, and marketing decisions, there are marked benefits for a mathematically correct process for integrating disparate conjoint surveys. In light of respondent-burden and despite the demand for many attributes (and levels), conjoint studies are generally limited to a few attributes of interest (such as price, efficacy, and safety), which can limit the application to real-world behavior. With a minimal amount of additional work, computation, and preplanning, results from disparate conjoint studies can be linked by designing conjoint surveys to have a repeated domain (e.g., price).
DESCRIPTION: This workshop will provide a walkthrough on the process necessary to conduct a linked-latent conjoint survey and analysis (Dang, Cole, & Magidson; 2007), including: the benefits for industry in using large pools of attributes in conjoint studies; establishing an appropriate attribute to serve as the link; understanding the basic demands on sample size for linked-latent conjoint surveys; coding nuances and output differences necessary to interpret the linked-latent conjoint survey results; and limits of the linked-latent conjoint approach. Exercises will be conducted in small groups to design and interpret results from linked-latent conjoint analyses, both from practical and statistical viewpoints. (Dang, J., Cole, J. C., & Magidson, J. (2007, June). A tale of two (or more) surveys: The application of latent variable modeling for linking attributes on separate conjoint surveys. Paper presented at the European Survey Research Association second meeting, Prague, Czech Republic.)
Use of Real World Data
W33: THE GROWTH OF SOCIAL NETWORKS AND OPPORTUNITIES FOR OUTCOMES RESEARCH
Discussion Leaders: Elisa Cascade MBA, Vice President, iGuard, Falls Church, VA, USA; Eric Gemmen MA, Senior Director, Quintiles, Falls Church, VA, USA; Stephen Doogan, Head of Business Development, WeAre.Us, San Francisco, CA, USA; Murtuza Bharmal PhD, MS, Director, Quintiles, Falls Church, VA, USA
PURPOSE: To characterize the emergence of the patient as a stakeholder in healthcare decision making via social networks, present case examples of how to work with these patients to conduct outcomes research, and to discuss the pros/cons of patient-centric vs. traditional physician-centric data collection methods.
DESCRIPTION: Over the past five years, the US has experienced a shift in the relationship between patients and physicians towards an environment where patients are empowered to take control of their own healthcare. This change in the healthcare landscape has been fueled in part by the growth in availability of healthcare information on the internet and the emergence of Social Networks. The patient’s motivation to share feedback on their healthcare experience coupled with the development of the infrastructure needed to access patients directly has given rise to a new data collection method, called Patient-centric Studies. Patient-centric Studies are defined as recruitment of and data collection from patients directly without the need for physician sites. The patient-centric approach offers the potential for collection of health resource utilization, HRQoL, disability, and treatment satisfaction data much more rapidly and at a lower cost in comparison to traditional methods. Understanding when and how to use a patient-centric approach as is critical to successful study completion, particularly with respect to: how to design a protocol, informed consent, and data collection instruments suitable for patient-centric studies; when to apply for IRB approval or waiver; and sources for patient recruitment. The panelists bring the combined experience of social networks and the conduct of patient-centric studies, we will be able to share examples of successful implementation as well as raise issues for the group to discuss such as: patient privacy, pharmacovigilence, what constitutes “source” data, when to collect confirmatory physician and/or laboratory data, and what to do when patient and physician reported data are contradictory.
WORKSHOPS - SESSION VI Wednesday, May 19, 2010: 3:00 PM-4:00 PM
Clinical Outcomes Research
W34: THE USE OF OBSERVATIONAL EVIDENCE FOR COMPARATIVE EFFECTIVENESS RESEARCH: IS IT FIT-FOR-PURPOSE?
Discussion Leaders: Rachael Fleurence PhD, Senior Research Scientist, Center for Health Economics and Science Policy, United BioSource Corporation, Bethesda, MD, USA; Kyle Fahrbach PhD, Senior Biostatistician, United BioSource Corporation, Lexington, MA, USA; David Vanness PhD, Assistant Professor, Population Health Sciences, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA
PURPOSE: CER requires us to answer the right questions for decision-making in routine clinical practice. While observational evidence has its limitations, it may fill in evidence gaps not addressed by RCTs. We will argue that all evidence should be considered, if it is “fit-for-purpose”.
DESCRIPTION: The CER agenda requires identifying relevant evidence and synthesizing it where appropriate to provide the right evidence on which to base decisions. Traditionally RCTs are considered to be the gold-standard at the top of the evidence hierarchy. Yet observational studies can provide generalizability often lacking in RCTs, as well as time and cost savings when RCT evidence is not available. Well-conducted observational studies may answer questions about effectiveness of interventions in actual clinical practice that are difficult with RCTs, and this evidence should be used as part of the evidence base for CER. In this workshop we will review the current use of observational evidence in the literature. We will start by describing the types of observational evidence that is available. We will briefly describe how bias and confounding are accounted for in individual observational studies. We will then review how observational evidence may be synthesized (e.g., use of raw data vs. pooled data, sub-group analyses and meta-regression to assess heterogeneity). Using observational evidence for decision-making is controversial, and we will review some of the criticisms leveraged against it, including discussing examples of studies comparing RCT pooled evidence with observational evidence (e.g. the Women Health Initiative findings). We conclude that observational evidence can be used but that strong methodological guidelines need to be followed. Audience participation will be solicited through the use of a simple worksheet showing comparisons of pooled estimates based on both observational studies and RCTs studies that will be the basis for a discussion on why there might be differences between them.
W35: IMPROVING THE PREDICTION OF MEDICAL RISK AND BENEFIT
Discussion Leaders: Patricia Cerrito DrPH, Professor, University of Louisville, Louisville, KY, USA; John Cerrito PharmD, Clinical Pharmacist, Kroger Pharmacy, Louisville, KY, USA
PURPOSE: No treatment or medication is 100% safe. There is always a trade-off between the risk of the disease and the risk of the treatment. We examine methods that are used to predict both risk and benefit, and to provide accurate trade-offs. We also discuss the problem of decision making when there is a misperception of risk.
DESCRIPTION: We examine the use of empirical data to investigate risk and benefit of treatments given patient conditions. As an example, we discuss the removal of the drug, Vioxx while allowing the drug, Celebrex to continue on the market. We want to examine the difference in predicted risk between the two drugs while using empirical data to improve upon the predictions. In addition, we examine the issue of rare occurrences and how they are usually not identified until post-marketing surveillance because of the needed sample sizes. At what point should physicians and patients make the decision concerning the trade off of risk versus benefit given the fact that most of the information is based upon predictions of both? We will discuss the implications of faulty risk assessments in medical decision making, and the consequences of faulty risk assessments. As an example, we will look to misperceptions in the treatment of ulcers, misperceptions in the use of HRT in the prevention of heart disease in women, and current misperceptions concerning stroke and medication benefit. Audience participation will include a discussion of the degree and occurrence of adverse events that should determine whether a drug is removed from the market.
Economic Outcomes Research
W36: FAILURE AT THE MEAN: THE IMPORTANCE OF REFLECTING HETEROGENEITY IN ECONOMIC EVALUATIONS
Discussion Leaders: Denis Getsios, Research Scientist, United BioSource Corporation, Halifax, NS, Canada; Phil McEwan PhD, Managing Director, CRC, Cardiff, UK
PURPOSE: Many pharmacoeconomic models rely on average values to specify input parameters and simplify clinical processes at the expense of considering variability in downstream outcomes. Incorporating variability in parameter estimates, clinical pathways and ultimate outcomes is critical to producing meaningful economic outcomes. This workshop will outline the failures of modeling that ignore important heterogeneity and describe methods that can be used to feasibly incorporate these into economic modeling studies.
DESCRIPTION: The workshop will examine issues that need to be considered when accounting for heterogeneity in key input variables such as treatment response, differences in population characteristics, and variations in medical decisions. The effect on predicted outcomes, how to define response, when to evaluate it, how responders might differ from non-responders in the absence of treatment, and what effects there are on subsequent treatment decisions and adherence will be discussed. A case study for a modeling study in Hepatitis C will be presented to illustrate the potential for error when modeling using average values. Further, to demonstrate the importance of considering variability in treatment response, a simple individual level simulation, built in stages, will demonstrate the impact on outcomes as each new element is considered. Participants will gain an understanding of the challenges in designing pharmacoeconomic evaluations that properly account for variability, but also be introduced to techniques that can be used to address these challenges while still keeping the scope of modeling work manageable. While of interest to researchers involved in designing health economic evaluations, the workshop is directed at health care decision makers and individuals commissioning pharmacoeconomic model evaluations, with the goal of highlighting the dangers of ignoring variability in economic evaluations and raising awareness that techniques are available to overcome these limitations.
Health Care Policy Development Using Outcomes Research
W37: USING INTERACTIVE PRESENTATIONS TO COMMUNICATE PHARMACOECONOMIC EVIDENCE TO PAYERS
Discussion Leaders: Mark Nuijten MD, PhD, MBA, Director, Ars Accessus Medica/Erasmus University Rotterdam, Jisp, The Netherlands; Gijs Hubben PhD, Health Economist, BaseCase Software, Berlin, Germany
PURPOSE: To explore how pharmacoeconomic evidence can be more effectively used in discussions with payers at the regional level. Economic models are more frequently used in the communications with regional payers, for example to perform a budget impact or business-case analysis for each individual payer. We will present an overview of case-studies that use models in this way, and identify the key challenges and potential solutions of using models at the regional level.
DESCRIPTION: The decentralisation of the health care decision making process in many countries resulted in an additional hurdle for new pharmaceuticals: inclusion in local formularies of payers. However, the decision criteria that payers apply are not explicitly stated and the process lacks transparency. An individual, case-by-case approach is needed. The approach we will explore in this workshop is to take economic models to the payer and perform a custom analysis for their setting. Economic models synthesise evidence from a wide range of sources, and model outcome reflects the consensus of the authors on numerous assumptions. But for such analyses be credible, consensus with payers on model assumptions is crucial. To build credibility with payers, a model needs to be explained and adapted to the local setting. Through dialogue and collaboration with payers, consensus can be reached on model outcome. By integrating the mathematical model into a presentation that is suitable to be used by non-experts, pharmacoeconomic evidence can be effectively used at the regional level.
Use of Real World Data
W38: IMPUTING HEALTH EXPENDITURE DATA ONTO EMR DATABASES: PROS AND CONS OF PROPENSITY MATCHING AND REGRESSION BASED TECHNIQUES
Discussion Leaders: John Rizzo PhD, Professor, Department of Economics, Stony Brook University, Stony Brook, NY, USA; Candace Gunnarsson EdD, President, S2 Statistical Solutions Inc, Cincinnati, OH, USA; Teresa Zyczynski PharmD, MBA, MPH, Research & Analytics Leader, GE Healthcare, Princeton, NJ, USA; Amy Ryan MS, Researcher, GE Healthcare Clinical Data Services, Princeton, NJ, USA
PURPOSE: To demonstrate how regression based technique and propensity matching can be valid methods of imputation across operational databases
DESCRIPTION: This workshop will discuss and analyze two important approaches for imputing health expenditure data onto health care databases that lack such information. In particular, it will employ both a regression-based imputation technique and propensity scoring methodology to assign yearly direct healthcare expenditures obtained from patients in the Medical Expenditure Panel Survey (MEPS) database onto a subset of patients in the GE Healthcare Centricity Electronic Medical Records (EMR). The MEPS database provides nationally-representative estimates of health care utilization and expenditures, health status, health insurance coverage, and sociodemographic and socioeconomic characteristics for the civilian, noninstitutionalized population in the United States. The MEPS sample was chosen as a nationally-representative sub-sample of the ongoing National Health Interview Survey (NHIS) conducted by the National Center for Health Statistics. It can be linked to the NHIS database as well. The MEPS survey respondents were interviewed in person. The surveys achieved response rates of approximately 75%. First, propensity score methods will be demonstrated to show how to match the MEPS database to the GE database. Match criteria will include sociodemographic characteristics and comorbidities that are common to both databases. We will discuss alternative matching regiments (e.g., 1-to-1, 1-to-2, 1-to-3, etc), considering the tradeoffs in added power versus decreased precision. Another method is to use this matched sample of MEPS data and take a regression based approach, whereby, multivariate regression models are estimated and coefficients that were estimated in MEPS could be applied to the GE database to obtain imputed values for annual insurer expenditures and annual out-of-pocket expenditures. The two methods will be compared and the limitations and advantage of each discussed.
|
