Does the Future Belong to MCDA?
A key challenge for health care decision makers is balancing the multiplicity of medical, social, and economic factors that have a bearing on their choices. These factors, and the importance attached to them, often vary from one decision to another and between the stakeholders. Ensuring accountability requires that the process for determining these factors and their relative importance is transparent. Multi-criteria Decision Analysis (MCDA) is one approach to this end that has recently received much attention.
MCDA covers a range of methods that structure decision problems such that the relevant evaluation criteria and their relative importance are explicit. In doing so MCDA can better inform decisions. Since the first attempts in the 1960s , MCDA has been applied in many settings, including transport, environmental protection, construction, defence and finance . To date, however, formal application in health care has been limited [3, 4].
This article provides a brief introduction to MCDA, outlines current proposals for its use in health care, and spotlights related challenges and opportunities for industry.
What is MCDA and Why Does It Interest Decision Makers?
The term ‘MCDA’ is used to refer to a range of different methods, and it is important to be clear about which definition of MCDA is being adopted. One definition of MCDA is a method used to structure group decision making . This approach is concerned with eliciting and making transparent the judgements made in the decision making process. An alternative, broader definition of MCDA is the set of methods that seek to score, weight and ultimately aggregate the various criteria into an overall composite measure of benefit . The second definition is inclusive of the first, but also includes a range of alternative approaches to weighting criteria, such as stated preference techniques. In the remainder of this paper, the latter definition of MCDA is adopted.
MCDA can inform a range of health care decisions – such as manufacturers’ judgements to invest in compounds, regulatory approvals, reimbursement decisions, health authority resource allocation decisions, and clinicians’ prescription decisions. MCDA can support these decisions in a number of ways, including [6,7]:
- Improving the transparency, predictability and consistency of decisions.
- Facilitating the incorporation of patients’ values in decision making.
- Supporting the communication of the benefits, risks, and costs of treatments.
- Informing the design of data collection.
- Understand differences in viewpoints between stakeholders.
- Sharpening signals to industry about what matters to decision makers.
The following four steps are common to all MCDA methods: identifying options, defining and weighting relevant criteria, and scoring each option on each criterion. Each of the steps in MCDA presents methodological challenges: Which options should be considered? How should the criteria be selected? How should weights be assigned and who should be responsible for this? How should options be scored on the criteria? How should uncertainty be assessed? These questions have been addressed in many different ways by the various MCDA methods , but two key differences are often used to distinguish methods. First, whether the result of the MCDA is a quantitative overall score, or whether the MCDA stops short of such a score and a structured deliberation of the data is undertaken instead. Second, if a quantitative score is produced, what method is used to estimate the weights required to combine criteria to generate this score.
The next two sections consider how MCDA is being considered for two areas of health care decision making – authorisation and reimbursement.
MCDA and Marketing Authorisation
High profile withdrawals of drugs over the past decade have led to a renewed focus on drug safety . These concerns about safety are only heightened by a drug assessment process that “does not include an explicit, consistent, transparent, and aggregate quantification of the risks and benefits and lacks clarity pertaining to the role of specific factors in the recommendations” . As a consequence both the US Food and Drug Administration (FDA) and the European Medicines Agency (EMA) are considering new ways to weigh the benefits and risks of drugs, including, notably, MCDA.
In 2006, the Institute for Medicine Report on Drug Safety recommended that the Centre for Drug Evaluation and Research (CDER) at the FDA develop a systematic approach to benefit-risk assessment (BRA) . As a consequence, enhancing BRA in regulatory decision-making is now one of the Prescription Drug User Free Act’s (PDUFA) Reauthorisation Performance Goals (http://www.fda.gov/ downloads/forindustry/userfees/prescriptiondruguserfee/ucm270412.pdf). The precise nature of the BRA method that will be adopted is not yet known; but this is expected to be announced in early 2013. Early communications suggest that the FDA’s preferred approach will be more qualitative than quantitative . Data is collected to populate a grid designed to standardise the way that benefits and risks are described, but no weighting of these data are undertaken to generate an overall benefit-risk score.
The EMA has provided more detail on how it proposes to undertake BRA. The Committee for Medicinal Products for Human Use (CHMP) was set up to provide recommendations on ways to improve the methodology, transparency, consistency and communication of BRA. It advised that a structured, but mainly qualitative, approach be used . It also recommended further research to develop BRA methods. Accordingly, the EMA initiated five work packages to develop tools and processes for balancing multiple benefits and risks to support informed, science-based regulatory decision making about medicinal products . Work Packages 1 to 3 reviewed BRA practice within the EU regulatory network; assessed the applicability of relevant frameworks and quantitative approaches; and field-tested preferred methods . The conclusions were that decision analysis provides a sound theoretical basis; that the so-called PrOACT-URL (Problem formulation, Objectives, Alternatives, Consequences, Trade-Offs, Uncertainties, Risk Attitude and Linked Decisions) framework should be employed; and that a quantitative model could be developed to support decisions .
Work Package 4 re-emphasised the importance of the PrOACT-URL framework and identified the EMA’s preference for effect tables that draw on reviews of relevant studies in order to score interventions, and quantitative MCDA for more contentious cases where the benefit-risk balance is marginal . Work package five will pilot these tools and processes, and provide for relevant training.
The EMA has gone a long way to specify how MCDA can be used to support BRA. In essence, it believes MCDA should be employed where decisions are contentious, that it should be based on a quantitative assessment of drugs against multiple criteria, and that this assessment should form the basis for scoring and weighting by workshop participants. Even so, the agency’s recommendations leave several questions unanswered. For example, who sets the criteria? How should users ensure criteria meet MCDA requirements, such as preference independence – the idea that an option’s score on one criterion can be determined independently of its score on other criteria? What methods are appropriate for populating effect tables? Exactly how should weights be elicited? Furthermore, EMA’s recommendations differ from some practice currently employed by industry. For instance, while EMA seem to prefer expert-based weights generated through workshops, there are examples of industry eliciting patients’ weights via surveys . Furthermore, a range of alternative quantitative approaches have been identified by the ISPOR Risk-Benefit Management Working Group .
Partly motivated by the need to answer these questions, a number of initiatives have been launched to further developed BRA methods. These often involve collaboration between industry, regulator, and academia. Examples of such initiatives include the Innovative Medicines Initiative’s (IMI) Pharmacoepidemiological Research on Outcomes of Therapeutics in a European Consortium (PROTECT) programme , the work of the Benefit Risk Action Team (BRAT) , and the CASS work . These initiatives hold out the promise of greater standardisation in the use of MCDA to inform BRA, which will allow the industry to better plan their investments and the corresponding evidence generation.
MCDA and Health Technology Assessment (HTA)
As with BRA, HTA faces the challenge of weighing the various costs, risks, and benefits of a drug. To date, the formal evidence generation undertaken to inform HTA has focused on only a portion of the risks and benefits that stakeholders consider relevant for this setting. In the UK, for instance, NICE’s reference case requests a cost-effectiveness analysis that quantifies the health benefits of a drug as far as these can be captured by the quality-adjusted life year (QALY). Stakeholders, however, are often interested in other, very different, sources of value . As a consequence, it has been argued that MCDA should be adopted to ensure HTA takes an appropriate account of all relevant factors in reaching judgements .
The EVIDEM Collaboration (https://www.evidem.org/) was launched to respond to this increasing demand for MCDA in HTA. Having reviewed decision-making processes in 20 jurisdictions, EVIDEM identified 15 criteria relevant to HTA and was designed to fulfil MCDA requirements . These criteria were used as the basis for an MCDA framework to inform HTA. The framework specifies that each criterion needs to be weighed by experts using a five-point scale. The scores for each alternative are then quantified using best practice synthesis methods. Once the criteria have been quantified, experts use this data to score the criteria on a four-point scale. These quantitative criteria are supplemented with qualitative contextual criteria intended to focus decision-makers’ attention on “colloquial” forms of evidence.
The authors of the EVIDEM framework highlight drawbacks within the framework, including a weighting and scoring system with potentially low discriminatory power, and the violation of the requirement for non-redundancy by the inclusion of cost-effectiveness as a criterion even though its components are themselves included as separate criteria . Despite such limitations, pilots of the framework have concluded that it can support deliberations as part of the appraisal of technologies .
The EVIDEM framework adopts the structured decision making definition of MCDA. Whether this is the appropriate form of MCDA for HTA, or whether, for instance, patient or public values should be used to weight criteria, is currently the subject of a consultation by NICE . The debate about MCDA and HTA in the UK tends to be framed around the question of how to broaden the benefits considered in HTA beyond the health gains captured in the QALY. Others, such as the Institute for Quality and Efficiency in Health Care (IQWiG) in Germany and the Patient-Centred Outcomes Research Institute (PCORI) in the U.S., have rejected the QALY. The challenge of weighing the benefits and risks of technologies still remains, and similar methodological questions inform the debate in these countries. For instance, IQWiG has explored conjoint analysis and the analytic hierarchy process as methods to prioritize and weigh patient-centred outcomes . IQWiG has not committed to either of these methods. Rather the Federal Joint Committee may request that manufacturers employ one of these methods where health economic analysis is submitted.
In recognition of the call for HTA with a broader perspective, a number of recent initiatives and studies have explored the possibility of incorporating several criteria into decision making, and will influence the nature of any MCDA incorporated into HTA. Emblematic of these is the value-based pricing (VBP) initiative in the UK , which plans to formally assess drugs based on their innovative nature, broader social value, and the severity of the illness being considered, as well as their cost-effectiveness. Similar concerns in Sweden led the National Pharmaceutical Strategy to emphasize that health investments should be judged against criteria relating to environmental sustainability, world class medical outcomes, equitable care and innovativeness. The UK’s VBP initiative would have important implications for MCDA methods. Following NICE’s preference for weighting endpoints based on general public preferences , the UK Department of Health has commissioning various academic institutions to undertake population surveys to generate cost-effectiveness weights for different severities of disease.
There is increasing support for using MCDA to support health care decisions to ensure these are more structured, consistent and transparent. It is not clear; however, which MCDA methods will become standard in the BRA and HTA processes. In this regard, the existing literature provides some indications as to the MCDA methods that may eventually be requested by decision makers, but much more detail is required before we can be sure what these methods will be. Further research and consultation is required in this area.
Ultimately, different approaches to MCDA will probably be adopted to support BRA and HTA. This assumption reflects the fact that the aims, and thus the criteria relevant to these processes differ, with HTA being concerned, for example, with a broader set of values, including equity and innovation. Also, while the EMA’s current framework suggests that BRA might lead to appraisal-specific weights, it seems likely that weights generated for HTA will be applied across appraisals and even across therapy areas.
The uncertainty surrounding the precise role of MCDA in BRA and HTA presents both a challenge and an opportunity to industry. Without clarity on the methods and processes, it is difficult to plan for the emergence of MCDA. Industry cannot know, for example, what data to collect; when these might be needed; how to process them; what weighting and scoring methods might need to be followed; and what pitfalls to avoid. These unknowns, however, also provide opportunities for industry to influence which MCDA methods are adopted and to research the implications of alternative methods. MCDA even holds out the ground-breaking possibility of moving away from the QALY, or at least using more transparent and appropriate weighting of its components, as well as other factors to formulate a new, better composite measure.
Since similar debates about MCDA methods are ongoing for both BRA and HTA, there are opportunities to identify and take advantage of synergies between the evidence required for both these decision points. In particular, alignments between these processes to produce efficiencies in the evidence-generation process are increasingly desirable as it becomes clearer that market authorisation can no longer exist in isolation from reimbursement decisions, and cooperation between regulators and HTA bodies is already on the rise . One example of this – a pilot program in Sweden between the national medicines agency (MPA) and the reimbursement body (TLV) – was judged as a step in the right direction; however, it was insufficiently co-ordinated such that processes happened more in parallel than with close integration . The interest in MCDA provides an opportunity to progress this agenda further, improving the transparency, rigour and consistency of BRA and HTA, but also to bring these processes closer together.
Thanks to Kelly Davis, Denis Getsios and Ike Iheanacho from UBC for their comments.
- Köksalan M, Wallenius J, Zionts S. Multiple criteria decision making. From early history to the 21st century. World Scientific, 2011.
- Communities and Local Government, Multi-criteria analysis: a manual. 2009. London: The Department for Communities and Local Government.
- Maciosek MV, Coffield AB, Edwards NM, et al. Priorities among effective clinical preventive services: Results of a systematic review and analysis. Am J Prevent Med 2006;31:52-61.
-  Marsh K, Dolan D, Kempster J, Lugon M Prioritising investments in public health: A multi criteria decision analysis, 2012 (in press), JPublic Health.
-  Cross JT and Garrison LP (2008) Challenges and Opportunities for Improving Benefit-Risk Assessment of Pharmaceuticals from an Economic Perspective. London: OHE. http:// www.ohe.org/publications/article/challenges-and-opportunities-for-improving-benefit-risk-assessment-38.cfm
-  NICE (2012), Briefing paper for methods review workshop on structured decision making. Available from: http://www.nice.org.uk/media/C67/40/
- TAMethodsGuideReviewSupportingDocuments.pdf. [Accessed September 28, 2012].
- Guo JJ, Pandey S, Doyle J, et al. A review of quantitative risk-benefit methodologies for assessing drug safety and efficacy – report of the ISPOR Risk-Benefit Management Working Group. Value Health 2010:13:5:657-66.
- IOM, The Future of Drug Safety: Promoting and Protecting the Health of the Public. 2006. Available from: http://www.iom.edu/Reports/2006/The-Future-of-Drug-Safety-Promoting-and-Protecting-the-Health-of-the-Public.aspx. [Accessed September 28, 2012].
- Frey P, Benefit-Risk Considerations in CDER: Development of a Qualitative Framework. 2012, The US Food and Drug Administration. Available from: http://www.fda.gov/downloads/ AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/CDER/UCM317788.pdf . [Accessed September 28, 2012].
- EMA, Report of the CHMP working group on benefit-risk assessment models and methods. 2007. Available from: http://www.ema.europa.eu/docs/en_GB/document_ library/Regulatory_and_procedural_guideline/2010/01/WC500069668.pdf. [Accessed September 28, 2012].
- EMA, Benefit-risk methodology project. Work package 4 report: Benefit-risk tools and processes, 2012. EMA/145646/2012.
- Mohammed, A.F., Hauber, A.B., Levitan, B. & Coplan, P. Patients’ trade-off preferences for migraine treatments. Podium Presentation at the Third Biennial Conference of the American Society of Health Economists. Ithaca, NY, 2010.
- Auclert L, PROTECT-Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium, Seminaire IFIS, Paris, 5 July 2011.
- Levitan B and Andrews EB, Example Application of PhRMA BRAT (Benefit-Risk Action Team) Framework, Assessing Benefits and Risks of Medicinal Products in Regulatory Decisions, DIA, November, 2009.
- Liberti L, MaxAuslane N and Walker SR, Progress on the Development of a Benefit/ Risk Framework for Evaluating Medicines. Availabel from: http://cirsci.org/system/files/ private/2010FocusLiberti_0.pdf. [Accessed November 30, 2012].
- Golan O, Hansen P, Kaplan G, Tal O. Health technology prioritization: Which criteria for prioritizing new technologies and what are their relative weights? Health Pol 2011;102:126-35.
- Delvin N and Sussex J. Incorporating multiple criteria into HTA. London: Office of Health Economics, 2011.
- Goetghebeur MM, Wagner M, Khoury H, et al. Bridging health technology assessment (HTA) and efficient health care decision making with multicriteria decision analysis (MCDA): Applying the EVIDEM framework to medicines appraisal. Med Decis Making 2012;32:376.
- Tony et al. Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada. BMC Health Serv Res 2011;11:329.
- Danner M, Hummel JM, van Manen JG, et al. Integrating patients’ views into health technology assessment: Analytical hierarchy process (AHP) as a method to elicit patient preferences. Int J Technol Assess Health Care 2011;4:1-7.
- Department of Health. A new value-based approach to the pricing of branded medicines. A consultation. 2012. Available from: http://www.dh.gov.uk/en/Consultations/ Liveconsultations/DH_122760. [Accessed September 28, 2012].
- NICE, Guide to the Methods of Technology Appraisal. 2008. London: NICE.
- Kermanu F, Rasi at the EMA: Redefining Benefit/Risk As Staggered Approvals Rise. The Pink Sheet DAILY, Jun 4, 2012 (a).
- Kermanu F, NICE Scientific Advice Program May Seek Broader Application Through Boosting Regulator Link-up. The Pink Sheet DAILY, Jul, 2, 2012(b).