Expert Elicitation Techniques: Informing Application in Health Technology Assessment Decision Making

Danielle Riley, MSc, Aimée Fox, PhD, Alex Hirst, MSc, Anne Marciniak, MD, PhD, Louise Heron, MSc, MA (Cantab), Adelphi Values PROVE™, Bollington, Cheshire, UK

Introduction to the benefits of Expert Elicitation

Health technology assessment (HTA) agencies require evidence relating to the burden of illness, comparative effectiveness, cost-effectiveness, and/or the budget impact of new therapies or technologies to assess their value and inform patient access decisions. However, often the required evidence is lacking or can be uncertain due to lack of robust or long-term comparative clinical trial data, challenges in areas like rare diseases with their small and heterogeneous patient populations, or lack of published evidence. In these instances, leveraging the insights of experts who have adequate and appropriate subject knowledge on a particular topic can be useful to elicit key data necessary to inform healthcare decision making.

The inherent characteristics of the decision-making process for reimbursement and access means that uncertainty is an inevitable element of HTA assessments, since at the time of the evaluation, not all data are available (eg, long-term follow-up data). Expert elicitation is an evidence-generation technique that, when applied appropriately, can be leveraged to reduce the impact of uncertainty in HTA submissions, provide reliable information, and inform the appraisal in the absence of patient-level data.¹ The application of expert elicitation in recent years has become increasingly prominent in value assessments. Previous research has highlighted vast heterogeneity in the methodologies used and a lack of clear guidance on the topic.² In addition, there is a lack of evidence available to determine which methods are most appropriate for use in healthcare decision making.¹ Consequently, reference protocols, such as the Medical Research Council (MRC) protocol, have been developed to provide clarity on methods for collecting and using experts’ judgements and to consider when alternative methodology may be required in particular contexts.¹

Recommendations regarding the appropriate use and correct implementation of existing expert elicitation methodologies are required to allow this field to achieve its potential of informing healthcare decisions. As such, a review of literature in MEDLINE exploring the use of expert elicitation methods within HTA was conducted to assess in which contexts elicitation techniques had been used and to identify methodological and practical recommendations included in best practice guidance and HTA guidelines. The learnings were used to develop a framework to inform method selection.

Through the literature review, typical situations where expert elicitation was used were identified. We present here an overview of those potential uses to guide and support appropriate methodological choices across a variety of contexts for those seeking to conduct expert elicitation.

Expert elicitation is an evidence-generation technique that, when applied appropriately, can be leveraged to reduce the impact of uncertainty in HTA submissions, provide reliable information, and inform the appraisal in the absence of patient-level data.

Use of expert elicitation in HTA decision making

The review highlighted that expert elicitation is recognized as an valuable methodology within health economics and outcomes research (HEOR) and is accepted by numerous HTA bodies. The use of expert elicitation was identified in cases where traditional evidence synthesis methods (such as head-to-head comparison and the use of published data) were impractical or insufficient (Table).

Table. Overview of expert elicitation methods and example uses

Acceptability of expert elicitation in HTA decision making

How accepted is expert elicitation currently?

As noted in existing reference protocols, no standard guidelines exist for the conduct of expert elicitation in HTA and on the conditions for the acceptability of the results, but a number of generic guidance documents or reference materials have been developed to help inform appropriate method selection and study design.¹

Across countries, variation exists as to whether expert elicitation guidance is provided by HTA agencies (Figure 1). Most HTA agencies across countries listed provided at least some level of guidance; while some provided extensive details, such as Australia and Portugal, others simply acknowledged the acceptance of expert elicitation techniques in the absence of patient-level data. This is with the notable exception of the United States, where guidance on expert elicitation is not provided nor acknowledged.³

Figure 1. Summary of the availability of guidance on expert elicitation methods across HTA organizations

Guidelines refer to the use of expert elicitation techniques to inform HTA submissions, with example uses including defining clinical or unmet need, assessing potential alterations to clinical management pathways and algorithms, assessing clinical importance and patient relevance of outcomes, modifying patterns of healthcare resource use, predicting healthcare resource impact, estimating proportions and or probabilities in relation to outcomes of interest, and predicting treatment use following the emergence of a new therapy.⁴

HTA organizations advise that formal elicitation methods must adhere to best practice principles to minimize bias and adhere to validated processes to obtain results with the highest level of objectivity feasible. There is variation in the detail included. For example, in Australia and Portugal, the guidance is prescriptive and includes details relating to design, conduct, expert election, aggregation, and analysis. In contrast, other countries provide minimal details on how to include expert elicitation within HTA. An overview of excerpts from HTA guidelines is presented below in Figure 2 (detailed guidance from Australia and Portugal have not been listed in this article; however, they can be found in HTA guidelines).^4,5

Figure 2. Excerpts from HTA guidelines

Commonalities exist across the recommendations provided in HTA guidelines; for example, the guidelines in France refer to the Australian guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee to outline exemplar methodologies, ranging from questionnaire-based surveys involving a statistical analysis, to the qualitative or quantitative summary of interviews across a selected panel of experts. Of note, the latter is in line with the methods outlined in the MRC protocol, suggesting convergence in the preferences of HTA bodies.^4,6 In contrast, some guidelines in Europe lean toward specific methodological directions, such as preference for structured quantitative methodologies in the National Institute for Health and Care Excellence (NICE) guidelines.⁷

How will expert elicitation be perceived in the future?

As the European Commission has adopted a framework of rules and guidance for joint clinical assessments (JCA) as part of its effort to implement the EU health technology assessment regulation, it is currently unclear where expert elicitation will sit within the expert involvement framework. However, it is likely that expert elicitation techniques will gain acceptance provided they follow accepted best practice guidance, such as the MRC protocol, and if guidelines are developed in the future, they will align with existing HTA guidelines.

It is likely that expert elicitation techniques will gain acceptance provided they follow accepted best practice guidance, such as the Medical Research Council protocol, and if guidelines are developed in the future, they will align with existing HTA guidelines.

Application in HTA decision making

As noted above, there is increasing interest in expert elicitation techniques as a means to inform healthcare decision making and reduce the uncertainty inherent to HTA assessments.

As uncertainty within technology submissions remains a major challenge for decision makers, techniques—such as expert elicitation—that can help mitigate the level of uncertainty associated with the development of new technologies provide an opportunity for health technology developers to improve the quality of their submissions—provided the studies are conducted and results utilized appropriately.

To supplement the recommendations and preferences identified within HTA guidelines, we have prepared an overview of potential uses as an elicitation road map (Figure 3) to guide appropriate and adequate methodology selection, study design, and implementation across an array of market access research questions, while following best practice principles detailed in published literature.

Figure 3. Road map of potential uses to help facilitate methodology choices

Call to Action

The wide-ranging requirements needed for HTA submissions to support patient access at an early stage of products’ life cycles, combined with the intricacies of challenges encountered in complex indications and innovative types of technologies, leads to the introduction of an inevitable level of uncertainty in HTA decision making.

This uncertainty can often impact HTA decision making negatively or lead to suboptimal decisions as decision makers look to reduce their risk, ultimately reducing patient access to useful therapies. As we continue to aim for earlier patient access to therapies, the uncertainty included in the process is substantial, especially when regulatory pathways are accelerated. Efforts should therefore be made to mitigate the extent of this uncertainty to prevent delays or restrictions to access.

Expert elicitation is an evidence-generation technique that, when applied appropriately, can be leveraged to reduce the impact of uncertainty on HTA submissions to provide stakeholders with reliable information to inform their decision making in the absence of patient-level data. With the increasing use of expert elicitation in HTA decision making, it provides an opportunity for patients and patient representatives to ensure their voices are integrated into evidence-generation activities, making evaluations of new technologies relevant, comprehensive, and patient-centered.

References

1. Bojke L, Soares M, Claxton K, et al. Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study. Health Technol Assess. 2021;25(37):1-124.
2. Soares MO, Sharples L, Morton A, Claxton K, Bojke L. Experiences of structured elicitation for model-based cost-effectiveness analyses. Value Health. 2018;21(6):715-723.
3. ICER’s Reference Case for Economic Evaluations: Elements and Rationale. Institute for Clinical and Economic Review (ICER). https://icer.org/wp-content/uploads/2024/02/ICER-Reference-Case_2024.pdf. Updated September 30, 2024. Accessed July 15, 2025.
4. Guidelines for preparing a submission to the Pharmaceutical Benefits Advisory Committee. Pharmaceutical Benefits Advisory Committee. https://pbac.pbs.gov.au/content/information/files/pbac-guidelines-version-5.pdf. Published September 2016. Accessed October 5, 2023.
5. Perelman J, Soares M, Mateus Céu, et al.  Orientações Metodológicas. INFARMED. https://www.infarmed.pt/documents/15786/4001413/. Published December 12, 2019. Accessed July 15, 2025.
6. Choices in Methods for Economic Evaluation—HAS. Haute Autorité de Santé. https://www.has-sante.fr/upload/docs/application/pdf/2020-11/methodological_guidance_2020_-choices_in_methods_for_economic_evaluation.pdf. Published April 6, 2020. Accessed October 5, 2023.
7. NICE health technology evaluations: the manual. National Institute for Health and Care Excellence (NICE). https://www.nice.org.uk/process/pmg36/chapter/introduction-to-health-technology-evaluation. Published January 31, 2022. Accessed May 1, 2024.
8. Guidelines for the Economic Evaluation of Health Technologies: Canada—4th Edition. Canada’s Drug and Health Technology Agency. https://www.cadth.ca/guidelines-economic-evaluation-health-technologies-canada-4th-edition. Accessed October 5, 2023.
9. Guideline for economic evaluations in healthcare. National Health Care Institute. https://english.zorginstituutnederland.nl/documents/2024/01/16/guideline-for-economic-evaluations-in-healthcare. Published January 16, 2024. Accessed June 25, 2024.