Are Best Practice Methods for Expert Elicitation Being Applied in NICE Highly Specialized Technology (HST) Submissions? A Review and Evaluation

OBJECTIVES: Expert elicitation is commonly used when there is a paucity of evidence for healthcare decision-making, especially for Highly Specialized Technologies (HST) for rare conditions. This method, however, can introduce uncertainty in or bias the results. Bojke et al. (2021) and INFARMED (Portugal’s health authority; 2019) issued guidance on how to best conduct elicitation. The current study critically examined how structured expert elicitation has been reported in HST submissions to the National Institute of Health and Care Excellence (NICE).

METHODS: A targeted review was conducted to identify NICE HST submissions involving expert elicitation between January 2015 and March 2023. Submissions without fully documented committee papers or being resubmitted were excluded.

RESULTS: Nineteen HSTs were included (four exclusions). Interview was the most common expert elicitation technique (14 HSTs), followed by advisory board (seven), Delphi panel (five) and survey (three). Most HSTs (79%) involved fewer than 10 experts. Health-related quality of life (64% of HSTs) and healthcare costs (58%) were the most frequent topics. Only one submission met most principles (six of nine) described in Bojke et al. (2021) and INFARMED (2019). Most submissions were transparent (summarized the process [95%], reported results [79%]; Principle 1) and provided information that addressed the decision problem (89%; Principle 2), with consistency among the process, context and capacity of the decision-making entities (84%; Principle 3). Experts were familiar with the target quantity or had relevant experience (Principle 7), as stated in 84% of submissions. Little information was provided regarding methods to explore uncertainty (Principle 4), to recognize and act on biases (Principle 5) and to account for differing levels of normative expertise (Principle 9).

CONCLUSIONS: Best practices for transparency and applicability of the information to the decision problem were consistently applied in NICE HST submissions, but little was stated on how to handle uncertainty and biases.