Piloting Structured Expert Elicitation: An Exploratory Case Study of Practical Implementation
Author(s)
Susanne G. Værnø1, Eline Aas, PhD2, Francisco Oteiza, PhD3, Magnus Bangum, MSc4, Fredrik Rustad Holmboe, MSc5, Erik Magnus Sather, PhD3, Christoffer Bugge, PhD3.
1Economist, Oslo Economics, Oslo, Norway, 2University of Oslo, Oslo, Norway, 3Oslo Economics, Oslo, Norway, 4AbbVie, Oslo, Norway, 5AbbVie, Lysaker, Norway.
1Economist, Oslo Economics, Oslo, Norway, 2University of Oslo, Oslo, Norway, 3Oslo Economics, Oslo, Norway, 4AbbVie, Oslo, Norway, 5AbbVie, Lysaker, Norway.
OBJECTIVES: Structured Expect Elicitation (SEE) techniques can be valuable tools to support healthcare decision making, e.g when faced with evidence from single-arm trials or for long-term extrapolation. New guidelines have recently been published by ISPOR and NICE DSU TSD, however SEE evaluations still show a significant amount of heterogeneity. The purpose of this study was to evaluate an SEE pilot project, aiming to gain local experience on SEE.
METHODS: An SEE pilot was conducted for pemigatinib in treatment of bile duct cancer based on a single arm trial (FIGHT-202). Relevant oncologists were recruited in a systematic way. Following existing SEE protocols, an evidence brief, including information on the relevant studies and a description of the SEE methodology, was distributed prior to the elicitation. Individual elicitations were conducted to elicit PFS and OS at 6, 12 and 36 months for patients receiving pemigitanib or SoC. Uncertainty was elicited using the Roulette method. A group consensus workshop followed, in which anonymized responses were presented and a projection agreed upon. Feedback from experts on the SEE methodology was collected throughout the pilot.
RESULTS: Seven experts were recruited for elicitation, where of five attended the group consensus workshop (three in person, two digitally). Three improvements to the implemented methodology were suggested.First, hybrid format consensus meetings may lead to skewed degrees of participation. Exclusively in person or digital participation should be the norm. Second, the degree of preparation that experts showed during their individual and consensus elicitation varied. An overview of the key materials at the start of each interview may be valuable. Finally, the consensus elicitation presented all individual estimates as well as an average for discussion due to time constraints. While efficient, the experts tended to agree with this average distribution, suggesting a potential anchoring effect.
CONCLUSIONS: This evaluation provides lessons towards refining SEE implementation in practice.
METHODS: An SEE pilot was conducted for pemigatinib in treatment of bile duct cancer based on a single arm trial (FIGHT-202). Relevant oncologists were recruited in a systematic way. Following existing SEE protocols, an evidence brief, including information on the relevant studies and a description of the SEE methodology, was distributed prior to the elicitation. Individual elicitations were conducted to elicit PFS and OS at 6, 12 and 36 months for patients receiving pemigitanib or SoC. Uncertainty was elicited using the Roulette method. A group consensus workshop followed, in which anonymized responses were presented and a projection agreed upon. Feedback from experts on the SEE methodology was collected throughout the pilot.
RESULTS: Seven experts were recruited for elicitation, where of five attended the group consensus workshop (three in person, two digitally). Three improvements to the implemented methodology were suggested.First, hybrid format consensus meetings may lead to skewed degrees of participation. Exclusively in person or digital participation should be the norm. Second, the degree of preparation that experts showed during their individual and consensus elicitation varied. An overview of the key materials at the start of each interview may be valuable. Finally, the consensus elicitation presented all individual estimates as well as an average for discussion due to time constraints. While efficient, the experts tended to agree with this average distribution, suggesting a potential anchoring effect.
CONCLUSIONS: This evaluation provides lessons towards refining SEE implementation in practice.
Conference/Value in Health Info
2025-11, ISPOR Europe 2025, Glasgow, Scotland
Value in Health, Volume 28, Issue S2
Code
EE609
Topic
Economic Evaluation, Health Policy & Regulatory, Health Technology Assessment
Topic Subcategory
Value of Information
Disease
Oncology