A Framework for Expert Elicitation Related to Counterfactual Values: A Case Study of Elicitation to Inform Subgroup Analyses in Clinical Trials.
Author(s)
Geue C1, Hawkins N1, Oakley J2, Grieve R3, Hernandez M4, Carpenter J5
1University of Glasgow, Glasgow, UK, 2University of Sheffield, Sheffield, UK, 3London School of Hygiene and Tropical Medicine (LSHTM), London, UK, 4University of Sheffield, Sheffield, NYK, UK, 5London School of Hygiene and Tropical Medicine, London, UK
OBJECTIVES:Subgroup analyses are important components of precision medicine, but typically lack power to distinguish between false negatives and false positives.Hence, prior plausibility, elicited from experts is an important criterion for their credibility. There is little guidance on how this should be elicited, leaving decision-makers to judge plausibility of subgroup effects after trial results have been published. Additionally, counterfactuals are typically unobservable, outside clinical trials, for subgroup effects. Existing frameworks for expert elicitation (Cook’s classical method, the Sheffield Elicitation Framework (SHELF), modified Delphi scheme) do not cover elicitation of unobservable counterfactuals. We introduce a framework for expert elicitation that classifies elicitation according to underlying epistemiology of the elicitation: i) observablebut not recorded; ii) currently unobserved but observable in future; iii) unobservable counterfactuals. METHODS:Based on prior analysis of likely epistemiology, we developed a progressive three-step process: 1) determine whether the experts have relevant knowledge, 2) where experts have relevant knowledge; elicit qualitative judgements regarding existence and direction of subgroup effects, 3) where experts have relevant knowledge; elicit quantitative judgements regarding the magnitude of subgroup effects. We test our framework using a case study of a clinical trial into permissive hypotension in critically ill patients over 65. RESULTS: Experts seemed reasonably confident to state the direction of subgroup-treatment effects for prognostic factors, but less confident to make a judgement about the likely direction of subgroup-treatment effects for predictive factors without having prior evidence. CONCLUSIONS: It appears likely that in many cases experts will not be able to state strong directional priors for subgroup effects. It is important to explore the form and extent of experts’ knowledge in elicitation exercises and to framing questions accordingly. A progressive approach is likely to be more informative than trying to force experts to speculate where they have little relevant knowledge to draw upon.
Conference/Value in Health Info
2021-11, ISPOR Europe 2021, Copenhagen, Denmark
Value in Health, Volume 24, Issue 12, S2 (December 2021)
Code
POSB421
Topic
Health Technology Assessment, Methodological & Statistical Research, Organizational Practices
Topic Subcategory
Academic & Educational, Decision & Deliberative Processes, Survey Methods
Disease
No Specific Disease