Why TSD26 Matters: How Accurate Was Clinical Expert Opinion in Estimating Long-Term Outcomes?
Author(s)
Rebecca Beale, MA1, Harry Freeman, MSc2.
1Costello Medical Consulting, Cambridge, United Kingdom, 2Costello Medical Consulting, London, United Kingdom.
1Costello Medical Consulting, Cambridge, United Kingdom, 2Costello Medical Consulting, London, United Kingdom.
OBJECTIVES: The recent National Institute for Health and Care Excellence (NICE) Decision Support Unit Technical Support Document (TSD) 26 details robust methods for eliciting long-term survival outcomes from clinical experts. Given the significant impact survival outcomes have on cost-effectiveness, it is vital predictions are accurate. We reviewed previous NICE appraisals to investigate the accuracy of clinical expert estimates of long-term survival outcomes prior to the publication of TSD26.
METHODS: NICE appraisals published between January 2020 and December 2023 in the four most common cancers (breast, lung, prostate and colorectal) were identified. The elicitation method and estimated progression free survival (PFS) or overall survival (OS) by clinical experts were extracted, where not redacted. Longer-term trial data were then extracted from the literature and compared to clinical expert estimates.
RESULTS: 49 appraisals were identified across breast (n=13), prostate (n=8), lung (n=23) and colorectal (n=5) cancer, of which 44 included expert opinion to inform survival curve selection. Expert opinion was elicited through telephone conferences (n=26), advisory boards (n=17) and surveys (n=1). Nine appraisals contained unredacted expert survival estimates with available subsequent long-term published trial data for comparison. Across these nine appraisals, 26 survival timepoints were evaluated. Clinical expert estimates were misaligned (defined as >5% difference from trial data) with realised survival at 13 of 26 timepoints, with underestimation at 12 timepoints and overestimation at one. For the remaining 13 timepoints, clinical expert survival estimates were within 5% of long-term trial data.
CONCLUSIONS: Clinical expert estimates were misaligned with realised long-term survival in half of assessed timepoints, with systematic underestimation in the vast majority of inaccurate cases. This suggests inherent limitations in current expert elicitation methods and potential conservative bias that may impact cost-effectiveness evaluations and reimbursement decisions. The results support implementing structured approaches as outlined in TSD26 to improve elicitation methods and potentially enhance prediction accuracy.
METHODS: NICE appraisals published between January 2020 and December 2023 in the four most common cancers (breast, lung, prostate and colorectal) were identified. The elicitation method and estimated progression free survival (PFS) or overall survival (OS) by clinical experts were extracted, where not redacted. Longer-term trial data were then extracted from the literature and compared to clinical expert estimates.
RESULTS: 49 appraisals were identified across breast (n=13), prostate (n=8), lung (n=23) and colorectal (n=5) cancer, of which 44 included expert opinion to inform survival curve selection. Expert opinion was elicited through telephone conferences (n=26), advisory boards (n=17) and surveys (n=1). Nine appraisals contained unredacted expert survival estimates with available subsequent long-term published trial data for comparison. Across these nine appraisals, 26 survival timepoints were evaluated. Clinical expert estimates were misaligned (defined as >5% difference from trial data) with realised survival at 13 of 26 timepoints, with underestimation at 12 timepoints and overestimation at one. For the remaining 13 timepoints, clinical expert survival estimates were within 5% of long-term trial data.
CONCLUSIONS: Clinical expert estimates were misaligned with realised long-term survival in half of assessed timepoints, with systematic underestimation in the vast majority of inaccurate cases. This suggests inherent limitations in current expert elicitation methods and potential conservative bias that may impact cost-effectiveness evaluations and reimbursement decisions. The results support implementing structured approaches as outlined in TSD26 to improve elicitation methods and potentially enhance prediction accuracy.
Conference/Value in Health Info
2025-11, ISPOR Europe 2025, Glasgow, Scotland
Value in Health, Volume 28, Issue S2
Code
HTA371
Topic
Health Technology Assessment
Topic Subcategory
Systems & Structure
Disease
No Additional Disease & Conditions/Specialized Treatment Areas