Presentation (CTI)

OBJECTIVES: Scenario analysis is central to characterising uncertainty in cost-effectiveness models, yet remains inconsistently implemented across HTA submissions. Our previous review of NICE oncology appraisals (ISPOR Europe 2022, MSR114) found frequent evidence assessment group (EAG) requests for additional analyses. We evaluated a retrieval-grounded LLM approach within a pre-submission EAG simulation framework to identify clinically and methodologically relevant omitted scenarios before submission.
METHODS: A reference standard was developed from completed NICE oncology single technology appraisals by extracting company-submitted scenarios and additional analyses requested or conducted by EAGs. For each appraisal, the approach reviewed the submitted model and evidence base to identify assumptions likely to be challenged during technical engagement or committee appraisal. Proposed scenarios were ranked and described by assumption, decision relevance, rationale, and model input. Two senior health economists assessed validity, materiality, traceability, and unsupported assumptions. Outcomes included recall of EAG-imposed scenarios, precision, unsupported-scenario rate, and novel-but-valid scenarios.
RESULTS: The approach identified most omitted scenarios subsequently requested or undertaken by EAGs. Recall was highest for recurring uncertainty domains from the 2022 review, including survival extrapolation, time horizon, treatment-effect waning, utilities, and treatment duration. Performance was weaker where analyses depended mainly on external clinical judgement rather than explicit model assumptions. All retained scenarios were traceable to submitted model inputs, with no unsupported inputs identified. Eight valid scenarios not assessed by either the company or EAG were also identified.
CONCLUSIONS: A retrieval-grounded LLM approach anticipated most EAG-imposed omitted scenario analyses in NICE oncology appraisals, supporting more systematic pre-submission uncertainty assessment. Used within an EAG simulation framework, it may help manufacturers identify likely reviewer concerns, strengthen model validation, and improve submission readiness while keeping prioritisation and interpretation under senior health economist oversight.