AI-Assisted Review of NICE Guidance for Highly Specialized Technologies: Economic Modelling of Ultra-Rare Conditions

OBJECTIVES: Reimbursement of treatments for ultra-rare diseases is associated with specific challenges, often due to limited availability of clinical and economic evidence. These challenges have led to the development of dedicated reimbursement frameworks, including Highly Specialized Technology (HST) processes. To better understand how ultra-rare conditions are modelled for the UK National Institute for Health and Care Excellence (NICE), we analyzed all 28 HST assessments published between January 2012 and May 2024.

METHODS: Human-supervised artificial intelligence (AI) (ChatGPT-4o, OpenAI) was used to extract clinical, statistical, economic analysis, and Evidence Review Group (ERG) or External Assessment Group (EAG) critique data from 28 NICE HST assessments. Manual extraction of three appraisals was conducted by two analysts independently to develop and validate prompts. All AI-extracted data were cross-checked by a third analyst before analysis.

RESULTS: Of the 28 submissions, 23 used a cost-utility analysis framework with cohort state transition model. Models used 3-10 health states categorized by intermediate endpoints such as body mass index (BMI)/BMI Z-score or functionality scores, with treatment effects captured through transition probabilities. Three used partitioned survival models, one used an individual patient-level model, and one used a cost-effectiveness framework expressed as Disability Adjusted Life Years (DALYs)-averted instead of Quality Adjusted Life Years (QALYs)-gained, due to a lack of robust utility data. Models typically employed discounting of 3.5% for both health benefits and costs, best supportive care as the comparator, and a lifetime horizon. Given the large proportion of single-arm clinical trials used in the submissions, comparator arm efficacy data were typically sourced from natural history studies or expert elicitation. The most widely-used methods for eliciting expert opinions were structured interviews, the Sheffield Elicitation Framework (SHELF), or Delphi procedure.

CONCLUSIONS: Analysis of HST submissions suggests modelling in ultra-rare diseases remains challenging. Use of simpler models, natural history studies, and expert elicitation can support efficient decision-making.