Understanding the Application of Various Statistical Analysis Methods to Inform Comparative Effectiveness of Ultra Orphan Treatments: A Review of NICE HST Guidance

OBJECTIVES: National Institute for Health and Care Excellence (NICE) evaluate medicines for ultra rare diseases, having a prevalence <1 per 50,000 persons, by the Highly Specialised Technologies (HST) evaluation process. We reviewed NICE HST guidance to understand the role of different statistical analysis methods used to inform the comparative effectiveness of ultra orphan treatments.

METHODS: Manufacturer submissions, ERG reports and final appraisal documents for all HST submissions, available till June 2023, were reviewed to assess clinical evidence submitted, statistical analyses performed, ERG comments, and final recommendations.

RESULTS: Twenty-three HSTs were identified. Majority of the guidance (61%) were focused on endocrine, nutritional, and metabolic rare diseases. All submissions received positive recommendation, however, included a simple discount-based patient access scheme. The clinical evidence included in the submissions comprised of RCTs (56%), single arm trials (69%), non-RCTs (8%) and observational studies (17%). Overall, 22% submissions included pooled analyses, naïve indirect comparisons (17%), propensity-score matching analyses (13%), or an unanchored matching-adjusted indirect comparison (MAIC; 4%). Mostly population-adjusted indirect comparisons were not performed due to lack of relevant clinical data, heterogeneity of the included studies and non-availability of comparator data. In two submissions, an MAIC was attempted but was deemed infeasible due to small effective sample size and lack of covariates for adjustment. In 22% submissions, ERG recommended to conduct additional population-adjusted analyses to support the clinical effectiveness and reduce the uncertainty around submitted evidence.

CONCLUSIONS: A high clinical uncertainty was observed in majority of the HST submissions. A few submissions included pooled analyses, propensity score matching analyses, and naïve comparisons to support clinical effectiveness. Advanced methods like MAIC, simulated treatment comparison (STC), and multi-level-network meta-regression (ML-NMR) could not be utilised due to smaller sample size and lack of covariate suitable for matching/adjustments.