OBJECTIVES: To develop evidence generation strategic insights and lessons for ultra orphan drugs based on obvious mistakes identified by NICE during Highly Specialized Technology (HST) assessments.

METHODS: A systematic review of all NICE HST HTAs was conducted. Case studies were designed to develop insights and lessons from NICE’s committee review of company submission. Assessments with mention of evidence lacking face validity were selected and reviewed in detail. The product, indication, category of evidence and specific concerns by NICE were reviewed to develop insights and lessons learned for future submissions.

RESULTS: In 10 HST assessments NICE criticized the company submission for lacking face validity. In 6 assessments NICE questioned utility values-(1) gain values (2) caregiver utilities more than 1 (3) other sources of utilities (4) use of general population utilities, (5) use of values from specialist survey and (6) infusion disutility values . In 4 assessments NICE questioned the transition probabilities (1) choice of prior distributions for transition probabilities (2) mortality hazard ratios not in line with previous natural history study (3) assuming difference in probabilities rates for transplantation in treated versus untreated and (4) use of an outcome for transition probabilities (6MWT vs FVC).

CONCLUSIONS: Case studies of mistakes in previous submissions provides valuable lessons for companies to design evidence generation strategies for rare disease products.