Lack of Face Validity in Health Technology Assessments: Systematic Review of NICE Assessments and Lessons Learned

OBJECTIVES: The objective of this analysis was to systematically review 2023-2024 health technology assessments (HTAs) which lacked face validity according to NICE.

METHODS: A systematic review of all 2023-June 2024 NICE HTAs was conducted. Assessments in which NICE mentioned a lack of face validity were included in the review. Committee comments related to face validity were extracted and categorized by indication, type of evidence, and reason for the comment. Lessons and insights were developed for future HTA submissions.

RESULTS: During 2023-June 2024, there were 137 Health Technology Assessments (HTAs) published by NICE. Overall, there were 19 assessments (14%) in which NICE mentioned 62 times that the presented evidence lacked face validity. HTA indications included Cancer (9), Rare Diseases (4), Infectious Diseases (2), Endocrine Disorders, Ophthalmology, Multiple Sclerosis, and Inflammatory Bowel Disease. Comments were related to incorrect utility values (52%), survival estimates (26%), QALY estimates (16%), model structure (6%), comparative efficacy assumptions (6%), and network meta-analysis (6%). Nearly all of the errors could have been avoided by either using a team approach for developing protocols, validation by comparison with other assessments, or by a quality control check.

CONCLUSIONS: The review of previous assessments provides valuable lessons for evidence generation strategy and helps to avoid common mistakes.