OBJECTIVES

: NICE single technology appraisal (STA) recommendations are informed by economic models submitted by companies. Technical errors identified by Evidence Review Groups (ERG) can reduce the credibility of the model for decision-making. This study aimed to conduct a comprehensive review of single technology appraisals (STA) to improve our understanding of the frequency and type of model errors reported by ERGs.

METHODS

: Technical error data were extracted from committee papers, including ERG reports and final appraisal determinations (FAD), published on the NICE website in 2017. STAs that were terminated or not initially undertaken in 2017 were excluded. Technical errors were classified into the following types: computational, logic, data handling, interpretation, other, or unknown. Errors were categorized as minor or major if reported as such by the ERG. Details of the model validation exercises conducted by the manufacturer were also extracted.

RESULTS

: A total of 79 STAs were reviewed of which 41 STAs met the study inclusion criteria. Companies reported having undertaken model validation in every STA. Only two ERG reports (5%) had no mention of any technical errors. 19 (46%) of the ERG reports identified 1 to 4 errors, 16 (39%) 5 to 9 errors and 4 (10%) reported more than 10 errors. A total of 206 errors were reported by the ERGs of which 9 (4%) were classified as ‘major’. The three most common errors reported by the ERGs were transcription (25%), computational (23%) and logic errors (23%). The FAD contained details of technical errors in 8 (20%) of the STAs reviewed.

CONCLUSIONS

: Despite validation being an integral part of the model development process, technical errors are frequently identified by ERGs. While errors may be an unavoidable part of the development process, there is a need for manufacturers to improve their model checking activities prior to submission to NICE.