Assessing Literature Reviews in NICE Single and Multiple Technology Appraisals

OBJECTIVES: Systematic identification of evidence is a key step in the development of HTA submissions. The aim of this review was to assess the limitations of companies’ review methodologies as critiqued by NICE Evidence Review Groups (ERGs) in published technology reports.

METHODS: ERG reports available on the NICE website were retrieved by searching manufacturer submissions published 2019–2021. The manufacturer’s approach to the clinical systematic literature review in terms of methodology/reporting and the critiques highlighted by ERG was extracted. We grouped extracted information into the following categories: search strategy, study selection, data extraction, and quality assessment.

RESULTS: In total, 96 submissions were identified with associated ERG reports: 33, 33, and 30 in 2019, 2020, and 2021, respectively. The ERG noted limitations in 58% of submissions. Limitations were most frequently noted for search strategies (26 reports) mostly due to missing free-text terms or omission of searching trial databases. Twenty-eight submissions had issues with the study selection criteria: intervention/comparators (15/28), population (12/28), outcomes (7/28), study design (4/28), language (3/28). ERG reported a substantial risk of data extraction errors in seven submissions. Seventy-four submissions reported use of a quality assessment tool. For appraisal of RCTs, the most frequently used tool was the NICE checklist (29 reports), the tool in the Centre for Review and Dissemination guidance (22 reports) or Cochrane Risk of Bias tool (20 reports); the Downs and Black checklist (12 reports) was most frequently used to appraise observational studies. The ERG judged there was a likely risk of bias in ten submissions due to methodological/reporting issues.

CONCLUSIONS: Shortcomings in review methodology and reporting in submissions are frequently noted by the ERG, which contribute to uncertainty in the evidence and require additional clarifications from the manufacturer. Meeting the SLR requirements of NICE ERGs would improve efficiency of the HTA process.