OBJECTIVES: Assessing study risk of bias (RoB) is vital in health technology assessment (HTA). However, it is unclear how companies account for high-risk studies in their submissions. By reviewing National Institute for Health and Care Excellence (NICE) Technology Appraisals (TAs), we aimed to determine how study RoB impacts HTA submissions and critique.

METHODS: Randomised controlled trial (RCT) evidence in a random sample of approximately 50% of NICE TAs (2019–May 2023; excluding terminated and multiple TAs) were reviewed. Company submissions and Evidence Assessment Group (EAG) reports were reviewed to determine RoB assessment tools used, RoB results, indirect treatment comparison (ITC) considerations and EAG critique.

RESULTS: Of 130 TAs reviewed, 106 reported RoB assessment of RCT evidence. The most frequently used checklists were ‘Centre for Reviews and Dissemination’ (57/106) and Cochrane’s RoB 1 tool (13/106). Only one TA reported including a high-risk pivotal clinical trial.

The EAG commented on the RoB assessments for most TAs (100/106), commonly agreeing with the company’s RoB findings (38/100) or agreeing but noting areas of improvement (49/100). Disagreement (7/100) and unclear judgements (6/100) were rare.

Pairwise meta-analyses or ITCs were performed in 73/106 TAs. The company typically did not specify if high-risk studies were included (52/73); a minority (16/73) reported only including low-risk studies and one TA did not assess RoB of studies in the ITC. In 4/73 TAs, the company reported the inclusion of high-risk studies in their ITC; of these, one TA conducted a sensitivity analysis removing high-risk studies following an EAG request.

CONCLUSIONS: Study RoB was rarely considered in evidence synthesis for NICE TAs and in subsequent EAG critique. However, RoB assessment results were often reported in appendices and were seldom readily accessible, making it difficult to determine if high-risk studies were adequately considered or if included studies were mostly low-risk.