OBJECTIVES: Risk of bias (RoB) assessment is an important part of a systematic review and hence the production of health technology assessment. However, it is a time consuming, subjective and labour intensive process, and disagreements on a study’s RoB between reviewers are common. In response, novel software tools have emerged which aim to support this process. RobotReviewer is a free web-based machine learning system that aims to automate RoB assessments of randomised controlled trials (RCTs). RobotReviewer has been tested internally by its developers where it performed well, but to date we have not identified any published independent evaluations. We compared and evaluated RobotReviewer against the current standard for RoB assessment, defined as double, independent, human researcher assessment with disagreements resolved by a third reviewer.

METHODS: A case study has been undertaken where RobotReviewer was tested on a subset of RCT papers that had been previously assessed by two independent human researchers, as part of a systematic review. The results of the automated (i.e. RobotReviewer) assessments were compared with the manual, human reviewer assessments for similarity at each RoB domain in accordance with the Cochrane Risk of Bias Tool.

RESULTS: CONCLUSIONS: This study has provided practical insight into the current effectiveness of employing a machine learning system to support RoB assessment in a systematic review, as part of a health technology assessment.