Assessing the Reliability of Data From Non-Randomized Studies: Where Are We Now?

OBJECTIVES: Risk of bias (RoB) tools aim to identify systematic error or deviation from the truth in primary studies. While RoB assessment (RoBA) of randomised controlled trials (RCTs) is well established, assessment of other trial designs is less standardised. We investigated which tools are currently used for RoBA in systematic reviews (SRs) of non-randomised studies. To consider how RoBA has changed over the past 20 years, we compared our findings with those of Deeks et al (2003), who conducted an evaluation of RoBA tools in SRs including non-randomised intervention studies.

METHODS: A pragmatic search of Medline identified 66 2023 SRs, including 940 non-randomised primary studies. Due to lack of consistency in review authors’ descriptions of primary studies, we conducted our own assessment, classifying them as non-randomised controlled trials (21), single arm trials (183), cohort studies (198), or case series (538).

RESULTS: Across 39 SRs including non-randomized comparative studies, ten different RoB tools were used. Six SRs (15%) conducted no RoBA of included studies. Across 65 SRs including non-comparative studies, 14 RoB tools were used, with 15 SRs (23%) conducting no RoBA of the included single group studies. For comparative and non-comparative studies, most SRs used a Newcastle–Ottawa, Joanna Briggs Institute (JBI), or MINORS tool for RoBA.

Compared with 2003, more authors are now conducting RoBA (86% v. 33% in 2003) and more authors are using an existing standardised tool (80% v. 14% in 2003). However, for up to 55% of the 2023 SRs, the choice of tool may not have been appropriate.

CONCLUSIONS: Specific tools used for RoBA of non-randomised studies have changed over the past two decades, with a positive trend towards a greater awareness of the importance of RoBA, and consistency in the tools used. However, matching each study design to the most appropriate tool remains challenging.