Accuracy and Efficiency of Automated or Artificial Intelligence Tools in Systematic Literature Reviews: A Systematic Literature Review

OBJECTIVES: The aim of this systematic literature review (SLR) was to assess the comparative accuracy and efficiency of commercially available automated tools to support screening and data extraction tasks versus human reviewers in the conduct of a SLR.

METHODS: Electronic database searches conducted in Embase®, from inception to February 2024, were supplemented by interrogation of grey literature to identify relevant literature. Studies exploring generic algorithms or models that were not clearly associated with a commercially available platform or large language models were excluded. Search results were uploaded to LaserAI for screening and data extraction. Two independent reviewers performed screening, with data extraction performed by one reviewer and checked by a second. Quantitative data for accuracy and efficiency of tools versus human reviewers and a qualitative summary of factors influencing rates were extracted.

RESULTS: In total, 45 studies investigating at least one of the following tools were included: Abstrackr (n=10), ASReview (n=6), DistillerSR (n=13), EPPI-Reviewer (n=2), ExaCT (n=1), LaserAI/Dextr (n=3), PICO Portal (n=1), Rayyan (n=5), Research Screener (n=2), RobotAnalyst (n=3), RobotReviewer (n=2), Robot Screener (n=1), SWIFT-Active Screener (n=2), SWIFT Review (n=2), and SYMPRO (n=1). Most tools focused on accuracy and efficiency outcomes for title and abstract screening (n=42), with consideration of full paper screening (n=4), data extraction (n=2), and risk of bias assessments (n=1) comparatively limited. Authors reported time and cost savings through implementation of a platform, however, due to notable study design and outcome heterogeneity, the impact on accuracy was less evident. Factors influencing accuracy and efficiency potential included review type and size, training set, algorithm decision thresholds, and research questions complexity.

CONCLUSIONS: Commercially available tools with automation and/or artificial intelligence capabilities offered efficiency savings when compared with conventional human reviewer methodology, however, conclusions regarding comparative accuracy were less clear, highlighting a need for further research with aligned outcomes.