OBJECTIVES: To assess artificial intelligence (AI) tools within the DistillerSR systematic review platform for title and abstract screening in systematic literature reviews (SLRs). The SLRs goals were to identify utility values for an infectious disease, cost-effectiveness models (CEMs) for treatment of vascular disease, and randomised controlled trials (RCTs) assessing specified drugs used in ophthalmology.

METHODS: Four AI tools were tested: Test (acting as a reviewer), Audit (audit of excluded references), Preview and Rank (preview the predictions of unscreened references), and Review (screen references based on the predictions). These tools checked the results of titles and abstracts screenings of utilities, CEMs, and RCTs (6,847, 851, and 2,054 references, respectively). Before the tests, references were assessed by two independent reviewers and a third analyst resolved conflicts.

RESULTS: When SLR of utilities was analysed, correct decisions made by the AI Test tool increased with the training set (from 16% to 31% with a training set of 10% and 70% references, respectively). The AI Audit tool did not identify any relevant utility studies excluded by human analysts by mistake; however, it was useful to double-check studies selected by the tool. AI Preview and Rank and AI Review tools were used to test results of CEMs and RCTs reviews. Results showed that 92-99% of AI decisions were correct, depending on the training set size and review. The level of discrepancies between the AI and humans was low, around 1%-3% for CEMs and 1%-5% for RCTs, similar to an analyst.

CONCLUSIONS: The audit tool has been found to be useful to reinforce our confidence in the selection of articles. AI Preview and Rank and AI Review tools are useful as they can be used partially as a reviewer. Tools and methods for AI-based screening are evolving rapidly and will require ongoing testing.