OBJECTIVES: The use of machine learning-driven AI tools has the potential to meet the increasing demand for comprehensive, up-to-date literature reviews. We aimed to explore the performance of two AI tools in commercially available platforms (DistillerSR and Rayyan), in making inclusion/exclusion decisions during title/abstract screening of an SLR.

METHODS: The sample set comprised a review (300 titles/abstracts) of dual versus triple-inhaled therapy in patients living with chronic obstructive pulmonary disease, with 100% screened by a single human reviewer and 10%, selected at random, screened by a second independent reviewer. AI training sets were varied from 20% to 50% (60–150 records), and the accuracy, precision and recall of the two AI-tools were compared with those of the human.

RESULTS: The accuracy of DistillerSR was 67%, 81%, and 89% with a training set of 20%, 30%, and 50%, respectively. Decisions were provided for 6%, 26%, and 29% of records in Rayyan (20%, 30% and 50% training set, respectively), with an accuracy of 40%, 96%, and 98%, respectively. Precision and recall were 33–84% and 19–84% in DistillerSR, and 27–100% and 75–80% in Rayyan, respectively. In a sensitivity analysis where human full-text review decisions relating to records with AI false negative (FN) decisions (resulting in incorrectly excluded titles/abstracts) were considered, recall increased to 88–100% in DistillerSR and 100% in Rayyan.

CONCLUSIONS: In AI training sets ≥90 records, approximately three-quarters of records were correctly included at title/abstract screening using either review platform. However, given all FNs were records that were excluded at human full-text review, no relevant information would have been lost in this instance. This highlights the potential of AI-assisted literature screening tools and the implications of their integration into existing literature review processes; however, the training sets we explored were relatively small.