OBJECTIVES: Targeted literature reviews require significant time and effort due to the vast amount of available scientific evidence. However, artificial intelligence (AI) could be used to efficiently identify relevant literature. This research evaluates the performance of the Rayyan AI tool (ie, Rayyan) for title/abstract screening and reports potential time-savings associated with an AI-assisted process.

METHODS: A large targeted literature review (8,755 references) previously screened by human reviewers was identified; Rayyan was trained using 3 subsets of the total references (5%, 10%, and 20%). Based on the training set, Rayyan predicted the relevance of the remaining references using a 5-level system ranging from “most likely to exclude” to “most likely to include.” Rayyan’s relevancy ratings were compared to the original inclusion/exclusion decisions to calculate sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV). Screening time was compared for an AI-assisted process vs human reviewers.

RESULTS: When references with Rayyan ratings of “most likely to include”, “likely to include”, and “no recommendation” were included, sensitivity was consistently high, ranging from 93%-97% across all training sets. Specificity increased with training set size at 34%, 52%, and 61% for the 5%, 10%, and 20% training sets, respectively. Accuracy ranged from 38%-63%, PPV ranged from 9%-13%, and NPV was 99% for all training sets. Time-savings also increased with greater training set size. The largest time-savings were reported for the 20% training set, where the AI-assisted process resulted in a 46% decrease in hours spent on title/abstract screening.

CONCLUSIONS: AI-assisted screening using Rayyan was highly sensitive and resulted in considerable time-savings. Therefore, this is a promising method for increasing screening efficiency for targeted literature reviews. Future research should confirm the performance and time-saving benefits of AI-assisted screening across targeted literature reviews varying in size (number of references) and topics of interest.