OBJECTIVES: To assess the performance of artificial intelligence (AI) tools within the DistillerSR systematic review platform for title and abstract screening in a systematic literature review (SLR) of randomized controlled trials (RCTs). The SLR goal was to identify clinical trials assessing specified drugs used in ophthalmology.

METHODS: The search strategy returned 2,054 references. References were assessed by two independent human reviewers. Two AI uses were tested: the AI Preview and Rank and AI Review tool. By using both tools, DistillerAI trained two classifiers [SVM and Naïve Bayes] using a percentage of manually reviewed references and predict if unreviewed references should be included, excluded, or if the decision is not clear. Different training sample sizes were applied (5, 8, 12 and 15% of all references), and results were compared.

RESULTS: The AI decided for 24 to 30% of the total number of references depending on the training set size. The percentage of correct AI decisions was high (from 92% to 99%). When the training sample size increased from 5 to 12%, fewer studies were excluded by the program by mistake: from 0.53% for 5% sample size to 0.23% for 12% sample size. The level of discrepancies between the AI and humans was low, around 1%-5%, which is similar compared to a human analyst. When training sample size was 15%, the level of neutral decisions for studies that should be included was the lowest.

CONCLUSIONS: AI tools are useful and can be used partially as a reviewer. Decisions made by tools were of good quality, and 24-30% of the references were answered. As the percentage of studies excluded by mistake by tools was low (equal or less than 0.5%), the AI can be used for the preliminary screening of RCTs to quickly exclude non-relevant studies.