OBJECTIVES: To assess the performance of artificial intelligence (AI) tools within the DistillerSR systematic review platform for title and abstract screening in a systematic literature reviews (SLRs). The SLRs objectives were to identify cost-effectiveness analyses (CEAs) and clinical evidence related with the chronic liver disease.

METHODS: The search strategy returned around 200 references for each review. References were assessed by two independent reviewers; a third analyst resolved conflicts. Two AI tools were tested: the AI acting as a reviewer using different size training sets (from 10% to 50% of hits), and as a validation tool: the AI compared excluded hits with included ones and searched for accidental exclusions.

RESULTS: The AI was trained by using training sets; for each test it was asked to screen 50% of the identified references. As the training set size increased, the percentage of correct AI decisions made for the review of CEAs increased (from 77% to 87%). Regarding the review of the clinical evidence, the percentage of correct decisions was also high: between 78% and 93% (93% was for the training set size of 30%). The AI Audit tool was used to test screening results of the SLR of CEAs, and it did not identify any relevant study excluded by mistake.

CONCLUSIONS: The AI Audit tool has been found to be useful in checking excluded hits and it can be used as an additional quality-check of analysts’ work to reinforce our confidence in the process of articles selection. AI acting as a reviewer is promising; decisions made by the tool were of good quality and independently of the training set size, AI was able to give good answers for more than 75% of analyzed references. Tools and methods for AI-based screening are evolving rapidly and can be a future of literature reviews.