OBJECTIVES: To assess the performance of artificial intelligence (AI) tools within the DistillerSR systematic review platform for title and abstract screening in a systematic literature review (SLR). The SLR objective was to identify the economic burden related with selected headache disorders.

METHODS: The search strategy returned around 1082 references. References were assessed by two independent reviewers; a third analyst resolved conflicts. Two AI tools were tested: the AI acting as a reviewer using different size training sets (from 10% to 50% of hits), and as a validation tool: the AI compared excluded hits with included ones and searched for accidental exclusions.

RESULTS: The AI was trained by using training sets; for each test it was asked to screen 50% of the identified references. The percentage of correct AI decisions made was high: between 77% and 79% (79% was for the training set size of 30% and 40%). The AI Audit tool was used to test screening results and indicated that 15 abstracts could be excluded by mistake. An experienced analyst reviewed them again: none of them was excluded by mistake. However, the majority of references identified by AI were duplicates of previously included abstracts, or there was no abstract available in the software so analysts made their final decisions based on abstracts identified by hand searches.

CONCLUSIONS: The AI Audit tool has been found to be useful in checking excluded hits and it can be used as an additional quality-check in the process of articles selection. AI acting as a reviewer is promising; independently of the training set size, AI was able to give good answers for more than 75% of analyzed references. AI tools are evolving rapidly and can be a future of literature reviews.