OBJECTIVES: To assess the performance of artificial intelligence (AI) tools within the DistillerSR systematic review platform for title and abstract screening in a systematic literature review (SLR). The SLR objective was to identify the humanistic burden related with selected headache disorders.

METHODS: The search strategy returned around 925 references. References were assessed by two independent reviewers; a third analyst resolved conflicts. Two AI tools were tested: the AI acting as a reviewer using different size training sets (from 10% to 50% of hits), and as a validation tool, i.e. the AI compared excluded hits with included ones and searched for accidental exclusions.

RESULTS: The AI was trained by using training sets; for each test, it was asked to screen 50% of the identified references. The percentage of correct AI decisions made was high: between 75% and 82% (for the training set size of 30%). The AI Audit tool was used to test screening results and indicated that 22 abstracts could be excluded by mistake. An experienced analyst reviewed them again: none of them was excluded by mistake. However, the majority of articles identified by AI were methodological papers, including all of the words of interest regarding population and/or outcomes, but focusing on the tool being used e.g. for assessment of patients’ quality of life, tools’ reliability and validity.

CONCLUSIONS: The AI Audit tool has been found to be useful in checking excluded hits and it can be used to reinforce our confidence in the process of articles selection. AI acting as a reviewer is promising; independently of the training set size, AI was able to give good answers for more than 75% of analyzed references. AI tools are evolving rapidly and can be a future of literature reviews.