OBJECTIVES:

To assess the performance of artificial intelligence (AI) tools within the DistillerSR systematic review platform for title and abstract screening in a systematic literature review (SLR). The SLR goal was to identify utility values for an infectious disease.

METHODS:

The search strategy returned 6,847 references for review. References were assessed by two independent reviewers; a third analyst resolved conflicts. Two AI uses were tested: the AI acting as a reviewer (3 possible outcomes: correct, wrong, unsure) using different size training sets (from 10% to 70% of hits), and as a validation tool: the AI compared excluded hits with included ones and searched for accidental exclusions.

RESULTS:

The AI made a decision for 9 to 15% of the total review depending on the training set size. As the training set size increased, the percentage of correct AI decisions increased (from 16% to 31% with a training set of 685 and 4794 references respectively) but the total number of AI decisions decreased (from 982 to 685 answers with a training set of 643 to 4,794 references respectively). The level of discrepancies between the AI and humans was low, around 4%-6%, which is similar to better compared to an analyst. The AI Audit tool did not identify relevant studies excluded by mistake, but the references spotted by DistillerSR were interesting (e.g. duplicates of a relevant hit, references with title only requiring a manual search to exclude the article).

CONCLUSIONS:

The audit tool has been found to be useful in checking excluded hits; it reinforced our confidence in our selection of articles. AI is acting as a reviewer is promising; decisions made by the tool were of good quality however only a small proportion of the references were answered. Tools and methods for AI-based screening are evolving rapidly and will require ongoing testing.