Objectives: The use of artificial intelligence (AI) may alleviate additional time and effort needed to conduct robust, fully comprehensive systematic literature reviews (SLRs) intended for submission to health technology assessment (HTA) bodies. This study aimed to determine the ability of AI to accurately identify relevant studies on costs, healthcare resource use (HCRU), economic evaluations (EE), and patient-reported outcomes (PRO).

Methods: Title/abstract screening from previously completed SLRs (2008-2018) on costs/HCRU, EE, and PRO in attention-deficit/hyperactivity disorder were replicated using DistillerSR AI reviewer. Sets of 50 and 150 references from each SLR were used to train the AI reviewer. Screening decisions for 3,201 references were compared between AI and human reviewers.

Results: With training sets of 50 references –completed as a calibration exercise by the human reviewers– approximately 77% of AI screening decisions were accurate and the AI correctly screened most (62-83%) records originally missed by human reviewers. However, several exclusions made by the AI were references that were ultimately included in the SLRs by human reviewers. These exclusion errors occurred most often with EE (85.7%) followed by costs/HCRU (58.3%), and PRO (46.8%). Increasing the training sets to 150 references improved the accuracy of AI screening to 84% for costs/HCRU and EE, and increased its ability to correctly identify EE (exclusion errors reduced to 30%). The accuracy did not improve for PRO, although AI was less likely to exclude studies that should have been included in the SLR.

Conclusions: Larger training sets impacted the ability of AI to accurately identify EE and cost/HCRU studies, but to a lesser extent for PRO. While this study assessed AI as a single reviewer in a small sample, the results suggest AI may be best utilized as a second reviewer or in conjunction with other DistillerSR AI capabilities to offset the human screening burden.