OBJECTIVES: Systematic literature reviews (SLRs) seek to answer research questions and form comprehensive, rigorous evidence-based conclusions. However, each step of an SLR is resource-intensive. To address SLR workload challenges, there are now multiple SLR applications that provide artificial intelligence (AI)-as-a-service capabilities (e.g. EPPI-Reviewer, DistillerSR, Abstrakr). However, it is unclear if researchers are utilizing the AI component of these applications. Our SLR assessed whether AI is being utilized in published SLRs.

METHODS: MEDLINE and EMBASE were searched systematically in June 2021 for SLRs that utilized AI. SLRs and scoping reviews were eligible if they addressed human healthcare-related questions and reported the utilization of AI and/or AI-as-a-service application (AIsAPP) in any of the SLR steps. Rayyan was used for abstract and full-text screening; data were abstracted into Google Sheets. Backwards citation-tracking and hand-searching were completed.

RESULTS: Fifty-six studies (reported in 59 publications) were included. Forty-nine studies used AIsAPP, while seven used bespoke algorithms. The most frequently utilized AIsAPPs were Rayyan (n=22) and DistillerSR (n=11), and Python packages (n=7) were used in most of the bespoke algorithms. Forty studies involving an AIsAPP did not mention how/if AI was used. Of the remaining 16 studies, 13 used AI for semi-automation, two for full-automation, and for one study, how AI was used was unclear. Semi-automation was mainly used for screening (n=12) and extraction (n=2). Full automation was used for abstract screening in the two studies.

CONCLUSIONS: Few SLRs reported utilizing AI in the SLR process. In contrast to current PRISMA guidelines, when an AIsAPP was utilized, details on automation steps were often not described. Despite the increasing effort to automate SLRs, it seems AI is not yet common practice. Further research should evaluate the limitations and barriers of fully incorporating and reporting the utilization of AI as standard in SLRs.