Assessing the Accuracy of Artificial Intelligence in Conducting Systematic Literature Reviews
Author(s)
Aastha Radotra, MPH1, Geetank Kamboj, M.Pharm.1, Surabhi Aggarwal, M.Pharm.1, Rajpal Singh, PhD1, Hemant Rathi, MSc2.
1Skyward Analytics, Gurugram, India, 2EasySLR, Gurugram, India.
1Skyward Analytics, Gurugram, India, 2EasySLR, Gurugram, India.
Presentation Documents
OBJECTIVES: To assess the performance of artificial intelligence (AI) in conducting systematic literature reviews (SLRs) using EasySLR™ and to compare its accuracy against previously published, manually conducted SLRs.
METHODS: An exploratory analysis was performed to evaluate the accuracy of AI-driven SLRs. Published SLRs were included based on the following criteria: (1) a clearly defined search strategy, (2) a comprehensive list of included studies, and (3) the use of a single database (PubMed) for the literature search. Five SLRs meeting these criteria were selected for the analysis: Nicholas (2020), Bruurs (2013), Kaegi (2020), Kaegi (2022), and Sharifian-Dorche (2021). The original search strategies were replicated in PubMed, and the retrieved results were cross-checked against the included studies from original SLRs. Screening rules were developed based on the inclusion and exclusion criteria described in the original SLRs. AI-based screening and study selection were subsequently conducted using EasySLR™. The AI’s performance was assessed by calculating its accuracy, defined as the percentage of correctly identified included studies relative to the original SLRs.
RESULTS: The AI tool demonstrated accuracy rates ranging from 73% to 100%. Perfect accuracy (100%) was achieved for Kaegi (2020), while Bruurs (2013) and Nicholas (2020) showed accuracy rates of 90% and 84%, respectively. Lower accuracy rates were observed for Kaegi (2022) and Sharifian-Dorche (2021), at 74% and 73%, respectively. Variability in accuracy likely reflects differences in study design, search strategies, and complexity in inclusion-exclusion criteria across the selected SLRs.
CONCLUSIONS: This study demonstrates the potential of AI-driven tools like EasySLR™ to enhance the efficiency of systematic literature reviews (SLRs). While the AI showed high accuracy, variability in performance suggests that it may not yet be a standalone solution. Instead, AI could serve as a valuable second reviewer in future SLRs, complementing human efforts during the screening process.
METHODS: An exploratory analysis was performed to evaluate the accuracy of AI-driven SLRs. Published SLRs were included based on the following criteria: (1) a clearly defined search strategy, (2) a comprehensive list of included studies, and (3) the use of a single database (PubMed) for the literature search. Five SLRs meeting these criteria were selected for the analysis: Nicholas (2020), Bruurs (2013), Kaegi (2020), Kaegi (2022), and Sharifian-Dorche (2021). The original search strategies were replicated in PubMed, and the retrieved results were cross-checked against the included studies from original SLRs. Screening rules were developed based on the inclusion and exclusion criteria described in the original SLRs. AI-based screening and study selection were subsequently conducted using EasySLR™. The AI’s performance was assessed by calculating its accuracy, defined as the percentage of correctly identified included studies relative to the original SLRs.
RESULTS: The AI tool demonstrated accuracy rates ranging from 73% to 100%. Perfect accuracy (100%) was achieved for Kaegi (2020), while Bruurs (2013) and Nicholas (2020) showed accuracy rates of 90% and 84%, respectively. Lower accuracy rates were observed for Kaegi (2022) and Sharifian-Dorche (2021), at 74% and 73%, respectively. Variability in accuracy likely reflects differences in study design, search strategies, and complexity in inclusion-exclusion criteria across the selected SLRs.
CONCLUSIONS: This study demonstrates the potential of AI-driven tools like EasySLR™ to enhance the efficiency of systematic literature reviews (SLRs). While the AI showed high accuracy, variability in performance suggests that it may not yet be a standalone solution. Instead, AI could serve as a valuable second reviewer in future SLRs, complementing human efforts during the screening process.
Conference/Value in Health Info
2025-05, ISPOR 2025, Montréal, Quebec, CA
Value in Health, Volume 28, Issue S1
Code
MSR111
Topic
Methodological & Statistical Research
Topic Subcategory
Artificial Intelligence, Machine Learning, Predictive Analytics
Disease
No Additional Disease & Conditions/Specialized Treatment Areas