MetaSLR in Automating the Data Collection for Systematic Literature Reviews
Author(s)
Barinder Singh, RPh1, Gagandeep Kaur, M.Pharm2, Rajdeep Kaur, PhD2, Diaby Karam, PhD3, Sumeet Attri, MPharm2.
1Pharmacoevidence, London, United Kingdom, 2Pharmacoevidence, SAS Nagar Mohali, India, 3Department of Pharmaceutical Sciences, University of Florida, Florida, FL, USA.
1Pharmacoevidence, London, United Kingdom, 2Pharmacoevidence, SAS Nagar Mohali, India, 3Department of Pharmaceutical Sciences, University of Florida, Florida, FL, USA.
Presentation Documents
OBJECTIVES: The aim of this study was to compare the performance of the automated tool, MetaSLR, with the traditional human review process. By leveraging the data from previously conducted SLR of economic evaluations (EEs) of health interventions in ADHD, this study seeks to validate the accuracy and efficiency of MetaSLR tool to automate the screening process and improve productivity compared to conventional methodologies.
METHODS: The EMBASE® database was searched to identify EEs published in the English language assessing adult patients with ADHD. A subject Matter Expert (SME) with a decade of experience in SLR prepared and optimized the prompt according to the predefined inclusion and exclusion criteria. The LLM Claude Sonnet 3.5 with an optimized prompt was deployed using a Python script to evaluate the citations as included or excluded. SME evaluated screening results to check the agreement level of human and AI reviewers.
RESULTS: A total of 4,581 citations retrieved from EMBASE® were screened by the AI model Claude Sonnet 3.5. The AI model achieved an overall accuracy of 94.8% compared to the human reviewer, with a sensitivity of 92% and specificity of 94.8%. During the initial title and abstract screening phase, the inclusion rate for the AI model was 3.4% higher than that of the human reviewers. At the full-text screening stage, the AI model included additional articles and did not miss any relevant publications. Both methodologies successfully identified all relevant studies. However, the time required for screening differed significantly between the human reviewers and AI model.
CONCLUSIONS: The findings demonstrate that MetaSLR tool using Claude Sonnet 3.5 enhances the efficiency of the SLR process by significantly reducing the time required for data collection while maintaining high accuracy and precision.
METHODS: The EMBASE® database was searched to identify EEs published in the English language assessing adult patients with ADHD. A subject Matter Expert (SME) with a decade of experience in SLR prepared and optimized the prompt according to the predefined inclusion and exclusion criteria. The LLM Claude Sonnet 3.5 with an optimized prompt was deployed using a Python script to evaluate the citations as included or excluded. SME evaluated screening results to check the agreement level of human and AI reviewers.
RESULTS: A total of 4,581 citations retrieved from EMBASE® were screened by the AI model Claude Sonnet 3.5. The AI model achieved an overall accuracy of 94.8% compared to the human reviewer, with a sensitivity of 92% and specificity of 94.8%. During the initial title and abstract screening phase, the inclusion rate for the AI model was 3.4% higher than that of the human reviewers. At the full-text screening stage, the AI model included additional articles and did not miss any relevant publications. Both methodologies successfully identified all relevant studies. However, the time required for screening differed significantly between the human reviewers and AI model.
CONCLUSIONS: The findings demonstrate that MetaSLR tool using Claude Sonnet 3.5 enhances the efficiency of the SLR process by significantly reducing the time required for data collection while maintaining high accuracy and precision.
Conference/Value in Health Info
2025-05, ISPOR 2025, Montréal, Quebec, CA
Value in Health, Volume 28, Issue S1
Code
MSR107
Topic
Methodological & Statistical Research
Topic Subcategory
Artificial Intelligence, Machine Learning, Predictive Analytics
Disease
SDC: Neurological Disorders