Leveraging AI for Evidence Synthesis: Assessing the Accuracy and Efficiency of AI-Supported Data Extraction and Reporting
Author(s)
Bianca Jackson, MPH1, Stephan Martin, MPH1, Kevin Kallmes, MA, JD2, Grace E. Fox, PhD1.
1OPEN Health HEOR & Market Access, New York, NY, USA, 2Nested Knowledge, St. Paul, MN, USA.
1OPEN Health HEOR & Market Access, New York, NY, USA, 2Nested Knowledge, St. Paul, MN, USA.
OBJECTIVES: Integration of artificial intelligence (AI) into evidence synthesis workflows promises improvements in accuracy and efficiency of literature reviews. However, evaluation of AI capabilities beyond title/abstract screening is needed. This study assessed the performance of an AI-supported data extraction tool in extracting quantitative data from a sample of studies reporting treatments for B-cell acute lymphoblastic leukemia (ALL).
METHODS: An AI-supported data extraction tool (Smart Meta-Analytical Extraction, Nested Knowledge) was used to extract survival, response, and safety data from text and tables within peer-reviewed observational studies and interventional trials evaluating ALL treatments. Quantitative extractions were validated by a human reviewer against manually extracted data. AI-generated summaries were produced for each study based on a prompted list of study details (e.g., study design, sample size, interventions/comparators) and independently reviewed for accuracy and comprehensiveness.
RESULTS: AI extraction was performed on a sample of 7 studies. The AI identified response-related and safety outcomes more readily than survival-related outcomes, potentially because of more standard language (e.g., “complete response,” ”response rate”) used by authors, compared with outcomes that were heterogeneously reported (e.g., “overall survival” vs. “OS”). The AI was less accurate in finding data reported in text or figures instead of tables. AI-generated textual summaries accurately conveyed key study elements, including study design, interventions/comparators, and results. AI extraction took ~5 minutes for all extractions before curation, saving over 95% of extraction time compared with an estimated manual extraction time of 25 minutes per study.
CONCLUSIONS: These data suggest that AI tools can substantially reduce time for initial data extraction in evidence synthesis. AI-generated summaries were accurate and demonstrated utility for data summarization and cross-checking. However, limitations in the completeness of AI-supported extractions underscore the need for human-in-the-loop quality checks by expert reviewers. Future testing of these extraction features can be repeated following the upcoming launch of a new version.
METHODS: An AI-supported data extraction tool (Smart Meta-Analytical Extraction, Nested Knowledge) was used to extract survival, response, and safety data from text and tables within peer-reviewed observational studies and interventional trials evaluating ALL treatments. Quantitative extractions were validated by a human reviewer against manually extracted data. AI-generated summaries were produced for each study based on a prompted list of study details (e.g., study design, sample size, interventions/comparators) and independently reviewed for accuracy and comprehensiveness.
RESULTS: AI extraction was performed on a sample of 7 studies. The AI identified response-related and safety outcomes more readily than survival-related outcomes, potentially because of more standard language (e.g., “complete response,” ”response rate”) used by authors, compared with outcomes that were heterogeneously reported (e.g., “overall survival” vs. “OS”). The AI was less accurate in finding data reported in text or figures instead of tables. AI-generated textual summaries accurately conveyed key study elements, including study design, interventions/comparators, and results. AI extraction took ~5 minutes for all extractions before curation, saving over 95% of extraction time compared with an estimated manual extraction time of 25 minutes per study.
CONCLUSIONS: These data suggest that AI tools can substantially reduce time for initial data extraction in evidence synthesis. AI-generated summaries were accurate and demonstrated utility for data summarization and cross-checking. However, limitations in the completeness of AI-supported extractions underscore the need for human-in-the-loop quality checks by expert reviewers. Future testing of these extraction features can be repeated following the upcoming launch of a new version.
Conference/Value in Health Info
2025-11, ISPOR Europe 2025, Glasgow, Scotland
Value in Health, Volume 28, Issue S2
Code
SA61
Topic
Methodological & Statistical Research, Study Approaches
Topic Subcategory
Literature Review & Synthesis
Disease
Oncology