Presentation (CTI)

OBJECTIVES: Evidence synthesis approaches such as targeted literature reviews (TLRs), scoping reviews, and umbrella reviews are rigorous but lower risk than HTAs, making them strong candidates for AI-assisted workflows. AI tools show promise in extracting qualitative information, binary outcomes, and organizing complex data into evidence maps. Combined with structured human QC, AI can streamline evidence identification, extraction, and visualization without compromising accuracy. This study evaluated an AI-assisted workflow for qualitative data extraction and evidence mapping within an umbrella review of safety outcomes in solid tumors.
METHODS: We conducted an umbrella review using an AI-assisted workflow. Custom extraction tags standardized qualitative data collection across domains (study characteristics, objectives, criteria, interventions, comparators, and binary safety outcomes). AI-generated extractions were produced via Nested Knowledge using GPT-4, followed by 100% human QC. Performance was assessed by extraction volume, accuracy versus human validation, and time savings compared with manual processes.
RESULTS: AI-assisted extraction generated 210 qualitative data points across 10 fields for 23 studies (91.3% extraction rate). Accuracy for study characteristics was 90%, with errors mainly due to missing extractions. For binary safety outcomes (207 expected data points), AI identified 150 (72.5%), with overall accuracy of 62.3% when missing extractions were treated as incorrect. Precision for successfully extracted data was 86.0%. Time to complete qualitative extraction and evidence mapping was reduced by 80% versus manual processes. Reviewers reported improved efficiency and reduced repetitive tasks.
CONCLUSIONS: AI-assisted qualitative extraction can enhance umbrella reviews, reducing time by 80% while maintaining high accuracy for study characteristics. Binary outcomes showed lower accuracy due to missing extractions, though precision was strong when data were captured. AI performs best for qualitative fields and has growing potential for evidence mapping in low-risk contexts such as TLRs, scoping reviews, and umbrella reviews.