EVALUATING THE PERFORMANCE AND OPTIMAL USE STRATEGY OF AN ARTIFICIAL INTELLIGENCE (AI)-ASSISTED TOOL FOR ONCOLOGY LITERATURE REVIEW
Author(s)
Ahmed M. Kamel, B.Pharm1, Josh (JT) Harvey, MPH2, Wenxi Tang, MS3, Xiaoliang Wang, MPH, PhD3, Gregory A. Maglinte, MPH, PhD3, Lin Zhan, PhD3;
1Auburn University Harrison School of Pharmacy, Auburn, AL, USA, 2University of Massachusetts Chan Medical School, Worcester, MA, USA, 3BeOne Medicines Ltd, San Carlos, CA, USA
1Auburn University Harrison School of Pharmacy, Auburn, AL, USA, 2University of Massachusetts Chan Medical School, Worcester, MA, USA, 3BeOne Medicines Ltd, San Carlos, CA, USA
OBJECTIVES: To evaluate the performance of an AI-assisted systematic review platform, Nested Knowledge, (NK) for health economics and outcomes research (HEOR) across two complementary projects: (1) an AI-only workflow benchmarked against a manual PubMed systematic review, and (2) an AI + human hybrid workflow integrating targeted oversight and validation.
METHODS: In project 1, systematic reviews on immunotherapy medications for non-small cell lung cancer (NSCLC) were conducted using manual and AI-only workflows, independently, with identical search strategies and inclusion/exclusion criteria. The manual process used PubMed keyword searches, manual title/abstract and full-text screening, and manual data extraction. The AI-only workflow used NK Smart Search with machine learning-based screening and AI tagging without human intervention. Efficiency, recall, and data-extraction accuracy were compared. In project 2, a targeted literature review was conducted for ibrutinib clinical outcomes in blood cancers. The AI-human hybrid workflow employed automated screening with manual verification and cross-referenced findings with the Centers for Medicare & Medicaid Services (CMS) Maximum Fair Price (MFP) document.
RESULTS: In NSCLC, manual review identified 699 studies out of 2,263 screened articles; while AI identified 776 out of 4,746 screened results, with 361 overlapping (51.6%). While screening time was reduced from 90 hours (manual) to 6 hours (AI-only), after manual verification, the precision of the AI extraction for overall survival (OS) was low (20%). In the AI-human hybrid workflow, of 52 clinical studies cited by CMS MFP document, the review identified 21 (40.4%), capturing all key head-to-head trials. Unmatched citations were predominantly foundational single-agent trials predating the Bruton's tyrosine kinase inhibitor era.
CONCLUSIONS: AI-assisted tools require human oversight for search optimization and manual input to ensure completeness. When combined with human oversight, NK improves extraction while substantially reducing total review time. Integrating AI and human expertise has the potential to enable the efficient generation of evidence for HEOR, informing healthcare decision-making.
METHODS: In project 1, systematic reviews on immunotherapy medications for non-small cell lung cancer (NSCLC) were conducted using manual and AI-only workflows, independently, with identical search strategies and inclusion/exclusion criteria. The manual process used PubMed keyword searches, manual title/abstract and full-text screening, and manual data extraction. The AI-only workflow used NK Smart Search with machine learning-based screening and AI tagging without human intervention. Efficiency, recall, and data-extraction accuracy were compared. In project 2, a targeted literature review was conducted for ibrutinib clinical outcomes in blood cancers. The AI-human hybrid workflow employed automated screening with manual verification and cross-referenced findings with the Centers for Medicare & Medicaid Services (CMS) Maximum Fair Price (MFP) document.
RESULTS: In NSCLC, manual review identified 699 studies out of 2,263 screened articles; while AI identified 776 out of 4,746 screened results, with 361 overlapping (51.6%). While screening time was reduced from 90 hours (manual) to 6 hours (AI-only), after manual verification, the precision of the AI extraction for overall survival (OS) was low (20%). In the AI-human hybrid workflow, of 52 clinical studies cited by CMS MFP document, the review identified 21 (40.4%), capturing all key head-to-head trials. Unmatched citations were predominantly foundational single-agent trials predating the Bruton's tyrosine kinase inhibitor era.
CONCLUSIONS: AI-assisted tools require human oversight for search optimization and manual input to ensure completeness. When combined with human oversight, NK improves extraction while substantially reducing total review time. Integrating AI and human expertise has the potential to enable the efficient generation of evidence for HEOR, informing healthcare decision-making.
Conference/Value in Health Info
2026-05, ISPOR 2026, Philadelphia, PA, USA
Value in Health, Volume 29, Issue S6
Code
SA3
Topic
Study Approaches
Topic Subcategory
Literature Review & Synthesis
Disease
SDC: Oncology