Presentation (CTI)

OBJECTIVES: Artificial intelligence (AI) may facilitate automation of systematic literature review (SLR) components. Data extraction (DE) from economic evaluations, commonly undertaken to inform health technology assessment models, presents unique challenges due to methodological heterogeneity and the breadth of required data. However, AI performance in economic data extraction remains insufficiently evaluated, leaving its potential role uncertain. This study evaluated structured AI-assisted data extraction within an economic SLR.
METHODS: DE was conducted in Nested Knowledge using AI-assisted Adaptive Smart Tags. DE prompts covered study characteristics, model results, cost data, resource use, and utility values. Ten economic evaluations included in an SLR of chronic lymphocytic leukaemia were assessed. AI-extracted results were compared with human-extracted data with consistency assessed using accuracy, precision, recall, and F1 score across key categories. Discrepancies between AI and human extraction were resolved through source verification. Data from primary manuscripts and supplements were combined to maximise information capture.
RESULTS: AI achieved the highest precision for cost data extraction (0.978), followed by model results (0.949), while the highest accuracy and F1 scores were observed for study characteristics (0.808; F1 = 0.879), followed by resource use (0.739; F1 = 0.878). Study characteristics also had the highest recall (0.919), with resource use (0.790) ranking second. AI limitations were observed for selected data items, including type of analysis (failure to distinguish CEA and CUA), resource use (irrelevant utility data), cost data (missing ranges), data sources (incomplete citations), and incorrect digitisation from figures. Nevertheless, AI identified data missed by manual reviewer, including rationale for model design, study inclusion criteria, sensitivity analysis results, and resource use reported in text.
CONCLUSIONS: AI-assisted extraction performed well across key economic evidence categories and may enhance the efficiency and completeness of economic SLR workflows. Further prompt refinement to address current limitations and support reuse across SLRs may increase its value.