Presentation (CTI)

OBJECTIVES: To evaluate the performance of artificial intelligence (AI)-assisted multi-row data extraction for meta-analysis compared with human data extraction.
METHODS: An umbrella review of published meta-analyses and network meta-analyses was performed. Data were extracted in a multi-row format from nine included reviews to capture study characteristics and outcome data. Extraction by humans was performed in Microsoft Excel, followed by quality control and adjudication to establish a reference standard. The same reviews were extracted by AI using EasySLR. AI-extracted and human-extracted data were compared against the full-text publications. For AI-assisted extraction, each extracted data point was classified as a true positive (TP; correctly extracted and reported in the review), false positive (FP; non-relevant data extracted), false negative (FN; relevant data reported in the review but not extracted), or true negative (TN; as the extraction task does not provide a finite set of non-relevant data points, true negatives cannot be computed and were therefore set to zero). Precision, recall, F1 score, and F2 score were calculated.
RESULTS: For study characteristics, the AI correctly extracted all relevant data, achieving 100% recall, although three additional non-relevant data points were extracted. For outcome data, AI achieved 100% recall in seven of nine reviews, with recall of 60% and 79% in the remaining two; precision was 100% across all reviews. F1 and F2 scores ranged from 0.86 to 1.00 and 0.94 to 1.00 for study characteristics, and from 0.75 to 1.00 and 0.65 to 1.00 for outcome data, respectively. Missed data points were primarily associated with data reported in more complex reporting formats.
CONCLUSIONS: AI-assisted multi-row data extraction demonstrated high recall and robust F1 and F2 scores across most reviews. However, some data were not extracted when information was presented in complex formats, highlighting the need for human review during data extraction.