Large Language Models for Data Extraction in a Targeted Review: A Case Study

OBJECTIVES: Accurate, consistent extraction and presentation of data is a time-consuming process. Large language models (LLMs) accessed via a chat interface require minimal user training and perform tasks without any setup overheads beyond an initial phase of prompt engineering. We assessed the chat interface to Claude 3 Opus for accuracy, consistency, presentation of data, and time savings in the context of high-level extraction for a targeted review.

METHODS: A targeted review was conducted to investigate disparities in patient characteristics in the diagnosis and treatment of one specific indication. We used the chat interface to Claude to extract data from 30 papers, with a human reviewer checking all data points. Study and population details were extracted, plus brief details of any study results or discussion regarding disparities in diagnosis or treatment. Papers were uploaded in pairs to minimize prompts, with the model explicitly tasked with labelling each set of data points with the name of the relevant paper.

RESULTS: Data were consistently extracted in a suitably structured format. Eleven papers required no edits to the data. Five papers required minimal edits, and nine papers contained minor errors or omissions in the data. One paper was extracted correctly but the answers reported by the model also contained additional data drawn from the second paper of the pair. Another pair of papers was extracted by the model and mislabeled, with data for each paper labelled with the file name of the other paper. Following this error, PDFs were uploaded singly.

CONCLUSIONS: Even allowing time for human checking and minor correction of the extracted data, use of the model enabled extraction and checking of 30 papers in a single day. Access to LLMs via a chat interface is typically relatively inexpensive and can offer significant resource savings in the context of suitable reviews.