Is GPT-4o Capable of Automating Detailed Data Extraction for Systematic Literature Reviews (SLRs)?

OBJECTIVES: Data extraction is an integral part of SLRs which takes significant time and effort. This study assesses the effectiveness of the recent GPT-4o model in performing data extraction, specifically targeting various elements of study design.

METHODS: We evaluated the accuracy of data extraction using previously extracted and quality checked information from 12 publications of clinical trials for interventions in metastatic non-small cell lung cancer (NSCLC). We selected 39 data entries that describe a study design, resulting in a total of 468 data elements.

Using Python coding we converted the publications to text, which were processed and used as context in a prompt containing the information about the data entries to be extracted. This prompt was then sent to GPT-4o via API hosted on Azure, and the response was processed to structure it into the data extraction template. The data extracted using GPT-4o was then compared to the manually extracted data to estimate accuracy.

RESULTS: GPT-4o successfully extracted 415 data elements (88.7% accuracy), and in 37 cases, GPT-4o provided more comprehensive information than the manual extracted data, demonstrating zero-shot learning potential without any technical expertise or prior training. In 53 cases, elements were generated erroneously: 15 significant errors, 8 minor errors, 6 fabricated data, and 24 partially missing data. A significant concentration of errors (29) was observed in 8 data entries that require complex contextual understanding, such as subgroup analyses and details of therapeutic protocols.

An additional highlight is the speed of our pipeline, averaging 27.75 seconds to extract data elements per publication, significantly reducing time and effort compared to manual extraction.

CONCLUSIONS: The study highlights the remarkable performance of the GPT-4o model in automating the extraction of complex data. The results underscore the model's potential to streamline the data extraction process for researchers, significantly reducing the time required.