OBJECTIVES: This study aims to investigate the use of large language models (LLMs) for classifying peer-reviewed publications. By leveraging LLMs, we aim to develop a system that can accurately extract metadata from publications and present it in a structured format.

METHODS: We used the llama-index framework and OpenAI's gpt-3.5-turbo model to extract structured output from publication abstracts. We evaluated the performance of traditional machine learning models such as Logistic Regression, Decision Trees, and Gradient Boosting against the LLM approach. These models were trained on a dataset of approximately 35,000 labeled publications, and for each type of publication metadata, all three models were trained and the one yielding the best performance was selected for comparison to the LLM approach. To assess the accuracy of data extraction, we formulated three classification questions, including surgical procedure type, study type, and an indicator of whether robotic-assisted surgery was studied. These questions were tested on a dataset of 98 abstracts related to surgical literature, and the results were evaluated by two independent researchers. The program was developed and validated using Python 3.12.

RESULTS: Each of the three questions was assessed individually. The precision of primary procedure, study type, and robotic surgery indicator was 0.89, 0.68, and 0.90, respectively. Correspondingly, the classification results from traditional machine learning models were 0.60, 0.75, and 0.89, respectively. The labels predicted by LLMs showed significantly better results compared to traditional machine learning methods in primary procedure and similar results in study type and robotic indicators.

CONCLUSIONS: This study demonstrates that LLMs can accelerate the extraction and analysis of vast amounts of information, increasing productivity and optimizing the literature review process. Clinical librarians can leverage these tools to shift their role from manually labeling and extracting data to validating model output.