Large Language Models for Automatic PICO Criteria Evaluation
Author(s)
Ron Filomeno, PhD, MPH1, Lenon Mendes Pereira, PhD, MS2, Louis Patrick Watanabe, PhD1, Niamh McGuinness, PhD3;
1IQVIA, Tokyo, Japan, 2IQVIA, Chicago, IL, USA, 3IQVIA, Ottawa, ON, Canada
1IQVIA, Tokyo, Japan, 2IQVIA, Chicago, IL, USA, 3IQVIA, Ottawa, ON, Canada
OBJECTIVES: The systematic assessment of published literature is a crucial step in the synthesis of evidence of scientific content. However, the process can be time-consuming and requires meticulous attention to inclusion criteria, often guided by the PICO (Population, Intervention, Comparison, and Outcome) framework. This study explores the potential of using a Large Language Model (LLM) to expedite the evaluation process of determining whether scientific articles meet PICO-based inclusion criteria.
METHODS: The methodology included two main steps. First, we created and validated a prompt to identify PICO elements in full-text documents. Next, we tested how accurately and efficiently this prompt could categorize articles for inclusion or exclusion. The specific focus was on categorizing 100 research papers published in 2024 in PubMed that studied the impact of GLP-1 receptor agonists on cardiovascular risk factors in obese patients.
RESULTS: The results demonstrated significant potential in identifying relevant studies, with a precision rate of 0.93, a recall of 1, and an F1-score of 0.96. Specifically, the LLM showed proficiency in recognizing key components of the PICO criteria, such as the study population, the nature of interventions, control or comparison groups, and specified outcomes. This suggests that LLMs can reliably assist in the initial stages of literature screening, thus reducing the burden on researchers and streamlining the review process.
CONCLUSIONS: In conclusion, LLMs present a promising tool for enhancing the efficiency of systematic literature reviews by accurately assessing PICO-based inclusion criteria, ultimately contributing to a more rapid and comprehensive synthesis of scientific knowledge. Future research will focus on expanding LLMs capabilities and exploring their integration into broader tasks of scientific literature reviews.
METHODS: The methodology included two main steps. First, we created and validated a prompt to identify PICO elements in full-text documents. Next, we tested how accurately and efficiently this prompt could categorize articles for inclusion or exclusion. The specific focus was on categorizing 100 research papers published in 2024 in PubMed that studied the impact of GLP-1 receptor agonists on cardiovascular risk factors in obese patients.
RESULTS: The results demonstrated significant potential in identifying relevant studies, with a precision rate of 0.93, a recall of 1, and an F1-score of 0.96. Specifically, the LLM showed proficiency in recognizing key components of the PICO criteria, such as the study population, the nature of interventions, control or comparison groups, and specified outcomes. This suggests that LLMs can reliably assist in the initial stages of literature screening, thus reducing the burden on researchers and streamlining the review process.
CONCLUSIONS: In conclusion, LLMs present a promising tool for enhancing the efficiency of systematic literature reviews by accurately assessing PICO-based inclusion criteria, ultimately contributing to a more rapid and comprehensive synthesis of scientific knowledge. Future research will focus on expanding LLMs capabilities and exploring their integration into broader tasks of scientific literature reviews.
Conference/Value in Health Info
2025-05, ISPOR 2025, Montréal, Quebec, CA
Value in Health, Volume 28, Issue S1
Code
SA58
Topic
Study Approaches
Topic Subcategory
Literature Review & Synthesis
Disease
SDC: Cardiovascular Disorders (including MI, Stroke, Circulatory), SDC: Diabetes/Endocrine/Metabolic Disorders (including obesity)