Artificial Intelligence-Based Population, Intervention, Comparator, and Outcome (PICO) Prediction for European Union (EU) Joint Clinical Assessment (JCA)
Author(s)
Gavin J. Outteridge, MA1, Justin Yu2, Catherine CHAMOUX, PharmD3.
1Aesara Europe, London, United Kingdom, 2Jersey City, NJ, USA, 3AESARA, CABOURG, France.
1Aesara Europe, London, United Kingdom, 2Jersey City, NJ, USA, 3AESARA, CABOURG, France.
OBJECTIVES: PICOs for a JCA vary, leading to uncertainty in HTA outcomes and subsequent reimbursement of new treatments. Manually reviewing HTA reports to predict PICOs can take months, which is not ideal given tight timelines for JCA submissions. The objective of this study was thus to create a LLM-based tool for rapid and accurate PICO prediction.
METHODS: A multi-agent workflow (“PICOs Predictor”) was developed in Python and used to extract PICO information from EU HTA reports (FR (HAS), n=19; DE (G-BA), n=9; ES (AEMPS), n=8) on metastatic colorectal cancer treatments published between 2010-2025. The PICOs Predictor was refined by iteratively comparing its outputs with target information manually extracted by a human HTA expert. Code updates, followed by retesting on all documents, were made until 100% agreement with the HTA expert was achieved. Lastly, PICOs Predictor outputs were compared to those from non-reasoning GPT-4o (OpenAI), reasoning o3 (OpenAI), and hybrid reasoning Claude Sonnet 4 (Anthropic) using identical prompts (except for multi-step ones), with initial validation focused on HAS reports.
RESULTS: In comparison to the PICOs Predictor, o3, GPT-4o, and Claude Sonnet 4 correctly identified the population(s) of interest for each included HAS report in 18/19 (95%), 10/19 (58%), and 9/10 (47%) of cases, respectively. Despite the high accuracy of o3, however, 10/19 (53%) outputs contained population descriptions that were sub-optimally defined based on HTA expert review.
CONCLUSIONS: AI-enabled PICOs prediction is a promising approach to support evidence generation and access strategy for treatments subject to the JCA process. The PICOs Predictor can accelerate manual review of HTA documents and generate high-quality PICO predictions when manual review of relevant documents is not viable. The PICOs Predictor is both more comprehensive and more reliable than off-the-shelf LLMs. Further evaluation of the PICOs Predictor across other disease states would demonstrate its generalizability.
METHODS: A multi-agent workflow (“PICOs Predictor”) was developed in Python and used to extract PICO information from EU HTA reports (FR (HAS), n=19; DE (G-BA), n=9; ES (AEMPS), n=8) on metastatic colorectal cancer treatments published between 2010-2025. The PICOs Predictor was refined by iteratively comparing its outputs with target information manually extracted by a human HTA expert. Code updates, followed by retesting on all documents, were made until 100% agreement with the HTA expert was achieved. Lastly, PICOs Predictor outputs were compared to those from non-reasoning GPT-4o (OpenAI), reasoning o3 (OpenAI), and hybrid reasoning Claude Sonnet 4 (Anthropic) using identical prompts (except for multi-step ones), with initial validation focused on HAS reports.
RESULTS: In comparison to the PICOs Predictor, o3, GPT-4o, and Claude Sonnet 4 correctly identified the population(s) of interest for each included HAS report in 18/19 (95%), 10/19 (58%), and 9/10 (47%) of cases, respectively. Despite the high accuracy of o3, however, 10/19 (53%) outputs contained population descriptions that were sub-optimally defined based on HTA expert review.
CONCLUSIONS: AI-enabled PICOs prediction is a promising approach to support evidence generation and access strategy for treatments subject to the JCA process. The PICOs Predictor can accelerate manual review of HTA documents and generate high-quality PICO predictions when manual review of relevant documents is not viable. The PICOs Predictor is both more comprehensive and more reliable than off-the-shelf LLMs. Further evaluation of the PICOs Predictor across other disease states would demonstrate its generalizability.
Conference/Value in Health Info
2025-11, ISPOR Europe 2025, Glasgow, Scotland
Value in Health, Volume 28, Issue S2
Code
HTA43
Topic
Clinical Outcomes, Health Technology Assessment, Study Approaches
Topic Subcategory
Systems & Structure, Value Frameworks & Dossier Format
Disease
Oncology