CAN JCA PICO SCOPE BE PREDICTED BEFORE IT IS PUBLISHED? A PRE-REGISTERED, BLINDED VALIDATION FRAMEWORK AGAINST SEALED GROUND TRUTH
Author(s)
Farzana Malik1, Mark Goodlad, BA2.
1Managing Partner, Cogience, London, United Kingdom, 2Cogience, London, United Kingdom.
1Managing Partner, Cogience, London, United Kingdom, 2Cogience, London, United Kingdom.
OBJECTIVES: Since January 2025, all new oncology medicines and ATMPs in the EU undergo Joint Clinical Assessment (JCA), and the consolidated PICO scope is released only 60-100 days before the dossier deadline. Developers must anticipate that scope at least 2 years earlier. Published prediction methods are validated informally, against the predictor's own reasoning or the JCA subgroup's reported ranges, and none show accuracy under blinded conditions against a result set fixed in advance. We tested whether JCA PICO scope can be predicted blind and scored against a sealed result set fixed before the prediction is made
METHODS: For each case, a scoring rubric and a structured result set (populations, comparators, outcomes, subgroup axes) was taken from the published JCA report, then time-stamped and sealed before any prediction existed. Predictions were produced in an independent workflow that used only pre-scope inputs: the pivotal trial and label, the EU-27 treatment landscape, prior national HTA decisions, and JCA guidance. The dossier and JCA report were withheld. Comparator precision and recall were the primary metrics
RESULTS: One case (tovorafenib) has completed full blinded validation, the only published JCA report available to score against so far under the new process. It predicted 3 populations and 8 PICOs against the sealed result set. A further 8 predictions have already been made and sealed, and will be validated as the JCA releases the corresponding reports through 2026. The recurrent errors were comparator over-prediction in narrow indications and comparator recall loss in heterogeneous later-line settings.
CONCLUSIONS: Sealing the result set before predicting removes the grading-to-fit risk of retrospective validation, and the design lets accuracy build as reports publish. Few reports exist yet, so validation is early by necessity. The method ran on Claude Opus 4.8; capability is improving exponentially year on year, which is what now makes this approach workable.
METHODS: For each case, a scoring rubric and a structured result set (populations, comparators, outcomes, subgroup axes) was taken from the published JCA report, then time-stamped and sealed before any prediction existed. Predictions were produced in an independent workflow that used only pre-scope inputs: the pivotal trial and label, the EU-27 treatment landscape, prior national HTA decisions, and JCA guidance. The dossier and JCA report were withheld. Comparator precision and recall were the primary metrics
RESULTS: One case (tovorafenib) has completed full blinded validation, the only published JCA report available to score against so far under the new process. It predicted 3 populations and 8 PICOs against the sealed result set. A further 8 predictions have already been made and sealed, and will be validated as the JCA releases the corresponding reports through 2026. The recurrent errors were comparator over-prediction in narrow indications and comparator recall loss in heterogeneous later-line settings.
CONCLUSIONS: Sealing the result set before predicting removes the grading-to-fit risk of retrospective validation, and the design lets accuracy build as reports publish. Few reports exist yet, so validation is early by necessity. The method ran on Claude Opus 4.8; capability is improving exponentially year on year, which is what now makes this approach workable.
Conference/Value in Health Info
2026-11, ISPOR Europe 2026, Vienna, Austria
Value in Health, Volume 29, Issue 12S
Code
HTA111
Topic
Clinical Outcomes, Health Policy & Regulatory, Health Technology Assessment
Topic Subcategory
Value Frameworks & Dossier Format
Disease
Neurological Disorders, Oncology, Pediatrics, Rare & Orphan Diseases