A DEDUCTIVE LARGE LANGUAGE MODEL (LLM) WORKFLOW FOR MULTI-TOPIC EVIDENCE LANDSCAPE MAPPING AT SCALE
Author(s)
Kim Wager, PhD1, Noemi Pasquarelli, PhD2, Anne-Marie Couto, PhD1, Gemma Carter, BSc, PhD1, David Moore, PhD1.
1Oxford PharmaGenesis, Oxford, United Kingdom, 2F. Hoffmann-La Roche Ltd, Basel, Switzerland.
1Oxford PharmaGenesis, Oxford, United Kingdom, 2F. Hoffmann-La Roche Ltd, Basel, Switzerland.
OBJECTIVES: Landscape mapping to continuously monitor literature and uncover evidence trends and gaps can be manually infeasible in disease areas with high publication activity. We developed and applied a deductive, LLM-assisted topic-modelling workflow to quantify and synthesize literature for five therapies and a drug class against an expert-defined framework of 28 clinical and strategic topics.
METHODS: Systematic searches of MEDLINE and Embase were followed by deduplication and LLM-assisted relevance screening. Abstracts underwent multi-topic classification (1-3 topics) and categorization tagging (comparative, long-term, real-world evidence). Records then underwent structured extraction, batched topic-level thematic synthesis and evidence gap analysis. A random sample of 50 abstracts was independently classified by a human rater to validate LLM topic assignments.
RESULTS: Of 1943 records, 1271 unique abstracts were included. In the validation sample, human and LLM assignments shared at least one topic in 98% of cases. Where both assigned at least two topics, 80% shared at least two; where both assigned three, 54% shared all three. Jaccard similarity was 0.68 (standard deviation 0.29) and recall 0.83, indicating moderate agreement. In the full sample, 2788 topic assignments were made across 28 topics (mean 2.2 per abstract). Corpus-wide, 69%, 34% and 10% of abstracts carried a real-world evidence, comparative or long-term tag, respectively. Overall, 346 evidence gaps classified by quality level (e.g. insufficient) and importance (e.g. high) were identified.
CONCLUSIONS: Deductive LLM-assisted topic modelling provides a scalable, transparent approach to characterizing evidence landscapes. Framework design and development required ~50 person-hours and the pipeline completes in ~3 hours of computation. Equivalent manual mapping would require substantially greater effort and would be infeasible. Limitations include the fixed framework scope, abstract-level granularity and variable inter-rater agreement across topics. The approach complements conventional systematic review where comprehensive mapping across multiple clinical themes is required for evidence planning and research prioritization.
METHODS: Systematic searches of MEDLINE and Embase were followed by deduplication and LLM-assisted relevance screening. Abstracts underwent multi-topic classification (1-3 topics) and categorization tagging (comparative, long-term, real-world evidence). Records then underwent structured extraction, batched topic-level thematic synthesis and evidence gap analysis. A random sample of 50 abstracts was independently classified by a human rater to validate LLM topic assignments.
RESULTS: Of 1943 records, 1271 unique abstracts were included. In the validation sample, human and LLM assignments shared at least one topic in 98% of cases. Where both assigned at least two topics, 80% shared at least two; where both assigned three, 54% shared all three. Jaccard similarity was 0.68 (standard deviation 0.29) and recall 0.83, indicating moderate agreement. In the full sample, 2788 topic assignments were made across 28 topics (mean 2.2 per abstract). Corpus-wide, 69%, 34% and 10% of abstracts carried a real-world evidence, comparative or long-term tag, respectively. Overall, 346 evidence gaps classified by quality level (e.g. insufficient) and importance (e.g. high) were identified.
CONCLUSIONS: Deductive LLM-assisted topic modelling provides a scalable, transparent approach to characterizing evidence landscapes. Framework design and development required ~50 person-hours and the pipeline completes in ~3 hours of computation. Equivalent manual mapping would require substantially greater effort and would be infeasible. Limitations include the fixed framework scope, abstract-level granularity and variable inter-rater agreement across topics. The approach complements conventional systematic review where comprehensive mapping across multiple clinical themes is required for evidence planning and research prioritization.
Conference/Value in Health Info
2026-11, ISPOR Europe 2026, Vienna, Austria
Value in Health, Volume 29, Issue 12S
Code
SA18
Topic
Methodological & Statistical Research, Study Approaches
Topic Subcategory
Literature Review & Synthesis
Disease
No Additional Disease & Conditions/Specialized Treatment Areas