COMPARING LLM-BASED TO EXPERT-CURATED EXTRACTION FOR BIOMARKER ATTRIBUTES IN LUNG AND BREAST CANCER
Author(s)
Sheenu Chandwani, MPH, PhD1, Payal Keswarpu, MBBS, MD2, Ashwani Ashwani, MSc2, Vivek P. Vaidya, BSc2, Jiby Joseph, MD, MS1, Ruth Pe Benito, MPH1;
1ConcertAI, LLC, Cambridge, MA, USA, 2ConcertAI, LLC, Bengaluru, India
1ConcertAI, LLC, Cambridge, MA, USA, 2ConcertAI, LLC, Bengaluru, India
OBJECTIVES: Biomarker details are essential for RWE and HEOR in oncology, however, this information is inherently multi-dimensional, tumor-specific, and embedded in unstructured EHR text, posing challenges for scalable abstraction. We conducted an internal validation of LLM/SLM-based models for automated extraction of oncology biomarker attributes from unstructured EHRs.
METHODS: We evaluated a multi-agent suite of decoder-only LLM and SLM models trained using supervised fine tuning and direct preference optimization to extract biomarker attributes from unstructured EHR notes for patients with breast or lung cancer. Extracted attributes included biomarker name, cancer type, categorical result, exon number, genomic alteration (e.g., mutation, expression, deletion, amplification, gene rearrangement, copy number variation, etc.), and variant type (e.g., V600, G12C, T790m). Model outputs were validated against expert curated data from 1,652 biomarker records across 12 biomarkers (EGFR, ALK, KRAS, ROS1, BRAF, MET, RET, TMB, ER, PR, HER2, and PD-L1) and 31 patients. Performance was assessed at the record level using precision, recall, and F1-score.
RESULTS: Overall, record-level F1 scores were 0.97 (biomarker name), 0.96 (cancer type), 0.94 (categorical result), 0.90 (exon number), 0.85 (genomic alteration), and 0.75 (variant type). Biomarker name and cancer type had ≥0.90 F1 scores across all biomarkers. Categorical results also performed well (≥0.92), except for KRAS (0.57) and PD-L1 (0.68). Variant type performance was ≥0.88 F1 for EGFR, KRAS, PD-L1, whereas 0.53 for ALK and 0.57 for ROS1. Genomic alternation performed at ≥0.81 F1 for EGFR and KRAS, and EGFR exon number reached 0.90 F1.
CONCLUSIONS: Multi-agent decoder-only models demonstrated strong performance for automated extraction of multidimensional oncology biomarker attributes from unstructured EHR. These findings highlight the potential of AI driven auto-curation to scale biomarker characterization for RWE/HEOR while reducing manual review burden. Future work will include broader validation at the patient and population level and correlation with clinical expectations in patient treatment journey.
METHODS: We evaluated a multi-agent suite of decoder-only LLM and SLM models trained using supervised fine tuning and direct preference optimization to extract biomarker attributes from unstructured EHR notes for patients with breast or lung cancer. Extracted attributes included biomarker name, cancer type, categorical result, exon number, genomic alteration (e.g., mutation, expression, deletion, amplification, gene rearrangement, copy number variation, etc.), and variant type (e.g., V600, G12C, T790m). Model outputs were validated against expert curated data from 1,652 biomarker records across 12 biomarkers (EGFR, ALK, KRAS, ROS1, BRAF, MET, RET, TMB, ER, PR, HER2, and PD-L1) and 31 patients. Performance was assessed at the record level using precision, recall, and F1-score.
RESULTS: Overall, record-level F1 scores were 0.97 (biomarker name), 0.96 (cancer type), 0.94 (categorical result), 0.90 (exon number), 0.85 (genomic alteration), and 0.75 (variant type). Biomarker name and cancer type had ≥0.90 F1 scores across all biomarkers. Categorical results also performed well (≥0.92), except for KRAS (0.57) and PD-L1 (0.68). Variant type performance was ≥0.88 F1 for EGFR, KRAS, PD-L1, whereas 0.53 for ALK and 0.57 for ROS1. Genomic alternation performed at ≥0.81 F1 for EGFR and KRAS, and EGFR exon number reached 0.90 F1.
CONCLUSIONS: Multi-agent decoder-only models demonstrated strong performance for automated extraction of multidimensional oncology biomarker attributes from unstructured EHR. These findings highlight the potential of AI driven auto-curation to scale biomarker characterization for RWE/HEOR while reducing manual review burden. Future work will include broader validation at the patient and population level and correlation with clinical expectations in patient treatment journey.
Conference/Value in Health Info
2026-05, ISPOR 2026, Philadelphia, PA, USA
Value in Health, Volume 29, Issue S6
Code
RWD19
Topic
Real World Data & Information Systems
Topic Subcategory
Health & Insurance Records Systems
Disease
SDC: Oncology