INTEGRATING GENERATIVE ARTIFICIAL INTELLIGENCE (AI) INTO BIOSTATISTICAL WORKFLOWS FOR TRIAL EMULATION: A FRAMEWORK TO ACCELERATE REAL-WORLD EVIDENCE GENERATION
Author(s)
Saamir Pasha, MPH1, Jessica Paulus, ScD2, Zhaohui Su, PhD3;
1Ontada, Boston, MA, USA, 2Ontada, Dedham, MA, USA, 3McKesson, Chestnut Hill, MA, USA
1Ontada, Boston, MA, USA, 2Ontada, Dedham, MA, USA, 3McKesson, Chestnut Hill, MA, USA
OBJECTIVES: Real world evidence (RWE) studies increasingly leverage AI-enabled workflows to accelerate timelines while maintaining rigor. We present a novel framework to apply AI to trial emulation RWE studies integrating deterministic pipelines, archived databases, automated quality checks, and large language model (LLM) evaluations to enhance reproducibility, scalability, and analytic quality.
METHODS: The framework specifies all steps of a trial emulation using real world data— including data curation, eligibility mapping, propensity score modeling, and time-to-event analysis. Automated data quality check (QC) rules and subject matter expert (SME) validated rubrics guided LLM evaluators for eligibility mapping and narrative checks. The framework was applied to a metastatic breast cancer trial emulation using electronic health records and claims data.
RESULTS: The generative AI trial emulation framework has four major features: (1) Automated specification of eligibility and variable derivation to ensure protocol fidelity (eligibility mapping accuracy and outcome definition concordance >70% per SME checklist); (2) Generation of reasoning chains and progressive summaries with visualizations for interpretability; (3) Automated data quality checks and irregularities flagging with >90% QC rule coverage and rapid resolution; and (4) Execution of trial emulation statistical analyses, producing strong covariate balance (mean standardized difference [MSD] ≤0.1 for ≥90% covariates). The framework incorporates reproducibility checks through repeated runs on identical datasets and includes LLM evaluator reliability using κ/α against SME‑labeled gold sets (20-40 items) plus consistency across multiple prompts. Efficiency gains (~80%) with same-day data insights and a re-allocation index >50% demonstrate reinvestment of time in QC, sensitivity analyses, and cross-discipline review (medical oncology and epidemiology), enhancing rigor.
CONCLUSIONS: AI assisted workflows can deliver reproducible, efficient, and transparent RWE analyses when paired with database archives, standardized QC, and auditable LLM evaluation. Efficiency gains enable more QC and SME engagement, reinforcing credibility. This framework offers a scalable path for high-quality trial emulation.
METHODS: The framework specifies all steps of a trial emulation using real world data— including data curation, eligibility mapping, propensity score modeling, and time-to-event analysis. Automated data quality check (QC) rules and subject matter expert (SME) validated rubrics guided LLM evaluators for eligibility mapping and narrative checks. The framework was applied to a metastatic breast cancer trial emulation using electronic health records and claims data.
RESULTS: The generative AI trial emulation framework has four major features: (1) Automated specification of eligibility and variable derivation to ensure protocol fidelity (eligibility mapping accuracy and outcome definition concordance >70% per SME checklist); (2) Generation of reasoning chains and progressive summaries with visualizations for interpretability; (3) Automated data quality checks and irregularities flagging with >90% QC rule coverage and rapid resolution; and (4) Execution of trial emulation statistical analyses, producing strong covariate balance (mean standardized difference [MSD] ≤0.1 for ≥90% covariates). The framework incorporates reproducibility checks through repeated runs on identical datasets and includes LLM evaluator reliability using κ/α against SME‑labeled gold sets (20-40 items) plus consistency across multiple prompts. Efficiency gains (~80%) with same-day data insights and a re-allocation index >50% demonstrate reinvestment of time in QC, sensitivity analyses, and cross-discipline review (medical oncology and epidemiology), enhancing rigor.
CONCLUSIONS: AI assisted workflows can deliver reproducible, efficient, and transparent RWE analyses when paired with database archives, standardized QC, and auditable LLM evaluation. Efficiency gains enable more QC and SME engagement, reinforcing credibility. This framework offers a scalable path for high-quality trial emulation.
Conference/Value in Health Info
2026-05, ISPOR 2026, Philadelphia, PA, USA
Value in Health, Volume 29, Issue S6
Code
MSR66
Topic
Methodological & Statistical Research
Topic Subcategory
Artificial Intelligence, Machine Learning, Predictive Analytics, Confounding, Selection Bias Correction, Causal Inference
Disease
SDC: Oncology