Presentation (CTI)

OBJECTIVES: The increasing use of electronic health records (EHR) and healthcare claims databases has expanded opportunities for real-world evidence (RWE) generation. However, developing RWE study designs and statistical analysis plans requires integrating information from multiple sources, including scientific literature, clinical guidance, coding systems, outcome definitions, and statistical methods. This resource-intensive, expertise-dependent process may contribute to variability. To streamline RWE study design and statistical analysis plan development using EHR and claims data, we developed and evaluated an artificial intelligence (AI)-assisted framework for automated evidence retrieval, consolidation, and prioritization.
METHODS: A prototype was developed using patients diagnosed with myelodysplastic syndromes (MDS) as a proof-of-concept use case. The framework used a multi-stage large language model (LLM) workflow to retrieve, harmonize, and prioritize evidence from publications, clinical and regulatory guidelines, healthcare coding repositories, and proprietary methodological knowledge bases. Extracted information included diagnosis and medication codes, procedures, clinical outcomes, endpoint definitions, and statistical methodologies. Performance was benchmarked against evidence generated in a previously completed human expert-led MDS real-world evidence study.
RESULTS: The framework generated a structured evidence matrix relevant to the study question. Compared with evidence identified through conventional expert-led review, the prototype achieved an agreement rate of 70% across key evidence elements and reduced the time required for evidence synthesis by 40-50%. The framework successfully consolidated information from multiple evidence domains into a single structured output, supporting a more efficient and robust study design and protocol development activities.
CONCLUSIONS: This proof-of-concept demonstrated the feasibility of AI-assisted evidence synthesis to support the design of RWE studies using EHR and claims databases. By integrating scientific, coding, clinical, and methodological evidence within a unified framework, the approach showed high concordance with expert-generated outputs while substantially reducing manual review effort. Such tools would improve the efficiency, consistency, and reproducibility of RWE study design and support more rapid evidence generation in healthcare research.