Presentation (CTI)

OBJECTIVES: To describe a quality assurance (QA) framework supporting the generation of regulatory-grade real-world evidence (RG-RWE) from multi-source electronic health records (EHR), enriching structured data with information extracted from clinical text using clinical natural language processing (cNLP).
METHODS: A multi-layer QA framework was developed covering the RWE lifecycle through three components: (1) data source, (2) variable construction, and (3) study-level RWE. The data source framework assesses fitness-for-purpose through predefined quality controls addressing site qualification, data completeness, data privacy and security (e.g., GDPR/HIPAA compliance and ISO-aligned safeguards), structural and semantic harmonization, and post-integration plausibility. The variable construction framework governs extraction of information from free-text records using cNLP, including entity recognition and contextualization (e.g., negation, temporality), and is supported by a terminology-based Disease Panel integrating standardized vocabularies with clinically curated extensions ensuring consistent identification of concepts. Extracted concepts are transformed into study variables through predefined rules and refined iteratively using annotated clinical records. Key Population, Intervention, and Outcome variables undergo validation against reference standards, incorporating clinician verification (human-in-the-loop), using precision, recall, and F1-score. Additional variables receive targeted term-level validation. The RWE framework further incorporates patient-level review of cohort eligibility and longitudinal consistency, complemented by comparison with external benchmarks.
RESULTS: The framework provides a standardized and auditable approach for evaluating data quality across heterogeneous EHR sources and for constructing validated study variables from structured and unstructured data. It supports predefined acceptance criteria at data source, model, and variable levels, enabling systematic identification and mitigation of missingness, heterogeneity, mapping inconsistencies, and potential misclassification before analysis.
CONCLUSIONS: This end-to-end QA framework supports transparent, reproducible, and fit-for-purpose generation of RG-RWE from multi-source EHR data. By integrating data quality assessment, cNLP-based variable construction, and study-level validation within a unified process, it may contribute to standardization of AI-enabled real-world data for regulatory and health technology assessment decision-making.