OBJECTIVES: The study assesses the contribution of using free-text from electronic medical records (EMR) by large language models (LLMs), to the completeness and validity of information on medication-use. We present an example of data on aspirin intake during pregnancy.

METHODS: Data on medication use (Acetylsalicylic Acid, “Aspirin”) was derived from available EMR sources, coded and free-text fields in several hospitals within Briya’s network. Medication listed in coded fields was identified using ATC codes. Free-text analysis of EMR from the documenting departments (emergency room, delivery room, high-risk and post-delivery wards) was performed using on-premise open-source LLM (LLama 3-8B). Frequency of aspirin use was described by source of data, department and hospital. Sample of notes were reviewed by human experts for validation of LLM and F1 score, precision and recall (sensitivity) were calculated.

RESULTS: The study population included 16,122 women who gave birth to 19,185 infants (18,836 deliveries) between 03.2020- 05.2024. 1,142 women (6.1% of deliveries) used aspirin during pregnancy. Positive aspirin use was documented 1,596 times, 1,111 in free-text (range 1-6 notes/women). Compared to experts, LLM validation metrics were: F1=0.9; precision=0.88, recall=0.92). About 66% of patients who received the medication (n=755) were identified exclusively via free-text, another third (n=356) were identified by both free-text and coded fields, resulting in free-text coverage of 97.3%. The remaining 2.7% of aspirin uptake was identified exclusively in coded fields. Comparison by documenting department and hospitals, dosage, and treatment duration will be presented.

CONCLUSIONS: The use of AI for free-text data abstraction could capture the vast majority of data on medication use. LLMs demonstrate excellent zero-shot performance without task-specific training, and deployed on-premise without GPUs, optimizing hospital resources. Completeness and quality of medication-use are highly dependent on source of data.