Presentation (CTI)

OBJECTIVES: Published studies suggest large language models (LLMs) can extract structured clinical information from unstructured physician notes. However, using LLM-derived variables in real-world evidence (RWE) studies intended for publication or HTA submissions requires traceability, standardized outputs, and human review. This pilot evaluated whether a structured LLM workflow could abstract key variables from physician notes with sufficient reliability for evidence generation in a study of patients with hematological malignancy.
METHODS: The pilot used de-identified longitudinal notes from 24 patients in an enterprise LLM environment. Records included several hundred to more than 1,000 notes per patient, with treatment history, lab values, and assessments repeated across notes. A structured master prompt extracted patient-level variables, applied treatment response definitions, and produced summaries, timelines, source snippets, and conflicts/uncertainties. Target variables included treatment response, ECOG, treatment dates/regimens, relapsed/refractory status, stem cell transplant status, and discontinuation reason. First-pass LLM outputs were compared with final adjudicated study values determined through reviewer reconciliation of structured study data, manual abstraction, LLM outputs, and source notes.
RESULTS: Across 96 evaluable non-date fields, first-pass LLM outputs matched adjudicated values in 70.8% of fields and had 5.2% missingness. Date agreement, defined as within ±7 days, was 86.3%. For treatment response, LLM agreement was 66.7%, with 12.5% missingness. LLM-derived treatment duration matched adjudicated duration, with a mean difference of 0.1 days. Performance varied by variable, with lower concordance for co-mutations and stem cell transplant status, highlighting the need for human review.
CONCLUSIONS: A structured prompt with source-referenced outputs helped organize complex longitudinal notes and reduced missingness for selected RWE variables, but human verification remains essential. By making clinically important information embedded in physician notes more accessible, LLM-assisted abstraction may expand the use of unstructured data for RWE analyses. Planned development will focus on batch processing and validation in larger patient samples.