163 - Cross-Institutional Validation of LLM-Based Cardiac Event Extraction from Electronic Health Records
Presenter(s)
W. Cao1, I. Lloyd2, M. Dichmann3, N. Halder3, D. Thomas4, Z. Chen3, E. Blomain5, C. Jiang2, F. Adejolu6, K. Beck7, D. M. Sopka7, P. Faherty8, M. Khan2, N. L. Simone4, V. Jain3, E. Storozynsky9, A. P. Dicker3, W. Choi4, and Y. Vinogradskiy4; 1Department of Radiation Oncology, Sidney Kimmel Medical College & Cancer Center at Thomas Jefferson University, Philadelphia, PA, 2Thomas Jefferson University, Philadelphia, PA, 3Department of Radiation Oncology, Sidney Kimmel Medical College at Thomas Jefferson University, Philadelphia, PA, 4Dept. of Radiation Oncology, Sidney Kimmel Medical College and Comprehensive Cancer Center, Thomas Jefferson University, Philadelphia, PA, 5Department of Radiation Oncology, Thomas Jefferson University, Philadelphia, PA, 6Department of Radiation Oncology, Thomas Jefferson University Hospital, Philadelphia, PA, 7Lehigh Valley Health Network, Allentown, PA, 8Thomas Jefferson University Hospital, Philadelphia, PA, 9Department of Cardiology, Sidney Kimmel Medical College at Thomas Jefferson University, Philadelphia, PA
Purpose/Objective(s): Cardiotoxicity is a significant concern for cancer patients following radiotherapy. Several barriers make cardio-oncology research time and resource intensive, including the time required to manually review Electronic Health Records (EHRs) and the need for large datasets. Having a reliable resource to identify cardiac events after radiation could facilitate this research. We developed and validated a novel large language model (LLM)-based framework to automate the identification of cardiac events using a large two-institution dataset.
Materials/Methods: A total of 411 lung and breast cancer patients from two institutions were analyzed. Institution 1 data (n=266) were divided into a development cohort (lung, n=178) and an internal validation cohort (breast, n=88). External validation used Institution 2 data (lung, n=145). Cardiac events were physician-adjudicated from longitudinal EHRs and included myocardial, ischemic, valvular, constrictive, and conduction events occurring both before and after cancer diagnosis and treatment. EHR data comprised structured problem lists and unstructured clinical notes. We introduced the Two-phase Reasoning for Automated Cardiac Event Recognition (TRACER) framework, which combines structured term matching with LLM-based analysis of unstructured notes. LLM prompts incorporated specialty-filtered notes to reduce irrelevant context, temporally aware queries to enforce event timing constraints, and few-shot examples to steer structured reasoning. Three open-source LLMs (DeepSeek-R1, Llama-3.3, Mistral-Large) were deployed locally. Performance was evaluated against physician-adjudicated ground truth using accuracy and processing time.
Results: Of 411 patients, 220 had at least one cardiac event documented at any time point in the longitudinal EHR (pre-existing or post-treatment). Mean accuracy was 80%, 81%, and 82% for development, internal validation, and external validation cohorts, respectively. DeepSeek-R1 achieved the highest accuracy on internal cohorts (84–85%), while Mistral-Large reached 88% accuracy on external validation. Mean TRACER processing time was 20–42 seconds per patient (2.3–4.8 hours for 411 patients) versus 2 hours per patient for manual review (822 person-hours total).
Conclusion: TRACER, a locally deployed LLM framework, accurately extracted cardiac events across institutions and disease sites, achieving accuracies up to 88% while reducing processing time 170-fold compared to manual review. This approach enables scalable, privacy-preserving cardio-oncology research by substantially reducing the resources required for cardiac event identification.