337 - Augmenting Patient Safety Surveillance in Radiation Oncology with Large Language Model-Based Root Cause Analysis
Presenter(s)
Y. Wang1, M. De Ornelas2, M. T. Studenski3, E. L. Bossart4, S. Nejad-Davarani5, and Y. Yang2; 1University of Miami - Sylvester Cancer Center, Coral Gables, FL, 2University of Miami, Miami, FL, United States, 3Department of Radiation Oncology, University of Miami/Sylvester Comprehensive Cancer Center, Miami, FL, 4Department of Radiation Oncology, University of Miami School of Medicine, Miami, FL, 5Department of Radiation Oncology, Sylvester Comprehensive Cancer Center, Miller School of Medicine, University of Miami, Miami, FL
Purpose/Objective(s): We investigated the potential utility of large language models (LLMs) in supporting patient safety efforts. Specifically, we evaluated the reasoning capabilities of LLMs in performing root cause analysis (RCA) of radiation oncology incidents using narrative reports from the Radiation Oncology Incident Learning System (RO-ILS).
Materials/Methods: We prompted four state-of-the-art LLMs, Gemini 2.5 Pro, GPT-4o, o3, and Grok 3, with the “Background and Incident Overview” sections from 19 publicly available RO-ILS cases. Each model was instructed to perform RCA and generate root causes, lessons learned, and suggested actions using a standardized prompt based on AAPM RCA guidelines. Model outputs were evaluated using a combination of objective semantic similarity metrics (cosine similarity via Sentence Transformer), semi-subjective assessments (precision, recall, F1-score, accuracy, hallucination rate and performance criteria including relevance, comprehensiveness, quality of justification and quality of solution), and subjective ratings (reasoning quality and overall performance) by five board-certified medical physicists.
Results: LLMs demonstrated satisfactory performance across evaluation metrics. GPT-4o achieved the highest cosine similarity (0.831), and Gemini 2.5 Pro had the highest recall (0.799) and accuracy (0.918). All models exhibited some degree of hallucination, ranging from 11% to 61%. Gemini 2.5 Pro, which outperformed all other models across performance evaluation criteria, received an overall performance rating of 4.8 out of 5 from expert reviewers. Statistically significant differences were observed among models in accuracy, hallucination rate, and subjective ratings (p < 0.05).
Conclusion: LLMs delivered promising results as assistive tools for RCA in radiation oncology, with the ability to generate relevant and accurate analyses aligned with expert expectations. LLMs may support incident analysis and contribute to quality improvement efforts to advance patient safety in clinical radiation oncology practice.