216 - Artificial Intelligence-Based Incident Learning System
Presenter(s)
A. J. Jinia1, K. L. Chapman1, S. Liu1, C. Della Biancia1, E. Hipp1, E. Lin1, R. Moulder2, D. R. Parikh3, J. Cordero3, C. Pistone3, M. Gil3, J. Ford4, A. Li1, and J. M. Moran1; 1Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY, 2Division of Quality and Safety, Memorial Sloan Kettering Cancer Center, New York, NY, 3Department of Radiation Oncology, Memorial Sloan Kettering Cancer Center, New York, NY, 4Department of Nursing, Memorial Sloan Kettering Cancer Center, New York, NY
Purpose/Objective(s):
Recently, the development of artificial intelligence (AI), particularly large language models (LLMs), has accelerated, enabling them to tackle complex tasks like analyzing incidents in medical incident learning systems (ILS). By automating incident analysis, AI helps organizations process vast amounts of data, uncovering valuable insights that improve treatment quality and patient safety in radiation therapy. In this study, we design and evaluate an AI-ILS framework using LLMs to analyze radiation therapy incidents with the Human Factors Analysis and Classification System (HFACS).
Materials/Methods:
The core component of the AI-ILS is the LLM, specifically Llama 3.1. This open-source model was trained on 1,241 mock incidents created by a diverse team of radiation oncologists, physicists, dosimetrists, therapists, quality assurance staff, and nurses. The dataset was designed to be comprehensive, balanced and representative of various incident scenarios, minimizing the risk of real-world bias. The model's performance was assessed using an additional 307 mock incidents and the HFACS framework, before and after training. The trained model was applied to 350 real-world incidents, with resulting classifications reviewed by two experts for accuracy. All incidents were pre-processed to expand acronyms and anonymize sensitive information.
Results:
Before training, the model achieved 53% accuracy, with an area under the receiver operating characteristic curve (AUROC) of 0.51 and a Matthew's correlation coefficient (MCC) of 0.38. After training, its performance significantly improved to 80% accuracy, with an AUROC of 0.92 and an MCC of 0.73. This improvement indicates that the model initially demonstrated a broad understanding of radiation oncology incidents but struggled to associate nuanced details with the HFACS framework - a limitation effectively addressed during training.
When classifications were reviewed by experts for real-world incidents, the reviewers agreed with the model's classification in 79% of cases, but were uncertain in 10% due to limited information in the incident reports. This uncertainty often requires a thorough investigation to determine the root cause. The analysis identified three primary contributing factors: Errors, Personnel Factors, and Organizational Processes, consistent with trends reported in the literature. Notably, the AI-ILS completed the analysis in just 31 minutes, whereas human review took over 10 hours.