3610 - Automating Patient Message Responses Using AI in Radiation Oncology Clinic: What is the Potential?

Purpose/Objective(s): Patient-provider communication via electronic health record (EHR) software systems like EPIC MyChart is resource-intensive. This study assesses the proportion of inbound patient messages in a radiation oncology practice that artificial intelligence (AI) could automate, with or without EHR integration, to optimize clinical efficiency.

Materials/Methods: Inbound MyChart messages from patients to nurses in a radiation oncology office were reviewed for April 2024. A total of 173 messages across 97 threads were manually evaluated by staff. Messages were categorized: (1) answerable by AI with no EHR integration (copy/paste response), (2) answerable with AI access to EPIC and other EHRs, and (3) not answerable by AI even with integration. Feasibility was based on content (e.g., pleasantries, scheduling, physician knowledge). Three AI models (GPT, Gemini, Meta) were tested on automatable pleasantries, rated on a 3-point scale (2=good, 1=ok, 0=not good), compared to nurse responses. Assumptions include consistent patterns and AI language capability. Chi-square tests assessed differences.

Results: Of 173 messages, 76 (44%) were answerable by AI without EHR integration, all of these were “pleasantries” during or at the end of a text exchange; GPT, Gemini, and Meta scored 1.5, 1.3, and 1.6, respectively, vs. nurses’ 1.6 (17/76 responses). Nurses did not respond to 59 /76 (78%) of these patient messages. An additional 49 messages (28%) could be automated with EHR access, including scheduling (33), medications (8), and history (3). The remaining 48 messages (28%) were not automatable, needing physician knowledge (28), insurance information (4), or requests requiring a phone call (7) (p<0.01). Total automatable messages were 125 (72%).

Conclusion: AI could potentially automate 72% of MyChart messages, with 44% being pleasantries and 28% using EHR data for scheduling, medications, and history. Complex queries needing human expertise were further 28%. Free-standing AI models not integrated into the EPIC interface currently bring limited value to our EHR patient interactions. Future work will explore potential integration challenges.