3689 - Expert Evaluation of Patient-Oriented AI-Generated Responses Regarding Radiotherapy for Nonmalignant Indications
Presenter(s)
N. G. Nelson, J. Massachi, and A. Raldow; Department of Radiation Oncology, University of California, Los Angeles, Los Angeles, CA
Purpose/Objective(s): The use of AI chatbots is on the rise by patients hoping to better understand their medical care. Likewise, there has been an upward trend in the application of radiotherapy in the management of nonmalignant or “benign” conditions. We hypothesized that chatbots would generate accurate but incomplete patient-oriented responses regarding the use of radiotherapy for nonmalignant diseases, as assessed by radiation oncologists.
Materials/Methods: Prompts were based on a list of questions by the American Cancer Society for patients with newly diagnosed cancer. These were modified into three simulated patient inquiries regarding radiotherapy for nonmalignant conditions, which were each entered into a new instance of OpenAI’s ChatGPT with a 200-word limit. Four nonmalignant indications were selected to represent different radiotherapy techniques: trigeminal neuralgia, refractory ventricular tachycardia, Dupuytren contracture, and osteoarthritis. ChatGPT responses were then reviewed by attending radiation oncologists who were blinded to the source of the responses. They rated their agreement with the accuracy and completeness of responses on a 5-point Likert scale from “Strongly Disagree” to “Strongly Agree.”
Results: AI-generated responses were evaluated by 10 attending radiation oncologists. A majority agreed or strongly agreed with both the accuracy (mean 4.35 ± 0.78, range 1–5) and the completeness (mean 4.34 ± 0.65, range 2–5) of each response for each nonmalignant condition, including the completeness of responses to “What are the treatment options for refractory ventricular tachycardia” (mean 4.20 ± 0.42), “Dupuytren contracture” (mean 4.30 ± 0.67), and “osteoarthritis” (mean 4.40 ± 0.70), which did not mention radiotherapeutic modalities.
Conclusion: Different levels of evidence support the use of radiotherapy in specific nonmalignant conditions. Regarding these indications, attending radiation oncologists tended to agree with responses to relevant patient questions that were generated by a popular AI chatbot, including responses to open-ended prompts that omitted mentioning radiotherapy.