Main Session
Sep
28
PQA 04 - Breast Cancer, Patient Reported Outcomes/QoL/Survivorship, Functional Radiation Medicine, Hematologic Malignancies, Palliative Care, and International/Global Oncology
2910 - Performance of an AI Language Model in Identifying Patients with High-Risk Asymptomatic Bone Metastases on PET/CT Imaging
Presenter(s)
Robert Walker, MD, MS - Thomas Jefferson University, Philadelphia, PA
R. Walker1, W. Forrest2, W. Cao3, K. W. Mund1, W. Choi1, Z. Chen1, Y. Vinogradskiy4, T. A. LaCouture1, G. S. Alexander1, and D. R. Cohen1; 1Department of Radiation Oncology, Sidney Kimmel Medical College at Thomas Jefferson University, Philadelphia, PA, 2Cooper Medical School of Rowan University, Camden, NJ, 3Department of Radiation Oncology, Sidney Kimmel Medical College & Cancer Center at Thomas Jefferson University, Philadelphia, PA, 4Dept. of Radiation Oncology, Sidney Kimmel Medical College and Comprehensive Cancer Center, Thomas Jefferson University, Philadelphia, PA
Purpose/Objective(s):
Skeletal-related events (e.g., pathologic fracture, spinal cord compression) due to bone metastases are a common source of morbidity in patients with Stage IV malignancies. Palliative radiation therapy (RT) is the standard of care and has historically been reserved for symptomatic bone metastases. Prophylactic treatment of asymptomatic high-risk metastases (HRMs) is associated with a reduction in hospitalizations and improved overall survival, yet is underreported in routine PET/CT reporting and under-referred for radiotherapy. Methods are needed to automatically identify patients who are candidates for prophylactic treatment of asymptomatic HRMs. This study tested whether local LLMs can accurately screen reports to flag candidates for prophylactic RT (PRT).Materials/Methods:
Patients with Stage IV malignancies evaluated by radiation oncology and who underwent at least 1 PET/CT scan at our single institution from 2020 to 2025 were retrospectively identified. PET/CT reports were manually reviewed by an expert and classified as binary regarding PRT candidacy to establish the ground truth. One PET/CT per patient was used. HRMs were defined using established criteria, including bulky disease (>2cm), involvement of high-risk joints (hip, shoulder, SI), long-bone cortical involvement (1/3–2/3 thickness), or junctional/posterior vertebral involvement. Six locally deployed LLMs (two clinically trained, four general purpose; 4-120B parameters) were evaluated. LLM predictions were compared to expert reviewer ground truth to establish model accuracy, sensitivity, specificity, and positive predictive value (PPV).Results:
Ninety-three patients were included; 19 (20%) met HRM criteria. The best-performing model achieved 92.8% accuracy, 88.2% sensitivity, and 96.3% specificity. Three models achieved 100% sensitivity, identifying all HRM candidates, with specificity ranging from 85% to 88%. PPV ranged from 37% to 75% across models. DeepSeek-R1 showed the most balanced performance, providing the highest accuracy and precision. However, Llama-3.3 and MedGemma-27b were the best screening LLMs by catching 100% of candidates with HRMs while maintaining high accuracy.Conclusion:
These findings demonstrate that local LLMs can accurately and safely identify asymptomatic HRMs that may benefit from PRT. Future integration into the electronic medical record (e.g., Epic) may enable automated screening of PET/CT reports and real-time flagging of high-risk metastases for radiation oncology evaluation.