2614 - LLM-Driven Extraction of NI-RADS and Imaging Tumor Characteristics to Enhance Oropharyngeal Cancer Survivorship Surveillance
Presenter(s)
W. N. Song1, L. Shbita2, I. Jang2, O. Starostina3, R. Lewis4, A. Sahli1, W. Floyd5, M. Mahin6, W. Rinsurongkawong7, C. E. Barbon1, S. Y. Lai8, J. J. Lee9, K. Shah2, M. Chen2, K. A. Hutcheson9, C. D. Fuller5, and A. C. Moreno8; 1The Department of Head and Neck Surgery, The University of Texas MD Anderson Cancer Center, Houston, TX, 2The University of Texas MD Anderson Cancer Center, Houston, TX, 3The Department of Enterprise Data Engineering and Analytics, The University of Texas MD Anderson Cancer Center, Houston, TX, 4The Department of Information Technology, The University of Texas MD Anderson Cancer Center, Houston, TX, 5Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, 6The Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, 7The Department of Quantitative Research Computing, The University of Texas MD Anderson Cancer Center, Houston, TX, 8Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, 9MD Anderson Cancer Center, Houston, TX
Purpose/Objective(s): Radiologic surveillance is essential for oropharyngeal cancer (OPC) survivors, guiding recurrence detection and informing follow-up strategies. The Neck Imaging Reporting and Data System (NI-RADS) provides a standardized framework for post-treatment risk assessment at both the primary tumor site and cervical lymph nodes. Comprehensive surveillance further requires assessment of disease status, including the primary tumor, nodal involvement, and distant metastases. These clinical parameters are often embedded as unstructured data within free-text radiology reports. We hypothesized that a large language model (LLM) can reliably extract NI-RADS scores and key imaging features from unstructured radiology text, achieving high concordance with expert review.
Materials/Methods: Previously untreated OPC patients who received definitive radiation therapy were identified. Eligible imaging reports included post-treatment head and neck CT, MRI, or FDG PET/CT scans containing narrative and impression text. Examinations lacking narrative or impression text, containing pre-existing NI-RADS annotations, or involving non-surveillance imaging modalities were excluded. A total of 200 reports were randomly selected from 7,076 eligible examinations for manual abstraction using a three-reviewer consensus framework to establish a gold standard dataset. Using the Palantir Foundry Pipeline Builder, a GPT-5-based LLM was deployed to extract pNI-RADS, nNI-RADS scores, and key imaging features of disease status from these reports. Agreement was assessed using exact match, with F1 scores reported as complementary metrics.
Results:
Agreement for pNI-RADS 1 was 94.9% (130/137; F1 = 0.94), and for nNI-RADS 1 was 84.0% (131/156; F1 = 0.88). For higher-risk categories (=2), agreement was 80.4% (41/60; macro-F1 = 0.79) for pNI-RADS, and 65.0% (26/40; macro-F1 = 0.59) for nNI-RADS, respectively. Agreement for identifying absence of residual or recurrent primary tumor was 86.8% (125/144; F1 = 0.89) while agreement for the absence of nodal disease was 84.5% (109/129; F1 = 0.88). For distant metastatic disease, agreement was 82.3% for cases without metastases (116/141; F1 = 0.82) and 100% for cases with confirmed metastases (7/7; F1 = 0.64). The LLM demonstrated variable performance in extracting detailed imaging descriptors, including primary tumor site, laterality, and size; nodal laterality and burden; and metastatic sites when reported.Conclusion: An LLM-based extraction approach demonstrates substantial concordance with expert review for NI-RADS stratification and confirmation of absence of disease in OPC survivorship surveillance. Its robust performance in identifying low-risk disease supports application in longitudinal monitoring and scalable survivorship research in head and neck oncology.