2589 - Prospective Assessment of the Impact of AI on Peer Review of Elective Pelvic Nodal Segmentations
Presenter(s)
M. C. Repka1, L. B. Marks1, E. Byrd2, B. M. Anderson1, S. K. Das3, Y. Gao2, K. Adapa4, R. McGurk5, and L. Mazur1; 1Department of Radiation Oncology, University of North Carolina, Chapel Hill, NC, 2UNC Chapel Hill, Chapel Hill, NC, 3University of North Carolina, Chapel Hill, NC, 4Department of Radiation Oncology, University of North Carolina at Chapel Hill, Chapel Hill, NC, 5UNC School of Medicine, Chapel Hill, NC, United States
Purpose/Objective(s): Peer review of MD-defined elective pelvic nodal volumes can be cognitively demanding due to large target volumes, ill-defined anatomic boundaries, and time constraints. We previously described an artificial intelligence (AI)-based tool that can accurately generate elective pelvic nodal volumes (validation Dice score of 0.79 [0.71–0.86]) We herein report on a prospective trial to assess whether these AI-generated pelvic nodal volumes could enhance peer review of MD-defined pelvic nodal volumes.
Materials/Methods: Sixteen participants (radiation oncologists, dosimetrists, and physicists) were assigned to four multidisciplinary teams. Each team reviewed the MD-defined elective pelvic nodal volumes on the CT images from 10 prostate cancer cases twice (20 total review sessions per team, and 80 sessions for the four teams). In the 1st session, the team reviewed MD-defined contours only. In the 2nd session, the MD-defined and AI-generated volumes were reviewed simultaneously. Sessions were separated in time by >3-4 weeks, and the case order was randomized to minimize recall bias during the second session. For each session, the quality of the peer review was assessed by the number of agreed-upon moderate or major recommendations (minor recommendations were not considered in this analysis); comparisons made with the ?2 test. For the second session, teams subjectively assessed whether they found AI contours useful (yes/no) and whether they preferred the MD- or AI-derived volumes. Two physicians who did not participate in the peer review sessions independently reviewed A/V recordings to validate the clinical recommendations and assess impact on clinical discussion.
Results: The total number of major and moderate peer review recommendations were similar across 1st and 2nd sessions (mean 1.2 per case per session). The identified recommendations were concordant across reviewers and commonly related to under-coverage of external iliac nodal basin, over-inclusion of bowel, and the superior border of the target volume. In the 2nd session, teams reported the AI-generated contour as useful in 68% of evaluations, and they preferred the AI-generated contour in 25% of evaluations. Compared to session 1, reviewers in session 2 identified new recommendations in 6% of cases. However, in 5% of the evaluations there were findings in session 1 that were not identified during session 2. Qualitative review of AV recordings suggested risks of distraction and attention tunneling when participants were presented with MD- and AI-volumes concurrently.
Conclusion: An AI-generated secondary pelvic nodal volume can enhance peer review. However, risks of distraction and attention tunneling were observed when MD- and AI-volumes were reviewed concurrently. Careful workflow integration is required to mitigate these issues and further research is needed to better understand how to incorporate safe and trusted human-AI workflows in radiation oncology peer review.