3738 - A Recent Evaluation on the Performance of LLMs on Radiation Oncology Physics Using Questions of Randomly Shuffled Options

Purpose/Objective(s): We present a study to evaluate the performance of large language models (LLMs) in answering radiation oncology physics questions, focusing on the recently released models.

Materials/Methods: A set of 100 multiple-choice radiation oncology physics questions, previously created by a well-experienced physicist following the official study guide of the American Board of Radiology, was used for this study. The answer options of the questions were randomly shuffled five times to create five "new" exam sets. Five LLMs -- OpenAI o1-preview, GPT-4o, LLaMA 3.1 (405B), Gemini 1.5 Pro, and Claude 3.5 Sonnet -- with the versions released before September 30, 2024, were queried through Application Programming Interface (API) services using these new exam sets. To further evaluate their deductive reasoning ability, the correct answer options in the questions were replaced with "None of the above." Then, the explain-first and step-by-step instruction prompts were used to test if this strategy improved their reasoning ability. The performance of the LLMs was compared with that of the medical physicists.

Results: All models demonstrated expert-level performance on these questions, with o1-preview even surpassing medical physicists with a majority vote. When replacing the correct answer options with "None of the above", all models exhibited a considerable decline in performance, suggesting room for improvement. The explain-first and step-by-step instruction prompts helped enhance the reasoning ability of the LLaMA 3.1 (405B), Gemini 1.5 Pro, and Claude 3.5 Sonnet models. Table 1 shows the performance of LLMs on radiation oncology physics, evaluated using questions with randomly shuffled options. The performance accuracy of each LLM is reported as the average score percentage across all five trials. The uncertainty associated with the reported accuracy is not included in the table due to word count limitations.

Conclusion: These recently released LLMs demonstrated expert-level performance in answering radiation oncology physics questions, exhibiting great potential to assist in radiation oncology physics education and training. The method of randomly shuffling answer options can be used a novel data augmentation technique for both model evaluation and pre-training in natural language processing.