2444 - Performance of Large Language Models in Identifying Eligible Radiation Oncology Clinical Trials from Unstructured Clinical Notes

Purpose/Objective(s):

Large language models (LLMs) have been used to recommend oncology clinical trials, including earlier systems such as GPT-3.5 and GPT-4. The performance of newer LLMs in radiation oncology clinical trial matching remains unclear. This study evaluates four LLMs and tests the hypothesis that these models can accurately identify eligible, actively recruiting radiation oncology clinical trials.

Materials/Methods:

One hundred de-identified radiation oncology patient cases derived from unstructured clinical notes were evaluated using four LLMs: Claude-4.5-Sonnet, GPT-5.2-Regular, GPT-5.2-Thinking, and Grok-4.1-Thinking. For each patient, models were prompted to identify matching trials from a master list of 1,484 radiation oncology studies obtained from ClinicalTrials.gov, including trials that were active recruiting, active not recruiting, completed, and not yet recruiting. Models were instructed to recommend only actively recruiting trials for which the patient met eligibility criteria and were permitted to make reasonable clinical assumptions when data were not explicitly available.

Due to API constraints and time-stalled outputs, not all models completed all 100 cases. Incomplete responses from the LLMs were excluded. Across models, 262 model-patient evaluations yielded 519 trial recommendations. Recommendations were independently reviewed by two reviewers with consensus resolution. The primary endpoint was precision, defined as the proportion of recommended trials confirmed eligible by both reviewers. Missed patients were defined as cases in which a model failed to recommend at least one trial deemed eligible by another model and confirmed by both reviewers.

Results:

Performance varied across models. GPT-5.2-Regular demonstrated the highest observed precision (84.8%), recommending 298 trials across 92 patients. Claude-4.5-Sonnet achieved 82.0% precision across 13 patients with a 9.1% missed-patient rate. Grok-4.1-Thinking achieved 73.5% precision across 64 patients and had the fastest mean response time of 72.6 seconds. GPT-5.2-Thinking demonstrated 75.0% precision but recommended only 70 trials across 93 patients, with 77.6% meeting the missed-patient definition and exhibiting the longest response time of 769.1 seconds.

Conclusion:

The evaluated LLMs achieved precision ranging from 73.5% to 84.8% in identifying eligible, actively recruiting radiation oncology trials. Findings are limited by incomplete model coverage due to API constraints and reliance on retrospective patient summaries. These results suggest LLMs show promise for automated clinical trial matching, though further evaluation and workflow integration will be necessary before potential clinical use.

Model	Patients (n)	Trials Recommendations	Precision	Missed Patients	Mean Response (s)
Claude 4.5 Sonnet	13	53	82.00%	9.10%	193.8
GPT-5.2 Regular	92	298	84.80%	6.40%	270.8
GPT-5.2 Thinking	93	70	75.00%	77.60%	769.1
Grok 4.1 Thinking	64	98	73.50%	17.90%	72.6