3691 - Improving Clinical Trial Screening for High-Risk Bone Metastasis Using a Large Language Model and Synthetic Data

Purpose/Objective(s): Screening participants for clinical trials is a critical yet resource-intensive process. This study investigates the effectiveness of GPT3.5-generated synthetic data in improving large language model (LLM) models for identifying high-risk bone metastasis patients.

Materials/Methods: A dataset of 423 oncology patients with bone metastases was used to train an LLM-based classifier to identify patients with CT-imaging reports showing at least one of five high-risk clinical criteria including involvement of the (1) junctional spine, (2) long bone, (3) hip/shoulder joint, (4) sacroiliac joint and (5) bulky disease. We trained BlueBERT, a biomedical language model, to recognize these five criteria under two training pipelines: a baseline pipeline, trained exclusively on real patient data, and a synthetic pipeline, incorporating GPT3.5-generated synthetic oncology notes. Synthetic data was generated by prompting GPT3.5 to create CT or PET descriptions of bone metastases without exposing GPT3.5 to real patient data. Model performance was assessed using the area under the receiver operating characteristic curve (AUC), sensitivity, positive predictive value (PPV), and F1 score, which is a harmonic mean of sensitivity and PPV.

Results: The inclusion of synthetic data significantly improved model performance. The baseline BlueBERT model achieved an AUC of 0.876, while models trained with 10k, 20k, and 40k synthetic samples achieved AUCs of 0.907, 0.913, and 0.905, respectively. Sensitivity improved from 0.759 (baseline) to a maximum of 0.876 (Synthetic-10k), while PPV increased from 0.634 to 0.685 (Synthetic-20k). F1 score also improved from 0.676 to a maximum of 0.751 (Synthetic-20k) with the inclusion of a synthetic dataset.