3707 - Transforming Clinical Reports into Actionable Insights: Report Querying Using LLMs and Assistants API

Purpose/Objective(s): Searching through clinical reports accumulated over many years for specific answers is a time-consuming and resource-intensive task. We propose a solution using the Application Programming Interface (API) tool to efficiently query and extract information from patient records, whether individually or in bulk. This approach streamlines tasks like monitoring conditions, detecting changes, or preparing data for research on vasculopathy, imaging findings and more enhancing both productivity and precision in clinical data analysis.

Materials/Methods: The study utilized a dataset comprising clinical reports from 160 pediatric craniopharyngioma patients enrolled in an Institutional Review Board-approved Intensity Modulated Proton Therapy protocol. To investigate the accuracy of the Assistants approach two groups were established: 30 patients with signs of vasculopathy and another 30 patients without vasculopathy as a comparison group. Patient data protection was ensured by removing all identifying information from the clinical reports and through a Non-Disclosure Agreement with the vendor. Text files were uploaded as separate vector stores to avoid cross-referencing. The latest version (v3) of the Assistants API File Search tool with the ChatGPT-4o model was used for querying the reports.

Results: The Assistants API completed the inquiry, “Based on the report, classify whether the patient: (1) received proton radiation treatment, (2) was not treated at this institution, or (3) lacks sufficient information to determine the treatment status,” for 160 reports in 19.4 minutes. The median inquiry time per report was 6.1 seconds, ranging from 3.4 to 95.7 seconds, with the maximum being an outlier due to network delays. It identified 155 cases that received proton radiation treatment and 5 lacked sufficient information. For the inquiry “Does this patient have vasculopathy?” the Assistants API identified 80% of vasculopathy cases and 93.3% without vasculopathy for established groups. Across all 160 reports, the API classified 34.4% as having vasculopathy and 65.6% as not having vasculopathy. When the inquiry was rephrased to a more detailed question, “Is there evidence of any vascular abnormalities, including blood vessel narrowing, stenosis, aneurysm, vasculopathy, hemorrhage, cavernoma, occlusion, hemosiderin deposition, or signs of stroke?” the API achieved improved performance, correctly classifying 93.3% of vasculopathy cases and 96.7% of non-vasculopathy cases. Of all 160 reports, it classified 37.5% as having vasculopathy and 62.5% as not.

Conclusion: We implemented the Assistants API tool to efficiently and accurately query clinical reports, significantly boosting productivity. Our findings highlight that precise and detailed prompt engineering and quality reports greatly enhance the API's performance and accuracy.