2627 - Graph-Augmented Context Engineering for Radiation Oncology Guideline Retrieval: A Quantitative Failure Analysis of Dense Vector Search

Purpose/Objective(s):

Retrieval-augmented generation (RAG) systems are increasingly proposed for clinical decision support in radiation oncology. Most systems rely on dense vector similarity for context retrieval. However, radiation oncology guidelines frequently encode recommendations within structured tables, algorithms, and conditional logic. We hypothesized that dense vector retrieval alone is insufficient for clinical knowledge extraction and that graph-augmented context engineering improves robustness.

Materials/Methods:

We constructed a radiation oncology knowledge base from AUA SUO guidelines using a LightRAG-based GraphRAG framework. Documents were segmented into 256-token overlapping chunks (32-token overlap) and indexed using cosine similarity (top-k=5). Two local LLMs (phi4:14b and llama3.1:8b) were evaluated for entity and relationship extraction. We compared (1) vector retrieval recall, (2) graph-augmented retrieval recall, (3) entity extraction density, (4) citation integrity, and (5) token-level indexing cost. A clinically relevant query from a prostate cancer treatment algorithm was used to test retrieval robustness. Scaling projections to 10,000 pages were estimated from empirical indexing times.

Results:

For the query, “What are the two bone-protective agents recommended for mCRPC patients with bony metastases?”, dense vector retrieval returned 0 of 5 relevant chunks (Recall@5 = 0%), despite the correct answer being present in the source PDF. In contrast, graph-based entity traversal successfully retrieved the relevant content (Recall@5 = 100% via entity-linked subgraph traversal), enabling correct answer generation. Entity extraction density differed between models: phi4:14b extracted 37 entities and 25 relations from a 2-page guideline versus 28 entities and 16 relations with llama3.1:8b (-24% entity density), reflecting a tradeoff between extraction depth and indexing speed (11.1 vs 3.2 minutes per document). Token-level analysis demonstrated that 75% of indexing tokens were instruction-template overhead rather than source text, indicating that semantic extraction—not embedding—is the dominant cost center. Serial local indexing of 10,000 pages was projected to require 11–38 days, whereas parallel API-based extraction reduced projected indexing time to 6–8 hours.

Conclusion:

Dense vector similarity alone may fail to retrieve clinically critical recommendations embedded within structured radiation oncology guidelines. Graph-augmented context engineering may improve retrieval robustness by modeling entity-level relationships that compensate for embedding brittleness. For radiation oncology informatics, where treatment decisions depend on algorithmic logic and exception handling, sophisticated context engineering is essential for clinically trustworthy AI systems. These findings suggest that reliable oncology knowledge systems may require structured semantic indexing rather than similarity search alone.