1265 - Unifying Vision-Language Foundation Models via Multi-Teacher Knowledge Distillation for Cross-Modality Medical Image Understanding and Interpretation
Presenter(s)
X. Yang1, S. Wang1, M. Safari1, M. Hu2, Y. Gao3, C. W. Chang1, R. L. J. Qiu1, J. R. Roper1, T. Liu4, H. A. Al-Hallaq1, R. Jagsi1, Z. Tian5, and D. S. Yu1; 1Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, 2Department of Radiation Oncology, Winship Cancer Institute of Emory University, Atlanta, GA, 3Department of Radiation Oncology, Emory University, Atlanta, GA, 4Department of Radiation Oncology, Icahn School of Medicine at Mount Sinai, New York, NY, 5Department of Radiation & Cellular Oncology, The University of Chicago, Chicago, IL
Purpose/Objective(s): Vision-language foundation models (VLFMs) offer a scalable approach to medical image understandingand interpretation, yet most biomedical contrastive language-image pretraining (CLIP) models are trained in narrow domains and exhibit limited and inconsistent generalization across modalities and tasks. To address this limitation, we propose MCKD-VLFM, a VLFM that leverages multi-CLIP knowledge distillation (MCKD) to integrate complementary strengths from multiple biomedical CLIP teachers into a single unified representation.
Materials/Methods: MCKD-VLFM uses a two-stage pipeline: 1) CLIP-style contrastive pretraining on 2.9M PMC-OA image-text pairs spanning 26 imaging modalities; 2) Offline distillation on 16.5M image-text feature quadruplets generated by 9 teacher CLIP models (BMCA, GenMedCLIP, MedCLIP, BiomedCLIP, UniMedCLIP, PLIP, PUBMEDCLIP, PMCCLIP, and QuiltNet). The student adopts a MetaCLIP ViT-B/16 visual encoder and BioMed-BERT text encoder. To reconcile various teacher embedding dimensions, all teacher features are mapped into a shared 512-D latent space using teacher-specific projections and a shared dual-stream autoencoder. Training optimizes a weighted objective combining CLIP alignment, feature distillation, and interactive contrastive learning. We evaluated MCKD-VLFM on 58 public benchmarks spanning 9 imaging modalities (MRI, CT, X-ray, endoscopy, fundus, OCT, ultrasound, dermatology, pathology), comprising >1M images, across 6 task categories: zero-shot and linear-probe classification, cross-modal retrieval, visual question answering (VQA), cancer diagnosis and survival prediction.
Results: MCKD-VLFM demonstrated the strongest overall generalization, outperforming all individual teachers in zero-shot classification and exceeding the best teacher in seven of nine modalities. In limited-label settings (1% training data), it achieved an AUC of 94.4% for MRI (+5.5% vs GenMedCLIP; p<0.01) and 79.8% for fundus (+5.7% vs UniMedCLIP; p<0.01), while maintaining robust pathology performance (AUC 88.6% vs PLIP 86.3; p<0.01). The model led cross-modal retrieval on MedTrinity-25M (Recall at 50: 15.2% text-to-image, 16.6% image-to-text; p<0.01) and BookSet (text-to-image Recall at 50: 53.0%; p<0.001). For VQA, it achieved 80.2% accuracy on SLAKE and 70.1% on VQA-RAD, and improved TCGA survival prediction (C-index 0.687 vs 0.668; p<0.01).
Conclusion: Multi-teacher knowledge distillation provides an effective pathway for unifying heterogeneous biomedical CLIP models into a single VLFM. MCKD-VLFM achieves robust cross-modality generalization, particularly in low-label settings, supporting clinically relevant oncology applications including image understanding and interpretation, retrieval- and VQA-based decision support, and treatment outcome prediction.