100 - Multimodal Modeling of Detailed Cancer Subtypes and Molecular Features from >60,000 Patients with Co-Registered H&E Images and Clinical Tumor Sequencing
Presenter(s)
K. M. Boehm1, M. Darmofal2, A. Aukerman2, A. Pasha2, R. Lim2, T. Pollard2, D. Moore2, J. F. Chen2, N. Rekhtman2, H. Al-Ahmadie2, J. Chang2, N. Y. Lee2, L. R. G. Pike1, H. Nagar1, J. Janopaul-Naylor1, L. Z. Braunstein2, M. Berger2, N. Schultz3, S. P. Shah2, and F. Sanchez-Vega2; 1Department of Radiation Oncology, Memorial Sloan Kettering Cancer Center, New York, NY, 2Memorial Sloan Kettering Cancer Center, New York, NY, 3Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, NY
Purpose/Objective(s): Detailed tumor subtyping and molecular profiling are required for optimal therapy. Prior digital pathology work in this field has suffered from small sample sizes and a lack of co-registered multimodal genomics and outcomes. The role of digital pathology as a complement to DNA sequencing rather than a surrogate also remains underexplored. This study develops multimodal artificial intelligence (AI) for H&E whole-slide images (WSIs) to model detailed cancer subtypes, identify actionable biomarkers, and evaluate the added value of digital pathology to clinical sequencing.
Materials/Methods: We curated a pan-cancer cohort of 367,535 H&E WSIs with matched clinical sequencing from 62,460 patients across 124 OncoTree subtypes. We trained two transformer-based models, Aeon and Paladin, to infer OncoTree codes and multiscale genomic features from WSIs, respectively. Three independent test sets were used.
Results: In inferring cancer subtypes, Aeon achieved AUROC 0.996 across 124 classes, outperforming contemporary models, including TITAN, CHIEF, and GDDENS (genomics model). Integrating Aeon and GDDENS inferences improved accuracy for challenging metastatic cases from 71% to 78%, and reclassified cancers of unknown primary with concordant genomic and prognostic distributions. In inferring multi-scale molecular features, Paladin identified characteristic H&E phenotypes for 165 (5%) of 3,541 tested biomarkers, improving on benchmarks. Paladin also identified likely oncogenic variants among variants of unknown significance (VUS) in STK11 in lung adenocarcinoma: patients with tumors harboring a VUS that exhibited this phenotype suffered shorter overall survival (log-rank p<0.05) compared to those without the phenotype, with lower RNA-seq STK11 abundance (Mann-Whitney U (MWU) p<0.05) and loss of STK11 on immunohistochemistry (MWU p<0.05). Furthermore, wildtype (WT) cases with STK11-mutant H&E phenotype formed an intermediate prognostic group between STK11-mutant cases and WT cases without STK11-mutant phenotype (log-rank p=0.01), suggesting prognostically significant phenocopying.
Conclusion: This work establishes multimodal AI models using a large multimodal tumor dataset, advancing AI-based cancer diagnostics to 124 detailed subtypes, with prognostic value for cancers of unknown primary. It also establishes digital pathology for interpretation of clinical sequencing results, including VUS and phenocopying states, with predictive and prognostic value.