Pith sign in

arxiv: 2606.28395 · v1 · pith:QCHY3GDA · submitted 2026-06-24 · cs.CV

JASPR: Joint Spatial Representation learning of histology and spatial genomics for improved virtual genomic screening and clinical prognostication

Marija Pizurica , Eric Zimmermann , Neil Tenenholtz , James Hall , Olivier Gevaert , Ava P. Amini , Lorin Crawford , Kristen A. Severson This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved 2026-06-30 01:44 UTCgrok-4.3pith:QCHY3GDArecord.jsonopen to challenge →

Figure 1
Figure 1. Figure 1: Overview of JASPR. a) The input to JASPR is a window of n × n tokens (n = 8 throughout), representing HE embeddings extracted with Virchow2 (d = 1280), or gene expression measurements from spatial transcriptomics data (d = 9248). (b)-g)) Learning objectives and input-output pairs for JASP… reproduced from arXiv: 2606.28395
classification cs.CV
keywords spatial transcriptomicshistology imagesjoint representation learninggene expression predictionbreast cancer prognosisself-supervised learningcross-modal reconstructionvirtual genomic screening
0
0 comments X

The pith

JASPR learns joint spatial representations from HE images and spatial transcriptomics to improve prediction of 9248 genes and breast cancer prognosis from images alone.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents JASPR, a self-supervised framework that combines hematoxylin and eosin whole slide images with spatial transcriptomics profiles. It trains by reconstructing one modality from the other while preserving spatial layout in both, using shared modules for properties common to both data types and separate experts for morphology-specific or genomics-specific details. The learned representations are reported to raise the accuracy of gene expression prediction from HE images alone across 9248 genes and to add value when forecasting patient outcomes in breast cancer. This matters because spatial transcriptomics remains costly and limited in availability, so stronger virtual screening from routine slides could broaden access to spatial molecular information in research and care.

Core claim

JASPR integrates HE images and ST data through a cross-modal reconstruction objective that incorporates spatial context within HE images and ST profiles. It employs shared modules to capture universal spatial properties across modalities, while modality-specific experts encode features unique to morphological and genomic data. Trained and validated on breast cancer datasets, its learned joint representation substantially improves HE-based prediction of 9248 genes and provides prognostic value for breast cancer outcomes.

What carries the argument

Cross-modal reconstruction objective that incorporates spatial context within HE images and ST profiles, using shared modules and modality-specific experts.

If this is right

  • HE images alone can predict expression levels for 9248 genes with substantially higher accuracy than prior methods.
  • The joint representation adds measurable value to clinical outcome prediction in breast cancer.
  • Spatial context is effectively preserved and utilized across the two modalities during learning.
  • The framework demonstrates practical utility for virtual genomic screening on standard pathology slides.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Routine pathology slides could serve as a cheaper proxy for expensive spatial molecular profiling in more settings.
  • The same joint-learning pattern might apply to other paired imaging and sequencing modalities in different diseases.
  • Universal spatial patterns captured by the shared modules could point to common principles of tumor organization.
  • Testing the representations on non-breast cancers would reveal whether the gains transfer beyond the training domain.

Load-bearing premise

The cross-modal reconstruction objective incorporating spatial context within HE images and ST profiles, using shared modules and modality-specific experts, will produce representations that generalize to substantially improved gene prediction and prognostication on breast cancer datasets.

What would settle it

A direct comparison on held-out breast cancer data showing that an HE-only baseline achieves the same or higher accuracy in predicting the 9248 genes and the same or better survival stratification as the JASPR joint representation.

Figures

Figures reproduced from arXiv: 2606.28395 by Ava P. Amini, Eric Zimmermann, James Hall, Kristen A. Severson, Lorin Crawford, Marija Pizurica, Neil Tenenholtz, Olivier Gevaert.

Figure 2
Figure 2. Figure 2: JASPR Virtual screening performance when varying hyperparameters, as specified on the x-axis, averaging over other hyper [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Tile-level Virchow2 baseline versus JASPR performance for virtual screening across all 9,248 genes, in the held-out HEST1k [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Clinical and biological applications of JASPR and baselines. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 1
Figure 1. Figure 1: Downstream application of JASPR features and compared baseline architectures. a) JASPR. b) Baseline. c) CLIP contrastive [PITH_FULL_IMAGE:figures/full_fig_p013_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Gene-level Pearson Correlation Coefficient (PCC) versus ground-truth, non-zscored standard deviation (STD), colored by Mean [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Histogram of gene-level Pearson Correlation Coefficient (PCC). [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
read the original abstract

Recent studies have shown that spatial properties of tumors are critical for understanding disease biology and predicting patient outcomes. These spatial properties are increasingly uncovered through complementary modalities: spatial transcriptomics (ST) captures spatially-resolved molecular states, while hematoxylin and eosin-stained whole slide images (HE) reveal tissue morphology. While approaches are emerging to fuse these modalities, effective methods that learn not only joint representations but also incorporate spatial context across modalities are lacking. Here, we present JASPR (Joint Spatial Representation learning), a self-supervised deep learning framework that integrates HE images and ST data through a cross-modal reconstruction objective that incorporates spatial context within HE images and ST profiles. It employs shared modules to capture universal spatial properties across modalities, while modality-specific experts encode features unique to morphological and genomic data. We train and validate JASPR on breast cancer datasets, demonstrating that its learned joint representation substantially improves HE-based prediction of 9,248 genes and provides prognostic value for breast cancer outcomes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper introduces JASPR, a self-supervised deep learning framework for learning joint spatial representations from HE-stained whole slide images and spatial transcriptomics (ST) profiles. It uses a cross-modal reconstruction objective that incorporates spatial context, with shared modules capturing universal spatial properties and modality-specific experts encoding modality-unique features. The central claim is that the resulting representations substantially improve HE-based prediction of 9,248 genes and provide prognostic value for breast cancer outcomes, as demonstrated on breast cancer datasets.

Significance. If the empirical results were shown to hold with appropriate baselines, metrics, and validation, the work could advance multimodal spatial integration in computational pathology by addressing the gap in joint representation learning that incorporates spatial context across modalities. The self-supervised cross-modal approach with shared and expert modules is a plausible direction for improving virtual genomic screening and prognostication, but the absence of any quantitative evidence prevents evaluation of whether these benefits are realized or exceed prior HE-only methods.

major comments (2)
  1. [Abstract] Abstract: The claim that the learned joint representation 'substantially improves HE-based prediction of 9,248 genes' is unsupported by any metrics (e.g., Pearson r, MAE), baseline comparisons, dataset sizes/splits, error bars, or statistical tests. This is load-bearing because the entire contribution rests on these unshown empirical improvements over existing methods.
  2. [Abstract] Abstract: No details are supplied on the cross-modal reconstruction objective, the training/validation protocol, ablation of the spatial-context or expert components, or how prognostic value is quantified (e.g., C-index, survival analysis). Without these, the weakest assumption—that the objective yields generalizable representations—cannot be assessed.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on the abstract. We address each point below and will revise the abstract to incorporate the requested quantitative and methodological details while preserving the manuscript's core contributions.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The claim that the learned joint representation 'substantially improves HE-based prediction of 9,248 genes' is unsupported by any metrics (e.g., Pearson r, MAE), baseline comparisons, dataset sizes/splits, error bars, or statistical tests. This is load-bearing because the entire contribution rests on these unshown empirical improvements over existing methods.

    Authors: We agree that the abstract's qualitative claim would be strengthened by quantitative support. The full manuscript reports these results (Pearson r, MAE, baselines including HE-only and prior multimodal approaches, dataset sizes and splits from breast cancer cohorts, error bars across runs, and statistical tests) in the Results section. We will revise the abstract to include key metrics and baseline comparisons to make the claim self-contained. revision: yes

  2. Referee: [Abstract] Abstract: No details are supplied on the cross-modal reconstruction objective, the training/validation protocol, ablation of the spatial-context or expert components, or how prognostic value is quantified (e.g., C-index, survival analysis). Without these, the weakest assumption—that the objective yields generalizable representations—cannot be assessed.

    Authors: We acknowledge the abstract is high-level. The manuscript details the cross-modal reconstruction objective (including spatial context terms) in Methods, the training/validation protocol and splits in the experimental setup, ablations of spatial-context and expert modules in Results, and prognostic quantification via C-index and survival analysis. We will revise the abstract to briefly reference the evaluation metrics (Pearson correlation for genes; C-index for prognosis) and note the self-supervised objective. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical claims rest on external validation, not self-definition or fitted inputs

full rationale

The abstract and described framework present JASPR as a self-supervised model with cross-modal reconstruction, shared modules, and modality-specific experts. Central claims of improved gene prediction (9,248 genes) and prognostic value are framed as outcomes of training/validation on breast cancer datasets, not as derivations that reduce to inputs by construction. No self-definitional loops, fitted parameters renamed as predictions, or load-bearing self-citations appear in the provided text. The derivation chain is self-contained against external benchmarks and does not invoke uniqueness theorems or ansatzes from prior author work.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 2 invented entities

Abstract-only review provides no explicit free parameters, fitted values, or detailed axioms; framework relies on standard self-supervised learning assumptions and introduces architectural components without independent evidence.

axioms (1)
  • domain assumption Self-supervised cross-modal reconstruction can capture universal spatial properties across modalities
    Core premise invoked in the framework description.
invented entities (2)
  • shared modules no independent evidence
    purpose: To capture universal spatial properties across modalities
    Introduced as part of JASPR architecture
  • modality-specific experts no independent evidence
    purpose: To encode features unique to morphological and genomic data
    Introduced as part of JASPR architecture

pith-pipeline@v0.9.1-grok · 5732 in / 1217 out tokens · 49918 ms · 2026-06-30T01:44:34.391977+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 4 canonical work pages

  1. [1]

    Communications Biol- ogy6(1), 304 (2023) 1, 2

    Alsaafin, A., Safarpoor, A., Sikaroudi, M., Hipp, J.D., Tizhoosh, H.: Learning to predict rna sequence ex- pressions from whole slide images with applications for search and classification. Communications Biol- ogy6(1), 304 (2023) 1, 2

    2023

  2. [2]

    Fron- tiers in Immunology15, 1499301 (2024) 1

    An, J., Lu, Y ., Chen, Y ., Chen, Y ., Zhou, Z., Chen, J., Peng, C., Huang, R., Peng, F.: Spatial transcriptomics in breast cancer: providing insight into tumor hetero- geneity and promoting individualized therapy. Fron- tiers in Immunology15, 1499301 (2024) 1

    2024

  3. [3]

    Advances in neural information pro- cessing systems35, 32897–32912 (2022) 3

    Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O.K., Aggarwal, K., Som, S., Piao, S., Wei, F.: Vlmo: Unified vision-language pre-training with mixture-of- modality-experts. Advances in neural information pro- cessing systems35, 32897–32912 (2022) 3

    2022

  4. [4]

    medRxiv pp

    Bareja, R., Carrillo-Perez, F., Zheng, Y ., Pizurica, M., Nandi, T.N., Shen, J., Madduri, R., Gevaert, O.: Evaluating vision and pathology foundation models for computational pathology: a comprehensive bench- mark study. medRxiv pp. 2025–05 (2025) 5

    2025

  5. [5]

    Clinical Cancer Research30(8), 1509–1517 (2024) 8

    Bartlett, J.M., Xu, K., Wong, J., Pond, G., Zhang, Y ., Spears, M., Salunga, R., Mallon, E., Taylor, K.J., Hasenburg, A., et al.: Validation of the prognostic per- formance of breast cancer index in hormone receptor– positive postmenopausal breast cancer patients in the team trial. Clinical Cancer Research30(8), 1509–1517 (2024) 8

    2024

  6. [6]

    In: Proceedings of the IEEE/CVF international conference on com- puter vision

    Caron, M., Touvron, H., Misra, I., J ´egou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on com- puter vision. pp. 9650–9660 (2021) 2

    2021

  7. [7]

    Cancer discovery2(5), 401–404 (2012) 5

    Cerami, E., Gao, J., Dogrusoz, U., Gross, B.E., Sumer, S.O., Aksoy, B.A., Jacobsen, A., Byrne, C.J., Heuer, M.L., Larsson, E., et al.: The cbio cancer genomics portal: an open platform for exploring multidimen- sional cancer genomics data. Cancer discovery2(5), 401–404 (2012) 5

    2012

  8. [8]

    NPJ precision oncology4(1), 14 (2020) 2

    Chen, M., Zhang, B., Topatana, W., Cao, J., Zhu, H., Juengpanich, S., Mao, Q., Yu, H., Cai, X.: Classifica- tion and mutation prediction based on histopathology h&e images in liver cancer using deep learning. NPJ precision oncology4(1), 14 (2020) 2

    2020

  9. [9]

    In: International conference on ma- chine learning

    Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on ma- chine learning. pp. 1597–1607. PmLR (2020) 2

    2020

  10. [10]

    Nature Methods pp

    Chen, W., Zhang, P., Tran, T.N., Xiao, Y ., Li, S., Shah, V .V ., Cheng, H., Brannan, K.W., Youker, K., Lai, L., et al.: A visual–omics foundation model to bridge histopathology with spatial transcriptomics. Nature Methods pp. 1–15 (2025) 1, 2, 5

    2025

  11. [11]

    Cancer research83(23), 3861–3867 (2023) 5

    De Bruijn, I., Kundra, R., Mastrogiacomo, B., Tran, T.N., Sikina, L., Mazor, T., Li, X., Ochoa, A., Zhao, G., Lai, B., et al.: Analysis and visualization of longi- tudinal genomic and clinical data from the aacr project genie biopharma collaborative in cbioportal. Cancer research83(23), 3861–3867 (2023) 5

    2023

  12. [12]

    Science signaling6(269), pl1–pl1 (2013) 5

    Gao, J., Aksoy, B.A., Dogrusoz, U., Dresdner, G., Gross, B., Sumer, S.O., Sun, Y ., Jacobsen, A., Sinha, R., Larsson, E., et al.: Integrative analysis of complex cancer genomics and clinical profiles using the cbio- portal. Science signaling6(269), pl1–pl1 (2013) 5

    2013

  13. [13]

    In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision

    Gindra, R.H., Palla, G., Nguyen, M., Wagner, S.J., Tran, M., Theis, F.J., Saur, D., Crawford, L., Peng, T.: A large-scale benchmark of cross-modal learn- ing for histology and gene expression in spatial tran- scriptomics. In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision. pp. 1182– 1192 (2025) 1, 2, 5

    2025

  14. [14]

    Ad- vances in neural information processing systems33, 21271–21284 (2020) 2

    Grill, J.B., Strub, F., Altch´e, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al.: Bootstrap your own latent-a new approach to self-supervised learning. Ad- vances in neural information processing systems33, 21271–21284 (2020) 2

    2020

  15. [15]

    Nature biomedical engineering4(8), 827–834 (2020) 1, 2

    He, B., Bergenstr ˚ahle, L., Stenbeck, L., Abid, A., Andersson, A., Borg, ˚A., Maaskola, J., Lundeberg, J., Zou, J.: Integrating spatial gene expression and breast tumour morphology via deep learning. Nature biomedical engineering4(8), 827–834 (2020) 1, 2

    2020

  16. [16]

    In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition

    He, K., Chen, X., Xie, S., Li, Y ., Doll ´ar, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition. pp. 16000–16009 (2022) 2, 6

    2022

  17. [17]

    In: Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition

    He, K., Fan, H., Wu, Y ., Xie, S., Girshick, R.: Mo- mentum contrast for unsupervised visual representa- tion learning. In: Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition. pp. 9729–9738 (2020) 2

    2020

  18. [18]

    In: International con- ference on machine learning

    Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: International con- ference on machine learning. pp. 2127–2136. PMLR (2018) 5

    2018

  19. [19]

    Neural Computing and Applications37(16), 9311– 9330 (2025) 1

    Jahan, I., Chowdhury, M.E., Vranic, S., Al Saady, R.M., Kabir, S., Pranto, Z.H., Mim, S.J., Nobi, S.F.: Deep learning and vision transformers-based frame- work for breast cancer and subtype identification. Neural Computing and Applications37(16), 9311– 9330 (2025) 1

    2025

  20. [20]

    In: Advances in Neural Infor- mation Processing Systems (Dec 2024) 2, 5, 6, 8

    Jaume, G., Doucet, P., Song, A.H., Lu, M.Y ., Almagro-Perez, C., Wagner, S.J., Vaidya, A.J., Chen, R.J., Williamson, D.F.K., Kim, A., Mahmood, F.: Hest-1k: A dataset for spatial transcriptomics and his- 10 tology image analysis. In: Advances in Neural Infor- mation Processing Systems (Dec 2024) 2, 5, 6, 8

    2024

  21. [21]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition

    Jaume, G., Oldenburg, L., Vaidya, A., Chen, R.J., Williamson, D.F., Peeters, T., Song, A.H., Mahmood, F.: Transcriptomics-guided slide representation learn- ing in computational pathology. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition. pp. 9632–9644 (2024) 2

    2024

  22. [22]

    Briefings in Bioinformatics25(1) (2023) 1, 2, 6

    Jia, Y ., Liu, J., Chen, L., Zhao, T., Wang, Y .: Thi- togene: a deep learning method for predicting spatial transcriptomics from histological images. Briefings in Bioinformatics25(1) (2023) 1, 2, 6

    2023

  23. [23]

    npj Breast Cancer (2025) 8

    Jilderda, M.F., Bartlett, J.M., Liefers, G.J., Zhang, Y ., Dunn-Davies, H., Rebattu, V ., Salunga, R., Meershoek-Klein Kranenbarg, E., de Munck, L., Hasenburg, A., et al.: Validation of minimal risk of recurrence classification by the breast cancer index in early stage breast cancer. npj Breast Cancer (2025) 8

    2025

  24. [24]

    NAR cancer6(4), zcae046 (2024) 1

    Jim ´enez-Santos, M.J., Garc ´ıa-Mart´ın, S., Rubio- Fern´andez, M., G ´omez-L´opez, G., Al-Shahrour, F.: Spatial transcriptomics in breast cancer reveals tumour microenvironment-driven drug responses and clonal therapeutic heterogeneity. NAR cancer6(4), zcae046 (2024) 1

    2024

  25. [25]

    Nature cancer1(8), 789–799 (2020) 2

    Kather, J.N., Heij, L.R., Grabsch, H.I., Loeffler, C., Echle, A., Muti, H.S., Krause, J., Niehues, J.M., Som- mer, K.A., Bankhead, P., et al.: Pan-cancer image- based detection of clinically actionable genetic alter- ations. Nature cancer1(8), 789–799 (2020) 2

    2020

  26. [26]

    ArXiv pp

    Kong, Z., Qiu, M., Boesen, J., Lin, X., Yun, S., Chen, T., Kellis, M., Zitnik, M.: Spatia: Multimodal model for prediction and generation of spatial cell pheno- types. ArXiv pp. arXiv–2507 (2025) 2

    2025

  27. [27]

    The Breast66, 183–190 (2022) 1

    Li, B., Li, F., Liu, Z., Xu, F., Ye, G., Li, W., Zhang, Y ., Zhu, T., Shao, L., Chen, C., et al.: Deep learning with biopsy whole slide images for pretreatment pre- diction of pathological complete response to neoad- juvant chemotherapy in breast cancer: A multicenter study. The Breast66, 183–190 (2022) 1

    2022

  28. [28]

    Bioin- formatics38(14), 3629–3637 (2022) 1

    Liu, H., Kurc, T.: Deep learning for survival analysis in breast cancer with whole slide image data. Bioin- formatics38(14), 3629–3637 (2022) 1

    2022

  29. [29]

    Clinical cancer re- search16(21), 5222–5232 (2010) 1

    Nielsen, T.O., Parker, J.S., Leung, S., V oduc, D., Ebbert, M., Vickery, T., Davies, S.R., Snider, J., Sti- jleman, I.J., Reed, J., et al.: A comparison of pam50 intrinsic subtyping with immunohistochemistry and clinical prognostic factors in tamoxifen-treated estro- gen receptor–positive breast cancer. Clinical cancer re- search16(21), 5222–5232 (2010) 1

    2010

  30. [30]

    medRxiv pp

    Nonchev, K., Dawo, S., Silina, K., Moch, H., An- dani, S., Consortium, T.P., Koelzer, V .H., R¨atsch, G.: Deepspot: Leveraging spatial context for enhanced spatial transcriptomics prediction from h&e images. medRxiv pp. 2025–02 (2025) 1, 2

    2025

  31. [31]

    BioRxiv pp

    Pang, M., Su, K., Li, M.: Leveraging information in spatial transcriptomics to predict super-resolution gene expression from histology images in tumors. BioRxiv pp. 2021–11 (2021) 1, 2

    2021

  32. [32]

    Commu- nications Engineering3(1), 104 (2024) 1

    Perera, R., Savas, P., Senanayake, D., Salgado, R., Joensuu, H., O’Toole, S., Li, J., Loi, S., Halgamuge, S.: Annotation-efficient deep learning for breast can- cer whole-slide image classification using tumour in- filtrating lymphocytes and slide-level labels. Commu- nications Engineering3(1), 104 (2024) 1

    2024

  33. [33]

    Cancer Research83(17), 2970–2984 (2023) 2

    Pizurica, M., Larmuseau, M., Van der Eecken, K., de Schaetzen van Brienen, L., Carrillo-Perez, F., Is- phording, S., Lumen, N., Van Dorpe, J., Ost, P., Ver- beke, S., et al.: Whole slide imaging-based predic- tion of tp53 mutations identifies an aggressive disease phenotype in prostate cancer. Cancer Research83(17), 2970–2984 (2023) 2

    2023

  34. [34]

    Na- ture Communications15(1), 9886 (2024) 1, 2

    Pizurica, M., Zheng, Y ., Carrillo-Perez, F., Noor, H., Yao, W., Wohlfart, C., Vladimirova, A., Marchal, K., Gevaert, O.: Digital profiling of gene expression from histology images with linearized attention. Na- ture Communications15(1), 9886 (2024) 1, 2

    2024

  35. [35]

    NPJ precision oncology5(1), 87 (2021) 2

    Qu, H., Zhou, M., Yan, Z., Wang, H., Rustgi, V .K., Zhang, S., Gevaert, O., Metaxas, D.N.: Genetic muta- tion and biological pathway prediction based on whole slide images in breast carcinoma using deep learning. NPJ precision oncology5(1), 87 (2021) 2

    2021

  36. [36]

    arXiv preprint arXiv:2506.21857 (2025) 1, 2

    Redekop, E., Pleasure, M., Wang, Z., Flores, K., Sisk, A., Speier, W., Arnold, C.W.: Spade: Spatial tran- scriptomics and pathology alignment using a mixture of data experts for an expressive latent space. arXiv preprint arXiv:2506.21857 (2025) 1, 2

    work page arXiv 2025

  37. [37]

    arXiv preprint arXiv:2312.02147 (2023) 2

    Ren, S., Wang, Z., Zhu, H., Xiao, J., Yuille, A., Xie, C.: Rejuvenating image-gpt as strong visual repre- sentation learners. arXiv preprint arXiv:2312.02147 (2023) 2

    work page arXiv 2023

  38. [38]

    BioRxiv p

    Schaumberg, A.J., Rubin, M.A., Fuchs, T.J.: H&e- stained whole slide image deep learning predicts spop mutation state in prostate cancer. BioRxiv p. 064279 (2016) 2

    2016

  39. [39]

    Nature communications11(1), 3877 (2020) 1, 2

    Schmauch, B., Romagnoni, A., Pronier, E., Saillard, C., Maill ´e, P., Calderaro, J., Kamoun, A., Sefta, M., Toldo, S., Zaslavskiy, M., et al.: A deep learning model to predict rna-seq expression of tumours from whole slide images. Nature communications11(1), 3877 (2020) 1, 2

    2020

  40. [40]

    Clinical cancer research26(17), 4682–4687 (2020) 1

    Sestak, I., Filipits, M., Buus, R., Rudas, M., Balic, M., Knauer, M., Kronenwett, R., Fitzal, F., Cuzick, J., Gnant, M., et al.: Prognostic value of endopre- dict in women with hormone receptor–positive, her2- 11 negative invasive lobular breast cancer. Clinical cancer research26(17), 4682–4687 (2020) 1

    2020

  41. [41]

    bioRxiv pp

    Shulman, E.D., Campagnolo, E.M., Lodha, R., Stem- mer, A., Cantore, T., Ru, B., Chang, T.G., Biswas, S., Dhruba, S.R., Wang, A., et al.: Ai-driven spa- tial transcriptomics unlocks large-scale breast cancer biomarker discovery from histopathology. bioRxiv pp. 2024–10 (2024) 1, 2, 6

    2024

  42. [42]

    Expert review of molecular diagnostics9(5), 417–422 (2009) 1

    Slodkowska, E.A., Ross, J.S.: Mammaprint™ 70- gene signature: another milestone in personalized medical care for breast cancer patients. Expert review of molecular diagnostics9(5), 417–422 (2009) 1

    2009

  43. [43]

    Communications Medicine3(1), 44 (2023) 5

    Steyaert, S., Qiu, Y .L., Zheng, Y ., Mukherjee, P., V o- gel, H., Gevaert, O.: Multimodal deep learning to predict prognosis in adult and pediatric brain tumors. Communications Medicine3(1), 44 (2023) 5

    2023

  44. [44]

    Molecu- lar diagnosis & therapy24(5), 621–632 (2020) 1

    Syed, Y .Y .: Oncotype dx breast recurrence score®: a review of its use in early-stage breast cancer. Molecu- lar diagnosis & therapy24(5), 621–632 (2020) 1

    2020

  45. [45]

    Cell genomics 1(3) (2021) 5

    Thennavan, A., Beca, F., Xia, Y ., Garcia-Recio, S., Allison, K., Collins, L.C., Tse, G.M., Chen, Y .Y ., Schnitt, S.J., Hoadley, K.A., et al.: Molecular analysis of tcga breast cancer histologic types. Cell genomics 1(3) (2021) 5

    2021

  46. [46]

    In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition

    Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O.K., Singhal, S., Som, S., et al.: Image as a foreign language: Beit pretraining for vision and vision-language tasks. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition. pp. 19175– 19186 (2023) 3

    2023

  47. [47]

    Na- ture communications15(1), 10232 (2024) 1

    Wang, X., Venet, D., Lifrange, F., Larsimont, D., Red- iti, M., Stenbeck, L., Dupont, F., Rouas, G., Gar- cia, A.J., Craciun, L., et al.: Spatial transcriptomics reveals substantial heterogeneity in triple-negative breast cancer with potential clinical implications. Na- ture communications15(1), 10232 (2024) 1

    2024

  48. [48]

    Cancer research81(19), 5115–5126 (2021) 2

    Wang, Y ., Kartasalo, K., Weitz, P., Acs, B., Valko- nen, M., Larsson, C., Ruusuvuori, P., Hartman, J., Rantalainen, M.: Predicting molecular pheno- types from histopathology images: a transcriptome- wide expression–morphology analysis in breast can- cer. Cancer research81(19), 5115–5126 (2021) 2

    2021

  49. [49]

    Nature genetics45(10), 1113– 1120 (2013) 5

    Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan- cancer analysis project. Nature genetics45(10), 1113– 1120 (2013) 5

    2013

  50. [50]

    Advances in Neural Infor- mation Processing Systems36, 70626–70637 (2023) 1, 2

    Xie, R., Pang, K., Chung, S., Perciani, C., MacPar- land, S., Wang, B., Bader, G.: Spatially resolved gene expression prediction from histology images via bi- modal contrastive learning. Advances in Neural Infor- mation Processing Systems36, 70626–70637 (2023) 1, 2

    2023

  51. [51]

    Zimmermann, E., V orontsov, E., Viret, J., Casson, A., Zelechowski, M., Shaikovski, G., Tenenholtz, N., Hall, J., Klimstra, D., Yousfi, R., et al.: Virchow2: Scaling self-supervised mixed magnification models in pathology. arXiv preprint arXiv:2408.00738 (2024) 5 12 JASPR: Joint Spatial Representation learning of histology and spatial genomics for improve...

    work page arXiv 2024

  52. [52]

    Downstream application of JASPR features and compared baseline architectures

    Supplementary Figures Figure 1. Downstream application of JASPR features and compared baseline architectures. a) JASPR. b) Baseline. c) CLIP contrastive alignment training. 1 Figure 2. Gene-level Pearson Correlation Coefficient (PCC) versus ground-truth, non-zscored standard deviation (STD), colored by Mean Squared Error (MSE). 2 Figure 3. Histogram of ge...

  53. [53]

    Overview of methods that model histology images and paired genomics data

    Supplementary Tables Table 1. Overview of methods that model histology images and paired genomics data. Existing methods Genomics technology Spatial ST context Spatial HE context Joint training Patch-based HE and bulk genomics [8, 25, 33, 35, 38, 48] Bulk DNA, bulk RNA ✗ ✗ ✗ Slide-based HE and bulk genomics [1, 34, 39] Bulk RNA✗ ✓ ✗ Patch-based HE and spa...

    work page arXiv

  54. [54]

    Supplementary Notes Suppl. Note 1: Spatial Transcriptomics prediction per- formance To evaluate ST prediction performance, we focused on mean PCC performance in the main text, as this is the stan- dard evaluation metric in the field [20]. This is largely be- cause many downstream analyses emphasize relative ex- pression patterns (e.g., differential expres...