Pith. sign in

REVIEW 2 major objections 4 minor 80 references

Self-supervision, not clinical labels, is what aligns medical AI encoders.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:16 UTC pith:PGALTQUF

load-bearing objection Strong empirical study of medical encoder convergence, but the abstract overstates the causal and scaling claims given one-seed controlled cells and a nonconverged consensus. the 2 major comments →

arxiv 2607.20274 v1 pith:PGALTQUF submitted 2026-07-22 cs.CV cs.AIcs.CLcs.LG

Self-supervision drives representational convergence in medical foundation models more than clinical supervision

Soroosh Tayebi Arasteh , Sebastian Ziegelmayer , Mahshad Lotfinia , Lisa Adams , Sven Nebelung , Jakob Nikolas Kather , Daniel Truhn This is my paper
classification cs.CV cs.AIcs.CLcs.LG
keywords representational convergenceself-supervised learningmedical foundation modelsrepresentation alignmentmodel interoperabilitychest radiographytransfer learningvision-language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether independently trained medical image encoders converge toward a shared representation, and what causes it. It argues that the training objective, specifically self-supervised learning, is the driver, while clinical-label supervision and model scale are not. The evidence comes from a controlled experiment that varies only the objective, a synthetic generative model, and a broad panel of encoders across five imaging modalities. If right, interoperability between medical AI models is something to design for through self-supervised pretraining, not a gift of scale or supervision.

Core claim

The paper claims that representational convergence among medical foundation models is real but modest, and that it is governed by the shared self-supervised pretraining objective rather than by clinical supervision. Matched self-supervised encoders align far more (40.4% on chest radiography) than label-supervised (21.1%) or image-text (3.3%) pairs, and the same ordering holds in a synthetic generative model and does not close as labels become more informative. Convergence does not grow with model size (Spearman 0.302, p=0.223) and does not extend to image-text alignment. Despite the weak geometry, a linear classifier transfers across encoders and to five held-out hospitals retaining about 85

What carries the argument

The key objects are the alignment metric centered-kernel nearest-neighbor alignment (CKNNA, agreement of k-nearest-neighbor graphs after centering) and the controlled-training design that fixes data, architecture, and scale while varying only the training objective. A consensus geometry built by generalized Procrustes analysis and an anchor-based common coordinate system (relative representations) carry the transfer and stitching analyses.

Load-bearing premise

The causal ordering rests on a controlled training experiment with a single initialization and one seed per cell, so the reported alignment differences could in principle be reversed by training-seed variability, which the bootstrap intervals do not capture.

What would settle it

Re-run the controlled-objective matrix with ten seeds per cell; if the between-seed spread of same-objective alignment overlaps across objectives, the claim that the objective governs convergence collapses. Also, train self-supervised encoders on non-overlapping data from different institutions; if alignment drops to the random floor, shared data rather than the objective may be the driver.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Convergence in medical imaging is set by the pretraining objective; adding clinical-label or report supervision moves an encoder away from its self-supervised peers.
  • Bigger and better encoders do not converge more, so interoperability cannot be bought by scaling.
  • A modest shared geometry is enough for functional interchangeability: linear classifiers transfer across encoders and to external hospitals, and affine stitching recovers oracle performance.
  • Convergence is within-modality: image and text encoders do not align beyond a random floor, so cross-modal convergence should not be assumed.
  • The shared representation is uneven across patient subgroups and least reliable for under-represented groups, so subgroup-specific validation is needed before it is trusted.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the objective, not scale, drives convergence, pretraining recipe choices (masked reconstruction versus contrastive) may be a more direct lever for interoperability than data volume; a testable extension is to measure convergence as a function of objective family in the same controlled setup.
  • The failure of alignment to match radiologist similarity suggests that case-level clinical similarity may not live in the same subspace as diagnostic co-occurrence; a finding-conditioned triplet task could separate ill-posedness from representational limits.
  • The near-chance image-text alignment implies that vision-language medical models may align with language only through fine-tuning on paired data, not from shared pretraining geometry; this could be tested by measuring alignment before and after vision-language instruction tuning.
  • The subgroup instability may reflect neighbor-count artifacts in sparsely populated regions; a density-matched evaluation could distinguish representation fracture from estimation noise.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies representational convergence across 18 open-weight medical image encoders and 7 biomedical text encoders, using pairwise neighbor-based alignment metrics (mKNN, CKNNA) on shared pools spanning five imaging modalities, with chest radiography as the primary substrate (650,982 images). It makes three main claims: (1) convergence is real but modest and is driven by the training objective rather than clinical supervision, scale, or downstream capability; (2) the shared geometry is clinically structured (it tracks disease co-occurrence but not ICD-10 taxonomy) and is uneven across patient subgroups; and (3) despite weak geometric alignment, a linear classifier transferred through relative representations retains about 85% of within-encoder performance across encoders and generalizes across five external sites, while feature stitching recovers oracle performance. The causal claim is supported by a controlled experiment that fixes data, architecture, scale, and initialization while varying only the objective, and by a synthetic generative model with five seeds. The authors report several negative results, including failure of cross-modal image-text convergence, failure of a consensus-deviation drift detector, and chance-level prediction of radiologist similarity triplets.

Significance. If the central claim holds, this is an important and largely novel result for medical foundation models: it inverts the common assumption that clinical-label or report supervision concentrates representations, and it argues that interoperability should be designed through the pretraining objective and validated in a common coordinate system rather than assumed from scale. The study is unusually broad in encoder coverage and evaluation substrates, and it has several methodological strengths: the controlled-objective training matrix, the synthetic generative model with label-informativeness sweep, the use of relative representations for transfer and stitching, the explicit random-initialization floor, and the unusually candid limitation statements. The transfer and stitching results are strong and internally consistent. The main risk is that the central causal ordering rests on a single training seed per controlled cell, and several secondary results rest on a consensus configuration that did not converge; both are acknowledged in the text but are load-bearing for some of the headline claims.

major comments (2)
  1. [Results, first subsection; Methods 'Controlled-objective training'; Limitation 5] The central causal claim — that the self-supervised objective drives convergence more than clinical supervision — rests on the controlled-objective experiment, where each cell uses one initialization and one seed, and the reported bootstrap intervals resample evaluation cases, not training seeds. The headline chest-radiography gap is 40.4% (SSL) versus 21.1% (supervised), a 19.3-point difference, but each value is a single ViT-S/ViT-B pair. Training a small ViT for 50 epochs from the same initialization can plausibly produce seed-level alignment swings of several points; if that swing is comparable to the gap, the ordering and the 'inverting the expectation' conclusion are not established. The synthetic model uses five seeds, but it is a two-layer MLP on synthetic data and does not quantify real-data seed variance. I would ask the authors to run at least 3–5 seeds for the key chest-radio
  2. [Methods 'Consensus geometry and residual alignment'; Supplementary Note 1; Fig. 5a–e] The Generalized Procrustes consensus did not reach its convergence tolerance, as stated in the Methods and Supplementary Note 1. Two results that are used to support secondary but still prominent claims depend on this consensus: the clinical-structure recovery (comorbidity correlation 0.615, p<0.001; ICD-10 0.135, p=0.211) and the scaling analysis (residual distance to consensus versus size, AUROC, and release year; Spearman 0.302, p=0.223; 0.131, p=0.604; 0.206, p=0.412). A nonconverged fit can bias a downstream statistic in either direction, so the 'does not grow with size' conclusion and the co-occurrence-versus-taxonomy contrast are sensitivity-limited. The paper says this, but the results are still presented as findings rather than as exploratory. I recommend either running the Procrustes iteration to convergence (e.g., centering/scaling relative representations, increasing iteratio
minor comments (4)
  1. [Abstract and Results, first subsection] The abstract states 'Convergence is modest but above a random floor' and then gives 40.4% on chest radiography. This floor comparison is formal only for chest radiography; for the other four modalities the floor was not computed on the matched pool, and the text later says the comparison is descriptive. Please state this qualification where the pooled claim is first made.
  2. [Fig. 2 and Supplementary Table 1] The caption to Fig. 2 says error bars in panels d,e are ±SD, but these SDs come from bootstrapping evaluation cases, not training seeds. To avoid misleading readers, label the error bars as 'case-bootstrap SD' or add a note that no seed variability is included.
  3. [Methods 'Statistical analysis'; Results 'Uniformity across subgroups'] The demographic omnibus tests treat encoder pairs as independent although pairs sharing an encoder are correlated, and the text correctly flags the p-values as anticonservative. This is an honest caveat, but the Results section still reports p<0.001 for race/ethnicity/insurance without the caveat in the same paragraph. Please move or repeat the caveat at the point of reporting.
  4. [Fig. 8 and reader study methods] The 'automated diagnostic read' in Fig. 8b is not defined in the main text or caption — which encoder and which classifier produce this read? Please specify, or refer to the Methods/Supplementary table where this is defined.

Circularity Check

0 steps flagged

No material circularity: the core convergence claim is measured with externally defined metrics and controlled experiments; self-citations are contextual only.

full rationale

The paper's alignment measures (mKNN and CKNNA, Eqs. 1-2), relative representations (Eq. 3), and transfer/stitching protocols are defined from external methodology and are not fitted to the target conclusion. The controlled-objective experiment varies the training objective under fixed data, architecture, and budget, and reports an observed ordering (e.g., chest-radiography matched-objective CKNNA: SSL 40.4 vs supervised 21.1 vs image-text 3.3); this ordering is an outcome, not an input. The synthetic model fixes generative parameters (Eqs. 6-7) and varies label informativeness α as a controlled variable; the SSL-vs-supervised gap is not encoded in the model specification. Self-citations (refs 14-18, 46-47) appear in the introduction and discussion as contextual examples and do not supply the measurement or the causal inference; the independent corroboration in ref. 44 is external. The paper itself honestly flags the single-seed limitation (Discussion, 'Fifth, the controlled encoders use a single seed per cell') and the nonconverged consensus ('First, the consensus configuration did not reach its convergence tolerance'); these are robustness/validity concerns rather than reductions of the result to its inputs. No specific step can be quoted in which a 'prediction' equals a fitted parameter or in which a self-citation chain forces the conclusion, so no circular step is identified.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

No new physical or architectural entities are introduced. The paper's load-bearing assumptions are domain choices: how convergence is measured, that native preprocessing is part of each encoder, that relative representations give a common coordinate system, and that one seed per controlled cell is sufficient to attribute causality. The synthetic generative model is an ad hoc construction and its stated proposition conflicts with the observed result.

free parameters (5)
  • k (neighbor count) = 10
    Headline neighbor count for mKNN/CKNNA; sensitivity checked at k=5,20,50, but headline values and the central ordering are reported at k=10.
  • Anchor count for relative representations = 1024
    Default anchor set for relative representations; verified at 256 and 4096, but 1024 is the default used for transfer and stitching.
  • Synthetic model dimensions and noise = D=160, d_s=32, d_n=128, σ=0.1
    Chosen to create a low-dimensional signal subspace and higher-dimensional nuisance subspace; no external calibration and not swept.
  • Controlled training budget = 50 epochs, lr 1e-4, batch 512, wd 0.05, masking 75%
    Held fixed across objectives so no objective is advantaged by training length; still a hand-set configuration that could interact with objective difficulty.
  • Synthetic label informativeness grid = α ∈ {0.1, 0.25, 0.5, 0.75, 1.0}, C=10
    Swept to test whether the objective gap closes as labels become more informative; the grid and the label-generation rule are modeling choices.
axioms (6)
  • domain assumption k-NN graph overlap at k=10 captures the clinically relevant notion of representational convergence
    Alignment is defined through neighbor overlap; a different definition of similarity could order encoder pairs differently, although mKNN and CKNNA agree closely (r=0.99).
  • domain assumption Native preprocessing is part of each encoder and is confounded with encoder identity
    The authors explicitly state that measured alignment reflects the representation plus its native preprocessing, not the representation in isolation. This weakens causal attribution of convergence to the objective alone.
  • domain assumption The CXR random-initialization floor approximates chance alignment for other pools
    Pool-specific floors were not computed for histopathology, fundus, dermoscopy, or mammography; the floor comparison is explicitly descriptive there.
  • domain assumption Relative representations with 1,024 anchors provide a valid common coordinate system for transfer and stitching
    Borrowed from Moschella et al. [33]; the paper does not re-derive its validity in this setting, though it verifies anchor counts 256 and 4096.
  • ad hoc to paper Synthetic generative model's signal/nuisance decomposition and label-generation rule
    Purpose-built for this study; the stated analytical proposition predicts supervised alignment grows with label informativeness, but the observed supervised-minus-self-supervised gap is negative at every α, so the model's mechanistic framing is internally inconsistent.
  • domain assumption One training seed per controlled cell is representative of the objective's effect
    The causal ordering of objectives rests on a single seed per cell; uncertainty intervals cover evaluation-case variability only, not training-seed variability.

pith-pipeline@v1.3.0-alltime-deepseek · 48322 in / 11765 out tokens · 108291 ms · 2026-08-01T10:16:42.214960+00:00 · methodology

0 comments
read the original abstract

Medical image encoders from different groups are increasingly treated as interchangeable, on the assumption that scale and clinical supervision concentrate their representations onto a shared structure. Whether this convergence is real, what produces it, and whether it is clinically usable are untested, and the similarity measures behind such claims are fragile. We present a controlled dissection across 18 image and 7 text encoders, all open-weight and run locally, spanning 7M to 27B parameters and five imaging modalities, including 650,982 chest radiographs from six datasets. To isolate cause, we train encoders that vary only the objective under fixed data, architecture, and scale, and reproduce the effect in a synthetic model. Convergence is modest but above a random floor, driven by the self-supervised objective, not clinical supervision: matched self-supervised encoders aligned most (40.4% on chest radiography), with label-supervised (21.1%) and image-text (3.3%) far lower, and did not grow with size (Spearman 0.302, p=0.223) or capability. It is within-modality, does not reach clinical language, and does not reproduce how radiologists judge case similarity. Yet a linear classifier transfers across encoders and to five held-out hospitals, retaining about 85% of within-encoder performance. Convergence in medical imaging is therefore set by the pretraining objective, not inherited from scale or clinical supervision. Interoperability is accordingly something to design for through that objective, and to validate where the shared geometry is weakest, across patient subgroups and against clinical judgment.

Figures

Figures reproduced from arXiv: 2607.20274 by Daniel Truhn, Jakob Nikolas Kather, Lisa Adams, Mahshad Lotfinia, Sebastian Ziegelmayer, Soroosh Tayebi Arasteh, Sven Nebelung.

Figure 1
Figure 1. Figure 1: Study overview: motivation and design. a, Medical image encoders are increasingly reused as if interchangeable, which presumes that their representations converge to a shared geometry; the study asks whether this convergence is real, what produces it, and whether it is clinically us￾able. b, The frozen encoder panel of 18 image and 7 biomedical text encoders, spanning 7 million to 27 billion parameters and… view at source ↗
Figure 2
Figure 2. Figure 2: Representational alignment among 12 image encoders trained under a controlled matrix of [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Representational alignment among 18 frozen image encoders within each of five image pools, [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Alignment of same-objective encoder pairs in a synthetic generative model as a function of [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Structure of the consensus representation over the 18 chest radiograph image encoders, [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Transfer of the shared representation across encoders and sites and feature stitching, chest [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Image-to-text representational alignment between 18 image encoders and 7 biomedical text [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Expert-reader validation of the shared geometry, chest radiography. Two board-certified ra [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

80 extracted references · 2 canonical work pages

  1. [1]

    Nature Machine Intelligence7(1), 119–130 (2025) 33

    Pérez-García, F., Sharma, H., Bond-Taylor, S., Bouzid, K., Salvatelli, V., Ilse, M., Bannur, S., Castro, D.C., Schwaighofer, A., Lungren, M.P., et al.: Exploring scalable medical image encoders beyond text supervision. Nature Machine Intelligence7(1), 119–130 (2025) 33

    2025

  2. [2]

    Nature biomedical engineering 6(12), 1399–1406 (2022)

    Tiu, E., Talius, E., Patel, P., Langlotz, C.P., Ng, A.Y., Rajpurkar, P.: Expert-level detection of patholo- gies from unannotated chest x-ray images via self-supervised learning. Nature biomedical engineering 6(12), 1399–1406 (2022)

    2022

  3. [3]

    Nejm Ai2(1), AIoa2400640 (2025)

    Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., et al.: A multimodal biomedical foundation model trained from fifteen million image–text pairs. Nejm Ai2(1), AIoa2400640 (2025)

    2025

  4. [4]

    Nature medicine30(3), 850–862 (2024)

    Chen, R.J., Ding, T., Lu, M.Y., Williamson, D.F., Jaume, G., Song, A.H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al.: Towards a general-purpose foundation model for computational pathology. Nature medicine30(3), 850–862 (2024)

    2024

  5. [5]

    Nature medicine30(10), 2924–2935 (2024)

    Vorontsov, E., Bozkurt, A., Casson, A., Shaikovski, G., Zelechowski, M., Severson, K., Zimmermann, E., Hall, J., Tenenholtz, N., Fusi, N., et al.: A foundation model for clinical-grade computational pathology and rare cancers detection. Nature medicine30(10), 2924–2935 (2024)

    2024

  6. [6]

    Nature 630(8015), 181–188 (2024)

    Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann, T., Wong, C., Gero, Z., González, J., Gu, Y., et al.: A whole-slide foundation model for digital pathology from real-world data. Nature 630(8015), 181–188 (2024)

    2024

  7. [7]

    Nature medicine 30(3), 863–874 (2024)

    Lu, M.Y., Chen, B., Williamson, D.F., Chen, R.J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L.P., Gerber, G., et al.: A visual-language foundation model for computational pathology. Nature medicine 30(3), 863–874 (2024)

    2024

  8. [8]

    Filiot, A., Jacob, P., Kain, A.M., Saillard, C.: Phikon-v2, a large and public feature extractor for biomarker prediction (2024),https://arxiv.org/abs/2409.09173

    Pith/arXiv arXiv 2024

  9. [9]

    Nature622(7981), 156–163 (2023)

    Zhou, Y., Chia, M.A., Wagner, S.K., Ayhan, M.S., Williamson, D.J., Struyven, R.R., Liu, T., Xu, M., Lozano, M.G., Woodward-Court, P., et al.: A foundation model for generalizable disease detection from retinal images. Nature622(7981), 156–163 (2023)

    2023

  10. [10]

    Nature machine intelligence6(3), 354–367 (2024)

    Pai, S., Bontempi, D., Hadzic, I., Prudente, V., Sokač, M., Chaunzwa, T.L., Bernatz, S., Hosny, A., Mak, R.H., Birkbak, N.J., et al.: Foundation model for cancer imaging biomarkers. Nature machine intelligence6(3), 354–367 (2024)

    2024

  11. [11]

    In: Proceedings of the 37th International Conference on Neural Information Processing Systems (2023)

    Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., Gao, J.: Llava- med: training a large language-and-vision assistant for biomedicine in one day. In: Proceedings of the 37th International Conference on Neural Information Processing Systems (2023)

    2023

  12. [12]

    In: Machine learning for healthcare conference

    Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive learning of medical visual representations from paired images and text. In: Machine learning for healthcare conference. pp. 2–25. PMLR (2022)

    2022

  13. [13]

    Nature616(7956), 259–265 (2023)

    Moor, M., Banerjee, O., Abad, Z.S.H., Krumholz, H.M., Leskovec, J., Topol, E.J., Rajpurkar, P.: Foundation models for generalist medical artificial intelligence. Nature616(7956), 259–265 (2023)

    2023

  14. [14]

    Radiology: Artificial Intelligence7(4), e240476 (2025)

    Tayebi Arasteh, S., Lotfinia, M., Bressem, K., Siepmann, R., Adams, L., Ferber, D., Kuhl, C., Kather, J.N., Nebelung, S., Truhn, D.: Radiorag: online retrieval–augmented generation for radiology question answering. Radiology: Artificial Intelligence7(4), e240476 (2025)

    2025

  15. [15]

    npj Digital Medicine8, 790 (2025)

    Wind, S., Sopa, J., Truhn, D., Lotfinia, M., Nguyen, T.T., Bressem, K., Adams, L., Rusu, M., Köstler, H., Wellein, G., et al.: Multi-step retrieval and reasoning improves radiology question answering with large language models. npj Digital Medicine8, 790 (2025)

    2025

  16. [16]

    Arasteh, S.T., Joodaki, M., Lotfinia, M., Nebelung, S., Truhn, D.: Case-grounded evidence verification: A framework for constructing evidence-sensitive supervision (2026),https://arxiv.org/abs/2604.0 9537

    2026

  17. [17]

    Radiology: Artificial Intelligence8(3), e260346 (2026)

    Tayebi Arasteh, S., Truhn, D.: When framing shapes the answer: Cognitive bias and large language model reliability in radiology. Radiology: Artificial Intelligence8(3), e260346 (2026)

    2026

  18. [18]

    Nature Communications 15(1), 1603 (2024)

    Tayebi Arasteh, S., Han, T., Lotfinia, M., Kuhl, C., Kather, J.N., Truhn, D., Nebelung, S.: Large language models streamline automated machine learning for clinical studies. Nature Communications 15(1), 1603 (2024)

    2024

  19. [19]

    In: ICLR 2021 (2021),https://openreview.net/forum ?id=YicbFdNTTy

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR 2021 (2021),https://openreview.net/forum ?id=YicbFdNTTy

    2021

  20. [20]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16000–16009 (2022) 34

    2022

  21. [21]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    Caron,M.,Touvron,H.,Misra,I.,Jégou,H.,Mairal,J.,Bojanowski,P.,Joulin,A.:Emergingproperties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 9650–9660 (October 2021)

    2021

  22. [22]

    Transactions on Machine Learning Research (2024),https://openreview.net/forum?id=a68SUt6zFt, featured Certification

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZ- IZA, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.Y., Li, S.W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: DINOv2: Learning robust visual fe...

    2024

  23. [23]

    In: Proceedings of the 37th International Conference on Machine Learning (2020)

    Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: Proceedings of the 37th International Conference on Machine Learning (2020)

    2020

  24. [24]

    In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representa- tion learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9726–9735 (2020).https://doi.org/10.1109/CVPR42600.2020.00975

    arXiv 2020

  25. [25]

    In: Proceedings of the 34th International Conference on Neural Information Processing Systems (2020)

    Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P.H., Buchatskaya, E., Doersch, C., Pires, B.A., Guo, Z.D., Azar, M.G., Piot, B., Kavukcuoglu, K., Munos, R., Valko, M.: Bootstrap your own latent a new approach to self-supervised learning. In: Proceedings of the 34th International Conference on Neural Information Processing Systems (2020)

    2020

  26. [26]

    In: Inter- national conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: Inter- national conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  27. [27]

    Huh, M., Cheung, B., Wang, T., Isola, P.: The platonic representation hypothesis (2024),https: //arxiv.org/abs/2405.07987

    Pith/arXiv arXiv 2024

  28. [28]

    In: Proceedings of the 31st International Conference on Neural Information Processing Systems

    Raghu, M., Gilmer, J., Yosinski, J., Sohl-Dickstein, J.: Svcca: singular vector canonical correlation analysis for deep learning dynamics and interpretability. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. p. 6078–6087. NIPS’17, Curran Associates Inc. (2017)

    2017

  29. [29]

    In: Proceedings of the 32nd International Conference on Neural Information Processing Systems

    Morcos, A.S., Raghu, M., Bengio, S.: Insights on representational similarity in neural networks with canonical correlation. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. p. 5732–5741. NIPS’18, Curran Associates Inc. (2018)

    2018

  30. [30]

    In: International conference on machine learning

    Kornblith, S., Norouzi, M., Lee, H., Hinton, G.: Similarity of neural network representations revisited. In: International conference on machine learning. pp. 3519–3529. PMlR (2019)

    2019

  31. [31]

    Lenc, K., Vedaldi, A.: Understanding image representations by measuring their equivariance and equiv- alence. Int. J. Comput. Vision127(5), 456–476 (2019).https://doi.org/10.1007/s11263-018-109 8-y

    work page doi:10.1007/s11263-018-109 2019

  32. [32]

    In: Proceedings of the 35th International Conference on Neural Information Processing Systems (2021)

    Bansal, Y., Nakkiran, P., Barak, B.: Revisiting model stitching to compare neural representations. In: Proceedings of the 35th International Conference on Neural Information Processing Systems (2021)

    2021

  33. [33]

    In: ICLR 2023 (2023),https://openreview.net/forum ?id=SrC-nwieGJ

    Moschella, L., Maiorca, V., Fumero, M., Norelli, A., Locatello, F., Rodolà, E.: Relative representations enable zero-shot latent space communication. In: ICLR 2023 (2023),https://openreview.net/forum ?id=SrC-nwieGJ

    2023

  34. [34]

    Koepke, A.S., Zverev, D., Ginosar, S., Efros, A.A.: Back into plato’s cave: Examining cross-modal representational convergence at scale (2026),https://arxiv.org/abs/2604.18572

    Pith/arXiv arXiv 2026

  35. [35]

    Nature medicine27(12), 2176–2182 (2021)

    Seyyed-Kalantari, L., Zhang, H., McDermott, M.B., Chen, I.Y., Ghassemi, M.: Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature medicine27(12), 2176–2182 (2021)

    2021

  36. [36]

    The Lancet Digital Health4(6), e406–e414 (2022)

    Gichoya,J.W.,Banerjee,I.,Bhimireddy,A.R.,Burns,J.L.,Celi,L.A.,Chen,L.C.,Correa,R.,Dullerud, N., Ghassemi, M., Huang, S.C., et al.: Ai recognition of patient race in medical imaging: a modelling study. The Lancet Digital Health4(6), e406–e414 (2022)

    2022

  37. [37]

    Proceedings of the National Academy of Sciences117(23), 12592–12594 (2020)

    Larrazabal, A.J., Nieto, N., Peterson, V., Milone, D.H., Ferrante, E.: Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proceedings of the National Academy of Sciences117(23), 12592–12594 (2020)

    2020

  38. [38]

    Science366(6464), 447–453 (2019) 35

    Obermeyer, Z., Powers, B., Vogeli, C., Mullainathan, S.: Dissecting racial bias in an algorithm used to manage the health of populations. Science366(6464), 447–453 (2019) 35

    2019

  39. [39]

    EBioMedicine89(2023)

    Glocker, B., Jones, C., Bernhardt, M., Winzeck, S.: Algorithmic encoding of protected characteristics in chest x-ray disease detection models. EBioMedicine89(2023)

    2023

  40. [40]

    New England Journal of Medicine385(3), 283–286 (2021)

    Finlayson, S.G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I.S., Saria, S.: The clinician and dataset shift in artificial intelligence. New England Journal of Medicine385(3), 283–286 (2021)

    2021

  41. [41]

    In: Breakthroughs in statistics: Method- ology and distribution, pp

    Efron, B.: Bootstrap methods: another look at the jackknife. In: Breakthroughs in statistics: Method- ology and distribution, pp. 569–593. Springer (1992)

    1992

  42. [42]

    Journal of the Royal statistical society: series B (Methodological)57(1), 289–300 (1995)

    Benjamini, Y., Hochberg, Y.: Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological)57(1), 289–300 (1995)

    1995

  43. [43]

    Cancer Research 27, 209–220 (1967)

    Mantel, N.: The detection of disease clustering and a generalized regression approach. Cancer Research 27, 209–220 (1967)

    1967

  44. [44]

    Ciernik, L., Linhardt, L., Morik, M., Dippel, J., Kornblith, S., Muttenthaler, L.: Objective drives the consistency of representational similarity across datasets (2025),https://arxiv.org/abs/2411.05561

    Pith/arXiv arXiv 2025

  45. [45]

    Research Square pp

    Aerts, H., Pai, S., Hadzic, I., Fedorov, A., Mak, R.: Foundation model embeddings for quantitative tumor imaging biomarkers. Research Square pp. rs–3 (2025)

    2025

  46. [46]

    Lotfinia, M., Ziegelmayer, S., Adams, L., Truhn, D., Maier, A., Arasteh, S.T.: Vision-language models for chest radiography do not always need the image (2026),https://arxiv.org/abs/2606.17710

    Pith/arXiv arXiv 2026

  47. [47]

    Wind, S., Nguyen, T.T., Sopa, J., Lotfinia, M., Bickelhaup, S., Uder, M., Köstler, H., Wellein, G., Nebelung, S., Truhn, D., Maier, A., Arasteh, S.T.: Safety and accuracy follow different scaling laws in clinical large language models (2026),https://arxiv.org/abs/2605.04039

    Pith/arXiv arXiv 2026

  48. [48]

    Advances in Neural Information Processing Systems38, 45963–45987 (2026)

    Jha, R., Zhang, C., Shmatikov, V., Morris, J.: Harnessing the universal geometry of embeddings. Advances in Neural Information Processing Systems38, 45963–45987 (2026)

    2026

  49. [49]

    Advances in Neural Information Processing Systems37, 52545–52567 (2024)

    Diao, H., Cui, Y., Li, X., Wang, Y., Lu, H., Wang, X.: Unveiling encoder-free vision-language models. Advances in Neural Information Processing Systems37, 52545–52567 (2024)

    2024

  50. [50]

    In: ProceedingsoftheIEEE/CVFconferenceoncomputervisionandpatternrecognition.pp.24960–24971 (2025)

    Luo, G., Yang, X., Dou, W., Wang, Z., Liu, J., Dai, J., Qiao, Y., Zhu, X.: Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training. In: ProceedingsoftheIEEE/CVFconferenceoncomputervisionandpatternrecognition.pp.24960–24971 (2025)

    2025

  51. [51]

    Team, G., Abd, S.E., Aggarwal, V., Algayres, R., et al.: Gemma 4 technical report (2026),https: //arxiv.org/abs/2607.02770

    Pith/arXiv arXiv 2026

  52. [52]

    Scientific data6(1), 317 (2019)

    Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: Mimic-cxr, a de-identified publicly available database of chest radiographs with free- text reports. Scientific data6(1), 317 (2019)

    2019

  53. [53]

    In: Proceedings of the AAAI conference on artificial intelligence

    Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33, pp. 590–597 (2019)

    2019

  54. [54]

    In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases . In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3462–3471. IEEE Computer Society (Jul 2017).https://doi.org/10.1109/...

    doi:10.1109/cvpr.2017.369 2017

  55. [55]

    Medical image analysis66, 101797 (2020)

    Bustos, A., Pertusa, A., Salinas, J.M., De La Iglesia-Vaya, M.: Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical image analysis66, 101797 (2020)

    2020

  56. [56]

    Scientific Data9(1), 429 (2022)

    Nguyen, H.Q., Lam, K., Le, L.T., Pham, H.H., Tran, D.Q., Nguyen, D.B., Le, D.D., Pham, C.M., Tong, H.T., Dinh, D.H., et al.: Vindr-cxr: An open dataset of chest x-rays with radiologist’s annotations. Scientific Data9(1), 429 (2022)

    2022

  57. [57]

    PhysioNet (version 1.0

    Pham, H.H., Tran, T.T., Nguyen, H.Q.: Vindr-pcxr: An open, large-scale pediatric chest x-ray dataset for interpretation of common thoracic diseases. PhysioNet (version 1.0. 0)10(2) (2022)

    2022

  58. [58]

    Zenodo (2018)

    Kather, J.N., Halama, N., Marx, A.: 100,000 histological images of human colorectal cancer and healthy tissue. Zenodo (2018)

    2018

  59. [59]

    In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II

    Veeling, B.S., Linmans, J., Winkens, J., Cohen, T., Welling, M.: Rotation equivariant cnns for digital pathology. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II. p. 210–218. Springer-Verlag (2018).https://doi.org/10.1007/978-3-030-0093...

    work page doi:10.1007/978-3-030-00934-2_24 2018

  60. [60]

    Kaggle (2019),https://www.kaggle .com/c/aptos2019-blindness-detection

    Karthik, M., Sohier, D.: Aptos 2019 blindness detection dataset. Kaggle (2019),https://www.kaggle .com/c/aptos2019-blindness-detection

    2019

  61. [61]

    Image Analysis & Stereology pp

    Decencière, E., Zhang, X., Cazuguel, G., Lay, B., Cochener, B., Trone, C., Gain, P., Ordóñez-Varela, J.R., Massin, P., Erginay, A., et al.: Feedback on a publicly distributed image database: the messidor database. Image Analysis & Stereology pp. 231–234 (2014)

    2014

  62. [62]

    Scientific data5(1), 180161 (2018)

    Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data5(1), 180161 (2018)

    2018

  63. [63]

    Hernández-Pérez, C., Combalia, M., Podlipnik, S., Codella, N.C., Rotemberg, V., Halpern, A.C., Re- iter,O.,Carrera,C.,Barreiro,A.,Helba,B.,etal.:Bcn20000:Dermoscopiclesionsinthewild.Scientific data11(1), 641 (2024)

    2024

  64. [64]

    In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018)

    Codella, N.C.F., Gutman, D., Celebi, M.E., Helba, B., Marchetti, M.A., Dusza, S.W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., Halpern, A.: Skin lesion analysis toward melanoma detection: A challengeatthe2017internationalsymposiumonbiomedicalimaging(isbi),hostedbytheinternational skin imaging collaboration (isic). In: 2018 IEEE 15th International ...

    arXiv 2018

  65. [65]

    Scientific Data10(1), 277 (2023)

    Nguyen, H.T., Nguyen, H.Q., Pham, H.H., Lam, K., Le, L.T., Dao, M., Vu, V.: Vindr-mammo: A large- scale benchmark dataset for computer-aided diagnosis in full-field digital mammography. Scientific Data10(1), 277 (2023)

    2023

  66. [66]

    Chambon, P., Delbrouck, J.B., Sounack, T., Huang, S.C., Chen, Z., Varma, M., Truong, S.Q., Chuong, C.T., Langlotz, C.P.: Chexpert plus: Augmenting a large chest x-ray dataset with text radiology reports, patient demographics and additional image formats (2024),https://arxiv.org/abs/2405.1 9538

    2024

  67. [67]

    In: Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2023), https://openreview.net/forum?id=OL2JQoO0kq

    Ikezogwo, W.O., Seyfioglu, M.S., Ghezloo, F., Geva, D.S.C., Mohammed, F.S., Anand, P.K., Kr- ishna, R., Shapiro, L.: Quilt-1m: One million image-text pairs for histopathology. In: Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2023), https://openreview.net/forum?id=OL2JQoO0kq

    2023

  68. [68]

    Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., Massa, F., Haziza, D., Wehrstedt, L., Wang, J., Darcet, T., Moutakanni, T., Sentana, L., Roberts, C., Vedaldi, A., Tolan, J., Brandt, J., Couprie, C., Mairal, J., Jégou, H., Labatut, P., Bojanowski, P.: Dinov3 (2025),https://ar...

    Pith/arXiv arXiv 2025

  69. [69]

    Tschannen, M., Gritsenko, A., Wang, X., Naeem, M.F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., Hénaff, O., Harmsen, J., Steiner, A., Zhai, X.: Siglip 2: Multilin- gual vision-language encoders with improved semantic understanding, localization, and dense features (2025),https://arxiv.org/abs/2502.14786

    Pith/arXiv arXiv 2025

  70. [70]

    In: International Conference on Medical Imaging with Deep Learning

    Cohen, J.P., Viviano, J.D., Bertin, P., Morrison, P., Torabian, P., Guarrera, M., Lungren, M.P., Chaud- hari, A., Brooks, R., Hashir, M., et al.: Torchxrayvision: A library of chest x-ray datasets and models. In: International Conference on Medical Imaging with Deep Learning. pp. 231–249. PMLR (2022)

    2022

  71. [71]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4700–4708 (2017)

    2017

  72. [72]

    Zimmermann, E., Vorontsov, E., Viret, J., Casson, A., Zelechowski, M., Shaikovski, G., Tenenholtz, N., Hall, J., Klimstra, D., Yousfi, R., Fuchs, T., Fusi, N., Liu, S., Severson, K.: Virchow2: Scaling self- supervised mixed magnification models in pathology (2024),https://arxiv.org/abs/2408.00738

    Pith/arXiv arXiv 2024

  73. [73]

    Sellergren, A., Kazemzadeh, S., Jaroensri, T., et al.: Medgemma technical report (2026),https: //arxiv.org/abs/2507.05201

    Pith/arXiv arXiv 2026

  74. [74]

    Transactions on Machine Learning Research (2025), https://openreview.net/forum?id=zKv8qULV6n

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., Li, C.: LLaVA-onevision: Easy visual task transfer. Transactions on Machine Learning Research (2025), https://openreview.net/forum?id=zKv8qULV6n

    2025

  75. [75]

    ACM Trans

    Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., Poon, H.: Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Healthcare3(1) (2021).https://doi.org/10.1145/3458754

    doi:10.1145/3458754 2021

  76. [76]

    Bioinformatics39(11), btad651 (2023) 37

    Jin, Q., Kim, W., Chen, Q., Comeau, D.C., Yeganova, L., Wilbur, W.J., Lu, Z.: Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformatics39(11), btad651 (2023) 37

    2023

  77. [77]

    In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

    Liu, F., Shareghi, E., Meng, Z., Basaldella, M., Collier, N.: Self-alignment pretraining for biomedical entity representations. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 4228–4238 (Jun 2021). https://doi.org/10.18653/v1/2021.naacl-main.334

    doi:10.18653/v1/2021.naacl-main.334 2021

  78. [78]

    In: International Conference on Learning Representations (2019),https://openreview.net/forum?id=Bkg6RiCqY7

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2019),https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  79. [79]

    In: ICLR (Poster) (2015),http: //dblp.uni-trier.de/db/conf/iclr/iclr2015.html#KingmaB14

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (Poster) (2015),http: //dblp.uni-trier.de/db/conf/iclr/iclr2015.html#KingmaB14

    2015

  80. [80]

    same” or “diff

    Ernst, M.D.: Permutation methods: a basis for exact inference. Statistical Science pp. 676–685 (2004) 38 Supplementary information Supplementary Note 1: Representational alignment metrics and the consensus manifold Pairwise alignment.For two encoders embedding the same set of cases, mutualk-nearest- neighbor alignment (mKNN) is the average overlap of the ...

    2004