Pith. sign in

REVIEW 2 major objections 5 minor 25 references

Pathology foundation models beat CNNs on robustness, but mid-sized ones already match or beat the billion-parameter systems.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 19:26 UTC pith:D6GONDYU

load-bearing objection Solid multi-model PFM robustness benchmark with a real scaling-saturation signal, but NR-Kfold is model-dependent and the clinical leap rests on frozen patch probes. the 2 major comments →

arxiv 2607.04401 v1 pith:D6GONDYU submitted 2026-07-05 cs.CV cs.LG

The Good, the Bad, and the Brittle: Benchmarking Robustness and Generalisation of Histopathology Foundation Models

Dhyey Yajnik , Amina Asif , Fayyaz Minhas This is my paper
classification cs.CV cs.LG
keywords histopathology foundation modelsrobustnessdomain generalisationperturbation performance indexnon-redundant k-foldmodel scalingcomputational pathology
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Pathology foundation models are sold as more reliable than older convolutional networks for reading tissue slides, yet clinical use demands they survive stain changes, scanner quirks, blur, and real differences across hospitals. This paper tests twelve of those models plus ResNet baselines on four patch-classification tasks using eleven controlled clinical-style image perturbations and a special fold-splitting method that deliberately breaks similarity between training and test data. The models do outperform the CNNs on both fronts, but once size reaches a few hundred million parameters further growth yields little or no gain; mid-sized models such as UNI2 and Virchow-2 sit on the robustness frontier while the billion-parameter systems plateau or dip. The practical message is that the field has hit a scaling limit: the next generation must improve data quality, multimodality, and domain alignment rather than simply train larger models if the systems are to become clinically trustworthy.

Core claim

Pathology foundation models consistently outperform convolutional baselines in both perturbation robustness and domain generalisation, yet robustness and out-of-domain accuracy show clear diminishing returns with parameter count: mid-sized models achieve comparable or greater resilience than billion-parameter systems, so genuine clinical reliability now requires prioritising data quality, multimodality and domain alignment over further scale.

What carries the argument

Perturbation Performance Index (PPI) — the average ROC-AUC of a model across a normalised sweep of constraint budgets for each clinically realistic image transform — together with Non-Redundant K-fold validation that forces entire feature-space clusters into different folds so that train–test similarity is deliberately broken.

Load-bearing premise

That freezing the foundation-model encoder and training only a linear head on patch-level binary tasks under this particular suite of synthetic perturbations and cluster-based folds is a fair proxy for how the models would behave in real multi-centre clinical deployment.

What would settle it

Re-run the same models with full fine-tuning or slide-level aggregation on a true multi-centre external cohort; if the billion-parameter models then clearly outrank the mid-sized ones on both perturbation robustness and domain-shift accuracy, the scaling-saturation claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper benchmarks twelve pathology foundation models (PFMs) plus ResNet baselines for robustness and domain generalisation on four patch-level histopathology classification tasks. Using the REET toolbox it applies eleven clinically motivated perturbations and summarises performance via a new Perturbation Performance Index (PPI), the average ROC-AUC over a normalised constraint sweep. Domain-shift behaviour is assessed with a Non-Redundant K-fold (NR-Kfold) protocol that clusters each model’s own embeddings and assigns whole clusters to folds. The central empirical claim is that PFMs outperform CNNs, yet both PPI and NR-Kfold ΔAUC exhibit diminishing returns with scale: mid-sized models (UNI2, Virchow-2, etc.) match or exceed billion-parameter systems, implying that future gains should come from data quality, multimodality and domain alignment rather than parameter count alone.

Significance. If the scaling-saturation result holds under more stringent protocols, it supplies a timely, multi-model empirical counterweight to the prevailing “bigger-is-better” narrative in computational pathology and supplies two reusable evaluation tools (PPI and NR-Kfold). The work is strengthened by the breadth of models (twelve PFMs spanning three orders of magnitude), the public REET codebase, and the explicit reporting of weak parameter–robustness correlations together with bootstrap slopes. These elements make the study a useful reference point for both model developers and clinical-validation teams, even if the absolute ranking of any single architecture remains provisional.

major comments (2)
  1. §2.4 and Fig. 1b: NR-Kfold folds are constructed by cosine k-means on the embeddings of the very encoder under test. Consequently the induced covariate/class-conditional shift is model-dependent; larger models can carve different cluster geometries and therefore experience different effective shifts. Without a fixed reference embedding space (or raw-pixel clustering) applied uniformly to all models, the claim that mid-sized PFMs match or exceed billion-parameter models in domain generalisation cannot be cleanly isolated from this circularity. A sensitivity experiment with a common partition is required before the scaling-saturation conclusion for generalisation can be regarded as established.
  2. §2.2 and Conclusions §4: All results rest on linear probing of frozen encoders on binary patch-level tasks. The manuscript itself notes this limitation, yet the clinical-reliability recommendation is framed as if the observed mid-size plateau generalises to full fine-tuning and slide-level aggregation. At least one ablation with end-to-end fine-tuning (or a multi-instance head) on a subset of models is needed to test whether the ranking reverses under more realistic training regimes.
minor comments (5)
  1. Abstract and title: “scaling limites” is a typographical error; correct to “limits”.
  2. Eq. (1): the discrete approximation of the PPI integral is written with an average over L points; state explicitly whether the τ-grid is uniform in the original constraint or after normalisation, and report L for each perturbation family.
  3. Fig. 1a(iii) and 1b(iii): the isotonic/LOWESS frontiers are visually helpful, but the exact parameter counts used for each model should be listed in a supplementary table so that the plotted points can be reproduced.
  4. §2.1: the NCT subsample fractions (10 % train/val, 2.5–10 % test) and the PanNuke neoplastic-nucleus threshold (≥5) are free parameters that affect class balance; a short sensitivity check would strengthen confidence that rankings are stable.
  5. References [10] and [11] are identical; remove the duplicate.

Circularity Check

2 steps flagged

Empirical benchmark with minor self-referential NR-Kfold construction (model embeddings define the folds) and self-citation of REET; rankings not forced by construction or tautology.

specific steps
  1. other [§2.4 NR-Kfold Distribution-Shift Evaluation; Fig. 1b(i)]
    "each dataset was partitioned into five non-redundant folds by clustering feature embeddings using cosine k-means (k=5) within each class and assigning entire clusters to distinct folds simulating various domain shifts as discussed earlier (see Fig. 1b(i) for an example using Virchow-2 over PanNuke where folds occupy different areas of the feature space)."

    The folds that define the 'domain shift' are constructed from the very embeddings of the model being scored. Consequently the covariate/class-conditional shift experienced by each model is a function of that model's own representation geometry; larger or differently trained encoders can induce systematically different cluster separations. The reported ranking of mid-sized vs. billion-parameter models on ΔAUC / fold-wise SD is therefore not measured against a fixed, model-independent distribution shift, introducing a mild self-referential dependence between the evaluation protocol and the quantity being ranked.

  2. self citation load bearing [Abstract; §1; §2.3; ref. [2]]
    "We benchmarked twelve pathology foundation models (PFMs) and ResNet baselines using our Robustness Evaluation and Enhancement Toolbox (REET) [2] across eleven clinically realistic perturbations"

    REET is introduced and cited as the authors' own prior toolbox (overlapping authors Asif/Minhas). While the citation supplies the perturbation generation machinery that underpins all PPI numbers, it is not a uniqueness theorem or ansatz that forces the mid-size superiority claim; the claim remains an empirical observation obtained by running the toolbox. The self-citation is therefore present and methodologically load-bearing but not circular in the strong sense that collapses the result to an input identity.

full rationale

This is an empirical benchmarking study, not a first-principles derivation. The core claims (PFMs beat ResNets; mid-sized models match or exceed billion-parameter ones on PPI and NR-Kfold ΔAUC) are measurements of public models on public datasets under defined protocols. PPI is simply the integral of observed ROC-AUC over a normalised constraint sweep (Eq. 1); it summarises data rather than predicting it. REET is self-cited ([2], overlapping authors) as the perturbation engine, but the toolbox is a reusable method whose outputs are new measurements, not a uniqueness theorem or ansatz that forces the mid-size ranking. The only mild circularity is that NR-Kfold partitions are built by cosine k-means on the embeddings of the encoder under test (illustrated with Virchow-2; §2.4 and Fig. 1b(i)), so the induced shift is model-dependent and the ΔAUC comparison is not fully isolated from how each model carves feature space. This confounds clean isolation of the scaling claim but does not make ΔAUC equal to an input by algebraic identity or fitted tautology. No fitted-input-as-prediction, no uniqueness imported from authors, no renaming of a known result. Score 2 reflects one non-load-bearing self-reference plus a methodological self-dependence that does not collapse the central empirical result.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The central claim is an empirical ranking under chosen evaluation protocols. It rests on domain assumptions that REET perturbations and embedding-cluster folds stand in for clinical shift, on the free design choices of k, sampling rates, and constraint normalisation, and on two invented summary constructs (PPI, NR-Kfold). No new physical entities are postulated; the ledger is methodological.

free parameters (5)
  • NR-Kfold cluster count k = 5
    Cosine k-means uses k=5 within each class to build non-redundant folds; the induced shift strength and ΔAUC depend on this choice.
  • NCT data subsample fractions = 10% train/val; 2.5–10% test
    Training/validation subsampled to 10% and test to 2.5–10% for efficiency; rankings could change with full data.
  • Perturbation constraint budgets and normalisation to τ∈[0,1]
    PPI integrates AUC over a hand-chosen [ε_min, ε_max] range linearly mapped to [0,1]; the index value depends on those ranges.
  • Linear-probe training details
    Single linear layer with cross-entropy; optimiser, learning rate, epochs, and regularisation are not fully specified in the text but affect absolute AUC.
  • PanNuke neoplastic-nucleus threshold = ≥5 nuclei
    Positive if ≥5 neoplastic nuclei, negative if none (1–4 excluded); class definition is a design choice that shapes the task.
axioms (5)
  • domain assumption Linear probing of frozen encoders is an adequate probe of foundation-model robustness and generalisation for clinical conclusions.
    Stated in §2.2 and used for all results; full fine-tuning or slide-level heads are not evaluated.
  • domain assumption The eleven REET transformations (noise, brightness, compression, stain/HED, blur, rotation, crop, zoom, etc.) are clinically realistic and cover the relevant perturbation space.
    §2.3 and Introduction; conclusions about clinical reliability inherit this coverage assumption.
  • domain assumption Assigning cosine k-means embedding clusters to distinct folds induces covariate and class-conditional shifts that emulate multi-centre deployment.
    §2.4 and Fig. 1b(i); NR-Kfold is treated as a direct measure of generalisation robustness under clinically meaningful shifts.
  • ad hoc to paper PPI, defined as the integral (average) of test ROC-AUC over normalised constraint τ∈[0,1], is a faithful scalar summary of robustness.
    Eq. (1) in §2.3; all robustness rankings and scaling plots use this definition.
  • domain assumption Parameter count is a meaningful primary axis for comparing capacity-related robustness gains across heterogeneous PFM architectures.
    Fig. 1a(iii) and 1b(iii); architecture and pretraining differences are acknowledged but scale is the plotted independent variable.
invented entities (2)
  • Perturbation Performance Index (PPI) no independent evidence
    purpose: Scalar summary of model accuracy across a full sweep of perturbation strengths for ranking and scaling analysis.
    Defined in Eq. (1) as the area under the AUC-vs-τ curve; not an external physical quantity, only an evaluation construct.
  • Non-Redundant K-fold (NR-Kfold) protocol no independent evidence
    purpose: Create train/test folds with deliberate feature-space dissimilarity to measure distribution-shift generalisation.
    Introduced in §2.4 via within-class cosine k-means clustering; the protocol is a methodological invention of the paper’s evaluation framework.

pith-pipeline@v1.1.0-grok45 · 11511 in / 3573 out tokens · 50732 ms · 2026-07-11T19:26:22.717339+00:00 · methodology

0 comments
read the original abstract

How robust and generalisable are pathology foundation models and have their scaling limites been reached? We benchmarked twelve pathology foundation models (PFMs) and ResNet baselines using our Robustness Evaluation and Enhancement Toolbox (REET) across eleven clinically realistic perturbations and a dissimilarity-driven Non-Redundant K-fold validation (NR-Kfold) protocol. We introduce a Perturbation Performance Index (PPI) to summarise accuracy trends under controlled perturbation sweeps and analyse robustness scaling with parameter count. We show that PFMs consistently outperform CNNs in both robustness and domain generalisation, yet model scaling shows diminishing returns: mid-sized models such (UNI2/Virchow-2 etc.) achieve comparable or greater resilience than larger systems. NR-Kfold analysis further reveals systematic accuracy loss and increased variability when training-test similarity is broken, underscoring the need for explicit distribution-shift evaluation. These findings suggest that the next generation of pathology foundation models must prioritise data quality, multimodality information and domain alignment over parameter count to achieve genuine clinical reliability.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 6 linked inside Pith

  1. [1]

    INTRODUCTION Deep learning has transformed computational pathol- ogy, enabling automated analysis of large-scale histol- ogy data and discovery of diagnostic and prognostic biomarkers. However, clinical reliability demands more than high test-set accuracy: models must remain stable under staining variation, scanner bias, and imag- ing artefacts that often...

  2. [2]

    MATERIALS AND METHODS To address these research questions, we implemented a unified robustness and generalisation framework based on the Robustness Evaluation and Enhancement Tool- box (REET) [2], evaluating twelve pathology founda- tion models (PFMs) and ResNet baselines across four different datasets covering diverse, clinically relevant patch-level cla...

    Pith/arXiv arXiv 2026

  3. [3]

    1b(i) for an example using Virchow-2 over PanNuke where folds occupy differ- ent areas of the feature space)

    within each class and assigning entire clusters to distinct folds simulating various domain shifts as dis- cussed earlier (see Fig. 1b(i) for an example using Virchow-2 over PanNuke where folds occupy differ- ent areas of the feature space). A linear probe was trained on four folds (with a stratified 30% valida- tion subset) and tested on the held-out fol...

  4. [4]

    Perturbation based Robustness As shown in Fig

    RESULTS AND DISCUSSION 3.1. Perturbation based Robustness As shown in Fig. 1a(ii), robustness patterns were struc- tured by both perturbation category and model fam- ily. Pixel-level distortions (noise, brightness, JPEG compression) remained the most destabilising, pro- ducing the lowest PPIs (≈0.4–0.6), whereas stain and colour-space variations induced m...

  5. [5]

    PFMs consistently surpass convolu- tional baselines in both robustness and domain general- isation, yet performance gains plateau with increasing scale

    CONCLUSIONS This study provides the first comprehensive benchmark of PFMs under controlled perturbations and structured distribution shifts. PFMs consistently surpass convolu- tional baselines in both robustness and domain general- isation, yet performance gains plateau with increasing scale. These results highlight that domain-specific pretraining, data ...

  6. [6]

    Current pathology foundation models are unrobust to medical center differences,

    Edwin D de Jong, Eric Marcus, and Jonas Teuwen, “Current pathology foundation models are unrobust to medical center differences,”arXiv preprint arXiv:2501.18055, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Reet: robustness evaluation and enhancement toolbox for computational pathology,

    Alex Foote, Amina Asif, Nasir Rajpoot, et al., “Reet: robustness evaluation and enhancement toolbox for computational pathology,”Bioinfor- matics, vol. 38, no. 12, pp. 3312–3314, 2022

    2022

  8. [8]

    Domain generalization in computational pathol- ogy: survey and guidelines,

    Mostafa Jahanifar, Manahil Raza, Kesi Xu, et al., “Domain generalization in computational pathol- ogy: survey and guidelines,”ACM Computing Surveys, vol. 57, no. 11, pp. 1–37, 2025

    2025

  9. [9]

    A clinical benchmark of pub- lic self-supervised pathology foundation mod- els,

    Gabriele Campanella, Shengjia Chen, Manbir Singh, et al., “A clinical benchmark of pub- lic self-supervised pathology foundation mod- els,”Nature Communications, vol. 16, no. 1, pp. 3640, 2025

    2025

  10. [10]

    100,000 histological images of human colorectal cancer and healthy tissue,

    Jakob Nikolas Kather, Niels Halama, and Alexander Marx, “100,000 histological images of human colorectal cancer and healthy tissue,” (No Title), 2018

    2018

  11. [11]

    Artificial intelligence for diagnosis and gleason grading of prostate cancer: the panda challenge,

    Wouter Bulten, Kimmo Kartasalo, Po- Hsuan Cameron Chen, et al., “Artificial intelligence for diagnosis and gleason grading of prostate cancer: the panda challenge,”Nature medicine, vol. 28, no. 1, pp. 154–163, 2022

    2022

  12. [12]

    Pannuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification,

    Jevgenij Gamper, Navid Alemi Koohbanani, Ksenija Benes, et al., “Pannuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification,” inEuropean Congress on Digital Pathology. Springer, 2019, pp. 11–19

    2019

  13. [13]

    Pannuke dataset exten- sion, insights and baselines,

    Jevgenij Gamper, Navid Alemi Koohbanani, Si- mon Graham, et al., “Pannuke dataset exten- sion, insights and baselines,”arXiv preprint arXiv:2003.10778, 2020

    Pith/arXiv arXiv 2003

  14. [14]

    Diagnostic assess- ment of deep learning algorithms for detection of lymph node metastases in women with breast cancer,

    Babak Ehteshami Bejnordi, Mitko Veta, Paul Jo- hannes Van Diest, et al., “Diagnostic assess- ment of deep learning algorithms for detection of lymph node metastases in women with breast cancer,”Jama, vol. 318, no. 22, pp. 2199–2210, 2017

    2017

  15. [16]

    Deep residual learning for image recogni- tion,

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, et al., “Deep residual learning for image recogni- tion,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  16. [18]

    Hibou: A family of founda- tional vision transformers for pathology,

    Dmitry Nechaev, Alexey Pchelnikov, and Eka- terina Ivanova, “Hibou: A family of founda- tional vision transformers for pathology,”arXiv preprint arXiv:2406.05074, 2024

    Pith/arXiv arXiv 2024

  17. [19]

    Scaling self-supervised learn- ing for histopathology with masked image mod- eling,

    Alexandre Filiot, Ridouane Ghermi, Antoine Olivier, et al., “Scaling self-supervised learn- ing for histopathology with masked image mod- eling,”medRxiv, 2023

    2023

  18. [20]

    Phikon-v2, a large and public feature ex- tractor for biomarker prediction,

    Alexandre Filiot, Paul Jacob, Alice Mac Kain, et al., “Phikon-v2, a large and public feature ex- tractor for biomarker prediction,” 2024

    2024

  19. [21]

    Exaonepath 1.0 patch-level foundation model for pathology,

    Juseung Yun, Yi Hu, Jinhyung Kim, et al., “Exaonepath 1.0 patch-level foundation model for pathology,”arXiv preprint arXiv:2408.00380, 2024

    Pith/arXiv arXiv 2024

  20. [23]

    A foundation model for clinical-grade computational pathology and rare cancers detec- tion,

    Eugene V orontsov, Alican Bozkurt, Adam Cas- son, et al., “A foundation model for clinical-grade computational pathology and rare cancers detec- tion,”Nature Medicine, 2024

    2024

  21. [24]

    Virchow2: Scaling self-supervised mixed magnification models in pathology,

    Eric Zimmermann, Eugene V orontsov, Julian Viret, et al., “Virchow2: Scaling self-supervised mixed magnification models in pathology,”arXiv preprint arXiv:2408.00738, 2024

    Pith/arXiv arXiv 2024

  22. [25]

    Towards a general-purpose foundation model for computational pathology,

    Richard J Chen, Tong Ding, Ming Y Lu, et al., “Towards a general-purpose foundation model for computational pathology,”Nature Medicine, 2024

    2024

  23. [26]

    A whole-slide foundation model for dig- ital pathology from real-world data,

    Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, et al., “A whole-slide foundation model for dig- ital pathology from real-world data,”Nature, 2024

    2024

  24. [27]

    H-optimus-0,

    Charlie Saillard, Rodolphe Jenatton, Felipe Llinares-L´opez, et al., “H-optimus-0,”https: //github.com/bioptimus/releases/ tree/main/models/h-optimus/v0, 2024

    2024

  25. [28]

    H-optimus-1,

    Bioptimus, “H-optimus-1,”https: //huggingface.co/bioptimus/ H-optimus-1, 2025

    2025