{"id":"fed1762d-8149-40c1-b5bc-0ec2f89338c7","arxiv_id":"2412.06365","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"End-to-end trained ResNet50 outperforms linear probing of five foundation models for mitotic figure classification across data sizes and domains.","lead":"This paper benchmarks five histopathology foundation models against a simple end-to-end trained ResNet50 for mitotic figure classification. It finds the simple baseline wins at every training set size and on cross-domain tests, challenging the idea that self-supervised embeddings reduce the need for labeled data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Comparison confounds fine-tuning and augmentation with source of pretraining; the end-to-end baseline is not matched in protocol, so the 'self-supervision not enough' claim overreaches.","rationale":"The paper is a timely, clearly reported benchmark, and the authors are appropriately cautious in the discussion ('better results might be achievable if the FMs were fine-tuned'). The reader's conditional verdict is reasonable. My stress-test agrees with the reader's identification of the frozen-embedding/linear-probe comparison as the critical assumption, but sharpens it: the comparison also varies data augmentation and whether the feature extractor is updated. The paper's own methods section (Section 3) documents that the end-to-end baseline used a standard augmentation pipeline (color jitter, Gaussian blur, flipping, rotations) while 'no additional data augmentation was applied' to the embedding-based training. Since linear probes on pre-extracted embeddings cannot benefit from augmentation, the observed gap may be due to the extra capacity and regularization rather than the choice of pretraining. This is not an internal inconsistency--the authors state the limitation--but it means the abstract's 'rendering both of the above assumptions incorrect' is too broad. A matched-protocol experiment (fine-tuning an FM, or adding augmentation to the linear probe) would settle whether the gap is about self-supervision or about training protocol. The proposed concrete tests also address the reader's request for significance testing. I would keep the verdict CONDITIONAL (UNCHANGED), with the condition that the authors either run a matched-protocol comparison or explicitly limit the conclusion to linear probing without augmentation.","tokens_in":5264,"tokens_out":9801,"duration_ms":94510,"concrete_test":"Run a matched-protocol experiment. Fine-tune the best-performing FM (e.g., H-optimus-0, or the smaller Phikon as a feasible proxy) end-to-end on the same training subsets, using the same augmentation pipeline and one-cycle schedule as the ResNet50 baseline, and re-measure AUROC at all four dataset sizes on the same splits. To isolate the augmentation effect without full fine-tuning, also train linear probes on embeddings extracted from augmented views (e.g., applying the same flips, rotations, and color jitter before patch embedding). Report paired significance tests (Wilcoxon signed-rank or paired bootstrap over the five Monte Carlo folds) for the ResNet50-versus-best-FM difference at each data size. If a fine-tuned FM reaches or exceeds the baseline at 10% or 100% data, the central claim fails; if the augmented linear probe still lags, the linear-probing conclusion is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that an end-to-end-trained ResNet50 outperforms all FM-based classifiers 'regardless of the amount of data provided' rests on a comparison that does not hold the training protocol fixed. In Section 3, FM embeddings are pre-extracted once from 224x224 patches at 0.25 mpp and evaluated by training a single linear layer with no data augmentation (following Chen et al.), while the end-to-end baseline is trained with the full network using a standard augmentation pipeline (color jitter, Gaussian blur, flipping, rotations) and a one-cycle learning-rate schedule. Three factors are thus confounded: fine-tuning versus frozen features, augmentation versus none, and CNN versus ViT architecture. The paper concedes in Section 5 that fine-tuning the FMs might improve results, yet the abstract concludes that the motivating assumptions are 'incorrect.' If the observed gap is caused by the extra trainable capacity and augmentation rather than by the source of pretraining, the headline claim overstates what the experiments show. Additionally, in the dataset-size experiment the 20% test split is made at the annotation level, not the whole-slide level, although the CCMCT data comprise only 32 WSIs; this may leak slide-specific appearance into training and could differentially favor the model with more trainable parameters.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks linear classifiers trained on frozen embeddings from public histopathology foundation models (Prov-GigaPath, H-optimus-0, Virchow, Virchow2, UNI, Phikon; the abstract says five, but Table 1 lists six) against ImageNet-pretrained feature extractors and an end-to-end-trained ResNet50 for mitotic figure classification. The authors evaluate performance as a function of training set size on MIDOG and CCMCT and evaluate cross-domain robustness on MIDOG. Their main findings are that the end-to-end baseline outperforms all FM-based classifiers at every data size and that FM-based classifiers are not more robust to domain shift. The paper concludes that the assumptions that foundation-model embeddings reduce labeling needs and improve domain robustness are incorrect for this task.","tokens_in":5480,"tokens_out":5368,"duration_ms":53202,"significance":"This paper addresses a timely and clinically relevant question: whether large self-supervised histopathology models can substitute for task-specific end-to-end training in mitotic figure classification. Its strengths are the use of public datasets and public foundation models, the systematic variation of training set size, and the inclusion of a domain-shift experiment. A careful, protocol-matched comparison would be a valuable negative result for the computational pathology community. However, as presented, the experimental protocol confounds the source of pretraining with fine-tuning, data augmentation, and training compute, so the headline claim is broader than what the evidence actually supports. The paper is reproducible in principle because all datasets and models are public, but the protocol limitations currently limit its impact.","major_comments":[{"comment":"The central comparison conflates the source of pretraining with three other factors. The FM embeddings are evaluated with a single linear layer on frozen features with no data augmentation, whereas the end-to-end ResNet50 is fully trainable, is trained with color jitter, Gaussian blurring, flipping, and rotations, and uses a one-cycle learning-rate schedule. This means the observed performance gap could be due to fine-tuning capacity, augmentation, or training compute rather than to the supervision signal alone. The discussion in Section 5 already concedes that fine-tuning the FMs might improve results, yet the abstract and title draw the much stronger conclusion that self-supervision is not enough. At a minimum, the authors should rephrase the claim to refer to linear probing of frozen embeddings, or better, provide a matched evaluation such as fine-tuning the FMs with the same augmentation pipeline or evaluating a frozen ResNet50 linear probe under identical conditions.","section":"Section 3, paragraphs 1 and 3"},{"comment":"The dataset-size experiment draws the test set as 20% of all annotations at the annotation level. The CCMCT dataset comprises only 32 whole-slide images, so this type of split can place patches from the same WSI in both training and test sets. This leaks slide-level staining and tissue appearance into training, which may differentially benefit the end-to-end model because it has more trainable parameters and can more easily memorize slide-specific artifacts. This could inflate the reported advantage of the end-to-end baseline on CCMCT. The authors should use a slide-level split for CCMCT and an ROI-level split for MIDOG, or explicitly justify why an annotation-level split is appropriate for the generalization claim.","section":"Section 3, dataset-size experiment"},{"comment":"All FM embeddings are computed from 224x224 patches at 0.25 microns per pixel, but the foundation models were pretrained at different resolutions and patch sizes (for example, Virchow and UNI use 0.5 microns per pixel). The single-resolution protocol therefore probably places some FMs outside their native operating range, while the end-to-end ResNet50 is trained and tested on the same extracted patches. Without a sensitivity analysis or a justification that the chosen resolution is appropriate for each FM, the comparison is not at each model's best operating point, and the conclusion that FMs underperform the baseline may be partly an artifact of suboptimal embedding extraction.","section":"Section 3, patch extraction paragraph"}],"minor_comments":[{"comment":"The abstract says 'five publicly available FMs' but Table 1 lists six foundation models; please correct the count.","section":"Abstract"},{"comment":"Figure 1 does not show error bars or confidence intervals even though the text reports standard deviations; adding error bars would make the comparative claims much easier to assess.","section":"Section 4, Figure 1"},{"comment":"The sentence 'The positive impact of the FMs’ pretraining- on the given tasks suggests a comparison...' contains an apparent typo ('pretraining-') and is difficult to parse; please rewrite it.","section":"Section 5"},{"comment":"The phrase 'better then' should be 'better than'.","section":"Section 5"},{"comment":"DINOv2 is described as training by 'contrasting differently augmented views'; a more precise description would mention self-distillation, since DINOv2 does not use a contrastive loss.","section":"Section 3, DINOv2 description"},{"comment":"The linear probing implementation is attributed to Chen et al. [3], but the original linear evaluation protocol is usually cited to the relevant self-supervised learning paper (e.g., DINOv2 or SimCLR); please add the appropriate reference.","section":"Section 3, linear probing protocol"},{"comment":"Please clarify how the 44,880 MF and 27,965 look-alike annotations were derived from the original 262,481 annotations, and whether any patches were discarded because they were out of slide bounds or otherwise unusable at the chosen resolution.","section":"Section 2, CCMCT dataset"}],"recommendation":"major_revision","confidential_remarks":"This paper is a useful and timely negative result for the foundation-model benchmarking literature, and the core question is well motivated. However, the headline claim currently overreaches, and the protocol confounds (fine-tuning, augmentation, and resolution mismatch) will need to be addressed or carefully retracted before the paper can be recommended for acceptance. I would encourage the editor to seek a revised version that either matches the protocols or narrows the claim to what the data actually show."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: this is a solid empirical benchmark that gives the pathology community a clear data point: frozen histopathology FM embeddings under linear probing don't beat a simple end-to-end ResNet50 for mitotic figure classification, and they don't confer obvious domain robustness. That is worth publishing. What is genuinely new is the systematic comparison of five public FMs plus two ImageNet baselines against a single end-to-end baseline, across training-set sizes and cross-domain splits, on two public datasets. The authors also deserve credit for bounding the claim in the discussion: they note that fine-tuning the FMs might improve results and that linear probing is the cheaper deployment mode. The data scaling experiment is a useful reference for practitioners deciding whether to spend annotation budget.\n\nSoft spots. The main one is that the comparison does not isolate the source of pretraining. The end-to-end baseline is trained with a full augmentation pipeline (color jitter, Gaussian blur, flip, rotation) and a one-cycle schedule, while the FM embeddings are frozen and fitted with a single linear layer and no augmentation. So the gap could come from fine-tuning, augmentation, or the extra capacity, not from self-supervision per se. The abstract's phrase 'regardless of the amount of data provided' rests on mean AUROC curves without significance tests; at the smallest data sizes the differences look within noise. The CCMCT test split is done at the annotation level on 32 whole slides, which risks slide-level appearance leakage and could differentially help the more flexible end-to-end model. A slide-level split or a clustered bootstrap would tighten this. Also, all FM embeddings are computed at a single 224x224 patch at 0.25 mpp, which may disadvantage FMs designed around different native magnifications. These are fixable.\n\nProportion: these concerns don't sink the paper. The core observation—that linear probing of these FMs underperforms a well-trained CNN on this task—is likely correct within its stated scope, and the authors clearly bound it. What is overstated is the general claim that self-supervision is not enough; what they actually show is that frozen embeddings with a linear head are not enough. That is a meaningful distinction.\n\nWho is this for? Practitioners building mitotic figure detection pipelines and researchers who use FM embeddings as a default starting point. Worth a serious referee.\n\nRecommendation: accept after revision, provided they add significance testing or at least honest error-bar interpretation, address the augmentation asymmetry either by adding an augmented linear probe as a control or explicitly framing the paper as comparing deployment protocols rather than pretraining paradigms, and discuss the slide-level split issue. I would engage with it.","headline":"A useful benchmark with an honest scope, but the headline claim overreaches because the comparison stacks fine-tuning and augmentation on top of the pretraining comparison.","tokens_in":6034,"tokens_out":1849,"would_cite":true,"duration_ms":19364,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark of mitotic figure classification finds that an end-to-end trained ResNet50 beats classifiers built on frozen histopathology foundation-model embeddings at every training-set size, and that the foundation models are not more…","keywords":["mitotic figure classification","foundation models","histopathology","self-supervised learning","linear probing","domain shift","digital pathology","benchmarking"],"falsifier":"Fine-tune one of the best-performing foundation models, such as H-optimus-0, on the MIDOG training domains and test on the held-out domains; if its out-of-domain AUROC reaches or exceeds the end-to-end baseline's 0.74, the claim that self-supervised pretraining cannot replace task-specific supervision for this task would be refuted.","tokens_in":5065,"feed_emoji":"🔬","tokens_out":13090,"duration_ms":101107,"temperature":0.7,"pith_summary":"Foundation models for histopathology promise that self-supervised pretraining on vast unlabeled data produces embeddings good enough that a simple classifier can solve downstream tasks with little labeled data and robust to new domains. The paper tests that promise for mitotic figure classification, a key tumor-prognosis biomarker, using two public datasets and classifiers trained on frozen embeddings from several foundation models. The result is negative: an end-to-end trained ResNet50, pretrained only on ImageNet and then fine-tuned on the task, outperforms every foundation-model-based classifier at every training-set size tested, from 0.1% to 100% of the data. The foundation-model classifiers also do not generalize better across the five domains of the MIDOG dataset, showing comparable or larger domain-shift drops. If correct, the paper establishes that for this task extensive self-supervised pretraining is not a substitute for task-specific supervision.","feed_headline":"End-to-end CNN beats self-supervised pathology foundation models","feed_subtitle":"The fine-tuned ResNet50 outperformed all foundation-model classifiers on two public datasets, at every data size.","key_machinery":"The machinery that carries the comparison is linear probing on frozen embeddings: each foundation model turns a $224 \\times 224$ pixel patch, extracted at 0.25 microns per pixel, into a vector (the class token, or for Virchow and Virchow2 the concatenation of class token and mean patch token), and a single linear layer is trained on those vectors. This isolates what the pretrained representations contribute, since only the final layer learns from labeled mitotic figure data. The evaluation protocol is equally load-bearing: identical train/validation/test splits across models, five-fold Monte Carlo cross-validation, training-set sizes from 0.1% to 100%, and a leave-one-domain-out scheme on the five MIDOG domains that measures in-domain and out-of-domain AUROC separately.","core_discovery":"The paper's central discovery is that frozen self-supervised histopathology embeddings do not carry the day for mitotic figure classification. Using shared train/validation/test splits and five-fold Monte Carlo cross-validation, the authors trained linear classifiers on embeddings from the compared foundation models and compared them with ImageNet-pretrained feature extractors and with a ResNet50 stem fine-tuned end to end. The end-to-end baseline reached mean AUROC of 0.87 on the CCMCT whole-slide dataset (canine cutaneous mast cell tumors) and 0.89 on MIDOG at full data, while the best foundation-model-based classifiers remained below that at every training-set size. In the cross-domain experiment, the end-to-end baseline again led both in-domain (0.87) and out-of-domain (0.74) AUROC, versus 0.79 and 0.66 for the best FM linear probe, and both model types suffered large and uneven domain-shift drops. The paper concludes that the two advertised benefits of foundation models for this task, reduced labeled-data need and increased domain robustness, did not materialize under linear probing.","pith_inferences":["This suggests a stronger test of the paper's question would be fine-tuning the foundation models themselves; the authors note this could improve results but require much more compute, and it remains open whether the gap would close.","Because all FMs were given $224 \\times 224$ patches at 0.25 microns per pixel regardless of the magnification they were trained on, per-model patch resolution might change the ranking, an untested confound.","This result cautions against assuming that any histopathology foundation model will transfer to a new, visually demanding task; similar linear-probing benchmarks on other rare-cell classification tasks would show whether the finding generalizes beyond mitotic figures."],"forward_implications":["For mitotic figure classification, adopting a frozen foundation-model embedding plus a linear layer will not save labeling effort: the end-to-end-trained network achieves higher AUROC at every training-set size from 0.1% to 100%.","Foundation-model embeddings do not remove the need for domain-specific data: on the five-domain MIDOG benchmark, their out-of-domain AUROC drops to 0.66, only slightly above the ImageNet baselines and below the end-to-end model's 0.74.","The fact that all FM embeddings beat the ImageNet feature extractors shows the self-supervised representations do capture histology-specific information; the shortfall is specific to the frozen-embedding, linear-probe mode of use.","The observed performance curves had not converged at the largest dataset size, so larger training sets could still narrow the gap between FM-based and end-to-end classifiers."],"supporting_citations":[{"why":"Supplies the MIDOG challenge dataset with five domains, the basis for both the dataset-size and cross-domain experiments.","marker":"[1]"},{"why":"Provides the linear-probing implementation used for all FM classifiers and the UNI model.","marker":"[3]"},{"why":"Introduces Virchow and the class-token-plus-mean-patch-token embedding recipe used for Virchow and Virchow2.","marker":"[4]"},{"why":"Provides Virchow2, one of the FM embeddings compared in the benchmark.","marker":"[5]"},{"why":"Provides Phikon, the iBOT-trained FM that performs among the weaker linear probes.","marker":"[6]"},{"why":"Provides H-optimus-0, the FM with the best dataset-size performance in the first experiment.","marker":"[7]"},{"why":"Provides Prov-GigaPath, a whole-slide-trained FM among the top-performing linear probes.","marker":"[8]"},{"why":"Supplies the CCMCT whole-slide dataset of canine cutaneous mast cell tumors used as the second benchmark.","marker":"[10]"},{"why":"Describes DINOv2, the self-supervised pretraining underlying most of the compared FMs and the premise being tested.","marker":"[11]"}],"fun_headline_variants":["End-to-end CNN outclasses pathology foundation models","Foundation models fall short for mitotic figure classification","Self-supervised FMs underperform end-to-end CNN on mitosis","No benefit from self-supervised FMs for mitotic figures","End-to-end training beats self-supervised FMs for mitosis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a single linear layer trained on frozen $224 \\times 224$ embeddings at 0.25 microns per pixel is a fair and representative way to use foundation models; if fine-tuning the models or adjusting resolution per model changes the ranking, the paper's conclusion would no longer hold.","fun_headline_variants_meta":{"raw":{"variants":["End-to-end CNN outclasses pathology foundation models","Foundation models fall short for mitotic figure classification","Self-supervised FMs underperform end-to-end CNN on mitosis","No benefit from self-supervised FMs for mitotic figures","End-to-end training beats self-supervised FMs for mitosis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000567,"raw_usage":{"total_tokens":2676,"prompt_tokens":927,"completion_tokens":1749,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":1668}},"tokens_in":543,"tokens_out":1749,"duration_ms":12597,"temperature":1.0,"reasoning_tokens":1668,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:43:33.984178+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune one of the best-performing foundation models, such as H-optimus-0, on the MIDOG training domains and test on the held-out domains; if its out-of-domain AUROC reaches or exceeds the end-to-end baseline's 0.74, the claim that self-supervised pretraining cannot replace task-specific supervision for this task would be refuted.","supporting_citations":[{"cited_title":"Do- main generalization across tumor types, laboratories, and species—Insights from the 2022 editionoftheMitosisDomainGeneralizationChallenge.MedicalImageAnalysis94(2024), p","cited_arxiv_id":null,"evidence_quote":"Supplies the MIDOG challenge dataset with five domains, the basis for both the dataset-size and cross-domain experiments."},{"cited_title":"Towards a general- purpose foundation model for computational pathology","cited_arxiv_id":null,"evidence_quote":"Provides the linear-probing implementation used for all FM classifiers and the UNI model."},{"cited_title":"A foundation model for clinical-grade computational pathology and rare cancers detection","cited_arxiv_id":null,"evidence_quote":"Introduces Virchow and the class-token-plus-mean-patch-token embedding recipe used for Virchow and Virchow2."},{"cited_title":"Scaling self-supervised learning for histopathology with masked image modeling","cited_arxiv_id":null,"evidence_quote":"Provides Phikon, the iBOT-trained FM that performs among the weaker linear probes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides H-optimus-0, the FM with the best dataset-size performance in the first experiment."},{"cited_title":"A whole-slide foundation model for digital pathology from real-world data","cited_arxiv_id":null,"evidence_quote":"Provides Prov-GigaPath, a whole-slide-trained FM among the top-performing linear probes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CCMCT whole-slide dataset of canine cutaneous mast cell tumors used as the second benchmark."}],"review_version":1}