Pith. sign in

REVIEW 4 major objections 5 minor 32 references

Towards a Comprehensive Benchmark for Pathological Lymph Node Metastasis in Breast Cancer Sections

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cleaning and re-annotating the Camelyon slides yields a four-class benchmark that changes how models are ranked and exposes weak performance on isolated tumor cells.

desk verdict A genuinely useful cleaned Camelyon benchmark, but the four-class labels are unverifiable until the authors state their size thresholds and show some validation of the relabeling. read the letter →

arxiv 2411.10752 v1 pith:GPY6FKUG submitted 2024-11-16 eess.IV cs.CV

Xitong Ling , Yuanyuan Lei , Jiawen Li , Junru Cheng , Wenting Huang , Tian Guan , Jian Guan , Yonghong He This is my paper · ORCID
classification eess.IVcs.CV
keywords breastcancerlymphnodemetastasiswholeslideimagesbenchmarkdatasetmultipleinstancelearningfoundationmodelsisolatedtumorcellshistopathology
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the widely used Camelyon datasets for breast cancer lymph node metastasis contain enough low-quality slides, labeling errors, and annotation gaps to distort model evaluation, and that a cleaned, merged four-class version, Camelyon+, provides a more reliable benchmark. It reports constructing this dataset from 1,399 original slides, removing 49, correcting labels, and adding expert pixel annotations, yielding 1,350 whole-slide images labeled negative, micro-metastasis, macro-metastasis, or isolated tumor cells. On this benchmark the paper re-evaluates twelve multiple-instance-learning methods and six feature extractors, finding that pathology-specific features outperform natural-image features and that model rankings shift after cleaning. A sympathetic reader would care because the field's most common evaluation set is claimed to be measurably cleaner, and the benchmark exposes that current methods label the rarest category, isolated tumor cells, poorly.

What carries the argument

The central mechanism is a dataset re-annotation pipeline: professional pathologists review each whole-slide image, exclude low-quality or treatment-altered slides, correct slide-level labels, and draw pixel-level tumor annotations where they were missing or wrong; from the size of these re-annotated regions the binary positive label is upgraded into the four classes negative, micro-metastasis, macro-metastasis, and isolated tumor cells. The evaluation machinery is embedding-based multiple instance learning: each slide is tiled into 256x256 patches at 20x magnification, features are extracted by a pre-trained encoder, and a MIL aggregator pools the patch features for slide classification.

What would settle it

Independently re-review a random sample of the 1,350 Camelyon+ slides with multiple pathologists and check whether the four-class labels match; if a substantial fraction, for example more than 10 percent, of positive slides are assigned different categories or reclassified as negative, the benchmark's validity fails. A more direct test is to recover the size thresholds from the released pixel annotations and verify them against the published ITC, micro, and macro definitions.

Watch

Extended reading notes

Core claim

The authors establish that Camelyon+, assembled from corrected Camelyon-16 and Camelyon-17 slides, constitutes a valid four-class benchmark for lymph node metastasis classification, and that on this benchmark the standard multiple-instance-learning pipeline performs well on negative, micro, and macro classes but fails on isolated tumor cells. They report removing slides with blur, poor staining, treatment artifacts, or ambiguous positivity; re-labeling slides based on the size of re-annotated tumor regions; and providing pixel annotations missed in the original data. Their experiments show that after this refinement, the ranking of MIL models changes, with CLAM-MB remaining the strongest, and that pathology-pretrained feature extractors, especially the contrastively trained CONCH model, match or exceed much larger feature extractors. The paper thereby positions Camelyon+ as a resource that should replace raw Camelyon data for fair evaluation of histopathology models.

Load-bearing premise

The load-bearing premise is that the pathologists' corrected slide labels and the size cutoffs used to assign micro, macro, and ITC classes are accurate and consistent; the paper does not state the cutoffs or report inter-observer agreement.

Editorial extensions

If this is right

  • Previous model comparisons run on the raw Camelyon-17 splits may rank methods differently once low-quality slides and label errors are removed.
  • Pathology-specific feature extractors consistently beat natural-image encoders for this four-class task, so future work should use them as the default.
  • The contrastive image-text model CONCH performs comparably to much larger visual encoders, indicating that training data quality can compensate for model scale.
  • The poor F1 score on isolated tumor cells, driven by a 16:1 imbalance and the size-based definition of the class, makes the ITC category a targeted challenge for future MIL research.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper does not publish the exact size thresholds that separate ITC, micro, and macro metastases, a natural next step is for the authors or others to state these cutoffs and validate them against the international tumor-node-metastasis criteria.
  • The released pixel annotations could be reused for a separate task: measuring tumor burden or metastasis size directly, which may be a more natural framing than four-class classification.
  • The observed ranking shift after cleaning implies that any future method claiming state-of-the-art on Camelyon-17 should be re-checked on Camelyon+ before the claim is trusted.
  • The benchmark's long-tailed class distribution mimics real clinical data, so it could also serve as a testbed for long-tail learning methods beyond multiple instance learning.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper introduces Camelyon+, a cleaned and re-annotated combination of the Camelyon-16 and Camelyon-17 datasets for breast cancer lymph node metastasis detection in whole slide images. The authors removed 49 low-quality slides, corrected slide-level labels, added pixel-level annotations to positive slides, and upgraded the binary classification task to a four-class task (negative, micro-metastasis, macro-metastasis, ITC) based on the sizes of re-annotated tumor regions. They then benchmarked 12 multiple instance learning (MIL) methods with 6 feature extractors, reporting metrics on the corrected Camelyon-17 and on the merged Camelyon+ dataset. The central claim is that the cleaned dataset provides a more reliable and clinically relevant benchmark than the original Camelyon series.

Significance. If the re-annotation and label corrections are validated, Camelyon+ would be a valuable community resource that merges the two Camelyon datasets, adds a clinically meaningful four-class label set, and provides extracted features and code to facilitate reproducible benchmarking. The paper also raises a substantive question about whether MIL is the right paradigm for size-based classification tasks. However, the dataset's core contribution depends on label corrections and threshold definitions that are not yet documented with sufficient protocol detail, validation, or inter-observer agreement measures, and the provenance of the Camelyon-17 test set labels is unclear. These gaps currently prevent the benchmark from being fully reproducible and trustworthy.

major comments (4)
  1. [Methods / Dataset Overview and Abstract] The four-class definitions are never operationalized. The paper states only that classes were assigned "based on the sizes of re-annotated tumor regions," but gives no numeric thresholds. Please specify the exact criteria used (e.g., AJCC 8th edition: ITC ≤0.2 mm or <200 cells; micro >0.2–2 mm; macro >2 mm), how tumor region size was measured (e.g., largest focus diameter, total area, or cell count), and how slides with multiple foci of different sizes were assigned to a class. Without this information, the class labels and all downstream comparisons in Tables 3–5 are not reproducible.
  2. [Methods / Technical Validation and Data Records] No inter-observer agreement or external validation is reported for the label corrections or the pixel-level annotations. Please provide the number and experience of the pathologists involved, the blinding procedure, the consensus method, and agreement statistics (e.g., Cohen's kappa) for slide-level classes and for pixel-level annotation overlap. An independent validation step, such as a second pathologist panel or comparison with a reference standard, is essential to support the claim that the original labels were "erroneous" and that the corrections are correct.
  3. [Methods / Dataset Overview and Technical Validation / Camelyon-17 Comparative Experiment] The paper states that the Camelyon-17 test set labels are "not publicly available," yet Table 1 reports performance metrics on the Camelyon-17-Origin test set. Please clarify how the official labels for this test set were obtained. If they were acquired through the challenge organizers, state this explicitly. If they were generated by the authors' re-annotation, then the Camelyon-17-Origin results are not against the official ground truth, and the "Origin vs. Refine" comparison in Tables 1–2 and Figures 3–4 must be reinterpreted accordingly.
  4. [Technical Validation / Camelyon-17 Comparative Experiment (Table 2)] Several entries in Table 2 show very large standard deviations (e.g., PLIP Max-MIL AUC 72.7 ± 11.09, UNI Max-MIL AUC 77.9 ± 11.95, and several F1 values with ±8–10), while other entries are much more stable. The paper does not discuss this instability, despite using these numbers to argue that dataset refinement improves the accuracy and fairness of model rankings. Please comment on the stability of the results and consider reporting additional random seeds or per-fold performance to ensure that the comparative claims are robust.
minor comments (5)
  1. [Tables 3–5 captions] The word "dastaset" is a typo for "dataset" in the captions of Tables 3, 4, and 5.
  2. [Usage Notes] The phrase "Camelon+ Dataset" should read "Camelyon+ Dataset."
  3. [Methods / Benchmark Experiment] The feature extractor name "PILP" in the first paragraph of the benchmark experiment section appears to be a typo for "PLIP." Also, the paper inconsistently uses "VIT-S" and "ViT-S" for the same model; please standardize the notation.
  4. [Data Records] The sentence "The original WSI data can be downloaded from the official websites of Camelyon161 and Camelyon-172" is missing hyphens and spaces: it should read "Camelyon-16" and "Camelyon-17."
  5. [Figures] Figure 5 contains many small text labels that appear garbled or overlapping in the manuscript version; please provide a higher-resolution figure and check the readability of the feature encoder names.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Camelyon+ labels are pathologist-derived inputs, not outputs of the benchmark models.

full rationale

The paper's central deliverable is a cleaned and re-labeled benchmark (Camelyon+) assembled from Camelyon-16/17, plus a comparative evaluation of twelve MIL aggregators and six feature extractors. Walking the derivation chain: the four-class labels are produced by pathologists from the sizes of re-annotated tumor regions; they are not derived from any model output in the paper. The evaluated MIL models are trained and tested on these labels, so the benchmark results do not feed back into the label definitions. The Camelyon-17-Origin versus Camelyon-17-Refine comparison is a before/after comparison of the same modeling protocols under two label/quality regimes; it is an observational evaluation, not a prediction that reduces to its inputs. The conclusion that rankings shift is read directly from Tables 1 and 2, and the claim that refinement improves fairness presupposes the corrected labels are accurate, which is a validity or reproducibility concern, not a circularity. The only self-citation is AMD-MIL (reference 27, same first author), included as one of twelve compared methods. No conclusion is forced by its inclusion; the paper does not invoke it to justify the dataset labels or to rule out alternative methods. No uniqueness theorem or ansatz is imported from prior work. The size thresholds for micro/macro/ITC and inter-observer agreement are not reported, but absent thresholds are a documentation gap, not an equation that makes one quantity equal to another by construction. Accordingly, no circular step can be quoted or exhibited from the manuscript, and the evidence does not support a score above zero.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No new physical entities or fitted parameters are introduced. The main assumptions are about the reliability of the re-annotation and the unstated size thresholds for the four classes.

assumptions (3)
  • domain assumption The original Camelyon labels are unreliable and require manual correction.
    The paper's cleaning effort presupposes that label errors are widespread enough to justify re-annotation; this is motivated by examples in Figure 1, not by a systematic error audit.
  • ad hoc to paper Metastasis category (ITC vs micro vs macro) is determined by tumor region size with thresholds that are not stated.
    The paper states 'Based on the sizes of re-annotated tumor regions, we upgraded...' but never gives the size thresholds, making the class definitions non-reproducible.
  • domain assumption Pathologist re-annotation serves as the ground truth for corrected labels and pixel annotations.
    No inter-observer agreement or second-reader validation is reported; the corrected labels are asserted as expert truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards a Comprehensive Benchmark for Pathological Lymph Node Metastasis in Breast Cancer Sections." pith.science (2026). https://pith.science/paper/GPY6FKUG

@misc{pith2026241110752,
  author       = {Pith},
  title        = {Pith review of: Towards a Comprehensive Benchmark for Pathological Lymph Node Metastasis in Breast Cancer Sections},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GPY6FKUG}},
  note         = {Machine review of arXiv:2411.10752}
}
read the original abstract

Advances in optical microscopy scanning have significantly contributed to computational pathology (CPath) by converting traditional histopathological slides into whole slide images (WSIs). This development enables comprehensive digital reviews by pathologists and accelerates AI-driven diagnostic support for WSI analysis. Recent advances in foundational pathology models have increased the need for benchmarking tasks. The Camelyon series is one of the most widely used open-source datasets in computational pathology. However, the quality, accessibility, and clinical relevance of the labels have not been comprehensively evaluated. In this study, we reprocessed 1,399 WSIs and labels from the Camelyon-16 and Camelyon-17 datasets, removing low-quality slides, correcting erroneous labels, and providing expert pixel annotations for tumor regions in the previously unreleased test set. Based on the sizes of re-annotated tumor regions, we upgraded the binary cancer screening task to a four-class task: negative, micro-metastasis, macro-metastasis, and Isolated Tumor Cells (ITC). We reevaluated pre-trained pathology feature extractors and multiple instance learning (MIL) methods using the cleaned dataset, providing a benchmark that advances AI development in histopathology.

Figures

Figures reproduced from arXiv: 2411.10752 by the authors.

Figure 1
Figure 1. Issues within the Camelyon-161 and Camelyon-172 datasets: therapeutic response, annotation omissions, blurred boundaries, and poor staining. (a) Negative (b) Micro (c) Macro (d) ITC [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Revised data distribution of the Camelyon dataset and the pathological characteristics of different categories. 7/14 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. AUC and F1-score Comparison of Different Methods in the Camelyon-17-Origin and Camelyon-17-Refine Comparative Experiments. Camelyon-17-Origin（PLIP） Camelyon-17-Origin（UNI） Camelyon-17-Origin（Gigapath） Camelyon-17-Origin（Total） Camelyon-17-Refine（PLIP） Camelyon-17-Refine（UNI） Camelyon-17-Refine（Gigapath） Camelyon-17-Refine（Total） F1 AUC AUC + F1 1st 2nd 3rd [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Model ranking radar chart based on AUC and F1-score on Camelyon-17-Origin and Camelyon-17-Refine datasets. 8/14 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Distribution of AUC and F1-score in benchmark results across different feature extractors and aggregators [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Confusion matrices of CLAM-MB19 and FR-MIL31 under the PLIP5 , CONCH6 , UNI9 , and Gigapath7 feature encoders. 9/14 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 16 canonical work pages

  1. [1]

    Bejnordi, B. E. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama 318, 2199–2210 (2017)

    2017

  2. [2]

    Bandi, P. et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE transactions on medical imaging 38, 550–560 (2018)

    2018

  3. [3]

    Song, A. H. et al. Artificial intelligence for digital and computational pathology. Nat. Rev. Bioeng. 1, 930–949 (2023)

    2023

  4. [4]

    & Ciompi, F

    Van der Laak, J., Litjens, G. & Ciompi, F. Deep learning in histopathology: the path to the clinic. Nat. medicine 27, 775–784 (2021). 4/14

    work page 2021

  5. [5]

    Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T. J. & Zou, J. A visual–language foundation model for pathology image analysis using medical twitter. Nat. medicine 29, 2307–2316 (2023)

    2023

  6. [6]

    Chen, R. J. et al. Towards a general-purpose foundation model for computational pathology. Nat. Medicine 30, 850–862 (2024)

    2024

  7. [7]

    Xu, H. et al. A whole-slide foundation model for digital pathology from real-world data. Nature 1–8 (2024)

    2024

  8. [8]

    Wang, X. et al. Transformer-based unsupervised contrastive learning for histopathological image classification. Med. image analysis 81, 102559 (2022)

    work page 2022

Show all 32 references
  1. [9]

    Y .et al

    Lu, M. Y .et al. A visual-language foundation model for computational pathology. Nat. Medicine 30, 863–874 (2024)

    2024

  2. [10]

    Kather, J. N. et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS medicine 16, e1002730 (2019)

    2019

  3. [11]

    Pataki, B. Á. et al. Huncrc: annotated pathological slides to enhance deep learning applications in colorectal cancer screening. Sci. Data 9, 370 (2022)

    2022

  4. [12]

    Barbano, C. A. et al. Unitopatho, a labeled histopathological dataset for colorectal polyps classification and adenoma dysplasia grading. In 2021 IEEE International Conference on Image Processing (ICIP) , 76–80 (IEEE, 2021)

    2021

  5. [13]

    Oquab, M. et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

    2023 arXiv

  6. [14]

    Ding, J. et al. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486 (2023)

    2023 arXiv

  7. [15]

    Radford, A. et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748–8763 (PMLR, 2021)

    2021

  8. [16]

    Yu, J. et al. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 (2022)

    2022 arXiv

  9. [17]

    & Welling, M

    Ilse, M., Tomczak, J. & Welling, M. Attention-based deep multiple instance learning. In International conference on machine learning, 2127–2136 (PMLR, 2018)

    2018

  10. [18]

    Shao, Z. et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Adv. neural information processing systems 34, 2136–2147 (2021)

    2021

  11. [19]

    Y .et al

    Lu, M. Y .et al. Data-efficient and weakly supervised computational pathology on whole-slide images. Nat. biomedical engineering 5, 555–570 (2021)

    2021

  12. [20]

    & Sun, J

    He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , 770–778 (2016)

    2016

  13. [21]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    2020 arXiv

  14. [22]

    Yan, R. et al. Shapley values-enabled progressive pseudo bag augmentation for whole-slide image classification. IEEE Transactions on Med. Imaging (2024)

    2024

  15. [23]

    Ouyang, M. et al. Mergeup-augmented semi-weakly supervised learning for wsi classification. arXiv preprint arXiv:2408.12825 (2024)

    2024 arXiv

  16. [24]

    Chu, H. et al. Retmil: Retentive multiple instance learning for histopathological whole slide image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention , 437–447 (Springer, 2024)

    2024

  17. [25]

    Qiehe, S. et al. Nciemil: Rethinking decoupled multiple instance learning framework for histopathological slide classifica- tion. In Medical Imaging with Deep Learning (2024)

    2024

  18. [26]

    & Chen, H

    Yang, S., Wang, Y . & Chen, H. Mambamil: Enhancing long sequence modeling with sequence reordering in computational pathology. In International Conference on Medical Image Computing and Computer-Assisted Intervention , 296–306 (Springer, 2024)

    2024

  19. [27]

    Ling, X. et al. Agent aggregator with mask denoise mechanism for histopathology whole slide image analysis. arXiv preprint arXiv:2409.11664 (2024)

    2024 arXiv

  20. [28]

    & Eliceiri, K

    Li, B., Li, Y . & Eliceiri, K. W. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 14318–14328 (2021)

    2021

  21. [29]

    Zhang, H. et al. Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 18802–18812 (2022). 5/14

    2022

  22. [30]

    Li, J. et al. Dynamic graph representation with knowledge-aware attention for histopathology whole slide image analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 11323–11332 (2024)

    2024

  23. [31]

    Chikontwe, P. et al. Fr-mil: Distribution re-calibration based multiple instance learning with transformer for whole slide image classification. IEEE Transactions on Med. Imaging 1–1, 10.1109/TMI.2024.3446716 (2024)

    2024

  24. [32]

    Deng, J. et al. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248–255 (Ieee, 2009). Acknowledgements This work was supported by the National Natural Science Foundation of China (NSFC) under Grant No. 82...

    2009

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.