{"id":"b8b25b73-aa58-496b-bb00-3ca36173d00c","arxiv_id":"2507.04681","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A challenge summary showing top-performing deep learning methods improve colorectal cancer tumor grade segmentation on the METU CCTGS dataset, with the winner reaching 70.2 macro F-score.","lead":"This paper reports the results of a challenge on colorectal cancer tumor grading and segmentation, where 39 teams competed to segment whole-slide images into five tissue classes. The best method achieved a macro F-score of 70.2, beating the previous 62.9 baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 70.2 vs 62.9 F-score gap lacks statistical grounding: it rests on ~15 test WSIs with no confidence intervals, and the paper's own acknowledged annotation-style inconsistency could bias the comparison.","rationale":"Good-faith reading: this is a challenge overview whose main empirical assertion is that the winning entry reached 70.2 macro F-score, beating the published Swin baseline of 62.9, and that whole-slide image use plus ensembling contributed to the gain. The paper is honest: it gives per-method training and inference details, names the annotation-style problem, and says Docker-based verification was performed, though artifacts are not linked. I see no internal contradiction in the method descriptions. The load-bearing spot is not the architecture but the evaluation: the entire comparison hinges on one small test split and a single scalar per method, in a setting where the authors themselves say ground-truth style varies. With about 15 test slides, the difference between 70.2 and 62.9 is within the range that could be produced by a few difficult slides or by style imbalance. A per-slide bootstrap and style-stratified reanalysis would settle this directly. If the difference survives, the paper's contribution is real; if not, the result should be framed as provisional. This matches, and slightly extends, the reader's concern, so the conditional verdict stands.","tokens_in":7838,"tokens_out":4829,"duration_ms":53398,"concrete_test":"Request from the organizers the per-slide, per-class F-scores for VAN+UperNet and the Swin baseline on all test WSIs, together with slide-level labels for annotation style (smoothed versus detailed). Then: (1) compute bootstrap 95% confidence intervals for the macro-F difference and run a paired permutation test across slides; (2) recompute the gap separately within each style stratum. If the 7.3-point margin persists in both strata and the bootstrap interval excludes zero, the central claim is supported. If the gap collapses in one style stratum or the interval includes zero, the reported superiority is not established and should be presented as provisional.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 reports a single macro-F number per method with no per-slide variance, no confidence intervals, and no significance test. The test split is 15% of 103 WSIs (about 15 slides), so macro-F over five classes can shift by several points with one or two atypical slides. The paper itself (Section 3) documents two visibly different annotation styles in the masks, smoothed boundaries versus highly detailed edges, and states that this complicates model evaluation because there is no clear guidance on which annotation domain should be prioritized. If the hidden test set is dominated by one style, the apparent ordering of methods could reflect how well each model matches that style rather than grading skill. The load-bearing premise is therefore that the test-set measurement is a stable, style-fair estimate of model quality, and the paper supplies no evidence for that premise. Additionally, the inference that whole-slide images are responsible for the top two teams' lead is confounded by their simultaneous use of ensembles, though that does not invalidate the headline score. The central claim could be true, but the reported 7.3-point advantage is not yet established as statistically or annotation-robust.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports on the ICIP Grand Challenge on Colorectal Cancer Tumor Grading and Segmentation, using the METU CCTGS dataset of 103 whole-slide images with five-class expert pixel-level annotations. The authors describe the challenge protocol, the top six participating methods, and their evaluation results. The headline result is that the winning method (VAN+UperNet) achieves a macro F-score of 70.2, outperforming the Swin Transformer baseline of 62.92. The paper also observes that the top two teams used full-resolution WSIs and ensembles and that ensembling generally helps. The dataset and evaluation infrastructure are made publicly available, and Docker-based reproducibility checks were performed.","tokens_in":8207,"tokens_out":4888,"duration_ms":51238,"significance":"If the reported results are reliable, the paper provides a useful public benchmark for tumor grade segmentation, with a well-defined challenge protocol, a moderately large multi-class histopathology dataset, and reproducible Docker submissions from six methods. The explicit release of the dataset and evaluation infrastructure is a valuable community contribution. However, the headline performance claims currently rest on a small hidden test set (about 15 WSIs) with no uncertainty quantification, and the paper itself acknowledges an annotation-style inconsistency that could bias comparisons. The significance of the benchmark would be substantially strengthened by addressing these statistical and annotation-robustness issues.","major_comments":[{"comment":"The headline claim that the winning method outperforms the Swin Transformer baseline by 7.3 macro-F points rests on a single aggregate score per method. The test split is 15% of 103 WSIs (approximately 15 slides), and macro-F over five classes is highly sensitive to one or two atypical slides. The paper should report per-slide macro-F scores, their standard deviation or confidence intervals, and a paired significance test (e.g., Wilcoxon signed-rank) comparing the top methods against the baseline. Without this, the reported advantage is not distinguished from sampling variability.","section":"Section 3, Table 1"},{"comment":"The paper explicitly acknowledges that the training annotations exhibit two distinct styles — one with smoothed boundaries and one with highly detailed edges — and states that this complicates model evaluation. However, it does not check whether the stratified random split balances these styles across training, validation, and test sets. If the hidden test set is dominated by one annotation style, the ranking of methods could reflect style-specific bias rather than grading skill. The authors should quantify the annotation-style distribution across splits and, if imbalanced, evaluate robustness (e.g., per-style scores or agreement with a re-annotated subset).","section":"Section 3"},{"comment":"The statement that using full-resolution WSIs provides a 'clear benefit' is confounded with ensembling: the top two teams used both WSIs and ensembles, while the third team used ensembling without WSIs. Without an ablation that varies resolution while holding the model and ensemble strategy fixed (e.g., running the winning pipeline on downsampled images), the observed margin cannot be attributed to WSI resolution. The paper should either provide such an ablation or present this observation as a hypothesis rather than a conclusion.","section":"Section 3"},{"comment":"The baseline Swin Transformer score (62.92) is cited from the METU CCTGS dataset paper [11], but the present paper does not state whether this baseline was trained on the same 70/15/15 split and evaluated on the same hidden test set used in the challenge. If the baseline was produced under a different data partition or preprocessing, the comparison in Table 1 is not apples-to-apples. Please specify the baseline's training and evaluation protocol, or rerun the baseline under the challenge protocol to confirm the comparison.","section":"Section 1 and Table 1"}],"minor_comments":[{"comment":"The hyperparameters of the Top-N Soft Biased Voting method (N, M, and the class-specific bias vector) and the area threshold epsilon in the postprocessing pipeline are not specified; please report their values and how they were chosen, to ensure reproducibility.","section":"Section 2.2"},{"comment":"Reporting only macro-averaged metrics hides class-level performance; for a tumor grading task, per-grade F-scores (Grade 1/2/3, normal, others) are essential for assessing clinically relevant confusion patterns and should be included.","section":"Table 1"},{"comment":"The stratification is described as 'pixel-level' class balance; please clarify whether this is computed across all pixels in the dataset or per-slide, and report the class distribution for each split.","section":"Section 1"},{"comment":"The method names 'V AN+UperNet' and 'DPT+MaxViT' contain typographic spaces; please use consistent formatting throughout the paper.","section":"Sections 2.1 and 2.2"},{"comment":"The qualitative analysis of Figure 2 would benefit from a quantitative confusion-matrix analysis, since subjective claims such as 'VAN+UperNet has high precision' are not directly traceable to Table 1.","section":"Section 3"},{"comment":"The paper does not state how many Codalab submissions were allowed or whether participants could see test-set metrics during the development phase; if repeated submissions were permitted, the risk of test-set overfitting should be discussed.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The author list includes both the challenge organizers and members of the two winning teams, and the baseline is taken from the organizers' own prior paper; a conflict-of-interest statement would be appropriate. Additionally, the challenge's Codalab infrastructure returned test metrics during development, so the number of allowed submissions and any measures against test-set overfitting should be disclosed to assess the validity of the rankings."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, this is a straightforward challenge report and should be read as one. The new thing is an independent, externally scored benchmark on METU CCTGS, with six teams beating the published Swin baseline (62.9) and a new best macro F-score of 70.2 from a VAN+UperNet ensemble. That is a real data point for a clinical task where public benchmarks are scarce. The paper is also honest: it flags the two annotation styles in the training masks and admits the evaluation is complicated by that.\n\nWhere I agree with the skeptic: the 7.3-point gap is not statistically grounded. The test set is about 15 whole-slide images, no per-slide variance or confidence intervals are reported, and a couple of atypical slides could move macro-F by several points. The acknowledged annotation-style inconsistency is a real confounder. If the hidden test set is dominated by one style, the method ranking could partly reflect style matching. I also think the claim that whole-slide images drive the top two teams' lead is confounded by their simultaneous use of ensembles, though the authors present it only as a plausible observation.\n\nThat said, the soft spots are typical of medical imaging challenges and mostly stem from the format, not carelessness. The paper describes six methods in enough detail to be reproduced, and the organizers say they verified Docker submissions. The baseline comes from the dataset paper, which is normal. The involvement of the top-2 teams' authors in the organizing group is a mild conflict, but the public leaderboard and hidden test set limit the damage.\n\nWould I accept this for peer review? Yes. It is exactly the kind of artifact a community needs: a documented public benchmark with reproducible evaluation. The main revision I would request is a per-slide breakdown and some estimate of variance, plus a more explicit analysis of annotation-style distribution across splits. The paper is not a methodological breakthrough and does not claim to be.\n\nFor a reading group, I'd only bring it if someone is working on histopathology benchmarks. Otherwise it is a solid but narrow empirical report. I'd cite it as the current state of the art on METU CCTGS.","headline":"Useful benchmark report with a believable new best score, but the headline gain lacks statistical support and the annotation-style confounder is real.","tokens_in":8668,"tokens_out":2012,"would_cite":true,"duration_ms":21850,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports that an open challenge on the CCTGS colorectal cancer dataset raised the best automated tumor grade segmentation macro F-score from 62.92 to 70.2, with the gain driven mainly by full-resolution whole-slide input and…","keywords":["colorectal cancer","tumor grade segmentation","whole-slide images","digital histopathology","semantic segmentation challenge","model ensembling","macro F-score"],"falsifier":"Have a second expert pathologist independently re-annotate the held-out test whole-slide images for the five tissue classes, then rescore the winning and baseline masks against those fresh labels; if the macro F-score gap between the winner and the Swin Transformer baseline shrinks or reverses, the reported improvement is substantially an artifact of annotation style rather than true segmentation quality.","tokens_in":7686,"feed_emoji":"🔬","tokens_out":7894,"duration_ms":78523,"temperature":0.7,"pith_summary":"The paper reports the results of an open challenge in which 39 teams built algorithms to segment colorectal cancer tumor grades directly from whole-slide histopathology images. The task matters because tumor grade guides prognosis and treatment, yet routine grading is subjective and pathologists are scarce. The winning pipeline, a Visual Attention Network paired with an UperNet decoder, reached a macro F-score of 70.2 on the held-out test set, clearly above the previous best of 62.92 recorded with a Swin Transformer baseline. The challenge also found that the two submissions using original high-resolution whole-slide images outperformed every method that trained on downsampled images, and that voting ensembles beat single models. These results establish a new reference point for automated colorectal cancer grading on this benchmark.","feed_headline":"Winner beats baseline 62.9 to 70.2 on colon tumor grading","feed_subtitle":"Full-resolution images and model voting lift colorectal cancer grade segmentation past the prior best.","key_machinery":"The load-bearing mechanism is the challenge evaluation itself: 103 whole-slide images split into training, validation, and test sets, hidden pixel-level test labels, and ranking by macro F-score across five classes, with mandatory containerized code submission so results are reproducible. Inside the winning method, the operative components are an active cropping scheme that samples 512-by-512 patches in proportion to class-weighted pixel content; a three-fold ensemble of Visual Attention Network backbones with UperNet decoders trained with Dice plus cross-entropy loss; overlapping 1024-by-1024 inference patches weighted by a Gaussian kernel and averaged across the three models; and a post-processing step that redraws external contours to fill holes and repair fragmented shapes.","core_discovery":"On the paper's own terms, the central claim is that the CCTGS benchmark has a new best result: 70.2 macro F-score, achieved by averaging three Visual Attention Network plus UperNet models trained on full-resolution whole-slide images with class-weighted active cropping. Six independently submitted methods exceeded the previous Swin Transformer baseline of 62.92, and the two highest-scoring methods were the only ones to use the original whole-slide files, finishing at least three points ahead of every downsampled-only method. The paper interprets this as evidence that high-resolution morphological detail and multi-model voting are the main levers for tumor grade segmentation accuracy, while noting that distinguishing Grade 1 from Grade 2 remains the dominant residual error.","pith_inferences":["Editorial inference: the reported advantage of full-resolution whole-slide images is observational rather than controlled, because the teams that chose to process the large files also invested more compute and engineering; the causal contribution of image resolution alone is not separated in the challenge data.","Editorial inference: the paper's observation of two annotation styles, smoothed boundaries versus highly detailed edges, implies that a future re-annotation with consensus labels could change absolute scores and possibly the ranking, not just the top-line number.","Editorial inference: because the hardest residual errors are between adjacent tumor grades, an architecture or loss function that treats grades as ordered, rather than as independent classes, is a concrete and testable next step."],"forward_implications":["Using original whole-slide images instead of downsampled copies is worth at least three macro F-score points on this benchmark, since both of the top two methods used the original files and outscored every downsampled-only method.","Ensembling by voting, whether hard majority voting on masks or soft averaging of probability maps, consistently improves test performance over any single model.","The residual error pattern is concentrated in Grade 1 versus Grade 2 confusion, while Grade 3 detection and normal-mucosa boundary fidelity are comparatively strong in the winning method.","The public dataset and evaluation server leave a fixed benchmark that future tumor grade segmentation methods can be measured against directly."],"supporting_citations":[{"why":"Supplies the CCTGS dataset, the stratified split, and the Swin Transformer baseline with 62.92 macro F-score that every submission is compared against.","marker":"[11]"},{"why":"Defines the UperNet architecture used as the segmentation head of the winning method.","marker":"[12]"},{"why":"Defines the Visual Attention Network (VAN) backbone used by the winning method.","marker":"[13]"},{"why":"Provides the Dense Prediction Transformer architecture used by the second-place ensemble.","marker":"[15]"},{"why":"Provides the MaxViT encoder used by the second-place ensemble.","marker":"[16]"},{"why":"Defines the HarDNet-DFUS segmentation model used by the third-place method.","marker":"[19]"},{"why":"Defines the Lawin transformer decoder used by the third-place method.","marker":"[21]"},{"why":"Supplies the frozen foundation-model feature extractor used in the PathVTA submission.","marker":"[26]"},{"why":"Supplies the adapter module through which PathVTA builds a multi-scale feature pyramid.","marker":"[27]"},{"why":"Provides the off-the-shelf pretrained implementations used by the Segmenter-L and SegFormer-B1 fine-tuning baselines.","marker":"[25]"}],"fun_headline_variants":["Colon tumor grading hits 70.2 F-score, up from 62.9 baseline","Full-res images and model voting lift colon cancer grading","New best on colorectal cancer grading: 70.2 macro F-score","High-res whole slides key to colorectal tumor grading win"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking depends on the hidden test labels being accurate and annotated in a style consistent with the training labels; the paper itself reports that training masks split into smoothed-boundary and detailed-edge styles, so if the same style gap reaches the test set, the point differences between methods could reflect annotation style rather than segmentation skill.","fun_headline_variants_meta":{"raw":{"variants":["Colon tumor grading hits 70.2 F-score, up from 62.9 baseline","Full-res images and model voting lift colon cancer grading","New best on colorectal cancer grading: 70.2 macro F-score","High-res whole slides key to colorectal tumor grading win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1325,"prompt_tokens":833,"completion_tokens":492,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":416}},"tokens_in":449,"tokens_out":492,"duration_ms":5895,"temperature":1.0,"reasoning_tokens":416,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:41:31.044363+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a second expert pathologist independently re-annotate the held-out test whole-slide images for the five tissue classes, then rescore the winning and baseline masks against those fresh labels; if the macro F-score gap between the winner and the Swin Transformer baseline shrinks or reverses, the reported improvement is substantially an artifact of annotation style rather than true segmentation quality.","supporting_citations":[{"cited_title":"The eighth edition ajcc cancer staging manual: continuing to build a bridge from a population-based to a more “personalized","cited_arxiv_id":null,"evidence_quote":"Supplies the CCTGS dataset, the stratified split, and the Swin Transformer baseline with 62.92 macro F-score that every submission is compared against."},{"cited_title":"Artificial intelligence for solid tu- mour diagnosis in digital pathology,","cited_arxiv_id":null,"evidence_quote":"Defines the UperNet architecture used as the segmentation head of the winning method."},{"cited_title":"Observer variation in the histolog- ical grading of rectal carcinoma.,","cited_arxiv_id":null,"evidence_quote":"Defines the Visual Attention Network (VAN) backbone used by the winning method."},{"cited_title":"Colorectal cancer tumor grade segmentation: A new dataset and baseline results,","cited_arxiv_id":null,"evidence_quote":"Provides the Dense Prediction Transformer architecture used by the second-place ensemble."},{"cited_title":"Unified perceptual parsing for scene un- derstanding,","cited_arxiv_id":null,"evidence_quote":"Provides the MaxViT encoder used by the second-place ensemble."},{"cited_title":"Towards robust monoc- ular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,","cited_arxiv_id":null,"evidence_quote":"Defines the HarDNet-DFUS segmentation model used by the third-place method."},{"cited_title":"Adaptive Augmentation Policy Optimization with LLM Feedback","cited_arxiv_id":"2410.13453","evidence_quote":"Defines the Lawin transformer decoder used by the third-place method."},{"cited_title":"100,000 histological images of human colorec- tal cancer and healthy tissue,","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen foundation-model feature extractor used in the PathVTA submission."},{"cited_title":"Nct-crc-he: Not all histopathological datasets are equally useful,","cited_arxiv_id":null,"evidence_quote":"Supplies the adapter module through which PathVTA builds a multi-scale feature pyramid."}],"review_version":1}