{"id":"cc1929a7-1101-4fa4-b38d-3eba31a0d459","arxiv_id":"2412.14619","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Current benchmarking of topology-aware segmentation is distorted by connectivity choices, label artifacts, and entangled metrics, which together can invert method rankings.","lead":"This paper shows that common evaluation practices for topology-aware medical image segmentation are unreliable: the choice of pixel connectivity, small labeling errors, and the choice of evaluation metric can flip method rankings. It offers recommendations for fairer benchmarking and a simple measure of how sensitive a dataset is to connectivity choices.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ranking-inversion claim rests on single-seed training with 6 methods; Table 2 Spearman correlations are within noise.","rationale":"The reader's weakest_assumption focuses on manual connectivity and artifact-threshold choices. I agree those are not externally validated, but they are not the most load-bearing issue for the paper's central claim. The claim that benchmarking is unreliable depends on showing that connectivity choice changes rankings. That evidence is statistically fragile: only six methods, one training run per condition, and no confidence intervals. The reader's rationale did call for multiple seeds and confidence intervals, so there is partial agreement, but the primary concern is different. The proposed concrete test would settle whether the ranking inversion is real. If the bootstrap CI for the mean Spearman rho overlaps zero, the headline effect would not be supported. The verdict remains CONDITIONAL because the paper's recommendations are still reasonable, but the quantitative ranking-inversion evidence needs strengthening.","tokens_in":12632,"tokens_out":5054,"duration_ms":87887,"concrete_test":"Retrain all six methods on CREMI under both A and D connectivity with at least 5 random seeds, compute per-metric Spearman rho between A and D rankings for each seed, and bootstrap a 95% confidence interval for the mean rho. If the interval includes zero, the ranking-inversion claim is unsupported. Also recompute the average from Table 2 to verify the reported -0.63.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative evidence for the connectivity pitfall (Section 3.1, Table 2) compares rankings of 6 methods on CREMI under A and D connectivity, with each method trained once per connectivity setting. The reported Spearman correlations range from -0.37 to -0.85. With n=6 methods, the critical Spearman value for significance at alpha=0.05 (two-tailed) is about 0.886, so none of the individual coefficients is significant, and the average -0.63 could easily arise from training stochasticity. Without multiple seeds or confidence intervals, the observed 'inversion' cannot be distinguished from random fluctuation. Additionally, the Discussion's 'avg. Spearman's rho = -0.63' does not match the mean of Table 2's reported values (-0.59), suggesting an inconsistency in the headline statistic. The manual connectivity labels (A for DRIVE, D for CREMI/Roads) are asserted rather than validated, but the ranking-sensitivity claim would survive even if the 'favorable' label were debated; the lack of statistical grounding is the more load-bearing issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that current benchmarking practices for topology-aware medical image segmentation are unreliable because of three pitfalls: (1) inadequate connectivity choices that can distort method rankings, (2) topological artifacts in ground-truth labels that inflate measured topological error, and (3) inappropriate use of evaluation metrics that entangle volumetric and topological information or aggregate different topological dimensions. The authors empirically study these issues using the DRIVE, CREMI, Roads, and MSSEG2 datasets and six topology-aware segmentation methods. They report that switching between the two standard connectivity conventions (A and D) can produce negative correlations between method rankings across metrics, that removing small ground-truth components reduces Betti matching error by up to ~43% (Table 4, BM1A), and that distributional metrics such as VOI/ARE are not consistently correlated with topological metrics. They propose recommendations: dataset-specific connectivity choices with transparency, removal of semantic artifacts, and reporting of disaggregated Betti matching errors alongside volumetric metrics. They also introduce a connectivity susceptibility metric.","tokens_in":12813,"tokens_out":5425,"duration_ms":36622,"significance":"If the findings are robust, the paper addresses a real and timely problem: many topology-aware segmentation papers report rankings that may be artifacts of evaluation choices rather than genuine method quality. The work is constructive in proposing concrete reporting standards and a susceptibility metric, and it ships code and uses publicly available datasets. The main claims are empirical and falsifiable. However, the central ranking-inversion claim currently rests on a small number of methods trained with single seeds, which undermines the strength of the conclusion as presented. The paper is nevertheless a valuable contribution to benchmarking methodology, provided the statistical grounding is improved.","major_comments":[{"comment":"The ranking-inversion claim is supported only by single training runs per method per connectivity setting. With n=6 methods, none of the reported Spearman coefficients (range -0.37 to -0.85) reaches the critical value of approximately 0.886 at alpha=0.05 (two-tailed), and the average of -0.63 could plausibly arise from training stochasticity. Please provide multi-seed experiments with confidence intervals or explicitly moderate the conclusion that 'the ranking of the methods changes drastically' to a statement that rankings are unstable under connectivity changes.","section":"Section 3.1, Table 2"},{"comment":"The headline statistic 'avg. Spearman's ρ = -0.63' does not match the mean of the seven values in Table 2, which is approximately -0.59. This inconsistency undermines the reported summary statistic; please correct the value or clarify how the average is computed (e.g., whether the average is over metrics or over some other grouping).","section":"Discussion (unnumbered paragraph)"},{"comment":"The artifact-removal experiment uses a manually chosen threshold of components with fewer than 5 pixels, and the reported claim that artifacts can comprise up to 43% of measured topological error (BM1A relative change) is threshold-dependent. The paper does not provide a sensitivity analysis over thresholds for the metric changes, although Table 5 reports component counts for several thresholds. Please either add such an analysis or explicitly state that the 43% figure is an example under a specific threshold, and discuss how the conclusion depends on this choice.","section":"Section 3.2, Table 4"}],"minor_comments":[{"comment":"The caption for Table 4 repeats the phrase 'adapted ground truth' for both blocks; the second block should refer to the original ground truth.","section":"Section 3.2, Table 4"},{"comment":"The connectivity susceptibility values in Table 3 appear to be per-image averages (e.g., for DRIVE, the total foreground component difference of 18,718 divided by 40 images gives 467.95), but the text defines the metric for 'a label G' without specifying whether the reported values are per-image averages or totals. Please clarify the normalization.","section":"Section 3.1, Eq. (1) and Table 3"},{"comment":"The claim that 'only one paper reports their connectivity choice' for the DRIVE dataset would benefit from a citation to that specific paper to make the statement verifiable.","section":"Section 3.1"},{"comment":"In Figure 4, the panels illustrate 'unfavorable' and 'favorable' predictions, but the caption does not state which prediction the authors consider preferable in each case; adding explicit labels would improve interpretability.","section":"Section 3.3, Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important benchmarking problem and is likely to be of interest to the MICCAI community. The main concern is the lack of statistical validation for the ranking-inversion claim; if the authors can add multi-seed experiments or carefully temper the conclusion, the paper could be acceptable. The paper includes several self-citations from the same group, which is not unusual in a methods critique but should be monitored for fairness. The availability of code and the use of public datasets are strengths."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a benchmarking-critique paper, not a new method. Its real contribution is the synthesis—showing that connectivity choice alone can invert method rankings on standard datasets, proposing a connectivity-susceptibility score, and a concrete set of reporting recommendations. The three pitfalls are genuine and the paper demonstrates them with clear examples. That is worth reading.\n\nWhat the paper does well: the artifact experiment on DRIVE is convincing. Removing small components changes Betti matching error by ~43% relative, and the examples in Fig. 3 make the issue concrete. The recommendation to report Betti matching errors per dimension instead of aggregated Betti numbers is well-argued and follows the literature (Stucki et al.). The code is released, and the Limitations section is honest about not empirically covering 3D.\n\nThe soft spots are mainly statistical. The central ranking-inversion claim (Table 2) comes from a single training run per method per connectivity, with only 6 methods. With n=6, no individual Spearman coefficient reaches significance (critical rho ≈ 0.886 at α=0.05), so the average of -0.63 could plausibly be training noise. The paper should have run multiple seeds and reported confidence intervals, or at least acknowledged this. Also the Discussion quotes avg. Spearman -0.63 but the mean of the Table 2 values is -0.59—a small inconsistency that suggests sloppy bookkeeping.\n\nThe manual choices—which connectivity is 'semantically favorable' and the 5-pixel artifact removal threshold—are asserted from visual inspection, not independently validated. That matters, but the paper doesn't hinge entirely on those exact numbers; Table 1 alone shows the connectivity effect is enormous for DRIVE.\n\nOn balance, the qualitative message holds. This is not a fatal flaw, but the quantitative support is weaker than the abstract's 'profound impact' language implies. The paper is most useful for researchers who evaluate topology-aware segmentation methods, and for reviewers checking whether reported rankings are robust. It deserves a serious referee, and I would send it out, but the authors should be asked to add multi-seed evaluation, confidence intervals, and fix the average inconsistency.","headline":"A benchmarking critique with a real message, but the headline ranking-inversion claim is built on single-seed statistics that need more support.","tokens_in":13329,"tokens_out":3255,"would_cite":true,"duration_ms":21943,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that standard benchmarking of topology-aware image segmentation is unreliable: connectivity choice alone can invert method rankings, ground-truth artifacts can account for a large share of measured topological error, and…","keywords":["topology-aware segmentation","model evaluation","connectivity","Betti numbers","Betti matching error","ground truth artifacts","benchmarking pitfalls","medical image segmentation"],"falsifier":"Compute the A-versus-D Spearman rank correlation for a new, topologically rich segmentation dataset using the authors' protocol; if the correlation is systematically positive or near zero, the claim that connectivity choice inverts rankings does not generalize. More directly, re-run the DRIVE artifact experiment with a 10-pixel or 3-pixel removal threshold and check whether the 43% reduction in BM1 error persists or disappears, which would test whether the effect is an artifact of the threshold.","tokens_in":12435,"feed_emoji":"🧠","tokens_out":4139,"duration_ms":27883,"temperature":0.7,"pith_summary":"This paper argues that the standard way of benchmarking topology-aware image segmentation methods is broken. It identifies three practice-level pitfalls: choosing the wrong voxel connectivity for a dataset, leaving topological artifacts in ground-truth labels, and using metrics such as VOI or aggregated Betti scores that mix volume and topology. On three widely used benchmarks (DRIVE, CREMI, Roads), the authors show that switching between the two common connectivity conventions alone flips method rankings, with an average Spearman correlation of -0.63, and that cleaning small label artifacts cuts measured topological error by up to 43%. The paper concludes with concrete reporting and evaluation guidelines designed to make future comparisons trustworthy.","feed_headline":"Connectivity choice flips segmentation method rankings","feed_subtitle":"A single evaluation choice flips rankings (Spearman -0.63); artifact cleanup cuts measured error up to 43%.","key_machinery":"The analysis is built on two formal tools. The first is the pair of connectivity conventions A and D, which correspond to opposite foreground/background adjacency choices and, in cubical complex terms, to T- versus V-construction; the paper proposes a simple susceptibility score, e.g., |β0(PD) − β0(PA)|, to quantify how much a dataset's topological metrics move when connectivity changes. The second is the comparison of before/after artifact removal on DRIVE, using a 5-pixel component-size threshold, to isolate the contribution of label noise and connectivity artifacts to Betti number and Betti matching errors.","core_discovery":"The central claim is that reported rankings of topology-aware segmentation methods are largely artifacts of arbitrary evaluation choices. By training and evaluating six methods on the CREMI dataset under both the A connectivity (all-connectivity foreground, direct background) and the D connectivity (the reverse), the authors find that ranks invert on nearly every metric, including negative Spearman correlations on Betti errors. On DRIVE, removing ground-truth components up to 5 pixels, which visually correspond to label noise and connectivity artifacts, reduces the Betti matching 1 error by about 43% and Betti number 1 error by about 29%, while leaving Dice and VOI nearly unchanged. The paper also shows that distributional metrics (VOI, ARE/ARI) entangle volumetric and topological errors, and that aggregating Betti errors across dimensions hides the metric that matters for the downstream task, e.g., BM0 for vessels and BM1 for neurons.","pith_inferences":["The susceptibility measure generalizes: any 2D or 3D segmentation benchmark could be screened with it, and datasets with high susceptibility should be reported with multiple connectivity choices until a semantic consensus is reached.","The finding that connectivity can invert rankings suggests that some published method comparisons in the topology-aware segmentation literature may need to be re-run under standardized connectivity before their conclusions can be considered stable.","The artifact-removal result implies that weakly supervised or iterative label-cleaning steps could be integrated into evaluation pipelines, though the 5-pixel threshold would need task-specific justification.","The paper's recommendation to use Betti matching errors over Betti numbers may itself depend on the matching algorithm's sensitivity to connectivity, which the paper does not fully dissect."],"forward_implications":["Method papers should report which connectivity they use, e.g., β0A versus β0D, and choose the semantics-preserving one per dataset.","Rankings in existing papers that omit connectivity reporting may not transfer to downstream applications.","Topological metrics should be reported without dimension aggregation, and always paired with a purely volumetric metric.","Datasets should be screened for small background components before benchmarking, except where such components carry semantic meaning, as in CREMI neuron beginnings.","The connectivity susceptibility score can be computed cheaply for any dataset before it is adopted as a benchmark."],"supporting_citations":[{"why":"Provides the DRIVE retinal vessel dataset used for connectivity and artifact experiments.","marker":"[41]"},{"why":"Provides the CREMI neuron segmentation dataset used in the connectivity ranking experiment.","marker":"[15]"},{"why":"Provides the Roads satellite image dataset used for connectivity and resolution artifact analysis.","marker":"[34]"},{"why":"Introduces the Betti matching error metric that the paper tests against Betti number errors.","marker":"[42]"},{"why":"Introduces the Betti number error and serves as a representative topology-aware method in the experiments.","marker":"[20]"},{"why":"Describes the relationship between T/V cubical complex constructions and connectivity, grounding the formalism.","marker":"[6]"},{"why":"Defines the variation of information metric that the paper criticizes for entangling volume and topology.","marker":"[32]"},{"why":"Supports the claim that distributional metrics irreversibly entangle topological and volumetric errors.","marker":"[14]"}],"fun_headline_variants":["Connectivity choice flips topology segmentation rankings","Artifact cleanup cuts topology error up to 43%","Rankings of topology-aware segmentation are arbitrary","Benchmark pitfalls invert segmentation method ranks","Topology evaluation flaws skew method rankings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative conclusions rest on the authors' manual choice, guided by visual inspection, of which connectivity is semantically correct for each dataset (A for DRIVE, D for CREMI and Roads) and of the 5-pixel component-size cutoff for treating ground-truth objects as artifacts; a different reasonable choice could change the magnitudes and possibly the ranking flips.","fun_headline_variants_meta":{"raw":{"variants":["Connectivity choice flips topology segmentation rankings","Artifact cleanup cuts topology error up to 43%","Rankings of topology-aware segmentation are arbitrary","Benchmark pitfalls invert segmentation method ranks","Topology evaluation flaws skew method rankings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1264,"prompt_tokens":831,"completion_tokens":433,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":366}},"tokens_in":447,"tokens_out":433,"duration_ms":3836,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:03:39.041447+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the A-versus-D Spearman rank correlation for a new, topologically rich segmentation dataset using the authors' protocol; if the correlation is systematically positive or near zero, the claim that connectivity choice inverts rankings does not generalize. More directly, re-run the DRIVE artifact experiment with a 10-pixel or 3-pixel removal threshold and check whether the 43% reduction in BM1 error persists or disappears, which would test whether the effect is an artifact of the threshold.","supporting_citations":[{"cited_title":"IEEE Transactions on Medical Imaging 23(4), 501–509 (2004)","cited_arxiv_id":null,"evidence_quote":"Provides the DRIVE retinal vessel dataset used for connectivity and artifact experiments."},{"cited_title":"IEEE transactions on pattern analysis and machine intelligence 41(7), 1669–1680 (2018)","cited_arxiv_id":null,"evidence_quote":"Provides the CREMI neuron segmentation dataset used in the connectivity ranking experiment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Roads satellite image dataset used for connectivity and resolution artifact analysis."},{"cited_title":"In: International Conference on Machine Learning","cited_arxiv_id":null,"evidence_quote":"Introduces the Betti matching error metric that the paper tests against Betti number errors."},{"cited_title":"Advances in neural information processing systems32 (2019)","cited_arxiv_id":null,"evidence_quote":"Introduces the Betti number error and serves as a representative topology-aware method in the experiments."},{"cited_title":"In: Research in Computational Topology 2, pp","cited_arxiv_id":null,"evidence_quote":"Describes the relationship between T/V cubical complex constructions and connectivity, grounding the formalism."},{"cited_title":"Journal of multivariate analysis98(5), 873–895 (2007)","cited_arxiv_id":null,"evidence_quote":"Defines the variation of information metric that the paper criticizes for entangling volume and topology."},{"cited_title":"Methods115, 119–127 (2017)","cited_arxiv_id":null,"evidence_quote":"Supports the claim that distributional metrics irreversibly entangle topological and volumetric errors."}],"review_version":1}