{"id":"1f627b04-1060-4884-b5b8-9333a0f9020a","arxiv_id":"2501.09101","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A U-Net variant with paired inputs outputs relation maps, and the Dice inconsistency between predicted union and intersection maps serves as an unsupervised confidence score for segmentation.","lead":"A segmentation network called Relation U-Net takes two images and outputs two segmentations plus union and intersection maps; the disagreement between the union and intersection maps becomes a per-image confidence score without needing ground truth. The paper reports on four medical datasets that this score ranks hard and easy test images, and that the multi-input model slightly improves Dice scores over vanilla U-Net.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The confidence score rests on an untrained reflexivity assumption for identical inputs; the paper does not rule out head bias, seed variance, or object-size confounding as the source of the correlation.","rationale":"I read the paper in good faith: the architecture is simple, the four datasets are public, and the idea of using relation-head disagreement as a confidence score is plausible and related to ensemble or MC-dropout uncertainty. The reader's verdict of CONDITIONAL is appropriate. My stress test agrees with the reader's weakest assumption: the entire confidence mechanism depends on the learned relation heads behaving reflexively on identical inputs even though they were never trained on identical pairs, and on the residual disagreement being a difficulty signal rather than noise or a size proxy. The manuscript does not supply the missing evidence: no numerical Pearson coefficients with confidence intervals, no partial correlations controlling for object size or contrast, no multi-seed reproducibility, and no direct evaluation of rp(x,x) and rc(x,x) against ground truth. I would not reject the paper; the empirical correlation may survive these checks. But the central claim is conditional on them, so the reader's CONDITIONAL verdict should stand unchanged. If the checks pass, I would accept; if the partial correlation vanishes, the confidence score is not measuring what the paper claims.","tokens_in":7099,"tokens_out":10268,"duration_ms":109648,"concrete_test":"On one held-out fold, compute for each test image: C=Dice(rp(x,x),rc(x,x)), the model segmentation Dice against ground truth, object size (voxel count), and boundary contrast. Then (a) regress Dice on log size and contrast and compute the partial correlation of C with the residual Dice; (b) retrain the same Relation U-Net with five different random seeds and compute the Spearman rank correlation of C across seeds. If the partial correlation is near zero, or if C rankings are not stable across seeds, the confidence score is not specifically measuring segmentation difficulty. Report the numerical Pearson values with 95% confidence intervals from the five-fold CV as well.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that for x1=x2, C=Dice(rp,rc) is a per-image difficulty score. Training always samples two distinct images (Section 3.2), so the x1=x2 diagonal is off-manifold. Section 2.3 says the relations are reflexive by definition, but that reflexivity holds for the ground-truth labels, not automatically for the learned subnetworks: the four output heads share an encoder but have independent classifiers and no loss term enforcing rp(x,x)=rc(x,x)=s(x). The residual discrepancy used as confidence could therefore be driven by (i) random head initialization, (ii) systematic bias between a union-trained head and an intersection-trained head, or (iii) object size or boundary length rather than segmentation difficulty. The reported Pearson coefficients in Fig. 4(a) are aggregate, lack confidence intervals, and are not accompanied by partial correlations or seed-to-seed reproducibility, so they cannot distinguish these alternatives. This is the load-bearing link between the network outputs and the claimed confidence score; if it fails, the central contribution reduces to a proxy correlation with image statistics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 'Relation U-Net', a U-Net variant with two input images and four output heads that predict the two input segmentation maps plus their Possible (union) and Consensus (intersection) relation maps. At test time, the same image is fed twice, and the Dice coefficient between the predicted union and intersection maps is used as a per-image confidence score. The method is evaluated with five-fold cross-validation on LiTS, Hippocampus, BraTS, and ISIC. The paper claims that Relation U-Net improves Dice over vanilla U-Net and that the estimated confidence score is linearly correlated with segmentation accuracy on test images. The central claim is that the discrepancy between the two relation maps ranks test images by difficulty without ground truth.","tokens_in":7332,"tokens_out":5583,"duration_ms":54944,"significance":"If established, the per-image confidence score would be practically useful for selective human review in clinical segmentation workflows. The method is simple, uses four public datasets with a standard five-fold protocol, and includes an MC Dropout baseline, which are clear strengths. The qualitative examples in Fig. 5 also support the intuitive plausibility of the approach. However, the load-bearing link between the learned relation heads and the claimed confidence score is not adequately verified, and the reported correlations are only graphical. The accuracy gains over the vanilla U-Net are also small and inconsistent. With additional experiments and proper statistical reporting, the contribution would be a useful lightweight extension; in its current form, the central claim is not fully supported.","major_comments":[{"comment":"The confidence score rests on an unverified reflexivity assumption. The ground-truth relations r_p = s1 v s2 and r_c = s1 ^ s2 are reflexive by definition, but the trained heads are learned only from pairs of independently sampled distinct training images, and no loss term or training pair enforces that r_p(x,x) = r_c(x,x) = s(x). Feeding the same image twice is therefore an off-manifold input. C = Dice(r_p, r_c) can be dominated by systematic bias between the union-trained and intersection-trained heads, by random initialization, or by object-size and boundary-length effects, rather than by segmentation difficulty. To support the central claim, the authors should (i) report the distribution of C and of its correlation with Dice across multiple training seeds, (ii) compare the relation heads on identical inputs against a fixed 'head-bias' baseline, for example by adding a reflexivity-consistency loss or including identical pairs in training, and (iii) report partial correlations or stratified analyses controlling for object size and contrast. Without this, the confidence-score equation in Section 2.3 is not established.","section":"Section 2.3 and Section 3.2"},{"comment":"The Pearson correlation coefficients are presented only graphically, without numerical values, confidence intervals, or p-values, and no rank correlation is reported. Since the intended use is to rank test images for review, the appropriate evidence includes Spearman or Kendall coefficients, per-fold and per-seed intervals, and the distribution of differences against the MC Dropout baseline. Fig. 4(b) would also benefit from error bars or shaded confidence bands over the five folds. The current reporting makes it impossible to assess whether the claimed linear correlation is statistically reliable.","section":"Section 3.4 and Fig. 4(a)"},{"comment":"The claimed accuracy advantage over vanilla U-Net is not consistent or statistically supported. In Table 1, the Possible relation r_p is worse than U-Net on ISIC (86.08 vs 87.89) and only marginally better on LiTS and Hippocampus. In Table 2, the x1 = x2 condition is worse than U-Net on Hippocampus (82.59 vs 82.76) and BraTS (82.27 vs 82.63). All reported differences are within one standard deviation of the five-fold means. Formal paired tests across folds, or a larger number of training runs, are needed before claiming 'better accuracy than vanilla U-Net' in the abstract.","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"The formula C = 2|r_p ^ r_c|/(r_p + r_c) is informal for binary maps; please define whether |.| denotes the number of positive pixels and clarify the denominator notation.","section":"Section 2.3"},{"comment":"Notation alternates among r_c, sc, and r_p; please use one consistent symbol for the Consensus relation map in text, equations, and tables.","section":"Section 2.3 and Table 1"},{"comment":"Reference [6] lists the dataset as 'LIST' and the first author as 'Patrick ilic'; the correct names are 'LiTS' and 'Patrick Bilic'.","section":"References"},{"comment":"The choice of 200 anchor images is not justified; please report sensitivity to n or explain why 200 forward passes are sufficient for a stable estimate.","section":"Section 3.3.2 and Table 2"},{"comment":"The caption does not clearly identify which curves correspond to which methods, and the grayscale rendering makes the comparison difficult to read in print.","section":"Fig. 4(a)"}],"recommendation":"major_revision","confidential_remarks":"I agree with the reader's take that there is no circularity in the confidence-score evaluation, but the diagonal reflexivity premise is a genuine gap that is load-bearing for the central claim. The missing numerical correlations and the inconsistent Dice gains make this a major revision rather than an acceptance. The proposed additional experiments are feasible within the manuscript's scope, so I would not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: the confidence-score idea is real and useful, but the evidence as presented is not yet tight enough to fully back it.\n\nWhat's new: using the discrepancy between predicted union and intersection maps from a multi-output U-Net as a per-image confidence score, when the same image is fed twice. The relation maps themselves come from the pairwise learning literature (Wang et al.), and the MIMO structure is borrowed, but the test-time reflexivity trick is new, as far as I know. The paper runs four public datasets with five-fold CV and compares against an MC-dropout baseline. That's a reasonable first pass.\n\nWhat it does well: the method is simple and cheap—one forward pass at test time gives both a segmentation and a ranking signal. Figure 5 shows the ranking visually tracks perceived difficulty. If the correlation holds, that's practically valuable for triage.\n\nWhere it's soft: the reported accuracy gains are small and inconsistent (ISIC actually drops on rp). The Pearson correlations in Fig 4(a) are only shown as bars; no numbers, no confidence intervals, no significance tests. The central theoretical justification—reflexivity of the learned relations at x1=x2—is asserted but never trained or enforced. The two relation heads share an encoder but have independent classifiers, so their discrepancy on identical input could reflect systematic head bias or random seed variation rather than segmentation difficulty. The paper doesn't check whether the confidence score is just tracking object size or boundary length. These are not fatal objections, but they are exactly the kind of thing a reviewer would want ruled out before trusting the ranking.\n\nThe stress-test note is fair. It puts the finger on the load-bearing assumption. Still, the empirical correlation across four datasets is suggestive, and the intended use case—flagging hard cases for human review—doesn't require the mechanism to be perfectly clean; the ranking just needs to be useful. So I'd send this to review, but with major-revision expectations: report numeric correlations with CIs, add a significance test, test reflexivity directly by training on some identical pairs (or enforcing it), and check confounders like object size. Code would help.\n\nBottom line: an incremental but practically motivated idea, with enough empirical signal to deserve referee time, and clear gaps to fix.","headline":"A useful per-image confidence signal for segmentation, but the evidence needs statistical rigor and a direct test of the untrained reflexivity assumption.","tokens_in":7792,"tokens_out":2400,"would_cite":false,"duration_ms":26768,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Relation U-Net turns the gap between union and intersection predictions into a per-image confidence score that tracks segmentation accuracy without ground truth.","keywords":["Relation U-Net","medical image segmentation","confidence score estimation","deep relation learning","uncertainty quantification","segmentation without ground truth","U-Net"],"falsifier":"Train Relation U-Net twice with different random seeds. If the confidence scores $C$ for the same test images have low rank correlation between the two training runs while the segmentation Dice stays similar, then $C$ is dominated by subnetwork variability rather than image difficulty; alternatively, on a test set of uniformly easy, high-quality images, a large spread in $C$ would show the same thing.","tokens_in":6916,"feed_emoji":"🩺","tokens_out":6904,"duration_ms":61837,"temperature":0.7,"pith_summary":"Relation U-Net generalizes U-Net to take two input images and output two segmentation maps plus the 'Possible' (union) and 'Consensus' (intersection) relation maps between them. The paper's central claim is that when the same test image is fed twice, the disagreement between the estimated union and intersection maps is a per-image confidence score that ranks segmentation difficulty without any ground truth. The authors report that this score is linearly correlated with true Dice accuracy on four public medical datasets (LiTS, Hippocampus, BraTS, ISIC) and that the relation outputs match or beat vanilla U-Net in Dice. The practical payoff would be a way for clinicians to know which automated segmentations need manual review.","feed_headline":"Segmentation network predicts per-image accuracy without labels","feed_subtitle":"By comparing the union and intersection of two predictions, it ranks test images by accuracy with no ground truth.","key_machinery":"The central mechanism is the Relation U-Net, built from a vanilla U-Net by concatenating the two input images at the input layer and adding four independent classifiers at the output layer: the segmentation heads $\\hat{s}_1, \\hat{s}_2$ and the relation heads $\\hat{r}_p$ (union/OR) and $\\hat{r}_c$ (intersection/AND). The confidence score is defined as the Dice coefficient between $\\hat{r}_p$ and $\\hat{r}_c$, written $C = 2|\\hat{r}_p \\cap \\hat{r}_c|/(|\\hat{r}_p| + |\\hat{r}_c|)$. The argument depends on reflexivity: with $x_1 = x_2$ the ground-truth relations coincide with the segmentation itself, so the residual discrepancy of the independently predicted relation maps is interpreted as a difficulty signal. Training samples pairs of distinct images with supervision on union and intersection of their ground truths; testing feeds one image twice and reads off $C$ as an unsupervised confidence estimate.","core_discovery":"On the paper's own terms, the discovery is that the consistency between two relation segmentation maps—the Possible relation $\\hat{r}_p = \\hat{s}_1 \\vee \\hat{s}_2$ and the Consensus relation $\\hat{r}_c = \\hat{s}_1 \\wedge \\hat{s}_2$—acts as a proxy for the accuracy of the segmentation of a single test image. Because the relations are reflexive, when $x_1 = x_2$ the two relation maps should both equal the segmentation; the trained network only approximates this, and the resulting discrepancy, measured as a Dice consistency score $C$, is claimed to be linearly correlated with the true Dice on the test image. The paper shows this correlation is stronger than that obtained from Monte Carlo Dropout uncertainty, and that thresholding test images by $C$ selects subsets with higher average Dice. The authors' strongest claim is that 'the discrepancy of the estimated relation segmentation maps is correlated to the segmentation accuracy of each test images.'","pith_inferences":["Editorial inference: the same discrepancy trick should transfer to any multi-output segmentation architecture whose outputs stand in a set-theoretic relation, not just U-Net, because it only requires two independently predicted maps with known union/intersection semantics.","Editorial inference: the paper's correlation results come from in-distribution five-fold splits; whether the confidence signal survives out-of-distribution shifts (different scanner, contrast, or population) is an open question a deployment study would need to test.","Editorial inference: a direct way to strengthen the reflexivity premise would be to add identical-pair training examples with the constraint $\\hat{r}_p = \\hat{r}_c = \\hat{s}$; this could reduce subnetwork noise and sharpen the confidence score.","Editorial inference: for clinical triage, what matters is not just linear correlation but calibration of $C$ against a clinical threshold such as Dice > 0.8; converting $C$ into a probability of acceptable segmentation would make the score directly actionable."],"forward_implications":["On the four evaluated datasets, Relation U-Net's outputs (particularly the Possible relation $\\hat{r}_p$ or the average of the two segmentation heads) reach Dice scores at least as high as vanilla U-Net, so adding relation heads does not sacrifice accuracy.","Without ground truth, the confidence score $C$ lets a user rank test images by difficulty; keeping only the images with the highest $C$ yields a subset with substantially higher average Dice.","The Pearson correlation between $C$ and true Dice is higher for Relation U-Net than for the MC Dropout baseline on all four datasets.","Using a second input image as an anchor (sampled from the training or test set) gives a small additional accuracy gain over feeding the same image twice, suggesting the relation heads can also act as an ensemble."],"supporting_citations":[{"why":"Defines the vanilla U-Net baseline architecture that Relation U-Net modifies and compares against.","marker":"[1]"},{"why":"Motivates ranking test examples by difficulty and supplies the coverage-style evaluation used in Fig. 4(b).","marker":"[3]"},{"why":"Provides the Possible/Consensus relation definitions (union and intersection) and the reflexivity property on which the confidence score is built.","marker":"[4]"},{"why":"Supplies the multiple-input multiple-output configuration used to construct Relation U-Net's four prediction heads.","marker":"[5]"},{"why":"LiTS is one of the four public evaluation datasets for liver tumor segmentation.","marker":"[6]"},{"why":"Hippocampus segmentation dataset from the Medical Segmentation Decathlon used for evaluation.","marker":"[7]"},{"why":"BraTS brain tumor segmentation benchmark used for evaluation.","marker":"[8]"},{"why":"ISIC skin lesion challenge dataset used for evaluation.","marker":"[9]"},{"why":"Provides the Monte Carlo Dropout baseline whose uncertainty scores are compared against Relation U-Net's confidence score.","marker":"[10]"},{"why":"Shows how to compute Possible and Consensus relation maps from multiple segmentations, used to construct the comparison confidence scores.","marker":"[11]"}],"fun_headline_variants":["Relation U-Net: Self-scoring segmentation without ground truth","Segmentation that grades itself: Relation U-Net's confidence trick","No labels needed: Relation U-Net knows its own accuracy","Self-aware segmentation: Relation U-Net estimates its own errors","Segmentation with a built-in accuracy oracle (no labels)"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that when the same image is fed twice, the difference between the predicted union and intersection maps reflects how hard the image is to segment, rather than random variation among the independently trained prediction heads.","fun_headline_variants_meta":{"raw":{"variants":["Relation U-Net: Self-scoring segmentation without ground truth","Segmentation that grades itself: Relation U-Net's confidence trick","No labels needed: Relation U-Net knows its own accuracy","Self-aware segmentation: Relation U-Net estimates its own errors","Segmentation with a built-in accuracy oracle (no labels)"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000625,"raw_usage":{"total_tokens":2845,"prompt_tokens":848,"completion_tokens":1997,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":1911}},"tokens_in":464,"tokens_out":1997,"duration_ms":12762,"temperature":1.0,"reasoning_tokens":1911,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:10:19.592330+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train Relation U-Net twice with different random seeds. If the confidence scores $C$ for the same test images have low rank correlation between the two training runs while the segmentation Dice stays similar, then $C$ is dominated by subnetwork variability rather than image difficulty; alternatively, on a test set of uniformly easy, high-quality images, a large spread in $C$ would show the same thing.","supporting_citations":[{"cited_title":"output-only","cited_arxiv_id":null,"evidence_quote":"Defines the vanilla U-Net baseline architecture that Relation U-Net modifies and compares against."},{"cited_title":"Datasets We use four public datasets to evaluate the proposed method, including LiTS [6], Hippocampus [7], BraTS [8] and ISIC [9]","cited_arxiv_id":null,"evidence_quote":"Motivates ranking test examples by difficulty and supplies the coverage-style evaluation used in Fig. 4(b)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Possible/Consensus relation definitions (union and intersection) and the reflexivity property on which the confidence score is built."},{"cited_title":"U-Net: Convolutional networks for biomedical image segmentation,","cited_arxiv_id":null,"evidence_quote":"Supplies the multiple-input multiple-output configuration used to construct Relation U-Net's four prediction heads."},{"cited_title":"Confi- dence calibration and predictive uncertainty estimation for deep medical image segmentation,","cited_arxiv_id":null,"evidence_quote":"LiTS is one of the four public evaluation datasets for liver tumor segmentation."},{"cited_title":"Es- timating example difficulty using variance of gradients,","cited_arxiv_id":null,"evidence_quote":"Hippocampus segmentation dataset from the Medical Segmentation Decathlon used for evaluation."},{"cited_title":"Pairwise learning for medical image seg- mentation,","cited_arxiv_id":null,"evidence_quote":"BraTS brain tumor segmentation benchmark used for evaluation."},{"cited_title":"Deep relation learning for regression and its appli- cation to brain age estimation,","cited_arxiv_id":null,"evidence_quote":"ISIC skin lesion challenge dataset used for evaluation."},{"cited_title":"The liver tumor segmentation benchmark (LIST),","cited_arxiv_id":null,"evidence_quote":"Provides the Monte Carlo Dropout baseline whose uncertainty scores are compared against Relation U-Net's confidence score."}],"review_version":1}