{"id":"30e36d54-1b6c-45de-9205-8c556cfcd5be","arxiv_id":"2504.17721","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Applying conformal risk control to Mask R-CNN yields empirical FDR and FNR values below pre-specified risk levels on two steel defect datasets, but the paper adds little beyond reusing the existing CRC theorem.","lead":"This paper applies conformal risk control, an existing statistical framework, to Mask R-CNN based steel surface defect detection, aiming to guarantee that false discovery and false negative rates stay below user-set levels. It reports experiments on the Severstal and NEU datasets across six backbone networks, and proposes average prediction set size as an uncertainty metric.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Monotonicity of the FDR loss in λ is asserted but not established; without it the central E[FDR]≤α guarantee in Eq. 9 defaults to a pointwise conditional bound that their experiments do not verify.","rationale":"The reader and I converge on the same weakest assumption: monotonicity of the per-sample FDR loss in λ (Eq. 5) is asserted without proof and is not generally true. A concrete failure mode is that expanding C_i(λ) by threshold lowering can add background pixels faster than true positives, so 1−precision can rise locally. Since CRC's threshold-selection theorem (Angelopoulos et al.) explicitly relies on monotonicity, the entire transfer of the guarantee (Eq. 9) is compromised in the absence of verification or a monotone surrogate. The paper's own experiments only report final FDR under the selected threshold, which cannot reveal non-monotonicity; a per-sample loss-versus-λ plot is the decisive test. I also note the reader's other correctness points: Eq. 7 and Eq. 9 are stated as equalities where CRC has an inequality-style upper bound, and Eq. 13's FNR expression is not a valid expectation formula as written. The abstract's stated negative correlation between average prediction-set size and risk level contradicts the positive correlation reported in §4.5.2 (Pearson r=0.91), so the empirical narrative is not internally consistent. None of these issues impeach the underlying CRC machinery or the plausibility of the empirical results; they are repair-and-verify issues, not a rejection. Hence CONDITIONAL remains the right verdict, with the monotonicity verification and corrections as conditions.","tokens_in":11083,"tokens_out":1921,"duration_ms":16634,"concrete_test":"Recompute l_i(λ) from Eq. 5 on the actual calibration images across a fine grid of λ (e.g., 0.01 to 0.99) for each calibration sample under each backbone, and confirm that every l_i(λ) is nonincreasing in λ (plot the per-sample curves and report the fraction violating monotonicity). Additionally, rerun the calibration-test split using the CRC fixed-point formula on a dataset where low-confidence background dominates the prediction sets, and compare the empirical E[FDR] against α for the data-driven λ̂ versus an oracle nonincreasing upper envelope of the loss; if the empirical FDR exceeds α or the fraction of non-monotone samples is nonzero, the guarantee in Eq. 9 needs repair.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central guarantee is Eq. 9: E[l_{n+1}(λ̂)]≤α. The step that makes λ̂ data-dependent valid is the claim in §3.2 that l_i(λ)=1−|C_i(λ)∩y_i*|/(|C_i(λ)|∨1) 'monotonically decreases with λ'. That claim is not generally true. As λ grows, the prediction set C_i(λ) expands; the numerator grows by adding true defect pixels, but the denominator also grows because low-confidence background pixels are included. If the added pixels contain fewer true positives than the current precision, the FDR loss actually increases on a non-monotone trajectory (e.g., a small true-defect region embedded in a large low-confidence background region). CRC's theorem requires monotone nonincreasing losses to replace the oracle λ* by the empirical λ̂; without monotonicity, Eq. 9 only holds for a fixed λ, and the data-dependent threshold can violate the bound. The paper supplies no proof, no counterexample check on their own calibration data (e.g., plotting l_i(λ) as a function of λ), and the abstract's claim of a negative correlation between prediction set size and risk level is internally inconsistent with the positive correlation shown in §4.5.2 (r=0.91), which further muddies the empirical support. Because the main theoretical contribution is the transferred CRC guarantee, an unverified monotonicity condition is the load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies Conformal Risk Control (CRC) to Mask R-CNN-based pixel-wise defect segmentation on two steel surface defect datasets (Severstal and NEU). It defines per-image FDR and FNR loss functions, computes an empirical risk on a calibration set, selects a data-dependent confidence threshold via Eq. (8), and claims that the expected FDR/FNR on test images is bounded by a user-specified level α. Experiments report empirical FDR/FNR curves for six backbone networks, prediction-set-size analyses, and an ablation study over calibration-to-test split ratios.","tokens_in":11335,"tokens_out":7742,"duration_ms":79845,"significance":"If the central guarantee holds, the paper provides a practically useful, model-agnostic procedure for controlling a user-specified error rate in industrial defect segmentation, which is a meaningful step beyond raw detection accuracy. The empirical evaluation covers two real datasets and multiple backbones, and the paper explicitly checks that empirical error rates stay below the target line. The paper does not claim new conformal theory; its contribution is an application of the existing CRC framework plus a prediction-set-size diagnostic. However, the paper does not provide code or machine-checked proofs, and the theoretical derivation contains a load-bearing gap that needs to be resolved before the main claim is established.","major_comments":[{"comment":"The assertion that l_i(λ) 'monotonically decreases with λ' is not proved and is not generally true for the FDR loss defined in Eq. (5). Since C_i(λ) = {(j,k): f(x_i)_{j,k} ≥ 1−λ}, increasing λ lowers the confidence threshold and expands the prediction set; each added false-positive pixel increases the false-discovery fraction, while each added true-positive pixel decreases it. Nothing in the problem guarantees that true positives are encountered before false positives as λ grows, so l_i(λ) can be non-monotone. This matters because the CRC theorem used to pass from Eq. (8) to Eq. (9) requires the per-sample losses to be monotone nonincreasing in λ; without this condition, the data-dependent threshold λ̂ is not covered by the guarantee. The authors should either prove monotonicity for this specific loss, verify it empirically on the calibration set (e.g., plot l_i(λ) versus λ for a sample of calibration images), or switch to a monotone loss function.","section":"Section 3.2, Eqs. (5) and (8)"},{"comment":"As written, these equations equate an expectation to a random variable. The left side E[l_{n+1}(λ)] is a deterministic number, while the right side (nL_n(λ)+l_{n+1}(λ))/(n+1) depends on the calibration sample and on the test loss. The correct argument uses exchangeability of the n+1 losses, for instance E[(Σ_{i=1}^{n+1} l_i(λ))/(n+1)] = E[l_{n+1}(λ)], and the proof of the threshold inequality must be stated with expectations over the appropriate random quantities. The current derivation is not a valid proof of the central E[FDR(S_test)]≤α claim.","section":"Section 3.2, Eqs. (7) and (9)"}],"minor_comments":[{"comment":"The abstract and conclusion state a 'negative correlation between the average prediction set size and the risk level,' but Section 4.5.2 reports a 'significant nonlinear positive correlation' with r = 0.91. The sign of the correlation should be corrected to match the experimental result.","section":"Abstract and Section 4.5.2"},{"comment":"The text says the optimal threshold is determined via 'constrained optimization as detailed in Eq. 3,' but Eq. (3) is the Mask R-CNN multi-task loss; the intended reference is Eq. (8).","section":"Section 3.3"},{"comment":"Equation (13) is poorly formatted: the denominator 'P y*_i' is not clearly defined, and the set notation should explicitly refer to pixels rather than image indices.","section":"Section 4.4.2, Eq. (13)"},{"comment":"The statement that the optimal λ follows 'the theoretical threshold update mechanism λ* = 1−α/k' is not derived from Eq. (8) and does not match the CRC quantile-based formula; the parameter k is not defined. This should be removed or properly derived.","section":"Section 4.6.2"},{"comment":"For small calibration sizes or small α, the right-hand side (α(n+1)−1)/n can be negative, making the infimum in Eq. (8) empty over the stated range of λ. The domain of λ and the convention for an empty set should be specified.","section":"Section 3.2, Eq. (8)"},{"comment":"There are several typos and wording issues, including 'strictly bounced' (should be 'bounded'), 'residul' (should be 'residual'), and 'effieciency' (should be 'efficiency').","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The monotonicity issue in Section 3.2 is the main correctness risk: the FDR loss defined in Eq. (5) is not generally monotone in λ, so the CRC theorem as stated does not apply. If the authors cannot prove monotonicity or replace the loss with a monotone surrogate, the central statistical guarantee collapses. The incorrect expectation/random-variable equations in Eqs. (7) and (9) also need a rigorous rewrite. Given that the empirical results are plausible and the underlying CRC framework is sound, I see this as fixable within a major revision rather than a rejection, provided the authors address the monotonicity requirement directly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a clearly-written application of Conformal Risk Control (Angelopoulos et al.) to steel defect segmentation, and the experiments do show empirical FDR/FNR under the target line across two datasets and several backbones. But the paper's restatement of the CRC theorem in Eqs. 7 and 9 is mathematically wrong, and the monotonicity condition that makes the data-dependent threshold valid is asserted for the FDR loss without proof. That is a load-bearing weakness, not a cosmetic one.\n\nWhat's actually new: not much methodologically. CRC is cited and correctly credited; FDR/FNR control for segmentation already appears in Mossina et al., Andéol et al., and Dai et al. The contribution is the industrial application and the careful comparison of backbone networks. The ablation on split ratios is a nice practical check. The writing is mostly accessible and the figures are informative.\n\nThe soft spots. First, Eq. 7 is nonsense as written: it equates an expectation to a quantity containing the random variable l_{n+1}(λ). Eq. 9 repeats the error. The correct CRC bound from [11] should be stated as a proper finite-sample guarantee with the empirical risk plus a correction term, not as an equality involving a random variable. The authors should copy the theorem verbatim and then specialize.\n\nSecond, the monotonicity claim for the FDR loss in §3.2 is not generally true. As λ increases, C_i(λ) grows; precision may fall. The CRC guarantee requires the loss to be monotone nonincreasing in λ. The authors say it 'monotonically decreases' but give no proof and no empirical plot of l_i(λ) versus λ. This is not a nitpick: without monotonicity, Eq. 8 has no justification and the guarantee collapses to a fixed-λ bound. The FNR loss is monotone because the denominator is fixed, but the FDR loss is not.\n\nThird, there's an internal contradiction: the abstract promises a 'negative correlation between the average prediction set size and the risk level,' while §4.5.2 reports a positive correlation (r=0.91). Someone reading closely will trip on this.\n\nFourth, reproducibility: no code, no data splits, no error bars on the main calibration curves. For an applied paper, that is fairly important.\n\nOn balance, the underlying idea is sound—apply CRC to segmentation and report the empirical behavior—and the experiments suggest the method works on these datasets even if the theory is misstated. But the paper as written cannot be accepted as a rigorous guarantee paper. A serious referee should engage with it, because the application is relevant and the flaws are fixable. I'd recommend major revision with a request for corrected math, a proof or empirical check of monotonicity, and artifact release.","headline":"Competent industrial application of Conformal Risk Control to steel defect segmentation, but the paper's restatement of the CRC theorem is wrong, the FDR loss monotonicity is asserted without proof, and the abstract contradicts the results on correlation direction.","tokens_in":11886,"tokens_out":3168,"would_cite":false,"duration_ms":30083,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Calibration keeps steel-defect detection errors at or below a user-set level.","keywords":["surface defect detection","conformal risk control","false discovery rate","false negative rate","uncertainty quantification","Mask R-CNN","statistical guarantees","steel defect segmentation"],"falsifier":"Compute $l_i(\\lambda)$ for every calibration image on a fine grid of $\\lambda$. If any image shows FDR rising as $\\lambda$ increases, that image violates the assumed monotonicity, and one can construct a calibration set in which the CRC-selected threshold fails to keep expected FDR at or below $\\alpha$, visible as an empirical FDR above the reference line.","tokens_in":10841,"feed_emoji":"🔩","tokens_out":7439,"duration_ms":62900,"temperature":0.7,"pith_summary":"This paper tries to establish that pixel-level defect detection on steel surfaces can carry a statistical guarantee: for any user-specified risk level $\\alpha$, the expected false discovery rate (the detected pixels that are not actually defective) and the expected false negative rate (the true defect pixels that are missed) on new test images stay at or below $\\alpha$. The method takes probability maps produced by a Mask R-CNN segmentation model and calibrates a single confidence threshold on a separate dataset drawn from the same distribution as the test data. If the claim is right, industrial inspectors can choose an acceptable error budget in advance and trust that the deployed detector meets it on average, without retraining the network. The paper also reports that the average prediction-set size rises with the allowed risk level, proposing this size as an uncertainty metric.","feed_headline":"Steel-defect error rates stay under a user-set cap","feed_subtitle":"Calibration tunes Mask R-CNN thresholds so expected false discovery and false negative rates meet user-set limits.","key_machinery":"The load-bearing object is the threshold-parameterized prediction set $C_i(\\lambda)$ together with the per-sample losses $l_i(\\lambda)$ that measure false-discovery and false-negative proportions. The argument rides on the monotonicity of these losses in $\\lambda$ and on the CRC threshold formula $\\hat\\lambda=\\inf\\{\\lambda: L_n(\\lambda)\\le(\\alpha(n+1)-1)/n\\}$, where $L_n$ is the calibration-average loss. The $+1/(n+1)$ correction inside the formula makes the selected threshold slightly more conservative than the raw empirical quantile; this correction is what converts an observed calibration risk into a bound on expected test risk under exchangeability.","core_discovery":"The central claim is that a conformal risk-control calibration step turns a raw Mask R-CNN probability map into a prediction set whose expected FDR and expected FNR on unseen test images are each bounded by the preset $\\alpha$. For each calibration image the authors define a per-sample FDR loss $l_i(\\lambda)=1-\\frac{|C_i(\\lambda)\\cap y_i^*|}{|C_i(\\lambda)|\\vee 1}$, where $C_i(\\lambda)=\\{(j,k): f(x_i)_{j,k}\\ge 1-\\lambda\\}$ is the thresholded prediction set, then choose $\\hat\\lambda$ as the smallest threshold satisfying the finite-sample corrected empirical risk bound $L_n(\\lambda)\\le(\\alpha(n+1)-1)/n$. This choice is shown to imply $E[\\mathrm{FDR}(S_{\\mathrm{test}})]\\le\\alpha$; changing the denominator to the ground-truth defect area gives the analogous FNR control. On the Severstal and NEU steel datasets all tested backbone networks keep empirical FDR below the reference line for $\\alpha\\in[0.1,0.9]$, and an ablation across calibration-to-test split ratios reports FNR control at every split.","pith_inferences":["If prediction-set size is read as an uncertainty metric, the reported negative correlation between set size and model parameters could become a practical model-selection heuristic for choosing among backbones; the paper does not establish why the correlation holds.","The same CRC construction should extend to per-class guarantees by defining separate FDR/FNR losses for each defect category, giving users a bound on each defect type rather than a pooled bound; this is a direct but untested extension.","A deliberate stress test with corrupted or shifted test images would show how quickly the guarantee degrades when the exchangeability premise is violated, since the paper only evaluates clean calibration/test splits.","The assumed monotonicity of the loss in the threshold is cheap to verify empirically by plotting $l_i(\\lambda)$ over a grid; if real images produce non-monotone losses, the finite-sample guarantee would need an additional bounded-rate-of-change condition."],"forward_implications":["Users can specify a maximum acceptable FDR or FNR before deployment and receive a confidence threshold that is guaranteed in expectation on new data from the same source.","The same calibration pipeline works on top of any segmentation network that outputs per-pixel confidence scores, not only Mask R-CNN.","At a fixed risk level, average prediction-set size becomes a model-comparison signal: a smaller set indicates a sharper feature representation, as the paper observes for ResNet-50 against lightweight backbones.","The FNR guarantee survives changes in calibration-to-test split ratio, so practitioners only need a calibration set from the same distribution, not a precisely tuned split.","Because the threshold adapts to each model and dataset, the framework supports deploying one detector across production lines with controlled error rates instead of hand-tuned confidence cutoffs."],"supporting_citations":[{"why":"Supplies the Conformal Risk Control theorem and the finite-sample corrected threshold formula that the paper builds on.","marker":"[11]"},{"why":"Introduces per-sample FDR and FNR loss definitions for medical instance segmentation that the paper adapts to surface defects.","marker":"[20]"},{"why":"Establishes the conformal prediction calibration setting and exchangeability basis for distribution-free guarantees.","marker":"[10]"},{"why":"Provides the Mask R-CNN model whose pixel probability maps are conformalized.","marker":"[21]"},{"why":"Demonstrates threshold-parameterized prediction sets and monotone losses for semantic segmentation with CRC, the template used here.","marker":"[18]"}],"fun_headline_variants":["Conformal calibration caps defect detection errors","Statistical bounds for steel defect segmentation","Guaranteed error control in steel defect detection","Mask R-CNN tuned to hit preset error caps","User-set risk levels bound defect detection errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee depends on the per-image false-discovery loss never increasing when the detection threshold is relaxed; the paper assumes this monotonicity without proving it or checking it on data.","fun_headline_variants_meta":{"raw":{"variants":["Conformal calibration caps defect detection errors","Statistical bounds for steel defect segmentation","Guaranteed error control in steel defect detection","Mask R-CNN tuned to hit preset error caps","User-set risk levels bound defect detection errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1655,"prompt_tokens":1058,"completion_tokens":597,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":674,"completion_tokens_details":{"reasoning_tokens":531}},"tokens_in":674,"tokens_out":597,"duration_ms":5445,"temperature":1.0,"reasoning_tokens":531,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:32:41.469392+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute $l_i(\\lambda)$ for every calibration image on a fine grid of $\\lambda$. If any image shows FDR rising as $\\lambda$ increases, that image violates the assumed monotonicity, and one can construct a calibration set in which the CRC-selected threshold fails to keep expected FDR at or below $\\alpha$, visible as an empirical FDR above the reference line.","supporting_citations":[{"cited_title":"Mask r-cnn","cited_arxiv_id":null,"evidence_quote":"Provides the Mask R-CNN model whose pixel probability maps are conformalized."},{"cited_title":"Statistical Management of the False Discovery Rate in Medical Instance Segmentation Based on Conformal Risk Control","cited_arxiv_id":"2504.04482","evidence_quote":"Introduces per-sample FDR and FNR loss definitions for medical instance segmentation that the paper adapts to surface defects."},{"cited_title":"Machine-learning applications of algorithmic randomness","cited_arxiv_id":null,"evidence_quote":"Establishes the conformal prediction calibration setting and exchangeability basis for distribution-free guarantees."},{"cited_title":"Conformal semantic image segmentation: Post-hoc quantifi- cation of predictive uncertainty","cited_arxiv_id":null,"evidence_quote":"Demonstrates threshold-parameterized prediction sets and monotone losses for semantic segmentation with CRC, the template used here."}],"review_version":1}