Pith. sign in

REVIEW 2 major objections 6 minor 26 references

Conformal Segmentation in Industrial Surface Defect Detection with Statistical Guarantees

T0 review · 2 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read Calibration keeps steel-defect detection errors at or below a user-set level.

desk verdict Competent industrial application of Conformal Risk Control to steel defect segmentation, but the paper's restatement of the CRC theorem is wrong, the FDR loss monotonicity is asserted without proof, and the abstract contradicts the results on correlation direction. read the letter →

arxiv 2504.17721 v1 pith:FEQW5MXM submitted 2025-04-24 cs.LG cs.AI

classification cs.LGcs.AI
keywords surfacedefectdetectionconformalriskcontrolfalsediscoveryratenegativeuncertaintyquantificationMaskR-CNNstatisticalguaranteessteelsegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that pixel-level defect detection on steel surfaces can carry a statistical guarantee: for any user-specified risk level $\alpha$, the expected false discovery rate (the detected pixels that are not actually defective) and the expected false negative rate (the true defect pixels that are missed) on new test images stay at or below $\alpha$. The method takes probability maps produced by a Mask R-CNN segmentation model and calibrates a single confidence threshold on a separate dataset drawn from the same distribution as the test data. If the claim is right, industrial inspectors can choose an acceptable error budget in advance and trust that the deployed detector meets it on average, without retraining the network. The paper also reports that the average prediction-set size rises with the allowed risk level, proposing this size as an uncertainty metric.

What carries the argument

The load-bearing object is the threshold-parameterized prediction set $C_i(\lambda)$ together with the per-sample losses $l_i(\lambda)$ that measure false-discovery and false-negative proportions. The argument rides on the monotonicity of these losses in $\lambda$ and on the CRC threshold formula $\hat\lambda=\inf\{\lambda: L_n(\lambda)\le(\alpha(n+1)-1)/n\}$, where $L_n$ is the calibration-average loss. The $+1/(n+1)$ correction inside the formula makes the selected threshold slightly more conservative than the raw empirical quantile; this correction is what converts an observed calibration risk into a bound on expected test risk under exchangeability.

What would settle it

Compute $l_i(\lambda)$ for every calibration image on a fine grid of $\lambda$. If any image shows FDR rising as $\lambda$ increases, that image violates the assumed monotonicity, and one can construct a calibration set in which the CRC-selected threshold fails to keep expected FDR at or below $\alpha$, visible as an empirical FDR above the reference line.

Watch

Extended reading notes

Core claim

The central claim is that a conformal risk-control calibration step turns a raw Mask R-CNN probability map into a prediction set whose expected FDR and expected FNR on unseen test images are each bounded by the preset $\alpha$. For each calibration image the authors define a per-sample FDR loss $l_i(\lambda)=1-\frac{|C_i(\lambda)\cap y_i^*|}{|C_i(\lambda)|\vee 1}$, where $C_i(\lambda)=\{(j,k): f(x_i)_{j,k}\ge 1-\lambda\}$ is the thresholded prediction set, then choose $\hat\lambda$ as the smallest threshold satisfying the finite-sample corrected empirical risk bound $L_n(\lambda)\le(\alpha(n+1)-1)/n$. This choice is shown to imply $E[\mathrm{FDR}(S_{\mathrm{test}})]\le\alpha$; changing the denominator to the ground-truth defect area gives the analogous FNR control. On the Severstal and NEU steel datasets all tested backbone networks keep empirical FDR below the reference line for $\alpha\in[0.1,0.9]$, and an ablation across calibration-to-test split ratios reports FNR control at every split.

Load-bearing premise

The guarantee depends on the per-image false-discovery loss never increasing when the detection threshold is relaxed; the paper assumes this monotonicity without proving it or checking it on data.

Editorial extensions

If this is right

  • Users can specify a maximum acceptable FDR or FNR before deployment and receive a confidence threshold that is guaranteed in expectation on new data from the same source.
  • The same calibration pipeline works on top of any segmentation network that outputs per-pixel confidence scores, not only Mask R-CNN.
  • At a fixed risk level, average prediction-set size becomes a model-comparison signal: a smaller set indicates a sharper feature representation, as the paper observes for ResNet-50 against lightweight backbones.
  • The FNR guarantee survives changes in calibration-to-test split ratio, so practitioners only need a calibration set from the same distribution, not a precisely tuned split.
  • Because the threshold adapts to each model and dataset, the framework supports deploying one detector across production lines with controlled error rates instead of hand-tuned confidence cutoffs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If prediction-set size is read as an uncertainty metric, the reported negative correlation between set size and model parameters could become a practical model-selection heuristic for choosing among backbones; the paper does not establish why the correlation holds.
  • The same CRC construction should extend to per-class guarantees by defining separate FDR/FNR losses for each defect category, giving users a bound on each defect type rather than a pooled bound; this is a direct but untested extension.
  • A deliberate stress test with corrupted or shifted test images would show how quickly the guarantee degrades when the exchangeability premise is violated, since the paper only evaluates clean calibration/test splits.
  • The assumed monotonicity of the loss in the threshold is cheap to verify empirically by plotting $l_i(\lambda)$ over a grid; if real images produce non-monotone losses, the finite-sample guarantee would need an additional bounded-rate-of-change condition.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper applies Conformal Risk Control (CRC) to Mask R-CNN-based pixel-wise defect segmentation on two steel surface defect datasets (Severstal and NEU). It defines per-image FDR and FNR loss functions, computes an empirical risk on a calibration set, selects a data-dependent confidence threshold via Eq. (8), and claims that the expected FDR/FNR on test images is bounded by a user-specified level α. Experiments report empirical FDR/FNR curves for six backbone networks, prediction-set-size analyses, and an ablation study over calibration-to-test split ratios.

Significance. If the central guarantee holds, the paper provides a practically useful, model-agnostic procedure for controlling a user-specified error rate in industrial defect segmentation, which is a meaningful step beyond raw detection accuracy. The empirical evaluation covers two real datasets and multiple backbones, and the paper explicitly checks that empirical error rates stay below the target line. The paper does not claim new conformal theory; its contribution is an application of the existing CRC framework plus a prediction-set-size diagnostic. However, the paper does not provide code or machine-checked proofs, and the theoretical derivation contains a load-bearing gap that needs to be resolved before the main claim is established.

major comments (2)
  1. [Section 3.2, Eqs. (5) and (8)] The assertion that l_i(λ) 'monotonically decreases with λ' is not proved and is not generally true for the FDR loss defined in Eq. (5). Since C_i(λ) = {(j,k): f(x_i)_{j,k} ≥ 1−λ}, increasing λ lowers the confidence threshold and expands the prediction set; each added false-positive pixel increases the false-discovery fraction, while each added true-positive pixel decreases it. Nothing in the problem guarantees that true positives are encountered before false positives as λ grows, so l_i(λ) can be non-monotone. This matters because the CRC theorem used to pass from Eq. (8) to Eq. (9) requires the per-sample losses to be monotone nonincreasing in λ; without this condition, the data-dependent threshold λ̂ is not covered by the guarantee. The authors should either prove monotonicity for this specific loss, verify it empirically on the calibration set (e.g., plot l_i(λ) versus λ for a sample of calibration images), or switch to a monotone loss function.
  2. [Section 3.2, Eqs. (7) and (9)] As written, these equations equate an expectation to a random variable. The left side E[l_{n+1}(λ)] is a deterministic number, while the right side (nL_n(λ)+l_{n+1}(λ))/(n+1) depends on the calibration sample and on the test loss. The correct argument uses exchangeability of the n+1 losses, for instance E[(Σ_{i=1}^{n+1} l_i(λ))/(n+1)] = E[l_{n+1}(λ)], and the proof of the threshold inequality must be stated with expectations over the appropriate random quantities. The current derivation is not a valid proof of the central E[FDR(S_test)]≤α claim.
minor comments (6)
  1. [Abstract and Section 4.5.2] The abstract and conclusion state a 'negative correlation between the average prediction set size and the risk level,' but Section 4.5.2 reports a 'significant nonlinear positive correlation' with r = 0.91. The sign of the correlation should be corrected to match the experimental result.
  2. [Section 3.3] The text says the optimal threshold is determined via 'constrained optimization as detailed in Eq. 3,' but Eq. (3) is the Mask R-CNN multi-task loss; the intended reference is Eq. (8).
  3. [Section 4.4.2, Eq. (13)] Equation (13) is poorly formatted: the denominator 'P y*_i' is not clearly defined, and the set notation should explicitly refer to pixels rather than image indices.
  4. [Section 4.6.2] The statement that the optimal λ follows 'the theoretical threshold update mechanism λ* = 1−α/k' is not derived from Eq. (8) and does not match the CRC quantile-based formula; the parameter k is not defined. This should be removed or properly derived.
  5. [Section 3.2, Eq. (8)] For small calibration sizes or small α, the right-hand side (α(n+1)−1)/n can be negative, making the infimum in Eq. (8) empty over the stated range of λ. The domain of λ and the convention for an empty set should be specified.
  6. [Throughout] There are several typos and wording issues, including 'strictly bounced' (should be 'bounded'), 'residul' (should be 'residual'), and 'effieciency' (should be 'efficiency').

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the FDR guarantee is an application of the external CRC theorem with calibration- and test-set separation, not a reduction of the conclusion into its inputs.

full rationale

The paper's derivation is a direct application of Conformal Risk Control (Angelopoulos et al., ref. [11]) to surface defect segmentation. The threshold lambda-hat is selected from the calibration set through Eq. 8 using the empirical risk L_n(lambda); the guarantee in Eq. 9 is the CRC transfer bound that follows from exchangeability, an external theorem not constructed by the present authors. The per-sample FDR loss in Eq. 5 is the standard false-discovery proportion and is not defined in terms of the test-set quantity it is later used to bound. No parameter is fitted to test data and then reported as a prediction: the empirical evaluations in Sec. 4.4 are performed on a separate test set, so the fitted-input-called-prediction pattern does not apply. There are no self-citations carrying the argument; the cited CRC and segmentation works are independent prior literature. The asserted monotonicity of l_i(lambda) in lambda is not proved and may fail in general, which is a correctness risk rather than a circularity, since the claim does not reduce to its own inputs by definition. Likewise, the abstract's wording about a negative correlation between prediction set size and risk level conflicts with the reported positive correlation r = 0.91 in Sec. 4.5.2, but that is an internal inconsistency, not a circular derivation. Because the central guarantee rests on an external theorem and a calibration-to-test split, the paper is self-contained against external benchmarks and receives a non-circularity score of 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The method relies on the external Conformal Risk Control theorem, the exchangeability of calibration and test data, the boundedness of the loss in [0,1], and an unproved monotonicity condition on the FDR loss. No invented entities or fitted model parameters are introduced; the threshold is selected by the calibration procedure itself.

assumptions (4)
  • domain assumption Calibration and test data are i.i.d. (or at least exchangeable).
    Invoked in Section 3.2 before Eq. 5; the finite-sample CRC guarantee depends on exchangeability of calibration and test loss values.
  • ad hoc to paper The FDR loss l_i(λ) is monotonically nonincreasing in λ.
    Asserted immediately after Eq. 5 ('The loss function monotonically decreases with λ') with no proof; required for the inf-formula threshold in Eq. 8 to control expected risk. The paper does not verify this on calibration data.
  • standard math Conformal Risk Control theorem of Angelopoulos et al. [11] is valid and applicable.
    The paper's guarantee (Eq. 9) is a restatement of this cited theorem; no independent proof is given.
  • standard math Loss values remain in [0,1].
    Needed for the finite-sample CRC bound; FDR loss (Eq. 5) and FNR loss (Eq. 13) satisfy this by construction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Conformal Segmentation in Industrial Surface Defect Detection with Statistical Guarantees." pith.science (2026). https://pith.science/paper/FEQW5MXM

@misc{pith2026250417721,
  author       = {Pith},
  title        = {Pith review of: Conformal Segmentation in Industrial Surface Defect Detection with Statistical Guarantees},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FEQW5MXM}},
  note         = {Machine review of arXiv:2504.17721}
}
read the original abstract

In industrial settings, surface defects on steel can significantly compromise its service life and elevate potential safety risks. Traditional defect detection methods predominantly rely on manual inspection, which suffers from low efficiency and high costs. Although automated defect detection approaches based on Convolutional Neural Networks(e.g., Mask R-CNN) have advanced rapidly, their reliability remains challenged due to data annotation uncertainties during deep model training and overfitting issues. These limitations may lead to detection deviations when processing the given new test samples, rendering automated detection processes unreliable. To address this challenge, we first evaluate the detection model's practical performance through calibration data that satisfies the independent and identically distributed (i.i.d) condition with test data. Specifically, we define a loss function for each calibration sample to quantify detection error rates, such as the complement of recall rate and false discovery rate. Subsequently, we derive a statistically rigorous threshold based on a user-defined risk level to identify high-probability defective pixels in test images, thereby constructing prediction sets (e.g., defect regions). This methodology ensures that the expected error rate (mean error rate) on the test set remains strictly bounced by the predefined risk level. Additionally, we observe a negative correlation between the average prediction set size and the risk level on the test set, establishing a statistically rigorous metric for assessing detection model uncertainty. Furthermore, our study demonstrates robust and efficient control over the expected test set error rate across varying calibration-to-test partitioning ratios, validating the method's adaptability and operational effectiveness.

Figures

Figures reproduced from arXiv: 2504.17721 by the authors.

Figure 1
Figure 1. Workflow of our Approach This framework ensures user-specified risk control while adaptively tuning detection sensitivity based on the data￾dependent threshold λˆ, effectively balancing statistical rigor with operational practicality. 4 Experiment Analysis 4.1 Datasets and Benchmarks This study is validated on two complementary steel surface defect datasets, [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Visual Dataset Analysis 4.2 Experimental Setup To evaluate the impact of different feature extraction modules, we implemented Mask R-CNN with ResNet-50 and several alternative backbone networks for feature extraction. For fair comparison, all backbones were initialized with ImageNet pretrained weights and fine-tuned under identical training protocols: • ResNet-50: A classic residual network comprising 50 layers (49 … view at source ↗
Figure 3
Figure 3. Guarantee of the FDR Metric 4.4.2 Guarantees of the FNR metric By adjusting the denominator in Equation 12 to the ground-truth defect pixel count y ∗ i , we formulate an enhanced FNR (False Negative Rate) control objective: FNR = E  |xi ∈ Sc : yi = 1| Py ∗ i  . (13) [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Guarantee of the FNR Metric 0.0 0.2 0.4 0.6 0.8 1.0 Risk Level (Alpha) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Average prediction set size (pixels) 1e6 Comparison of prediction set sizes for different network structures (Severstal) ResNet50 ResNet34 SqueezeNet GhostNetV3 Shuff…
Figure 5
Figure 5. Figure 5: Comparison of Prediction Set Sizes across Network Architectures [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 18 canonical work pages

  1. [11]

    Conformal risk control

    Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. arXiv preprint arXiv:2208.02814, 2022

  2. [1]

    Y . Liu, X. Zhang, and J. Wang. Deep learning-based defect detection in steel manufacturing: A review. IEEE Transactions on Industrial Informatics, 17(5):3061–3072, 2021

  3. [2]

    H. Wang, J. Li, and F. Zhou. Deep learning for industrial defect detection: A comprehensive review. Pattern Recognition, 107:107254, 2020

  4. [3]

    Towards trustworthy rotating machinery fault diagnosis via attention uncertainty in transformer.Journal of Manufacturing Systems, 70:186–201, 2023

    Yiming Xiao, Haidong Shao, Minjie Feng, Te Han, Jiafu Wan, and Bin Liu. Towards trustworthy rotating machinery fault diagnosis via attention uncertainty in transformer.Journal of Manufacturing Systems, 70:186–201, 2023. 11 Under Review

  5. [4]

    A survey of uncertainty in deep neural networks

    Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56(Suppl 1):1513–1589, 2023

  6. [5]

    Word-sequence entropy: Towards uncertainty estimation in free-form medical question answering applications and beyond

    Zhiyuan Wang, Jinhao Duan, Chenxi Yuan, Qingyu Chen, Tianlong Chen, Yue Zhang, Ren Wang, Xiaoshuang Shi, and Kaidi Xu. Word-sequence entropy: Towards uncertainty estimation in free-form medical question answering applications and beyond. Engineering Applications of Artificial Intelligence, 139:109553, 2025

  7. [6]

    Steel surface defect detection using a new haar–weibull-variance model in unsupervised manner

    Kun Liu, Heying Wang, Haiyong Chen, Erqing Qu, Ying Tian, and Hexu Sun. Steel surface defect detection using a new haar–weibull-variance model in unsupervised manner. IEEE transactions on instrumentation and measurement, 66(10):2585–2596, 2017

  8. [7]

    Invariant feature extraction method based on smoothed local binary pattern for strip steel surface defect

    Maoxiang Chu and Rongfen Gong. Invariant feature extraction method based on smoothed local binary pattern for strip steel surface defect. ISIJ International, 55(9):1956–1962, 2015

Show all 26 references
  1. [8]

    Edrnet: Encoder–decoder residual network for salient object detection of strip steel surface defects

    Guorong Song, Kechen Song, and Yunhui Yan. Edrnet: Encoder–decoder residual network for salient object detection of strip steel surface defects. IEEE Transactions on Instrumentation and Measurement, 69(12):9709– 9719, 2020

  2. [9]

    Automatic surface defect segmentation for hot-rolled steel strip using depth-wise separable u-shape network

    Zheng Huang, Jiajun Wu, and Feng Xie. Automatic surface defect segmentation for hot-rolled steel strip using depth-wise separable u-shape network. Materials Letters, 301:130271, 2021

  3. [10]

    Machine-learning applications of algorithmic randomness

    V olodya V ovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of algorithmic randomness. 1999

  4. [12]

    Conu: Conformal uncertainty in large language models with correctness coverage guarantees

    Zhiyuan Wang, Jinhao Duan, Lu Cheng, Yue Zhang, Qingni Wang, Xiaoshuang Shi, Kaidi Xu, Heng Tao Shen, and Xiaofeng Zhu. Conu: Conformal uncertainty in large language models with correctness coverage guarantees. In Findings of the Association for Computational Linguistics: EMNL...

  5. [13]

    Conformal risk control for pulmonary nodule detection

    Roel Hulsman, Valentin Comte, Lorenzo Bertolini, Tobias Wiesenthal, Antonio Puertas Gallardo, and Mario Ceresa. Conformal risk control for pulmonary nodule detection. arXiv preprint arXiv:2412.20167, 2024

  6. [14]

    Sample then identify: A general framework for risk control and assessment in multimodal large language models

    Qingni Wang, Tiantian Geng, Zhiyuan Wang, Teng Wang, Bo Fu, and Feng Zheng. Sample then identify: A general framework for risk control and assessment in multimodal large language models. In The Thirteenth International Conference on Learning Representations, 2025

  7. [15]

    Sconu: Selective conformal uncertainty in large language models

    Zhiyuan Wang, Qingni Wang, Yue Zhang, Tianlong Chen, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. Sconu: Selective conformal uncertainty in large language models. arXiv preprint arXiv:2504.14154, 2025

  8. [16]

    An electronic nose-based assistive diagnostic prototype for lung cancer detection with conformal prediction

    Xianghao Zhan, Zhan Wang, Meng Yang, Zhiyuan Luo, You Wang, and Guang Li. An electronic nose-based assistive diagnostic prototype for lung cancer detection with conformal prediction. Measurement, 158:107588, 2020

  9. [17]

    Uncertainty-aware real-time visual anomaly detection with conformal prediction in dynamic indoor environments

    Arya Saboury and Mustafa Kemal Uyguroglu. Uncertainty-aware real-time visual anomaly detection with conformal prediction in dynamic indoor environments. IEEE Robotics and Automation Letters, 2025

  10. [18]

    Conformal semantic image segmentation: Post-hoc quantifi- cation of predictive uncertainty

    Luca Mossina, Joseba Dalmau, and Léo Andéol. Conformal semantic image segmentation: Post-hoc quantifi- cation of predictive uncertainty. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3574–3584, 2024

  11. [19]

    Confident object detection via conformal prediction and conformal risk control: an application to railway signaling

    Léo Andéol, Thomas Fel, Florence De Grancey, and Luca Mossina. Confident object detection via conformal prediction and conformal risk control: an application to railway signaling. In Conformal and Probabilistic Prediction with Applications, pages 36–55. PMLR, 2023

  12. [20]

    Statistical guarantees of false discovery rate in medical instance segmentation tasks based on conformal risk control

    Mengxia Dai, Wenqian Luo, and Tianyang Li. Statistical guarantees of false discovery rate in medical instance segmentation tasks based on conformal risk control. arXiv preprint arXiv:2504.04482, 2025

  13. [21]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2961–2969, 2017

  14. [22]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

  15. [23]

    Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size

    Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016

  16. [24]

    Shufflenet v2: Practical guidelines for efficient cnn architecture design

    Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 116–131, 2018. 12 Under Review

  17. [25]

    Searching for mobilenetv3

    Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019

  18. [26]

    Ghostnetv3: Exploring the training strategies for compact models

    Zhenhua Liu, Zhiwei Hao, Kai Han, Yehui Tang, and Yunhe Wang. Ghostnetv3: Exploring the training strategies for compact models. arXiv preprint arXiv:2404.11202, 2024. 13

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.