REVIEW 2 major objections 6 minor 26 references
Conformal Segmentation in Industrial Surface Defect Detection with Statistical Guarantees
T0 review · 2 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Calibration keeps steel-defect detection errors at or below a user-set level.
desk verdict Competent industrial application of Conformal Risk Control to steel defect segmentation, but the paper's restatement of the CRC theorem is wrong, the FDR loss monotonicity is asserted without proof, and the abstract contradicts the results on correlation direction. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the threshold-parameterized prediction set $C_i(\lambda)$ together with the per-sample losses $l_i(\lambda)$ that measure false-discovery and false-negative proportions. The argument rides on the monotonicity of these losses in $\lambda$ and on the CRC threshold formula $\hat\lambda=\inf\{\lambda: L_n(\lambda)\le(\alpha(n+1)-1)/n\}$, where $L_n$ is the calibration-average loss. The $+1/(n+1)$ correction inside the formula makes the selected threshold slightly more conservative than the raw empirical quantile; this correction is what converts an observed calibration risk into a bound on expected test risk under exchangeability.
What would settle it
Compute $l_i(\lambda)$ for every calibration image on a fine grid of $\lambda$. If any image shows FDR rising as $\lambda$ increases, that image violates the assumed monotonicity, and one can construct a calibration set in which the CRC-selected threshold fails to keep expected FDR at or below $\alpha$, visible as an empirical FDR above the reference line.
Extended reading notes
Core claim
The central claim is that a conformal risk-control calibration step turns a raw Mask R-CNN probability map into a prediction set whose expected FDR and expected FNR on unseen test images are each bounded by the preset $\alpha$. For each calibration image the authors define a per-sample FDR loss $l_i(\lambda)=1-\frac{|C_i(\lambda)\cap y_i^*|}{|C_i(\lambda)|\vee 1}$, where $C_i(\lambda)=\{(j,k): f(x_i)_{j,k}\ge 1-\lambda\}$ is the thresholded prediction set, then choose $\hat\lambda$ as the smallest threshold satisfying the finite-sample corrected empirical risk bound $L_n(\lambda)\le(\alpha(n+1)-1)/n$. This choice is shown to imply $E[\mathrm{FDR}(S_{\mathrm{test}})]\le\alpha$; changing the denominator to the ground-truth defect area gives the analogous FNR control. On the Severstal and NEU steel datasets all tested backbone networks keep empirical FDR below the reference line for $\alpha\in[0.1,0.9]$, and an ablation across calibration-to-test split ratios reports FNR control at every split.
Load-bearing premise
The guarantee depends on the per-image false-discovery loss never increasing when the detection threshold is relaxed; the paper assumes this monotonicity without proving it or checking it on data.
Editorial extensions
If this is right
- Users can specify a maximum acceptable FDR or FNR before deployment and receive a confidence threshold that is guaranteed in expectation on new data from the same source.
- The same calibration pipeline works on top of any segmentation network that outputs per-pixel confidence scores, not only Mask R-CNN.
- At a fixed risk level, average prediction-set size becomes a model-comparison signal: a smaller set indicates a sharper feature representation, as the paper observes for ResNet-50 against lightweight backbones.
- The FNR guarantee survives changes in calibration-to-test split ratio, so practitioners only need a calibration set from the same distribution, not a precisely tuned split.
- Because the threshold adapts to each model and dataset, the framework supports deploying one detector across production lines with controlled error rates instead of hand-tuned confidence cutoffs.
Reading between the lines
- If prediction-set size is read as an uncertainty metric, the reported negative correlation between set size and model parameters could become a practical model-selection heuristic for choosing among backbones; the paper does not establish why the correlation holds.
- The same CRC construction should extend to per-class guarantees by defining separate FDR/FNR losses for each defect category, giving users a bound on each defect type rather than a pooled bound; this is a direct but untested extension.
- A deliberate stress test with corrupted or shifted test images would show how quickly the guarantee degrades when the exchangeability premise is violated, since the paper only evaluates clean calibration/test splits.
- The assumed monotonicity of the loss in the threshold is cheap to verify empirically by plotting $l_i(\lambda)$ over a grid; if real images produce non-monotone losses, the finite-sample guarantee would need an additional bounded-rate-of-change condition.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper applies Conformal Risk Control (CRC) to Mask R-CNN-based pixel-wise defect segmentation on two steel surface defect datasets (Severstal and NEU). It defines per-image FDR and FNR loss functions, computes an empirical risk on a calibration set, selects a data-dependent confidence threshold via Eq. (8), and claims that the expected FDR/FNR on test images is bounded by a user-specified level α. Experiments report empirical FDR/FNR curves for six backbone networks, prediction-set-size analyses, and an ablation study over calibration-to-test split ratios.
Significance. If the central guarantee holds, the paper provides a practically useful, model-agnostic procedure for controlling a user-specified error rate in industrial defect segmentation, which is a meaningful step beyond raw detection accuracy. The empirical evaluation covers two real datasets and multiple backbones, and the paper explicitly checks that empirical error rates stay below the target line. The paper does not claim new conformal theory; its contribution is an application of the existing CRC framework plus a prediction-set-size diagnostic. However, the paper does not provide code or machine-checked proofs, and the theoretical derivation contains a load-bearing gap that needs to be resolved before the main claim is established.
major comments (2)
- [Section 3.2, Eqs. (5) and (8)] The assertion that l_i(λ) 'monotonically decreases with λ' is not proved and is not generally true for the FDR loss defined in Eq. (5). Since C_i(λ) = {(j,k): f(x_i)_{j,k} ≥ 1−λ}, increasing λ lowers the confidence threshold and expands the prediction set; each added false-positive pixel increases the false-discovery fraction, while each added true-positive pixel decreases it. Nothing in the problem guarantees that true positives are encountered before false positives as λ grows, so l_i(λ) can be non-monotone. This matters because the CRC theorem used to pass from Eq. (8) to Eq. (9) requires the per-sample losses to be monotone nonincreasing in λ; without this condition, the data-dependent threshold λ̂ is not covered by the guarantee. The authors should either prove monotonicity for this specific loss, verify it empirically on the calibration set (e.g., plot l_i(λ) versus λ for a sample of calibration images), or switch to a monotone loss function.
- [Section 3.2, Eqs. (7) and (9)] As written, these equations equate an expectation to a random variable. The left side E[l_{n+1}(λ)] is a deterministic number, while the right side (nL_n(λ)+l_{n+1}(λ))/(n+1) depends on the calibration sample and on the test loss. The correct argument uses exchangeability of the n+1 losses, for instance E[(Σ_{i=1}^{n+1} l_i(λ))/(n+1)] = E[l_{n+1}(λ)], and the proof of the threshold inequality must be stated with expectations over the appropriate random quantities. The current derivation is not a valid proof of the central E[FDR(S_test)]≤α claim.
minor comments (6)
- [Abstract and Section 4.5.2] The abstract and conclusion state a 'negative correlation between the average prediction set size and the risk level,' but Section 4.5.2 reports a 'significant nonlinear positive correlation' with r = 0.91. The sign of the correlation should be corrected to match the experimental result.
- [Section 3.3] The text says the optimal threshold is determined via 'constrained optimization as detailed in Eq. 3,' but Eq. (3) is the Mask R-CNN multi-task loss; the intended reference is Eq. (8).
- [Section 4.4.2, Eq. (13)] Equation (13) is poorly formatted: the denominator 'P y*_i' is not clearly defined, and the set notation should explicitly refer to pixels rather than image indices.
- [Section 4.6.2] The statement that the optimal λ follows 'the theoretical threshold update mechanism λ* = 1−α/k' is not derived from Eq. (8) and does not match the CRC quantile-based formula; the parameter k is not defined. This should be removed or properly derived.
- [Section 3.2, Eq. (8)] For small calibration sizes or small α, the right-hand side (α(n+1)−1)/n can be negative, making the infimum in Eq. (8) empty over the stated range of λ. The domain of λ and the convention for an empty set should be specified.
- [Throughout] There are several typos and wording issues, including 'strictly bounced' (should be 'bounded'), 'residul' (should be 'residual'), and 'effieciency' (should be 'efficiency').
Circularity Check
No circularity: the FDR guarantee is an application of the external CRC theorem with calibration- and test-set separation, not a reduction of the conclusion into its inputs.
full rationale
The paper's derivation is a direct application of Conformal Risk Control (Angelopoulos et al., ref. [11]) to surface defect segmentation. The threshold lambda-hat is selected from the calibration set through Eq. 8 using the empirical risk L_n(lambda); the guarantee in Eq. 9 is the CRC transfer bound that follows from exchangeability, an external theorem not constructed by the present authors. The per-sample FDR loss in Eq. 5 is the standard false-discovery proportion and is not defined in terms of the test-set quantity it is later used to bound. No parameter is fitted to test data and then reported as a prediction: the empirical evaluations in Sec. 4.4 are performed on a separate test set, so the fitted-input-called-prediction pattern does not apply. There are no self-citations carrying the argument; the cited CRC and segmentation works are independent prior literature. The asserted monotonicity of l_i(lambda) in lambda is not proved and may fail in general, which is a correctness risk rather than a circularity, since the claim does not reduce to its own inputs by definition. Likewise, the abstract's wording about a negative correlation between prediction set size and risk level conflicts with the reported positive correlation r = 0.91 in Sec. 4.5.2, but that is an internal inconsistency, not a circular derivation. Because the central guarantee rests on an external theorem and a calibration-to-test split, the paper is self-contained against external benchmarks and receives a non-circularity score of 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Calibration and test data are i.i.d. (or at least exchangeable).
- ad hoc to paper The FDR loss l_i(λ) is monotonically nonincreasing in λ.
- standard math Conformal Risk Control theorem of Angelopoulos et al. [11] is valid and applicable.
- standard math Loss values remain in [0,1].
Cite this review
Pith. "Pith review of Conformal Segmentation in Industrial Surface Defect Detection with Statistical Guarantees." pith.science (2026). https://pith.science/paper/FEQW5MXM
@misc{pith2026250417721,
author = {Pith},
title = {Pith review of: Conformal Segmentation in Industrial Surface Defect Detection with Statistical Guarantees},
year = {2026},
howpublished = {\url{https://pith.science/paper/FEQW5MXM}},
note = {Machine review of arXiv:2504.17721}
}
read the original abstract
In industrial settings, surface defects on steel can significantly compromise its service life and elevate potential safety risks. Traditional defect detection methods predominantly rely on manual inspection, which suffers from low efficiency and high costs. Although automated defect detection approaches based on Convolutional Neural Networks(e.g., Mask R-CNN) have advanced rapidly, their reliability remains challenged due to data annotation uncertainties during deep model training and overfitting issues. These limitations may lead to detection deviations when processing the given new test samples, rendering automated detection processes unreliable. To address this challenge, we first evaluate the detection model's practical performance through calibration data that satisfies the independent and identically distributed (i.i.d) condition with test data. Specifically, we define a loss function for each calibration sample to quantify detection error rates, such as the complement of recall rate and false discovery rate. Subsequently, we derive a statistically rigorous threshold based on a user-defined risk level to identify high-probability defective pixels in test images, thereby constructing prediction sets (e.g., defect regions). This methodology ensures that the expected error rate (mean error rate) on the test set remains strictly bounced by the predefined risk level. Additionally, we observe a negative correlation between the average prediction set size and the risk level on the test set, establishing a statistically rigorous metric for assessing detection model uncertainty. Furthermore, our study demonstrates robust and efficient control over the expected test set error rate across varying calibration-to-test partitioning ratios, validating the method's adaptability and operational effectiveness.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[11]
Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. arXiv preprint arXiv:2208.02814, 2022
arXiv 2022
-
[1]
Y . Liu, X. Zhang, and J. Wang. Deep learning-based defect detection in steel manufacturing: A review. IEEE Transactions on Industrial Informatics, 17(5):3061–3072, 2021
work page 2021
-
[2]
H. Wang, J. Li, and F. Zhou. Deep learning for industrial defect detection: A comprehensive review. Pattern Recognition, 107:107254, 2020
work page 2020
-
[3]
Yiming Xiao, Haidong Shao, Minjie Feng, Te Han, Jiafu Wan, and Bin Liu. Towards trustworthy rotating machinery fault diagnosis via attention uncertainty in transformer.Journal of Manufacturing Systems, 70:186–201, 2023. 11 Under Review
work page 2023
-
[4]
A survey of uncertainty in deep neural networks
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56(Suppl 1):1513–1589, 2023
2023
-
[5]
Zhiyuan Wang, Jinhao Duan, Chenxi Yuan, Qingyu Chen, Tianlong Chen, Yue Zhang, Ren Wang, Xiaoshuang Shi, and Kaidi Xu. Word-sequence entropy: Towards uncertainty estimation in free-form medical question answering applications and beyond. Engineering Applications of Artificial Intelligence, 139:109553, 2025
work page 2025
-
[6]
Steel surface defect detection using a new haar–weibull-variance model in unsupervised manner
Kun Liu, Heying Wang, Haiyong Chen, Erqing Qu, Ying Tian, and Hexu Sun. Steel surface defect detection using a new haar–weibull-variance model in unsupervised manner. IEEE transactions on instrumentation and measurement, 66(10):2585–2596, 2017
work page 2017
-
[7]
Maoxiang Chu and Rongfen Gong. Invariant feature extraction method based on smoothed local binary pattern for strip steel surface defect. ISIJ International, 55(9):1956–1962, 2015
work page 1956
Show all 26 references
-
[8]
Edrnet: Encoder–decoder residual network for salient object detection of strip steel surface defects
Guorong Song, Kechen Song, and Yunhui Yan. Edrnet: Encoder–decoder residual network for salient object detection of strip steel surface defects. IEEE Transactions on Instrumentation and Measurement, 69(12):9709– 9719, 2020
2020
-
[9]
Automatic surface defect segmentation for hot-rolled steel strip using depth-wise separable u-shape network
Zheng Huang, Jiajun Wu, and Feng Xie. Automatic surface defect segmentation for hot-rolled steel strip using depth-wise separable u-shape network. Materials Letters, 301:130271, 2021
2021
-
[10]
Machine-learning applications of algorithmic randomness
V olodya V ovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of algorithmic randomness. 1999
1999
-
[12]
Conu: Conformal uncertainty in large language models with correctness coverage guarantees
Zhiyuan Wang, Jinhao Duan, Lu Cheng, Yue Zhang, Qingni Wang, Xiaoshuang Shi, Kaidi Xu, Heng Tao Shen, and Xiaofeng Zhu. Conu: Conformal uncertainty in large language models with correctness coverage guarantees. In Findings of the Association for Computational Linguistics: EMNL...
2024
-
[13]
Conformal risk control for pulmonary nodule detection
Roel Hulsman, Valentin Comte, Lorenzo Bertolini, Tobias Wiesenthal, Antonio Puertas Gallardo, and Mario Ceresa. Conformal risk control for pulmonary nodule detection. arXiv preprint arXiv:2412.20167, 2024
2024
-
[14]
Sample then identify: A general framework for risk control and assessment in multimodal large language models
Qingni Wang, Tiantian Geng, Zhiyuan Wang, Teng Wang, Bo Fu, and Feng Zheng. Sample then identify: A general framework for risk control and assessment in multimodal large language models. In The Thirteenth International Conference on Learning Representations, 2025
2025
-
[15]
Sconu: Selective conformal uncertainty in large language models
Zhiyuan Wang, Qingni Wang, Yue Zhang, Tianlong Chen, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. Sconu: Selective conformal uncertainty in large language models. arXiv preprint arXiv:2504.14154, 2025
2025 arXiv
-
[16]
An electronic nose-based assistive diagnostic prototype for lung cancer detection with conformal prediction
Xianghao Zhan, Zhan Wang, Meng Yang, Zhiyuan Luo, You Wang, and Guang Li. An electronic nose-based assistive diagnostic prototype for lung cancer detection with conformal prediction. Measurement, 158:107588, 2020
2020
-
[17]
Uncertainty-aware real-time visual anomaly detection with conformal prediction in dynamic indoor environments
Arya Saboury and Mustafa Kemal Uyguroglu. Uncertainty-aware real-time visual anomaly detection with conformal prediction in dynamic indoor environments. IEEE Robotics and Automation Letters, 2025
2025
-
[18]
Conformal semantic image segmentation: Post-hoc quantifi- cation of predictive uncertainty
Luca Mossina, Joseba Dalmau, and Léo Andéol. Conformal semantic image segmentation: Post-hoc quantifi- cation of predictive uncertainty. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3574–3584, 2024
2024
-
[19]
Confident object detection via conformal prediction and conformal risk control: an application to railway signaling
Léo Andéol, Thomas Fel, Florence De Grancey, and Luca Mossina. Confident object detection via conformal prediction and conformal risk control: an application to railway signaling. In Conformal and Probabilistic Prediction with Applications, pages 36–55. PMLR, 2023
2023
-
[20]
Statistical guarantees of false discovery rate in medical instance segmentation tasks based on conformal risk control
Mengxia Dai, Wenqian Luo, and Tianyang Li. Statistical guarantees of false discovery rate in medical instance segmentation tasks based on conformal risk control. arXiv preprint arXiv:2504.04482, 2025
2025 arXiv
-
[21]
Mask r-cnn
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2961–2969, 2017
2017
-
[22]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016
2016
-
[23]
Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size
Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016
2016 arXiv
-
[24]
Shufflenet v2: Practical guidelines for efficient cnn architecture design
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 116–131, 2018. 12 Under Review
2018
-
[25]
Searching for mobilenetv3
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019
2019
-
[26]
Ghostnetv3: Exploring the training strategies for compact models
Zhenhua Liu, Zhiwei Hao, Kai Han, Yehui Tang, and Yunhe Wang. Ghostnetv3: Exploring the training strategies for compact models. arXiv preprint arXiv:2404.11202, 2024. 13
2024 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.