REVIEW 3 major objections 6 minor 1 cited by
Robust Vision-Based Runway Detection through Conformal Prediction and Conformal mAP
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Conformal prediction gives runway detectors a statistical containment guarantee, raising a new strict C-mAP metric from under 2% to over 53% while keeping mAP above 92%.
desk verdict Standard conformal prediction applied to runway detection; the new C-mAP metric is sensible but the calibration/evaluation split is under-specified and that ambiguity is load-bearing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the conformal bounding box, constructed from coordinate-wise nonconformity scores. For each side of a matched predicted box, the signed difference (additive) or relative difference (multiplicative) to the ground-truth box is recorded over the calibration set, and the Bonferroni-corrected quantile at level 1 - alpha/4 is used to expand that side at inference. This expansion is what converts a raw YOLO box into a conformal box with a containment guarantee. The companion object is the C-AP/C-mAP matching rule, which requires both IoU >= t and IoA = 1, where IoA is the intersection divided by the ground-truth area; this forces a detection to fully enclose the runway, matching the downstream need for complete runway crops in pose estimation.
What would settle it
Split a held-out runway set into two disjoint halves, calibrate the per-side quantiles on one half, then evaluate containment on the other half at alpha = 0.3; if the empirical coverage falls below 70% on the untouched half, the reported guarantee is not reproducible under a clean calibration/evaluation separation.
Extended reading notes
Core claim
The central claim is that conformal prediction, applied to the four coordinates of a detected bounding box, produces a conformal box that contains the true runway box with probability at least 1 - alpha, without retraining the underlying detector. The method computes per-side nonconformity scores (additive or multiplicative) between matched predicted and ground-truth boxes on a calibration set, takes Bonferroni-corrected quantiles at level alpha/4 for each side, and expands new predictions accordingly. At alpha = 0.3, the paper reports empirical containment rates of 73.93–77.06% across the four conformalized YOLO variants. The same procedure raises the proposed C-mAP score from 0.77–1.31% for the base YOLO models to 52.71–56.86% for the conformalized versions, while standard mAP remains above 92% in every case. The paper also reports that conformalization expands predicted boxes modestly, with a Stretch ratio between 1.11 and 1.16 and average side margins under 19 pixels.
Load-bearing premise
The coverage guarantee depends on the calibration and test images being exchangeable, i.e., drawn from the same runway-approach conditions, and the paper does not clearly demonstrate a disjoint calibration/evaluation split on its 20% test subset.
Editorial extensions
If this is right
- At the reported alpha = 0.3, a vision-based landing system can treat each conformal runway box as containing the true runway with about 74–77% empirical reliability, giving pose estimation a quantitative safety margin.
- Operators can trade coverage against box size by lowering alpha: smaller alpha raises both coverage and C-mAP but enlarges the conformal boxes further.
- The C-mAP metric provides a single-number evaluation that rewards full enclosure alongside localization, so detector comparisons for landing tasks can reflect downstream requirements rather than IoU overlap alone.
- Because conformalization is applied after training, existing YOLOv5 and YOLOv6 runway detectors can be retrofitted with statistical guarantees without retraining or architectural changes.
Reading between the lines
- If the calibration pool matches real landing approaches, the same coordinate-wise conformalization could be applied to other critical detections in the aircraft pipeline, such as obstacles or ground markers, where full containment is more important than tight boundaries.
- The reported coverage (74–77%) exceeds the 70% target, which suggests the Bonferroni correction is conservative; a method that controls the joint coverage risk more efficiently could obtain the same guarantee with smaller boxes.
- The paper's experimental description says conformal experiments are run on 'the remaining 20% of the test set,' which leaves it unclear whether calibration and evaluation are disjoint; if both are drawn from the same test pool, the advertised out-of-sample coverage could be overstated.
- A natural next test is multi-object scenes: with several runways or false alarms per image, independent per-box conformalization may produce overlapping or inflated boxes that distort C-mAP, exactly the regime the paper names as a limitation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper applies conformal prediction to runway detection in aerial imagery, using fine-tuned YOLOv5 and YOLOv6 models on the LARD dataset. It uses coordinate-wise additive and multiplicative nonconformity scores with a Bonferroni correction, following de Grancey et al., and reports empirical coverage between 73.93% and 77.06% at alpha=0.3. It also introduces a new metric, Conformal mAP (C-mAP), in which a detection is a true positive only if IoU is above a threshold and IoA equals 1, and reports that conformalized models raise C-mAP from below 2% to roughly 53-57% while keeping standard mAP above 92%. The paper argues that conformal prediction provides statistical guarantees useful for vision-based landing systems.
Significance. If the calibration/evaluation protocol is properly specified, the paper is a useful application case study of conformal prediction in aerospace perception: it follows an established methodology, uses a public dataset, and releases code and trained models. The reported coverage values are internally consistent with the Bonferroni-corrected target, and the introduction of C-mAP is a simple but application-relevant evaluation metric for downstream tasks that require full containment of the runway. The incremental methodological novelty is modest, since the conformal procedure is imported from prior work and the C-mAP metric is a direct modification of mAP, but the application context and the empirical comparison of additive versus multiplicative penalties add value for the COPA community.
major comments (3)
- [Section 5, Dataset paragraph; Tables 3 and 5] The calibration split for computing the conformal quantiles is never specified. The text states that "the experiments on conformal prediction are done on the remaining 20% of the test set," but it does not state where the nonconformity-score quantiles in Section 4.2 are computed. A split-conformal coverage guarantee requires that calibration and evaluation samples be exchangeable and that the quantiles be fixed before seeing the evaluation boxes. If the same 20% subset was used for both calibration and evaluation, the coverage numbers in Table 5 are an in-sample check and provide no evidence for the out-of-sample guarantee advertised in Section 4.2; if the other 80% of the test set was used for calibration, that should be stated with sample sizes. The choice also determines whether the real-image results are in-distribution, since real images appear only in the test set. This ambiguity is load-bearing for the central numerical claims.
- [Section 5.2, coverage definition; Section 4.2, formal guarantee] Coverage is reported as "the proportion of predicted boxes that cover their associated ground truth," which conditions on a detection existing. The formal statement in Section 4.2, P(b_gt_{n+1} is contained in b_conf_{n+1}) >= 1-alpha, applies to a paired prediction and ground-truth box; if the detector outputs no box for an image, there is no conformalized box and that ground truth is excluded from the denominator. The paper should state this conditioning explicitly and, ideally, report the detection failure rate and unconditional coverage over all test ground truths, since the downstream VLS pipeline needs a box for every approach image.
- [Section 4.2 and Table 3] The C-mAP gain is partly by construction. The C-AP true-positive condition is IoU >= t and IoA = 1, and conformalization is precisely a procedure that expands boxes to enforce containment; therefore the jump in C-mAP from 0.77-1.31% to 52.71-56.86% is expected. The paper should not present this increase as independent evidence of improved reliability. The primary evidence should be empirical coverage relative to the target 1-alpha, together with the cost in standard mAP and box inflation. The metric remains useful as an application-specific evaluation criterion, but the paper should discuss how its definition aligns with the conformalization objective when interpreting the results.
minor comments (6)
- [Section 5.2] The sentence "Our next study will compare the Conformal Average precision of all models" appears to be a leftover from an outline; it should be revised to "We next compare...".
- [Table 5] The header "squared root of box area" should be "square root of box area".
- [Figures 8 and 9 captions] The captions use "YOLO'S predictions"; the possessive should be "YOLO's predictions" or, more simply, "YOLO predictions".
- [Section 2.1] The notation "mAP@50:95" should be "mAP@0.5:0.95" to match the table and the text.
- [Section 5.2, metric definitions] The phrase "across three metrics: standard mean Average Precision (mAP), Conformal mean Average Precision (C-mAP, naturally extended from C-mAP)" contains a typo; the second occurrence should refer to C-AP or a correctly named extension.
- [Abstract and Conclusion] The abstract's claim about "paving the way for certification" is stronger than the evidence presented; the conclusion appropriately limits the claims, so the abstract should be aligned with that more cautious framing.
Circularity Check
No material circularity: the conformal coverage results apply a cited external CP theorem, and C-mAP transparently measures the containment property conformalization is designed to provide.
full rationale
The central quantitative claims (coverage 73.93–77.06% at α=0.3) are produced by split-conformal quantiles: coordinate-wise nonconformity score vectors are computed between predicted and ground-truth boxes, Bonferroni-corrected quantiles q̂(j) are formed at level α/4, and boxes are expanded by those quantiles. No parameter is fitted to C-mAP or to the coverage value; the 1−α containment guarantee is a cited theorem from de Grancey et al. (2022), not re-derived or assumed by the present authors. The observed over-coverage is consistent with Bonferroni conservatism and is empirically checked on test boxes. C-mAP is explicitly defined as IoU≥t and IoA=1 (full containment), which is the same containment event the conformal procedure is constructed to control; the paper openly states this alignment in Section 4.2. The large C-mAP gain therefore largely reflects the metric rewarding the property conformalization enforces, but the gain is not forced by construction alone because it also requires IoU≥0.5 to survive conformal expansion and a workable precision–recall ranking; Table 3 shows mAP only drops to roughly 92–96%, an empirical, non-tautological fact. Self-citations (LARD dataset, Ducoffe et al. 2023; Stretch from Andéol et al. 2023) are descriptive and not load-bearing. The only serious concern is experimental reporting: Section 5 says CP experiments are done on 'the remaining 20% of the test set' without stating where calibration quantiles are computed. If the same 20% were used for both calibration and evaluation, Table 5 would be an in-sample check and the advertised out-of-sample guarantee would be untested; the paper's conclusion even concedes that guarantees 'rely on calibration distributions matching test conditions.' That ambiguity is a reproducibility/validity flaw, not a demonstrated circularity, so it does not raise the circularity score beyond the minor self-citation baseline.
Assumptions & free parameters
free parameters (3)
- alpha (risk level) =
0.3
- IoU threshold t in C-mAP =
0.5
- IoA thresholds for C-mAP@50@80:100 =
0.8 to 1.0
assumptions (3)
- domain assumption Exchangeability of calibration and test data
- domain assumption Validity of the 4D Bonferroni-corrected quantile method from de Grancey et al. (2022)
- domain assumption Hungarian matching correctly pairs predicted and ground-truth boxes for nonconformity computation
invented entities (1)
-
Conformal mAP (C-mAP)
Cite this review
Pith. "Pith review of Robust Vision-Based Runway Detection through Conformal Prediction and Conformal mAP." pith.science (2026). https://pith.science/paper/ZKVBVY7T
@misc{pith2026250516740,
author = {Pith},
title = {Pith review of: Robust Vision-Based Runway Detection through Conformal Prediction and Conformal mAP},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZKVBVY7T}},
note = {Machine review of arXiv:2505.16740}
}
read the original abstract
We explore the use of conformal prediction to provide statistical uncertainty guarantees for runway detection in vision-based landing systems (VLS). Using fine-tuned YOLOv5 and YOLOv6 models on aerial imagery, we apply conformal prediction to quantify localization reliability under user-defined risk levels. We also introduce Conformal mean Average Precision (C-mAP), a novel metric aligning object detection performance with conformal guarantees. Our results show that conformal prediction can improve the reliability of runway detection by quantifying uncertainty in a statistically sound way, increasing safety on-board and paving the way for certification of ML system in the aerospace domain.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 1 Pith paper
-
Conformal Prediction Sets for Instance Segmentation
A conformal method outputs a set of candidate instance masks per pixel query, guaranteeing that with probability ≥1−α at least one mask has IoU above a threshold.
Reference graph
Works this paper leans on
-
[1]
L \'e o And \'e ol, Thomas Fel, Florence De Grancey, and Luca Mossina. Confident object detection via conformal prediction and conformal risk control: an application to railway signaling. In Conformal and Probabilistic Prediction with Applications, pages 36--55. PMLR, 2023
work page 2023
-
[2]
Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. In The Twelfth International Conference on Learning Representations, 2024
work page 2024
-
[3]
Neural network based runway landing guidance for general aviation autoland
Giovanni Balduzzi, Martino Ferrari Bravo, Anna Chernova, Calin Cruceru, Luuk van Dijk, Peter de Lange, Juan Jerez, Nathana \"e l Koehler, Mathias Koerner, Corentin Perret-Gentil, et al. Neural network based runway landing guidance for general aviation autoland. Technical report, United States. Department of Transportation. Federal Aviation Administration …, 2021
work page 2021
-
[4]
Deep learning techniques—r-cnn to mask r-cnn: a survey
Puja Bharati and Ankita Pramanik. Deep learning techniques—r-cnn to mask r-cnn: a survey. Computational Intelligence in Pattern Recognition: Proceedings of CIPR 2019, pages 657--668, 2020
work page 2019
-
[5]
End-to-end object detection with transformers
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213--229. Springer, 2020
2020
-
[6]
The uncertain object: Application of conformal prediction to aerial and satellite images
Vicky Copley, Greg Finlay, and Ben Hiett. The uncertain object: Application of conformal prediction to aerial and satellite images. In The 13th Symposium on Conformal and Probabilistic Prediction with Applications, pages 73--89. PMLR, 2024
work page 2024
-
[7]
Wei Dai, Zhengjun Zhai, Dezhong Wang, Zhaozi Zu, Siyuan Shen, Xinlei Lv, Sheng Lu, and Lei Wang. Yomo-runwaynet: A lightweight fixed-wing aircraft runway detection algorithm combining yolo and mobilerunwaynet. Drones, 8 0 (7): 0 330, 2024
work page 2024
-
[8]
Object detection with probabilistic guarantees
Florence de Grancey, Jean-Luc Adam, Lucian Alecu, S \'e bastien Gerchinovitz, Franck Mamalet, and David Vigouroux. Object detection with probabilistic guarantees. In Fifth International Workshop on Artificial Intelligence Safety Engineering (WAISE 2022), 2022
work page 2022
Show all 26 references
-
[9]
A high probability safety guarantee for shifted neural network surrogates
M \'e lanie Ducoffe, S \'e bastien Gerchinovitz, and Jayant Sen Gupta. A high probability safety guarantee for shifted neural network surrogates. In SafeAI@ AAAI, pages 74--82, 2020
2020
-
[10]
Lard--landing approach runway detection--dataset for vision based landing
M \'e lanie Ducoffe, Maxime Carrere, L \'e o F \'e liers, Adrien Gauffriau, Vincent Mussot, Claire Pagetti, and Thierry Sammour. Lard--landing approach runway detection--dataset for vision based landing. arXiv preprint arXiv:2304.09938, 2023
2023 arXiv
-
[11]
Applying the conformal prediction paradigm for the uncertainty quantification of an end-to-end automatic speech recognition model (wav2vec 2.0)
Fares Ernez, Alexandre Arnold, Audrey Galametz, Catherine Kobus, and Nawal Ould-Amer. Applying the conformal prediction paradigm for the uncertainty quantification of an end-to-end automatic speech recognition model (wav2vec 2.0). In Conformal and Probabilistic Prediction with...
2023
-
[12]
Bayesod: A bayesian approach for uncertainty estimation in deep object detectors
Ali Harakeh, Michael Smart, and Steven L Waslander. Bayesod: A bayesian approach for uncertainty estimation in deep object detectors. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 87--93. IEEE, 2020
2020
-
[13]
ultralytics/yolov5: v3
Glenn Jocher, Alex Stoken, Jirka Borovec, Liu Changyu, Adam Hogan, Laurentiu Diaconu, Jake Poznanski, Lijun Yu, Prashant Rai, Russ Ferriday, et al. ultralytics/yolov5: v3. 0. Zenodo, 2020
2020
-
[14]
Yolov6: A single-stage object detection framework for industrial applications
Chuyi Li, Lulu Li, Hongliang Jiang, Kaiheng Weng, Yifei Geng, Liang Li, Zaidan Ke, Qingyuan Li, Meng Cheng, Weiqiang Nie, et al. Yolov6: A single-stage object detection framework for industrial applications. arXiv preprint arXiv:2209.02976, 2022
2022 arXiv
-
[15]
Probabilistic object detection via deep ensembles
Zongyao Lyu, Nolan Gutierrez, Aditya Rajguru, and William J Beksi. Probabilistic object detection via deep ensembles. In European Conference on Computer Vision, pages 67--75. Springer, 2020
2020
-
[16]
Efficient robust conformal prediction via lipschitz-bounded networks
Thomas Massena, L \'e o And \'e ol, Thibaut Boissin, Corentin Friedrich, Franck Mamalet, Mathieu Serrurier, and S \'e bastien Gerchinovitz. Efficient robust conformal prediction via lipschitz-bounded networks. 2025
2025
-
[17]
Puncc: a python library for predictive uncertainty calibration and conformalization
Mouhcine Mendil, Luca Mossina, and David Vigouroux. Puncc: a python library for predictive uncertainty calibration and conformalization. In Conformal and Probabilistic Prediction with Applications, pages 582--601. PMLR, 2023
2023
-
[18]
Dropout sampling for robust object detection in open-set conditions
Dimity Miller, Lachlan Nicholson, Feras Dayoub, and Niko S \"u nderhauf. Dropout sampling for robust object detection in open-set conditions. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3243--3249. IEEE, 2018
2018
-
[19]
Efficient non-maximum suppression
Alexander Neubeck and Luc Van Gool. Efficient non-maximum suppression. In 18th international conference on pattern recognition (ICPR'06), volume 3, pages 850--855. IEEE, 2006
2006
-
[20]
Intriguing properties of neural networks
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In The Second International Conference on Learning Representations, 2014
2014
-
[21]
A comprehensive review of yolo architectures in computer vision: From yolov1 to yolov8 and yolo-nas
Juan Terven, Diana-Margarita C \'o rdova-Esparza, and Julio-Alejandro Romero-Gonz \'a lez. A comprehensive review of yolo architectures in computer vision: From yolov1 to yolov8 and yolo-nas. Machine learning and knowledge extraction, 5 0 (4): 0 1680--1716, 2023
2023
-
[22]
Fcos: A simple and strong anchor-free object detector
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: A simple and strong anchor-free object detector. IEEE transactions on pattern analysis and machine intelligence, 44 0 (4): 0 1922--1933, 2020
1922
-
[23]
Adaptive bounding box uncertainties via two-step conformal prediction
Alexander Timans, Christoph-Nikolas Straehle, Kaspar Sakmann, and Eric Nalisnick. Adaptive bounding box uncertainties via two-step conformal prediction. In European Conference on Computer Vision, pages 363--398. Springer, 2024
2024
-
[24]
Algorithmic learning in a random world, volume 29
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world, volume 29. Springer, 2005
2005
-
[25]
Confidence score: The forgotten dimension of object detection performance evaluation
Simon Wenkel, Khaled Alhazmi, Tanel Liiv, Saud Alrshoud, and Martin Simon. Confidence score: The forgotten dimension of object detection performance evaluation. Sensors, 21 0 (13): 0 4350, 2021
2021
-
[26]
Monte carlo dropblock for modeling uncertainty in object detection
Sai Harsha Yelleni, Deepshikha Kumari, et al. Monte carlo dropblock for modeling uncertainty in object detection. Pattern Recognition, 146: 0 110003, 2024
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.