{"id":"350338aa-1bcb-426b-b747-0914867deaca","arxiv_id":"2508.09732","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Runway keypoint detection with calibrated uncertainty estimates and a RAIM-based geometric check that flags inconsistent predictions.","lead":"This paper describes a vision system that locates runway corners in landing images while also predicting how confident it is in each detection. It adds a geometric consistency check, adapted from GPS integrity monitoring, that can reject faulty outputs before they are used for navigation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The runtime-integrity claim rests on an independent-Gaussian error model that the authors concede can fail; a common-mode keypoint error is geometrically unobservable to the RAIM test, so fault detection is not demonstrated for correlated failure modes.","rationale":"I read the paper as a practical engineering contribution rather than a formal certification argument. The accuracy and calibration claims are supported by experiments on a validation set, though the lack of a held-out test set and the 20% recalibration are limitations. The most load-bearing concern is the RAIM assumption of independent Gaussian keypoint errors. This is not an internal inconsistency; the authors are transparent about the degradation under correlated errors. However, it is load-bearing because the abstract and introduction claim runtime detection and rejection of faulty outputs without qualifying that the method only detects faults incompatible with the runway shape under an independence assumption. A common-mode translation of all keypoints is a plausible failure mode that can be absorbed by the pose parameters, and the current empirical validation (Fig. 5) only checks the marginal distribution of nominal statistics, not the behavior under correlated faults. The concrete test proposed would settle whether the RAIM test retains its detection power under realistic error correlations. Since the reader already marked the paper CONDITIONAL and identified the same weakest assumption, my stress-test does not change the verdict; it reinforces the condition that correlated-error performance must be demonstrated.","tokens_in":12027,"tokens_out":3091,"duration_ms":40098,"concrete_test":"On the validation set, compute per-keypoint prediction errors relative to ground truth and estimate the correlation matrix across keypoints and coordinate axes. Then run a Monte Carlo test: (i) draw errors from a Gaussian with this empirical correlation and the predicted diagonal variances, compute the Algorithm 1 statistic, and compare the false-alarm threshold against the chi-square quantile; (ii) inject a common 2D shift applied to all four keypoints, with magnitude comparable to observed errors, and measure detection probability at the target false-alarm rate. If detection probability is not substantially above the false-alarm rate, the RAIM test fails for a plausible correlated fault mode.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim includes automatic detection and rejection of faulty predictions via RAIM (Algorithm 1). The test statistic in step 6 is treated as chi-squared with n-m degrees of freedom. This is valid only if, after projection linearization, the weighted residuals are independent standard Gaussians. The predicted covariance Sigma is diagonal per keypoint coordinate (Sec. IV-B), so any correlation between keypoints or between x/y errors is unmodeled. The paper's own conclusion (Sec. VI) concedes that performance degrades as pixel-error correlation increases, and that perfectly correlated errors make faults undetectable. This is not a remote edge case: a common-mode error such as all four predicted corners shifted by the same image translation can be absorbed by a perturbed pose (a translation of the aircraft and/or small rotation), leaving small reprojection residuals. The Fig. 5 demonstration only shows that the nominal statistic's marginal density matches chi-squared; it does not test the conditional distribution under correlated or non-Gaussian faults. Without quantifying the correlation structure of failures on the LARD data or another realistic benchmark, the claimed runtime assurance is conditional on an unverified independence assumption, weakening the central fault-detection claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a landing-scene pose-estimation pipeline for runway keypoints, combining a Soft Argmax (SAM) regression head with a heteroscedastic Gaussian negative-log-likelihood loss, and adds a RAIM-inspired geometric integrity check. The authors claim the SAM head outperforms fully-connected baselines in accuracy, the predicted uncertainties are well calibrated with sub-pixel precision, and the RAIM adaptation can detect and reject faulty keypoint predictions at runtime. Experiments use the LARD dataset with three CNN backbones and evaluate accuracy, calibration, sharpness, and nominal/off-nominal residual statistics.","tokens_in":12259,"tokens_out":9978,"duration_ms":117570,"significance":"If the claims hold, the paper offers a practical, real-time-compatible template for coupling per-keypoint predictive uncertainty with a geometric consistency test that does not require ground-truth annotations at runtime. The use of calibration curves and the explicit comparison to the theoretical chi-squared residual distribution are appropriate and transparent. The SAM + Gaussian-head design is simple and likely reusable across keypoint-based pose estimation tasks. However, the current strength of the claims is limited by three load-bearing gaps: the validation protocol does not use the official test split and may leak video-sequence information; the Algorithm 1 residual statistic is not the standard weighted RAIM residual and likely does not follow the claimed chi-squared distribution; and the fault-detection capability is demonstrated only on a single artificial failure mode despite the paper's own concession that correlated errors defeat the method.","major_comments":[{"comment":"The validation protocol is a random 80/20 split of the LARD training data; the provided 1261-image validation set is not used. If LARD contains video sequences or repeated airport approaches, a random frame split can place near-duplicate frames in both training and validation, inflating all reported metrics (Table I, Fig. 4, Fig. 5). Furthermore, the 'approximately 20% standard-deviation increase' for full calibration is fit to this same split, so the claimed calibration improvement is not evaluated on an independent set. Please report results on the official validation/test split or a sequence-disjoint split, and re-derive the calibration adjustment on that split.","section":"Section V.D"},{"comment":"For weighted least squares with covariance Σ and W=Σ^{-1}, the residual after pose estimation is e = y - h(θhat), and the weighted residual norm ||W^{1/2}e||² follows approximately χ²_{n-m} under the stated Gaussian model. Algorithm 1 instead computes r = (I - H H†)(yreproject - µ) with H† the unweighted pseudo-inverse of the Jacobian H. Since yreproject - µ = -e, this is -(I - H H†)e. When W is not a multiple of the identity, the projection (I - H H†) does not preserve the weighted residual distribution; the quadratic form ||L^{-1} r||² is not the standard RAIM test statistic and need not have the claimed χ²_{n-m} distribution. This undermines the threshold decision in steps 7-10. Please either define the residual as e = y - h(θhat) and use ||L^{-1}e||², or use the weighted projection P_W = H(H^TWH)^{-1}H^TW and derive the distribution of the resulting statistic explicitly.","section":"Algorithm 1, steps 5-6"},{"comment":"The fault-detection claim is central ('automatically detect and flag predictions that are incompatible with the known runway shape'), but the only off-nominal experiment is a single artificial 184 m shift of the far threshold. The paper's conclusion concedes that correlated pixel errors degrade RAIM and that perfectly correlated errors make faults undetectable. A common-mode error such as a uniform translation of all four corners is geometrically absorbed by a small pose perturbation and leaves tiny reprojection residuals, so the proposed test cannot detect it. Please add synthetic fault-injection experiments with controlled correlation structures (e.g., common-mode translation, scaled runway, single-keypoint outliers) and report detection probability versus false-alarm rate, or explicitly scope the runtime-assurance claim to uncorrelated/independent keypoint errors.","section":"Section VI and Fig. 5"}],"minor_comments":[{"comment":"The caption says 'Median Performance Results' but the columns are labeled 'NLL' and 'Mean Pixel Error.' The reported NLL values are negative, which is possible only if they are medians and the typical predicted uncertainty is well below one pixel. Please clarify which aggregate (mean/median) each column reports and include confidence intervals or per-image distributions.","section":"Table I"},{"comment":"The Jacobian is described as H ∈ R^{K×6}; since each projection contributes two coordinates, H should be R^{2K×6}. As written, the dimensions of H, r, and Σ do not align in steps 5-6.","section":"Algorithm 1"},{"comment":"The statement that nominal and off-nominal cases are 'clearly separated' would be stronger with quantitative separation metrics (e.g., AUC, minimum margin, or false-alarm rate at a chosen threshold).","section":"Section V.C"},{"comment":"There are several typographical issues: 'the simplicity of training which and model architecture', 'the validate the Gaussian error model assumption', and the 'Bogo crop' terminology. Please proofread.","section":"General"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a well-executed integration of known pieces—Soft Argmax keypoint regression, NLL-based heteroscedastic uncertainty, and a RAIM-style geometric consistency check—applied to runway landing. The new thing is the combination and the LARD evaluation, and the paper mostly delivers: sub-pixel accuracy on the crop, near-diagonal calibration, and a residual statistic that separates nominal from grossly erroneous predictions.\n\nWhat is worth credit: the empirical methodology is honest. The authors compare against a standard FC head across three backbones, report both NLL and pixel error, check calibration and sharpness, and validate the chi-squared assumption. The RAIM adaptation is a sensible transfer from GPS integrity monitoring, and the demonstration in Fig. 5 is clean as far as it goes.\n\nSoft spots: the RAIM test rests on an independent-Gaussian error model, and the authors themselves concede in the conclusion that correlated errors degrade detection, with perfectly correlated shifts being undetectable. That is not a remote corner case—a common-mode translation of all four corners can be absorbed by a pose perturbation and leave small residuals. The paper does not quantify detection rates or false alarms under realistic correlated failure modes; the off-nominal test is one artificial 184 m fault. Also, validation uses a 20% split of the training set rather than an independent test set, and 'sub-pixel' needs a concrete definition given the downsampling in the Bogo crop. The 20% standard-deviation inflation for full calibration is a post-hoc fit to the validation set, though the authors present it as optional, which is fair.\n\nWho this is for: people working on ML safety for aviation, specifically runtime assurance and uncertainty quantification. It is not a conceptual breakthrough, but it is a useful worked example of building an integrity monitor around a vision model, with limitations stated rather than hidden. The central claims are plausible and the flaws are addressable: a proper test set, quantitative fault-detection metrics, and a discussion of correlated-failure impact would strengthen it considerably.\n\nRecommendation: send it to peer review. It is a solid incremental contribution that deserves referee time, with the expectation of revision rather than desk rejection.","headline":"Solid engineering integration of Soft Argmax + NLL + RAIM for runway landing; the RAIM independence caveat is real but the authors already own it.","tokens_in":12800,"tokens_out":2453,"would_cite":true,"duration_ms":24421,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A real-time runway-keypoint detector can return calibrated, sub-pixel uncertainties, and an adapted GPS-style RAIM monitor can reject faulty predictions that are inconsistent with the known runway shape.","keywords":["predictive uncertainty","runtime assurance","RAIM","soft argmax","keypoint regression","calibration","pose estimation","autonomous landing"],"falsifier":"Generate a suite of runway images where the four corner predictions are corrupted by errors drawn from a multivariate Gaussian with increasing off-diagonal covariance, and measure the RAIM rejection rate at a fixed false-alarm threshold. Under the paper's assumptions the statistic remains chi-squared-distributed in nominal cases and the monitor flags incompatible geometry; if the nominal statistic already departs from chi-squared, or if rejection stays high under perfectly correlated errors, the independence and Gaussianity premise would be falsified.","tokens_in":11887,"feed_emoji":"🛬","tokens_out":6847,"duration_ms":72207,"temperature":0.7,"pith_summary":"The paper tries to establish that a vision-only landing system can do more than estimate runway corners: it can also say how sure it is about each corner, in pixel units, and use that calibrated uncertainty to reject predictions that no physically plausible aircraft pose can explain. The authors build a keypoint regressor with a spatial Soft Argmax head that reads out sub-pixel coordinates from any standard CNN backbone, train it with a negative log-likelihood loss so the predicted per-keypoint variances are faithful to actual errors, and then wrap the pose solver in an adaptation of the RAIM integrity test used in GPS navigation. On the paper's runway-image dataset, the SAM head beats fully-connected regression heads on accuracy while running at 30-60 Hz, and the RAIM-style residual statistic follows its theoretical chi-squared distribution in nominal cases while separating clearly from a simulated 184 m far-endpoint misprediction. If this holds up, a perception system could provide an integrity flag comparable to satellite navigation integrity monitoring, exactly the kind of runtime assurance that aviation certification frameworks ask for.","feed_headline":"A runway-vision pipeline gains GPS-style integrity checks for landings","feed_subtitle":"Per-keypoint uncertainties power a chi-squared check that rejects predictions no true aircraft pose can explain.","key_machinery":"Three pieces carry the argument. The spatial Soft Argmax operator turns each keypoint's activation heatmap into a probability distribution and returns the expected grid coordinate, enabling differentiable sub-pixel regression with almost no added parameters. The negative log-likelihood loss with a diagonal Gaussian output head produces a predicted $\\sigma_x$, $\\sigma_y$ per keypoint; it is a proper scoring rule, so training it encourages calibrated uncertainties. The RAIM adaptation (Algorithm 1) estimates pose by weighted least squares, reprojects the known runway corners, forms the residual $r = (I - HH^\\dagger)(y_{\\text{reproject}} - \\mu)$, and compares the variance-normalized norm to a $","core_discovery":"The central claim is that a probabilistic keypoint regressor can support runtime integrity monitoring for landing: the Soft Argmax operator extracts expected coordinates from low-resolution feature maps with sub-pixel precision, the negative log-likelihood loss with a diagonal Gaussian covariance gives uncertainty estimates whose coverage matches observed errors, and Algorithm 1's residual-based RAIM check decides ACCEPT or REJECT by comparing a variance-normalized reprojection residual to a chi-squared distribution. In the nominal validation set, the test statistic matches the theoretical chi-squared density; when the far runway threshold is mispredicted 184 m too close, the statistic separ","pith_inferences":["The same geometric-compatibility test transfers to any perception task with known 3D landmarks, such as aircraft docking, satellite inspection, or warehouse fiducials, so the paper's integrity monitor is a general runtime safety layer rather than a landing-specific fix.","Correlated keypoint errors are the natural stress test: because the paper concedes that perfectly correlated errors are undetectable, a synthetic benchmark injecting spatially coherent shifts would quantify how much correlation the monitor tolerates before detection power collapses.","Because the nominal calibration is validated only in distribution, a deployment-time calibration audit would be needed to distinguish a genuine fault from a distribution shift that makes the network overconfident."],"forward_implications":["Across ResNet18, ResNet50, and EfficientNet, the SAM head lowers median pixel error from 1.46, 2.50, and 10.59 pixels to 0.80, 0.65, and 0.50 pixels, with a minimal parameter footprint and 30-60 Hz inference.","Calibration curves sit close to the identity, so the Gaussian error model is adequate; inflating the predicted standard deviations by about 20% would make the model fully calibrated on the validation set.","The RAIM residual statistic follows the theoretical chi-squared distribution in nominal runs, which is what makes a fixed threshold meaningful rather than ad hoc.","A far-threshold misprediction by 184 m produces a cleanly separated residual statistic, demonstrating that geometrically incompatible outputs can be rejected at runtime without ground-truth projections."],"supporting_citations":[{"why":"Supplies the spatial Soft Argmax operator used to regress sub-pixel keypoints from low-resolution feature maps.","marker":"[49]"},{"why":"Establishes the heteroscedastic Gaussian negative-log-likelihood training objective that yields predictive per-keypoint uncertainties.","marker":"[50]"},{"why":"Provides the residual-based RAIM formulation and chi-squared test that Algorithm 1 adapts for vision.","marker":"[37]"},{"why":"Supplies the runway-approach image dataset and four-corner annotations used to train and evaluate the pipeline.","marker":"[7]"},{"why":"Defines the downstream probabilistic pose estimation that consumes the predicted keypoint uncertainties and motivates runtime integrity.","marker":"[6]"},{"why":"Motivates weighting each reprojection residual by the inverse predicted standard deviation in the pose solve.","marker":"[41]"}],"fun_headline_variants":["Landing AI gets self-checking uncertainty for safer runways","Uncertainty-aware vision keeps landings on track with built-in fault rejection","Real-time landing vision now rejects its own bad predictions","Probabilistic runway vision with integrity checks for aviation safety"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The whole fault-detection scheme rests on each keypoint's pixel error being an independent, Gaussian-distributed quantity with the variance the network predicts; if real errors are correlated or non-Gaussian, the chi-squared threshold no longer means what it claims.","fun_headline_variants_meta":{"raw":{"variants":["Landing AI gets self-checking uncertainty for safer runways","Uncertainty-aware vision keeps landings on track with built-in fault rejection","Real-time landing vision now rejects its own bad predictions","Probabilistic runway vision with integrity checks for aviation safety"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1173,"prompt_tokens":704,"completion_tokens":469,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":398}},"tokens_in":448,"tokens_out":469,"duration_ms":5167,"temperature":1.0,"reasoning_tokens":398,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:50:16.320906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a suite of runway images where the four corner predictions are corrupted by errors drawn from a multivariate Gaussian with increasing off-diagonal covariance, and measure the RAIM rejection rate at a fixed false-alarm threshold. Under the paper's assumptions the statistic remains chi-squared-distributed in nominal cases and the monitor flags incompatible geometry; if the nominal statistic already departs from chi-squared, or if rejection stays high under perfectly correlated errors, the independence and Gaussianity premise would be falsified.","supporting_citations":[{"cited_title":"Estimating the mean and variance of the target probability distribution,","cited_arxiv_id":null,"evidence_quote":"Establishes the heteroscedastic Gaussian negative-log-likelihood training objective that yields predictive per-keypoint uncertainties."},{"cited_title":"Solution Separation Versus Residual-Based RAIM,","cited_arxiv_id":null,"evidence_quote":"Provides the residual-based RAIM formulation and chi-squared test that Algorithm 1 adapts for vision."},{"cited_title":"LARD - landing approach runway detection - dataset for vision based landing,","cited_arxiv_id":null,"evidence_quote":"Supplies the runway-approach image dataset and four-corner annotations used to train and evaluate the pipeline."},{"cited_title":"Probabilistic Parameter Estimators and Calibration Metrics for Pose Estimation from Image Features,","cited_arxiv_id":null,"evidence_quote":"Defines the downstream probabilistic pose estimation that consumes the predicted keypoint uncertainties and motivates runtime integrity."},{"cited_title":"Strutz, Data Fitting and Uncertainty","cited_arxiv_id":null,"evidence_quote":"Motivates weighting each reprojection residual by the inverse predicted standard deviation in the pose solve."}],"review_version":1}