REVIEW 4 major objections 5 minor 39 references
Neural Error Covariance Estimation for Precise LiDAR Localization
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A neural network predicts per-scan LiDAR matching covariance, and feeding it into a Kalman filter cuts final localization error from 4.1 cm to 2.1 cm on KITTI.
desk verdict Solid idea for learning per-scan LiDAR covariance, but the single-sequence KITTI evaluation with no calibration check does not yet support the headline 2 cm claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Monte Carlo dataset generation stage, which turns a point cloud and a locally built map into a covariance label by running ICP 1,000 times from perturbed initial transformations and computing the empirical covariance of the resulting errors. The network architecture is a 3D feature extractor followed by a regression MLP whose output is a lower-triangular Cholesky factor, guaranteeing that the predicted covariance is always symmetric positive definite. The training objective combines a KL-divergence term that penalizes mismatch between predicted and true zero-mean Gaussians with a Huber loss on the upper-triangular entries, with weighted sampling that up-weights scans carrying large covariances. This machinery connects raw scans directly to the measurement covariance used in the Kalman filter update.
What would settle it
Run the trained covariance predictor in the Kalman filter on KITTI sequences with a different initial-perturbation scale (for example 2 m translation, 10 degrees rotation) and compare against the fixed-covariance baseline; if the roughly 2 cm improvement disappears or the filter's innovations become inconsistent with the predicted covariances, the central claim would be falsified.
Extended reading notes
Core claim
The central claim is that the uncertainty of ICP map matching, expressed as a $6\times 6$ covariance on the Lie algebra $\mathfrak{se}(3)$, is predictable from the point cloud itself, and that using the predicted covariance in the Kalman filter gives more precise localization than using a fixed average covariance. The ground-truth covariances are produced by a Monte Carlo procedure: each scan is matched to a locally built map 1,000 times from initial guesses sampled from a Gaussian with $\sigma=1$ m translation and $5^\circ$ rotation, and the empirical covariance of the residual errors becomes the label. A point-cloud network (Cylinder3D or PointNet++, optionally pretrained on SemanticKITTI) is trained end-to-end with a Cholesky parameterization to keep predictions symmetric positive definite and a loss combining KL divergence with a Huber term on the upper-triangular entries. The evaluation shows the predicted covariances reduce ADE from $0.02998$ m to $0.02097$ m and FDE from $0.04127$ m to $0.02072$ m against the fixed-covariance baseline.
Load-bearing premise
The covariance labels are generated by perturbing the ICP initial guess with a fixed Gaussian (1 m, 5 degrees), and the paper does not verify that this matches the distribution of ICP errors that actually occurs during online filtering, so the predicted covariances may be miscalibrated outside that setting.
Editorial extensions
If this is right
- Kalman filter localization improves when the measurement covariance is updated per scan rather than held fixed at the sequence average.
- The method makes ICP's uncertainty predictable from the scan alone, so no parametric sensor-noise model is required for the matching step.
- With Cylinder3D and pretrained semantic-segmentation weights, the KL-divergence metric drops, suggesting that richer features help covariance prediction.
- Because the covariance is predicted in an end-to-end fashion, the same training loop can be rerun for any LiDAR dataset that provides ground truth.
- Data augmentation via the adjoint transform allows expensive Monte Carlo labels to be reused across poses.
Reading between the lines
- A natural extension the paper does not test: the same Monte Carlo labeling could produce covariances for NDT or other matchers, not only ICP, since the labeling only needs a matcher and a perturbation distribution.
- The claimed gain likely depends on how well the 1 m / 5 degree perturbation matches the real initial-error distribution; testing with other perturbation scales would reveal whether the improvement is robust or an artifact of the training distribution.
- The paper evaluates sensor fusion on one KITTI sequence; a multi-sequence study with different environments would clarify whether the learned covariances generalize beyond scenes that resemble the training set.
- One could also feed the predicted covariance into a calibration check: filtering with it should produce innovations whose normalized squared error matches a chi-squared distribution.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a deep-learning framework that predicts a 6x6 error covariance for LiDAR ICP map matching, using a Monte Carlo dataset-generation procedure to create covariance labels, a Cholesky-parameterized neural-network output to preserve positive definiteness, and a KL-plus-Huber loss for training. The predicted covariances are then fed into a Kalman filter with an IMU motion model and ICP corrections, and the paper reports improved localization accuracy on KITTI Odometry relative to ICP-only and to a fixed-covariance baseline. The central claim is that the neural-network-predicted per-scan covariance improves Kalman-filter localization accuracy, with the abstract stating a '2 cm improvement'.
Significance. If the claim were established, the paper would make a useful practical contribution: map-matching covariance is genuinely hard to obtain, and a learned covariance that is calibrated and cheap to evaluate could benefit LiDAR localization systems. The paper also has strengths worth acknowledging: the Monte Carlo labeling procedure is concrete, the Cholesky parameterization is a sound way to enforce symmetric positive definiteness, and the evaluation includes two backbone architectures and an external Kalman-filter benchmark. However, the current evidence does not establish the central claim. The localization evaluation uses only one KITTI sequence with no error bars, no calibration check of the predicted covariances against actual ICP errors, and no explicit guarantee that the evaluation sequence is held out from training. These are load-bearing gaps, not presentation issues, so the paper needs substantial additional validation before the claimed improvement can be accepted.
major comments (4)
- [Section 4.4 / Table 3] The headline '2 cm improvement' is supported only by the FDE metric (0.02072 m vs. 0.04127 m fixed covariance), while the ADE improvement is 0.9 cm (0.02097 vs. 0.02998). The abstract and conclusion state the result as a general '2 cm improvement in localization accuracy,' which overstates the evidence. Moreover, the entire sensor-fusion evaluation is on one KITTI sequence with no standard deviations, no multiple sequences, and no statistical significance. At minimum, the authors should report per-sequence results with error bars over several held-out KITTI sequences and should state precisely which metric supports the '2 cm' claim.
- [Section 3.3 / Table 1 / Section 4.4] The covariance labels are the sample covariance of ICP errors when ICP is initialized from perturbations drawn from a hand-chosen Gaussian, O, with sigma_x/y/z = 1 m and sigma_phi/theta/psi = 5 degrees (Table 1). The paper acknowledges in Section 3.1 that O depends on initial guesses from other sensors and vehicle velocity, but the Kalman-filter evaluation never measures the actual distribution of ICP initial errors produced by the IMU prediction, nor does it check whether the predicted covariance is calibrated against empirical errors. Without a calibration metric such as normalized estimation error squared (NEES) or a comparison of predicted vs. empirical covariance over the evaluation trajectory, the improved FDE could be due to a fortunate choice of O rather than to the network's ability to predict per-scan uncertainty. The authors should validate calibration and report sensitivity of the localization results to the perturbation size in Table 1.
- [Section 4.1 / Section 4.4] The dataset split is described as randomly selecting 10,000 training, 5,000 testing, and 2,500 validation samples from the first 11 KITTI sequences, but the sensor-fusion evaluation then uses 'one sequence from the KITTI-Odometry dataset.' If that sequence's point clouds were included in the training split, the evaluation can benefit from memorization of that sequence's scans and covariances. The paper should either specify the exact sequence used, confirm it was not in the training set, or adopt a sequence-level split so that all training scans come from sequences disjoint from the evaluation sequence.
- [Section 4.4 / Section 2.2] The localization evaluation compares the proposed covariance only against a fixed average covariance and ICP-only. Since the related work surveys existing learned and analytical covariance estimators for ICP (CELLO, CELLO-3D, and analytical approaches), the paper should compare against at least one established covariance estimation method. Without such a comparison, the paper cannot show that its data-driven covariance is better than existing alternatives; it only shows that a covariance-informed Kalman filter can improve over fixed covariance in one sequence.
minor comments (5)
- [Figure 1] The figure label '3D Feature Etraction' contains a typo; it should read '3D Feature Extraction.'
- [Section 3.3 / Algorithm 1] The pseudocode uses inconsistent notation: the covariance is written as Y_k in the text but as Ŷ in Algorithm 1, and the summation symbol appears as 'Í' in the pseudocode. Also, the expression '𝑏𝑜𝑙𝑑𝑠𝑦𝑚𝑏𝑜𝑙𝑇' in Section 3.3 is corrupted and should be a clear symbol for the estimated transformation.
- [Section 3.5 / Equation (4)] The hyperparameters alpha = 0.1 and beta = 0.9 are described as 'empirically determined,' but no ablation or sensitivity analysis is provided. A short sensitivity table for alpha and beta would help establish that the reported results are not highly sensitive to these choices.
- [Section 4.4] The Kalman filter experiment lacks implementation details: the state dimension, the IMU noise model, the exact ICP configuration, and how the 6x6 covariance is mapped into the filter's measurement noise are not specified. These details are needed for reproducibility.
- [Section 4.1] The paper states that the map is built from 20 previous and 10 subsequent point clouds relative to the input, but it does not specify how the map point clouds are transformed to the global frame or whether the same map-building parameters are used in the Kalman-filter evaluation. This should be clarified for reproducibility.
Circularity Check
No significant circularity; the covariance labels are a modeling choice, and the Kalman filter evaluation is an external benchmark.
full rationale
The paper's derivation chain is not circular. Ground-truth covariance labels are produced by the Monte Carlo procedure in Sec. 3.3, where ICP is re-run from initial perturbations sampled from N(0, O) with Table 1 parameters, and the sample covariance of the resulting pose errors (Eq. in Sec. 3.3) is the training target. The network is trained to predict those labels, and the learned predictor is then evaluated inside a Kalman filter using IMU prediction plus ICP correction (Sec. 4.4). This is a standard supervised-learning pipeline: the KITTI filter evaluation is an external benchmark, and the reported ADE/FDE values are not algebraically forced by the label-generation equation. The only methodological weakness is that the 'fixed covariance' baseline in Table 3 is defined as the average of the proposed model's predicted covariances for that sequence, which makes that baseline non-independent but does not make the proposed result equivalent to its inputs. Concerns that the hand-chosen perturbation distribution O (1 m, 5 deg) may not match the actual ICP initial-error distribution during online filtering are calibration and external-validity risks, not circularity; no equation in the paper reduces the claimed 2 cm improvement to the training labels by construction. The self-citations in the related work (Javanmardi et al. [19,20]) provide background and are not load-bearing for the central claim.
Assumptions & free parameters
free parameters (3)
- Initial perturbation covariances (sigma_x, sigma_y, sigma_z, sigma_phi, sigma_theta, sigma_psi) =
1 m, 1 m, 1 m, 5 deg, 5 deg, 5 deg
- Loss weights alpha and beta =
0.1, 0.9
- Huber loss threshold delta_0 =
not specified
assumptions (4)
- domain assumption ICP pose errors are zero-mean and Gaussian.
- domain assumption The map built from ground truth is accurate enough for covariance labels.
- standard math The MC sample covariance with n=1000 converges to the true ICP covariance under the chosen perturbation distribution.
- domain assumption The chosen initial perturbation distribution is representative of real initial guess errors during Kalman filter operation.
Cite this review
Pith. "Pith review of Neural Error Covariance Estimation for Precise LiDAR Localization." pith.science (2026). https://pith.science/paper/K4ER54S3
@misc{pith2026250102558,
author = {Pith},
title = {Pith review of: Neural Error Covariance Estimation for Precise LiDAR Localization},
year = {2026},
howpublished = {\url{https://pith.science/paper/K4ER54S3}},
note = {Machine review of arXiv:2501.02558}
}
read the original abstract
Autonomous vehicles have gained significant attention due to technological advancements and their potential to transform transportation. A critical challenge in this domain is precise localization, particularly in LiDAR-based map matching, which is prone to errors due to degeneracy in the data. Most sensor fusion techniques, such as the Kalman filter, rely on accurate error covariance estimates for each sensor to improve localization accuracy. However, obtaining reliable covariance values for map matching remains a complex task. To address this challenge, we propose a neural network-based framework for predicting localization error covariance in LiDAR map matching. To achieve this, we introduce a novel dataset generation method specifically designed for error covariance estimation. In our evaluation using a Kalman filter, we achieved a 2 cm improvement in localization accuracy, a significant enhancement in this domain.
Figures
Reference graph
Works this paper leans on
-
[1]
Daniel Adolfsson, Martin Magnusson, Qianfang Liao, Achim J Lilienthal, and Henrik Andreasson. 2021. Coral–are the point clouds correctly aligned?. In 2021 European conference on mobile robots (ECMR) . IEEE, 1–7
work page 2021
-
[2]
Behley, M
J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall
-
[3]
José Luis Blanco-Claraco. 2021. A tutorial on SE(3) transformation parameteriza- tions and on-manifold optimization. arXiv preprint arXiv:2103.15980 (2021)
arXiv 2021
-
[4]
Martin Brossard, Silvere Bonnabel, and Axel Barrau. 2020. A new approach to 3D ICP covariance estimation. IEEE Robotics and Automation Letters 5, 2 (2020), 744–751
work page 2020
-
[5]
Anas Charroud, Karim El Moutaouakil, Vasile Palade, Ali Yahyaouy, Uche Onyekpe, and Eyo U Eyo. 2024. Localization and Mapping for Self-Driving Vehicles: A Survey. Machines 12, 2 (2024), 118
work page 2024
-
[6]
Weifeng Chen, Guangtao Shang, Aihong Ji, Chengjun Zhou, Xiyang Wang, Chonghui Xu, Zhenxiong Li, and Kai Hu. 2022. An overview on visual slam: From tradition to semantic. Remote Sensing 14, 13 (2022), 3010
work page 2022
-
[7]
Xieyuanli Chen, Thomas Läbe, Andres Milioto, Timo Röhling, Jens Behley, and Cyrill Stachniss. 2022. OverlapNet: A siamese network for computing LiDAR scan similarity with applications to loop closing and localization. Autonomous Robots (2022), 1–21
work page 2022
-
[8]
HyunGi Cho, Suyong Yeon, Hyunga Choi, and Nakju Doh. 2018. Detection and compensation of degeneracy cases for imu-kinect integrated continuous slam with plane features. Sensors 18, 4 (2018), 935
work page 2018
Show all 39 references
-
[9]
Jean-Emmanuel Deschaud. 2018. IMLS-SLAM: Scan-to-model matching based on 3D data. In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2480–2485
2018
-
[10]
Lingfeng Dong, Weidong Chen, and Jingchuan Wang. 2021. Efficient feature ex- traction and localizability based matching for lidar slam. In2021 IEEE International Conference on Robotics and Biomimetics (ROBIO) . IEEE, 820–825
2021
-
[11]
Kamak Ebadi, Matteo Palieri, Sally Wood, Curtis Padgett, and Ali-akbar Agha- mohammadi. 2021. DARE-SLAM: Degeneracy-aware and resilient loop closing in perceptually-degraded environments. Journal of Intelligent & Robotic Systems 102 (2021), 1–25
2021
-
[12]
Yuki Endo, Ehsan Javanmardi, and Shunsuke Kamijo. 2021. Analysis of occlusion effects for map-based self-localization in urban areas. Sensors 21, 15 (2021), 5196
2021
-
[13]
Yanwen Fang, Philip LH Yu, and Yaohua Tang. 2021. CNN-based realized covari- ance matrix forecasting. arXiv preprint arXiv:2107.10602 (2021)
2021 arXiv
-
[14]
Andreas Geiger, Philip Lenz, and Raquel Urtasun. 2012. Are we ready for au- tonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition . IEEE, 3354–3361
2012
-
[15]
Natasha Gelfand, Leslie Ikemoto, Szymon Rusinkiewicz, and Marc Levoy. 2003. Geometrically stable sampling for the ICP algorithm. In Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings. IEEE, 260–267
2003
-
[16]
Yun Hao, Jiacheng Liu, Yuzhen Liu, Xinyuan Liu, Ziyang Meng, and Fei Xing
-
[17]
Pierre Heroux. 2008. Principles of GNSS, Inertial, and Multisensor Integrated Navigation Systems. Geomatica 62, 2 (2008), 207–209
2008
-
[18]
Akshay Hinduja, Bing-Jui Ho, and Michael Kaess. 2019. Degeneracy-aware factors with applications to underwater slam. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 1293–1299
2019
-
[19]
Ehsan Javanmardi, Mahdi Javanmardi, Yanlei Gu, and Shunsuke Kamijo. 2018. Factors to evaluate capability of map for vehicle localization.IEEE Access 6 (2018), 49850–49867
2018
-
[20]
Ehsan Javanmardi, Mahdi Javanmardi, Yanlei Gu, and Shunsuke Kamijo. 2020. Pre-estimating self-localization error of NDT-based map-matching from map only. IEEE Transactions on Intelligent Transportation Systems 22, 12 (2020), 7652–7666
2020
-
[21]
Yin-Chiu Kan, Li-Ta Hsu, and Edward Chung. 2021. Performance evaluation on map-based NDT scan matching localization using simulated occlusion datasets. IEEE Sensors Letters 5, 3 (2021), 1–4
2021
-
[22]
Tsz-Ho Kwok and Kai Tang. 2016. Improvements to the iterative closest point algorithm for shape registration in manufacturing. Journal of Manufacturing Science and Engineering 138, 1 (2016), 011014
2016
-
[23]
David Landry, François Pomerleau, and Philippe Giguere. 2019. CELLO-3D: Esti- mating the Covariance of ICP in the Real World. In 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 8190–8196
2019
-
[24]
Katherine Liu, Kyel Ok, William Vega-Brown, and Nicholas Roy. 2018. Deep inference for covariance estimation: Learning gaussian noise models for state estimation. In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 1436–1443
2018
-
[25]
Ying Liu, Jingchuan Wang, and Yi Huang. 2021. A localizability estimation method for mobile robots based on 3d point cloud feature. In 2021 IEEE International Conference on Real-time Computing and Robotics (RCAR) . IEEE, 1035–1041
2021
-
[26]
Simona Nobili, Georgi Tinchev, and Maurice Fallon. 2018. Predicting alignment risk to prevent localization failure. In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 1003–1010
2018
-
[27]
Julian Nubert, Etienne Walther, Shehryar Khattak, and Marco Hutter. 2022. Learning-based localizability estimation for robust lidar localization. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 17–24
2022
-
[28]
C. Qi, L. Yi, Hao Su, and Leonidas J. Guibas. 2017. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. InNeural Information Processing Systems. https://api.semanticscholar.org/CorpusID:1745976
2017
-
[29]
Zheng Rong and Nathan Michael. 2016. Detection and prediction of near-term state estimation degradation via online nonlinear observability analysis. In 2016 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR) . IEEE, 28–33
2016
-
[30]
Turcan Tuna, Julian Nubert, Yoshua Nava, Shehryar Khattak, and Marco Hutter
-
[31]
Claudio Urrea and Rayko Agramonte. 2021. Kalman filter: historical overview and review of its use in robotics 60 years after its creation. Journal of Sensors 2021, 1 (2021), 9674015
2021
-
[32]
William Vega-Brown, Abraham Bachrach, Adam Bry, Jonathan Kelly, and Nicholas Roy. 2013. Cello: A fast algorithm for covariance estimation. In 2013 IEEE International Conference on Robotics and Automation . IEEE, 3160–3167
2013
-
[33]
IEEE Transactions on Robotics (2023)
X-icp: Localizability-aware lidar registration for robust localization in extreme environments. IEEE Transactions on Robotics (2023)
2023
-
[34]
Ji Zhang, Sanjiv Singh, et al . 2014. LOAM: Lidar odometry and mapping in real-time.. In Robotics: Science and systems , Vol. 2. Berkeley, CA, 1–9
2014
-
[35]
Weikun Zhen and Sebastian Scherer. 2019. Estimating the localizability in tunnel- like environments using LiDAR and UWB. In 2019 International Conference on Robotics and Automation (ICRA). IEEE, 4903–4908
2019
-
[36]
Ji Zhang, Michael Kaess, and Sanjiv Singh. 2016. On degeneracy of optimization- based state estimation problems. In 2016 IEEE international conference on robotics and automation (ICRA). IEEE, 809–816
2016
-
[39]
Hui Zhou, Xinge Zhu, Xiao Song, Yuexin Ma, Zhe Wang, Hongsheng Li, and Dahua Lin. 2020. Cylinder3d: An effective 3d framework for driving-scene lidar semantic segmentation. arXiv preprint arXiv:2008.01550 (2020)
2020 arXiv
-
[2019]
SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences. In Proc. of the IEEE/CVF International Conf. on Computer Vision (ICCV)
-
[2023]
Sensors 23, 9 (2023), 4510
Global Visual–Inertial Localization for Autonomous Vehicles with Pre-Built Map. Sensors 23, 9 (2023), 4510
2023
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.