REVIEW 4 major objections 8 minor 1 cited by
A Comprehensive Evaluation of LiDAR Odometry Techniques
T0 review · 4 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A systematic ablation of LiDAR odometry components finds planar features with plane-to-plane residuals and IMU dewarping give the most accurate scan-to-scan odometry.
desk verdict A careful, useful component-level ablation of LiDAR odometry, but the oracle-input protocol means the headline recommendations are stronger than the evidence supports. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the modular LO pipeline decomposed into four components: dewarping, feature extraction, initialization, and optimization. The identity that carries the argument is the residual weighting matrix in the nonlinear least-squares objective: point-to-plane uses the projection onto the estimated normal, pseudo-point-to-plane interpolates between point-to-plane and point-to-point through a parameter epsilon, and plane-to-plane adds the source normal's projection so that both target and source surfaces constrain the match. The evaluation mechanism is scan-to-scan ICP with ground-truth initialization and offline-computed IMU velocities and biases, which isolates each component's effect and measures drift with windowed relative trajectory error (RTEj) over a 10-second window.
What would settle it
Run the same component ablations inside a full scan-to-map pipeline with online state estimates, no ground-truth initialization, and online IMU bias estimation on the same seven datasets. If the ordering flips on any major recommendation, such as point-to-plane beating plane-to-plane in structured environments or constant-velocity dewarping beating IMU dewarping on a low-grade IMU, the central claim that these scan-to-scan, oracle-initialized results guide real pipeline design is falsified. A more targeted variant: on a structured campus sequence, replace plane-to-plane with point-to-plane in a full scan-to-map system and measure RTEj; if point-to-plane is not worse, the headline recommendation does not transfer.
Extended reading notes
Core claim
The authors' central claim is that the component choices in a LiDAR odometry pipeline can be ranked empirically, and that the best-performing combination is planar features extracted with any reasonable curvature estimator, optimized with a plane-to-plane residual, with IMU-based dewarping when an IMU is available, constant-velocity dewarping otherwise, and constant-velocity initialization when motion is smooth and IMU initialization under aggressive motion. They state the result directly: planar features with a plane-to-plane residual perform the best in structured environments, with additional features required for unstructured environments, there is little impact in the curvature computation method chosen, and IMU dewarping performs the best, with constant velocity also being worthwhile given sufficient compute. In unstructured environments such as forest trails, edge features detected along tree trunks add useful constraints and improve results, while in structured environments they add little or can degrade accuracy.
Load-bearing premise
The recommendations assume that what works best when each scan is matched to the previous scan using perfect starting positions and offline-tuned sensor corrections also works best in a real-time system that builds a map and estimates positions and corrections online as it goes.
Editorial extensions
If this is right
- A new LiDAR odometry pipeline for structured indoor or campus environments can default to planar features plus a plane-to-plane residual and drop edge features, saving computation without sacrificing accuracy.
- Designers with an IMU should use IMU dewarping, and designers without one should only add constant-velocity dewarping when they have spare compute, since the gains are modest.
- Curvature estimation can be chosen on implementation convenience and cost rather than accuracy, with classical scanline curvature recommended for rotating LiDARs.
- Initialization matters far less with planar features than with point features, so constant-velocity initialization is a safe default for smooth vehicle motion, with IMU initialization reserved for aggressive motion.
- In unstructured environments such as forest trails, adding edge or point features is necessary to prevent drift; a purely planar pipeline will underperform.
Reading between the lines
- If the component ranking transfers to scan-to-map pipelines, many current LiDAR-inertial systems are carrying unnecessary feature complexity in structured environments; removing edge features could cut runtime with little accuracy cost.
- Because the paper uses offline-optimized IMU biases and velocities, its IMU-dewarping advantage is an upper bound; with low-cost IMUs and online bias estimation the gap over constant-velocity dewarping could shrink, a comparison the paper leaves open.
- The finding that point-to-point residuals are far more sensitive to initialization than planar residuals suggests robustness testing of LO pipelines should stress initialization; one could add artificial pose perturbations and measure how quickly each residual type recovers.
- A testable extension of the plane-to-plane result is whether it persists with fewer beams or sparser scans, since the paper notes nearest-neighbor eigenvalue features failed on low-beam sensors.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a component-level empirical evaluation of LiDAR odometry (LO) building blocks: initialization schemes, scan dewarping, curvature/feature extraction methods, and optimization residuals (point-to-point, point-to-plane, pseudo-point-to-plane, plane-to-plane). The authors run scan-to-scan matching experiments across seven datasets with varied environments, LiDAR sensors, and vehicle motions, and report windowed Relative Trajectory Error (RTEj). They make design recommendations, most notably that plane-to-plane residuals are best in structured environments, that IMU dewarping and IMU initialization perform best when available, that classical scanline curvature is sufficient, and that point features are more initialization-sensitive than planar features. The codebase is released as open source. The core claim is that these ablation results provide empirically grounded guidance for building future LO and LIO pipelines.
Significance. If the conclusions hold, the paper fills a real gap: most prior evaluations compare full LO/LIO pipelines rather than isolating the effect of individual components. The dataset diversity, the use of a drift-oriented metric (RTEj), and the public code release are clear strengths. The paper also gives a useful tutorial overview of common LO techniques. However, the practical significance depends on an untested transfer assumption: the experiments are run with ground-truth and offline-optimized oracle inputs, while the recommendations target real online pipelines. This, together with the absence of uncertainty quantification on the central ranking claims, means the paper currently supports a promising methodology and a set of preliminary conclusions rather than a definitive design guide.
major comments (4)
- [§V.E, Fig. 11] The central recommendations rest on an oracle protocol. Section IV states that constant-velocity initialization and dewarping use ground-truth velocities and that IMU biases and velocities come from a full-trajectory factor graph with ground-truth pose priors. Section V.A explicitly calls this the 'ideal scenario that will be achieved if the LO or LIO pipeline is functioning properly.' The best-performing components, especially IMU dewarping and IMU initialization, are exactly those that in a real pipeline receive noisy, lagging, or drift-affected inputs. The paper acknowledges real-world IMU risk in Section V.A but does not test whether the rankings are preserved when oracle inputs are replaced by the causal, online estimates that a deployed system would actually use. This is load-bearing for the paper's practical recommendations. Please add experiments with online-style inputs (e.g., constant velocity computed from previous odometry estimates, and IMU bias/velocity from an actual forward-running LIO estimator), or at minimum a sensitivity analysis that perturbs the oracle velocities, biases, and initializations by realistic amounts and reports whether the ranking of dewarping and initialization methods changes.
- [§V.D, Fig. 10] The claim that 'plane-to-plane always outperformed point-to-plane methods' is stated without any uncertainty quantification. Section IV.B notes that ground-truth precision is near the precision of scan-to-scan odometry, so RTEj values are noisy, yet the figures report single per-trajectory values with no confidence intervals, bootstrap resamples, or paired statistical tests. Some reported percentage changes in Fig. 11 are small and could be within the noise floor of the metric. Please provide error bars or bootstrap intervals for the RTEj values, and state the number of independent runs or segments used. If repeated runs are not feasible because the experiments are deterministic, a bootstrap over trajectory segments or a paired significance test across the dataset collection would help support the 'always' claim.
- [§III, §IV] The pseudo-feature experiment, which motivates a central negative recommendation (that constraining the directions orthogonal to the normal is detrimental), displays results for only two trajectories. The text says more trajectories were tested but omits them 'for figure clarity.' This is not verifiable from the manuscript, and it leaves open the possibility that the displayed pair is not representative. Please include the full set of trajectories, either in the main text as an aggregate plot (e.g., boxplots or mean percent change with spread) or in a clearly referenced supplement, and report how many trajectories were used and whether the conclusion held in all of them.
- [§V.B, §IV.B] The paper's scope is scan-to-scan matching, but many of the recommendations are phrased as guidance for building whole LO or LIO pipelines, which commonly use scan-to-map matching. Section III asserts that the conclusions 'should also extend to, or at least guide development of, scan-to-map matching' without testing that claim. The sensitivity of different feature types and residual definitions can differ when a map accumulates and when the optimization basin changes. This is a load-bearing scope limitation for the stated goal. Please either test at least one representative scan-to-map configuration to confirm the main rankings (especially for features and residuals), or visibly restrict the conclusions to scan-to-scan matching and clearly state that scan-to-map transfer is a hypothesis rather than an empirical result.
minor comments (8)
- [§III.B] The abstract contains a typo: 'fine the most accurate method' should read 'find the most accurate method.'
- [§IV.B] There is a typo in 'leasts-squares'; it should be 'least-squares.'
- [§V.C] The choice of a 10-second window for RTEj is motivated by Fig. 6, but no sensitivity analysis is reported for the window duration. A brief check of whether the main rankings are stable for, say, 5 s and 20 s windows would strengthen the metric choice.
- [§V.D] The curvature experiment says thresholds were tuned to emit 'roughly the same number of features,' but the actual thresholds or feature counts are not reported. Please include the tuned parameter values or the resulting feature counts, even in a supplementary table, to make the comparison reproducible.
- [§V.F] In Fig. 10, the curves for different initialization strategies are not individually distinguishable for small values of epsilon; consider using distinct markers or a zoomed inset so the ordering at epsilon near zero is visible.
- [§II] The statement that point features 'required more iterations to converge' is not quantified. Reporting the average number of iterations or runtime for each feature type would make this concrete.
- [§I] The related-work section mentions that some surveys lack empirical results and some compare pipelines, but it does not cite recent large-scale odometry benchmarks (e.g., KITTI odometry, or more recent SLAM evaluation suites) that also provide some component-level insights. A brief discussion of where those benchmarks differ from the present ablation would help position the contribution.
- [§IV] The contribution bullet says some results 'run contrary to many current preferred choices,' but the introduction does not say which choices these are. Naming one or two examples there would make the contribution statement more informative.
Circularity Check
No circularity: the paper is an empirical ablation whose recommendations summarize measured component comparisons, not derivations from self-defined quantities.
full rationale
This paper performs component-level ablation experiments and derives recommendations from the resulting measurements. No load-bearing circular step is present in the derivation chain. The authors explicitly use ground-truth poses and offline-optimized bias/velocity estimates as oracle inputs, labeling this the "ideal scenario" in Section V.A, and they do not present these oracle-conditioned results as an online prediction; the conclusions are stated as empirical findings under that protocol. The skeptical concern about whether oracle-input rankings transfer to deployable pipelines is a question of external validity and experimental design, not circularity: the rankings are not equivalent to their inputs by construction, and the paper openly acknowledges real-world IMU risks and the scan-to-map transfer assumption without claiming that assumption is proven. The curvature, feature, residual, and initialization comparisons are independent measurements across held-out trajectories and datasets. Self-citation is not load-bearing here; the only close cases are citations to standard methods such as LOAM and Generalized-ICP, which are used as baselines and comparisons, not as justification for the paper's conclusions. Therefore, no circular step meeting the evidentiary standard can be identified.
Assumptions & free parameters
free parameters (3)
- Feature extraction thresholds =
not reported
- RTEj window duration =
10 s
- Trajectory truncation length =
5 min / ~3000 scans
assumptions (3)
- domain assumption Scan-to-scan matching results transfer to scan-to-map matching.
- domain assumption Ground-truth-based initialization and dewarping represent the ceiling of real component performance and the comparative rankings hold with noisy estimates.
- domain assumption The provided datasets have sufficiently accurate ground truth for windowed relative trajectory error at 10 s windows.
Cite this review
Pith. "Pith review of A Comprehensive Evaluation of LiDAR Odometry Techniques." pith.science (2026). https://pith.science/paper/5BOR767G
@misc{pith2026250716000,
author = {Pith},
title = {Pith review of: A Comprehensive Evaluation of LiDAR Odometry Techniques},
year = {2026},
howpublished = {\url{https://pith.science/paper/5BOR767G}},
note = {Machine review of arXiv:2507.16000}
}
read the original abstract
Light Detection and Ranging (LiDAR) sensors have become the sensor of choice for many robotic state estimation tasks. Because of this, in recent years there has been significant work done to fine the most accurate method to perform state estimation using these sensors. In each of these prior works, an explosion of possible technique combinations has occurred, with each work comparing LiDAR Odometry (LO) "pipelines" to prior "pipelines". Unfortunately, little work up to this point has performed the significant amount of ablation studies comparing the various building-blocks of a LO pipeline. In this work, we summarize the various techniques that go into defining a LO pipeline and empirically evaluate these LO components on an expansive number of datasets across environments, LiDAR types, and vehicle motions. Finally, we make empirically-backed recommendations for the design of future LO pipelines to provide the most accurate and reliable performance.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 1 Pith paper
-
COSMO-Bench: A Benchmark for Collaborative SLAM Optimization
A new suite of 24 realistic collaborative SLAM optimization benchmarks derived from real LiDAR data with simulated inter-robot communication.
Reference graph
Works this paper leans on
-
[1]
LOAM: Lidar Odometry and Mapping in Real-time,
J. Zhang and S. Singh, “LOAM: Lidar Odometry and Mapping in Real-time,” in Proc. Robot.: Sci. & Syst. Conf. Robotics: Science and Systems Foundation, July 2014
work page 2014
-
[2]
LIO- SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping,
T. Shan, B. Englot, D. Meyers, W. Wang, C. Ratti, and D. Rus, “LIO- SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots and Syst. , Oct. 2020, pp. 5135–5142
work page 2020
-
[3]
CT-ICP: Real-time Elastic LiDAR Odometry with Loop Closure,
P. Dellenbach, J.-E. Deschaud, B. Jacquet, and F. Goulette, “CT-ICP: Real-time Elastic LiDAR Odometry with Loop Closure,” in Proc. IEEE Int. Conf. Robot. and Automation , May 2022, pp. 5580–5586
work page 2022
-
[4]
A. Segal, D. Haehnel, and S. Thrun, “Generalized-icp.” in Proc. Robot.: Sci. & Syst. Conf. , vol. 2. Seattle, W A, 2009, p. 435
work page 2009
-
[5]
Review on LiDAR-based SLAM Techniques,
L. Huang, “Review on LiDAR-based SLAM Techniques,” in Int. Conf. on Signal Process. and Mach. Learning , Nov. 2021, pp. 163–168
work page 2021
-
[6]
Y . Zhang, P. Shi, and J. Li, “3D L IDAR SLAM : A survey,” The Photogrammetric Record, vol. 39, no. 186, pp. 457–517, June 2024
work page 2024
-
[7]
Efficient variants of the ICP algo- rithm,
S. Rusinkiewicz and M. Levoy, “Efficient variants of the ICP algo- rithm,” in Proceedings Third International Conference on 3-D Digital Imaging and Modeling . Quebec City, Que., Canada: IEEE Comput. Soc, 2001, pp. 145–152
work page 2001
-
[8]
LiDAR odometry survey: Recent advancements and remaining challenges,
D. Lee, M. Jung, W. Yang, and A. Kim, “LiDAR odometry survey: Recent advancements and remaining challenges,” Intelligent Service Robotics, vol. 17, no. 2, pp. 95–118, Mar. 2024
work page 2024
Show all 26 references
-
[9]
LiDAR Odometry Methodologies for Autonomous Driving: A Survey,
N. Jonnavithula, Y . Lyu, and Z. Zhang, “LiDAR Odometry Methodologies for Autonomous Driving: A Survey,” 2021. [Online]. Available: https://arxiv.org/abs/2109.06120
2021 arXiv
-
[10]
KISS-ICP: In Defense of Point-to-Point ICP – Simple, Accurate, and Robust Registration If Done the Right Way,
I. Vizzo, T. Guadagnino, B. Mersch, L. Wiesmann, J. Behley, and C. Stachniss, “KISS-ICP: In Defense of Point-to-Point ICP – Simple, Accurate, and Robust Registration If Done the Right Way,” IEEE Robot. and Automation Letters , vol. 8, no. 2, pp. 1029–1036, Feb. 2023
2023
-
[11]
MULLS: Versatile LiDAR SLAM via multi-metric linear least square,
Y . Pan, P. Xiao, Y . He, Z. Shao, and Z. Li, “MULLS: Versatile LiDAR SLAM via multi-metric linear least square,” in Proc. IEEE Int. Conf. Robot. and Automation . IEEE, 2021, pp. 11 633–11 640
2021
-
[12]
Wildcat: Online Continuous- Time 3D Lidar-Inertial SLAM,
M. Ramezani, K. Khosoussi, G. Catt, P. Moghadam, J. Williams, P. Borges, F. Pauling, and N. Kottege, “Wildcat: Online Continuous- Time 3D Lidar-Inertial SLAM,” 2022. [Online]. Available: https: //arxiv.org/abs/2205.12595
2022 arXiv
-
[13]
Super Odometry: IMU-centric LiDAR-Visual-Inertial Estimator for Chal- lenging Environments,
S. Zhao, H. Zhang, P. Wang, L. Nogueira, and S. Scherer, “Super Odometry: IMU-centric LiDAR-Visual-Inertial Estimator for Chal- lenging Environments,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots and Syst. , Sept. 2021, pp. 8729–8736
2021
-
[14]
Evaluation of the ICP Algorithm in 3D Point Cloud Registration,
P. Li, R. Wang, Y . Wang, and W. Tao, “Evaluation of the ICP Algorithm in 3D Point Cloud Registration,” IEEE Access , vol. 8, pp. 68 030–68 048, 2020
2020
-
[15]
Mad-icp: It is all about matching data–robust and informed lidar odometry,
S. Ferrari, L. Di Giammarino, L. Brizi, and G. Grisetti, “Mad-icp: It is all about matching data–robust and informed lidar odometry,” IEEE Robot. and Automation Letters , 2024
2024
-
[16]
Discrete differential geometry: An applied introduction,
K. Crane, “Discrete differential geometry: An applied introduction,” Notices of the AMS, Communication , vol. 1153, 2018
2018
-
[17]
The Newer College Dataset: Handheld LiDAR, Inertial and Vision with Ground Truth,
M. Ramezani, Y . Wang, M. Camurri, D. Wisth, M. Mattamala, and M. Fallon, “The Newer College Dataset: Handheld LiDAR, Inertial and Vision with Ground Truth,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots and Syst. , Oct. 2020, pp. 4353–4360
2020
-
[18]
Multi-Camera LiDAR Inertial Extension to the Newer College Dataset,
L. Zhang, M. Camurri, D. Wisth, and M. Fallon, “Multi-Camera LiDAR Inertial Extension to the Newer College Dataset,” 2022. [Online]. Available: https://arxiv.org/abs/2112.08854
2022 arXiv
-
[19]
Hilti-Oxford Dataset: A Millimeter- Accurate Benchmark for Simultaneous Localization and Mapping,
L. Zhang, M. Helmberger, L. F. T. Fu, D. Wisth, M. Camurri, D. Scaramuzza, and M. Fallon, “Hilti-Oxford Dataset: A Millimeter- Accurate Benchmark for Simultaneous Localization and Mapping,” IEEE Robot. and Automation Letters , vol. 8, no. 1, pp. 408–415, Jan. 2023
2023
-
[20]
The Oxford Spires Dataset: Benchmarking Large- Scale LiDAR-Visual Localisation, Reconstruction and Radiance Field Methods,
Y . Tao, M. ´Angel Mu ˜noz-Ba˜n´on, L. Zhang, J. Wang, L. F. T. Fu, and M. Fallon, “The Oxford Spires Dataset: Benchmarking Large- Scale LiDAR-Visual Localisation, Reconstruction and Radiance Field Methods,” 2024. [Online]. Available: https://arxiv.org/abs/2411.10546
2024 arXiv
-
[21]
MCD: Diverse Large-Scale Multi-Campus Dataset for Robot Perception,
T.-M. Nguyen, S. Yuan, T. H. Nguyen, P. Yin, H. Cao, L. Xie, M. Woz- niak, P. Jensfelt, M. Thiel, J. Ziegenbein, and N. Blunder, “MCD: Diverse Large-Scale Multi-Campus Dataset for Robot Perception,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog. Seattle, W A, USA: IEEE, June ...
2024
-
[22]
HeLiPR: Heterogeneous LiDAR dataset for inter-LiDAR place recognition under spatiotemporal variations,
M. Jung, W. Yang, D. Lee, H. Gil, G. Kim, and A. Kim, “HeLiPR: Heterogeneous LiDAR dataset for inter-LiDAR place recognition under spatiotemporal variations,” Int. J. Robot. Res. , vol. 43, no. 12, pp. 1867–1883, Oct. 2024
2024
-
[23]
Botanicgarden: A high-quality dataset for robot navigation in unstructured natural environments,
Y . Liu, Y . Fu, M. Qin, Y . Xu, B. Xu, F. Chen, B. Goossens, P. Z. Sun, H. Yu, C. Liu, et al. , “Botanicgarden: A high-quality dataset for robot navigation in unstructured natural environments,” IEEE Robot. and Automation Letters , vol. 9, no. 3, pp. 2798–2805, 2024
2024
-
[24]
On-Manifold Preintegration for Real-Time Visual–Inertial Odometry,
C. Forster, L. Carlone, F. Dellaert, and D. Scaramuzza, “On-Manifold Preintegration for Real-Time Visual–Inertial Odometry,” IEEE Trans. on Robotics , vol. 33, no. 1, pp. 1–21, Feb. 2017
2017
-
[25]
On measuring the accuracy of SLAM algorithms,
R. K ¨ummerle, B. Steder, C. Dornhege, M. Ruhnke, G. Grisetti, C. Stachniss, and A. Kleiner, “On measuring the accuracy of SLAM algorithms,” Autonomous Robots , vol. 27, no. 4, pp. 387–407, Nov. 2009
2009
-
[26]
Vbr: a vision benchmark in rome,
L. Brizi, E. Giacomini, L. Di Giammarino, S. Ferrari, O. Salem, L. De Rebotti, and G. Grisetti, “Vbr: a vision benchmark in rome,” in Proc. IEEE Int. Conf. Robot. and Automation . IEEE, 2024, pp. 15 868–15 874
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.