Pith. sign in

REVIEW 2 major objections 5 minor 2 cited by

Selective Kalman Filter: When and How to Fuse Multi-Sensor Information to Overcome Degeneracy in SLAM

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A selective Kalman filter that fuses visual data only when LiDAR degenerates matches full-fusion accuracy at a fraction of the cost.

desk verdict Covariance-based degeneracy detection is a real contribution, but the selective update misweights the projected visual information and is overconfident. read the letter →

arxiv 2412.17235 v1 pith:H3OUMOYG submitted 2024-12-23 cs.RO

Jie Xu , Guanyu Huang , Wenlu Yu , Xuanxuan Zhang , Lijun Zhao , Ruifeng Li , Shenghai Yuan , Lihua Xie This is my paper · ORCID
classification cs.RO
keywords selectivekalmanfiltermulti-sensorfusionSLAMdegeneracydetectionLiDAR-inertial-visualodometrycovariance-basedreal-timeperformance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that multi-sensor fusion in SLAM does not need to use all data all the time. It proposes a Selective Kalman Filter that feeds visual measurements into a LiDAR-inertial system only when the LiDAR subsystem becomes degenerate, and only along the degenerate directions. The authors show that this selective strategy achieves accuracy comparable to the all-in fusion approach while drastically reducing the visual processing cost and avoiding errors from visual data in well-conditioned situations. The paper also introduces a covariance-based degeneracy detection method that accounts for the coupling between rotational and translational constraints, which it argues is more accurate than existing Hessian-based methods.

What carries the argument

The load-bearing object is the covariance matrix $\Sigma = H_I^{-1}$ of the LiDAR point-to-plane measurement, split into rotation and translation blocks $\Sigma_{rr}$ and $\Sigma_{tt}$. Eigen-decomposition of these blocks gives the principal directions of uncertainty and their variances; a variance above a threshold marks a degenerate direction. A block-diagonal rotation matrix $V = \mathrm{diag}(V_r, V_t)$ aligns the state coordinates with these principal directions, and a diagonal selective matrix $S$ retains only the degenerate diagonal entries. The filter then replaces the full visual information matrix $J_I = J^\top Q^{-1} J$ and measurement vector $b$ with the projected quantities $J'_I = V S V^\top J_I V S V^\top$ and $b' = V S V^\top J_I V S V^\top J_I^{-1} b$, and runs the standard Kalman update with these replacements.

What would settle it

Construct a synthetic LiDAR-visual configuration where the visual Jacobian J has nonzero coupling between a degenerate and a non-degenerate direction, so that the information matrix J_I has off-diagonal blocks. Compute the true posterior covariance from the projected pseudo-measurement y = V S V^T $J_I^{{-1}}$ b and compare it with the covariance produced by Eq. (34). If the two disagree, the selective update is overconfident and the paper's uncertainty model is false.

Watch

Extended reading notes

Core claim

The central claim is that the information contained in visual measurements is only needed in the directions where the LiDAR constraints are weak; fusing all visual data in all directions adds cost and can actively degrade accuracy. The paper formalizes this as a projection operation: after detecting degenerate directions from the eigenvalue decomposition of the covariance blocks, it defines a selective matrix S that zeroes out non-degenerate directions, and it derives a modified Kalman update (Eqs. 33–34) that uses this projected pseudo-measurement. The proposed degeneracy detector uses the inverse of the LiDAR information matrix, so that the diagonal blocks of the covariance reflect the actual variance of rotation and translation after accounting for the coupling between them, unlike Hessian-based methods that treat the blocks separately. Experiments on the R3LIVE datasets show that embedding this selective filter (SKF-Fusion) yields end-to-end errors as good as or better than R3LIVE while cutting per-frame visual processing time by roughly an order of magnitude.

Load-bearing premise

The filter's uncertainty after projecting visual data is correct only if the visual information in the directions we keep is uncorrelated with the information in the directions we discard; if that correlation is nonzero, the filter claims to know more than the visual measurement actually tells it.

Editorial extensions

If this is right

  • The computation time of the visual front end becomes proportional to the frequency of degeneracy rather than the full frame rate; in normal conditions the visual subsystem only maintains state without updating it.
  • The system becomes robust to visual failures (motion blur, lighting changes) because visual measurements are only trusted when LiDAR is degenerate, reducing the chance of injecting visual errors.
  • The method generalizes beyond LiDAR-visual fusion to any pair of sensors where one is more reliable but sometimes degenerate, such as wheel encoders or GPS, as the authors note.
  • The proposed degeneracy detector provides a physically meaningful, threshold-based test (variance in radians squared and meters squared) that can be used independently as a degeneracy monitoring tool.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the assumption of block-diagonality (independence between selected and unselected directions) fails, the filter's posterior covariance understates uncertainty; a practitioner could recover the correct weight by using the pseudo-inverse of the projected covariance rather than $J'_I$.
  • The speed gain in the visual subsystem could allow the freed computation to be spent on higher-resolution visual processing, denser mapping, or more frequent LiDAR updates, potentially improving accuracy further.
  • The degeneracy detection principle — using the inverse information matrix rather than its blocks — could also improve observability analysis in other filtering-based fusion problems where state variables have different units.
  • The threshold selection is currently manual and tuned per voxel resolution; an adaptive or learned threshold could make the method work across different LiDAR configurations without retuning.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper proposes a LiDAR-visual-inertial SLAM fusion strategy that fuses visual measurements only when the LiDAR subsystem is deemed degenerate. Degeneracy is detected by eigendecomposing the marginal covariance blocks of Sigma = H_I^{-1}, which correctly accounts for rotation-translation coupling through the Schur complement, in contrast to methods that decompose the diagonal blocks of H_I separately. For fusion, the visual normal equations are projected onto the degenerate subspace with V S V^T and inserted into an information-form Kalman update. Experiments on R3LIVE datasets report accuracy comparable to R3LIVE with substantially reduced visual processing time.

Significance. The degeneracy detector in Section IV is mathematically sound and could be useful beyond this paper: the marginal covariances Sigma_rr and Sigma_tt are the correct rotation and translation uncertainties of the LiDAR least-squares estimate, and the paper explicitly shows why they differ from the inverse blocks of H_I. The paper also states that its code is openly available and reports a clear real-time benefit. However, the selective-update derivation in Section V is incorrect as written, so the central theoretical claim requires substantial revision.

major comments (2)
  1. [V, Eqs. (24)-(26); VI, Eqs. (33)-(34)] The projected visual estimate tilde x = V S V^T x_hat has covariance V S V^T J_I^{-1} V S V^T, so its information matrix is the Moore-Penrose inverse of that covariance, not J'_I = V S V^T J_I V S V^T. These two expressions agree only when V S V^T and J_I commute, which the paper does not prove and which is false in general. For example, with L = diag(1,0) and J_I = [[a,b],[b,c]], one obtains L J_I L = a, whereas (L J_I^{-1} L)^+ = a - b^2/c < a. Consequently, Eq. (34) underestimates the posterior covariance whenever the selected and discarded visual directions are correlated. This is not a presentation issue: the claim that the Selective Kalman Filter introduces fewer errors from visual measurements depends on this update being a valid Kalman update. The authors should re-derive Eqs. (33)-(34) using the correct information matrix of the projected measurement, or prove that the cross terms vanish.
  2. [VIII, Table II and accompanying text] The accuracy comparison does not support the broad claim of fewer errors from visual measurements. In Table II, SKF-Fusion is sometimes worse than R3LIVE (hku park 00: 0.080 m vs 0.076 m) and only marginally better in most other sequences, and Section VIII notes that the datasets do not introduce visual disturbances such as drastic lighting changes. The well-supported benefit is the large reduction in visual subsystem computation time (Fig. 10, Table II); the accuracy and robustness claims should be stated more cautiously or tested under the visual failure modes named in the Introduction.
minor comments (5)
  1. [V, Eq. (25)] The word 'implified' should be 'simplified', and the notation 'JIV SV^T' should be typeset as J_I V S V^T for readability.
  2. [IV, thresholds] The thresholds theta_r and theta_t are described only as set based on practical requirements and experience; given that the Conclusion admits they do not adapt across voxel resolutions, a sensitivity analysis or calibration procedure should be added.
  3. [VIII, Figs. 7 and 10] The timing comparisons report only visual-subsystem cost, not total system runtime; the real-time claim should be scoped accordingly.
  4. [VIII, experimental protocol] The evaluation uses a single run per R3LIVE sequence and no statistical analysis; reporting multiple runs or at least error bars would strengthen the comparison.
  5. [Abstract] The abstract states that the code is openly available, but no repository URL is given in the manuscript.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; the Selective Kalman Filter update is an algebraic substitution, with only a non-load-bearing self-citation and experience-tuned thresholds.

full rationale

No significant circularity. The core update Eqs. (33)-(34) is obtained by substituting Eq. (26) into the standard FAST-LIO Kalman update Eqs. (27)-(32); this is an algebraic substitution, not an equivalence-to-inputs. The selective matrix S is chosen from the eigenvector decomposition of covariance blocks (Eqs. (13)-(14)), independent of the visual processing chain, so the 'when/how' selection is not defined in terms of the predicted outcome. The one self-citation to MM-LINS [15] appears in the introduction only as positioning ('MM-LINS [15] is our previously proposed work, which includes degeneracy detection, but it does not have a dedicated analysis and experiments specifically for degeneracy detection'), and no load-bearing theorem or uniqueness claim is imported from it. Thresholds theta_r and theta_t are 'set based on practical requirements and experience' (Sec. IV) and held fixed across experiments; this is a tuning limitation, and the paper's own conclusion admits the thresholds 'do not robustly adapt to LIO across different voxel resolutions' (Sec. IX). A reviewer concern worth noting but not circular: J'_I in Eq. (26) is treated as the information matrix of the projected visual estimate without proof that selected and unselected directions are uncorrelated; this is a mathematical correctness/reliability issue, not a self-referential derivation. Because the method is benchmarked against external R3LIVE datasets and the central derivation is an algebraic manipulation of a standard Kalman filter, no prediction reduces to a fitted input or to a self-citation chain.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim depends on two fitted thresholds and on a pseudo-measurement construction whose information matrix is inconsistent with the covariance of the projected visual estimate. No new physical entities are introduced. The covariance-based degeneracy detector itself relies only on standard linear algebra and the inherited sensor models.

free parameters (2)
  • theta_r (rotational degeneracy threshold) = not reported
    Section IV: 'set two variance thresholds theta_r and theta_t based on practical requirements and experience.' Controls when visual fusion is triggered.
  • theta_t (translational degeneracy threshold) = not reported
    Same as theta_r; conclusion admits thresholds do not robustly adapt to different voxel resolutions.
assumptions (5)
  • standard math Eigenvalue decomposition and Schur complement identities for block covariance matrices
    Equations (10)-(14); standard linear algebra.
  • domain assumption LiDAR point-to-plane measurement model linearized as z = Hx + v
    Inherited from FAST-LIO, Section II; standard LIO assumption.
  • domain assumption Visual measurement model linearized as b = Jx + w
    Section II; standard VIO assumption.
  • domain assumption LiDAR-inertial odometry is more accurate and robust than visual-inertial odometry when not degenerate
    Section III; used to justify not fusing visual data normally; supported only by cited surveys, not by local validation.
  • ad hoc to paper The projected visual estimate D x_hat_vis has information matrix D JI D
    Section V Eq. (24)-(26) and Section VI Eq. (33)-(34); this is the load-bearing assumption and is false in general; the correct information weight is (D JI^{-1} D)^+.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Selective Kalman Filter: When and How to Fuse Multi-Sensor Information to Overcome Degeneracy in SLAM." pith.science (2026). https://pith.science/paper/H3OUMOYG

@misc{pith2026241217235,
  author       = {Pith},
  title        = {Pith review of: Selective Kalman Filter: When and How to Fuse Multi-Sensor Information to Overcome Degeneracy in SLAM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H3OUMOYG}},
  note         = {Machine review of arXiv:2412.17235}
}
read the original abstract

Research trends in SLAM systems are now focusing more on multi-sensor fusion to handle challenging and degenerative environments. However, most existing multi-sensor fusion SLAM methods mainly use all of the data from a range of sensors, a strategy we refer to as the all-in method. This method, while merging the benefits of different sensors, also brings in their weaknesses, lowering the robustness and accuracy and leading to high computational demands. To address this, we propose a new fusion approach -- Selective Kalman Filter -- to carefully choose and fuse information from multiple sensors (using LiDAR and visual observations as examples in this paper). For deciding when to fuse data, we implement degeneracy detection in LiDAR SLAM, incorporating visual measurements only when LiDAR SLAM exhibits degeneracy. Regarding degeneracy detection, we propose an elegant yet straightforward approach to determine the degeneracy of LiDAR SLAM and to identify the specific degenerative direction. This method fully considers the coupled relationship between rotational and translational constraints. In terms of how to fuse data, we use visual measurements only to update the specific degenerative states. As a result, our proposed method improves upon the all-in method by greatly enhancing real-time performance due to less processing visual data, and it introduces fewer errors from visual measurements. Experiments demonstrate that our method for degeneracy detection and fusion, in addressing degeneracy issues, exhibits higher precision and robustness compared to other state-of-the-art methods, and offers enhanced real-time performance relative to the all-in method. The code is openly available.

Figures

Figures reproduced from arXiv: 2412.17235 by the authors.

Figure 1
Figure 1. Performance on localization and mapping after applying four different [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Pipeline of the Selective Kalman Filter in the LIVO system: focusing [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Two-dimensional cross-section diagram illustrating LiDAR [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Pipeline of SKF-Fusion. non-degenerative dimensions. We revert the state to its origi￾nal coordinate system and simultaneously restore the original measurement equation: JIV SV ⊤xˆ = JIV SV T J −1 I b. (24) To preserve the symmetry of the information matrix, we un￾dert…
Figure 6
Figure 6. Figure 6: Degeneracy assessment of six degrees of freedom in the [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: A comparison of the time consumption of the visual subsystems in [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Localization and mapping of SKF-Fusion in the challenging outdoor [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 10
Figure 10. Figure 10: A comparison of the time consumption of the visual subsystems in [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Good Weights: Proactive, Adaptive Dead Reckoning Fusion for Continuous and Robust Visual SLAM

    cs.RO 2025-09 conditional novelty 5.0 of 10

    Gate a dead-reckoning prior by the number of tracked visual features, and visual SLAM stays continuous and accurate in low-texture environments.

  2. Large-Scale UWB Anchor Calibration and One-Shot Localization Using Gaussian Process

    cs.RO 2024-12 conditional novelty 5.0 of 10

    A Gaussian process trained on UWB ranges paired with a LiDAR-inertial trajectory calibrates UWB anchors to about 2 m in a 600 by 450 m obstructed site, and the calibrated anchors improve one-shot LiDAR localization.

Reference graph

Works this paper leans on

22 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    A review of multi-sensor fusion slam systems based on 3d lidar,

    X. Xu, L. Zhang, J. Yang, C. Cao, W. Wang, Y . Ran, Z. Tan, and M. Luo, “A review of multi-sensor fusion slam systems based on 3d lidar,” Remote Sensing , vol. 14, no. 12, p. 2835, 2022

    2022

  2. [2]

    R 3 live: A robust, real-time, rgb-colored, lidar- inertial-visual tightly-coupled state estimation and mapping package,

    J. Lin and F. Zhang, “R 3 live: A robust, real-time, rgb-colored, lidar- inertial-visual tightly-coupled state estimation and mapping package,” in 2022 International Conference on Robotics and Automation (ICRA) . IEEE, 2022, pp. 10 672–10 678

    work page 2022

  3. [3]

    Fast-livo: Fast and tightly-coupled sparse-direct lidar-inertial-visual odometry,

    C. Zheng, Q. Zhu, W. Xu, X. Liu, Q. Guo, and F. Zhang, “Fast-livo: Fast and tightly-coupled sparse-direct lidar-inertial-visual odometry,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 4003–4009

    2022

  4. [4]

    Lvio-fusion:tightly- coupled lidar-visual-inertial odometry and mapping in degenerate environments,

    H. Zhang, L. Du, S. Bao, J. Yuan, and S. Ma, “Lvio-fusion:tightly- coupled lidar-visual-inertial odometry and mapping in degenerate environments,” IEEE Robotics and Automation Letters , vol. 9, no. 4, pp. 3783–3790, 2024

    work page 2024

  5. [5]

    Liver: A tightly coupled lidar-inertial-visual state estimator with high robustness for underground environments,

    T. Wen, Y . Fang, B. Lu, X. Zhang, and C. Tang, “Liver: A tightly coupled lidar-inertial-visual state estimator with high robustness for underground environments,” IEEE Robotics and Automation Letters , vol. 9, no. 3, pp. 2399–2406, 2024

    work page 2024

  6. [6]

    Sensor data fusion using kalman filter,

    J. Sasiadek and P. Hartana, “Sensor data fusion using kalman filter,” in Proceedings of the Third International Conference on Information Fusion, vol. 2. IEEE, 2000, pp. WED5–19

    work page 2000

  7. [7]

    Extended kalman filtering for fuzzy modelling and multi-sensor fusion,

    G. Rigatos and S. Tzafestas, “Extended kalman filtering for fuzzy modelling and multi-sensor fusion,” Mathematical and computer modelling of dynamical systems , vol. 13, no. 3, pp. 251–266, 2007

    work page 2007

  8. [8]

    Multi-sensor optimal information fusion kalman filter,

    S.-L. Sun and Z.-L. Deng, “Multi-sensor optimal information fusion kalman filter,” Automatica, vol. 40, no. 6, pp. 1017–1023, 2004

    work page 2004

Show all 22 references
  1. [9]

    Driftless 3-d attitude determination and positioning of mobile robots by integration of imu with two rtk gpss,

    F. Aghili and A. Salerno, “Driftless 3-d attitude determination and positioning of mobile robots by integration of imu with two rtk gpss,” IEEE/ASME Transactions on Mechatronics , vol. 18, no. 1, pp. 21–31, 2011

    2011

  2. [10]

    On degeneracy of optimization-based state estimation problems,

    J. Zhang, M. Kaess, and S. Singh, “On degeneracy of optimization-based state estimation problems,” in 2016 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2016, pp. 809–816

    2016

  3. [11]

    Degeneracy-aware factors with applications to underwater slam,

    A. Hinduja, B.-J. Ho, and M. Kaess, “Degeneracy-aware factors with applications to underwater slam,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2019, pp. 1293– 1299

    2019

  4. [12]

    Lion: Lidar-inertial observability-aware navigator for vision-denied environments,

    A. Tagliabue, J. Tordesillas, X. Cai, A. Santamaria-Navarro, J. P. How, L. Carlone, and A.-a. Agha-mohammadi, “Lion: Lidar-inertial observability-aware navigator for vision-denied environments,” in Experimental Robotics: The 17th International Symposium . Springer, 2021, pp. 380–390

    2021

  5. [13]

    Switch-slam: Switching-based lidar-inertial-visual slam for degenerate environments,

    J. Lee, R. Komatsu, M. Shinozaki, T. Kitajima, H. Asama, Q. An, and A. Yamashita, “Switch-slam: Switching-based lidar-inertial-visual slam for degenerate environments,” IEEE Robotics and Automation Letters , vol. 9, no. 8, pp. 7270–7277, 2024

    2024

  6. [14]

    X-icp: Localizability-aware lidar registration for robust localization in extreme environments,

    T. Tuna, J. Nubert, Y . Nava, S. Khattak, and M. Hutter, “X-icp: Localizability-aware lidar registration for robust localization in extreme environments,” IEEE Transactions on Robotics , vol. 40, pp. 452–471, 2024

    2024

  7. [15]

    Mm-lins: a multi-map lidar-inertial system for over-degenerate environments,

    Y . Ma, J. Xu, S. Yuan, T. Zhi, W. Yu, J. Zhou, and L. Xie, “Mm-lins: a multi-map lidar-inertial system for over-degenerate environments,” IEEE Transactions on Intelligent V ehicles, pp. 1–11, 2024

    2024

  8. [16]

    Learning-based localizability estimation for robust lidar localization,

    J. Nubert, E. Walther, S. Khattak, and M. Hutter, “Learning-based localizability estimation for robust lidar localization,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2022, pp. 17–24

    2022

  9. [17]

    Fast-lio: A fast, robust lidar-inertial odometry package by tightly-coupled iterated kalman filter,

    W. Xu and F. Zhang, “Fast-lio: A fast, robust lidar-inertial odometry package by tightly-coupled iterated kalman filter,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 3317–3324, 2021

    2021

  10. [18]

    Fast-lio2: Fast direct lidar- inertial odometry,

    W. Xu, Y . Cai, D. He, J. Lin, and F. Zhang, “Fast-lio2: Fast direct lidar- inertial odometry,” IEEE Transactions on Robotics , vol. 38, no. 4, pp. 2053–2073, 2022

    2022

  11. [19]

    A practical survey on visual odometry for autonomous driving in challenging scenarios and conditions,

    L. R. Agostinho, N. M. Ricardo, M. I. Pereira, A. Hiolle, and A. M. Pinto, “A practical survey on visual odometry for autonomous driving in challenging scenarios and conditions,” IEEE Access, vol. 10, pp. 72 182– 72 205, 2022

    2022

  12. [20]

    Viral-fusion: A visual-inertial-ranging-lidar sensor fusion approach,

    T.-M. Nguyen, M. Cao, S. Yuan, Y . Lyu, T. H. Nguyen, and L. Xie, “Viral-fusion: A visual-inertial-ranging-lidar sensor fusion approach,” IEEE Transactions on Robotics , vol. 38, no. 2, pp. 958–977, 2022

    2022

  13. [21]

    R3LIVE Dataset,

    Z. Lin, “R3LIVE Dataset,” https://github.com/ziv-lin/r3live dataset, 2023, accessed: 2023-02-27

    2023

  14. [22]

    R 3 live++: A robust, real-time, radiance reconstruction package with a tightly-coupled lidar-inertial-visual state estimator,

    J. Lin and F. Zhang, “R 3 live++: A robust, real-time, radiance reconstruction package with a tightly-coupled lidar-inertial-visual state estimator,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–18, 2024

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.