Pith. sign in

REVIEW 3 major objections 6 minor 25 references

Chalito: An Extensible Library for Filtering-Based State Estimation in Quadruped Robots

T0 review · 3 major / 6 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Chalito is presented as the first open-source library built only for fair, extensible benchmarking of filter-based state estimators on quadruped robots.

desk verdict Solid systems paper: first dedicated open library for benchmarking quadruped filter estimators, with real interfaces and demos that do what they claim. read the letter →

arxiv 2607.09968 v1 pith:7XSSHESN submitted 2026-07-10 cs.RO

Hilton Marques Souza Santana , João Carlos Virgolino Soares , Marco Antonio Meggiolaro , Claudio Semini This is my paper · ORCID
classification cs.RO
keywords quadrupedrobotsstateestimationKalmanfilteringLiegroupsinvariantEKFbenchmarkingURDFproprioceptiveodometry
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

State estimation for four-legged robots is hard to compare fairly because each published filter is usually locked to one robot, one sensor suite, or one software stack. The authors claim this gap slows innovation and hurts reproducibility. They introduce Chalito (with a Python twin) as a URDF-driven, filter-agnostic library whose core interfaces for Lie groups, filters, measurements, and contact managers let researchers swap robots, estimators, and contact rules without rewriting common code. The library already ships two representative filters, three standard proprioceptive measurement models, a simple force-threshold contact detector, and loaders for both synthetic and real public datasets. Three experiments then show how the same interfaces can run convergence tests under large initial error, Monte-Carlo consistency checks, and position estimation across six different real robots and datasets. The practical promise is that a new filter can be dropped in, scored for accuracy and consistency against prior methods, and only then reimplemented for real-time use.

What carries the argument

The four abstract interfaces (ManifBase for matrix Lie groups including dynamic SE_{2+N}(3), FilterBase with prediction/update hooks, MeasurementBase for motion and observation models, and ContactManagerBase) together with the URDF-driven QuadrupedRobot class; they isolate the novel part of any new estimator so the rest of the pipeline can be reused.

What would settle it

If independent researchers implement a new filter or contact detector inside the published interfaces and find they still must rewrite large amounts of common code, or if side-by-side runs on the same public datasets produce results that cannot be reproduced from the released configuration files, the claim of a usable general benchmarking platform fails.

Watch

Extended reading notes

Core claim

The paper claims that a single extensible MATLAB/Python library with shared interfaces for Lie groups, quadruped kinematics, filters, measurements, and contact managers is sufficient to enable systematic, robot-agnostic benchmarking of filter-based state estimators, and that no prior open-source library has been dedicated exclusively to that task.

Load-bearing premise

The three built-in measurement models and a simple ground-reaction-force threshold for contact already form a representative enough core that comparisons run through them will generalize to the broader class of quadruped filters.

Editorial extensions

If this is right

  • New filters can be scored for accuracy and consistency on the same robots and public datasets without reimplementing kinematics, IMU integration, or visualization.
  • Researchers can isolate the effect of contact estimation or measurement models by swapping only the corresponding interface while keeping the rest of the pipeline fixed.
  • Monte-Carlo and large-error convergence studies become routine rather than one-off reimplementations.
  • A reference trajectory and NEES profile can be generated offline before any real-time C++/ROS version is written.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the library already exports TUM trajectories, the same pipeline can later feed standard absolute- and relative-pose error tools without further glue code.
  • The same interface pattern could be reused for smoothing algorithms or for exteroceptive measurements once a memory-efficient export path is added, as the authors themselves sketch in the conclusions.
  • If the contact-manager interface is left unchanged, learned contact detectors can be dropped in and scored against the force-threshold baseline on every dataset already supported.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Chalito (and PyChalito), a MATLAB/Python library for benchmarking filter-based state estimation on quadruped robots. It provides URDF-driven robot models, a ManifBase interface covering matrix Lie groups including dynamic SE_{2+N}(3), FilterBase prediction/update methods, MeasurementBase for InertialOdometry, LegKinematics and BaseVelocity, and ContactManagerBase (default GRF threshold). Two example filters (IEKF and SO(3)-EKF) are implemented and evaluated in three settings: large-initial-error convergence in MuJoCo, synthetic Monte Carlo with NEES, and position estimation on six real public datasets spanning multiple robots. The central claim is that this is the first open-source library exclusively dedicated to such benchmarking, with extensible interfaces that allow new filters, measurements, and contact managers without changing the exposed API.

Significance. If the library is released as described and remains usable, the contribution is practically useful for the quadruped state-estimation community: fair cross-robot, cross-filter comparisons of accuracy and consistency have been hindered by robot- and stack-specific C++/ROS implementations. Shipping ManifBase with dynamic SE_{2+N}(3), a clear FilterBase/MeasurementBase contract, and reproducible demos that recover the known IEKF vs SO(3)-EKF consistency advantage (including NEES) is a concrete systems contribution. The work is more infrastructure than algorithmic novelty, but that is appropriate for a library paper and can accelerate prototyping and education. Significance is contingent on actual open-source availability, documentation quality, and whether the interfaces prove easy to extend in practice.

major comments (3)
  1. The central claim (Abstract; Contributions; Sec. II) is that Chalito is the first open-source library exclusively for benchmarking filter-based quadruped estimators and is easily extended. The manuscript does not report a public repository URL, license, installation instructions, or a minimal worked example of adding a third filter or a non-GRF contact manager. Without that evidence, the extensibility and "first dedicated library" claims cannot be independently verified by readers. A revision should include the release link and a short appendix or README-level walkthrough that exercises the extension path claimed in Sec. III.
  2. Sec. III-C/D and Table V ship only three proprioceptive measurement models and a single GRF-threshold contact manager, while Sec. II and the conclusions advertise the library as a general benchmarking platform for the broader class of filter-based quadruped estimators (including methods that use learned contact or richer sensing). The demos (Sec. IV) successfully show accuracy/consistency evaluation for the two example filters, but they do not demonstrate that a new measurement type or contact estimator can be plugged in without modifying FilterBase. Either add one non-trivial extension example (e.g., a second contact manager or an additional measurement) or narrow the claim language so that "benchmarking" is scoped to the currently implemented proprioceptive core.
  3. Sec. IV-C reports qualitative trajectory plots (Fig. 5) and states that IEKF "outperformed" SO(3)-EKF on most datasets, attributing failures of the latter to false observability. The text mentions TUM trajectory output and evo-style ATE/RPE evaluation but does not report numerical ATE/RPE (or similar) tables for the six datasets under a fixed protocol. Without those numbers, the real-world benchmarking claim is only partially substantiated. Adding a compact quantitative table (same noise parameters as Eq. (21), same contact threshold) would make the third evaluation load-bearing rather than illustrative.
minor comments (6)
  1. Table II lists SO(3)×R^{6+3N} and related products; the body text (Eq. (1)) uses SO(3)×R^{12+3N}. Align the dimension notation consistently.
  2. Fig. 1 caption and architecture diagram are helpful; ensure the Python (PyChalito) side is either mirrored in a figure or clearly stated as sharing the same interfaces via QuadrupedPyMPC, since the MATLAB-only QuadrupedRobot class is otherwise a portability gap.
  3. Eq. (4) and the subsequent Monte Carlo conversion δx_{0} ≈ J ξ_{0} cite [22, Eq. 76]; a one-line definition or pointer in the main text would help readers who do not have that reference open.
  4. Several arXiv preprints in the reference list (e.g., [6], [7], [19], [22]) are dated 2023–2026; verify final bibliographic details and that self-citations to the authors’ prior filter work are clearly marked as example filters rather than independent baselines.
  5. Typographical consistency: "SO3EKF" vs "SO(3)-EKF", "S0_3" vs "SO(3)" in Fig. 1 class names, and occasional spacing issues (e.g., "SE 2+N(3)") should be cleaned for camera-ready.
  6. Sec. V future work on exteroceptive measurements via a parallel C++ exporter is reasonable; a sentence on how MeasurementBase would remain the single integration point would strengthen the extensibility narrative.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: systems/library claim is self-contained; demos reuse known filters and public data without reducing predictions to fitted inputs or self-citation chains.

full rationale

Chalito is a software contribution whose central claim (first extensible open-source library for benchmarking filter-based quadruped estimators via URDF-driven ManifBase/FilterBase/MeasurementBase/ContactManagerBase interfaces) is a systems existence and usability claim, not a first-principles derivation or numerical prediction. The two shipped filters (IEKF on SE_{2+N}(3) imes R^6 and SO(3)-EKF) are standard formulations taken from the literature (Hartley, Bloesch, Barrau–Bonnabel, etc.); their prediction/update equations and Jacobians are written out explicitly and exercised on MuJoCo, synthetic Monte-Carlo (NEES), and six public real datasets. Self-citations ([22], [23], [7]) point to the authors’ prior filter or benchmark work used only as example implementations or related context; they are not invoked as uniqueness theorems, load-bearing premises, or the sole support for the library claim. No parameter is fitted to data and then re-labeled a prediction; no ansatz is smuggled via self-citation; no known empirical pattern is merely renamed. The paper is therefore free of the enumerated circularity patterns. Scope limitations (only three proprioceptive measurements and a GRF-threshold contact manager) are acknowledged by the authors as intentional starting points for an extensible interface and do not create definitional circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The paper is primarily a software systems contribution. Its load-bearing premises are standard robotics modeling choices (URDF kinematics, IMU and encoder noise models, GRF-threshold contact) plus the claim that abstracting Lie groups, filters, measurements, and contact managers is enough for useful offline benchmarking. Free parameters are the usual filter noise covariances and thresholds chosen for the demos; no new physical entities are invented.

free parameters (2)
  • Process and measurement noise covariances (Qa, Qg, Qba, Qbg, Qv, Qf, Qk) and initial P0 = e.g. Qv = diag(0.280, 0.009, 0.018) on real data; other diagonal entries listed in (20)–(21)
    Hand-chosen or estimated from a preliminary run for each experiment (Eqs. 20–21 and MuJoCo section); they control reported convergence and NEES but are not derived from first principles.
  • GRF contact threshold λ
    Private member of the ThresholdGRF contact manager; contact set S depends on this hand-set threshold.
assumptions (4)
  • domain assumption Matrix Lie groups and right-invariant error-state Kalman filtering correctly capture the geometry of quadruped pose/velocity/foot-position estimation.
    Invoked throughout Section III-A/C via ManifBase and the IEKF / SO(3)-EKF formulations citing Barrau/Bonnabel and prior quadruped IEKF work.
  • domain assumption URDF plus joint encoders and foot force sensors supply sufficient proprioceptive information for the shipped measurement models.
    Section III-B/C and Table V; library scope is restricted to proprioceptive filtering.
  • ad hoc to paper A GRF z-threshold is an acceptable default contact manager for benchmarking, with the interface left open for better methods.
    Section II and III-B explicitly call this the simplest method while abstracting ContactManagerBase.
  • domain assumption Piecewise-constant IMU measurements with additive white noise and random-walk biases are an adequate process model for the prediction step.
    Equations (5)–(8) in the prediction step.
invented entities (1)
  • Chalito / PyChalito library and its core interfaces (ManifBase, FilterBase, MeasurementBase, ContactManagerBase, QuadrupedRobot)
    purpose: Provide a reusable, extensible offline benchmarking stack for filter-based quadruped state estimation.
    The library is the paper's primary contribution; independent evidence will be external adoption and third-party extensions, not yet shown in the manuscript.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chalito: An Extensible Library for Filtering-Based State Estimation in Quadruped Robots." pith.science (2026). https://pith.science/paper/7XSSHESN

@misc{pith2026260709968,
  author       = {Pith},
  title        = {Pith review of: Chalito: An Extensible Library for Filtering-Based State Estimation in Quadruped Robots},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7XSSHESN}},
  note         = {Machine review of arXiv:2607.09968}
}
read the original abstract

State estimation is essential for quadruped robots, enabling robust locomotion, navigation, and control. While many estimators have been proposed in the literature, existing implementations are often tied to specific robots or software stacks, making fair comparisons difficult. This lack of a general-purpose benchmarking framework hinders reproducibility and slows down algorithmic innovation. In this paper, we introduce Chalito, an extensible MATLAB/Python library for benchmarking filter-based state estimation algorithms in quadruped robots. Chalito imports robot models directly from URDF, supports multiple filtering approaches, and is designed to be easily extended with new methods. The framework runs on both simulated and real datasets, enabling systematic evaluation across robots and filters. To the best of our knowledge, this is the first open-source library exclusively dedicated to benchmarking filtering algorithms for quadruped robots.

Figures

Figures reproduced from arXiv: 2607.09968 by the authors.

Figure 1
Figure 1. MATLAB implementation of the Chalito architecture. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Trajectories, GRFs and relative position of the feet in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Time of convergence of the two filters considered in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Results of the numerical Monte Carlo experiment with synthetic data. a) Average normalized error for the velocity in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Position estimates obtained with IEKF and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 9 linked inside Pith

  1. [1]

    Contact-aided invariant extended kalman filtering for robot state es- timation,

    R. Hartley, M. Ghaffari Jadidi, R. M. Eustice, and J. W. Grizzle, “Contact-aided invariant extended kalman filtering for robot state es- timation,”Int. J. Robot. Res., vol. 39, 2019

    2019

  2. [2]

    Pronto: A multi- sensor state estimator for legged robots in real-world scenarios,

    M. Camurri, M. Ramezani, S. Nobili, and M. Fallon, “Pronto: A multi- sensor state estimator for legged robots in real-world scenarios,”Front. Robot. AI, vol. 7, pp. 1–18, 2020

    2020

  3. [3]

    Muse: A real-time multi-sensor state estimator for quadruped robots,

    Y . Nistic`o, J. C. V . Soares, L. Amatucci, G. Fink, and C. Semini, “Muse: A real-time multi-sensor state estimator for quadruped robots,”IEEE Robot. Autom. Lett., vol. 10, no. 5, 2025

    2025

  4. [4]

    State estimation for legged robots: Consis- tent fusion of leg kinematics and imu,

    M. Bloesch, M. Hutter, M. Hoepflinger, S. Leutenegger, C. Gehring, C. Remy, and R. Siegwart, “State estimation for legged robots: Consis- tent fusion of leg kinematics and imu,” inRSS, 2012

    2012

  5. [5]

    State estimation for legged robots on unstable and slippery terrain,

    M. Bloesch, C. Gehring, P. Fankhauser, M. Hutter, M. A. Hoepflinger, and R. Siegwart, “State estimation for legged robots on unstable and slippery terrain,” inIROS, 2013, pp. 6058–6064

    2013

  6. [6]

    Proprioceptive invariant robot state estimation,

    T.-Y . Lin, T. Li, W. Tong, and M. Ghaffari, “Proprioceptive invariant robot state estimation,” 2023,arXiv:2311.04320

    arXiv 2023

  7. [7]

    A proprioceptive- only benchmark for quadruped state estimation: Ate, rpe, and runtime trade-offs between filters and smoothers,

    Y . Nistic `o, J. C. V . Soares, J. Sol `a, and C. Semini, “A proprioceptive- only benchmark for quadruped state estimation: Ate, rpe, and runtime trade-offs between filters and smoothers,” 2026,arXiv:2605.11674

    arXiv 2026

  8. [8]

    Mujoco: A physics engine for model- based control,

    E. Todorov, T. Erez, and Y . Tassa, “Mujoco: A physics engine for model- based control,” inIROS, 2012, pp. 5026–5033

    2012

Show all 25 references
  1. [9]

    On the benefits of gpu sample-based stochastic predictive controllers for legged locomotion,

    G. Turrisi, V . Modugno, L. Amatucci, D. Kanoulas, and C. Semini, “On the benefits of gpu sample-based stochastic predictive controllers for legged locomotion,” inIROS, 2024, pp. 13 757–13 764

    2024

  2. [10]

    An ekf-slam algorithm with consistency properties,

    A. Barrau and S. Bonnabel, “An ekf-slam algorithm with consistency properties,” 2015,arXiv:1510.06263

    2015 arXiv

  3. [11]

    Analysis and improvement of the consistency of extended kalman filter based slam,

    G. P. Huang, A. I. Mourikis, and S. I. Roumeliotis, “Analysis and improvement of the consistency of extended kalman filter based slam,” inICRA, 2008, pp. 473–479

    2008

  4. [12]

    Kalman filters on differentiable mani- folds,

    D. He, W. Xu, and F. Zhang, “Kalman filters on differentiable mani- folds,” 2021,arXiv:2102.03804

    2021 arXiv

  5. [13]

    Manif: A micro lie theory library for state estimation in robotics applications,

    J. Deray and J. Sol `a, “Manif: A micro lie theory library for state estimation in robotics applications,”JOSS, 2020

    2020

  6. [14]

    Cerberus: Low-drift visual-inertial-leg odometry for agile locomotion,

    S. Yang, Z. Zhang, Z. Fu, and Z. Manchester, “Cerberus: Low-drift visual-inertial-leg odometry for agile locomotion,” 2022, arXiv:2209.07654

    2022 arXiv

  7. [15]

    Probabilistic contact estimation and impact detection for state estimation of quadruped robots,

    M. Camurri, M. Fallon, S. Bazeille, A. Radulescu, V . Barasuol, D. G. Caldwell, and C. Semini, “Probabilistic contact estimation and impact detection for state estimation of quadruped robots,”IEEE Robot. Autom. Lett., vol. 2, no. 2, pp. 1023–1030, 2017

    2017

  8. [16]

    Legged robot state estimation using invariant kalman filtering and learned contact events,

    T.-Y . Lin, R. Zhang, J. Yu, and M. Ghaffari, “Legged robot state estimation using invariant kalman filtering and learned contact events,” 2021,arXiv:2106.15713

    2021 arXiv

  9. [17]

    Multi-imu proprio- ceptive odometry for legged robots,

    S. Yang, Z. Zhang, B. Bokser, and Z. Manchester, “Multi-imu proprio- ceptive odometry for legged robots,” inIROS, 2023, pp. 774–779

    2023

  10. [18]

    Leg-kilo: Robust kinematic-inertial-lidar odometry for dynamic legged robots,

    G. Ou, D. Li, and H. Li, “Leg-kilo: Robust kinematic-inertial-lidar odometry for dynamic legged robots,”IEEE Robot. Autom. Lett., vol. 9, no. 10, pp. 8194–8201, 2024

    2024

  11. [19]

    Grandtour: A legged robotics dataset in the wild for multi-modal perception and state estimation,

    J. Frey, T. Tuna, F. Fu, K. Patterson, T. Xu, M. Fallon, C. Cadena, and M. Hutter, “Grandtour: A legged robotics dataset in the wild for multi-modal perception and state estimation,” 2026,arXiv:2602.18164

    2026 arXiv

  12. [20]

    A micro lie theory for state estimation in robotics,

    J. Sola, J. Deray, and D. Atchuthan, “A micro lie theory for state estimation in robotics,” 2018,arXiv:1812.01537. 9

    2018 arXiv

  13. [21]

    The geometry of navigation problems,

    A. Barrau and S. Bonnabel, “The geometry of navigation problems,” IEEE Trans. Autom. Control, vol. 68, no. 2, 2022

    2022

  14. [22]

    Iterated invariant ekf for quadruped robot odometry,

    H. M. S. Santana, J. C. V . Soares, S. Goffin, Y . Nistic `o, S. Bonnabel, C. Semini, and M. A. Meggiolaro, “Iterated invariant ekf for quadruped robot odometry,” 2026,arXiv:2604.15449

    2026 arXiv

  15. [23]

    Proprioceptive state estimation for quadruped robots using invariant kalman filtering and scale-variant robust cost functions,

    H. M. S. Santana, J. C. V . Soares, Y . Nistic `o, M. A. Meggiolaro, and C. Semini, “Proprioceptive state estimation for quadruped robots using invariant kalman filtering and scale-variant robust cost functions,” in Humanoids, 2024

    2024

  16. [24]

    A benchmark for the evaluation of rgb-d slam systems,

    J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of rgb-d slam systems,” inIROS, 2012

    2012

  17. [25]

    evo: Python package for the evaluation of odometry and slam,

    M. Grupp, “evo: Python package for the evaluation of odometry and slam,” 2017

    2017

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.