pith. sign in

arxiv: 2606.09237 · v1 · pith:ROT7W5QPnew · submitted 2026-06-08 · 💻 cs.RO · cs.SY· eess.SY

Can we stabilize an inverted pendulum with feedback from a time-of-flight camera?

Anthony Czubarow , Antonio Terpin , Raffaello D'Andrea This is my paper

Pith reviewed 2026-06-27 16:46 UTC · model grok-4.3

classification 💻 cs.RO cs.SYeess.SY
keywords time-of-flight camerainverted pendulumfeedback controlcart-poledepth sensingroboticsunstable dynamics
0
0 comments X

The pith

An inexpensive low-resolution time-of-flight camera supplies enough depth feedback to reliably balance an inverted pendulum on a cart.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that depth data from a compact, low-cost time-of-flight camera suffices for precise stabilization of an inverted pendulum on a cart, a standard test of fast unstable dynamics. This runs against the prevailing view that the camera's coarse spatial resolution and depth noise rule out high-performance feedback. A sympathetic reader would see the result as evidence that affordable sensors can handle control tasks previously thought to require finer or more expensive instrumentation. The demonstration is performed on the canonical cart-pole benchmark, so success directly addresses a well-understood performance target.

Core claim

An inexpensive, low-resolution time-of-flight camera provides sufficient feedback to reliably and precisely balance an inverted pendulum on a cart—a canonical benchmark for fast, unstable dynamics.

What carries the argument

Depth measurements from the time-of-flight camera used as state feedback in the cart-position control loop.

If this is right

  • Cart-pole balancing becomes feasible with a single compact depth sensor instead of higher-resolution or multi-sensor setups.
  • The same sensor class can be considered for other fast unstable plants that need real-time position or angle feedback.
  • Cost and size constraints on the sensing hardware can be relaxed while still meeting benchmark stability requirements.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The result suggests similar low-resolution depth cameras could be tested on related benchmarks such as balancing a two-wheeled robot or stabilizing a quadrotor attitude.
  • If the noise characteristics prove tolerable here, the same camera might be evaluated for tasks that combine depth sensing with slower but still dynamic motion, such as object tracking on a moving platform.

Load-bearing premise

Low spatial resolution and depth noise in time-of-flight cameras do not prevent precise feedback control of fast unstable systems.

What would settle it

The pendulum falls or drifts outside a tight angle bound when the controller receives only the camera's depth stream and no other sensors.

Figures

Figures reproduced from arXiv: 2606.09237 by Anthony Czubarow, Antonio Terpin, Raffaello D'Andrea.

Figure 1
Figure 1. Figure 1: Our inverted pendulum (the gold pen standing upright) on a cart [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Top: 3D-printed carriage with red highlights indicating the cylin [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: ToF camera (left) and its positioning (right). [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Schematic of our inverted pendulum on a cart. [PITH_FULL_IMAGE:figures/full_fig_p003_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Raw angle estimation with the ToF camera. We show the depth [PITH_FULL_IMAGE:figures/full_fig_p004_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: States trajectories over 30 trials. We report the pendulum angle as quantified via the encoder. values of RMSE(ϕraw, ϕgt) = 3.0 × 10−3 rad. Overall, we believe these results show the potential of ToF cameras for precise feedback control. IV. CONCLUSIONS In this paper, we revisit a canonical control benchmark with a practical question in mind: can an inexpensive, low-resolution ToF camera deliver measuremen… view at source ↗
read the original abstract

Time-of-flight cameras are popular in robotics for providing direct depth information while being compact, inexpensive, and robust to lighting conditions, but their low spatial resolution and depth noise are widely believed to preclude precise feedback control. In this paper, we show that an inexpensive, low-resolution time-of-flight camera provides sufficient feedback to reliably and precisely balance an inverted pendulum on a cart--a canonical benchmark for fast, unstable dynamics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 1 minor

Summary. The paper claims that an inexpensive, low-resolution time-of-flight camera supplies sufficient feedback to reliably and precisely stabilize an inverted pendulum on a cart, a canonical benchmark for fast unstable dynamics, contrary to the widespread view that ToF depth noise and low spatial resolution preclude such precise control.

Significance. If substantiated with methods, noise modeling, and experimental validation, the result would demonstrate that compact low-cost ToF sensors can handle high-bandwidth stabilization tasks, broadening their use in robotics beyond perception to closed-loop control of unstable systems and potentially lowering hardware costs for benchmark problems.

minor comments (1)
  1. The provided text consists solely of the abstract; no sections, equations, sensor models, control laws, experimental setup, or results are available to evaluate the central claim or the weakest assumption regarding depth noise.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their summary and assessment of our manuscript. No specific major comments were enumerated in the report, so we have no point-by-point responses to provide. We note that the manuscript already contains the methods, noise characterization, and experimental results needed to support the claims.

Circularity Check

0 steps flagged

No derivation chain present; empirical demonstration only

full rationale

The paper's abstract and context present a purely empirical claim: an inexpensive ToF camera suffices for balancing an inverted pendulum, with no equations, parameter fits, predictions, self-citations, or theoretical derivations exhibited. The central result is a physical demonstration rather than a reduction of any output to fitted inputs or prior self-referential steps. No load-bearing mathematical steps exist to inspect for circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Review based solely on abstract; no free parameters, axioms, or invented entities are described.

pith-pipeline@v0.9.1-grok · 5597 in / 856 out tokens · 21076 ms · 2026-06-27T16:46:03.025121+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 1 linked inside Pith

  1. [1]

    Grzegorzek, C

    M. Grzegorzek, C. Theobalt, R. Koch, and A. Kolb, Eds.,Time-of- Flight and Depth Imaging: Sensors, Algorithms, and Applications, ser. Lecture Notes in Computer Science. Springer, 2013

    2013

  2. [2]

    Hansard, S

    M. Hansard, S. Lee, O. Choi, and R. Horaud,Time-of-Flight Cam- eras: Principles, Methods and Applications, ser. Springer Briefs in Computer Science. Springer, 2013

    2013

  3. [3]

    Robust 3d- mapping with time-of-flight cameras,

    S. May, D. Droeschel, S. Fuchs, D. Holz, and A. Nüchter, “Robust 3d- mapping with time-of-flight cameras,” in2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2009, pp. 1673–1678

    2009

  4. [4]

    Tof cameras for active vision in robotics,

    G. Alenyà, S. Foix, and C. Torras, “Tof cameras for active vision in robotics,”Sensors and Actuators A: Physical, vol. 218, pp. 10–22, 2014

    2014

  5. [5]

    Autonomous grasp and manipulation planning using a tof camera,

    Z. Xue, S. W. Ruehl, A. Hermann, T. Kerscher, and R. Dillmann, “Autonomous grasp and manipulation planning using a tof camera,” Robotics and Autonomous Systems, vol. 60, no. 3, pp. 387–395, 2012, autonomous Grasping

    2012

  6. [6]

    Robotic visual servoing for tracking structureless moving parts,

    C. Schroder, M. Schumann, R. Loser, P. Klimant, and M. Dix, “Robotic visual servoing for tracking structureless moving parts,” in 2024 7th Iberian Robotics Conference (ROBOT), 2024, pp. 1–6

    2024

  7. [7]

    Uncalibrated visual servoing for intuitive human guidance of robots,

    M. Marshall, M. Matthews, A.-P. Hu, G. McMurray, and H. Lipkin, “Uncalibrated visual servoing for intuitive human guidance of robots,” in2012 IEEE International Conference on Robotics and Automation, 2012, pp. 4463–4468

    2012

  8. [8]

    Using a 3d time-of-flight range camera for visual tracking,

    U. Reiser and J. Kubacki, “Using a 3d time-of-flight range camera for visual tracking,”IFAC Proceedings Volumes, vol. 40, no. 15, pp. 355– 360, 2007, 6th IFAC Symposium on Intelligent Autonomous Vehicles

    2007

  9. [9]

    Visual control of robots using range images,

    J. Pomares, P. Gil, and F. Torres, “Visual control of robots using range images,”Sensors, vol. 10, no. 8, pp. 7303–7322, 2010

    2010

  10. [10]

    Lock-in time-of-flight (tof) cameras: A survey,

    S. Foix, G. Alenya, and C. Torras, “Lock-in time-of-flight (tof) cameras: A survey,”IEEE Sensors Journal, vol. 11, no. 9, pp. 1917– 1926, 2011

    1917

  11. [11]

    Environmental effects on measurement uncertainties of time-of-flight cameras,

    S. A. Gudmundsson, H. Aanaes, and R. Larsen, “Environmental effects on measurement uncertainties of time-of-flight cameras,” in2007 International Symposium on Signals, Circuits and Systems, vol. 1, 2007, pp. 1–4

    2007

  12. [12]

    Swinging up a pendulum by energy control,

    K. Åström and K. Furuta, “Swinging up a pendulum by energy control,”Automatica, vol. 36, no. 2, pp. 287–295, 2000

    2000

  13. [13]

    Feedback control of dynamic systems,

    G. Franklin, J. D. Powell, and A. Emami-Naeini, “Feedback control of dynamic systems,” inASME International Mechanical Engineering Congress and Exposition, vol. 14148. American Society of Mechan- ical Engineers, 1994, pp. 1053–1054

    1994

  14. [14]

    Underactuated robotics: Learning, planning, and control for efficient and agile machines: Course notes for mit 6.832,

    R. Tedrake, “Underactuated robotics: Learning, planning, and control for efficient and agile machines: Course notes for mit 6.832,”Working draft edition, vol. 3, no. 4, p. 2, 2009

    2009

  15. [15]

    Nonlinear controller for an inverted pendulum having restricted travel,

    Q. Wei, W. Dayawansa, and W. Levine, “Nonlinear controller for an inverted pendulum having restricted travel,”Automatica, vol. 31, no. 6, pp. 841–850, 1995

    1995

  16. [16]

    Variable inverted pendulum applied to humanoid motion design,

    T. Zielinska, G. Coba, and W. Ge, “Variable inverted pendulum applied to humanoid motion design,”Robotica, vol. 39, pp. 1–22, 02 2021

    2021

  17. [17]

    Nonlinear control of a swinging pendu- lum,

    C. C. Chung and J. Hauser, “Nonlinear control of a swinging pendu- lum,”Automatica, vol. 31, no. 6, pp. 851–862, 1995

    1995

  18. [18]

    Optimal control of a spatial inverted pendulum using the adjoint method,

    P. Maci ˛ ag, P. Malczyk, and J. Fr ˛ aczek, “Optimal control of a spatial inverted pendulum using the adjoint method,” in2nd International Conference on Machine Design (MD2023), 2023

    2023

  19. [19]

    Real-time h∞ control of networked inverted pendulum visual servo systems,

    D. Du, C. Zhang, Y . Song, H. Zhou, X. Li, and M. Fei, “Real-time h∞ control of networked inverted pendulum visual servo systems,” IEEE Transactions on Cybernetics, vol. 50, no. 12, pp. 5113–5126, 2020

    2020

  20. [20]

    Secure control of networked inverted pendulum visual servo systems based on active disturbance rejection control,

    D. Wu and Q. Lu, “Secure control of networked inverted pendulum visual servo systems based on active disturbance rejection control,” Actuators, vol. 11, no. 12, p. 355, 2022

    2022

  21. [21]

    Stabilization of a 2-dof inverted pendulum by a low cost visual feedback,

    H. Wang, A. Chamroo, C. Vasseur, and V . Koncar, “Stabilization of a 2-dof inverted pendulum by a low cost visual feedback,” in2008 American Control Conference, 2008, pp. 3851–3856

    2008

  22. [22]

    Visual servoing of an inverted pendulum on cart using a mounted smartphone,

    A. Brill, J. A. Frank, and V . Kapila, “Visual servoing of an inverted pendulum on cart using a mounted smartphone,” in2016 American Control Conference (ACC). IEEE, 2016, pp. 1323–1328

    2016

  23. [23]

    Just-in-time control of image-based inverted pendulum systems with a time-delay,

    K.-i. Fukuda, S. Ushida, and K. Deguchi, “Just-in-time control of image-based inverted pendulum systems with a time-delay,” in2006 SICE-ICASE International Joint Conference, 2006, pp. 4016–4021

    2006

  24. [24]

    Design and implementation of robust visual servoing control of an inverted pendulum with an fpga-based image co-processor,

    Y .-W. Tu and M.-T. Ho, “Design and implementation of robust visual servoing control of an inverted pendulum with an fpga-based image co-processor,”Mechatronics, vol. 21, no. 7, pp. 1170–1182, 2011

    2011

  25. [25]

    From pixels to torques with linear feedback,

    J. H. Lee, S. Schoedel, A. Bhardwaj, and Z. Manchester, “From pixels to torques with linear feedback,”arXiv preprint arXiv:2406.18699, 2024

    arXiv 2024

  26. [26]

    Position control of cart-driven inverted pendulum by visual feedback controller using non-calibrated images taken through fisheye lens,

    M. Sato, K. Hatada, and S. Otomo, “Position control of cart-driven inverted pendulum by visual feedback controller using non-calibrated images taken through fisheye lens,”2024 18th International Confer- ence on Control, Automation, Robotics and Vision (ICARCV), pp. 626– 631, 2024

    2024

  27. [27]

    Synthesis of a calibration-free visual feedback controller for an inverted pendulum using a fisheye lens,

    K. Hatada, M. Sato, K. Hirata, and Y . Masui, “Synthesis of a calibration-free visual feedback controller for an inverted pendulum using a fisheye lens,”IEEE Transactions on Industrial Electronics, vol. 69, no. 12, pp. 13 348–13 358, 2022

    2022

  28. [28]

    Vision-based control of a flying spherical inverted pendulum,

    S. Krafes, Z. Chalh, and A. Saka, “Vision-based control of a flying spherical inverted pendulum,” in2018 4th International Conference on Optimization and Applications (ICOA), 2018, pp. 1–6

    2018

  29. [29]

    A flying inverted pendulum,

    M. Hehn and R. D’Andrea, “A flying inverted pendulum,” in2011 IEEE International Conference on Robotics and Automation, 2011, pp. 763–770

    2011

  30. [30]

    Learning fast and precise pixel- to-torque control: A platform for reproducible research of learning on hardware,

    S. Bleher, S. Heim, and S. Trimpe, “Learning fast and precise pixel- to-torque control: A platform for reproducible research of learning on hardware,”IEEE Robotics & Automation Magazine, vol. 29, no. 2, pp. 75–84, 2022

    2022

  31. [31]

    Zero-shot sim-to-real transfer of tactile control policies for aggressive swing-up manipulation,

    T. Bi, C. Sferrazza, and R. D’Andrea, “Zero-shot sim-to-real transfer of tactile control policies for aggressive swing-up manipulation,”IEEE Robotics and Automation Letters, vol. 6, pp. 5761–5768, 2021

    2021

  32. [32]

    Balancing a 3d in- verted pendulum using remote magnetic manipulation,

    J. Zughaibi, B. Nelson, and M. Muehlebach, “Balancing a 3d in- verted pendulum using remote magnetic manipulation,”arXiv preprint arXiv:2402.06012, 2024

    arXiv 2024

  33. [33]

    Understanding robust control theory via stick balancing,

    Y . P. Leong and J. C. Doyle, “Understanding robust control theory via stick balancing,” in2016 IEEE 55th Conference on Decision and Control (CDC). IEEE, 2016, pp. 1508–1514

    2016

  34. [34]

    Using reinforcement learning to probe the role of feedback in skill acquisition,

    A. Terpin and R. D’Andrea, “Using reinforcement learning to probe the role of feedback in skill acquisition,”arXiv preprint arXiv:2512.08463, 2025

    arXiv 2025

  35. [35]

    Robot operating system 2: Design, architecture, and uses in the wild,

    S. Macenski, T. Foote, B. Gerkey, C. Lalancette, and W. Woodall, “Robot operating system 2: Design, architecture, and uses in the wild,” Science robotics, vol. 7, no. 66, p. eabm6074, 2022

    2022

  36. [36]

    Guaranteed margins for lqg regulators,

    J. Doyle, “Guaranteed margins for lqg regulators,”IEEE Transactions on Automatic Control, vol. 23, no. 4, pp. 756–757, 1978

    1978

  37. [37]

    J. C. Butcher,Numerical methods for ordinary differential equations. John Wiley & Sons, 2016

    2016

  38. [38]

    An interior trust region approach for nonlin- ear minimization subject to bounds,

    T. F. Coleman and Y . Li, “An interior trust region approach for nonlin- ear minimization subject to bounds,”SIAM Journal on optimization, vol. 6, no. 2, pp. 418–445, 1996

    1996

  39. [39]

    B. D. Anderson and J. B. Moore,Optimal control: linear quadratic methods. Courier Corporation, 2007

    2007

  40. [40]

    Szeliski,Computer vision: algorithms and applications

    R. Szeliski,Computer vision: algorithms and applications. Springer Nature, 2022

    2022

  41. [41]

    Distance transformations in digital images,

    G. Borgefors, “Distance transformations in digital images,”Computer Vision, Graphics, and Image Processing, vol. 34, no. 3, pp. 344–371, 1986

    1986

  42. [42]

    Algorithms for hyper-parameter optimization,

    J. Bergstra, R. Bardenet, Y . Bengio, and B. Kégl, “Algorithms for hyper-parameter optimization,”Advances in neural information pro- cessing systems, vol. 24, 2011

    2011

  43. [43]

    Optimality of linear policies for distributionally robust linear quadratic gaussian regulator with stationary distributions,

    N. Lanzetti, A. Terpin, and F. Dörfler, “Optimality of linear policies for distributionally robust linear quadratic gaussian regulator with stationary distributions,”arXiv preprint arXiv:2410.22826, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    K. J. Åström,Introduction to stochastic control theory. Courier Corporation, 2012

    2012

  45. [45]

    Model predictive heuristic control: Applications to industrial processes,

    J. Richalet, A. Rault, J. Testud, and J. Papon, “Model predictive heuristic control: Applications to industrial processes,”Automatica, vol. 14, no. 5, pp. 413–428, 1978

    1978

  46. [46]

    Pinet: Optimizing hard-constrained neural networks with orthogonal projection layers,

    P. D. Grontas, A. Terpin, E. C. Balta, R. D’Andrea, and J. Lygeros, “Pinet: Optimizing hard-constrained neural networks with orthogonal projection layers,”arXiv preprint arXiv:2508.10480, 2026

    arXiv 2026

  47. [47]

    K. Zhou, J. C. Doyle, and K. Glover,Robust and Optimal Control. Prentice Hall, 1996

    1996

  48. [48]

    Variational analysis in the wasserstein space,

    N. Lanzetti, A. Terpin, and F. Dörfler, “Variational analysis in the wasserstein space,”arXiv preprint arXiv:2406.10676, 2024

    arXiv 2024