Pith. sign in

REVIEW 2 major objections 6 minor 30 references

Reachability-Preserving Bellman Operator for the Discounted Reach-Cost Value Function: Uniting Hamilton-Jacobi Reachability and Reinforcement Learning

T0 review · 2 major / 6 minor · reviewed 2026-07-10 · grok-4.5

Pith's one-line read A non-additive Bellman operator for discounted reach-cost has a unique fixed point that is exactly the Hamilton-Jacobi reachability value function.

desk verdict Clean, usable completion of the HJ–RL bridge: a contractive non-additive Bellman operator whose fixed point exactly matches Choi’s semantics-preserving discounted reach value. read the letter →

arxiv 2607.07893 v1 pith:HPKZLDQI submitted 2026-07-08 eess.SY cs.SY

classification eess.SYcs.SY
keywords Hamilton-Jacobireachabilitynon-additiveBellmanoperatorreinforcementlearningreachcostsafety-criticalcontrolviscositysolutiondiscountedvaluefunctionfixed-pointtheorem
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Hamilton-Jacobi reachability gives formal safety and reachability certificates for continuous-time systems, but grid-based solvers hit the curse of dimensionality. Reinforcement learning scales better, yet standard RL is built for additive cumulative rewards, while reachability is a non-additive stopping problem. Earlier discounted formulations either changed the meaning of the reachable set or never produced a true Bellman fixed-point equation. This paper starts from a discounted reach-cost value function that keeps the original sign-based semantics and constructs a matching non-additive Bellman operator. Discounting makes the operator a contraction, so Banach's theorem yields a unique fixed point that coincides with the Hamilton-Jacobi viscosity solution. Reinforcement learning is then simply a sample-based method for solving that same fixed-point equation, which means learned value functions can still be read as rigorous reachable sets.

What carries the argument

The reachability-preserving Bellman operator T_{σ,λ} (and its infinite-horizon counterpart). It replaces the usual sum of rewards by a min of a stopping branch and a discounted continuation branch; the discount factor e^{-λσ} makes the whole map a contraction whose unique fixed point is the true reachability value function.

What would settle it

On a low-dimensional system whose true Hamilton-Jacobi value function can be computed accurately (for example the double-integrator or Dubins-car examples already used in the paper), run fitted value iteration with the proposed operator and check whether the learned zero level set fails to align with the reference zero level set, or whether the pointwise residual fails to go to zero as the discretization parameters vanish.

Watch

Extended reading notes

Core claim

The discounted reach-cost value function that already appears in the Hamilton-Jacobi literature is exactly the unique fixed point of a non-additive Bellman operator. Discounting alone is enough to make that operator contractive on a complete space of bounded functions, so existence, uniqueness and convergence of value iteration follow, and the Hamilton-Jacobi and Bellman characterizations are identical.

Load-bearing premise

The system dynamics must be globally Lipschitz continuous in the state, uniformly in the control, and the reach-cost function must be bounded; without those two conditions the contraction proof and the uniqueness of the fixed point no longer hold on the whole space.

Editorial extensions

If this is right

  • Learned reachability value functions can be interpreted as rigorous safety certificates rather than heuristic scores.
  • Value iteration and fitted-value methods become legitimate numerical solvers for continuous-time reachable sets once the non-additive operator is used.
  • The same contraction argument applies to both finite-horizon and infinite-horizon stationary problems, giving a uniform theory.
  • High-dimensional systems that are intractable for grid-based Hamilton-Jacobi solvers become candidates for data-driven reachable-set computation while preserving semantics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same construction should extend almost immediately to reach-avoid problems by swapping the min for a suitable max-min structure, yielding a single operator that encodes both safety and liveness.
  • Once the contraction property is established, residual-based post-hoc certification methods already developed for additive Bellman operators become available for these non-additive reachability operators.
  • The explicit separation of Bellman step σ from integration step Δt suggests a practical schedule that trades contraction speed against numerical consistency, which could be optimized automatically during learning.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper derives a non-additive Bellman operator for the semantics-preserving discounted reach-cost value function of Choi et al., proves that discounting renders the operator a contraction on a complete space of bounded functions, and shows that its unique fixed point coincides with the unique viscosity solution of the associated Hamilton–Jacobi variational inequality (finite- and infinite-horizon). Discrete-time consistent approximations and a fitted-value-iteration scheme are developed, and RL is interpreted as sample-based approximation of the same fixed-point equation. Numerical experiments on a double-integrator reach problem and a Dubins-car avoid problem show close agreement with HJ reference solutions and alignment of zero level sets.

Significance. If the results hold, the paper closes a genuine gap between HJ reachability and RL: prior discounted formulations either sacrificed exact reachability semantics for contraction (e.g., MDR) or preserved semantics on the HJ side without a matching Bellman fixed-point characterization. The operator-theoretic bridge (Definitions 5 and 7; Theorems 3–8) is clean, uses standard Banach and viscosity tools under classical Lipschitz/boundedness assumptions, and is complementary to concurrent travel-cost formulations. The explicit flagging of the open FVI convergence question (Remark 8) is a strength. The contribution is primarily theoretical; empirical support is limited to low-dimensional systems but is consistent with the claims that are actually proven.

major comments (2)
  1. Lemma 9 (and the parallel Lemma 12): the written contraction proof treats the rollout horizon as σ throughout and pulls out the factor e^{-λσ}, but Definition 5 uses h(τ)=min{τ,σ}. When τ<σ the continuation term lands on the fixed boundary Ψ(0,·)=g(·), so the difference is actually zero; Remark 4 correctly notes that the boundary condition is essential for uniform contraction near τ=0. The claim is true, but the proof as written does not case-split on h(τ). Please rewrite Lemma 9 (and the discrete analogue Lemma 14) with an explicit case analysis so that the argument is self-contained and matches Definition 5.
  2. Abstract, Introduction, and Section VIII: the paper repeatedly frames the contribution as uniting HJ reachability with reinforcement learning and as enabling “scalable, data-driven computation … in high-dimensional systems.” What is actually constructed is fitted value iteration with exact rollouts of a known dynamics model (explicitly identified as ADP in Section VIII), and Remark 8 correctly leaves open whether the composite “regress-then-apply-operator” map converges to the discrete fixed point. The operator-theoretic core (Theorems 3–8) does not depend on this claim, but the abstract and title overstate what is proven. Soften the RL/scalability language to match the proven fixed-point equivalence and the acknowledged open FVI question, or supply partial error bounds / high-dimensional evidence.
minor comments (6)
  1. Table I is a useful running reference; consider adding a one-line pointer to it at the first appearance of W_Bell / ĉW / Ψ_θ so readers do not miss the tier structure.
  2. Section IX-B: for the double integrator the infinite-horizon reachable set fills the domain, so zero-level-set comparison is impossible; this is explained, but a short finite-horizon double-integrator panel (or a reach-avoid variant) would make the zero-level-set claim more uniform across both examples.
  3. Eq. (42) / Definition 5: the notation U[0,h(τ)] vs U[0,σ] is slightly inconsistent with the later discrete operators; unify the control-interval notation.
  4. Proposition 3 writes V(x) rather than V(t,x); fix the argument list for consistency with Definition 1.
  5. Algorithms 1–2 are clear; a brief note on how U_d is chosen (grid density, effect on the min) would help reproducibility of the numerical scheme in Section VII.
  6. Related-work discussion of Hsu et al. and the concurrent travel-cost paper [27] is fair; a single sentence clarifying that the present operator is non-additive min-structured (not a running-cost integral) would further reduce possible confusion with additive discounted formulations.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the Bellman fixed-point and HJ equivalence are derived from the DPP plus Banach, not forced by definition or self-citation.

full rationale

The derivation chain is self-contained. The discounted reach-cost value functions W and W^∞ are defined directly from the dynamics and g (Defs. 1–4). The DPP (Props. 1–2) is obtained by splitting the infimum over time and using the semigroup property of the flow; the non-additive Bellman operators T_{σ,λ} and T^∞_{σ,λ} are then written down from those DPPs (Defs. 5, 7). Contraction follows from the elementary 1-Lipschitz property of min together with the factor e^{-λσ}<1 (Lems. 9, 12); existence/uniqueness of the fixed points is Banach’s theorem on the complete spaces B and B^∞ (Thms. 3, 5). That the true value functions are fixed points is immediate from the DPP, so uniqueness yields W = W_Bell and W^∞ = W^∞_Bell (Thms. 4, 6). The HJ characterizations are classical viscosity results (Thms. 1–2), with the avoid-case version already in Choi et al.; equivalence of the two characterizations is therefore just the observation that both objects equal the same W (Thms. 7–8). Self-citations to the authors’ concurrent travel-cost paper and residual-certification preprint appear only as complementary remarks and are not used in any proof. Experiments are pure numerical validation against an independent semi-Lagrangian HJ solver; no parameters are fitted and then re-predicted. The open FVI convergence question (Rem. 8) is correctly left open and does not affect the operator-theoretic claims. Hence the central results do not reduce to their inputs by construction.

Assumptions & free parameters 3 free parameters · 6 assumptions · 1 invented entities

The central theorems rest on standard viscosity-solution and fixed-point machinery plus eight domain assumptions on the dynamics and cost. No free parameters are fitted to obtain the theoretical claims; experimental hyperparameters are chosen by hand but do not enter the proofs. The only invented object is the operator itself, which is derived rather than postulated.

free parameters (3)
  • discount rate λ
    Chosen by the user; must be positive and, for infinite-horizon Lipschitz bounds, larger than the Lipschitz constant of f. Not fitted to data for the theoretical claims.
  • Bellman step σ
    Free discretization parameter that trades contraction speed against numerical consistency (Remark 5). Hand-chosen in experiments (σ=0.1).
  • SIREN frequencies ω0 and network widths
    Architecture hyperparameters tuned for the numerical examples; do not affect the theoretical fixed-point results.
assumptions (6)
  • standard math Banach fixed-point theorem on complete metric spaces
    Invoked in Theorems 3 and 5 to obtain uniqueness and geometric convergence of value iteration.
  • standard math Comparison principle / uniqueness of bounded continuous viscosity solutions for HJ variational inequalities
    Used in Theorems 1–2 and 7–8; classical results from Bardi & Capuzzo-Dolcetta.
  • domain assumption Global Lipschitz continuity of f in x, uniformly in u (Assumption 2)
    Required for trajectory estimates (Grönwall) that feed every Lipschitz and contraction proof.
  • domain assumption Boundedness of the reach cost g (Assumption 8)
    Guarantees the value function stays in the Banach space of bounded functions on which the operator is a contraction.
  • domain assumption Sign calibration of g: g<0 inside target, g≥0 outside (Assumption 5)
    Ensures the zero sublevel set of the value function coincides with the backward reachable tube (Proposition 3).
  • domain assumption Time-invariance of f and g (Assumptions 1, 6)
    Allows reduction to a time-to-go formulation and stationarity of the infinite-horizon value function.
invented entities (1)
  • Reachability-preserving Bellman operator T_{σ,λ} / T^∞_{σ,λ} independent evidence
    purpose: Provides the missing fixed-point characterization of the discounted reach-cost value function so that RL can be applied without altering semantics.
    Derived directly from the dynamic-programming principle; not an extra physical or mathematical postulate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reachability-Preserving Bellman Operator for the Discounted Reach-Cost Value Function: Uniting Hamilton-Jacobi Reachability and Reinforcement Learning." pith.science (2026). https://pith.science/paper/HPKZLDQI

@misc{pith2026260707893,
  author       = {Pith},
  title        = {Pith review of: Reachability-Preserving Bellman Operator for the Discounted Reach-Cost Value Function: Uniting Hamilton-Jacobi Reachability and Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HPKZLDQI}},
  note         = {Machine review of arXiv:2607.07893}
}
read the original abstract

Hamilton-Jacobi (HJ) reachability provides rigorous safety and reachability guarantees for continuous-time dynamical systems, but its numerical solution suffers from the curse of dimensionality. Deep reinforcement learning (DRL), by contrast, offers scalable sample-based methods. However, RL is typically built around additive cumulative rewards; whereas, reachability objectives are inherently non-additive. This mismatch makes a direct bridge between HJ reachability and RL nontrivial. Recent discounted formulations have either introduced contraction by altering the original reachability semantics, or preserved exact semantics on the HJ side without a corresponding Bellman fixed-point characterization. In this paper, we close this gap by building on a semantics-preserving discounted reach-based value function and deriving a non-additive Bellman operator whose unique fixed point exactly matches the value function in the HJ formulation. We prove that discounting makes this operator contractive, yielding existence, uniqueness, and convergence of value iteration. Furthermore, we establish the equivalence between the HJ and Bellman characterizations, and show that RL can be interpreted as a sample-based approximation scheme for the same fixed-point equation. This yields a principled and semantically exact connection between HJ reachability and RL, enabling learning-based methods to approximate reachability value functions while preserving their safety-critical meaning. As a result, the proposed framework opens the door to scalable, data-driven computation of reachable sets and safety certificates in high-dimensional systems. Numerical experiments demonstrate close agreement with HJ solutions, confirm preservation of reachability semantics via alignment of zero level sets, and support the interpretation of reinforcement learning as a sample-based solver of the proposed Bellman operator.

Figures

Figures reproduced from arXiv: 2607.07893 by the authors.

Figure 1
Figure 1. Comparison between the stationary solution of the HJ [PITH_FULL_IMAGE:figures/full_fig_p017_1.png] view at source ↗
Figure 2
Figure 2. Dubins car avoid problem. Comparison between the [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 30 canonical work pages

  1. [1]

    Akametalu, Shromona Ghosh, Jaime F

    Anayo K. Akametalu, Shromona Ghosh, Jaime F. Fisac, Vicenc Rubies- Royo, and Claire J. Tomlin. A minimum discounted reward hamil- ton–jacobi formulation for computing reachable sets.IEEE Transactions on Automatic Control, 69(2):1097–1103, 2024

  2. [2]

    Learning dexterous in-hand manipulation.The International Journal of Robotics Research, 39(1):3– 20, 2020

    OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation.The International Journal of Robotics Research, 39(1):3– 20, 2020

  3. [3]

    Hamilton- jacobi reachability: A brief overview and recent advances

    Somil Bansal, Mo Chen, Sylvia Herbert, and Claire J Tomlin. Hamilton- jacobi reachability: A brief overview and recent advances. In2017 IEEE 56th annual conference on decision and control (CDC), pages 2242–

  4. [4]

    Deepreach: A deep learning approach to high-dimensional reachability

    Somil Bansal and Claire J Tomlin. Deepreach: A deep learning approach to high-dimensional reachability. In2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1817–1824. IEEE, 2021

  5. [5]

    Springer, 1997

    Martino Bardi, Italo Capuzzo Dolcetta, et al.Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equations, volume 12. Springer, 1997

  6. [6]

    Bertsekas and John N

    Dimitri P. Bertsekas and John N. Tsitsiklis.Neuro-Dynamic Program- ming. Athena Scientific, Belmont, MA, 1996

  7. [7]

    Robust control barrier–value functions for safety- critical control

    Jason J Choi, Donggun Lee, Koushil Sreenath, Claire J Tomlin, and Sylvia L Herbert. Robust control barrier–value functions for safety- critical control. In2021 60th IEEE Conference on Decision and Control (CDC), pages 6814–6821. IEEE, 2021

  8. [8]

    Jérôme Darbon and Stanley Osher. Algorithms for overcoming the curse of dimensionality for certain hamilton–jacobi equations arising in control theory and elsewhere.Research in the Mathematical Sciences, 3(1):19, 2016

Show all 30 references
  1. [9]

    Magnetic control of tokamak plasmas through deep reinforcement learning.Nature, 602(7897):414– 419, 2022

    Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning.Nature, 602(7897):414– 419, 2022

  2. [10]

    Bridging hamilton-jacobi safety analysis and reinforcement learning

    Jaime F Fisac, Neil F Lugovoy, Vicenç Rubies-Royo, Shromona Ghosh, and Claire J Tomlin. Bridging hamilton-jacobi safety analysis and reinforcement learning. In2019 International Conference on Robotics and Automation (ICRA), pages 8550–8556. IEEE, 2019

  3. [11]

    Hamilton-jacobi reachability estimation in reinforcement learning

    Milan Ganai. Hamilton-jacobi reachability estimation in reinforcement learning. Master’s thesis, University of California, San Diego, 2024

  4. [12]

    Hamilton-jacobi reachability in reinforcement learning: A survey.IEEE Open Journal of Control Systems, 3:310–324, 2024

    Milan Ganai, Sicun Gao, and Sylvia L Herbert. Hamilton-jacobi reachability in reinforcement learning: A survey.IEEE Open Journal of Control Systems, 3:310–324, 2024

  5. [13]

    Springer, 2003

    Andrzej Granas, James Dugundji, et al.Fixed point theory, volume 14. Springer, 2003

  6. [14]

    Scalable learning of safety guarantees for autonomous systems using hamilton-jacobi reachability

    Sylvia Herbert, Jason J Choi, Suvansh Sanjeev, Marsalis Gibson, Koushil Sreenath, and Claire J Tomlin. Scalable learning of safety guarantees for autonomous systems using hamilton-jacobi reachability. In2021 IEEE International Conference on Robotics and Automation (ICRA), page...

  7. [15]

    Safety and liveness guarantees through reach-avoid reinforcement learning.arXiv preprint arXiv:2112.12288, 2021

    Kai-Chieh Hsu, Vicenç Rubies-Royo, Claire J Tomlin, and Jaime F Fisac. Safety and liveness guarantees through reach-avoid reinforcement learning.arXiv preprint arXiv:2112.12288, 2021

  8. [16]

    A hopf-lax formula in hamilton– jacobi analysis of reach-avoid problems.IEEE Control Systems Letters, 5(3):1055–1060, 2020

    Donggun Lee and Claire J Tomlin. A hopf-lax formula in hamilton– jacobi analysis of reach-avoid problems.IEEE Control Systems Letters, 5(3):1055–1060, 2020

  9. [17]

    Con- verse barrier certificates for finite-time safety verification of continuous- time perturbed deterministic systems.Systems & Control Letters, 209:106357, 2026

    Yonghan Li, Chenyu Wu, Taoran Wu, Shijie Wang, and Bai Xue. Con- verse barrier certificates for finite-time safety verification of continuous- time perturbed deterministic systems.Systems & Control Letters, 209:106357, 2026

  10. [18]

    On reachability and minimum cost optimal control

    John Lygeros. On reachability and minimum cost optimal control. Automatica, 40(6):917–927, 2004

  11. [19]

    Kavukcuoglu, David Silver, Andrei A

    V olodymyr Mnih, K. Kavukcuoglu, David Silver, Andrei A. Rusu, J. Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, A. Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, D. Kumaran, Daan Wierstra, S. Legg, and D....

  12. [20]

    Error bounds for approximate value iteration.Proceedings of the National Conference on Artificial Intelligence (AAAI), 2005

    Rémi Munos. Error bounds for approximate value iteration.Proceedings of the National Conference on Artificial Intelligence (AAAI), 2005

  13. [21]

    Finite-time bounds for fitted value iteration.Journal of Machine Learning Research, 9:815–857, 2008

    Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration.Journal of Machine Learning Research, 9:815–857, 2008

  14. [22]

    Dual-objective reinforcement learning with novel hamilton-jacobi-bellman formulations.arXiv preprint arXiv:2506.16016, 2025

    William Sharpless, Dylan Hirsch, Sander Tonkens, Nikhil Shinde, and Sylvia Herbert. Dual-objective reinforcement learning with novel hamilton-jacobi-bellman formulations.arXiv preprint arXiv:2506.16016, 2025

  15. [23]

    Bellman value decomposition for task logic in safe optimal control

    William Sharpless, Oswin So, Dylan Hirsch, Sylvia Herbert, and Chuchu Fan. Bellman value decomposition for task logic in safe optimal control. arXiv preprint arXiv:2602.19532, 2026

  16. [24]

    Mastering the game of go with deep neural networks and tree search.nature, 529(7587):484–489, 2016

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search.nature, 529(7587):484–489, 2016

  17. [25]

    A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.Science, 362(6419):1140–1144, 2018

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.Science, 362(641...

  18. [26]

    Mastering the game of go without human knowledge.nature, 550(7676):354–359, 2017

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge.nature, 550(7676):354–359, 2017

  19. [27]

    Formalizing the relationship between hamilton-jacobi reachability and reinforcement learning.arXiv preprint arXiv:2601.08050, 2026

    Prashant Solanki, Isabelle El-Hajj, Jasper van Beers, Erik-Jan van Kampen, and Coen de Visser. Formalizing the relationship between hamilton-jacobi reachability and reinforcement learning.arXiv preprint arXiv:2601.08050, 2026

  20. [28]

    van Beers, Erik-Jan van Kampen, and Coen C

    Prashant Solanki, Isabelle El-Hajj, Jasper J. van Beers, Erik-Jan van Kampen, and Coen C. de Visser. Certifying hamilton-jacobi reachability learned via reinforcement learning.arXiv preprint arXiv:2602.16475, 2026

  21. [29]

    Grandmaster level in starcraft ii using multi-agent reinforcement learning.nature, 575(7782):350–354, 2019

    Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Math- ieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning.nature, 575(7782):350–354, 2019

  22. [30]

    Inner-approximating reachable sets for polynomial systems with time-varying uncertainties

    Bai Xue, Martin Fränzle, and Naijun Zhan. Inner-approximating reachable sets for polynomial systems with time-varying uncertainties. IEEE Transactions on Automatic Control, 65(4):1468–1483, 2019

Pith tools

Reviewed July 10, 2026 · model on record in the stance chip above.