Pith. sign in

REVIEW 4 major objections 5 minor 26 references

Solving nonconvex Hamilton--Jacobi--Isaacs equations with PINN-based policy iteration

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read This paper proves that a mesh-free policy-iteration scheme built on physics-informed neural networks converges locally uniformly to the unique viscosity solution of nonconvex Hamilton–Jacobi–Isaacs equations, and bounds the neural…

desk verdict The method is sensible and the 2D/5D/10D results are decent, but the main convergence proof is missing the step that identifies the limit equation, Proposition 1 leans on an unproved gradient bound, and the experiments run outside Assumption 1; this deserves a serious referee but not acceptance as-is. read the letter →

arxiv 2507.15455 v2 pith:ALNMOW5O submitted 2025-07-21 math.NA cs.AIcs.NAmath.AP

classification math.NAcs.AIcs.NAmath.AP MSC 49L2565M9935D4091A23
keywords Hamilton–Jacobi–Isaacsequationspolicyiterationphysics-informedneuralnetworksviscositysolutionsnonconvexHamiltonianszero-sumstochasticdifferentialgamesuniformellipticityhigh-dimensionalPDEs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Policy iteration, a classical dynamic-programming scheme, is usually analyzed on grids; this paper brings it into the mesh-free PINN world for nonconvex Hamilton–Jacobi–Isaacs (HJI) equations. The authors show that when the game Lagrangian is uniformly strongly convex in one player's control and strongly concave in the other's, the value iterates converge locally uniformly to the unique bounded viscosity solution, with no convexity assumed on the Hamiltonian. For the practical neural-network implementation they prove $\sup_{t\in[0,T]}\|\tilde v_n(t,\cdot)-v(t,\cdot)\|_2 \le C(p_n+\rho^n)$, separating the network residual $p_n$ from an exponentially decaying iteration error $\rho^n$. If correct, this gives high-dimensional zero-sum differential games a solver with explicit $L^2$ error control and a built-in near-optimal feedback policy, and the 2D/5D/10D experiments support the scalability claim.

What carries the argument

The engine is the policy-iteration loop for the HJI equation. In the policy-evaluation step, the current feedback pair $(\alpha_n,\beta_n)$ is frozen and the linear parabolic PDE $\partial_t v_n + L(t,x,\nabla v_n)(\alpha_n,\beta_n) = -\tfrac{1}{2}\operatorname{Tr}(\sigma\sigma^\top D^2_{xx}v_n)$ is solved; the PINN represents $v_n$ with the hard-constraint ansatz $v_n = g + (T-t)N_n(t,x;\theta_n)$, so the terminal condition is enforced exactly and all derivatives come from automatic differentiation. In the policy-improvement step, the new controls are the pointwise minimax pair of the Lagrangian at $p=\nabla_x v_n$, whose existence, uniqueness, and global Lipschitz dependence on $p$ are the content of Lemma 1. That Lipschitz feedback selector, together with the Feynman–Kac representation and the parabolic $L^2$ energy estimate of Proposition 2, is what carries the equi-Lipschitz regularity and the contraction rate $\rho$ in Proposition 1 and Theorem 2.

What would settle it

To test the core convergence claim directly, take a one-dimensional zero-sum game with explicit solution—say $L(t,x,p)(a,b)=\frac{\mu_A}{2}a^2-\frac{\mu_B}{2}b^2+p(a+b)$, $\sigma=0.1$, terminal cost $g(x)=x^2$—and verify that idealized policy iteration (Algorithm 1) converges locally uniformly to the viscosity solution for positive $\mu_A,\mu_B$. Then shrink $\mu_A=\mu_B$ toward zero: Lemma 1's Lipschitz constant grows as $1/\mu$, so failure of convergence while Assumption 1 still holds would falsify Theorem 1, while continued convergence would show the strong convexity–concavity hypothesis is not necessary.

Watch

Extended reading notes

Core claim

The paper's central claim is that policy iteration can solve nonconvex HJI equations without grids and without convexity of the Hamiltonian, provided the dynamics satisfy Assumption 1. Each iteration solves a linear parabolic PDE for the current feedback pair—a policy-evaluation step that inherits Feynman–Kac representation and Schauder regularity—and then updates the controls by the pointwise saddle point of the Lagrangian at the current value gradient; Lemma 1 shows this minimax feedback selector is globally Lipschitz because of the strong convexity–concavity of the Lagrangian. Theorem 1 states the resulting value iterates converge locally uniformly to the unique bounded continuous viscosity solution, and Theorem 2 quantifies the error of the practical algorithm as $C(p_n+\rho^n)$, where $p_n$ is the $L^2$ norm of the PINN residual and $\rho<1$. The experiments in two, five, and ten dimensions are offered as evidence that the framework is accurate and scalable, with lower residuals and smoother value functions than direct PINN training.

Load-bearing premise

The whole argument rests on Assumption 1: the game's payoff must be uniformly strongly convex in the minimizing player's control and uniformly strongly concave in the maximizing player's control at every state, while the drift, cost, and diffusion remain bounded, Lipschitz, and uniformly elliptic—if any of that fails, the Lipschitz feedback selector and the equi-Lipschitz estimates that carry the proof are not available.

Editorial extensions

If this is right

  • Under Assumption 1, the practical PINN policy-iteration algorithm has total error at most $C(p_n+\rho^n)$, so a user can monitor the PDE residual $p_n$ and stop iterating once it plateaus.
  • Every iteration returns a near-optimal feedback policy by a pointwise minimax formula, so no separate control optimization is needed after the value network is trained.
  • The hard-constraint ansatz $v_n=g+(T-t)N_n$ removes the terminal loss term from the PINN objective, which the paper links to more stable training and smoother value functions.
  • The method matches a finite-difference reference in the 2D moving-obstacle game with relative $L^2$ errors around $10^{-3}$, and in the 5D and 10D publisher–subscriber games it outperforms direct PINN training in residual and relative $L^2$ error.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 5D and 10D benchmarks use linear controls with $\ell^1$ penalties and polynomial drift, which do not satisfy Assumption 1's strong convexity–concavity or global boundedness; if those results hold up, the convergence theory is sufficient but not necessary, and may be extendable to merely convex–concave Lagrangians with compact controls.
  • Appendix B shows the publisher–subscriber value function splits into a sum of two-dimensional problems when noise is isotropic; the reported 5D/10D accuracy may therefore chiefly measure the approximation of 2D subproblems, not genuine high-dimensional approximation.
  • The bound $C(p_n+\rho^n)$ suggests a natural stopping criterion—iterate until the residual plateaus—and also implies that the neural-network approximation error, not the policy iteration, is the ultimate bottleneck of the practical solver.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a mesh-free policy iteration method for nonconvex Hamilton-Jacobi-Isaacs (HJI) equations, in which each iteration solves a linear parabolic PDE with a fixed feedback policy via a PINN and then updates the feedback controls by pointwise minimax optimization using automatic differentiation. The main theoretical claims are Theorem 1 (local uniform convergence of the policy-evaluation value functions to the unique viscosity solution of the HJI equation), Proposition 1 (an exponential L2 convergence rate for the exact policy iteration), and Theorem 2 (a global error bound for the practical PINN implementation in terms of the PDE residual and the iteration error). Numerical experiments are reported for a 2D moving-obstacle path-planning game and for 5D and 10D publisher-subscriber games. The paper is clearly organized and the algorithmic description is concrete, but the central convergence arguments contain several load-bearing gaps, and the high-dimensional experiments are run outside the stated assumptions.

Significance. If the convergence theorems were established, the paper would make a useful contribution: a PINN-based policy iteration with a residual-controlled error bound is attractive for nonconvex HJI equations, and the hard-terminal-condition ansatz in (3.9) plus the pointwise minimax update via automatic differentiation are sensible algorithmic choices. The proof of Lemma 1 is sound under the strong convexity-concavity assumption, and the 2D experiment provides a quantitative finite-difference comparison. However, the significance is conditional, because Theorem 1, Proposition 1, and Theorem 2 all rely on regularity or stability steps that are not proven, and because the 5D/10D experiments violate Assumption 1. The paper should be revised so that the theoretical results are either completed or explicitly restricted to settings where the assumptions hold.

major comments (4)
  1. [§3.1, proof of Theorem 1] The final step of the proof is not justified. The argument establishes only that a subsequence of {v_n} converges locally uniformly by Arzela-Ascoli. However, the policy-evaluation PDE solved by v_n contains L(t,x,∇v_n)(α_n,β_n), where (α_n,β_n) is the feedback pair computed from ∇v_{n-1} in Algorithm 1, not from ∇v_n. By Lemma 1, |L(t,x,∇v_n)(π_n)-H(t,x,∇v_n)| is bounded by κ||f||∞|∇v_n-∇v_{n-1}|, and equi-Lipschitz convergence of v_n does not imply uniform convergence of ∇v_n. Therefore the cited stability property of viscosity solutions ([26]) cannot be applied as written; one would need to prove, for example, uniform convergence of ∇v_n or a viscosity notion of convergence of the Hamiltonians. Without this step, the limit v is not identified as the solution of the HJI equation (3.1).
  2. [§3.1, Proposition 1 (Eq. 3.7)] The exponential contraction F_n ≤ ρF_{n-1} depends entirely on the asserted gradient bound ||∇xδ_n(t,·)||_2 ≤ C̃√T (F_n + F_{n-1}), which is attributed to [6] without a statement or proof. The cited reference is a standard PDE textbook and does not contain this specific estimate. Moreover, the bound as written has F_n on both sides, so the displayed derivation only gives (1-C̃√T)F_n ≤ C̃√T F_{n-1}; a valid proof must establish the estimate with a constant that does not presuppose the desired contraction. Since Proposition 1 is also used in Theorem 2, this gap is load-bearing for both the exponential rate and the global error bound.
  3. [Appendix A, Proposition 2] The parabolic L2 estimate in Proposition 2 is applied to viscosity solutions, but the energy inequality d/dt||v||_2^2 + λ||∇v||_2^2 ≤ ... is derived by formal integration by parts and requires v(t,·) ∈ H^1 (and enough regularity to justify the manipulation). No regularization or approximation argument is given to bridge the gap from viscosity solutions to this energy class. In addition, when σσ^T depends on x, the integration by parts produces extra terms involving ∇·(σσ^T) that are not accounted for in the stated inequality. Theorem 2 uses Proposition 2 to control both A_n and B_n, so the global error estimate (3.10) is not rigorously established.
  4. [§4.3, publisher-subscriber experiments vs. Assumption 1] The 5D and 10D publisher-subscriber problems do not satisfy Assumption 1. The control and disturbance enter the drift linearly as Bu and Cd, so the Lagrangian L(t,x,p)(u,d) is not strongly convex-concave in (u,d); moreover, the drift contains ψ(x) with cubic terms, so f is neither globally bounded nor globally Lipschitz on R^N. Consequently Theorems 1 and 2 and Proposition 1 do not apply to the experiments in Section 4.3. The paper does not flag this mismatch, and the claim that the iterative framework yields provable L2 bounds in these high-dimensional nonconvex settings is therefore not supported.
minor comments (5)
  1. [§3.1, proof of Theorem 1] There is a typo 'v′ns' in the equi-Lipschitz estimate paragraph; it should read 'v_n's'.
  2. [§3.1, proof of Theorem 1] The stability property of viscosity solutions is cited to [26], an actor-critic flow paper; a standard reference such as [5] (Crandall-Ishii-Lions) would be more appropriate and would make the intended argument easier to verify.
  3. [§3.2, Theorem 2 statement] The theorem statement says 'where (α_n,β_n) is the exact feedback pair obtained from v_{n-1}', but Algorithm 2 computes policies from the approximate value function ṽ_{n-1}. This notational inconsistency should be clarified, since the proof of the B_n term depends on comparing exact and approximate policies.
  4. [Appendix B] The value-function decomposition proof assumes 'sufficient regularity for the chain rule' and block-diagonal or isotropic σ, but these assumptions are not stated in the main text and are not satisfied by the anisotropic experiments in Section 4.3. The global claim that the decomposition holds should be qualified accordingly.
  5. [Table 4] The 'Direct PINN epoch' entry for the publisher-subscriber setting is listed as a bare number 2,500,000 without indicating whether it is the total number of epochs or a product; this is confusing and should be written explicitly as E × M.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence analysis is self-contained and does not reduce to fitted inputs or self-citations.

full rationale

The load-bearing results are derived from the stated assumptions rather than from the paper's own conclusions. Theorem 1 starts from the policy-evaluation PDEs and attempts a viscosity limit using standard compactness and stability arguments; although the final limit passage is not fully justified (equi-Lipschitz convergence does not identify the limiting Hamiltonian), that is a mathematical gap, not a circular definition. Proposition 1 and Theorem 2 reduce the error to quantities such as the PDE residual p_n and the policy-iteration contraction rho^n, with Proposition 2 proved in Appendix A; these are stability estimates, not fitted predictions disguised as new results. The only self-citation is the background mention of [18] (Lee and Kim, co-authored by Y. Kim) in the introduction, which is not used in any proof and therefore not load-bearing. Reference [9] is an external prior-work citation for the iterative scheme and Theorem 1 is framed explicitly as a specialized analogue, not as an imported uniqueness theorem. No parameter is fit to a target value function and then reported as a prediction. The paper's own limitation statement about non-degenerate diffusion is acknowledged and does not affect the circularity assessment.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central claim relies on Assumption 1, standard viscosity solution and Feynman-Kac theory, and two unproved analytic inputs (Proposition 2 and a gradient bound cited to [6] but not stated). No data-fitting parameters or new entities are introduced.

assumptions (6)
  • domain assumption Assumption 1: bounded Lipschitz coefficients f,c,sigma; g in C^{2+beta}(R^d) intersect L^2; sigma sigma^T >= lambda I; Lagrangian strongly convex-concave in controls
    Invoked in Section 3.1 and used in Lemma 1 and Theorems 1-2 and Proposition 1.
  • standard math Viscosity solution theory for second-order PDEs (Crandall-Ishii-Lions)
    Used to define the HJI solution and the policy evaluation PDE solutions, and for uniqueness and stability.
  • standard math Feynman-Kac representation of linear parabolic PDEs
    Used in Theorem 1's proof to derive uniform boundedness of v_n.
  • standard math Parabolic L2 energy estimates (Proposition 2)
    Stated without proof in Appendix A; used in Proposition 1 and Theorem 2 for residual-to-error control.
  • domain assumption Gradient bound for linear parabolic equations with zero initial data (cited to Evans [6])
    Invoked in Proposition 1's proof to obtain ||grad delta_n||_2 <= C_tilde sqrt(T) (F_n + F_{n-1}); the precise statement is absent.
  • standard math Stability of viscosity solutions under locally uniform convergence
    Cited to [26] in Theorem 1's proof to pass to the limit; the convergence of the discrete Hamiltonians is not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Solving nonconvex Hamilton--Jacobi--Isaacs equations with PINN-based policy iteration." pith.science (2026). https://pith.science/paper/ALNMOW5O

@misc{pith2026250715455,
  author       = {Pith},
  title        = {Pith review of: Solving nonconvex Hamilton--Jacobi--Isaacs equations with PINN-based policy iteration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ALNMOW5O}},
  note         = {Machine review of arXiv:2507.15455}
}
abstract

We propose a mesh-free policy iteration framework that combines classical dynamic programming with physics-informed neural networks (PINNs) to solve high-dimensional, nonconvex Hamilton--Jacobi--Isaacs (HJI) equations arising in stochastic differential games and robust control. The method alternates between solving linear second-order PDEs under fixed feedback policies and updating the controls via pointwise minimax optimization using automatic differentiation. Under standard Lipschitz and uniform ellipticity assumptions, we prove that the value function iterates converge locally uniformly to the unique viscosity solution of the HJI equation. The analysis establishes equi-Lipschitz regularity of the iterates, enabling provable stability and convergence without requiring convexity of the Hamiltonian. Numerical experiments demonstrate the accuracy and scalability of the method. In a two-dimensional stochastic path-planning game with a moving obstacle, our method matches finite-difference benchmarks with relative $L^2$-errors below %10^{-2}%. In five- and ten-dimensional publisher-subscriber differential games with anisotropic noise, the proposed approach consistently outperforms direct PINN solvers, yielding smoother value functions and lower residuals. Our results suggest that integrating PINNs with policy iteration is a practical and theoretically grounded method for solving high-dimensional, nonconvex HJI equations, with potential applications in robotics, finance, and multi-agent reinforcement learning.

Figures

Figures reproduced from arXiv: 2507.15455 by the authors.

Figure 1
Figure 1. Comparison of the policy-iterative PINN solution and the reference FDM solution for the two [PITH_FULL_IMAGE:figures/full_fig_p013_1.png] view at source ↗
Figure 2
Figure 2. Time evolution of optimal trajectories derived from the policy-iterative PINN solution. Robots [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Comparison of policy-iterative and direct PINN approaches on the 2D isotropic benchmark prob [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of policy-iterative and direct PINN methods on the five-dimensional anisotropic [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: Comparison of policy-iterative and direct PINN methods on the ten-dimensional anisotropic prob [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 23 canonical work pages

  1. [26]

    Zhou and J

    M. Zhou and J. Lu. Solving time-continuous stochastic optimal control problems: Algorithm design and convergence analysis of actor-critic flow. arXiv preprint arXiv:2402.17208 , 2024. A Parabolic L2 estimate Proposition 2 (Parabolic L2–estimate). Let d ∈ N and T > 0. Suppose Assumption 1 holds. Let b ∈ L∞((0, T) × Rd), P ∈ L2((0, T); L2(Rd)) and Q ∈ L2(Rd...

  2. [6]

    L. C. Evans. Partial Differential Equations , volume 19 of Grad. Stud. Math. Amer. Math. Soc., 2022

  3. [1]

    Bansal and C

    S. Bansal and C. J. Tomlin. Deepreach: A deep learning approach to high-dimensional reachability. In Proc. IEEE Int. Conf. Robot. Autom. , pages 1817–1824. IEEE, 2021

  4. [2]

    Barles and P

    G. Barles and P. E. Souganidis. Convergence of approximation schemes for fully nonlinear second order equations. Asymptot. Anal. , 4(3):271–283, 1991

  5. [3]

    T. J. Barth and J. A. Sethian. Numerical schemes for the hamilton–jacobi and level set equations on triangulated domains. J. Comput. Phys. , 145(1):1–40, 1998. 18

  6. [4]

    Cecil, J

    T. Cecil, J. Qian, and Stanley Osher. Numerical methods for high dimensional Hamilton–Jacobi equa- tions using radial basis functions. J. Comput. Phys. , 196(1):327–347, 2004

  7. [5]

    M. G. Crandall, H. Ishii, and P.L. Lions. User’s Guide to viscosity solutions of second order partial differential equations. Bull. Amer. Math. Soc. , 27(1):1–67, 1992

  8. [7]

    Ferretti and O

    R. Ferretti and O. Junge. An adaptive multilevel radial basis function scheme for the hjb equation. In Proceedings of the SSSC Workshop (SpecSem 2016) , 2016

Show all 26 references
  1. [8]

    W. H. Fleming and H. M. Soner. Controlled Markov processes and viscosity solutions , volume 25. Springer Science & Business Media, 2006

  2. [9]

    X. Guo, H. V. Tran, and Y. P. Zhang. Policy iteration for nonconvex viscous Hamilton–Jacobi equations. arXiv preprint arXiv:2503.02159 , 2025

  3. [10]

    J. Han, A. Jentzen, and W. E. Solving high-dimensional partial differential equations using deep learning. Proc. Natl. Acad. Sci. , 115(34):8505–8510, 2018

  4. [11]

    D. J. Higham. An algorithmic introduction to numerical simulation of stochastic differential equations. SIAM Review , 43(3):525–546, 2001

  5. [12]

    E. L. Kawecki and T. Sprekeler. Discontinuous galerkin and c0-ip finite element approximation of peri- odic hamilton–jacobi–bellman–isaacs problems with application to numerical homogenization. ESAIM: Math. Model. Numer. Anal. , 56(2):679–704, 2022

  6. [13]

    Kerimkulov, D

    B. Kerimkulov, D. Siska, and L. Szpruch. Exponential convergence and stability of howard’s policy improvement algorithm for controlled diffusions. SIAM J. Control Optim. , 58(3):1314–1340, 2020

  7. [14]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

  8. [15]

    M. V. Klibanov, L. H. Nguyen, and H. V. Tran. Numerical viscosity solutions to hamilton–jacobi equations via a carleman estimate and the convexification method. J. Comput. Phys. , 451:110828, 2022

  9. [16]

    H. J. Kushner. Numerical methods for stochastic control problems in continuous time. SIAM J. Control Optim., 28(5):999–1048, 1990

  10. [17]

    I. E. Lagaris, A. Likas, and Dimitrios I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Trans. Neural Netw. Learn. Syst. , 9(5):987–1000, 1998

  11. [18]

    J. Y. Lee and Y. Kim. Hamilton–jacobi based policy-iteration via deep operator learning. Neurocom- puting, page 130515, 2025

  12. [19]

    Y. Liu, L. Cai, Y. Chen, and B. Wang. Physics-informed neural networks based on adaptive weighted loss functions for hamilton-jacobi equations. Math. Biosci. Eng. , 19(12):12866–12896, 2022

  13. [20]

    Mitake and H.V

    H. Mitake and H.V. Tran. Dynamical properties of Hamilton–Jacobi equations via the nonlinear ad- joint method: large time behavior and discounted approximation. Dynamical and geometric aspects of Hamilton–Jacobi and linearized Monge–Ampere equations—VIASM , 2183:125–228, 2016

  14. [21]

    Raissi, P

    M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. , 378:686–707, 2019

  15. [22]

    Implicit neural representations with periodic activation functions

    Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Adv. Neural. Inf. Process. Syst., 33:7462–7473, 2020. 19

  16. [23]

    P. E. Souganidis. Approximation schemes for viscosity solutions of hamilton-jacobi equations. . Differ- ential Equations , 59(1):1–43, 1985

  17. [24]

    W. Tang, H. V. Tran, and Y. P. Zhang. Policy iteration for the deterministic control problems—a viscosity approach. SIAM J. Control Optim. , 63(1):375–401, 2025

  18. [25]

    H. V. Tran, Z. Wang, and Y. P. Zhang. Policy iteration for exploratory hamilton–jacobi–bellman equations. Appl. Math. Optim. , 91(2):50, 2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.