{"id":"72339cce-5eea-452f-bdf6-3ec77c7ed97d","arxiv_id":"2412.01591","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A kernel regression of the controlled diffusion generator is combined with a convex HJB recursion to produce data-driven, approximately globally optimal feedback policies.","lead":"This paper learns the infinitesimal generator of a noisy control system from samples of its dynamics and cost, then solves a data-driven Hamilton-Jacobi-Bellman equation to get a feedback policy. The resulting kernel-based method produces global approximations of optimal controls and beats a classical NMPC solver on simulated pendulum and cartpole swing-up tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The finite-horizon truncation is not justified for the infinite-horizon problem: the constant function is not in the RKHS, so the policy from Algorithm 1 may retain a spurious T-proportional term that is never analyzed.","rationale":"The reader's weakest assumption (Assumption 2, derivative labels) concerns the practical data requirements; it is a real limitation but is explicitly stated and acknowledged by the authors. The reader's rationale also mentions the lack of convergence or error analysis for the finite-rank HJB approximation, which is where my concern lies. My focus is more specific: even with perfect derivative data, the transition from a finite-horizon HJB to a stationary policy for the infinite-horizon average-cost problem has no theoretical support. The exact HJB cancellation of the λT term relies on B^*1=0, which the finite-dimensional B does not inherit exactly because the constant function is not representable in the RKHS. This is not a manufactured or adversarial concern: it is a concrete mechanism by which the algorithm's output could depend on the arbitrarily chosen T. The proposed T-sweep test is cheap and would directly reveal whether the benchmarks are in the stationary regime. I keep the verdict UNCHANGED (CONDITIONAL) because the paper's experiments may well satisfy the test; the concern is that the claim is not yet justified, not that it is demonstrably false. The paper does have independent support: Lemma 1 is a correct derivation of the generator action on kernel sections, and Proposition 2 is an exact algebraic reduction of the approximated HJB, so the core construction is sound. The missing piece is the asymptotic link to the infinite-horizon problem, which is precisely the load-bearing assertion of an approximately globally optimal stationary policy.","tokens_in":13564,"tokens_out":23706,"duration_ms":209388,"concrete_test":"Rerun the inverted pendulum benchmark (Section 5.2) with identical data, kernel, γ, and Δt but horizons T ∈ {5, 10, 20, 40, 80} seconds (H ∈ {500, 1000, 2000, 4000, 8000} steps). Evaluate the resulting stationary policies on the same 10×50 initial conditions and report the mean accumulated cost. If the cost varies by more than about 10% as T doubles, the finite-horizon truncation is not in the stationary regime and the claim is unsupported; if it plateaus, the concern is resolved for this benchmark. Additionally, compute the vector e_γ minimizing ||K e_γ - 1||^2_γ (the regularized representation of the constant function) and report ||B e_γ|| to quantify the spurious T-proportional term in the policy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper solves the finite-horizon FVP (Eq. \\HJB-FVP) with zero terminal condition and then uses the t=0 policy π*(x)=u*(⟨Bv0,k(x)⟩) (Algorithm 1) as a stationary policy for the infinite-horizon average-cost problem (2). For the exact HJB, the finite-horizon value function satisfies V_T(0,x)=λT+w(x)+o(1) as T→∞, and because B^*1=0, the T-dependent constant cancels in the policy, which converges to u*(B^*w(x)). In the finite-dimensional approximation, the constant function 1 is not in the RKHS H for the Gaussian/Laplace kernels used in the experiments (such kernels' RKHS functions vanish at infinity). Consequently no vector e∈R^N satisfies \\hat{E}e=1 exactly; the coefficient vector v0 contains a component proportional to T that represents the constant only approximately. The policy then includes an unexamined term λT⟨Be,k(x)⟩, which may be nonzero for finite N and γ. The paper provides no theorem on the T→∞ limit, no bound on ∥Be∥, and no numerical sweep over the horizon T. Since H and Δt in Table 1 are fixed hyperparameters, the extracted policy could depend on an arbitrary finite T, undermining the claim that Algorithm 1 yields an approximately globally optimal feedback policy for the infinite-horizon problem (2).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a data-driven method for stochastic optimal control by learning the infinitesimal generator of a controlled diffusion in an RKHS. The key theoretical elements are Lemma 1, which expresses the empirical target kernel matrices K_pi in terms of drift derivatives and the kernel's gradient and Hessian, and Proposition 2, which reduces an infinite-dimensional HJB final-value problem to an N-dimensional ODE system, the \"HJB-FVP\" recursion. Algorithm 1, named IG-KHJB, then outputs a feedback policy from the solution of this finite-dimensional recursion. The paper reports experiments on a Van der Pol oscillator, an inverted pendulum, and a cartpole, comparing against a discrete-time kernel HJB method and against Altro-NMPC.","tokens_in":13850,"tokens_out":7134,"duration_ms":62854,"significance":"The paper's main contribution is a clean operator-theoretic derivation that avoids explicit grid-based PDE discretization: the derivative-reproducing property turns generator regression into kernel matrix computations, and the control-affine structure yields a finite-dimensional HJB recursion. If the infinite-horizon and approximation issues are resolved, the approach would be a practical data-driven alternative for nonlinear stochastic optimal control. The paper is explicit about the need for derivative data in Assumption 2 and uses finite differences in the simulator experiments, but the consequences of that approximation are not analyzed. The central claims are plausible, but they currently lack the error analysis needed to support the phrase \"approximately globally optimal\".","major_comments":[{"comment":"The paper formulates an infinite-horizon average-cost problem (2) but computes the policy from the solution of a finite-horizon FVP with zero terminal condition, V(T)=0. For the exact infinite-horizon HJB, V_T(0,x) = lambda*T + w(x) + o(1) as T tends to infinity, and the T-dependent term cancels in the policy because B^* 1 = 0. In the finite-rank approximation, the constant function 1 is not in the RKHS for the Gaussian or Laplace kernels used in the experiments, so no vector e in R^N satisfies the approximate identity exactly; the solution v0 therefore contains a component proportional to T, and the policy contains the unexamined term lambda*T*<B e, k(x)>. The manuscript provides no theorem on the T-to-infinity limit, no bound on ||B e||, and no numerical sweep over the horizon H (Table 1 fixes H to 1000, 500, and 3000). As a result, the claim that Algorithm 1 solves the infinite-horizon problem (2) is not supported. The authors should either prove that the T-dependence cancels in the finite-dimensional recursion, add a T-sensitivity analysis, or reframe the contribution as finite-horizon optimal control.","section":"Section 3.1, Eq. (HJB-FVP), and Algorithm 1"},{"comment":"Assumption 2 requires exact instantaneous derivative measurements at the data points. In the Dojo experiments these derivatives are approximated by finite differences, as stated in Section 5, but no bias or error bound is given. Lemma 1 uses the derivative-reproducing property on the exact drift values; a finite-difference approximation biases the target matrices K_pi and hence the estimated operators in Eq. (12) in a way that is not captured by the RKHS regression framework. Since this assumption is the main practical limitation of the method, the paper should quantify the resulting error, for example in terms of the finite-difference step and the smoothness of f and G, or provide empirical validation of the finite-difference approximation.","section":"Assumption 2 and Section 5 (Implementation details)"},{"comment":"Proposition 2 establishes an algebraic equivalence between the infinite-dimensional estimated HJB and the finite-dimensional recursion, but it is not an approximation result: it does not state how the solution v0 and the resulting policy relate to the true value function or optimal policy as N tends to infinity, gamma tends to zero, or as the empirical estimator converges. The abstract and Section 6 use the phrase \"approximately globally optimal\" without a quantitative error bound. At minimum, a consistency statement, such as convergence of the reconstructed value function to the true relative value function in an appropriate norm under Assumptions 1 and 2, is needed to support the central claim.","section":"Section 4, Proposition 2"}],"minor_comments":[{"comment":"In the definition of the matrix B_i, the index m should be i; the current subscript is inconsistent with the surrounding notation.","section":"Section 4, Proposition 2"},{"comment":"The phrase \"Structure1.\" contains a missing space and should read \"Structure. 1.\" or similar.","section":"Section 1"},{"comment":"The text refers to Figure 4(a) and Figure 4(b), but the displayed figure is captioned Figure 3; the figure numbering should be corrected.","section":"Sections 5.2 and 5.3"},{"comment":"The column headers and entries in Table 1 are difficult to parse because the delimiters are not aligned; the authors should reformat the table so that each column is unambiguous.","section":"Table 1"},{"comment":"The notation D_r(Bv) is overloaded: in the proposition it denotes a vector in R^N, while in Algorithm 1 it is defined through the regularized inverse K_gamma^{-1}. Please use distinct symbols for the pointwise function D_r and its coefficient vector.","section":"Section 4, Proposition 2 and Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on Bevanda et al. The new thing is Lemma 1: a clean application of Itô's formula and derivative reproducing kernels to estimate the adjoint generator of a controlled diffusion from instantaneous derivative data. That, plus the control-affine decomposition B = K^{-1}(K_{e_j} - K_0), is a neat trick and sits correctly next to Kostic et al.'s uncontrolled generator regression. Proposition 2's reduction to an N-dimensional FVP is correct as far as it goes. The numerical results show the method can beat a discrete-time kernel HJB and a well-tuned NMPC on two swing-up tasks. That earns credit.\n\nThe soft spots. The biggest issue is the infinite-horizon claim. The paper solves a finite-horizon FVP with zero terminal condition, then takes the t=0 policy as an approximation of the ergodic optimal policy. For the exact problem the T-dependent part of the value function drops out of the policy because B*1 = 0. In the RKHS with Gaussian/Laplace kernels, constants are not in the space, so the finite-rank B does not exactly annihilate the approximate constant, and the policy can carry a spurious term proportional to T. The paper gives no T→∞ analysis and no bound on ∥B e∥; Table 1 just fixes H. This matters because the method's claim is global optimality for (2). The authors should either restrict to a setting where the value function lies in the RKHS (compact domain, decaying stage cost) or provide an explicit error estimate in T.\n\nSecond, Assumption 2 requires derivative labels f(x(i)) and f_{e_j}(x(i)). They use finite differences in the Dojo experiments and don't quantify the induced bias. That's a correct observation; it limits the method mostly to simulators or models. Third, no code or data, and all hyperparameters hand-tuned. That weakens the empirical comparison.\n\nAll that said, the core derivation is sound and the paper is honest about the derivative-label assumption. It just needs to address the horizon issue before the main claim is supportable. I'd send it to review: it deserves a serious referee, but with major revision likely.","headline":"Novel generator-regression kernel HJB with a real infinite-horizon gap; deserves referee time but needs major revision.","tokens_in":14432,"tokens_out":3072,"would_cite":true,"duration_ms":28406,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["49L20","93E20","46E22","47B32"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that learning the diffusion generator from drift snapshots turns stochastic optimal control into a finite ODE.","keywords":["stochastic optimal control","infinitesimal generator","reproducing kernel Hilbert space","Hamilton-Jacobi-Bellman equation","Fokker-Planck-Kolmogorov operator","kernel ridge regression","operator learning","data-driven control"],"falsifier":"Run the method on a one-dimensional Ornstein-Uhlenbeck process whose optimal policy is known analytically, and compare the learned policy against the analytic one while varying how the drift derivatives in the dataset are obtained (exact, finite-difference, or noisy). If the finite-difference version does not converge to the analytic solution as $N$ grows and $\\epsilon$ shrinks, then the derivative-snapshot requirement of Assumption 2, not the kernel regularity condition, is the decisive limitation.","tokens_in":13337,"feed_emoji":"🎛️","tokens_out":16165,"duration_ms":117463,"temperature":0.7,"pith_summary":"The paper claims that the stochastic optimal control problem for a control-affine nonlinear system with small noise can be solved directly from data, without fitting a model in a physical state space or gridding the state space. The load-bearing object is the adjoint infinitesimal generator (the Fokker-Planck-Kolmogorov operator) of the controlled diffusion, estimated in a reproducing kernel Hilbert space from pointwise drift measurements under zero and one-hot inputs. The authors prove that the action of this generator on kernel sections is a kernel matrix whose entries require only drift values, kernel gradients, and kernel Laplacians, and they substitute that matrix into a convex Hamilton-Jacobi-Bellman recursion. This substitution reduces the infinite-dimensional HJB final-value problem to an $N$-dimensional ordinary differential equation whose terminal value produces an explicit feedback policy. A sympathetic reader would care because, if the claim is right, approximately globally optimal feedback policies become computable from modest state-derivative data rather than from a high-dimensional PDE discretization.","feed_headline":"Kernel formula turns optimal control into a finite ODE","feed_subtitle":"A kernel matrix from drift snapshots yields optimal policies without solving a high-dimensional PDE.","key_machinery":"The load-bearing identity is the kernel target matrix $(K_\\pi)_{ij}$ of Lemma 1, which encodes the restriction of the adjoint infinitesimal generator $L_\\pi^*$ to the canonical feature maps $k(\\cdot,x)$. The first term is the drift-gradient (transport) contribution and the second term, $\\epsilon$ times the trace of the kernel Hessian, is the diffusion contribution. The derivative reproducing property of the kernel converts what would be an infinite-dimensional Hilbert-Schmidt regression into finite matrix constructions using only the Gram matrix and derivative evaluations of the kernel. This is what allows the Hamilton-Jacobi-Bellman recursion to collapse to an $N$-dimensional ordinary differential equation.","core_discovery":"The paper's central claim is that the adjoint infinitesimal generator $L_\\pi^*$ of the controlled diffusion $dX_t=(f(X_t)+G(X_t)\\pi(X_t))dt+\\sqrt{2\\epsilon}\\,dW_t$ can be learned nonparametrically in a reproducing kernel Hilbert space, and that the learned generator suffices to solve the stochastic optimal control problem. The key identity (Lemma 1) states that for a smooth symmetric positive-definite kernel $k$, the action of $L_\\pi^*$ on the kernel section $k(\\cdot,x^{(j)})$, evaluated at $x^{(i)}$, is $(K_\\pi)_{ij}=\\langle f(x^{(i)})+G(x^{(i)})\\pi(x^{(i)}),\\,\\nabla_{x^{(i)}}k(x^{(i)},x^{(j)})\\rangle+\\epsilon\\,\\mathrm{Tr}(\\nabla^2_{x^{(i)}}k(x^{(i)},x^{(j)}))$. With $A=K_\\gamma^{-1}K_0$ and $B_m=K_\\gamma^{-1}(K_{e_m}-K_0)$, and with $D_r$ the dual control penalty $D_r(\\lambda)=\\min_{u\\in U}\\{r(u)+\\langle\\lambda,u\\rangle\\}$, the infinite-dimensional HJB recursion becomes the finite-dimensional final-value problem $-\\dot v=Av+q+D_r(Bv)$, $v(T)=0$, on $(0,T)$, and Algorithm 1 sets $\\hat\\pi^\\star(x)=u^\\star(\\langle Bv_0,k(x)\\rangle)$. The paper asserts that this $N$-dimensional ODE, built entirely from Gram matrices and derivative evaluations, approximates the infinite-dimensional solution and yields approximately globally optimal feedback policies.","pith_inferences":["Beyond the paper: the same kernel generator matrices could be used for data-driven certification, since the matrices represent $L^*$ on the sample points and therefore let one evaluate the sign of the generator applied to a candidate Lyapunov or barrier function.","Beyond the paper: the unquantified finite-difference bias under Assumption 2 could be characterized by comparing exact and approximated kernel matrices on a system with a known generator; the paper does not attempt this comparison.","Beyond the paper: the final-value ODE can be re-solved over a moving horizon, yielding an online data-driven model-predictive controller with a kernel terminal cost, an extension the paper leaves implicit."],"forward_implications":["The stochastic optimal control problem reduces to integrating an $N$-dimensional ordinary differential equation backward in time, so the computational complexity scales with the number of data points rather than with the dimension of the state space.","The estimator includes nonparametric estimators of uncontrolled infinitesimal generators as the special case $\\pi=0$, so the same construction yields a data-driven method for approximating Fokker-Planck-Kolmogorov operators.","Because the representation is in continuous time, the method does not depend on a chosen time lag of the data, in contrast to discrete-time Koopman and transfer-operator methods, and it returns explicit value and policy functions.","In the reported experiments, the learned kernel policy achieves lower accumulated stage costs than a receding-horizon nonlinear MPC solver on the same pendulum and cartpole dynamics, and the method needs fewer data points and is less sensitive to the kernel lengthscale than the discrete-time kernel HJB baseline."],"supporting_citations":[{"why":"Supplies the convex operator-theoretic FPK/HJB duality, the strong-duality conditions, and the ergodic formulation that the finite-dimensional HJB recursion builds on.","marker":"Houska (2025)"},{"why":"Provides Assumption 1 and the discrete-time kernel HJB method that this paper extends to continuous time, and serves as the main baseline in the unstable-oscillator experiment.","marker":"Bevanda et al. (2024b)"},{"why":"Prior work on learning infinitesimal generators of stochastic diffusion processes; supplies the Itô-formula connection used in the proof of Lemma 1.","marker":"Kostić et al. (2024)"},{"why":"Gives the derivative reproducing property used in Lemma 1 to evaluate the adjoint generator on kernel sections.","marker":"Zhou (2008)"},{"why":"The stochastic-calculus reference for Itô's formula, invoked in the derivation of the kernel target matrix.","marker":"Arnold (1974)"},{"why":"Establishes the Hilbert-Schmidt regression framework in which the generator risk functional (7) is formulated.","marker":"Mollenhauer et al. (2022)"}],"fun_headline_variants":["Data-driven generator: control via ODE, not PDE","RKHS generator yields finite ODE for optimal control","Kernel adjoint generator: ODE replaces high-dim PDE","Learn the generator, solve control with an ODE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method only works as stated if the dataset contains exact instantaneous derivative measurements of the drift vector field at the sampled states under zero control and under each one-hot input; when those derivatives are approximated by finite differences, as in the authors' simulated-robot experiments, the resulting bias in the learned generator is left unquantified.","fun_headline_variants_meta":{"raw":{"variants":["Data-driven generator: control via ODE, not PDE","RKHS generator yields finite ODE for optimal control","Kernel adjoint generator: ODE replaces high-dim PDE","Learn the generator, solve control with an ODE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000328,"raw_usage":{"total_tokens":1870,"prompt_tokens":1024,"completion_tokens":846,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":780}},"tokens_in":640,"tokens_out":846,"duration_ms":7726,"temperature":1.0,"reasoning_tokens":780,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:17:48.337275+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the method on a one-dimensional Ornstein-Uhlenbeck process whose optimal policy is known analytically, and compare the learned policy against the analytic one while varying how the drift derivatives in the dataset are obtained (exact, finite-difference, or noisy). If the finite-difference version does not converge to the analytic solution as $N$ grows and $\\epsilon$ shrinks, then the derivative-snapshot requirement of Assumption 2, not the kernel regularity condition, is the decisive limitation.","supporting_citations":[{"cited_title":"Derivative reproducing properties for kernel methods in learning theory","cited_arxiv_id":null,"evidence_quote":"Gives the derivative reproducing property used in Lemma 1 to evaluate the adjoint generator on kernel sections."},{"cited_title":"Stochastic differential equations: theory and applications, volume 2","cited_arxiv_id":null,"evidence_quote":"The stochastic-calculus reference for Itô's formula, invoked in the derivation of the kernel target matrix."}],"review_version":1}