{"id":"224736d3-5124-4cf9-ad59-5349b4540956","arxiv_id":"2506.08121","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A new coupled continuous-time policy-value iteration dynamics is shown to converge to the optimal value and control for relaxed and classical stochastic control problems, under Hamiltonian monotonicity conditions.","lead":"This paper introduces a continuous policy-value iteration algorithm for stochastic control, in which the estimate of the value function and the estimate of the optimal control are updated simultaneously by Langevin-type differential equations. It proves policy improvement and convergence under a monotonicity condition on the Hamiltonian, which matters because it connects stochastic control to Langevin-sampling and non-convex optimization ideas used in machine learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 22's Wasserstein bound is dimensionally inconsistent: at τ=0 it claims W2(π0,π̃0) ≤ E||u0−ũ0||², which fails for nearby Gaussians; the proof's Gronwall step bounds a squared distance, not W2.","rationale":"I judged the dimensional inconsistency in Proposition 22 to be the most load-bearing because it is a genuine false statement in the paper's main rate theorem, not merely an unverified condition. The proof itself reveals the error: after deriving a bound for E||uτ−ũτ||², the last line equates this with W2. The τ=0 check is decisive and independent of which models satisfy Assumption 20, so the concern does not disappear if the monotonicity conditions are strengthened. This does not overturn the reader's conditional verdict, since the underlying algorithm and policy-improvement idea may be repairable by restating the theorem as a squared-Wasserstein estimate and adjusting the rate to min(κ,β). However, it raises the correctness risk for the headline quantitative claim. The reader's weakest-assumption choice (monotonicity) captures a separate gap, and the two issues together support keeping the paper conditional rather than accepting it.","tokens_in":33566,"tokens_out":26704,"duration_ms":343619,"concrete_test":"At τ=0, use the scalar LQ model of Section 6 with f=−N u²/2, b=0, σ=0, λ>0, β>0, and N large enough for condition (30), with π0=N(0,σ²), π̃0=N(ε,σ²) and Ũ0=U0+ε. Then W2(π0,π̃0)=ε and E||u0−ũ0||²=ε², contradicting Proposition 22 for ε<1. To test the intended repair, re-derive the last line of Proposition 22: replace W2 by W2² or take square roots in the Gronwall estimate, and verify that the resulting exponential rate is min(κ,β), not κ or 2κ.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 22 states W2(πτ,π̃τ) ≤ e^{-2κτ}E||u0−ũ0||² + (e^{-2βτ} − e^{-2κτ})/(2(κ−β)) C0. The proof after (35) applies Gronwall to d/dτ ||uτ−ũτ||², producing a bound on E||uτ−ũτ||², and then the last line changes the left-hand side to W2 without taking a square root. The displayed theorem is therefore false as written. At τ=0 the second term vanishes and the claim reduces to W2(π0,π̃0) ≤ E||u0−ũ0||². In the scalar LQ example of Section 6, take π0=N(0,σ²), π̃0=N(ε,σ²) with the synchronous coupling Ũ0=U0+ε; then W2(π0,π̃0)=ε while E||u0−ũ0||²=ε², so the inequality fails for every ε in (0,1). The correct conclusion from the proof is a squared-Wasserstein bound; if that bound is used, the asymptotic rate is min(κ,β), not 2κ. Since Proposition 22 is the main advertised exponential Wasserstein convergence result, the Section 5 rate claim is unsupported as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a continuous-time coupled policy-value iteration for infinite-horizon stochastic control problems. The policy is updated by a Langevin-type SDE in the control variable while the value function, together with its derivatives, is updated simultaneously by an ODE in the iteration parameter. The framework is developed for entropy-regularized relaxed control problems in Section 3 and for classical control problems in Section 4. Under monotonicity conditions on the Hamiltonian, the paper claims exponential Wasserstein convergence of the policy distribution (Propositions 22 and 27) and optimality of the limiting policy via HJB verification (Proposition 10). A linear-quadratic example is analyzed in Section 6 as a partial verification of the assumptions.","tokens_in":33911,"tokens_out":7698,"duration_ms":98690,"significance":"If the main results were rigorously established, this paper would make a useful contribution to the continuous-time reinforcement-learning and stochastic-control literature: it offers a genuinely simultaneous continuous update of value and policy, connects relaxed control to mean-field Langevin sampling, and provides a rate statement for the coupled dynamics. The policy-improvement derivations in Section 3 are elaborate, and the linear-quadratic example is instructive. However, the central convergence and optimality claims are currently not established as written: Proposition 22 contains a false Wasserstein bound, Proposition 9 has a pointwise-convergence gap, and the monotonicity assumptions are not verified from primitive model data except in a special example. No code or numerical experiments are provided, and the authors are honest that annealing, discretization, and the policy-inhomogeneous diffusion analysis are left to future work.","major_comments":[{"comment":"The displayed Wasserstein bound is false as stated. The proof applies Gronwall to d/dτ E||uτ−ũτ||² and obtains a bound on the squared distance, and then the last line substitutes W2(πτ,π̃τ) on the left-hand side without extracting a square root. At τ=0 the bound reduces to W2(π0,π̃0) ≤ E||u0−ũ0||², which fails, for example, in the scalar LQ setting with π0=N(0,σ²), π̃0=N(ε,σ²) and the synchronous coupling Ũ0=U0+ε, where W2(π0,π̃0)=ε while E||u0−ũ0||²=ε² for ε∈(0,1). The correct conclusion of the proof is a squared-Wasserstein bound, and the exponential rate that follows is min(κ,β), not 2κ. Since Proposition 22 is the main advertised rate theorem of Section 5, this is a load-bearing error that must be repaired.","section":"Section 5.1, Proposition 22"},{"comment":"The proof asserts that because the expectation gτ converges and the dissipation term is nonnegative, ∇_u ln πτ converges pointwise to ∇_u H(x,u,v*_x,v*_xx)/λ. Convergence of gτ gives only an L2-type dissipation estimate along the trajectory after (19); it does not imply pointwise convergence of the score ∇_u ln πτ. The subsequent representation of πτ and the identification of π* as the Gibbs density therefore lack proof. Since Proposition 10 uses Proposition 9 to verify the HJB equation, the optimality conclusion is not established as written. Closing this gap requires additional compactness or stronger uniform estimates, which are not present in the manuscript.","section":"Section 3, Proposition 9"},{"comment":"The monotonicity conditions MC I–MC VI are assumed directly on the coupled iterates and are not derived from primitive conditions on f, b, and σ. They are load-bearing: Lemma 21 and Lemma 26 use them to contract vτ, vτx, and vτxx, and those contractions feed into Propositions 22 and 27. For a general model the paper gives no criterion by which a user can check these conditions. In the LQ example, Proposition 33 verifies only conditions (II) and (III), and the argument relies on the auxiliary ODE system for Yτ and Zτ with boundedness claims that are asserted rather than proved. The text also states that 'the first term is negative for sufficiently large τ as shown above' although the preceding computation establishes only that ∂xxE[Hτ]−∂xxE[H̃τ]=0. The example therefore does not provide a complete verification of the monotonicity hypotheses.","section":"Assumptions 20 and 25; Section 6"},{"comment":"The LQ example explicitly violates the boundedness Assumption 2, since f, b, and σ are unbounded in x, while the propositions of Section 3 are proved under that assumption. The text says that 'all the results in that section, in particular the policy improvement and convergence results, still hold while we omit the proofs.' Omitting the verification of the standing assumptions for the example leaves the example's support of the general claims incomplete. The paper should either extend the proofs to the unbounded-coefficient case or clearly present the example as heuristic rather than as a full verification.","section":"Section 6, first paragraph"}],"minor_comments":[{"comment":"The superscript and subscript conventions are inconsistent, especially for ρτ, πτ−s, and vτ(t,x); please harmonize the notation so that the time direction of the state SDE and the iteration direction of the policy are unambiguous.","section":"Notation throughout"},{"comment":"Assumption 8(ii) assumes local boundedness of ∇_u ln πτ uniformly in τ, but the proof of Proposition 9 needs a stronger control to obtain pointwise convergence; the exact regularity requirement should be stated explicitly.","section":"Assumption 8"},{"comment":"In the classical control case, the value function in (23) depends on a realization of the control path, but the probability space and the conditional expectation are not fully specified; the statement should clarify the dependence on the Brownian motion driving the policy iterations.","section":"Definition 13"},{"comment":"The footnote contains a stray superscript and an unproved implication about concavity of H; please correct the typo and provide the argument or remove the footnote.","section":"Proposition 17, footnote"},{"comment":"The remark claims that the continuous dynamics provide theoretical guarantees for discrete algorithm design, but no discretization or finite-step convergence analysis is provided; this should be rephrased as a research program rather than a guarantee.","section":"Remark 24"},{"comment":"The symbol ε(x) is used first as the maximum of the Young inequality constants and later as the minimum; this inconsistent notation should be fixed, and the dimensions in the matrix inequality (30) should be specified precisely.","section":"Proof of Proposition 22"},{"comment":"There are numerous typographical errors (e.g., 'sovle', 'startegies', '2000 AMS Mathematics subject classification') that should be corrected in a revision.","section":"Typos"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of math.OC and the proposed continuous-iteration idea is timely, but the central rate theorem is false as stated and the optimality proof has a substantial gap. I would not recommend rejection on novelty grounds; however, the authors should be asked to repair Proposition 22, justify the pointwise convergence in Proposition 9, and either verify the monotonicity assumptions more convincingly or state clearly that they are purely ad-hoc hypotheses."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is worth taking seriously: a genuinely new iteration-continuous coupled dynamics for value and policy, with Langevin-type updates. It is not in the cited discrete policy-iteration or gradient-flow literature. The policy improvement result (Proposition 5) is coherent, and the connection to mean-field Langevin sampling is a real bridge. Credit where due: the framework is new, the relaxed-control-to-sampling link is plausible, and the authors are honest that the convergence analysis relies on monotonicity conditions they do not derive.\n\nNow the soft spots. The stress-test note is right. Proposition 22 states W2(πτ,π̃τ) ≤ e^{-2κτ}E||u0−ũ0||² + (e^{-2βτ}−e^{-2κτ})/(2(κ−β))C0. The proof bounds ||uτ−ũτ||² via Gronwall and then relabels the left side as W2 without taking a square root. At τ=0 the claimed inequality reduces to W2(π0,π̃0) ≤ E||u0−ũ0||², which fails for nearby Gaussians with the synchronous coupling: W2=ε while E||u0−ũ0||²=ε². What the proof actually gives is a squared-Wasserstein bound, with a rate governed by min(κ,β). That is still a usable rate, but Proposition 22 as displayed is false. Fixable, but it is the main advertised convergence claim.\n\nSecond, Proposition 9 has an unjustified step from L2-type dissipation to pointwise convergence of ∇u ln πτ. The argument may be repairable with more regularity, but as written it does not go through.\n\nThird, the monotonicity Assumptions 20 and 25 are load-bearing. They are assumed, not derived from model data. The LQ example verifies only conditions (II) and (III), for large parameter regimes, and does not verify (I) or the structural condition (30) used in Proposition 22. This limits the paper's reach. To be fair, the assumptions are stated openly, and the policy improvement and HJB verification provide some independent support for the framework.\n\nOverall: the central framework is novel; the convergence-rate section is not yet sound. A serious referee should engage with it because the bugs look fixable and the idea deserves a proper treatment. The paper is primarily for stochastic control and RL theorists interested in continuous-time policy iteration. I would not cite the current rate theorem in my own work until it is corrected.\n\nRecommendation: send to peer review, with a request to fix the Wasserstein/square-root issue, tighten Proposition 9, and either verify or weaken the monotonicity conditions.","headline":"Genuinely new continuous policy-value iteration framework, but the headline Wasserstein rate theorem has a dimensional bug and the monotonicity assumptions are largely unverified.","tokens_in":34385,"tokens_out":2178,"would_cite":false,"duration_ms":27617,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["93E20","93E35","60H10"],"pacs":[],"model":"deepseek-v4-flash","headline":"A continuous policy-value iteration updates control and value together and converges to the optimal stochastic control.","keywords":["stochastic control","relaxed control","policy iteration","value iteration","Langevin dynamics","Hamilton-Jacobi-Bellman equation","policy improvement","Wasserstein distance"],"falsifier":"Run the coupled dynamics on a linear-quadratic model with small values of $|A|$, $M$, or $N$, where the paper's verification of Assumption 20 does not apply, and measure the Wasserstein distance between $\\pi_\\tau$ and $\\pi^*$ together with $v^\\tau$ along $\\tau$. If the distance does not decay exponentially, or if $v^\\tau$ ever decreases, the monotonicity condition is violated and the advertised convergence rate fails.","tokens_in":33313,"feed_emoji":"🎯","tokens_out":5455,"duration_ms":56519,"temperature":0.7,"pith_summary":"This paper proposes a continuous policy-value iteration algorithm for infinite-horizon stochastic control problems, in which the value-function approximation and the policy are updated simultaneously along an iteration variable $\\tau$, rather than in discrete rounds. The policy update is a Langevin-type stochastic differential equation in the control variable, and the value update is a coupled differential equation driven by the current Hamiltonian. The authors prove that the value sequence increases along $\\tau$ (policy improvement), that the limiting policy is the optimal relaxed control, and that the limiting value solves the Hamilton-Jacobi-Bellman equation. Under a sign condition called the monotonicity condition on the Hamiltonian, they prove exponential convergence of the policy distribution to its Gibbs invariant distribution in Wasserstein distance. If correct, the method removes the need to evaluate the value function of each policy iterate and connects stochastic control to Langevin sampling.","feed_headline":"Continuous value-policy iteration converges to the optimal control","feed_subtitle":"Coupled Langevin dynamics eliminate per-iteration PDE solves and reach the optimal policy with exponential convergence.","key_machinery":"The load-bearing object is the coupled dynamics (28) for relaxed control and (37) for classical control: the value $v^\\tau$, its derivatives $v^\\tau_x$ and $v^\\tau_{xx}$ follow ODEs whose drifts are the expected Hamiltonian minus $\\beta v^\\tau$, while the control $u_\\tau$ follows $du_\\tau = \\nabla_u H(x,u_\\tau,v^\\tau_x,v^\\tau_{xx})d\\tau+\\sqrt{2\\lambda}dB_\\tau$. The invariant distribution of this Langevin diffusion at the optimal value is the Gibbs measure $e^{H/\\lambda}/\\int e^{H/\\lambda}du$, which is precisely the optimal relaxed control. The proof of policy improvement uses Itô's formula along the coupled processes; the exponential convergence proof couples two copies of the dynamics with the same Brownian motion and uses the monotonicity conditions to obtain a contraction with rate $\\beta$ on value differences and rate $\\kappa(x)$ on control differences.","core_discovery":"The central claim is that value iteration and policy iteration can be run simultaneously as continuous dynamics: an ODE for the value function and an overdamped Langevin SDE for the control, coupled through the Hamiltonian. For relaxed (entropy-regularized) control, the limiting policy is the Gibbs distribution $e^{H/\\lambda}/\\int e^{H/\\lambda}du$, where $H$ is the Hamiltonian evaluated at the limiting value function, and the limiting value solves the HJB equation (Proposition 10). For classical control, under concavity or an additional condition, the expected value is nondecreasing in $\\tau$, and in the deterministic $\\lambda=0$ case the control converges to the maximizer of the limiting Hamiltonian (Proposition 17). Under Assumptions 20 and 25 the authors prove the exponential Wasserstein convergence rate $W_2(\\pi_\\tau,\\tilde\\pi_\\tau)\\le e^{-2\\kappa(x)\\tau}\\mathbb{E}[\\|u_0-\\tilde u_0\\|^2]+\\frac{e^{-2\\beta\\tau}-e^{-2\\kappa(x)\\tau}}{2(\\kappa(x)-\\beta)}C_0$ using synchronous coupling. In the linear-quadratic example, monotonicity conditions (II) and (III) are verified for sufficiently large values of $|A|$, $M$, and $N$.","pith_inferences":["The monotonicity conditions (MC II and MC III) are stated abstractly and are verified only for a linear-quadratic model in a large-parameter regime; whether they hold for broader classes, such as non-quadratic costs or state-dependent diffusion, remains open.","The analysis treats the continuous iteration directly; any practical discrete-time Euler-Maruyama discretization of (28) or (37) will introduce discretization error, and the paper leaves that error analysis for future work.","The authors flag the policy-inhomogeneous nature of the diffusion (the Hamiltonian changes with $\\tau$) as an open mathematical problem; proving convergence for general time-inhomogeneous Langevin dynamics would directly strengthen this framework.","A testable extension is to run the classical dynamics with an annealing schedule $\\lambda(\\tau)\\to 0$ and check whether the coupled system exits suboptimal wells and reaches the global optimum, connecting the method to simulated-annealing theory."],"forward_implications":["No PDE solve per iteration: the algorithm updates $v$ and $\\pi$ simultaneously, so each step only needs current derivatives, making high-dimensional implementation plausible.","The same continuous iteration handles classical controls with controlled diffusion coefficients, which discrete relaxed-control policy iteration cannot easily handle.","Under the monotonicity condition, value differences decay at rate $e^{-2\\beta\\tau}$ and the policy's Wasserstein distance decays at a rate governed by $\\kappa(x)$.","The Gibbs invariant structure connects stochastic control to Langevin-based distribution sampling, enabling sampling and non-convex-learning tools.","With $\\lambda\\to 0$ annealing, the classical-control dynamics form a continuous bridge to non-convex optimization."],"supporting_citations":[{"why":"Supplies the discrete-time policy iteration for entropy-regularized relaxed control that this paper makes iteration-continuous, including the linear-quadratic baseline.","marker":"Wang et al. (2020)"},{"why":"Extends policy-improvement convergence to a broader class of entropy-regularized control problems; the paper builds on its convergence arguments.","marker":"Huang et al. (2022)"},{"why":"Establishes the relaxed control framework in which controls are randomized at each time and state.","marker":"Fleming and Nisio (1984)"},{"why":"Provides mean-field Langevin dynamics exponential convergence and annealing, the sampling analogue whose invariant Gibbs measure matches the optimal relaxed policy.","marker":"Chizat (2022)"},{"why":"Gives uniform-in-time propagation of chaos for mean-field Langevin dynamics, used to justify the connection to distribution sampling.","marker":"Chen et al. (2022)"},{"why":"Supplies the Schauder interior estimates used in Lemma 7 to obtain locally uniform convergence of the value derivatives.","marker":"Gilbarg and Trudinger (1977)"},{"why":"Recent policy improvement algorithm that removes the per-iteration value evaluation, motivating the simultaneous-update design.","marker":"Ma et al. (2024a)"},{"why":"Establishes existence of optimal relaxed controls, underpinning the admissible-set formulation used in the paper.","marker":"Zhou (1992)"}],"fun_headline_variants":["Continuous value-policy iteration converges to optimal control","Langevin-coupled value-policy iteration converges to optimal control","Simultaneous value-policy iteration converges without PDE solves","Continuous control iteration via Langevin dynamics converges","Value-policy iteration with Langevin updates reaches optimum"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The exponential convergence results rest on the monotonicity condition that the expected change in the Hamiltonian and the expected change in the value function (or its derivatives) always have opposite signs; the paper verifies this only for a linear-quadratic example and only when $|A|$, $M$, and $N$ are sufficiently large.","fun_headline_variants_meta":{"raw":{"variants":["Continuous value-policy iteration converges to optimal control","Langevin-coupled value-policy iteration converges to optimal control","Simultaneous value-policy iteration converges without PDE solves","Continuous control iteration via Langevin dynamics converges","Value-policy iteration with Langevin updates reaches optimum"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000447,"raw_usage":{"total_tokens":2240,"prompt_tokens":912,"completion_tokens":1328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":1252}},"tokens_in":528,"tokens_out":1328,"duration_ms":10801,"temperature":1.0,"reasoning_tokens":1252,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:20:22.158934+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the coupled dynamics on a linear-quadratic model with small values of $|A|$, $M$, or $N$, where the paper's verification of Assumption 20 does not apply, and measure the Wasserstein distance between $\\pi_\\tau$ and $\\pi^*$ together with $v^\\tau$ along $\\tau$. If the distance does not decay exponentially, or if $v^\\tau$ ever decreases, the monotonicity condition is violated and the advertised convergence rate fails.","supporting_citations":[],"review_version":1}