{"id":"cd093ab6-f0c5-485a-93da-3d177ccc59b2","arxiv_id":"2505.03706","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Online policy-gradient updates for unknown LQR systems are shown to be sequentially stable and convergent to the optimal gain, for indirect, direct, natural-gradient, Gauss-Newton and regularized versions.","lead":"This paper designs adaptive controllers that update the feedback gain of an unknown linear system by descending the LQR cost gradient using online data, and it proves these updates keep the system stable while the gain converges to the optimal one. The work matters because it gives reinforcement-learning-style policy updates rigorous stability and convergence certificates in a classic control problem.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract claims stability and convergence for all PGAC variants, but the direct stability certificate (Lemma 22), Gauss-Newton Theorem 4, and several regularization proofs are omitted; the 'all approaches' claim is conditional pending those proofs.","rationale":"Read in good faith, the central theoretical contribution is the indirect vanilla-gradient Theorem 1, and the appendix gives a complete proof for it. The convergence and stability certificates for the other methods are presented as corollaries 'with different values of nu_i' or 'same vein.' Those statements are likely fillable, but they are not demonstrations. I agree with the reader's CONDITIONAL verdict, though I locate the load-bearing issue slightly differently: the reader's weakest assumption concerns existential constants nu_i and uncomputable stepsize/SNR thresholds. That is a real practical limitation, but it does not undermine the logical claim, since existence of such constants is the standard form of these guarantees. The missing proofs do bear on the logical claim, because the abstract asserts stability and convergence for every PGAC variant and Figure 1 classes them all as 'certified.' Lemma 22 is the only certificate for direct PGAC stability; without it, Theorem 2's state bound is asserted, not proved. Similarly Theorem 4 has no proof. Thus the paper's content is best described as: a fully proven indirect vanilla-gradient result plus credible but unproven extensions. That matches a conditional accept subject to completing those proofs, not a rejection. The authors clearly state what is omitted, and this is a verifiability issue rather than evidence of an error.","tokens_in":28115,"tokens_out":15479,"duration_ms":153281,"concrete_test":"Ask for a complete written proof of Lemma 22. The crucial step is to reproduce Lemma 17 with the update K_{t+1}-K_t = eta_t M_{t+1} grad-hat-C_{t+1}(K_t) and Lemma 20 in place of Lemma 14, showing that under eta_t <= ||M_t||^{-1} min{p4 gamma_t/(p3 delta_t), p7, 1/(2 p5 p8), ...} both ||K_{t+1}-K_t|| <= p4 and ||Sigma_{t+1}-Sigma_t|| < 1/2 hold; a successful reconstruction settles that Theorem 2's stability claim is sound, while any step that requires an additional bound on M_t beyond Lemma 6 would make the direct stability certificate incomplete. Separately, request the promised proofs of Theorem 4 and of Lemmas 23-24 to close the Gauss-Newton and natural-gradient claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The vanilla indirect PGAC result in Theorem 1 has a complete proof chain (Lemmas 10-18), so the core idea is credible. The load-bearing gap is that the paper's headline claim extends that guarantee to every method in Figure 1, while several of those proofs are not in the manuscript. Lemma 22, which certifies sequential stability for direct PGAC and is needed for the state bound (22) in Theorem 2, is stated with only 'proof follows same vein ... omitted due to space limitation.' Theorem 4 (indirect Gauss-Newton) is stated with no proof at all; Lemmas 23-24 underpinning the natural-gradient Theorem 3 are also stated as 'proof follows ... omitted'; Lemma 8 and Lemma 9 for the regularized gradients are likewise omitted; and Theorem 6's proof ends with 'rest follows ... omitted.' The stability of Algorithm 2 therefore rests on an analogy with Algorithm 1 rather than a verified argument, and the abstract's 'for all considered PGAC approaches' overstates what is currently established. This is a verifiability gap, not an observed contradiction: the omitted proofs may go through, and the paper already provides many ingredients. The existential constants nu_i are a further practical limitation but not the same kind of gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes policy gradient adaptive control (PGAC) for the linear quadratic regulator, unifying indirect (model-based identification followed by gradient computation) and direct (sample-covariance parameterization) approaches. The algorithm alternates between applying a state-feedback policy plus probing noise and performing one gradient step on the LQR cost, with variants using vanilla gradient, natural gradient, Gauss-Newton updates, and a variance-based regularizer. The central claim is that, under a sufficiently large data signal-to-noise ratio and a sufficiently small stepsize, the policy sequence is sequentially stable and the optimality gap converges to a bias that scales inversely with the SNR. The main theoretical result with a complete proof chain is Theorem 1 for indirect vanilla PGAC, proved in Appendices A-B. Theorems 2-7 extend the guarantee to direct PGAC, natural gradient, Gauss-Newton, and regularized variants, but several load-bearing proofs are omitted or only sketched.","tokens_in":28311,"tokens_out":3999,"duration_ms":43183,"significance":"If all the stated theorems are correct, this is a significant contribution: it provides stability and optimality certificates for online gradient-based LQR adaptation, bridges indirect and direct data-driven formulations, connects Gauss-Newton PGAC to an adaptive Hewer's algorithm, and improves the convergence rate over the authors' earlier DeePO result. The proof of Theorem 1 is detailed and makes explicit use of gradient dominance, local smoothness, and Lyapunov perturbation bounds, which lends credibility to the core idea. The simulation study is small but informative, and the authors provide code. The main weakness is that the headline 'for all considered PGAC approaches' claim rests on several omitted proofs, notably Lemma 22, Lemmas 23-24, Lemma 27, and the proofs of Theorems 4, 6, and 7.","major_comments":[{"comment":"The proof of Lemma 22, which certifies sequential stability and the state bound for direct PGAC, is omitted with the statement 'follows the same vein of that of Lemmas 17 and 18'. This is load-bearing for Theorem 2: direct PGAC updates K by K' = K - eta M grad, where M is data-dependent, so one cannot simply reuse the indirect proof without additional bounds on M and its variation. The stability of Algorithm 2 is therefore not currently verified. A full proof is required.","section":"Appendix A, Lemma 22"},{"comment":"The natural-gradient PGAC guarantee in Theorem 3 depends on Lemmas 23 and 24, which bound ||E_hat - E|| and the resulting cost difference; both proofs are omitted. Since the natural-gradient update has a different gradient error structure than the vanilla gradient, the claimed generalization of Theorem 1 does not follow automatically. These proofs must be supplied before Theorem 3 can be considered established.","section":"Appendix B, Lemmas 23-24 and Theorem 3"},{"comment":"Theorem 4, which gives convergence and stability for indirect Gauss-Newton PGAC, is stated without proof, and Lemma 27, the perturbation bound for the inverse term (R + B^T P B)^{-1} E, is stated as 'proof follows from that of Lemma 12 and is omitted'. The Gauss-Newton update is not a small perturbation of the vanilla gradient; the inverse term introduces additional nonlinear dependence on the estimate. The stated convergence certificate (35) is therefore unverified.","section":"Section IV-B, Theorem 4 and Lemma 27"},{"comment":"The regularization results are not proven. Lemma 8 and Lemma 9 (gradient expressions for the regularized costs) are omitted, Lemma 31 is given but the rest of the proof of Theorem 6 is omitted, and Theorem 7 has no proof at all. Since the decay condition lambda_t <= nu_6 delta_t is essential for the claimed convergence to the unregularized optimal gain, the proofs of these statements must be included.","section":"Section V, Theorems 6-7"},{"comment":"The constants nu_i and the polynomial bounds p_i in all theorems depend on the unknown plant (A,B,Q,R) and the initial gain K_{t0}. Consequently, the stepsize condition eta <= min{nu_2, 2 mu} and the SNR condition SNR_t >= nu_1 cannot be verified from data alone. This is a practical limitation that should be stated clearly, and the paper would be stronger if it gave even conservative explicit estimates for the constants or a procedure to certify them online.","section":"Theorems 1-7, conditions on nu_i"}],"minor_comments":[{"comment":"The abstract states 'For all the considered PGAC approaches, we show closed-loop stability and convergence', but the manuscript currently omits proofs for several of these approaches. The wording should be softened or the missing proofs added before publication.","section":"Abstract and Section I"},{"comment":"In the statement of Lemma 6, the lower bound is written as 'sigma(M_t) >= gamma_t^2' but the surrounding text and proof suggest 'sigma(M_t) >= gamma_t^2' should be 'sigma(M_t) >= gamma_t^2' with a superscript; please clarify the notation.","section":"Section III-C, Lemma 6"},{"comment":"The sentence 'the condition on the initial policy K_{t0} can be replaced with a condition on the SNR' is asserted without proof. If this claim is used, it should be proven or explicitly marked as a conjecture.","section":"Theorem 5, discussion after (36)"},{"comment":"The figures do not show the natural-gradient and Gauss-Newton curves in Fig. 2, and the text says they 'lie between those of the one-shot-based method and indirect PGAC with the vanilla gradient'. It would be clearer to include them or state explicitly why they are omitted.","section":"Section VI-A, Figures 2-3"},{"comment":"The condition SNR_t >= max{nu_1, nu_2 ||M_t||/sigma(M_t)} combines the information metric with the conditioning of M_t. Since M_t depends on online data, the paper should clarify whether this condition can be monitored online and what happens when it is violated.","section":"Section III-C, Theorem 2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely problem and the proof of Theorem 1 is a genuine contribution. My main concern is verifiability: the headline claims for direct PGAC, natural gradient, Gauss-Newton, and regularization are not supported by proofs in the current manuscript. If the authors can supply full proofs for Lemma 22, Lemmas 23-24, Lemma 27, and Theorems 4, 6, and 7, and those proofs check out, I would view the paper as suitable for publication. The existential constants issue is a known limitation in this literature but should be acknowledged."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: the indirect vanilla-gradient PGAC theorem is real and the proof chain is mostly there; the paper overstates its coverage by claiming every method in Figure 1 is certified while several load-bearing proofs are left out. Still worth a serious referee.\n\nWhat is genuinely new: an indirect PGAC with sequential stability and an SNR-dependent convergence bound (Theorem 1), the natural-gradient equivalence between indirect and direct updates (Lemma 7), the Gauss-Newton connection to an adaptive Hewer's algorithm (Algorithm 3, Theorem 5), and the regularization scheme that preserves the guarantees. The linear convergence over the earlier DeePO sublinear rate is a real improvement, and the simulations back the claims. Code is provided, which helps.\n\nThe soft spots are the omitted proofs. Lemma 22, the direct-PGAC stability certificate, is essential for Theorem 2 and is dismissed with 'proof follows the same vein.' Theorem 4 has no proof at all. Lemmas 23, 24, 27, 8, 9, and the tail of Theorem 6 are similarly abbreviated. The abstract says 'for all the considered PGAC approaches' stability and convergence are shown, but the direct variant and the natural-gradient/Gauss-Newton variants rest on analogy rather than verified arguments. I do not see an observed contradiction, and the missing pieces look fillable from the ingredients already given, but the manuscript as it stands does not deliver the advertised coverage.\n\nA second, lesser concern is that all the constants nu_i and the stepsize conditions depend on the unknown plant and initial gain, so the user cannot compute a valid stepsize from data alone. That is a tuning limitation rather than a flaw in the convergence logic—it is common in this literature and does not undermine the central argument.\n\nWho is this for? Anyone working on adaptive LQR, direct data-driven control, or policy-gradient RL for control. The core result on indirect vanilla PGAC is a genuine contribution, and the framework will likely become a baseline. It deserves a serious referee, but the referee should ask for the complete appendices before accepting. My recommendation: send it to peer review with the expectation of heavy revision, and insist that the omitted proofs appear in the final version.","headline":"The indirect vanilla-gradient PGAC proof is credible and the framework is new, but the paper's 'all approaches' guarantee outruns the proofs actually present; still worth a serious referee.","tokens_in":28925,"tokens_out":1477,"would_cite":true,"duration_ms":16013,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["93C40","93E35","49N10"],"pacs":[],"model":"deepseek-v4-flash","headline":"Online gradient steps can learn the optimal LQR gain while never losing closed-loop stability.","keywords":["adaptive control","policy gradient","linear quadratic regulator","closed-loop stability","signal-to-noise ratio","data-driven control","certainty equivalence","online learning"],"falsifier":"On the benchmark of Section VI with i.i.d. Gaussian noise and constant probing excitation, run Algorithm 1 and record $C(K_t)-C^*$ together with $\\mathrm{SNR}_t$; the theorem predicts the optimality gap is bounded by an exponentially decaying term plus a term proportional to $\\sum(1-\\eta/2\\mu)^{t-i}/\\mathrm{SNR}_i$, which is $O(1/\\sqrt{t})$. If the empirical gap decays slower than $O(1/\\sqrt{t})$, or diverges while $\\mathrm{SNR}_t$ stays above the threshold and $\\eta$ below its bound, the certified bound is false.","tokens_in":27833,"feed_emoji":"🎛️","tokens_out":8138,"duration_ms":79556,"temperature":0.7,"pith_summary":"The paper sets out to show that a feedback controller can learn the optimal linear-quadratic regulator gain online by taking small gradient steps on the LQR cost using only closed-loop data, while staying inside the set of stabilizing gains. This matters because optimality-seeking adaptive controllers, including reinforcement-learning-style policy updates, have generally lacked stability guarantees. The central result is a per-time bound: the optimality gap decays exponentially from its initial value plus a bias that scales with the inverse signal-to-noise ratio of the data, and the state trajectory stays bounded by an exponentially decaying term plus a noise-dependent bias. The same template is shown to cover indirect and direct formulations, vanilla and natural gradients, a Gauss-Newton update that becomes an adaptive Hewer's algorithm, and a variance-based regularizer.","feed_headline":"Gradient updates learn optimal LQR control while staying stable","feed_subtitle":"Stability and optimality coexist; the optimality gap shrinks exponentially plus a bias set by the data's signal-to-noise ratio.","key_machinery":"The load-bearing object is the policy-gradient structure of the LQR cost $C(K)$ together with the signal-to-noise ratio $\\mathrm{SNR}_t=\\gamma_t/\\delta_t$ of the collected data. The non-convex cost is gradient dominated, so exact gradient descent on the true plant converges linearly, and the paper's perturbation lemmas show that replacing the true model with the least-squares estimate perturbs the per-step cost decrease only by a term proportional to $1/\\mathrm{SNR}_t$. Sequential stability of the switched closed-loop system is then enforced by choosing the stepsize small enough that consecutive gains change slowly, in the precise sense of strong stability. A further identity links the two families: the direct projected gradient update is exactly the indirect update premultiplied by a data-dependent positive definite matrix $M_t$, which is why both methods share the same stability-and-convergence template.","core_discovery":"On its own terms, the paper claims that policy gradient adaptive control solves a two-part problem: starting from a stabilizing gain and offline persistently exciting data, alternate between applying a feedback-plus-probing input and taking one gradient step on a certainty-equivalence LQR cost. In the indirect version the gradient is computed through a recursive least-squares model estimate; in the direct version it is computed through a sample-covariance parameterization of the policy. For every variant the authors prove sequential stability of the switched closed-loop system and non-asymptotic convergence of the gain to the optimal LQR gain. Theorem 1 states the result for the indirect vanilla-gradient case: if the data signal-to-noise ratio stays above a threshold and the stepsize is small enough, the state obeys the bound in (22) and the optimality gap obeys (23), an exponential contraction plus an SNR-weighted bias. The signal-to-noise ratio is the information metric that sets the convergence rate, so the guarantees degrade smoothly as data become noisier relative to excitation, in line with the monotonicity principles of adaptive control.","pith_inferences":["Beyond the paper, the SNR-weighted bias term suggests a general recipe for adaptive reinforcement learning: whenever an estimated gradient is used, stepsizes inversely proportional to the current SNR yield both a stability certificate and a rate, with the LQR serving as the case where gradient dominance is available.","The equivalence between direct and indirect updates through the matrix $M_t$ indicates that the conditioning of the data matrix, not just its minimum singular value, controls how close a model-free update is to a model-based one; shaping probing noise to improve the spectrum of $\\Phi_t$ is a plausible accelerator for direct PGAC.","The simulations show the optimality gap decreasing as $O(1/t)$ even though the certified bound is $O(1/\\sqrt{t})$, so the analysis is plausibly loose; a sharper Lyapunov or bias-decay argument might match the known one-shot adaptive LQR rate."],"forward_implications":["Indirect PGAC with the vanilla gradient converges at the certified rate $O(1/\\sqrt{t})$ when noise is i.i.d. Gaussian and excitation is constant, improving on the earlier sublinear rate for the direct method.","Natural-gradient PGAC costs less per step, since it solves one Lyapunov equation instead of two, and provably bridges the indirect and direct updates.","Setting the Gauss-Newton stepsize to $1/2$ recovers an adaptive version of Hewer's algorithm, which coincides with online identification-based policy iteration and inherits local quadratic convergence near the optimal gain.","Regularizing with a coefficient $\\lambda_t \\le O(\\delta_t)$ preserves all stability and convergence guarantees while compensating uncertainty in the closed-loop covariance; in the paper's simulations it raises the fraction of convergent trials from roughly 81-83 percent to 98-99 percent."],"supporting_citations":[{"why":"Supplies the gradient-dominance and local-smoothness inequalities for the LQR cost on which the entire convergence argument is built.","marker":"[18]"},{"why":"Introduces the sample-covariance parameterization and the direct PGAC update whose convergence rate Theorem 2 improves.","marker":"[21]"},{"why":"Provides the sequential-stability definition and state-bounding machinery used to certify closed-loop stability under switching policies.","marker":"[12]"},{"why":"Hewer's algorithm is the named iterative method that the Gauss-Newton PGAC update becomes at stepsize $1/2$.","marker":"[33]"},{"why":"Online identification-based policy iteration; the adaptive Hewer's algorithm coincides with it and inherits its small-identification-error stability result.","marker":"[15]"},{"why":"Supplies the variance-based regularization for certainty-equivalence LQR whose decay condition $\\lambda_t \\le O(\\delta_t)$ preserves the guarantees.","marker":"[34]"},{"why":"Gives the one-shot adaptive LQR convergence rates and SNR model used for comparison and for interpreting the PGAC rates.","marker":"[11]"},{"why":"Formulates certainty equivalence and regularization in direct data-driven LQR, setting up the design basis for both indirect and direct costs.","marker":"[26]"}],"fun_headline_variants":["Policy gradients adapt LQR gains while preserving stability","Indirect and direct policy gradients both converge to optimal LQR","Stable policy gradient learning for LQR: convergence proven","Natural gradient unifies indirect and direct adaptive LQR","Gradient descent finds optimal LQR gain without destabilizing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proven stability and convergence hold only when the data are informative enough and the policy is updated slowly enough, and the constants defining those thresholds depend on the unknown system and the initial controller, so a user cannot verify the condition from measured data alone.","fun_headline_variants_meta":{"raw":{"variants":["Policy gradients adapt LQR gains while preserving stability","Indirect and direct policy gradients both converge to optimal LQR","Stable policy gradient learning for LQR: convergence proven","Natural gradient unifies indirect and direct adaptive LQR","Gradient descent finds optimal LQR gain without destabilizing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":3022,"prompt_tokens":956,"completion_tokens":2066,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1987}},"tokens_in":572,"tokens_out":2066,"duration_ms":13865,"temperature":1.0,"reasoning_tokens":1987,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:44:19.444307+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the benchmark of Section VI with i.i.d. Gaussian noise and constant probing excitation, run Algorithm 1 and record $C(K_t)-C^*$ together with $\\mathrm{SNR}_t$; the theorem predicts the optimality gap is bounded by an exponentially decaying term plus a term proportional to $\\sum(1-\\eta/2\\mu)^{t-i}/\\mathrm{SNR}_i$, which is $O(1/\\sqrt{t})$. If the empirical gap decays slower than $O(1/\\sqrt{t})$, or diverges while $\\mathrm{SNR}_t$ stays above the threshold and $\\eta$ below its bound, the certified bound is false.","supporting_citations":[{"cited_title":"Global convergence of policy gradient methods for the linear quadratic regulator,","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient-dominance and local-smoothness inequalities for the LQR cost on which the entire convergence argument is built."},{"cited_title":"Learning linear-quadratic regu- lators efficiently with only √ T regret,","cited_arxiv_id":null,"evidence_quote":"Provides the sequential-stability definition and state-bounding machinery used to certify closed-loop stability under switching policies."},{"cited_title":"Robustness of Online Identification-based Policy Iteration to Noisy Data","cited_arxiv_id":"2504.07627","evidence_quote":"Online identification-based policy iteration; the adaptive Hewer's algorithm coincides with it and inherits its small-identification-error stability result."},{"cited_title":"Regularization for Covariance Parameterization of Direct Data-Driven LQR Control","cited_arxiv_id":"2503.02985","evidence_quote":"Supplies the variance-based regularization for certainty-equivalence LQR whose decay condition $\\lambda_t \\le O(\\delta_t)$ preserves the guarantees."},{"cited_title":"Naive exploration is optimal for online LQR,","cited_arxiv_id":null,"evidence_quote":"Gives the one-shot adaptive LQR convergence rates and SNR model used for comparison and for interpreting the PGAC rates."},{"cited_title":"On the certainty-equivalence ap- proach to direct data-driven lqr design,","cited_arxiv_id":null,"evidence_quote":"Formulates certainty equivalence and regularization in direct data-driven LQR, setting up the design basis for both indirect and direct costs."}],"review_version":1}