Pith. sign in

REVIEW 3 major objections 4 minor 39 references

Global Optimality of Single-Timescale Actor-Critic under Continuous State-Action Space: A Study on Linear Quadratic Regulator

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper establishes that the classic single-sample single-timescale actor-critic reaches the global optimum on LQR, the first such guarantee on continuous state-action space.

desk verdict First serious finite-time global-optimality claim for single-sample single-timescale actor-critic on LQR, with largely sound algebra, but the theorem overreaches because Assumption 1 is not enforced on the unprojected actor. read the letter →

arxiv 2505.01041 v1 pith:46SLE6XL submitted 2025-05-02 cs.LG

Xuyang Chen , Jingliang Duan , Lin Zhao This is my paper · ORCID
classification cs.LG
keywords actor-criticsingle-timescalelinearquadraticregulatorglobaloptimalitysamplecomplexitycontinuousstate-actionspacenaturalpolicygradientfinite-timeconvergence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to prove that the classic single-sample single-timescale actor-critic—one sample per iteration, actor and critic stepsizes kept in a constant ratio—can find a globally optimal policy on a continuous, unbounded state-action space. Using the linear quadratic regulator as the testbed, it shows that with step sizes $\alpha_t = c/\sqrt{T}$ and $\beta_t = \gamma_t = 1/\sqrt{T}$, the average squared cost-estimator error, the average squared critic error, and the best actor suboptimality gap all shrink as $O(1/\sqrt{T})$. Interpreting the average gap as an $\epsilon$-accuracy criterion, this is an $O(\epsilon^{-2})$ sample-complexity guarantee, the same order previously obtained only for local convergence or finite state-action spaces. The proof's core move is to stop analyzing actor and critic separately and instead solve a coupled system of inequalities for the three estimation errors.

What carries the argument

The central object is the three-variable interconnected error system $(A_T,B_T,C_T)$, where $A_T$ is the average squared cost-estimator error, $B_T$ is the average squared critic error, and $C_T$ is the average squared natural-gradient norm. The paper derives implicit bounds of the form $A_T\le O(1/\sqrt{T})+h_2B_T+h_2C_T$, $B_T\le O(1/\sqrt{T})+h_4\sqrt{A_TB_T}+h_5C_T$, and $C_T\le O(1/\sqrt{T})+h_7\sqrt{B_TC_T}$, then shows these inequalities force all three averages to $O(1/\sqrt{T})$ when the actor/critic stepsize ratio is below an explicit threshold. Carrying the argument are two LQR-specific identities: the natural gradient $\nabla^N_K J(K)=E_K=(R+B^\top P_K B)K-B^\top P_K A$, and the coercive, almost-smooth, gradient-dominated landscape of $J(K)$, capped by $J(K)-J(K^*)\le \frac{1}{\sigma_{\min}(R)}\|D_{K^*}\|\operatorname{Tr}(E_K^\top E_K)$.

What would settle it

Run Algorithm 1 on the two-dimensional LQR example while recording $\rho(A-BK_t)$ at every iteration: if any iterate violates $\rho(A-BK_t)\le\lambda<1$ or leaves the assumed norm bound, the theorem's premise fails; alternatively, set the actor/critic stepsize ratio above the threshold in (51) and check whether the three average errors stop decaying at $O(1/\sqrt{T})$.

Watch

Extended reading notes

Core claim

Stated on the paper's own terms, the discovery is that under Assumption 1 (every iterate $K_t$ stays norm-bounded with $\rho(A-BK_t)\le\lambda<1$), Algorithm 1 achieves $\min_{0\le t<T}\mathbb{E}[J(K_t)-J(K^*)] = O(1/\sqrt{T})$, together with $\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}(\eta_t-J(K_t))^2=O(1/\sqrt{T})$ and $\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\|\omega_t-\omega^*_{K_t}\|^2=O(1/\sqrt{T})$. The paper reads this as the first global optimality proof for the single-sample single-timescale actor-critic on an infinite continuous state-action space, with sample complexity $O(\epsilon^{-2})$ for an $\epsilon$-optimal policy. The reason such a guarantee is not immediate is that the critic and cost estimates are noisy and coupled to the actor update; the paper's claim is that the coupling can be bounded as one interconnected iteration system rather than decoupled.

Load-bearing premise

The load-bearing premise is that every controller produced by the unprojected actor update stays inside a fixed bounded set of stabilizing controllers; the proof assumes this uniform stability rather than enforcing it, and a single excursion outside the set would undo the Lyapunov estimates the convergence argument relies on.

Editorial extensions

If this is right

  • For LQR with unbounded continuous state and action spaces, the single-sample single-timescale actor-critic is globally convergent, not merely locally convergent, at the same $O(\epsilon^{-2})$ sample complexity previously attached to local or finite-space results.
  • The actor, critic, and cost estimator can all run online with one sample per step and proportional step sizes; no inner-loop policy evaluation or two-timescale separation is needed for the guarantee.
  • The step-size ratio between actor and critic must lie below a constant threshold; the paper gives this threshold in closed form, converting a heuristic tuning choice into a stated condition.
  • Relative to double-loop actor-critic for LQR, the paper claims an improvement in sample complexity from $O(\epsilon^{-5})$ to $O(\epsilon^{-2})$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not in the paper: if Assumption 1 were replaced by a projection onto the uniformly stable set $\{K:\rho(A-BK)\le\lambda\}$, the proof might become self-contained; the paper defers exactly such a projection, so testing it would show whether the global claim can survive without the unprojected stable-trajectory premise.
  • Not in the paper: the same coupled-inequality scheme may transfer to other control problems whose cost is coercive, almost smooth, and gradient-dominated; checking whether the three inequalities close on an output-feedback or nonlinear benchmark would locate the boundary of the mechanism.
  • Not in the paper: the stationary-distribution sampling oracle (Line 3) could be relaxed by drawing each state after a short warm-up, since LQR trajectories mix geometrically; quantifying the extra burn-in cost would turn the algorithm into one implementable without an oracle.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies the classic single-sample, single-timescale actor-critic algorithm applied to the infinite-horizon time-average linear quadratic regulator (LQR) with continuous state and action spaces. The actor uses a natural-policy-gradient update based on a critic parameter vector updated by TD(0) with a single sample per iteration, together with a running cost estimator. The main claim (Theorem 1) is that, under Assumption 1 and with step sizes α_t = c/√T and β_t = γ_t = 1/√T, the average squared cost-estimation error, the average squared critic error, and the minimum over t of E[J(K_t)-J(K*)] are all O(1/√T), yielding O(ε^{-2}) sample complexity. The proof derives three coupled error recursions (Theorems 2-4 in the supplement), solves the resulting interconnected inequality system (Theorem 5), and uses the gradient-domination property of LQR to convert natural-gradient convergence into global optimality. The paper frames this as the first global optimality guarantee for single-sample single-timescale actor-critic on unbounded continuous state-action space.

Significance. If the result is established, it is a meaningful step beyond prior work: earlier single-sample single-timescale actor-critic analyses are local and restricted to finite state or action spaces, while the present paper targets global convergence for the LQR benchmark with the same O(ε^{-2}) sample complexity as those local results. The proof machinery is also noteworthy: rather than decoupling actor and critic, it analyzes their estimation errors as an interconnected system and identifies an explicit threshold on the actor/critic step-size ratio that guarantees contraction (condition h2 h4^2 + h2 h4^2 h7^2 + 2 h5 h7^2 < 1). The paper includes reproducible-looking numerical experiments and gives a transparent comparison with zeroth-order and double-loop baselines. However, the current manuscript leaves a load-bearing assumption unenforced: Assumption 1 is imposed on every iterate K_t, but Algorithm 1 does not project the actor and the proof does not close the induction. For this reason the central claim, as stated for Algorithm 1, is not yet established.

major comments (3)
  1. [Assumption 1 and Algorithm 1, Theorem 1] Assumption 1 requires every policy K in the class to satisfy ||K|| ≤ K̄ and ρ(A-BK) ≤ λ < 1, and the proof of Theorem 1 uses this uniform stability for all t to bound D_{K_t}, P_{K_t}, A_{K_t}, the Lipschitz constants l1, l2, and the perturbation bound (Supp. eq. (34)). However, Algorithm 1 projects only η_t and ω_t (Lines 7-8); the actor update on Line 9 is unprojected. The proof shows only that ||K_{t+1}-K_t|| is small (Supp. eq. (34)), which bounds the size of the perturbation but not its direction. Since K_t may be arbitrarily close to the boundary ρ(A-BK)=λ (or even to ρ(A-BK)=1), a one-step perturbation of the size allowed by (34) can move the iterate outside the stabilizing set, where J(K_t), D_{K_t}, P_{K_t}, and the TD recursion are undefined. No induction argument establishes K_t ∈ K for all t; the remark after Assumption 1 explicitly defers the projection map and omits the high-probability argument that would handle bounded trajectories. Thus every Lyapunov-type estimate in Lemmas 5-11 and the gradient-domination conclusion in Lemma 13 are conditional on a property that Algorithm 1 does not enforce and that the proof does not close. This is the central gap: Theorem 1 is not established for Algorithm 1 as stated.
  2. [Algorithm 1, Line 3 and Section 3] Line 3 requires that at each iteration the algorithm sample x_t from the exact stationary distribution ρ_{K_t} of the current policy. This is a strong oracle assumption: in an online, single-trajectory implementation the state distribution is not stationary, and the proof's martingale and variance bounds rely on the sampled pair (x_t,u_t) being drawn from the stationary distribution of K_t. The text cites geometric β-mixing to argue that a sufficiently long burn-in approximates stationarity, but no finite-time bias analysis is provided. Consequently, the advertised O(ε^{-2}) sample complexity is for an idealized sampling oracle, not for the single-trajectory algorithm described in the introduction. The authors should either state this idealization explicitly as a limitation or extend the analysis to the non-stationary (or transient) setting.
  3. [Theorem 1 statement vs. proof, Supp. A.5] The theorem states min_{0≤t<T} E[J(K_t)-J(K*)] = O(1/√T), and the proof obtains this by bounding the average of E[J(K_t)-J(K*)] via the average of E||E_{K_t}||^2 and then using min ≤ average. This step is valid. However, in Supp. A.5 the text says 'From the convergence of C_T, we know that 1/T Σ E||E_{K_t}|| = O(1/√T)', which should read E||E_{K_t}||^2. The correct statement is used implicitly in the displayed inequality, so this is a typographical error rather than a mathematical flaw, but it should be corrected for clarity.
minor comments (4)
  1. [Supp. A.1, A.2, A.3] Theorems 2, 3, and 4 each begin 'Suppose that Assumptions 1 and 1 hold'; the second '1' appears to be a typo for a differently numbered assumption or intended to be repeated Assumption 1.
  2. [Theorem 4 and Supp. eq. (43)] In the statement of Theorem 4, the constant in the first term is written as (U + 2 c_4 c_α^2)/(2 σ_min(D0) c), while the proof uses c_α and c interchangeably. Since α_t = c/√T was introduced, the notation should be unified to avoid confusion about which constant is being optimized.
  3. [Section 4.1, proof sketch] In the proof sketch, the line 'we project η into a ball of radius U and project ω into a ball of radius ω̄' is helpful, but the text does not mention that the actor is not projected; given that this is the source of the main gap, the sketch should flag this explicitly.
  4. [Abstract and Section 1] The abstract says 'we show that the popular single-timescale actor-critic can attain an epsilon-optimal solution with ... sample complexity for solving LQR'; given the sampling oracle and the Assumption 1 issue, this statement should be qualified as holding under the stated idealized assumptions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence target is external to the algorithm, and the main lemmas are proved in the paper rather than reduced to fitted inputs or author-imposed definitions.

full rationale

The paper's claimed convergence quantities J(K*), omega*_K = svec(Omega_K), and E_K are fixed by the LQR problem (Lemmas 1, 2, 4) and the gradient-domination benchmark (Lemma 13), not by any fitted parameter or by the algorithm's own iterates. Theorem 1 does not define its target in terms of eta_t or omega_t, and no step of the proof renames a fitted value as a prediction. The derivation is largely self-contained: Lemmas 1, 2, 4, 12, and 13 are proved from the LQR Lyapunov/Bellman equations; Lemma 3's coercivity proof is included in the supplement despite citing [Duan et al. 2023]; the perturbation bound Lemma 8 is quoted from [Yang et al. 2019], an external source, and all subsequent Lipschitz constants are derived from it. The self-citations ([Chen and Zhao 2022], [Duan et al. 2023]) are contextual or for lemmas whose proofs are given, so they are not load-bearing. The serious weakness---Assumption 1 is assumed for all iterates while Algorithm 1 projects no actor---is a gap between assumptions and algorithm, not a circularity: the convergence target is still external to the algorithm, and the proof does not use the algorithm's output to define J(K*). Therefore no circular step is identifiable.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

This is a convergence proof rather than a derivation of a constant. The central claim leans on the trajectory-level stability assumption (Assumption 1), the stationary-sampling oracle, classical LQR/ARE results imported from [Fazel et al. 2018] and [Yang et al. 2019], and Gaussian moment identities from the statistical literature. No parameters are fitted to data; c and σ are a priori hyperparameters. No new entities are postulated.

free parameters (2)
  • actor step-size ratio c = chosen below threshold (51); no explicit value given
    The theorem sets α_t = c/√T and requires c small enough that h2 h4² + h2 h4² h7² + 2 h5 h7² < 1 (Supp (51)); the advertised O(ε^-2) rate hides this and other problem-dependent constants.
  • exploration noise standard deviation σ = σ = 1 in experiments; otherwise any fixed σ > 0
    The Gaussian policy (6) needs σ > 0 for exploration and to make σ_min(D~_K) strictly positive in Lemma 10; the theory holds for any fixed positive σ, and σ = 1 is used in the experiments.
assumptions (6)
  • standard math Gaussian quadratic-form moment identities and chi-square product moment bounds (Nagar 1959, Magnus 1978, Joarder-Omar 2011).
    Used in Lemmas 4 and 6 to evaluate E[φφ^T], E[φφ'^T] and to bound feature and reward moments over the stationary distribution.
  • standard math svec/smat and symmetric Kronecker product algebra (Schacke 2004), including the (A⊗s B)(C⊗s D) expansion.
    Used to derive A_K = 2(D~_K ⊗s D~_K)(I − L^T ⊗s L^T) in Lemma 4 and to express the Q-function in the critic feature basis.
  • standard math Fact ρ(L) = ρ(A − BK) for the lifted matrix L = [[A,B],[−KA,−KB]].
    Converts stability of K into ρ(L) < 1, which is needed for invertibility of A_K and the uniform lower bound μ in Lemma 10.
  • domain assumption Trajectory-level Assumption 1: every iterate K_t satisfies ||K_t|| bounded and ρ(A − BK_t) ≤ λ < 1, although Algorithm 1 does not project the actor.
    All uniform bounds (Lemmas 5-11) and the gradient-domination step require the spectral radius of A − BK_t to stay uniformly below 1; the proof supplies only a one-step perturbation bound (Supp eq. (34)), so the induction that the unprojected actor stays in the stable set is not closed. The authors note the missing projection and defer it.
  • domain assumption Stationary-distribution sampling oracle (Algorithm 1, Line 3): each iteration draws x_t directly from ρ_{K_t}.
    The TD feature matrices A_K, b_K in (20) are defined under ρ_K; the O(ε^-2) sample count does not include mixing or burn-in costs.
  • domain assumption LQR model assumptions: (A,B) stabilizable, (A,Q^{1/2}) observable, D0 ≻ 0, Q,R symmetric positive definite (problem (4)).
    These guarantee the unique global optimum K* via the ARE and the gradient-domination property (Lemma 13); standard in the LQR RL literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Global Optimality of Single-Timescale Actor-Critic under Continuous State-Action Space: A Study on Linear Quadratic Regulator." pith.science (2026). https://pith.science/paper/46SLE6XL

@misc{pith2026250501041,
  author       = {Pith},
  title        = {Pith review of: Global Optimality of Single-Timescale Actor-Critic under Continuous State-Action Space: A Study on Linear Quadratic Regulator},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/46SLE6XL}},
  note         = {Machine review of arXiv:2505.01041}
}
read the original abstract

Actor-critic methods have achieved state-of-the-art performance in various challenging tasks. However, theoretical understandings of their performance remain elusive and challenging. Existing studies mostly focus on practically uncommon variants such as double-loop or two-timescale stepsize actor-critic algorithms for simplicity. These results certify local convergence on finite state- or action-space only. We push the boundary to investigate the classic single-sample single-timescale actor-critic on continuous (infinite) state-action space, where we employ the canonical linear quadratic regulator (LQR) problem as a case study. We show that the popular single-timescale actor-critic can attain an epsilon-optimal solution with an order of epsilon to -2 sample complexity for solving LQR on the demanding continuous state-action space. Our work provides new insights into the performance of single-timescale actor-critic, which further bridges the gap between theory and practice.

Figures

Figures reproduced from arXiv: 2505.01041 by the authors.

Figure 1
Figure 1. (a) Learning results of Algorithm 1. In the fig￾ure, the cost error refers to 1 T PT −1 t=0 (ηt − J(Kt))2 , Critic er￾ror refers to 1 T PT −1 t=0 ∥ωt − ω ∗ Kt ∥ 2 , and the Actor error refers to 1 T PT −1 t=0 [J(Kt) − J(K∗ )], corresponding to the conclusion in Theorem 1 empirically. (b) Comparison of Algorithm 1 with two other algorithms. The ac￾tor norm error refers to ∥K − K∗ ∥F . In this figure, the solid lines … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 28 canonical work pages

  1. [1]

    Optimal control: linear quadratic methods

    Brian DO Anderson and John B Moore. Optimal control: linear quadratic methods . Courier Corporation, 2007

    work page 2007

  2. [2]

    Analysis of a target-based actor-critic algorithm with linear function approximation

    Anas Barakat, Pascal Bianchi, and Julien Lehmann. Analysis of a target-based actor-critic algorithm with linear function approximation. In International Conference on Artificial Intelligence and Statistics , pages 991--1040. PMLR, 2022

    work page 2022

  3. [3]

    Natural actor--critic algorithms

    Shalabh Bhatnagar, Richard S Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor--critic algorithms. Automatica , 45(11):2471--2482, 2009

    work page 2009

  4. [4]

    Adaptive linear quadratic control using policy iteration

    Steven J Bradtke, B Erik Ydstie, and Andrew G Barto. Adaptive linear quadratic control using policy iteration. In Proceedings of 1994 American Control Conference-ACC'94 , volume 3, pages 3475--3479. IEEE, 1994

    work page 1994

  5. [5]

    A convergent online single time scale actor critic algorithm

    Dotan Di Castro and Ron Meir. A convergent online single time scale actor critic algorithm. The Journal of Machine Learning Research , 11:367--410, 2010

    work page 2010

  6. [6]

    Finite-time analysis of single-timescale actor-critic

    Xuyang Chen and Lin Zhao. Finite-time analysis of single-timescale actor-critic. arXiv preprint arXiv:2210.09921 , 2022

    work page Pith review arXiv 2022

  7. [7]

    Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems

    Tianyi Chen, Yuejiao Sun, and Wotao Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. Advances in Neural Information Processing Systems , 34:25294--25307, 2021

    work page 2021

  8. [8]

    Learning linear-quadratic regulators efficiently with only T regret

    Alon Cohen, Tomer Koren, and Yishay Mansour. Learning linear-quadratic regulators efficiently with only T regret. In International Conference on Machine Learning , pages 1300--1309. PMLR, 2019

    work page 2019

Show all 39 references
  1. [9]

    Regret bounds for robust adaptive control of the linear quadratic regulator

    Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu. Regret bounds for robust adaptive control of the linear quadratic regulator. Advances in Neural Information Processing Systems , 31, 2018

    2018

  2. [10]

    On the sample complexity of the linear quadratic regulator

    Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu. On the sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics , 20(4):633--679, 2020

    2020

  3. [11]

    Optimization landscape of policy gradient methods for discrete-time static output feedback

    Jingliang Duan, Jie Li, Xuyang Chen, Kai Zhao, Shengbo Eben Li, and Lin Zhao. Optimization landscape of policy gradient methods for discrete-time static output feedback. IEEE Transactions on Cybernetics , 2023

    2023

  4. [12]

    Global convergence of policy gradient methods for the linear quadratic regulator

    Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In International Conference on Machine Learning , pages 1467--1476. PMLR, 2018

    2018

  5. [13]

    On statistical characteristics of the product of two correlated chi-square variables

    Anwar H Joarder and M Hafidz Omar. On statistical characteristics of the product of two correlated chi-square variables. Journal of Applied Statistical Science , 19(4):89--101, 2011

    2011

  6. [14]

    A natural policy gradient

    Sham M Kakade. A natural policy gradient. Advances in neural information processing systems , 14, 2001

    2001

  7. [15]

    Two time-scale stochastic approximation with controlled markov noise and off-policy temporal-difference learning

    Prasenjit Karmakar and Shalabh Bhatnagar. Two time-scale stochastic approximation with controlled markov noise and off-policy temporal-difference learning. Mathematics of Operations Research , 43(1):130--151, 2018

    2018

  8. [16]

    Actor-critic algorithms

    Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems , 12, 1999

    1999

  9. [17]

    Finite-time analysis of approximate policy iteration for the linear quadratic regulator

    Karl Krauth, Stephen Tu, and Benjamin Recht. Finite-time analysis of approximate policy iteration for the linear quadratic regulator. Advances in Neural Information Processing Systems , 32, 2019

    2019

  10. [18]

    On the sample complexity of actor-critic method for reinforcement learning with function approximation

    Harshat Kumar, Alec Koppel, and Alejandro Ribeiro. On the sample complexity of actor-critic method for reinforcement learning with function approximation. arXiv preprint arXiv:1910.08412 , 2019

    1910 arXiv

  11. [19]

    Deep learning

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature , 521(7553):436--444, 2015

    2015

  12. [20]

    The moments of products of quadratic forms in normal variables

    Jan R Magnus. The moments of products of quadratic forms in normal variables. Statistica Neerlandica , 32(4):201--210, 1978

    1978

  13. [21]

    Derivative-free methods for policy optimization: Guarantees for linear quadratic systems

    Dhruv Malik, Ashwin Pananjady, Kush Bhatia, Koulik Khamaru, Peter Bartlett, and Martin Wainwright. Derivative-free methods for policy optimization: Guarantees for linear quadratic systems. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2...

    2019

  14. [22]

    Certainty equivalence is efficient for linear quadratic control

    Horia Mania, Stephen Tu, and Benjamin Recht. Certainty equivalence is efficient for linear quadratic control. Advances in Neural Information Processing Systems , 32, 2019

    2019

  15. [23]

    Asynchronous methods for deep reinforcement learning

    Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928--1937. PMLR, 2016

    1928

  16. [24]

    The bias and moment matrix of the general k-class estimators of the parameters in simultaneous equations

    Anirudh L Nagar. The bias and moment matrix of the general k-class estimators of the parameters in simultaneous equations. Econometrica: Journal of the Econometric Society , pages 575--595, 1959

    1959

  17. [25]

    A small gain analysis of single timescale actor critic

    Alex Olshevsky and Bahman Gharesifard. A small gain analysis of single timescale actor critic. SIAM Journal on Control and Optimization , 61(2):980--1007, 2023

    2023

  18. [26]

    Linear models in statistics

    Alvin C Rencher and G Bruce Schaalje. Linear models in statistics . John Wiley & Sons, 2008

    2008

  19. [27]

    On the kronecker product

    Kathrin Schacke. On the kronecker product. Master's thesis, University of Waterloo , 2004

    2004

  20. [28]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889--1897. PMLR, 2015

    2015

  21. [29]

    Mastering the game of go without human knowledge

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354--359, 2017

    2017

  22. [30]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018

    2018

  23. [31]

    Policy gradient methods for reinforcement learning with function approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999

    1999

  24. [32]

    Least-squares temporal difference learning for the linear quadratic regulator

    Stephen Tu and Benjamin Recht. Least-squares temporal difference learning for the linear quadratic regulator. In International Conference on Machine Learning , pages 5005--5014. PMLR, 2018

    2018

  25. [33]

    Neural policy gradient methods: Global optimality and rates of convergence

    Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. arXiv preprint arXiv:1909.01150 , 2019

    1909 arXiv

  26. [34]

    A finite-time analysis of two time-scale actor-critic methods

    Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems , 33:17617--17628, 2020

    2020

  27. [35]

    Non-asymptotic convergence analysis of two time-scale (natural) actor-critic algorithms

    Tengyu Xu, Zhe Wang, and Yingbin Liang. Non-asymptotic convergence analysis of two time-scale (natural) actor-critic algorithms. arXiv preprint arXiv:2005.03557 , 2020

    2005 arXiv

  28. [36]

    Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost

    Zhuoran Yang, Yongxin Chen, Mingyi Hong, and Zhaoran Wang. Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost. Advances in neural information processing systems , 32, 2019

    2019

  29. [37]

    Provably convergent two-timescale off-policy actor-critic with function approximation

    Shangtong Zhang, Bo Liu, Hengshuai Yao, and Shimon Whiteson. Provably convergent two-timescale off-policy actor-critic with function approximation. In International Conference on Machine Learning , pages 11204--11213. PMLR, 2020

    2020

  30. [38]

    Single timescale actor-critic method to solve the linear quadratic regulator with convergence guarantees

    Mo Zhou and Jianfeng Lu. Single timescale actor-critic method to solve the linear quadratic regulator with convergence guarantees. Journal of Machine Learning Research , 24(222):1--34, 2023

    2023

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.