Pith. sign in

REVIEW 2 major objections 5 minor 75 references

Decoupled Functional Central Limit Theorems for Two-Time-Scale Stochastic Approximation

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Decoupled functional CLTs give two Ornstein-Uhlenbeck limits for two-time-scale stochastic approximation.

desk verdict A credible, genuinely new decoupled FCLT for two-time-scale SA, with a smart auxiliary-sequence device, but the main theorem leans on non-asymptotic rate bounds imported from the authors' earlier paper under stronger assumptions than those stated. read the letter →

arxiv 2412.17070 v5 pith:K7PPXIPG submitted 2024-12-22 math.PR math.OCstat.ML

Yuze Han , Xiang Li , Jiadong Liang , Zhihua Zhang This is my paper · ORCID
classification math.PRmath.OCstat.ML MSC 60F1760F0562L2060J60
keywords two-time-scalestochasticapproximationfunctionalcentrallimittheoremOrnstein-Uhlenbeckprocessdecoupledconvergencemartingaleproblemweaktightness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Two-time-scale stochastic approximation updates a fast and a slow iterate with different step sizes, each influencing the other. This paper proves that, after rescaling each error term by the square root of its own step size, the fast trajectory converges weakly to a stationary Ornstein-Uhlenbeck process with drift $-B_1$ and diffusion $\Sigma_\xi$, and the slow trajectory converges weakly to a stationary Ornstein-Uhlenbeck process with drift $-(B_3-\tilde{\beta}I/2)$ and diffusion $\tilde{\Sigma}_\psi$. The limiting dynamics on each time scale has the same form as standard stochastic approximation run with that step size alone; the only trace of the coupling between the two scales is in the coefficients of the slow limit. A reader should care because trajectory-level limits, not just single-point central limit theorems, describe how two-time-scale algorithms fluctuate over time and support inference for methods such as actor-critic and gradient temporal-difference learning.

What carries the argument

The central object is an auxiliary sequence $\check z_n=\check y_n-\sqrt{\kappa_{n-1}}B_2B_1^{-1}\check x_n$, where $\check x_n=(x_n-H(y_n))/\sqrt{\alpha_{n-1}}$, $\check y_n=(y_n-y_\star)/\sqrt{\beta_{n-1}}$, and $\kappa_n=\beta_n/\alpha_n$. Subtracting this correction removes the dominant $O(\sqrt{\beta_n\alpha_n})$ influence of the fast iterate on the slow recursion, so $\check z_n$ evolves like a standard stochastic approximation iterate. The proof builds one-step recursions for $\check x_n$, $\check y_n$, and $\check z_n$ with explicit residual bounds, establishes tightness of the continuous-time trajectories formed by linear interpolation, and then uses the martingale problem approach to identify the unique limiting Ornstein-Uhlenbeck processes.

What would settle it

Run two-time-scale SA on a linear problem with known $B_1,B_2,B_3$ and known noise covariances, using step sizes $\alpha_n=(n+1)^{-a}$ and $\beta_n=(n+1)^{-b}$ with $b/a<1$, so Assumption 3.4(v) fails. If the rescaled slow trajectory still converges weakly to the stationary OU solution of (16), the condition is not necessary; if convergence fails or the limit changes, it is load-bearing. A second check is to estimate $E\|\hat x_n\|^4$ directly: if it is not $O(\alpha_n^2)$, Assumption 4.1 fails and the residual terms $R_n^y$ and $R_n^z$ in Lemmas 5.1–5.2 no longer vanish at the required orders.

Watch

Extended reading notes

Core claim

Theorem 4.1 shows that, under Assumptions 3.1–3.6 and 4.1, the linearly interpolated rescaled fast trajectory $\bar X_n(\cdot)$ converges weakly to the stationary solution of $dX(t)=-B_1X(t)\,dt+\Sigma_\xi^{1/2}\,dW_{d_x}(t)$, and the rescaled slow trajectory $\bar Y_n(\cdot)$ converges weakly to the stationary solution of $dY(t)=-(B_3-\tilde{\beta}I/2)Y(t)\,dt+\tilde{\Sigma}_\psi^{1/2}\,dW_{d_y}(t)$, where $\tilde{\Sigma}_\psi=\Sigma_\psi-B_2B_1^{-1}\Sigma_{\xi,\psi}-\Sigma_{\xi,\psi}^{\top}B_1^{-\top}B_2^{\top}+B_2B_1^{-1}\Sigma_\xi B_1^{-\top}B_2^{\top}$. Equivalently, each rescaled iterate converges in distribution to the invariant Gaussian distribution $N(0,\Sigma_x)$ or $N(0,\Sigma_y)$ solving the Lyapunov equations (15) and (18). This is the first decoupled functional central limit theorem for two-time-scale stochastic approximation in which each time scale is rescaled by its own step size, rather than by a common factor.

Load-bearing premise

The paper assumes as given the non-asymptotic rates $E\|\hat x_n\|^2=O(\alpha_n)$, $E\|\hat y_n\|^2=O(\beta_n)$, and $E\|\hat x_n\|^4+E\|\hat y_n\|^4=O(\alpha_n^2)$ from an earlier paper; it does not prove these rates from Assumptions 3.1–3.6, and every residual bound collapses if they fail.

Editorial extensions

If this is right

  • Corollary 4.1 recovers the classical single-point central limit theorems: $\alpha_n^{-1/2}(x_n-H(y_n)) \Rightarrow N(0,\Sigma_x)$ and $\beta_n^{-1/2}(y_n-y_\star) \Rightarrow N(0,\Sigma_y)$.
  • For algorithms such as SGD with Polyak-Ruppert averaging, normalized stochastic heavy ball, GTD2, and TDC, the trajectory-level limit is the stationary Ornstein-Uhlenbeck process (14) or (16), so asymptotic fluctuations over finite time horizons are characterized, not just the marginal distributions.
  • The fast-scale limit is independent of the slow scale, while the slow-scale limit depends on the fast scale only through the coefficient matrices $B_1,B_2$ and the noise cross-covariance, plus the step-size constant $\tilde{\beta}$ when $\beta_n$ decays like $1/n$.
  • When $\beta_n \asymp 1/n$, the initial slow step size enters the limiting slow drift through $\tilde{\beta}=\beta_0^{-1}$, so the asymptotic slow dynamics depends on the step-size schedule in a concrete, testable way.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A next step the paper does not take is to use the auxiliary sequence $\check z_n$ as a debiased slow iterate for constructing confidence intervals or tests in two-time-scale algorithms; the FCLT for $\check z_n$ makes its asymptotic distribution explicit.
  • If Assumption 4.1's non-asymptotic rates are established under Markovian or state-dependent noise, the same tightness-and-martingale skeleton should yield decoupled FCLTs there; the residual bounds in Lemmas 5.1 and 5.2 are the main place the noise model enters.
  • The coefficient-only coupling suggests that in actor-critic and TD-style algorithms, the fast auxiliary variable and the slow parameter have asymptotically independent Gaussian trajectory fluctuations on their own timescales, which could be checked empirically by comparing simulated rescaled paths to the predicted OU limits.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper establishes decoupled functional central limit theorems for two-time-scale stochastic approximation. The authors construct continuous-time processes by rescaling the fast error x_n - H(y_n) by sqrt(alpha_n) and the slow error y_n - y* by sqrt(beta_n), and prove, under Assumptions 3.1-3.6 together with an imported non-asymptotic rate assumption (Assumption 4.1), that these processes converge weakly to stationary Ornstein-Uhlenbeck processes with drift and diffusion coefficients given in (14)-(17). The proof proceeds through one-step recursions, an auxiliary sequence that removes the dominant fast-scale influence on the slow scale, tightness of the trajectory sequences, and the martingale problem approach.

Significance. If the result is valid, it is a meaningful advance over earlier single-point decoupled convergence results and over the prior FCLT of Faizal and Borkar [18], because it captures a stochastic limit on each time scale and identifies the coupling only through the coefficients of the limiting SDEs. The paper contains detailed appendix proofs of the recursions, tightness, and approximate martingale problem, and the limiting SDEs are explicit, falsifiable predictions with no fitted parameters. The main reservation is that the central theorem is conditional on Assumption 4.1, whose formal verification in Appendix B.1 requires strong monotonicity conditions and step-size restrictions that do not appear in the main hypotheses of Theorem 4.1.

major comments (2)
  1. [Section 4.1 and Appendix B.1] The central claim of Theorem 4.1 is conditional on Assumption 4.1, which imports the rate bounds E||xhat_n||^2 = O(alpha_n), E||yhat_n||^2 = O(beta_n), and E||xhat_n||^4 + E||yhat_n||^4 = O(alpha_n^2) from [23, Theorem 3.1]. These bounds are never derived from Assumptions 3.1-3.6; the only formal statement, Proposition B.1, requires global strong monotonicity of F and of G(H(y),y), and it obtains the second-moment rates only under the additional step-size restriction b/a <= 1 + delta_F/2. Neither the strong monotonicity nor the step-size restriction appears in Theorem 4.1. Since every residual control in Lemmas 5.1 and 5.2, and therefore the tightness Lemma 5.3 and the approximate martingale problem Lemma 5.4, is built on Assumption 4.1, a system satisfying Assumptions 3.1-3.6 but not the stronger rate conditions could have residual terms R^x_n, R^y_n, R^z_n that do not vanish at the required orders, so the limiting SDEs in (14)-(16) need not follow. Please either prove Assumption 4.1 under the main assumptions or add the strong monotonicity and step-size conditions to the hypotheses of Theorem 4.1 and discuss the impact on Examples 4.1-4.3.
  2. [Section 5.1] The residual properties stated in Lemmas 5.1 and 5.2 are used for p in the interval (2, 4/(1+(delta_H vee delta_F vee delta_G)/2)], but the moment computations in the proof of Lemma 5.1 in Appendix C.1 are only sketched. In particular, the bounds for E||R^y_n||^p and E||R^x_n||^p require a careful interpolation between the second-moment and fourth-moment rates of Assumption 4.1 under the step-size condition (v) of Assumption 3.4; the text says this follows from Young and Jensen inequalities but does not display the full argument. Please provide the complete interpolation step or state the needed moment hypotheses explicitly, so that the residual bounds are fully verifiable.
minor comments (5)
  1. [Section 1, Related Work] The phrase 'depend sorely on their respective step sizes' should read 'depend solely on their respective step sizes'.
  2. [Lemma 5.4(ii)] In the displayed decomposition for the slow scale, the generator term is written as A_y f(bar{Z}_n(s)); since the test function is g, this should be A_y g(bar{Z}_n(s)).
  3. [Section C.4, Proof of Theorem 4.1(i)] In Step 1 and Step 2 of the proof, the text refers to 'the SDE in (16)' when the generator A_x corresponds to equation (14); the same substitution is needed wherever the invariant distribution and semigroup of the fast process are discussed. Also, near the end of Step 2, 'Letting epsilon -> infinity' should be 'Letting epsilon -> 0'.
  4. [Equations (16)-(17)] The diffusion matrix of the slow SDE appears in inconsistent notation: tilde Sigma psi, ~Sigma psi, and Sigma psi are all used. Please unify the notation.
  5. [Section 4.1] Proposition 4.1 is labelled 'informal' and Assumption 4.1 refers back to it; to help the reader, point directly to the formal Proposition B.1 in the main body when introducing Assumption 4.1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the rate bounds are a transparent assumption, and the OU limits are derived from the recursions rather than assumed.

full rationale

Theorem 4.1 is explicitly conditional: it assumes Assumptions 3.1–3.6 and 4.1, and then proves weak convergence of the rescaled trajectories. Assumption 4.1 imports non-asymptotic rate bounds from the authors' prior work [23], but it is stated as an assumption, not silently derived, and the paper discloses in Proposition B.1 that obtaining these rates from primitives requires additional strong monotonicity and step-size conditions. That is a hypothesis gap for Theorem 4.1 as stated, but not a circular reduction. The limiting OU processes in (14) and (16) are not used to define the one-step recursions: Lemma 5.1 derives the linearized recursions (24)–(25) with coefficients B1, B3 and noise terms; Lemma 5.2 introduces the auxiliary sequence (26), whose correction coefficient B2 B1^{-1} is algebraically forced by the coupling term in (25); Lemma 5.4 then proves that the generators (30)–(31) are approximately solved, with tilde_Sigma_psi computed as the covariance of psi - B2 B1^{-1} xi in (17). The matrices B1, B2, B3 and the noise covariances come from the local linearity assumptions and the noise assumptions, not from any fitted parameter or from the desired OU limit. No step renames a fitted quantity as a prediction, and no uniqueness theorem of the authors is invoked to force the form of the limit. The central FCLT derivation is therefore self-contained given its stated assumptions; the only self-referential element is the borrowed rate bound, which is a legitimate external theorem and not circuitous here.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central theorem introduces no fitted parameters and no new physical entities. It relies on the authors' previous rate theorem (Assumption 4.1) and on a standard SA regularity framework, so the axiom ledger is dominated by domain assumptions rather than ad hoc constructs.

assumptions (3)
  • domain assumption Assumptions 3.1-3.6: Lipschitz and local linearity conditions, step-size conditions, Hurwitz matrices, and martingale-difference noise with fourth moments.
    These are the standard regularity conditions under which two-time-scale SA converges and residual terms can be controlled.
  • ad hoc to paper Assumption 4.1: non-asymptotic decoupled convergence rates from [23, Theorem 3.1].
    The paper assumes E||xhat||^2=O(alpha), E||yhat||^2=O(beta), and the fourth-moment bound O(alpha^2) instead of proving them; the formal statement (Proposition B.1) needs additional strong monotonicity conditions not listed in Assumptions 3.1-3.6.
  • ad hoc to paper Strong monotonicity conditions in Proposition B.1.
    These are needed to guarantee Assumption 4.1 but are absent from the main theorem's hypothesis set, so the theorem is conditional on an external result with stronger hypotheses.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decoupled Functional Central Limit Theorems for Two-Time-Scale Stochastic Approximation." pith.science (2026). https://pith.science/paper/K7PPXIPG

@misc{pith2026241217070,
  author       = {Pith},
  title        = {Pith review of: Decoupled Functional Central Limit Theorems for Two-Time-Scale Stochastic Approximation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K7PPXIPG}},
  note         = {Machine review of arXiv:2412.17070}
}
read the original abstract

In two-time-scale stochastic approximation (SA), two iterates are updated at different rates, governed by distinct step sizes, with each update influencing the other. Previous studies have demonstrated that the convergence rates of the error terms for these updates depend solely on their respective step sizes, a property known as decoupled convergence. However, a functional version of this decoupled convergence has not been explored. Our work fills this gap by establishing decoupled functional central limit theorems for two-time-scale SA, offering a more precise characterization of its asymptotic behavior. Our results show that, on each time scale, the limiting dynamics has the same form as in standard SA, and the coupling between the two iterates enters the limit only through the associated coefficients. To achieve these results, we leverage the martingale problem approach and establish tightness as a crucial intermediate step. Furthermore, to address the interdependence between different time scales, we introduce an innovative auxiliary sequence to eliminate the primary influence of the fast-time-scale update on the slow-time-scale update.

Figures

Figures reproduced from arXiv: 2412.17070 by the authors.

Figure 1
Figure 1. Construction of X¯ n(·) and Y¯ n(·) (i) The stochastic process X¯ n(·) defined in (12) converges weakly to the stationary solution of the following SDE dX(t) = −B1X(t)dt + Σ1/2 ξ dWdx (t). (14) The rescaled iterate xˇn defined in (11) converges weakly to the invariant distribution of (14), i.e., N (0, Σx), where Σx satisfies the following Lyapunov equation B1Σx + ΣxB1 = Σξ. (15) (ii) The stochastic process Y¯ n(·) d… view at source ↗
Figure 2
Figure 2. Illustration for the framework of proof. [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Illustration for the last step of proof. [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 57 canonical work pages

  1. [18]

    Functional centr al limit theorem for two timescale stochastic approximation

    Fathima Zarin Faizal and Vivek Borkar. Functional centr al limit theorem for two timescale stochastic approximation. arXiv preprint arXiv:2306.05723 , 2023

    arXiv 2023

  2. [1]

    Adaptive algorithms and stochastic approximations, volume 22

    Albert Benveniste, Michel Métivier, and Pierre Priouret . Adaptive algorithms and stochastic approximations, volume 22. Springer Science & Business Media, 2012

    work page 2012

  3. [2]

    Convergence of probability measures

    Patrick Billingsley. Convergence of probability measures . 1968

    work page 1968

  4. [3]

    Convergence of probability measures

    Patrick Billingsley. Convergence of probability measures . John Wiley & Sons, 2013

    2013

  5. [4]

    Li mit theorems for stochastic gradient descent with infinite variance

    Jose Blanchet, Aleksandar Mijatović, and Wenhao Yang. Li mit theorems for stochastic gradient descent with infinite variance. arXiv preprint arXiv:2410.16340 , 2024

    arXiv 2024

  6. [5]

    The ODE method for asymptotic statistics in stochastic approxi mation and reinforcement learning

    Vivek Borkar, Shuhang Chen, Adithya Devraj, Ioannis Kont oyiannis, and Sean Meyn. The ODE method for asymptotic statistics in stochastic approxi mation and reinforcement learning. arXiv preprint arXiv:2110.14427 , 2021

    arXiv 2021

  7. [6]

    Stochastic approximation with two time sc ales

    Vivek S Borkar. Stochastic approximation with two time sc ales. Systems & Control Letters , 29(5):291–294, 1997

    work page 1997

  8. [7]

    Stochastic approximation: A dynamical systems viewpoint , volume 48

    Vivek S Borkar. Stochastic approximation: A dynamical systems viewpoint , volume 48. Springer, 2009

    work page 2009

Show all 75 references
  1. [8]

    The actor-critic al gorithm as multi-time-scale stochastic approximation

    Vivek S Borkar and Vijaymohan R Konda. The actor-critic al gorithm as multi-time-scale stochastic approximation. Sadhana, 22:525–543, 1997

    1997

  2. [9]

    Sharp inequalities for martingales and stochastic integrals

    Donald L Burkholder. Sharp inequalities for martingales and stochastic integrals. Astérisque, 157(158):75–94, 1988

    1988

  3. [10]

    A generalization of reg ularized dual averaging and its dynamics

    Shih-Kang Chao and Guang Cheng. A generalization of reg ularized dual averaging and its dynamics. arXiv preprint arXiv:1909.10072 , 2019

    1909 arXiv

  4. [11]

    Closing the gap : Tighter analysis of alternating stochastic gradient methods for bilevel problems

    Tianyi Chen, Yuejiao Sun, and Wotao Yin. Closing the gap : Tighter analysis of alternating stochastic gradient methods for bilevel problems. In Advances in Neural Information Processing Systems, volume 34, pages 25294–25307, 2021

    2021

  5. [12]

    Online stati stical inference for stochastic optimization via Kiefer-Wolfowitz methods

    Xi Chen, Zehua Lai, He Li, and Yichen Zhang. Online stati stical inference for stochastic optimization via Kiefer-Wolfowitz methods. arXiv e-prints , pages arXiv–2102, 2021

    2021

  6. [13]

    A tale of tw o-timescale reinforcement learn- ing with the tightest finite-time bound

    Gal Dalal, Balazs Szorenyi, and Gugan Thoppe. A tale of tw o-timescale reinforcement learn- ing with the tightest finite-time bound. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3701–3708, 2020

    2020

  7. [14]

    Finite sample analysis of two- timescale stochastic approximation with applications to r einforcement learning

    Gal Dalal, Gugan Thoppe, Balázs Szörényi, and Shie Manno r. Finite sample analysis of two- timescale stochastic approximation with applications to r einforcement learning. In Conference On Learning Theory , pages 1199–1233. PMLR, 2018. 41

    2018

  8. [15]

    Polic y evaluation with temporal dif- ferences: A survey and comparison

    Christoph Dann, Gerhard Neumann, and Jan Peters. Polic y evaluation with temporal dif- ferences: A survey and comparison. Journal of Machine Learning Research , 15(24):809–883, 2014

    2014

  9. [16]

    Nonlinear two-time-scale stochastic app roximation convergence and finite-time performance

    Thinh T Doan. Nonlinear two-time-scale stochastic app roximation convergence and finite-time performance. IEEE Transactions on Automatic Control , 2022

    2022

  10. [17]

    Markov processes: characterization and convergence

    Stewart N Ethier and Thomas G Kurtz. Markov processes: characterization and convergence . John Wiley & Sons, 2009

    2009

  11. [19]

    S tochastic heavy ball

    Sébastien Gadat, Fabien Panloup, and Sofiane Saadane. S tochastic heavy ball. Electronic Journal of Statistics , 12:461–529, 2018

    2018

  12. [20]

    Approximation methods f or bilevel programming

    Saeed Ghadimi and Mengdi Wang. Approximation methods f or bilevel programming. arXiv preprint arXiv:1802.02246 , 2018

    2018 arXiv

  13. [21]

    Understanding the role of mo- mentum in stochastic gradient methods

    Igor Gitman, Hunter Lang, Pengchuan Zhang, and Lin Xiao . Understanding the role of mo- mentum in stochastic gradient methods. In Advances in neural information processing systems , volume 32, 2019

    2019

  14. [22]

    A. M. Gupal and L. T. Bazhenov. A stochastic analog of the c onjugate gradient method. Cybernetics, 8:138–140, 1972

    1972

  15. [23]

    Finite-time decou pled convergence in nonlinear two- time-scale stochastic approximation

    Yuze Han, Xiang Li, and Zhihua Zhang. Finite-time decou pled convergence in nonlinear two- time-scale stochastic approximation. arXiv preprint arXiv:2401.03893 , 2024

    2024 arXiv

  16. [24]

    Tight finite time bounds of two-time-scale linear stochastic approximation with ma rkovian noise

    Shaan Ul Haque, Sajad Khodadadian, and Siva Theja Magul uri. Tight finite time bounds of two-time-scale linear stochastic approximation with ma rkovian noise. arXiv preprint arXiv:2401.00364, 2023

    2023 arXiv

  17. [25]

    A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor- critic

    Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang . A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor- critic. SIAM Journal on Optimization , 33(1):147–180, 2023

    2023

  18. [26]

    Central limit theorem fo r two-timescale stochastic approxima- tion with markovian noise: Theory and applications

    Jie Hu, Vishwaraj Doshi, et al. Central limit theorem fo r two-timescale stochastic approxima- tion with markovian noise: Theory and applications. In International Conference on Artificial Intelligence and Statistics , pages 1477–1485. PMLR, 2024

    2024

  19. [27]

    Finite time analysis of linear two-timescale stochastic approximatio n with Markovian noise

    Maxim Kaledin, Eric Moulines, Alexey Naumov, Vladisla v Tadic, and Hoi-To Wai. Finite time analysis of linear two-timescale stochastic approximatio n with Markovian noise. In Conference on Learning Theory , pages 2144–2203. PMLR, 2020

    2020

  20. [28]

    Two time-sca le stochastic approximation with controlled Markov noise and off-policy temporal-difference learning

    Prasenjit Karmakar and Shalabh Bhatnagar. Two time-sca le stochastic approximation with controlled Markov noise and off-policy temporal-difference learning. Mathematics of Operations Research, 43(1):130–151, 2018

    2018

  21. [29]

    Limit behavior of two-time-sc ale diffusions revisited

    RZ Khasminskii and G Yin. Limit behavior of two-time-sc ale diffusions revisited. Journal of Differential Equations , 212(1):85–113, 2005. 42

    2005

  22. [30]

    R. Z. Khasminskij. On the principle of averaging the ito v’s stochastic differential equations. Kybernetika, 4:260–279, 1968

    1968

  23. [31]

    Applications of singular perturbat ion techniques to control problems

    Petar V Kokotović. Applications of singular perturbat ion techniques to control problems. SIAM review, 26(4):501–550, 1984

    1984

  24. [32]

    On actor-critic alg orithms

    Vijay R Konda and John N Tsitsiklis. On actor-critic alg orithms. SIAM journal on Control and Optimization , 42(4):1143–1166, 2003

    2003

  25. [33]

    Convergence rate of linear two-time-scale stochastic approximation

    Vijay R Konda and John N Tsitsiklis. Convergence rate of linear two-time-scale stochastic approximation. The Annals of Applied Probability , 14(2):796–819, 2004

    2004

  26. [34]

    Stochastic approximation and recursive algorithms and applications, volume 35

    Harold Kushner and G George Yin. Stochastic approximation and recursive algorithms and applications, volume 35. Springer Science & Business Media, 2003

    2003

  27. [35]

    Harold J. Kushner. Almost optimal controls for wideban d noise driven systems. In Wendell Fleming and Pierre-Louis Lions, editors, Stochastic Differential Systems, Stochastic Control Theory and Applications , pages 255–273, New York, NY, 1988. Springer New York

    1988

  28. [36]

    Stochastic approxim ation with averaging of the iterates: Optimal asymptotic rate of convergence for general process es

    Harold J Kushner and Jichuan Yang. Stochastic approxim ation with averaging of the iterates: Optimal asymptotic rate of convergence for general process es. SIAM Journal on Control and Optimization, 31(4):1045–1062, 1993

    1993

  29. [37]

    Two-timescale linear stochastic approximation: Constant stepsizes go a long way

    Jeongyeol Kwon, Luke Dotson, Yudong Chen, and Qiaomin X ie. Two-timescale linear stochastic approximation: Constant stepsizes go a long way. arXiv preprint arXiv:2410.13067 , 2024

    2024 arXiv

  30. [38]

    Fast and robust online inference with stochastic gradient descent via random scaling

    Sokbae Lee, Yuan Liao, Myung Hwan Seo, and Youngki Shin. Fast and robust online inference with stochastic gradient descent via random scaling. In the AAAI Conference on Artificial Intelligence, volume 36, pages 7381–7389, 2022

    2022

  31. [39]

    Fast inference for quantile regression with tens of millions of observations

    Sokbae Lee, Yuan Liao, Myung Hwan Seo, and Youngki Shin. Fast inference for quantile regression with tens of millions of observations. Available at SSRN 4263158 , 2022

    2022

  32. [40]

    High- probability sample complexities for policy evaluation wit h linear function approximation

    Gen Li, Weichen Wu, Yuejie Chi, Cong Ma, Alessandro Rina ldo, and Yuting Wei. High- probability sample complexities for policy evaluation wit h linear function approximation. IEEE Transactions on Information Theory , 2024

    2024

  33. [41]

    Statistical inferen ce in massive data sets

    Runze Li, Dennis KJ Lin, and Bing Li. Statistical inferen ce in massive data sets. Applied Stochastic Models in Business and Industry , 29(5):399–409, 2013

    2013

  34. [42]

    Statistical estimation and online inference via Local SGD

    Xiang Li, Jiadong Liang, Xiangyu Chang, and Zhihua Zhan g. Statistical estimation and online inference via Local SGD. In Conference on Learning Theory , pages 1613–1661. PMLR, 2022

    2022

  35. [43]

    Online stati stical inference for nonlinear stochastic approximation with Markovian data

    Xiang Li, Jiadong Liang, and Zhihua Zhang. Online stati stical inference for nonlinear stochastic approximation with Markovian data. arXiv preprint arXiv:2302.07690 , 2023

    2023 arXiv

  36. [44]

    A statistical analysis of Polyak- Ruppert averaged Q-learning

    Xiang Li, Wenhao Yang, Zhihua Zhang, and Michael I Jorda n. A statistical analysis of Polyak- Ruppert averaged Q-learning. In International Conference on Artificial Intelligence and St atis- tics, volume 206, 2023. 43

    2023

  37. [45]

    Asy mptotic behaviors and phase transitions in projected stochastic approximation: A jump diffusion approach

    Jiadong Liang, Yuze Han, Xiang Li, and Zhihua Zhang. Asy mptotic behaviors and phase transitions in projected stochastic approximation: A jump diffusion approach. arXiv preprint arXiv:2304.12953, 2023

    2023 arXiv

  38. [46]

    Convergent temporal-difference learning with arbi trary smooth function approximation

    Hamid Maei, Csaba Szepesvari, Shalabh Bhatnagar, Doina Precup, David Silver, and Richard S Sutton. Convergent temporal-difference learning with arbi trary smooth function approximation. In Advances in neural information processing systems , volume 22, 2009

    2009

  39. [47]

    Gradient temporal-difference learning algorithms

    Hamid Reza Maei. Gradient temporal-difference learning algorithms . PhD thesis, University of Alberta, 2011

    2011

  40. [48]

    Ergodicity for s des and approximations: locally lips- chitz vector fields and degenerate noise

    JC Mattingly, AM Stuart, and DJ Higham. Ergodicity for s des and approximations: locally lips- chitz vector fields and degenerate noise. Stochastic Processes and their Applications, 101(2):185– 232, 2002

    2002

  41. [49]

    Convergen ce rate and averaging of nonlinear two-time-scale stochastic approximation algorithms

    Abdelkader Mokkadem and Mariane Pelletier. Convergen ce rate and averaging of nonlinear two-time-scale stochastic approximation algorithms. Annals of Applied Probability , 16(3):1671– 1702, 2006

    2006

  42. [50]

    Optimal variance-reduced stochastic approximation i n Banach spaces

    Wenlong Mou, Koulik Khamaru, Martin J Wainwright, Pete r L Bartlett, and Michael I Jor- dan. Optimal variance-reduced stochastic approximation i n Banach spaces. arXiv preprint arXiv:2201.08518, 2022

    2022 arXiv

  43. [51]

    On linear stochastic approximation: Fine-grained Pol yak-Ruppert and non-asymptotic concentration

    Wenlong Mou, Chris Junchi Li, Martin J Wainwright, Pete r L Bartlett, and Michael I Jor- dan. On linear stochastic approximation: Fine-grained Pol yak-Ruppert and non-asymptotic concentration. In Conference on Learning Theory , pages 2947–2997. PMLR, 2020

    2020

  44. [52]

    Optimal oracle inequalities for projected fixed-point equations, with applications to p olicy evaluation

    Wenlong Mou, Ashwin Pananjady, and Martin J Wainwright . Optimal oracle inequalities for projected fixed-point equations, with applications to p olicy evaluation. Mathematics of Operations Research, 2022

    2022

  45. [53]

    Non-asymptotic analysi s of stochastic approximation algo- rithms for machine learning

    Eric Moulines and Francis Bach. Non-asymptotic analysi s of stochastic approximation algo- rithms for machine learning. In Advances in Neural Information Processing Systems , volume 24, 2011

    2011

  46. [54]

    Statistical inference with stochastic gradient algorithms

    Jeffrey Negrea, Jun Yang, Haoyue Feng, Daniel M Roy, and J onathan H Huggins. Statistical inference with stochastic gradient algorithms. arXiv preprint arXiv:2207.12395 , 2022

    2022 arXiv

  47. [55]

    Acceleration of st ochastic approximation by averaging

    Boris T Polyak and Anatoli B Juditsky. Acceleration of st ochastic approximation by averaging. SIAM journal on control and optimization , 30(4):838–855, 1992

    1992

  48. [56]

    Fokker-planck equation

    Hannes Risken. Fokker-planck equation. Springer, 1996

    1996

  49. [57]

    A stochastic approxi mation method

    Herbert Robbins and Sutton Monro. A stochastic approxi mation method. The annals of mathematical statistics , pages 400–407, 1951

    1951

  50. [58]

    Efficient estimations from a slowly conve rgent robbins-monro process

    David Ruppert. Efficient estimations from a slowly conve rgent robbins-monro process. Techni- cal report, Cornell University Operations Research and Ind ustrial Engineering, 1988

    1988

  51. [59]

    Two-timescale stochastic approximat ion for bilevel optimisation problems in continuous-time models

    Louis Sharrock. Two-timescale stochastic approximat ion for bilevel optimisation problems in continuous-time models. arXiv preprint arXiv:2206.06995 , 2022. 44

    2022 arXiv

  52. [60]

    Multidimensional diffusion processes , volume

    Daniel W Stroock and SR Srinivasa Varadhan. Multidimensional diffusion processes , volume

  53. [61]

    Fast gradient-descent meth ods for temporal-difference learning with linear function approximation

    Richard S Sutton, Hamid Reza Maei, Doina Precup, Shalab h Bhatnagar, David Silver, Csaba Szepesvári, and Eric Wiewiora. Fast gradient-descent meth ods for temporal-difference learning with linear function approximation. In Proceedings of the 26th annual international conference...

    2009

  54. [62]

    A convergent o(n) algorithm for off-policy temporal-difference learning with linear functi on approximation

    Richard S Sutton, Csaba Szepesvári, and Hamid Reza Maei . A convergent o(n) algorithm for off-policy temporal-difference learning with linear functi on approximation. Advances in neural information processing systems , 21(21):1609–1616, 2008

    2008

  55. [63]

    Almost sure convergence of two time- scale stochastic approximation algo- rithms

    Vladislav B Tadic. Almost sure convergence of two time- scale stochastic approximation algo- rithms. In Proceedings of the 2004 American Control Conference , volume 4, pages 3802–3807. IEEE, 2004

    2004

  56. [64]

    A r eview of off-policy evaluation in reinforcement learning

    Masatoshi Uehara, Chengchun Shi, and Nathan Kallus. A r eview of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.06355 , 2022

    2022 arXiv

  57. [65]

    Asymptotic analysis via stoch astic differential equations of gradient descent algorithms in statistical and computatio nal paradigms

    Yazhen Wang and Shang Wu. Asymptotic analysis via stoch astic differential equations of gradient descent algorithms in statistical and computatio nal paradigms. Journal of Machine Learning Research, 21(199):1–103, 2020

    2020

  58. [66]

    Non-asymptotic ana lysis for two time-scale TDC with general smooth function approximation

    Yue Wang, Shaofeng Zou, and Yi Zhou. Non-asymptotic ana lysis for two time-scale TDC with general smooth function approximation. Advances in Neural Information Processing Systems , 34:9747–9758, 2021

    2021

  59. [67]

    Weak convergence of probability measures o n the function space C[0, ∞)

    Ward Whitt. Weak convergence of probability measures o n the function space C[0, ∞). The Annals of Mathematical Statistics , 41(3):939–944, 1970

    1970

  60. [68]

    A fin ite-time analysis of two time-scale actor-critic methods

    Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A fin ite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems , 33:17617– 17628, 2020

    2020

  61. [69]

    Asymptotic time-uniform infer- ence for parameters in averaged stochastic approximation

    Chuhan Xie, Kaicheng Jin, Jiadong Liang, and Zhihua Zha ng. Asymptotic time-uniform infer- ence for parameters in averaged stochastic approximation. arXiv preprint arXiv:2410.15057 , 2024

    2024 arXiv

  62. [70]

    A statistical online infer ence approach in averaged stochastic approximation

    Chuhan Xie and Zhihua Zhang. A statistical online infer ence approach in averaged stochastic approximation. In Advances in Neural Information Processing Systems , 2022

    2022

  63. [71]

    Sample complexity bounds f or two timescale value-based reinforcement learning algorithms

    Tengyu Xu and Yingbin Liang. Sample complexity bounds f or two timescale value-based reinforcement learning algorithms. In International Conference on Artificial Intelligence and Statistics, pages 811–819. PMLR, 2021

    2021

  64. [72]

    Non-asymptotic convergence analysis of two time- scale (natural) actor-critic algorithms

    Tengyu Xu, Zhe Wang, and Yingbin Liang. Non-asymptotic convergence analysis of two time- scale (natural) actor-critic algorithms. arXiv preprint arXiv:2005.03557 , 2020

    2005 arXiv

  65. [73]

    Two time-sc ale off-policy TD learning: Non- asymptotic analysis over Markovian samples

    Tengyu Xu, Shaofeng Zou, and Yingbin Liang. Two time-sc ale off-policy TD learning: Non- asymptotic analysis over Markovian samples. Advances in Neural Information Processing Sys- tems, 32, 2019. 45

    2019

  66. [74]

    Stochastic recur sive inclusions in two timescales with nonadditive iterate-dependent Markov noise

    Vinayaka G Yaji and Shalabh Bhatnagar. Stochastic recur sive inclusions in two timescales with nonadditive iterate-dependent Markov noise. Mathematics of Operations Research, 45(4):1405– 1444, 2020. 46

    2020

  67. [233]

    Springer Science & Business Media, 1997

    1997

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.