Pith. sign in

REVIEW 2 major objections 2 minor 77 references

Sharp asymptotic theory for Q-learning with LDTZ learning rate and its generalization

T0 review · 2 major / 2 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Download PDF Read on arXiv ↗

Pith's one-line read Q-learning with power-law decay-to-zero steps gets both fast initial drop and asymptotic consistency, plus a CLT and strong invariance for inference.

desk verdict Coherent theory package for PD2Z/LD2Z Q-learning (non-asymptotics + tail PR CLT + strong invariance), but the supplied full text is unreadable so we cannot audit the proofs. read the letter →

arxiv 2604.04218 v1 submitted 2026-04-05 stat.ML cs.LGmath.STstat.TH

Soham Bonnerjee , Zhipeng Lou , Wei Biao Wu This is my paper · ORCID
classification stat.MLcs.LGmath.STstat.TH MSC 62L2062M0568T0590C40
keywords Q-learningLD2ZPD2Zstep-sizeschedulesPolyak-Ruppertaveragingstronginvarianceprinciplecentrallimittheoremreinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most Q-learning theory uses either constant step sizes, which leave a permanent bias, or polynomially decaying steps, which converge slowly. This paper studies a broader family called power-law decay to zero (PD2Z-ν), which includes the popular linear decay to zero (LD2Z) schedule, and proves that these schedules deliver a best-of-both-worlds rate: they forget bad initializations as quickly as constant steps while still converging like decaying steps. The authors give a sharp non-asymptotic error bound, then use it to obtain a central limit theorem for a new tail average of the Q-iterates and a time-uniform strong invariance principle for the partial-sum process. Together these results justify bootstrap-based inference and explain why LD2Z has worked well in practice. Numerical experiments support the theory and give concrete guidance on how to average and how to choose the decay exponent.

What carries the argument

The PD2Z-ν family of learning rates η_{t,n}=η(1-t/n)^ν, together with a tail Polyak–Ruppert averaging estimator that averages only the late iterates; these objects turn the non-asymptotic error bound into asymptotic normality and a strong invariance principle.

What would settle it

Run tabular Q-learning with LD2Z (or PD2Z-ν) on a small, fully known MDP whose optimal Q-function is known exactly; if the observed mean-squared error does not decay at the rate predicted by the non-asymptotic bound, or if the tail average fails to be asymptotically normal at the predicted scale, the central claims are false.

Watch

Extended reading notes

Core claim

For tabular Q-learning driven by PD2Z-ν step sizes η_{t,n}=η(1-t/n)^ν (including LD2Z when ν=1), the paper establishes a sharp non-asymptotic error bound that simultaneously captures rapid decay of the initialization bias and vanishing asymptotic bias. From that bound it derives a central limit theorem for a novel tail Polyak–Ruppert average and a time-uniform Gaussian approximation (strong invariance principle) for the partial-sum process of the Q-iterates, enabling rigorous bootstrap inference.

Load-bearing premise

The underlying Markov decision process must be finite-state and finite-action, and the sampling chain under the behavior policy must mix fast enough for the usual contraction and moment conditions of Q-learning to hold; if mixing fails or the space is continuous, the claimed rates and limit theorems collapse.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper studies tabular Q-learning under a power-law decay-to-zero step-size family PD2Z-ν (η_{t,n}=η(1-t/n)^ν), which includes the linear schedule LD2Z when ν=1. From the abstract it claims three main results: a sharp non-asymptotic error bound, a central limit theorem for a new tail Polyak–Ruppert averaging estimator, and a time-uniform strong invariance principle for the partial-sum process of the iterates that enables bootstrap inference. These are said to establish a best-of-both-worlds property—rapid decay from initialization like constant steps together with asymptotic consistency like polynomially decaying steps—and are supported by numerical experiments. The supplied full-text artifact is almost entirely corrupted by encoding errors, so the precise theorem statements, assumptions, and proofs cannot be audited beyond the abstract.

Significance. If the claimed rates, CLT, and strong invariance principle hold under standard finite-state MDP and mixing conditions, the work would fill a genuine gap: most existing theory treats only constant or polynomially decaying steps, while LD2Z has already shown empirical promise. A rigorous non-asymptotic bound plus usable asymptotic inference tools (tail PR averages and bootstrap via strong approximation) would be of clear interest to the stochastic-approximation and RL-theory communities and would give practitioners concrete guidance on step-size choice and uncertainty quantification. Because the body is unreadable, these contributions remain provisional; the abstract alone is coherent and well-motivated.

major comments (2)
  1. The full manuscript body supplied for review is essentially unreadable (severe Unicode/encoding corruption throughout). Consequently the central non-asymptotic bound, the precise definition and CLT for the tail Polyak–Ruppert estimator, the strong-invariance statement, and all supporting assumptions (mixing rates, moment conditions, admissible range of η and ν) cannot be verified. Without a clean, auditable text the load-bearing claims cannot be assessed for correctness or sharpness.
  2. Even from the abstract it is clear that the results rest on standard but non-trivial premises (finite state-action space, contraction of the Bellman operator under a fixed behavior policy, suitable ergodicity/mixing of the sampling chain). The paper must state these hypotheses explicitly, quantify the dependence of the rates on the mixing time and the contraction modulus, and indicate the admissible range of the free parameters η and ν; none of this can be checked in the present artifact.
minor comments (2)
  1. Once a clean manuscript is available, the abstract’s claim of a “best-of-both-worlds” property should be accompanied by an explicit comparison table (or corollary) that places the PD2Z-ν rates next to the classical constant-step and polynomial-step rates under identical assumptions.
  2. Notation for the horizon-dependent schedule η_{t,n} and for the tail averaging window should be introduced early and kept consistent; the abstract already uses both LD2Z and PD2Z-ν, so a short glossary would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: abstract claims are standard SA/Q-learning asymptotics under a named step-size class, not fits or definitional identities.

full rationale

The paper’s advertised contribution is a chain of mathematical results for tabular Q-learning under the PD2Z-ν (including LD2Z) step-size schedule: a sharp non-asymptotic error bound, a CLT for a tail Polyak–Ruppert average, and a time-uniform strong invariance principle for the partial-sum process. These are asymptotic/non-asymptotic statistical statements about an algorithm under stated (standard) MDP and noise assumptions, not empirical predictions obtained by fitting free parameters to data, nor quantities defined in terms of the very objects they claim to derive. The full manuscript body supplied in the cache is heavily encoding-corrupted and unreadable, so no theorem statements, proof steps, or self-citations can be audited line-by-line; under the hard rule that circularity may be claimed only when a specific reduction can be quoted and exhibited, no circular step can be asserted. From the abstract alone there is no self-definitional loop, no fitted-input-called-prediction, and no load-bearing uniqueness theorem imported from the authors. Residual dependence on prior stochastic-approximation lemmas (possibly by overlapping authors) is ordinary and not circularity under the stated criteria. Score 0 is therefore the honest finding.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

Only the abstract is usable. The theory almost certainly rests on standard MDP/SA axioms (finite tabular setting, Bellman contraction, step-size form, noise moments, mixing of the sampling chain) plus the authors’ definitions of PD2Z-ν and a tail Polyak–Ruppert average. Free parameters include the schedule constants η and ν and any burn-in/tail cutoffs for averaging. No new physical entities are introduced; invented objects are mathematical estimators and the named schedule class.

free parameters (3)
  • η (base learning-rate scale in PD2Z-ν)
    User-chosen scale in η_{t,n}=η(1−t/n)^ν; theory and finite-sample constants depend on it.
  • ν (power in PD2Z-ν)
    Schedule exponent; LD2Z is the special case ν=1. Rates and CLT regimes depend on ν.
  • tail/burn-in cutoffs for Polyak–Ruppert averaging
    The new tail averaging estimator requires choosing which late iterates to average; those cutoffs affect bias–variance and bootstrap validity.
assumptions (5)
  • domain assumption Finite-state, finite-action MDP with Q-learning updates under a (behavior) policy inducing sufficiently mixing samples.
    Standard for sharp tabular Q-learning asymptotics; required for contraction and ergodic averages underlying CLT/SIP.
  • domain assumption Bellman optimality/evaluation operator is a contraction in a suitable norm (discount factor γ<1 or analogous).
    Core of Q-learning convergence; without it the error bounds and Gaussian limits do not hold in the usual form.
  • ad hoc to paper Step sizes follow PD2Z-ν: η_{t,n}=η(1−t/n)^ν with η,ν in a regime where the paper’s rates apply.
    The schedule class is the paper’s object of study; results are conditional on this exact horizon-dependent form.
  • domain assumption Noise/moment and dependence conditions sufficient for a strong invariance principle of the Q-learning martingale (or near-martingale) noise.
    Needed for time-uniform Gaussian approximation and bootstrap; typical but load-bearing and not checkable here.
  • standard math Standard probability inequalities and stochastic-approximation limit theorems used as black boxes.
    Background analysis tools; not invented by the paper.
invented entities (2)
  • PD2Z-ν learning-rate class (power-law decay to zero over horizon n) independent evidence
    purpose: Unifies LD2Z and related schedules for which sharp Q-learning theory is developed.
    Named schedule family central to all claims; mathematical definition rather than a physical entity.
  • Tail Polyak–Ruppert averaging estimator for Q-learning under PD2Z
    purpose: Obtain a CLT-scale estimator with reduced early-iterate contamination under horizon-dependent steps.
    Described as new in the abstract; exists as an estimator definition with claimed Gaussian limit.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sharp asymptotic theory for Q-learning with LDTZ learning rate and its generalization." pith.science (2026). https://pith.science/paper/2604.04218

@misc{pith2026260404218,
  author       = {Pith},
  title        = {Pith review of: Sharp asymptotic theory for Q-learning with LDTZ learning rate and its generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2604.04218}},
  note         = {Machine review of arXiv:2604.04218}
}
abstract

Despite the sustained popularity of Q-learning as a practical tool for policy determination, a majority of relevant theoretical literature deals with either constant ($\eta_{t}\equiv \eta$) or polynomially decaying ($\eta_{t} = \eta t^{-\alpha}$) learning schedules. However, it is well known that these choices suffer from either persistent bias or prohibitively slow convergence. In contrast, the recently proposed linear decay to zero (\texttt{LD2Z}: $\eta_{t,n}=\eta(1-t/n)$) schedule has shown appreciable empirical performance, but its theoretical and statistical properties remain largely unexplored, especially in the Q-learning setting. We address this gap in the literature by first considering a general class of power-law decay to zero (\texttt{PD2Z}-$\nu$: $\eta_{t,n}=\eta(1-t/n)^{\nu}$). Proceeding step-by-step, we present a sharp non-asymptotic error bound for Q-learning with \texttt{PD2Z}-$\nu$ schedule, which then is used to derive a central limit theory for a new \textit{tail} Polyak-Ruppert averaging estimator. Finally, we also provide a novel time-uniform Gaussian approximation (also known as \textit{strong invariance principle}) for the partial sum process of Q-learning iterates, which facilitates bootstrap-based inference. All our theoretical results are complemented by extensive numerical experiments. Beyond being new theoretical and statistical contributions to the Q-learning literature, our results definitively establish that \texttt{LD2Z} and in general \texttt{PD2Z}-$\nu$ achieve a best-of-both-worlds property: they inherit the rapid decay from initialization (characteristic of constant step-sizes) while retaining the asymptotic convergence guarantees (characteristic of polynomially decaying schedules). This dual advantage explains the empirical success of \texttt{LD2Z} while providing practical guidelines for inference through our results.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 2 canonical work pages

  1. [1]

    Bergsma, N

    S. Bergsma, N. S. Dey, G. Gosal, G. Gray, D. Soboleva, and J. Hestness. Straight to zero: Why linearly decaying the learning rate to zero works best for llms. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net, 2025. URL https://openreview.net/forum?id=hrOlBgHsMI

    2025

  2. [2]

    Berkes, W

    I. Berkes, W. Liu, and W. B. Wu. Koml \'o s--major--tusn \'a dy approximation under dependence. The Annals of Probability, 42 0 (2): 0 794--817, 2014

    2014

  3. [3]

    Bhandari, D

    J. Bhandari, D. Russo, and R. Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference on learning theory, pages 1691--1692. PMLR, 2018

    2018

  4. [4]

    Bonnerjee, S

    S. Bonnerjee, S. Karmakar, and W. B. Wu. Gaussian approximation for nonstationary time series with optimal rate and explicit construction. Ann. Statist., 52 0 (5): 0 2293--2317, 2024. ISSN 0090-5364,2168-8966. doi:10.1214/24-aos2436. URL https://doi.org/10.1214/24-aos2436

    work page doi:10.1214/24-aos2436 2024

  5. [5]

    Bonnerjee, S

    S. Bonnerjee, S. Karmakar, and W. B. Wu. Sharp gaussian approximations for decentralized federated learning. arXiv preprint arXiv:2505.08125, 2025

    arXiv 2025

  6. [6]

    V. S. Borkar. Stochastic approximation: a dynamical systems viewpoint, volume 48 of Texts and Readings in Mathematics. Springer, Singapore; Hindustan Book Agency, New Delhi, second edition, 2023. ISBN 978-981-99-8276-9; 978-981-99-8277-6. doi:10.1007/978-981-99-8277-6. URL https://doi.org/10.1007/978-981-99-8277-6

    doi:10.1007/978-981-99-8277-6 2023

  7. [7]

    Bottou, F

    L. Bottou, F. E. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning. SIAM Rev., 60 0 (2): 0 223--311, 2018. ISSN 0036-1445,1095-7200. doi:10.1137/16M1080173. URL https://doi.org/10.1137/16M1080173

    doi:10.1137/16m1080173 2018

  8. [8]

    X. Chen, J. D. Lee, X. T. Tong, and Y. Zhang. Statistical inference for model parameters in stochastic gradient descent. Ann. Statist., 48 0 (1): 0 251--273, 2020 a . ISSN 0090-5364,2168-8966. doi:10.1214/18-AOS1801. URL https://doi.org/10.1214/18-AOS1801

    doi:10.1214/18-aos1801 2020

Show all 77 references
  1. [9]

    Z. Chen. Non-asymptotic guarantees for average-reward q-learning with adaptive stepsizes. arXiv preprint arXiv:2504.18743, 2025

    2025 arXiv

  2. [10]

    Z. Chen, S. T. Maguluri, S. Shakkottai, and K. Shanmugam. Finite-sample analysis of contractive stochastic approximation using smooth convex envelopes. Advances in Neural Information Processing Systems, 33: 0 8223--8234, 2020 b

    2020

  3. [11]

    Z. Chen, S. T. Maguluri, S. Shakkottai, and K. Shanmugam. A lyapunov theory for finite-sample guarantees of asynchronous q-learning and td-learning variants. arXiv preprint arXiv:2102.01567, 2021

    2021 arXiv

  4. [12]

    Y. Chi, Y. Chen, and Y. Wei. Statistical and algorithmic foundations of reinforcement learning. arXiv preprint arXiv:2507.14444, 2025

    2025 arXiv

  5. [13]

    o rg o and P. R \'e v \'e sz. A new method to prove strassen type laws of invariance principle. 1. Zeitschrift f \

    M. Cs \"o rg o and P. R \'e v \'e sz. A new method to prove strassen type laws of invariance principle. 1. Zeitschrift f \"u r Wahrscheinlichkeitstheorie und verwandte Gebiete , 31 0 (4): 0 255--259, 1975 a

    1975

  6. [14]

    o rg o and P. R \'e v \'e sz. A new method to prove strassen type laws of invariance principle. ii. Zeitschrift f \

    M. Cs \"o rg o and P. R \'e v \'e sz. A new method to prove strassen type laws of invariance principle. ii. Zeitschrift f \"u r Wahrscheinlichkeitstheorie und verwandte Gebiete , 31 0 (4): 0 261--269, 1975 b

    1975

  7. [15]

    Cs \"o rgo and P

    M. Cs \"o rgo and P. R \'e v \'e sz. Strong approximations in probability and statistics. Academic press, 2014

    2014

  8. [16]

    Cs \"o rg \"o and P

    S. Cs \"o rg \"o and P. Hall. The koml \'o s-major-tusn \'a dy approximations and their applications. Australian Journal of Statistics, 26 0 (2): 0 189--218, 1984

    1984

  9. [17]

    Dedecker, P

    J. Dedecker, P. Doukhan, and F. Merlev \`e de. Rates of convergence in the strong invariance principle under projective criteria. Electron. J. Probab, 17 0 (16): 0 1--31, 2012

    2012

  10. [18]

    Defazio, A

    A. Defazio, A. Cutkosky, H. Mehta, and K. Mishchenko. Optimal linear decay learning rate schedules and further refinements. arXiv preprint arXiv:2310.07831, 2023

    2023 arXiv

  11. [19]

    Devlin, M

    J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computati...

    2019 doi

  12. [20]

    U. Einmahl. A useful estimate in the multidimensional invariance principle. Probability theory and related fields, 76 0 (1): 0 81--101, 1987

    1987

  13. [21]

    Erd \"o s and M

    P. Erd \"o s and M. Kac. On certain limit theorems of the theory of probability. Bulletin of the American Mathematical Society, 52 0 (4): 0 292--302, 1946

    1946

  14. [22]

    Goldreich, Z

    O. Goldreich, Z. Wei, S. Bonnerjee, J. Li, and W. B. Wu. Asymptotic theory of sgd with a general learning-rate. Preprint, 2025

    2025

  15. [23]

    G\" o tze and A

    F. G\" o tze and A. Y. Zaitsev. Bounds for the rate of strong approximation in the multidimensional invariance principle. Theory of Probability & Its Applications, 53 0 (1): 0 59--80, 2009

    2009

  16. [24]

    H \" a gele, E

    A. H \" a gele, E. Bakouch, A. Kosson, L. B. Allal, L. von Werra, and M. Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, editors, Advances in Neural Info...

    2024

  17. [25]

    Heyde and D

    C. Heyde and D. Scott. Invariance principles for the law of the iterated logarithm for martingales and processes with stationary increments. The Annals of Probability, pages 428--436, 1973

    1973

  18. [26]

    Hoffmann, S

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. W. Rae, and L. Sifr...

    2022

  19. [27]

    Howard and S

    J. Howard and S. Ruder. Universal language model fine-tuning for text classification. In I. Gurevych and Y. Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long...

    2018 doi

  20. [28]

    N. Iyer, V. Thejas, N. Kwatra, R. Ramjee, and M. Sivathanu. Wide-minima density hypothesis and the explore-exploit learning rate schedule. J. Mach. Learn. Res., 24: 0 65:1--65:37, 2023. URL https://jmlr.org/papers/v24/21-0549.html

    2023

  21. [29]

    Jaakkola, M

    T. Jaakkola, M. Jordan, and S. Singh. Convergence of stochastic iterative dynamic programming algorithms. Advances in neural information processing systems, 6, 1993

    1993

  22. [30]

    Karmakar and W

    S. Karmakar and W. B. Wu. Optimal gaussian approximation for multiple time series. Statistica Sinica, 30 0 (3): 0 1399--1417, 2020

    2020

  23. [31]

    Karmakar, S

    S. Karmakar, S. Richter, and W. B. Wu. Simultaneous inference for time-varying models. Journal of Econometrics, 227 0 (2): 0 408--428, 2022

    2022

  24. [32]

    Khamaru, A

    K. Khamaru, A. Pananjady, F. Ruan, M. J. Wainwright, and M. I. Jordan. Is temporal difference learning optimal? A n instance-dependent analysis. SIAM J. Math. Data Sci., 3 0 (4): 0 1013--1040, 2021. ISSN 2577-0187. doi:10.1137/20M1331524. URL https://doi.org/10.1137/20M1331524

    2021 doi

  25. [33]

    Koml \'o s, P

    J. Koml \'o s, P. Major, and G. Tusn \'a dy. An approximation of partial sums of independent rv'-s, and the sample df. i. Zeitschrift f \"u r Wahrscheinlichkeitstheorie und verwandte Gebiete , 32: 0 111--131, 1975

    1975

  26. [34]

    Koml \'o s, P

    J. Koml \'o s, P. Major, and G. Tusn \'a dy. An approximation of partial sums of independent rv's, and the sample df. ii. Zeitschrift f \"u r Wahrscheinlichkeitstheorie und verwandte Gebiete , 34: 0 33--58, 1976

    1976

  27. [35]

    S. Lee, Y. Liao, M. H. Seo, and Y. Shin. Fast and robust online inference with stochastic gradient descent via random scaling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7381--7389, 2022

    2022

  28. [36]

    G. Li, Y. Wei, Y. Chi, Y. Gu, and Y. Chen. Sample complexity of asynchronous q-learning: Sharper analysis and variance reduction. IEEE Transactions on Information Theory, 68 0 (1): 0 448--473, 2021

    2021

  29. [37]

    G. Li, C. Cai, Y. Chen, Y. Wei, and Y. Chi. Is q-learning minimax optimal? a tight sample complexity analysis. Operations Research, 72 0 (1): 0 222--236, 2024 a

    2024

  30. [38]

    J. Li, Z. Lou, S. Richter, and W. B. Wu. The stochastic gradient descent from a nonlinear time series perspective. Preprint, 2024 b

    2024

  31. [39]

    X. Li, J. Liang, and Z. Zhang. Online statistical inference for nonlinear stochastic approximation with markovian data. arXiv preprint arXiv:2302.07690, 2023 a

    2023 arXiv

  32. [40]

    X. Li, W. Yang, J. Liang, Z. Zhang, and M. I. Jordan. A statistical analysis of polyak-ruppert averaged q-learning. In International Conference on Artificial Intelligence and Statistics, pages 2207--2261. PMLR, 2023 b

    2023

  33. [41]

    Liu and Z

    W. Liu and Z. Lin. Strong approximation for a class of stationary processes. Stochastic Processes and their Applications, 119 0 (1): 0 249--280, 2009

    2009

  34. [42]

    Liu and W

    W. Liu and W. B. Wu. Simultaneous nonparametric inference of time series. The Annals of Statistics, 38 0 (4): 0 2388--2421, 2010

    2010

  35. [43]

    Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  36. [44]

    Lu and Q.-M

    C. Lu and Q.-M. Shao. Strong approximations for partial sums of weakly dependent random variables. Sci. Sinica Ser. A, 1987

    1987

  37. [45]

    Merlev \`e de and E

    F. Merlev \`e de and E. Rio. Strong approximation of partial sums under dependence conditions with application to dynamical systems. Stochastic Processes and their applications, 122 0 (1): 0 386--417, 2012

    2012

  38. [46]

    Mies and A

    F. Mies and A. Steland. Sequential gaussian approximation for nonstationary time series in high dimensions. Bernoulli, 29 0 (4): 0 3114--3140, 2023

    2023

  39. [47]

    W. Mou, C. J. Li, M. J. Wainwright, P. L. Bartlett, and M. I. Jordan. On linear stochastic approximation: Fine-grained polyak-ruppert and non-asymptotic concentration. In Conference on learning theory, pages 2947--2997. PMLR, 2020

    2020

  40. [48]

    S. K. Panda, T. Li, R. Liu, and Y. Xiang. Online statistical inference of constant sample-averaged q-learning. In First Reinforcement Learning Safety Workshop, 2024

    2024

  41. [49]

    B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30 0 (4): 0 838--855, 1992

    1992

  42. [50]

    M. L. Puterman and S. L. Brumelle. On the convergence of policy iteration in stationary dynamic programming. Mathematics of Operations Research, 4 0 (1): 0 60--69, 1979

    1979

  43. [51]

    Qu and A

    G. Qu and A. Wierman. Finite-time analysis of asynchronous stochastic approximation and q -learning. In Conference on Learning Theory, pages 3185--3205. PMLR, 2020

    2020

  44. [52]

    D. Ruppert. Efficient estimations from a slowly convergent robbins-monro process. Technical Report, 1988

    1988

  45. [53]

    Samsonov, E

    S. Samsonov, E. Moulines, Q.-M. Shao, Z.-S. Zhang, and A. Naumov. Gaussian approximation and multiplier bootstrap for polyak-ruppert averaged linear stochastic approximation with applications to td learning. In The Thirty-eighth Annual Conference on Neural Information Processi...

    2024

  46. [54]

    Samsonov, M

    S. Samsonov, M. Sheshukova, E. Moulines, and A. Naumov. Statistical inference for linear stochastic approximation with markovian noise. arXiv preprint arXiv:2505.19102, 2025

    2025 arXiv

  47. [55]

    Q.-M. Shao. Strong approximation theorems for independent random variables and their applications. Journal of multivariate analysis, 52 0 (1): 0 107--130, 1995

    1995

  48. [56]

    L. Shi, G. Li, Y. Wei, Y. Chen, and Y. Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. In International conference on machine learning, pages 19967--20025. PMLR, 2022

    2022

  49. [57]

    Strassen

    V. Strassen. An invariance principle for the law of the iterated logarithm. Zeitschrift f \"u r Wahrscheinlichkeitstheorie und verwandte Gebiete , 3 0 (3): 0 211--226, 1964

    1964

  50. [58]

    R. S. Sutton and A. G. Barto. Reinforcement learning: an introduction. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, second edition, 2018. ISBN 978-0-262-03924-6

    2018

  51. [59]

    Szepesv \'a ri

    C. Szepesv \'a ri. The asymptotic convergence-rate of q-learning. Advances in neural information processing systems, 10, 1997

    1997

  52. [60]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi \`e re, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  53. [61]

    J. N. Tsitsiklis. Asynchronous stochastic approximation and Q-learning, volume 16. Springer, 1994

    1994

  54. [62]

    C. J. Watkins and P. Dayan. Q-learning. Machine learning, 8: 0 279--292, 1992

    1992

  55. [63]

    C. J. C. H. Watkins et al. Learning from delayed rewards. 1989

    1989

  56. [64]

    Waudby-Smith, D

    I. Waudby-Smith, D. Arbour, R. Sinha, E. H. Kennedy, and A. Ramdas. Time-uniform central limit theory and asymptotic confidence sequences. The Annals of Statistics, 52 0 (6): 0 2613--2640, 2024

    2024

  57. [65]

    Z. Wei, W. Zhu, and W. B. Wu. Weighted averaged stochastic gradient descent: Asymptotic normality and optimality. arXiv preprint arXiv:2307.06915, 2023

    2023 arXiv

  58. [66]

    S. Wu, G. Zhang, and X. Liu. SwinSOD : Salient object detection using swin-transformer. Image Vis. Comput., 146 0 (105039): 0 105039, June 2024 a

    2024

  59. [67]

    W. Wu, G. Li, Y. Wei, and A. Rinaldo. Statistical inference for temporal difference learning with linear function approximation. arXiv preprint arXiv:2410.16106, 2024 b

    2024 arXiv

  60. [68]

    W. Wu, Y. Wei, and A. Rinaldo. Uncertainty quantification for markov chains with application to temporal difference learning. arXiv preprint arXiv:2502.13822, 2025

    2025 arXiv

  61. [69]

    W. B. Wu. Strong invariance principles for dependent random variables. The Annals of Probability, pages 2294--2320, 2007

    2007

  62. [70]

    W. B. Wu and Z. Zhao. Inference of trends in time series. Journal of the Royal Statistical Society Series B: Statistical Methodology, 69 0 (3): 0 391--410, 2007

    2007

  63. [71]

    W. B. Wu and Z. Zhou. Gaussian approximations for non-stationary multiple time series. Statistica Sinica, pages 1397--1413, 2011

    2011

  64. [72]

    E. Xia, K. Khamaru, M. J. Wainwright, and M. I. Jordan. Instance-optimality in optimal value estimation: Adaptivity via variance-reduced q-learning. IEEE Transactions on Information Theory, 2024

    2024

  65. [73]

    Xie and Z

    C. Xie and Z. Zhang. A statistical online inference approach in averaged stochastic approximation. Advances in Neural Information Processing Systems, 35: 0 8998--9009, 2022

    2022

  66. [74]

    C. Xie, K. Jin, J. Liang, and Z. Zhang. Asymptotic time-uniform inference for parameters in averaged stochastic approximation. arXiv preprint arXiv:2410.15057, 2024

    2024 arXiv

  67. [75]

    Zhang and Q

    Y. Zhang and Q. Xie. Constant stepsize q-learning: Distributional convergence, bias and extrapolation. arXiv preprint arXiv:2401.13884, 2024

    2024 arXiv

  68. [76]

    W. Zhu, X. Chen, and W. B. Wu. Online covariance matrix estimation in stochastic gradient descent. J. Amer. Statist. Assoc., 118 0 (541): 0 393--404, 2023. ISSN 0162-1459,1537-274X

    2023

  69. [77]

    W. Zhu, Z. Lou, Z. Wei, and W. B. Wu. High confidence level inference is almost free using parallel stochastic optimization. arXiv preprint arXiv:2401.09346, 2024

    2024

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.