Pith. sign in

REVIEW 2 major objections 6 minor 38 references

Dual Averaging Converges for Nonconvex Smooth Stochastic Optimization

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Stochastic dual averaging converges on smooth nonconvex stochastic optimization at rate $O(1/T + \sigma \log T/\sqrt{T})$, matching SGD up to a logarithmic factor.

desk verdict Promising reduction with a real boundary gap: Lemma A.3 is false at t=2, so Theorem 4.3 is not proven as written, but the hole looks repairable. read the letter →

arxiv 2505.21394 v1 pith:NCDJSOBU submitted 2025-05-27 math.OC

Tuo Liu , El Mehdi Saad , Wojciech Kot{l}owski , Francesco Orabona This is my paper
classification math.OC MSC 90C2690C1562L20
keywords stochasticdualaveragingnonconvexsmoothoptimizationstronggrowthconditiongradientnormconvergenceAdaGradstepsizeshigh-probabilityboundsfirst-ordermethods
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Stochastic dual averaging (SDA), a first-order method that updates its iterate from the uniform average of all past noisy gradients, has long been known to match gradient descent in convex problems, but no finite-time guarantee existed for smooth nonconvex objectives. This paper closes that gap: under the $(\rho,\sigma)$-strong growth condition on the gradient noise, a tuned SDA with decreasing step sizes drives the average squared gradient norm to zero at rate $O(1/T + \sigma \log T/\sqrt{T})$, matching SGD up to a logarithmic factor. The argument works by rewriting SDA as SGD on a sequence of objectives that are the original function plus a time-varying quadratic regularizer, and then exploiting the telescoping structure of the regularizer. A sympathetic reader should care because it gives the averaging-based method the same theoretical status as SGD for nonconvex training, removing the last major algorithmic gap between the two recipe books.

What carries the argument

The load-bearing identity is the equivalence of SDA with SGD on implicitly regularized objectives: writing $\gamma_t = 1/\eta_t - 1/\eta_{t-1}$, the SDA update $x_{t+1} = -\eta_t \sum_{i=1}^t g_i$ is identical to one SGD step with learning rate $\eta_t$ on $f_t(x) = f(x) + (\gamma_t/2)\|x\|^2$. Because $f$ is $L$-smooth, each surrogate $f_t$ is $L_t = L + \gamma_t$ smooth, and the standard descent inequality becomes an online descent inequality over a time-varying sequence. The proof then selects the step sizes of Theorem 4.3 so that the drift terms $\gamma_{t-1}-\gamma_t$ telescope and dominate the curvature and noise terms, yielding $A_t \ge (\eta_t/8)\||\nabla f(x_t)\|^2$; summing $A_t$ over $t$ converts bounds on the surrogates into bounds on the true gradient. For the adaptive variant, the AdaGrad-norm step size $\eta_t = \eta/\sqrt{\gamma + \sum_{i=1}^t \|g_i\|^2}$ plays the same role, with the bound controlled by the maximal iterate radius.

What would settle it

Run deterministic dual averaging with the step sizes of Theorem 4.3 on a smooth nonconvex function such as $f(x)=\sin(x)+x^2/4$. Since $\sigma=0$, the theorem predicts $(1/T)\sum_{t=1}^T(\nabla f(x_t))^2 = O(1/T)$; if a numerical evaluation at large $T$ shows a visibly slower decay, the lemma linking surrogate gradients to true gradients is broken.

Watch

Extended reading notes

Core claim

The paper's central claim is that stochastic dual averaging is not confined to convex optimization: for any $L$-smooth, lower-bounded $f$ with unbiased gradients satisfying $E[\|\hat{\nabla}f(x)-\nabla f(x)\|^2 \mid x] \le \rho\|\nabla f(x)\|^2 + \sigma^2$, the SDA iterates $x_{t+1} = -\eta_t \sum_{i=1}^t \hat{\nabla}f(x_i)$ with step sizes $\eta_t = 1/(L(1+\rho)(1+\rho+\alpha\sqrt{t}))$ and $\alpha = \min\{\sigma/(L(1+\rho)),1\}$ satisfy an explicit bound on $(1/T)\sum_{t=1}^T E[\|\nabla f(x_t)\|^2]$ that is $O(1/T + \sigma \log T/\sqrt{T})$. This is the first iterate-level convergence theorem for dual averaging in the smooth nonconvex stochastic setting; it also recovers the fast $O(1/T)$ rate in the low-noise case $\sigma=0$. The same reduction yields a high-probability bound under sub-Gaussian noise and an adaptive AdaGrad-norm variant, ADA-DA, that reaches the same rate without knowing $\sigma$ whenever the iterates stay bounded.

Load-bearing premise

The load-bearing premise is that gradient noise obeys the $(\rho,\sigma)$-strong growth condition, meaning its variance may grow with the squared true gradient, but only linearly plus a constant, and, for the high-probability bound, that the noise is sub-Gaussian.

Editorial extensions

If this is right

  • SDA now has a finite-time complexity guarantee for nonconvex smooth stochastic problems: $O(1/T + \sigma \log T/\sqrt{T})$ under the strong growth condition, matching SGD up to a log factor.
  • In the low-noise regime $\sigma=0$, the guarantee improves to $O(1/T)$, so averaging-based iterations inherit the fast rate of gradient descent.
  • With sub-Gaussian noise, the same analysis gives a high-probability bound: a single run, not just the ensemble average, has small average gradient with probability at least $1-\delta$.
  • Using AdaGrad-norm step sizes, ADA-DA attains the same rate adaptively, without knowing $\sigma$ or $L$, whenever the iterates remain bounded.
  • The prescribed step sizes do not require knowing the horizon $T$ in advance, so the bound is anytime rather than fixed-horizon.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The regularized-surrogate identity may carry over to other methods that accumulate past gradients with equal weights, such as momentum or Adam-type update rules; the paper notes these face the same coupling challenge but does not make the extension.
  • The logarithmic factor in the stochastic term likely comes from the time-varying step sizes rather than from the averaging mechanism itself, since SGD with time-varying learning rates suffers a similar log; if so, a constant-step-size variant would give the sharp $O(\sigma/\sqrt{T})$ without the log.
  • The bounded-iterate restriction on ADA-DA is probably removable: because the extra $x_t$ term in the SDA recursion lacks uniform decay under AdaGrad step sizes, a bias-correction or a slightly different adaptive schedule might yield an iterate-independent adaptive bound.
  • Practically, the result offers a theoretical basis for the empirical finding that dual-averaging optimizers match SGD on deep learning tasks: uniform weighting of past gradients does not sacrifice the convergence rate, even though it changes the trajectory.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper studies stochastic dual averaging (SDA) for smooth, possibly nonconvex, unconstrained optimization. The central contribution is an in-expectation convergence rate O(1/T + sigma log T / sqrt(T)) for the average squared gradient norm under the (rho, sigma)-strong growth condition, a high-probability bound under sub-Gaussian noise, and an adaptive AdaGrad-style variant whose bound depends on the maximal iterate norm. The main proof technique is a reduction of SDA to SGD on a time-varying sequence of implicitly regularized objectives f_t(x) = f(x) + (gamma_t / 2) ||x||^2, combined with a descent inequality and telescoping over the surrogates.

Significance. If the main theorem is correct, the paper closes a long-standing gap by providing the first finite-time gradient-norm complexity for standard stochastic dual averaging in nonconvex smooth stochastic optimization, matching SGD up to a logarithmic factor and recovering the fast 1/T rate in the low-noise regime. The paper has real strengths: the reduction in Proposition 1 is algebraically transparent, the step-size schedule is explicit and not fitted to the target bound, and the appendix contains substantial detail. However, the proof of Theorem 4.3 currently has a genuine gap at the first nontrivial iterate, so the central claim is not established as written; the significance is therefore conditional on a repair of that gap.

major comments (2)
  1. [Appendix A.4, Lemma A.3, Eq. (29)] Inequality (29) is false as stated for t=2 under the paper's convention eta_0 = eta_1, which gives gamma_1 = 0. For L=1, rho=0, alpha=1, one has eta_2 = 1/(1+sqrt(2)), gamma_2 = sqrt(2)-1, L_2 = 1 + gamma_2, and the left-hand side of (29) equals (gamma_1 - gamma_2)/2 - (1/2)gamma_2^2 eta_2 + (1/4)gamma_2^2 eta_2^2 L_2, which is approximately -0.232 < 0. With f(x) = c^T x + (1/2)||x||^2 and x_1 = 0, the quantity A_2 used in the proof of Theorem 4.3 is approximately -0.0267||c||^2, whereas the claimed lower bound eta_2/8 ||grad f(x_2)||^2 is approximately 0.0129||c||^2. Thus the inequality A_t >= eta_t/8 ||grad f(x_t)||^2, which is load-bearing for the proof of Theorem 4.3, is not established at t=2. The lemma's proof begins 'Fix t>=3', so the base case t=2 is simply not covered; a separate argument for t=2 is needed.
  2. [Appendix A.4, proof of Lemma A.3, last displayed inequality] Even for t>=3, the proof of (29) as written contains an invalid inference: the line 'we used in the last line L_t >= L and eta_t <= 1/(L(1+rho)) to have eta_t L_t >= 1/(1+rho)' uses eta_t <= 1/(L(1+rho)) together with L_t >= L to deduce a lower bound on eta_t L_t, which is the wrong direction. For example, with L=1, rho=0, alpha=1, eta_2 L_2 is about 0.586 < 1 = 1/(1+rho). The intended bound appears salvageable because the first positive term in (29) dominates, but the proof needs a corrected inequality chain. This is not a cosmetic issue because (29) is essential to the lower bound on A_t in Theorem 4.3.
minor comments (6)
  1. [Section 4, Lemma 4.2] The displayed inequality 'gamma_{t-1} - gamma_t - gamma_t^2 eta_t >= 0' is inconsistent with the appendix's inequality (29) and is false for the Section 4 schedule when gamma_1 = 0; for L=1, rho=0, alpha=1 and t=2 the left-hand side is about -0.485. The lemma should state the corrected coefficient inequality used in the appendix.
  2. [Appendix A.1, Eq. (14)] Equation (14) gives the right-hand side as E[f_1(x_2) - f^*], but the preceding telescoping of the terms f_{t-1}(x_t) - f_t(x_{t+1}) produces f_0(x_1) - f_T(x_{T+1}) = f(0) - f_T(x_{T+1}), and the next line of the proof uses f(0) - f^*. The displayed right-hand side should be corrected to f(0) - f^* for consistency.
  3. [Appendix A.2, proof of Theorem 5.4] The proof says 'by choosing eta_t = 1/(L+sigma t)', but the theorem and all surrounding algebra use eta_t = 1/(L+sigma sqrt(t)). This is a typo, but it should be fixed to avoid confusion.
  4. [Appendix A.4, proof of Lemma A.2] In the proof of Lemma A.2, the text says 'To show that B_t is non-positive', but the argument that follows establishes B_t >= 0; the word should be 'non-negative'. The same proof also contains a sign statement about psi'(tau) that is consistent with psi decreasing, so the conclusion B_t >= 0 is correct once the typo is fixed.
  5. [Theorem 4.3, statement] The bound is typeset ambiguously as 'L(1+rho) 2+sigma sqrt(T) / T'; it should be L(1+rho)(2+sigma sqrt(T))/T (or an equivalent expression). As printed, the formula is hard to parse.
  6. [Appendix A.2, proof of Lemma 5.3] The displayed inequality '1 - eta L_t/(2 sqrt(t)) - (1/2)(1-eta_t L_t)^2 >= 1/2' contains a spurious sqrt(t) in the denominator; with the intended term 1 - eta_t L_t/2 - (1/2)(1-eta_t L_t)^2 the inequality is valid for 0 <= eta_t L_t <= 1.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the main derivation is self-contained; the flagged Lemma A.3 failure at t=2 is a correctness gap, not a circular step.

full rationale

The central derivation chain is self-contained. Proposition 1, the reduction of SDA to SGD on the surrogates f_t(x) = f(x) + (γ_t/2)||x||^2 with γ_t = 1/η_t − 1/η_{t−1}, is attributed to Jelassi and Defazio (2020), an external source, and is also proven inline within the paper ('This result is an immediate consequence of the update of SDA: x_{t+1} = −η_t(g_t + Σ_{i<t}g_i) = x_t − η_t(g_t + (1/η_{t−1} − 1/η_t)x_t)'), so no ansatz is smuggled in via citation. The step-size schedule η_t = 1/(L(1+ρ)(1+ρ+α√t)) with α = min{σ/(L(1+ρ)), 1} is a proof choice rather than a fitted input: the σ-dependence of the final bound arises algebraically from the A4 noise term (σ^2/2)Σ η_t^2 L_t, and no term in the bound is fitted to the iterates or to the target quantity. The authors' self-citations (Orabona 2019; Li and Orabona 2019; Orabona and Pál 2021) appear only in background or related-work framing and are not load-bearing; the concentration step in Theorem 5.2 uses a standard MGF bound 'due to the helper Lemma 2.2 in Liu et al. (2023)', an external result. One flagged issue is a genuine omission but is not circular: Lemma A.3 states inequality (29) 'for any t≥2', yet its proof begins 'Fix t≥3', and at t=2 with L=1, ρ=0, α=1 the left side equals (γ_1−γ_2)/2 − γ_2^2 η_2/2 + γ_2^2 η_2^2 L_2/4 ≈ −0.232 < 0, so the asserted bound A_t ≥ (η_t/8)||∇f(x_t)||^2 in Theorem 4.3 is not established at the first nontrivial step. This is an internal correctness gap (likely repairable, e.g., by taking η_0 < η_1), not a reduction of the conclusion to its inputs, and it does not raise the circularity score. The paper's own limitations statements (heavy-tailed noise outside A5; adaptive Ada-DA only provably convergent with bounded iterates) further confirm that the claims are scoped rather than definitionally forced. Verdict: no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no free parameters. The step-size constants depend on problem data L, ρ, σ, not on fitted values. Assumptions A1-A5 are domain assumptions that the paper explicitly states. No new entities are postulated.

assumptions (5)
  • domain assumption L-smoothness of f (Assumption A1)
    The function is differentiable with L-Lipschitz gradient; used throughout for the descent inequality.
  • domain assumption Lower boundedness (Assumption A2)
    inf f > -∞ ensures telescoping bounds stay finite.
  • domain assumption Unbiased gradient estimator (Assumption A3)
    E[∇̂f(x)|x]=∇f(x) for the oracle used in SDA.
  • domain assumption Strong growth condition with additive noise (Assumption A4)
    Noise variance bounded by ρ||∇f(x)||^2 + σ^2; load-bearing for Theorem 4.3 and Theorem 6.1.
  • domain assumption Sub-Gaussian noise (Assumption A5)
    Used in Section 5 for high-probability bounds; replaces A4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dual Averaging Converges for Nonconvex Smooth Stochastic Optimization." pith.science (2026). https://pith.science/paper/NCDJSOBU

@misc{pith2026250521394,
  author       = {Pith},
  title        = {Pith review of: Dual Averaging Converges for Nonconvex Smooth Stochastic Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NCDJSOBU}},
  note         = {Machine review of arXiv:2505.21394}
}
abstract

Dual averaging and gradient descent with their stochastic variants stand as the two canonical recipe books for first-order optimization: Every modern variant can be viewed as a descendant of one or the other. In the convex regime, these algorithms have been deeply studied, and we know that they are essentially equivalent in terms of theoretical guarantees. On the other hand, in the non-convex setting, the situation is drastically different: While we know that SGD can minimize the gradient of non-convex smooth functions, no finite-time complexity guarantee for Stochastic Dual Averaging (SDA) was known in the same setting. In this paper, we close this gap by a reduction that views SDA as SGD applied to a sequence of implicitly regularized objectives. We show that a tuned SDA exhibits a rate of convergence $\mathcal{O}(1 / T + \sigma \log T/ \sqrt{T})$, similar to that of SGD under the same assumptions. To our best knowledge, this is the first complete convergence theory for dual averaging on non-convex smooth stochastic problems without restrictive assumptions, closing a long-standing open problem in the field. Beyond the base algorithm, we also discuss ADA-DA, a variant that marries SDA with AdaGrad's auto-scaling, which achieves the same rate without requiring knowledge of the noise variance.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 27 canonical work pages

  1. [1]

    J. D. Abernethy, E. Hazan, and A. Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. In Rocco A. Servedio and Tong Zhang, editors, Proc. of Conference on Learning Theory (COLT), pages 263--274. Omnipress, 2008

    work page 2008

  2. [2]

    L. Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pages 177--186. Springer, 2010

    work page 2010

  3. [3]

    Bottou, F

    L. Bottou, F. E. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60 0 (2): 0 223--311, 2018

    2018

  4. [4]

    Heavy-tailed noise does not explain the gap between sgd and adam on transformers

    Jacques Chen, Frederik Kunstner, and Mark Schmidt. Heavy-tailed noise does not explain the gap between sgd and adam on transformers. In 13th annual workshop on optimization for machine learning, 2021

    work page 2021

  5. [5]

    Chen and E

    X. Chen and E. Hazan. Open problem: Black-box reductions & adaptive gradient methods. Proceedings of Machine Learning Research vol, 196: 0 1--8, 2024

    work page 2024

  6. [6]

    X. Chen, S. Liu, R. Sun, and M. Hong. On the convergence of a class of Adam -type algorithms for non-convex optimization. In International Conference on Learning Representations, 2019

    work page 2019

  7. [7]

    Cutkosky

    A. Cutkosky. Anytime online-to-batch, optimism and acceleration. In International conference on machine learning, pages 1446--1454. PMLR, 2019

    work page 2019

  8. [8]

    J. C. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12: 0 2121--2159, 2011

    work page 2011

Show all 38 references
  1. [9]

    Ghadimi and G

    S. Ghadimi and G. Lan. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization I : A generic algorithmic framework. SIAM Journal on Optimization, 22 0 (4): 0 1469--1492, 2012

    2012

  2. [10]

    G. J. Gordon. Regret bounds for prediction problems. In Proc. of the twelfth annual conference on Computational learning theory (COLT), pages 29--40, 1999

    1999

  3. [11]

    Stochastic approximation and recursive algorithm and applications

    J Harold, G Kushner, and George Yin. Stochastic approximation and recursive algorithm and applications. Application of Mathematics, 35 0 (10), 1997

    1997

  4. [12]

    Hazan and S

    E. Hazan and S. Kale. Extracting certainty from uncertainty: Regret bounded by variation in costs. In Proc. of the 21st Conference on Learning Theory, 2008

    2008

  5. [13]

    P. Jain, S. M. Kakade, R. Kidambi, P. Netrapalli, and A. Sidford. Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification. Journal of machine learning research, 18 0 (223): 0 1--42, 2018

    2018

  6. [14]

    Jelassi and A

    S. Jelassi and A. Defazio. Dual averaging is surprisingly effective for deep learning optimization. arXiv preprint arXiv:2010.10502, 2020

    2010 arXiv

  7. [15]

    Li and F

    X. Li and F. Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. In Proc. of the 22nd International Conference on Artificial Intelligence and Statistics, AISTATS , 2019

    2019

  8. [16]

    Z. Liu, T. D. Nguyen, T. H. Nguyen, A. Ene, and H. Nguyen. High probability convergence of stochastic gradient methods. In International Conference on Machine Learning, pages 21884--21914. PMLR, 2023

    2023

  9. [17]

    H. B. McMahan and M. J. Streeter. Adaptive bound optimization for online convex optimization. In COLT, 2010

    2010

  10. [18]

    A. Mishkin. Interpolation, growth conditions, and stochastic gradient descent. PhD thesis, University of British Columbia, 2020

    2020

  11. [19]

    Nesterov

    Y. Nesterov. Primal-dual subgradient methods for convex problems. Mathematical programming, 120 0 (1): 0 221--259, 2009. Received: 29 September 2005 / Accepted: 13 January 2007 / Published online: 19 June 2007

    2009

  12. [20]

    First exit time analysis of stochastic gradient descent under heavy-tailed gradient noise

    Thanh Huy Nguyen, Umut Simsekli, Mert Gurbuzbalaban, and Ga \"e l Richard. First exit time analysis of stochastic gradient descent under heavy-tailed gradient noise. Advances in neural information processing systems, 32, 2019

    2019

  13. [21]

    F. Orabona. A modern introduction to online learning. arXiv preprint arXiv:1912.13213, 2019

    1912 arXiv

  14. [22]

    Orabona and D

    F. Orabona and D. P\'al. Parameter-free stochastic optimization of variationally coherent functions. arXiv preprint arXiv:2102.00236, 2021

    2021 arXiv

  15. [23]

    Schmidt and N

    M. Schmidt and N. Le Roux. Fast convergence of stochastic gradient descent under a strong growth condition. arXiv preprint arXiv:1308.6370, 2013

    2013 arXiv

  16. [24]

    Shalev-Shwartz and Y

    S. Shalev-Shwartz and Y. Singer. Online learning meets optimization in the dual. In International Conference on Computational Learning Theory, pages 423--437. Springer, 2006 a

    2006

  17. [25]

    Shalev-Shwartz and Y

    S. Shalev-Shwartz and Y. Singer. Convex repeated games and Fenchel duality. In Advances in neural information processing systems, pages 1265--1272, 2006 b

    2006

  18. [26]

    Simsekli, L

    U. Simsekli, L. Sagun, and M. Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning, pages 5827--5837. PMLR, 2019

    2019

  19. [27]

    Solodkin, S

    V. Solodkin, S. Chezhegov, R. Nazikov, A. Beznosikov, and A. Gasnikov. Accelerated stochastic gradient method with applications to consensus problem in Markov -varying networks. In International Conference on Mathematical Optimization Theory and Operations Research, pages 69--...

    2024

  20. [28]

    Streeter and H

    M. Streeter and H. B. McMahan. Less regret via online conditioning. arXiv preprint arXiv:1002.4862, 2010

    2010 arXiv

  21. [29]

    A. S. Suggala and P. Netrapalli. Online non-convex learning: Following the perturbed leader is optimal. In Algorithmic Learning Theory, pages 845--861. PMLR, 2020

    2020

  22. [30]

    Vaswani, F

    S. Vaswani, F. Bach, and M. Schmidt. Fast and faster convergence of SGD for over-parameterized models (and an accelerated P erceptron). In The 22nd international conference on artificial intelligence and statistics, pages 1195--1204. PMLR, 2019

    2019

  23. [31]

    Vershynin

    R. Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018

    2018

  24. [32]

    R. Ward, X. Wu, and L. Bottou. AdaGrad stepsizes: Sharp convergence over nonconvex landscapes. In International Conference on Machine Learning, pages 6677--6686. PMLR, 2019

    2019

  25. [33]

    L. Xiao. Dual averaging methods for regularized stochastic learning and online optimization. Journal of Machine Learning Research, 11: 0 2543--2596, 2010

    2010

  26. [34]

    Zhang, S

    J. Zhang, S. Pr. Karimireddy, A. Veit, S. Kim, S. Reddi, S. Kumar, and S. Sra. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33: 0 15383--15393, 2020

    2020

  27. [35]

    Zhang and Z.-H

    L. Zhang and Z.-H. Zhou. Stochastic approximation of smooth and strongly convex functions: Beyond the O(1/t) convergence rate. In Conference on Learning Theory, pages 3160--3179. PMLR, 2019

    2019

  28. [36]

    D. Zhou, J. Chen, Y. Cao, Z. Yang, and Q. Gu. On the convergence of adaptive gradient methods for nonconvex optimization. arXiv preprint arXiv:1808.05671, 2018

    2018 arXiv

  29. [37]

    Zinkevich

    M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proc. of the International Conference on Machine Learning, pages 928--936, 2003

    2003

  30. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.