Pith. sign in

REVIEW 2 major objections 5 minor 100 references

Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size

T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding heavy-ball momentum to the curvature-adaptive NGN step-size preserves its convergence rate and step-size robustness under far weaker assumptions than earlier momentum-plus-Polyak analyses required.

desk verdict The main theorem only covers negligible momentum (β ≈ 0.01), while the paper's own experiments use β = 0.9; the gap is real and structural, but the paper admits it and the empirical robustness results are substantive. read the letter →

arxiv 2508.15071 v1 pith:ZRAJA2OL submitted 2025-08-20 cs.LG math.OCstat.ML

Rustem Islamov , Niccolo Ajroldi , Antonio Orvieto , Aurelien Lucchi This is my paper · ORCID
classification cs.LGmath.OCstat.ML MSC 90C2590C15
keywords NGNstep-sizeheavy-ballmomentumadaptivestochasticconvexoptimizationconvergenceraterobustnesscoordinate-wisepreconditioningdeeplearningoptimizers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that heavy-ball momentum and the curvature-adaptive NGN step-size can be combined without giving up the step-size's best property: robustness to the choice of the learning-rate hyperparameter. The new algorithm, NGN-M, is proved to converge at the standard $O(1/\sqrt{K})$ rate on convex $L$-smooth losses under only bounded variance of the per-batch optimal values, with no interpolation condition, no bounded stochastic gradients, and no bounded iterates — all of which earlier momentum-plus-Polyak-step-size analyses needed. Two coordinate-wise extensions, NGN-MDv1 and NGN-D, bring the same idea to the diagonal-preconditioning regime dominated by Adam. The paper acknowledges that the main theorem holds only for small momentum, while the recommended setting $\beta = 0.9$ lies outside the proven regime, with a one-dimensional stability result as the only theoretical bridge to large momentum. In experiments on CNNs, vision transformers, and transformers up to a billion parameters, the proposed optimizers tolerate step-sizes one to two orders of magnitude larger than the optimum, matching or beating Adam and Momo-Adam.

What carries the argument

Two devices carry the proof. The first is the Iterative Moving Average (IMA) reformulation of momentum: with $\beta = \lambda/(1+\lambda)$, the update is equivalent to $z_{k+1} = z_k - \gamma_k \nabla f_{S_k}(x_k)$ followed by $x_{k+1} = \frac{\lambda}{1+\lambda} x_k + \frac{1}{1+\lambda} z_{k+1}$, which turns a second-order recursion into a contraction inequality in squared distances to the optimum. The second is a splitting of the NGN step-size into a fixed baseline $\rho = c/((1+cL)(1+2cL))$ and an adaptive fluctuation $\tilde{\gamma}_k$ bounded by $O(c^2 L)$: the baseline drives the descent term, while the fluctuation contributes only noise terms weighted by $c^2$ times the variances $\sigma^2_{f*}$ and $\sigma^2_f$. The small-momentum condition $\lambda \le \min\{cL, \tfrac12(1+cL)^{-1}(1+2cL)^{-1}\}$ is exactly what makes $\rho - c\lambda > 0$, so the momentum-induced error telescopes away and the bound closes on the $O(1/\sqrt{K})$ rate.

What would settle it

Run NGN-M with $\beta = 0.9$ and $cL$ around 5 to 10 on a convex $L$-smooth problem with visible batch noise — for instance a separable quadratic with varied per-coordinate curvature and small mini-batches — and record whether the function values stay bounded over many iterations. The theorem's momentum condition fails for these settings, so divergence would refute the practical stability claim, while bounded convergence near the predicted $O(cL\sigma^2)$ neighborhood would indicate that the small-$\lambda$ requirement is an artifact of the proof technique, exactly as the paper conjectures.

Watch

Extended reading notes

Core claim

The central claim is that the NGN step-size, which in its scalar form reads $\gamma_k = c / (1 + \tfrac{c}{2 f_S(x_k)} \|\nabla f_S(x_k)\|^2)$ and automatically shrinks when the batch loss or the gradient is large, can be wrapped in the heavy-ball update $x_{k+1} = x_k - (1-\beta)\gamma_k \nabla f_{S_k}(x_k) + \beta (x_k - x_{k-1})$, and the resulting NGN-M provably satisfies $\mathbb{E}[f(\bar{x}_K) - f(x^*)] \le O(1/\sqrt{K})$ when $c = O(1/\sqrt{K})$. The theorem assumes each $f_i$ is convex and $L$-smooth, and that the variances $\sigma^2_{f*} = \mathbb{E}[(f^* - f^*_S)^2]$ and $\sigma^2_f = \mathbb{E}[(f^*_S)^2]$ of the per-batch optimal values are finite; it holds with $\beta = \lambda/(1+\lambda)$ and $\lambda \le \min\{cL, \tfrac12 (1+cL)^{-1}(1+2cL)^{-1}\}$. For any fixed $c$, the iterates provably reach and stay within a neighborhood of the optimum whose size scales with $c$ and $c^2$ times the two variances, so momentum inherits rather than destroys the non-divergence property of NGN. The paper also claims the first diagonal Polyak-type step-size, NGN-D, with an $O(1/\sqrt{K})$ guarantee under standard smoothness and variance assumptions, without interpolation or bounded gradients.

Load-bearing premise

The load-bearing premise is that the convergence proof only covers small momentum — with $\lambda \le \min\{cL, \tfrac12(1+cL)^{-1}(1+2cL)^{-1}\}$ and $\beta = \lambda/(1+\lambda)$ — whereas the momentum value used in every deep-learning experiment, $\beta = 0.9$ (that is, $\lambda = 9$), violates this condition for any realistic $cL$; the paper explicitly acknowledges this and supports large momentum only with a one-dimensional stability theorem, so the headline stability of the practical configuration rests on experiments rather than on the main proof.

Editorial extensions

If this is right

  • Users can run NGN-M on convex or convex-like problems without ever checking interpolation, gradient boundedness, or iterate boundedness: any convex $L$-smooth problem with finite per-batch optimal-value variance qualifies for the $O(1/\sqrt{K})$ guarantee.
  • Momentum does not ruin NGN's safety property: for every fixed step-size hyperparameter $c$, the iterates enter and stay in a neighborhood of the optimum whose radius vanishes as $c \to 0$, so a too-large $c$ degrades accuracy without causing divergence.
  • NGN-D extends the same guarantee to coordinate-wise (diagonal) step-sizes, the regime where Adam operates, and is claimed to be the first diagonal Polyak-type method with an $O(1/\sqrt{K})$ rate under standard smoothness and variance assumptions.
  • In the reported experiments, the stable step-size window of NGN-M and NGN-MDv1 is one to two orders of magnitude wider than that of SGDM, Adam, Momo, and Momo-Adam, with equal or better best-case accuracy or perplexity across CIFAR, ImageNet, and SlimPajama-trained transformers up to 1B parameters.
  • The effective step-size of NGN-MDv1 self-schedules: it rises, peaks early, and decays over training, resembling cosine annealing without any scheduler, which the paper identifies as the mechanism behind the resilience.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the small-$\beta$ restriction turns out to be an artifact, as the paper's own 1D result hints, then the practical recipe $\beta = 0.9$ would be backed by theory; the likely route is to exploit that near a minimizer the NGN step-size collapses to order $c/(1+cL)$, which would damp the momentum drift automatically.
  • The baseline-plus-fluctuation splitting is a transferable template: any adaptive step-size of the form $\rho + \tilde{\gamma}$ with $|\tilde{\gamma}| \le O(c^2 L)$ and a known lower bound could inherit the same momentum analysis, potentially covering RMSprop- or Adam-style rules.
  • The paper's own timing data show the naive implementation of NGN-MDv1 is about 2.5 times slower per update than a tuned AdamW implementation, leaving an open question the paper does not resolve: whether fused implementations or the wider stable hyperparameter range close that wall-clock gap in practice.
  • The empirical success at 1B scale suggests a workflow the paper does not spell out: fix one step-size hyperparameter per model family and scale model size without retuning, a form of hyperparameter transfer that the automatic shrinkage of the NGN step-size could provide for free.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes NGN-M, a momentum variant of the NGN adaptive step-size method, along with two coordinate-wise extensions (NGN-MDv1/v2) and a diagonal-only variant (NGN-D). The main theoretical claim is that under convex L-smooth losses and bounded variance of per-batch optimal values, NGN-M achieves the standard O(1/√K) convergence rate without interpolation, bounded stochastic gradients, or bounded iterates (Theorem 1, Section 4). This is supported by a proof using the iterative moving average (IMA) formulation and a decomposition of the NGN step-size into constant and fluctuating parts. The paper further gives a diminishing-step-size variant (Theorem 4, Appendix E) and a special 1D large-momentum stability result (Appendix F). Empirically, the authors evaluate NGN-M and NGN-MDv1 across CIFAR, ImageNet, and Transformer language models up to 1B parameters, demonstrating improved robustness to step-size hyperparameter choice and competitive or better performance than Adam, Momo, and related optimizers.

Significance. If the claims hold, this is a useful contribution to adaptive step-size methods: it extends the non-divergence and robustness properties of NGN to a momentum variant and to diagonal preconditioning, and it does so under weaker assumptions (no interpolation, no bounded gradients) than prior SPS-with-momentum analyses. The empirical study is unusually extensive for this line of work, covering up to 1B-parameter language models, and consistently shows that NGN-M/NGN-MDv1 tolerate a wider range of step-size hyperparameters than baselines. The proof of Theorem 1 is self-contained given standard smoothness and variance assumptions, and the paper transparently states its limitations, including the restricted momentum regime and the lack of large-scale parallelization support. The main weakness, discussed below, is that the central theoretical result is proven only for asymptotically vanishing momentum, whereas all experiments use the standard β=0.9, leaving the practically relevant regime supported only by experiments and a 1D special case.

major comments (2)
  1. [Section 4, Theorem 1; Eqs. (26)–(29)]
  2. [Section 5, first paragraph]
minor comments (5)
  1. [Section 3, Ver.1/Ver.2 design]
  2. [Appendix J, hyperparameters]
  3. [Section 4, Theorem 1 statement]
  4. [Appendix F]
  5. [Assumption 2]

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the NGN-M O(1/sqrt(K)) bound is derived from explicit smoothness and variance assumptions, with no fitted parameter renamed as a prediction.

full rationale

I walked the derivation chain of Theorem 1 (Section 4) and the supporting appendices. The NGN-M update is defined in Algorithm 1; the proof uses the IMA reformulation (Appendix A) and the algebraic identity gamma_k ||nabla f_S(x_k)||^2 = 2(c-gamma_k)/c f_S(x_k), which is a rearrangement of the definition of gamma_k, not an assumption of the conclusion. The step-size is decomposed as gamma_k = rho + delta gamma_k with rho = c/((1+cL)(1+2cL)), and the condition lambda <= min{cL, 0.5(1+cL)^{-1}(1+2cL)^{-1}} is used to keep the coefficient (rho - c lambda) positive in Eq. (26); this is a proof condition, not a parameter fitted to the target rate. The bound E[f(x_bar)-f(x*)] <= ... follows by telescoping and Jensen, with c = O(1/sqrt(K)) chosen in the theorem statement. Lemma B.6 is imported from Orvieto and Xiao (a co-author), but it is a parameter-free L-smoothness bound that does not contain the target rate, so it qualifies as independent support rather than a self-citation chain. The paper explicitly acknowledges that Theorem 1 covers only small beta and that the practically used beta = 0.9 is outside the proven regime (Section 4, item (iv); Appendix F), and it provides only a 1D large-beta stability result; this is a scope limitation, not circularity. No equation in the paper reduces the claimed O(1/sqrt(K)) rate to the definition of the algorithm or to a fitted input, and no empirical result is relabeled as a prediction. Score 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces algorithms, not new physical or mathematical entities. The central proof depends on standard convex smooth optimization assumptions plus a small-momentum technical condition. There are no invented mediators or new conserved quantities. The hyperparameters c, beta, and the diagonal preconditioner constants are the main free choices the user must set.

free parameters (4)
  • step-size hyperparameter c = tuned over powers of 10 in practice; theory sets c = O(1/sqrt(K)) or c_k = O(1/sqrt(k))
    The central step-size scale for NGN-M, NGN-MD, and NGN-D. The convergence rate and the size of the asymptotic neighborhood depend on c.
  • momentum parameter beta = 0.9 in experiments; theorem requires small beta (lambda small)
    Controls heavy-ball momentum. The main convergence theorem only covers beta small, while the empirical evaluation uses the standard beta = 0.9.
  • NGN-MD hyperparameters beta1, beta2, epsilon = defaults (0.9, 0.999) with unspecified epsilon
    Used in the RMSprop-style preconditioner of NGN-MDv1 and v2. No convergence proof is provided for these variants.
  • per-coordinate constants c_j in NGN-D = set via RMSprop preconditioner in experiments
    The theory assumes fixed c_j; the experiments set c_j = c/(v_j)^{1/2} adaptively, so the theoretical guarantee does not directly cover the practical choice.
assumptions (5)
  • domain assumption Each loss f_i is convex and L-smooth
    Theorem 1 relies on convexity and L-smoothness for the descent inequalities. Convexity of neural loss landscapes is debated, but the authors motivate it by prior empirical work.
  • domain assumption Losses are nonnegative and each batch problem has a finite optimal value
    The NGN step-size denominator uses f_S(x) and f_S*; nonnegativity is needed for the step-size bounds gamma in [c/(1+cL), c].
  • domain assumption Bounded variance of batch optima: E[f(x*) - f_S*] = sigma_f^2 < infinity and E[f_S*] = sigma_g^2 < infinity
    The rate contains these variance terms. If f_S* can be heavy-tailed or negative, the bound breaks and the convergence claim is not supported.
  • domain assumption Coordinate-wise smoothness (separability) for NGN-D
    The NGN-D theorem uses per-coordinate L_j smoothness and per-coordinate variance, which is stronger than standard L-smoothness.
  • ad hoc to paper Small momentum condition lambda <= min{cL, 0.5(1+cL)^{-1}(1+2cL)^{-1}}
    A technical restriction in the proof of Theorem 1 that makes the momentum error term negative. It excludes the practically used beta = 0.9.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size." pith.science (2026). https://pith.science/paper/ZRAJA2OL

@misc{pith2026250815071,
  author       = {Pith},
  title        = {Pith review of: Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZRAJA2OL}},
  note         = {Machine review of arXiv:2508.15071}
}
abstract

Modern optimization algorithms that incorporate momentum and adaptive step-size offer improved performance in numerous challenging deep learning tasks. However, their effectiveness is often highly sensitive to the choice of hyperparameters, especially the step-size. Tuning these parameters is often difficult, resource-intensive, and time-consuming. Therefore, recent efforts have been directed toward enhancing the stability of optimizers across a wide range of hyperparameter choices [Schaipp et al., 2024]. In this paper, we introduce an algorithm that matches the performance of state-of-the-art optimizers while improving stability to the choice of the step-size hyperparameter through a novel adaptation of the NGN step-size method [Orvieto and Xiao, 2024]. Specifically, we propose a momentum-based version (NGN-M) that attains the standard convergence rate of $\mathcal{O}(1/\sqrt{K})$ under less restrictive assumptions, without the need for interpolation condition or assumptions of bounded stochastic gradients or iterates, in contrast to previous approaches. Additionally, we empirically demonstrate that the combination of the NGN step-size with momentum results in enhanced robustness to the choice of the step-size hyperparameter while delivering performance that is comparable to or surpasses other state-of-the-art optimizers.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

100 extracted references · 36 canonical work pages

  1. [1]

    Why do we need weight decay in modern deep learning? arXiv preprint arXiv:2310.04415, 2023

    Maksym Andriushchenko, Francesco D'Angelo, Aditya Varre, and Nicolas Flammarion. Why do we need weight decay in modern deep learning? arXiv preprint arXiv:2310.04415, 2023

    arXiv 2023

  2. [2]

    Complexity guarantees for polyak steps with momentum

    Mathieu Barr\'e, Adrien Taylor, and Alexandre d'Aspremont. Complexity guarantees for polyak steps with momentum. In Proceedings of Thirty Third Conference on Learning Theory, 2020

    2020

  3. [3]

    signsgd: Compressed optimisation for non-convex problems

    Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, 2018

    2018

  4. [4]

    Gpt-neox-20b: An open-source autoregressive language model

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv: 2204.06745, 2022

    arXiv 2022

  5. [5]

    On the fast convergence of minibatch heavy ball momentum

    Raghu Bollapragada, Tyler Chen, and Rachel Ward. On the fast convergence of minibatch heavy ball momentum. arXiv preprint arXiv:2206.07553, 2022

    work page Pith review arXiv 2022

  6. [6]

    Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit

    Blake Bordelon, Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit. arXiv preprint arXiv:2309.16620, 2023

    arXiv 2023

  7. [7]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  8. [8]

    Symbolic discovery of optimization algorithms

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 2024

    2024

Show all 100 references
  1. [9]

    On empirical comparisons of optimizers for deep learning

    Dami Choi, Christopher J Shallue, Zachary Nado, Jaehoon Lee, Chris J Maddison, and George E Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint arXiv:1910.05446, 2019

    1910 arXiv

  2. [10]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 2023

    2023

  3. [11]

    Adaptive methods through the lens of SDE s: Theoretical insights on the role of noise

    Enea Monzio Compagnoni, Tianlin Liu, Rustem Islamov, Frank Norbert Proske, Antonio Orvieto, and Aurelien Lucchi. Adaptive methods through the lens of SDE s: Theoretical insights on the role of noise. In The Thirteenth International Conference on Learning Representations, 2025....

    2025

  4. [12]

    Momentum improves normalized sgd

    Ashok Cutkosky and Harsh Mehta. Momentum improves normalized sgd. In International conference on machine learning. PMLR, 2020

    2020

  5. [13]

    Momentum-based variance reduction in non-convex sgd

    Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd. Advances in neural information processing systems, 2019

    2019

  6. [14]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009

    2009

  7. [15]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  8. [16]

    Adaptive subgradient methods for online learning and stochastic optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 2011

    2011

  9. [17]

    Momentum provably improves error feedback! Advances in Neural Information Processing Systems, 2024

    Ilyas Fatkhullin, Alexander Tyurin, and Peter Richt \'a rik. Momentum provably improves error feedback! Advances in Neural Information Processing Systems, 2024

    2024

  10. [18]

    Lecture 6: Matrix norms and spectral radii

    Simon Foucart. Lecture 6: Matrix norms and spectral radii. lecture notes for the course NSTP187 at Drexel University, Philadelphia, PA, Fall, 2012, 2012

    2012

  11. [19]

    When and why momentum accelerates sgd: An empirical study

    Jingwen Fu, Bohan Wang, Huishuai Zhang, Zhizheng Zhang, Wei Chen, and Nanning Zheng. When and why momentum accelerates sgd: An empirical study. arXiv preprint arXiv:2306.09000, 2023

    2023 arXiv

  12. [20]

    Handbook of convergence theorems for (stochastic) gradient methods

    Guillaume Garrigos and Robert M Gower. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint arXiv:2301.11235, 2023

    2023 arXiv

  13. [21]

    Global convergence of the heavy-ball method for convex optimization

    Euhanna Ghadimi, Hamid Reza Feyzmahdavian, and Mikael Johansson. Global convergence of the heavy-ball method for convex optimization. In 2015 European control conference (ECC), 2015

    2015

  14. [22]

    Goodfellow, Yoshua Bengio, and Aaron Courville

    Ian J. Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016

    2016

  15. [23]

    Sgd for structured nonconvex functions: Learning rates, minibatching and interpolation

    Robert Gower, Othmane Sebbouh, and Nicolas Loizou. Sgd for structured nonconvex functions: Learning rates, minibatching and interpolation. In International Conference on Artificial Intelligence and Statistics, 2021

    2021

  16. [24]

    Analysis of an idealized stochastic polyak method and its application to black-box model distillation

    Robert M Gower, Guillaume Garrigos, Nicolas Loizou, Dimitris Oikonomou, Konstantin Mishchenko, and Fabian Schaipp. Analysis of an idealized stochastic polyak method and its application to black-box model distillation. arXiv preprint arXiv:2504.01898, 2025

    2025 arXiv

  17. [25]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016

    2016

  18. [26]

    Neural networks for machine learning lecture 6a overview of mini-batch gradient descent

    Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Lecture notes, 2012

    2012

  19. [27]

    Empirical tests of optimization assumptions in deep learning

    Tran Hoang, Qinzi Zhang, and Ashok Cutkosky. Empirical tests of optimization assumptions in deep learning. arXiv preprint arXiv:2407.01825, 2024

    2024

  20. [28]

    Long short-term memory

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997

    1997

  21. [29]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  22. [30]

    Loss landscape characterization of neural networks without over-parametrization

    Rustem Islamov, Niccol\' o Ajroldi, Antonio Orvieto, and Aurelien Lucchi. Loss landscape characterization of neural networks without over-parametrization. In Advances in Neural Information Processing Systems, 2024 a

    2024

  23. [31]

    Near optimal decentralized optimization with compression and momentum tracking

    Rustem Islamov, Yuan Gao, and Sebastian U Stich. Near optimal decentralized optimization with compression and momentum tracking. arXiv preprint arXiv:2405.2011, 2024 b

    2011

  24. [32]

    Double momentum and error feedback for clipping with fast rates and differential privacy

    Rustem Islamov, Samuel Horvath, Aurelien Lucchi, Peter Richtarik, and Eduard Gorbunov. Double momentum and error feedback for clipping with fast rates and differential privacy. arXiv preprint arXiv: 2502.11682, 2025

    2025

  25. [33]

    Accelerating stochastic gradient descent for least squares regression

    Prateek Jain, Sham M Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Accelerating stochastic gradient descent for least squares regression. In Conference On Learning Theory, 2018

    2018

  26. [34]

    Towards understanding how momentum improves generalization in deep learning

    Samy Jelassi and Yuanzhi Li. Towards understanding how momentum improves generalization in deep learning. In International Conference on Machine Learning, 2022

    2022

  27. [35]

    Adaptive sgd with polyak stepsize and line-search: Robust convergence and variance reduction

    Xiaowen Jiang and Sebastian U Stich. Adaptive sgd with polyak stepsize and line-search: Robust convergence and variance reduction. Advances in Neural Information Processing Systems, 2024

    2024

  28. [36]

    Muon: An optimizer for hidden layers in neural networks, 2024

    Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/

    2024

  29. [37]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv: 2001.08361, 2020

    2001 arXiv

  30. [38]

    char-rnn

    Andrej Karpathy. char-rnn. https://github.com/karpathy/char-rnn, 2015

    2015

  31. [39]

    Andrej Karpathy. Nanogpt. https://github.com/karpathy/nanoGPT, 2022

    2022

  32. [40]

    Adam: A method for stochastic optimization

    Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015

    2015

  33. [41]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Learning multiple layers of features from tiny images. Scientific Report, 2009

    2009

  34. [42]

    Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be

    Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. In The Eleventh International Conference on Learning Representations, 2023

    2023

  35. [43]

    Heavy-tailed class imbalance and why adam outperforms gradient descent on language models

    Frederik Kunstner, Robin Yadav, Alan Milligan, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why adam outperforms gradient descent on language models. arXiv preprint arXiv: 2402.19449, 2024

    2024 arXiv

  36. [44]

    Trajectory of mini-batch momentum: batch size saturation and convergence in high dimensions

    Kiwon Lee, Andrew Cheng, Elliot Paquette, and Courtney Paquette. Trajectory of mini-batch momentum: batch size saturation and convergence in high dimensions. Advances in Neural Information Processing Systems, 2022

    2022

  37. [45]

    Convergence of adam under relaxed assumptions

    Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie. Convergence of adam under relaxed assumptions. Advances in Neural Information Processing Systems, 2024

    2024

  38. [46]

    Pytorch distributed: Experiences on accelerating data parallel training

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704, 2020

    2006 arXiv

  39. [47]

    Sp2: A second order stochastic polyak method

    Shuang Li, William J Swartworth, Martin Tak \'a c , Deanna Needell, and Robert M Gower. Sp2: A second order stochastic polyak method. arXiv preprint arXiv:2207.08171, 2022

    2022 arXiv

  40. [48]

    An improved analysis of stochastic gradient descent with momentum

    Yanli Liu, Yuan Gao, and Wotao Yin. An improved analysis of stochastic gradient descent with momentum. Advances in Neural Information Processing Systems, 2020

    2020

  41. [49]

    Stochastic polyak step-size for sgd: An adaptive learning rate for fast convergence

    Nicolas Loizou, Sharan Vaswani, Issam Hadj Laradji, and Simon Lacoste-Julien. Stochastic polyak step-size for sgd: An adaptive learning rate for fast convergence. In International Conference on Artificial Intelligence and Statistics, 2021

    2021

  42. [50]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 1711.05101, 2019

    2019 arXiv

  43. [51]

    Adaptive gradient methods with dynamic bound of learning rate

    Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. arXiv preprint arXiv:1902.09843, 2019

    1902 arXiv

  44. [52]

    Quasi-hyperbolic momentum and adam for deep learning

    Jerry Ma and Denis Yarats. Quasi-hyperbolic momentum and adam for deep learning. arXiv preprint arXiv:1810.06801, 2018

    2018 arXiv

  45. [53]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In Proceedings of 4th International Conference on Learning Representations (ICLR 2016), 2016

    2016

  46. [54]

    Recurrent neural network based language model

    Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. Recurrent neural network based language model. Proceedings of the 11th Annual Conference of the International Speech Communication Association, INTERSPEECH 2010, 2010

    2010

  47. [55]

    A theory on adam instability in large-scale machine learning

    Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, et al. A theory on adam instability in large-scale machine learning. arXiv preprint arXiv:2304.09871, 2023

    2023 arXiv

  48. [56]

    Signal propagation in transformers: Theoretical perspectives and the role of rank collapse

    Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 2022

    2022

  49. [57]

    Stochastic polyak step-sizes and momentum: Convergence guarantees and practical performance

    Dimitris Oikonomou and Nicolas Loizou. Stochastic polyak step-sizes and momentum: Convergence guarantees and practical performance. arXiv preprint arXiv:2406.04142, 2024

    2024 arXiv

  50. [58]

    The cost of training nlp models: A concise overview

    Sharir Or, Barak Peleg, and Yoav Shoham. The cost of training nlp models: A concise overview. arXiv preprint arXiv:2004.08900, 2020

    2004 arXiv

  51. [59]

    An adaptive stochastic gradient method with non-negative gauss-newton stepsizes

    Antonio Orvieto and Lin Xiao. An adaptive stochastic gradient method with non-negative gauss-newton stepsizes. arXiv preprint arXiv: 2407.04358, 2024

    2024 arXiv

  52. [60]

    Dynamics of sgd with stochastic polyak stepsizes: Truly adaptive variants and convergence to exact solution

    Antonio Orvieto, Simon Lacoste-Julien, and Nicolas Loizou. Dynamics of sgd with stochastic polyak stepsizes: Truly adaptive variants and convergence to exact solution. Advances in Neural Information Processing Systems, 2022

    2022

  53. [61]

    Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales

    Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACL, 2005

    2005

  54. [62]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. NIPS 2017 Workshop Autodiff, 2017

    2017

  55. [63]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 2024

    2024

  56. [64]

    Boris T. Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics, 1964

    1964

  57. [65]

    Introduction to optimization

    Boris T Polyak. Introduction to optimization. New York, Optimization Software, 1987

    1987

  58. [66]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. Technical report, OpenAI, 2019

    2019

  59. [67]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1--16. IEEE, 2020

    2020

  60. [68]

    Local curvature descent: Squeezing more curvature out of standard and polyak gradient descent

    Peter Richt \'a rik, Simone Maria Giancola, Dymitr Lubczyk, and Robin Yadav. Local curvature descent: Squeezing more curvature out of standard and polyak gradient descent. arXiv preprint arXiv:2405.16574, 2024

    2024 arXiv

  61. [69]

    An adaptive polyak heavy-ball method

    Samer Saab, Shashi Phoha, Minghui Zhu, and Asok Ray. An adaptive polyak heavy-ball method. Machine Learning, 2022

    2022

  62. [70]

    Stochastic sign descent methods: New algorithms and better theory

    Mher Safaryan and Peter Richt \'a rik. Stochastic sign descent methods: New algorithms and better theory. In International Conference on Machine Learning, 2021

    2021

  63. [71]

    Fabian Schaipp, Ruben Ohana, Michael Eickenberg, Aaron Defazio, and Robert M. Gower. M o M o: Momentum models for adaptive learning rates. In Proceedings of the 41st International Conference on Machine Learning, 2024

    2024

  64. [72]

    The surprising agreement between convex optimization theory and learning-rate scheduling for large model training

    Fabian Schaipp, Alexander H \"a gele, Adrien Taylor, Umut Simsekli, and Francis Bach. The surprising agreement between convex optimization theory and learning-rate scheduling for large model training. arXiv preprint arXiv:2501.18965, 2025

    2025 arXiv

  65. [73]

    Almost sure convergence rates for stochastic gradient descent and stochastic heavy ball

    Othmane Sebbouh, Robert M Gower, and Aaron Defazio. Almost sure convergence rates for stochastic gradient descent and stochastic heavy ball. In Conference on Learning Theory, 2021

    2021

  66. [74]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv: 2002.05202, 2020

    2002 arXiv

  67. [75]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  68. [76]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  69. [77]

    J. W. Smith, J. E. Everhart, W. C. Dickson, W. C. Knowler, and R. S. Johannes. Using the adap learning algorithm to forecast the onset of diabetes mellitus. In Symposium on Computer Applications and Medical Care, 1988

    1988

  70. [78]

    SlimPajama: A 627B token cleaned and deduplicated version of RedPajama , 2023

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama , 2023

    2023

  71. [79]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023

    2023

  72. [80]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv \'e J \'e gou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, 2021

    2021

  73. [81]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023 arXiv

  74. [82]

    Closing the gap between the upper bound and lower bound of adam's iteration complexity

    Bohan Wang, Jingwen Fu, Huishuai Zhang, Nanning Zheng, and Wei Chen. Closing the gap between the upper bound and lower bound of adam's iteration complexity. Advances in Neural Information Processing Systems, 2024

    2024

  75. [83]

    A modular analysis of provable acceleration via polyak’s momentum: Training a wide relu network and a deep linear network

    Jun-Kun Wang, Chi-Heng Lin, and Jacob D Abernethy. A modular analysis of provable acceleration via polyak’s momentum: Training a wide relu network and a deep linear network. In International Conference on Machine Learning, pages 10816--10827. PMLR, 2021

    2021

  76. [84]

    Provable acceleration of heavy ball beyond quadratics for a class of polyak-lojasiewicz functions when the non-convexity is averaged-out

    Jun-Kun Wang, Chi-Heng Lin, Andre Wibisono, and Bin Hu. Provable acceleration of heavy ball beyond quadratics for a class of polyak-lojasiewicz functions when the non-convexity is averaged-out. In International conference on machine learning, 2022

    2022

  77. [85]

    Generalized polyak step size for first order optimization with momentum

    Xiaoyu Wang, Mikael Johansson, and Tong Zhang. Generalized polyak step size for first order optimization with momentum. In International Conference on Machine Learning, 2023

    2023

  78. [86]

    Adagrad stepsizes: Sharp convergence over nonconvex landscapes

    Rachel Ward, Xiaoxia Wu, and Leon Bottou. Adagrad stepsizes: Sharp convergence over nonconvex landscapes. Journal of Machine Learning Research, 2020

    2020

  79. [87]

    Pytorch image models

    Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2019

    2019

  80. [88]

    The marginal value of adaptive gradient methods in machine learning

    Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. Advances in neural information processing systems, 2017

    2017

  81. [89]

    Small-scale proxies for large-scale transformer training instabilities

    Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309.14322, 2023

    2023 arXiv

  82. [90]

    Rethinking conventional wisdom in machine learning: From generalization to scaling

    Lechao Xiao. Rethinking conventional wisdom in machine learning: From generalization to scaling. arXiv preprint arXiv: 2409.15156, 2024

    2024 arXiv

  83. [91]

    Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer

    Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022

    2022 arXiv

  84. [92]

    A spectral condition for feature learning

    Greg Yang, James B Simon, and Jeremy Bernstein. A spectral condition for feature learning. arXiv preprint arXiv:2310.17813, 2023

    2023 arXiv

  85. [93]

    Root mean square layer normalization, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019

    2019

  86. [94]

    Three mechanisms of weight decay regularization

    Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv:1810.12281, 2018

    2018 arXiv

  87. [95]

    Why gradient clipping accelerates training: A theoretical justification for adaptivity

    Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. arXiv preprint arXiv:1905.11881, 2019

    1905 arXiv

  88. [96]

    Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 2020

    Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 2020

    2020

  89. [97]

    Convergence guarantees for rmsprop and adam in generalized-smooth non-convex optimization with affine noise variance

    Qi Zhang, Yi Zhou, and Shaofeng Zou. Convergence guarantees for rmsprop and adam in generalized-smooth non-convex optimization with affine noise variance. arXiv preprint arXiv:2404.01436, 2024 a

    2024 arXiv

  90. [98]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  91. [99]

    Why transformers need adam: A hessian perspective

    Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. Why transformers need adam: A hessian perspective. arXiv preprint arXiv:2402.16788, 2024 b

    2024 arXiv

  92. [100]

    Adabelief optimizer: Adapting stepsizes by the belief in observed gradients

    Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. In Advances in Neural Information Processing Systems, 2020

    2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.