Pith. sign in

REVIEW 4 major objections 5 minor 60 references

SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a single-momentum-coefficient, p-th order sign-like update matches AdamW quality in about half the steps while suppressing loss spikes.

desk verdict Plaustible optimizer and a direct test of the loss-spike story, but the paper's headline convergence theorem is mathematically broken as written and the experiments lack reproducibility. read the letter →

arxiv 2507.06464 v1 pith:WDAQA53J submitted 2025-07-09 cs.LG cs.AI

Hanyang Peng , Shuang Qin , Yue Yu , Fangqing Jiang , Hui Wang , Wen Gao This is my paper
classification cs.LGcs.AI MSC 68T0790C26
keywords SoftSignSGDAdamdescentlossspikesoptimizerdesignnonconvexstochasticoptimizationNesterovaccelerationlargelanguagemodeltraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that Adam's practical strength is not second-order preconditioning but its sign-like update, and that its weakness, sudden loss spikes in large-model training, comes from coordinates whose update magnitude grows uncontrolled. To keep the benefit and remove the failure, the authors introduce SoftSignSGD (S3), whose update is the ratio of a first-order Nesterov momentum to a p-th order momentum of absolute gradients, with both movers using the same averaging coefficient $\beta$. Setting beta1=beta2 forces every coordinate update to lie in [-1,1], so the maximum possible step size is capped. The authors prove an O(1/$T^{{1/4}}$) convergence rate for general nonconvex stochastic objectives under a non-uniform smoothness assumption, and report that on ResNet-50, ViT-B/16, GPT-2 (345M), and GPT-2 (7B), S3 reaches AdamW-quality results in about half the steps without loss spikes.

What carries the argument

The load-bearing object is the S3 update $u_t = n_t / b_t(p)$, with $n_t = \beta m_t + (1-\beta) g_t$ and $b_t(p) = (\beta s_{t-1} + (1-\beta)|g_t|^p)^{1/p}$, using a single EMA coefficient $\beta$ for both accumulator pairs. Theorem 2 shows that when the numerator and denominator share $\beta$, each coordinate of $u_t$ is bounded in $[-1,1]$; this bound is what the paper invokes to suppress loss spikes, to remove bias correction and clipping, and to enable larger learning rates. The Nesterov equivalence (Theorem 3) shows that this formulation is computation-free NAG, needing no extra memory. Theorem 4 then bounds the averaged $\ell^1$ gradient norm by $O(1/T^{1/4})$ under a non-uniform smoothness condition, provided every coordinate of $u_t$ stays above a positive floor $1/U_{\max}$, which supplies the optimal nonconvex stochastic rate according to the paper's lower-bound match.

What would settle it

Run Algorithm 1 on the GPT-2 (7B) setup while recording, at every step, the coordinate-wise ratio $|n_t^{(j)}|/b_t^{(j)}(p)$; if the minimum over coordinates ever falls below the theorem's required floor $1/U_{\max}$, the $O(1/T^{1/4})$ bound's premise is violated. A second check is to reproduce the claimed loss-spike suppression at a $10\times$ learning rate and count spike events against the reported behavior.

Watch

Extended reading notes

Core claim

On the paper's terms, the central claim is that a sign-like update, not variance preconditioning, is the load-bearing ingredient in Adam, and that the same sign-like update can be made safe by bounding it. S3 keeps the quasi-binary behavior of $m_t/\sqrt{v_t}$ but replaces the $\sqrt{v_t}$ denominator with $(\beta s_{t-1} + (1-\beta)|g_t|^p)^{1/p}$, a p-th order momentum of absolute gradients, and uses the same $\beta$ for numerator and denominator. The paper shows that with $\beta_1=\beta_2$ the per-coordinate ratio $|n_t|/b_t(p)$ is at most 1, eliminating the large-update mechanism it identifies as the cause of loss spikes; a larger $p$ also gives a smaller denominator bound, which the paper connects to the ability to use larger learning rates. Experimentally the paper reports that S3 matches or beats AdamW at $2\times$ fewer steps on ImageNet and GPT-2 pretraining, that it rarely spikes even at $10\times$ the learning rate, and that its Nesterov formulation costs no extra memory.

Load-bearing premise

The convergence theorem assumes that at every step every coordinate of the normalized update stays at least $1/U_{\max}$ above zero; the algorithm does not enforce this, and a coordinate whose gradient or momentum crosses zero would violate it.

Editorial extensions

If this is right

  • S3 removes bias correction and gradient clipping and drops one hyperparameter, since the shared-$\beta$ design already caps every update.
  • If the empirical claim holds, LLM pretraining with S3 reaches AdamW-quality losses in roughly half the steps, a direct compute saving.
  • The $[-1,1]$ coordinate bound implies a hard ceiling on any single parameter's step, which the paper links to the absence of loss spikes even at $10\times$ the learning rate.
  • A larger $p$ gives a smaller denominator bound (Theorem 2(3)), which the paper connects to stable use of larger learning rates and faster early convergence.
  • The theorem's $O(1/T^{1/4})$ rate is presented as optimal for nonconvex stochastic optimization under non-uniform smoothness, matching the lower bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: instrument a standard S3 run to record $\min_{j,t} |n_t^{(j)}|/b_t^{(j)}(p)$; the theorem's premise requires this to stay above a positive floor, so the run would either confirm the assumption or show the published rate is not actually in force.
  • If the sign-like mechanism is the full story, a natural next step is to schedule $p$ or $\beta$ per layer rather than globally; the paper's own analysis suggests layers have very different gradient scales.
  • A port of the shared-$\beta$ cap to existing adaptive methods (or to low-precision and quantized training) could generalize the loss-spike suppression the paper demonstrates for S3.
  • Should the reported speedups hold at larger scale, the practice of checkpoint-and-rollback around spikes, and the extra gradient-reduction heuristics used in LLM training, may become unnecessary for S3 runs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes SoftSignSGD (S3), an optimizer that replaces Adam's second-moment denominator with a p-th order EMA of |g|^p, uses a single EMA coefficient beta for numerator and denominator so that coordinatewise updates are bounded by 1, and adds a Nesterov acceleration step. The authors argue that Adam's practical success is due to its sign-like behavior and that its instability is caused by occasional large updates, and they claim (Theorem 4) an O(1/T^{1/4}) convergence rate for S3 under Assumptions 1-4. Empirically, they compare S3 with AdamW, SGDM, NAdam, Adan, and Lion on ImageNet (ResNet-50, ViT-B/16) and GPT-2 (345M, 7B), reporting faster convergence, better final accuracy/perplexity, and fewer loss spikes, including at a 10x larger learning rate.

Significance. If the theoretical and empirical claims held, S3 would be a valuable contribution: it would provide an adaptive sign-based optimizer with bounded updates, potentially halving LLM pretraining compute while reducing loss spikes. The paper also contains a genuine strength: Section VI.F and Figure 10 offer a direct, independent test of the proposed loss-spike mechanism by clipping Adam updates and using equal betas, rather than relying solely on the success of the proposed optimizer. However, the central convergence theorem is not established, and the experimental section lacks error bars, seeds, code, and data. The practical claims are therefore not currently verifiable.

major comments (4)
  1. [Section V, Theorem 4; Appendix E] The theorem's hypothesis 'if u_t = |n_t^{(j)}| / b_t^{(j)} >= 1/Umax' is an unenforced trajectory condition, not a consequence of Algorithm 1 or of any stated problem assumption. The update n_t^{(j)} can be zero or arbitrarily small whenever the EMA of past gradients crosses zero, and the denominator b_t(p) can vanish when gradients vanish, so ordinary training runs violate the premise. Assumption 4, which the theorem invokes, is never stated, and the proof's u_min is defined only through this condition. Consequently, the advertised O(1/T^{1/4}) rate is not applicable to S3 as implemented; the stress-test concern is confirmed by the manuscript text itself.
  2. [Appendix E, Eq. (38)] The proof's threshold on T is T >= max{(2dL1/(L0 umin))^{4/3}, (8 beta^2 sqrt(d) L1 / ((1-beta) L0 umin))^4}. Substituting the theorem's own hyperparameter beta = 1 - 1/sqrt(T) makes the second term equal to C^4 T^2 with C = 8 sqrt(d) L1/(L0 umin), since 1/(1-beta) = sqrt(T). No positive T can satisfy T >= C^4 T^2 whenever C > 0, so the guarantee is vacuous even if the unenforced u_min assumption is granted. The optimal-rate claim is therefore not derived by the printed proof.
  3. [Algorithm 1] The update rule x_{t+1} = x_t - gamma_t n_t / b_t(p) divides by b_t(p) = (beta s_t(p) + (1-beta)|g_t|^p)^{1/p} with no epsilon floor. Along any coordinate where all recent gradients are exactly zero, b_t^{(j)} = 0 and the update is undefined. This is not a minor implementation detail: it occurs in the stationary regime that the convergence theorem concerns, and it also means the assumption u_t^{(j)} >= 1/Umax cannot hold for such coordinates.
  4. [Section VI, Tables I-II] The empirical results are reported as single runs without error bars, multiple seeds, or released code and data. Given the strength of the claims (e.g., S3 at 50k steps improves over AdamW at 100k steps on GPT-2 345M, and S3 uses a 10x larger learning rate), the absence of standard reproducibility evidence prevents independent verification. The downstream evaluation paragraph in VI.C already concedes randomness for the 345M model, but the main tables report only point estimates.
minor comments (5)
  1. [Throughout] There are numerous typos and grammatical errors, including 'Adamhas proven remarkable successful' in the abstract, 'generatedS3' in Theorem 2, 'of ar beta_1 and beta_2' in Section IV, 'ReNet-50' and 'NAadm' in Figure 4, and 'iteLion2023' in Appendix F.
  2. [Figure 7] The legend contains a duplicate entry for 'S3, lr=6e-3, same , w/o NAG, p=2', which should be corrected.
  3. [Section III, Theorem 1] Theorem 1's bound in Eq. (3) is deterministic, but the text describes the risk of large updates in probabilistic terms ('with non-trivial probability') without specifying a distributional model; this informal language should be made precise or replaced.
  4. [Section VII] The conclusion's claim that the paper provides 'the first theoretical proof of adaptive optimizer convergence from the perspective of sign descent' is overstated given the existing literature on sign-based methods cited in Section II (e.g., [24], [25]).
  5. [Theorem 4] The theorem statement should be self-contained: Assumption 4 is referenced but never defined, and u_min is only introduced informally in the proof. This must be fixed even if the substantive proof issues are addressed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the theoretical and empirical claims are derived from stated assumptions and independent benchmarks, and the proof defects identified are correctness gaps rather than input-output equivalences.

full rationale

The paper's central derivation chain is not circular. S3's bounded-update property is proved in Theorem 2 from the algorithm's own update rule and the choice of a shared EMA coefficient; it is not assumed as the conclusion. The claimed O(T^{-1/4}) rate in Theorem 4 is conditioned on the explicit hypothesis u_t^(j) >= 1/U_max for every coordinate at every step, and the proof derives the gradient-norm bound from Assumptions 1-3 plus that hypothesis. The lower-bound condition is not enforced by Algorithm 1 and can fail when a coordinate momentum crosses zero, so the theorem is conditional and likely vacuous for ordinary runs; that is a correctness defect, not circularity. Similarly, the threshold before Eq. (38), combined with beta = 1 - 1/sqrt(T) and gamma = 1/(L0 T^{3/4}), becomes T >= C^4 T^2 for a constant C, making the printed proof internally inconsistent; again this is a proof error, not a circular reduction. Theorem 4 also cites Assumption 4, which is never stated in the paper; that is a missing-support defect, not a self-referential derivation. There are no load-bearing self-citations: the cited lower bound [57], the Adam and Lion results, and the loss-spike references are all external works, and no argument reduces to the present authors' prior results. The experimental loss-spike mechanism is tested directly in Figure 10 by comparing Adam with clipping and Adam with equal betas, which is an independent falsifiable check and does not assume S3's convergence theorem. The empirical claim that S3 often matches AdamW at 2x steps is supported by the reported benchmark comparisons and is not derived by fitting a parameter and then renaming it as a prediction. Overall, no step in the paper's derivation is equivalent by construction to its own input, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The proof rests on four assumptions, three of which are standard. The fourth, a uniform lower bound on every update coordinate, is an ad hoc condition that is not guaranteed by the algorithm and is not verified empirically. The free parameters p and beta are tuned by hand, and the theory's beta differs from the experimental beta.

free parameters (3)
  • p (momentum order) = 3
    Chosen by coarse grid search on ViT-B/16 (Figure 9) and transferred to other tasks; the theory allows arbitrary p.
  • beta (EMA coefficient) = 0.95
    Selected from the same grid search; the convergence theorem instead sets beta = 1 - 1/sqrt(T), which is not what experiments use.
  • Umax (lower bound on update magnitude) = Not estimated
    Theorem 4 assumes every coordinate update stays above 1/Umax. The paper does not estimate this or show it holds, so the rate bound is conditional on an unverified constant.
assumptions (4)
  • standard math Objective F has a finite lower bound F* (Assumption 1).
    Standard for nonconvex optimization lower-bound analysis.
  • domain assumption Generalized smoothness: ||grad F(y) - grad F(x)|| <= (L0 + L1||grad F(x)||) ||y-x|| for ||y-x|| <= R (Assumption 2).
    Weaker than uniform smoothness, but non-standard and used to get the L1-norm bound in Lemma 5.
  • domain assumption Stochastic gradients are unbiased with bounded second moment (Assumption 3).
    Standard in stochastic optimization analysis.
  • ad hoc to paper Every coordinate update satisfies u_t^{(j)} >= 1/Umax for all t and j (Theorem 4).
    Introduced specifically to make the proof close; it is not implied by the algorithm and can be violated for near-zero gradient components.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam." pith.science (2026). https://pith.science/paper/WDAQA53J

@misc{pith2026250706464,
  author       = {Pith},
  title        = {Pith review of: SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WDAQA53J}},
  note         = {Machine review of arXiv:2507.06464}
}
abstract

Adam has proven remarkable successful in training deep neural networks, but the mechanisms underlying its empirical successes and limitations remain underexplored. In this study, we demonstrate that the effectiveness of Adam stems largely from its similarity to SignSGD in robustly handling large gradient fluctuations, yet it is also vulnerable to destabilizing loss spikes due to its uncontrolled update scaling. To enhance the advantage of Adam and mitigate its limitation, we propose SignSoftSGD (S3), a novel optimizer with three key innovations. \emph{First}, S3 generalizes the sign-like update by employing a flexible $p$-th order momentum ($p \geq 1$) in the denominator, departing from the conventional second-order momentum (variance) preconditioning. This design enables enhanced performance while achieving stable training even with aggressive learning rates. \emph{Second}, S3 minimizes the occurrences of loss spikes through unified exponential moving average coefficients for numerator and denominator momenta, which inherently bound updates to $[-1, 1]$ and simplify hyperparameter tuning. \emph{Third}, S3 incorporates an equivalent Nesterov's accelerated gradient(NAG) module, accelerating convergence without memory overhead. Theoretically, we prove that S3 achieves the optimal convergence rate of $O\left(\frac{1}{T^{\sfrac{1}{4}}}\right)$ for general nonconvex stochastic optimization under weak assumptions. Extensive experiments across a range of vision and language tasks show that \textsf{\small S3} not only converges more rapidly and improves performance but also rarely experiences loss spikes, even with a \textbf{$\bm{10 \times}$} larger learning rate. In fact, S3 delivers performance comparable to or better than AdamW with \textbf{$2 \times$} the training steps, establishing its efficacy in both efficiency and final task performance.

Figures

Figures reproduced from arXiv: 2507.06464 by the authors.

Figure 1
Figure 1. The (a) trajectories, (b) loss convergence curves, and (c) mean update curves of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Visualization of gradient norms within different layers in ViT-B/16 at initialization. While Adam effectively trains complex DNNs, it also esca￾lates the risk of training instability and loss spikes with non￾trivial probability. This can be inferred from Theorem 1. Theorem 1. The sequences {mt} and {vt} are generated by Adam in Eq. (2). If the moving average coefficients satisfy β 2 1 < β2, then it holds that |m (j … view at source ↗
Figure 3
Figure 3. Visualization of the mean update (i.e., Avg(|m(j) t |/ q v (j) t )), the maximum update (i.e., maxj∈[d](|m(j) t |/ q v (j) t )), and the training loss over 50,000 iterations during GPT-2 (345M) training on OpenWebText using AdamW (β1 = 0.9, β2 = 0.999) with a cosine learning rate schedule. The figure illustrates that all loss spikes are preceded by abrupt increases in the mean update, following a sharp rise in the m… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Comparison of train loss and test accuracy on ImageNet for training ReNet-50 and ViT-B/16 with AdamW, SGDM, NAadm, Adan, Lion and S3. 0 25k 50k 75k 100k Iter 1.5 1.8 2.1 Train Loss AdamW, iter=100k, lr=3e-4 AdamW, iter=50k, lr=3e-4 NAdam, iter=50k, lr=1e-3 Adan, iter=5…
Figure 5
Figure 5. Figure 5: Comparison of train loss and validation loss for pre-training GPT-2 (345M) and GPT-2 (7B) with [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Zero-shot evaluation of the pre-trained GPT-2 (345M) and GPT-2 (7B) with AdamW, Adan, aLion, and S3 on downstream reasoning tasks. accuracies that are 1.47% and 1.36% higher when training ResNet-50 and ViT-B, respectively. This represents a sub￾stantial improvement in …
Figure 7
Figure 7. Figure 7: Ablation study on train loss of S3 for training ViT-B/16 on ImageNet. 0 10k 20k 30k 40k 50k Iter 1.5 1.8 2.1 2.4 Train Loss AdamW, lr=3e-4 AdamW, lr=1e-3 S3, lr=3e-4, same , w/o NAG, p=2 S3, lr=1e-3, same , w/o NAG, p=2 S3,lr=1e-3, same , w/ NAG, p=2 S3,lr=3e-3, same ,…
Figure 8
Figure 8. Figure 8: Ablation study on train loss of S3 for training GPT-2(345M) on OpenWebText. p=1 p=2 p=3 p=4 =0.90 =0.95 =0.99 80.27 80.35 80.59 80.51 80.62 80.82 80.93 80.84 80.24 80.49 80.47 80.28 [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Impact of the momentum order (p) and the momentum coefficient (β) on the Accuracy of S3 training ViT-B/16 on ImageNet. We implement ablation experiments for training ViT-B/16 to clarify the contributions of each modification of S3 over Adam [PITH_FULL_IMAGE:figures/fu…
Figure 10
Figure 10. Figure 10: The Loss spikes phenomenons during Training GPT-2 (345M) on OpenWebText using AdamW. β2 completely eliminates loss spikes. This further confirms the validity of our analysis in Sections 3 and 4. VII. CONCLUSION AND DISCUSSION In this paper, we thoroughly examine the s…
Figure 11
Figure 11. Figure 11: Search the optimal peak learning rates for AdamW, Adan, Lion and S3 pre-training GPT-2 (345M) on OpenWebText. 1 T X T t=1 E[∥∇F(xt)∥1] ≤ 2L0Umax(F(x1) − F(x ∗ )) T 1/4 + 4βUmax √ dE [∥∇F(x1)∥2 ] T 1/2 + 4Umax √ dσ T 1/4 + 4β 2Umaxd T 1/4 + Umaxd T 7/4 . (39) F. Detail…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 25 canonical work pages

  1. [1]

    Adam: A method for stochastic optimization,

    D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” inInternational Conference on Learning Representations (ICLR), 2015

    work page 2015

  2. [2]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017. JOURNAL OF LATEX CLASS FILES, 2025 11

    work page 2017

  3. [3]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language mod- els are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  4. [4]

    Palm: Scal- ing language modeling with pathways,

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmannet al., “Palm: Scal- ing language modeling with pathways,”Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023

    2023

  5. [5]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  6. [6]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    arXiv 2024

  7. [7]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning, 2021, pp. 8748–8763

    2021

  8. [8]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Loet al., “Segment anything,” arXiv preprint arXiv:2304.02643, 2023

    arXiv 2023

Show all 60 references
  1. [9]

    A convnet for the 2020s,

    Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 11 976–11 986

    2022

  2. [10]

    Convnext v2: Co-designing and scaling convnets with masked autoen- coders,

    S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, “Convnext v2: Co-designing and scaling convnets with masked autoen- coders,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 16 133–16 142

    2023

  3. [11]

    Imagenet classification with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,”Communications of the ACM, vol. 60, no. 6, pp. 84–90, 2017

    2017

  4. [12]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inIEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778

    2016

  5. [13]

    Symbolic discovery of opti- mization algorithms,

    X. Chen, C. Liang, D. Huang, E. Real, K. Wang, Y . Liu, H. Pham, X. Dong, T. Luong, C.-J. Hsiehet al., “Symbolic discovery of opti- mization algorithms,”arXiv preprint arXiv:2302.06675, 2023

    2023 arXiv

  6. [14]

    Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be,

    F. Kunstner, J. Chen, J. W. Lavington, and M. Schmidt, “Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be,”arXiv preprint arXiv:2304.13960, 2023

    2023 arXiv

  7. [15]

    Adaptive subgradient methods for online learning and stochastic optimization

    J. Duchi, E. Hazan, and Y . Singer, “Adaptive subgradient methods for online learning and stochastic optimization.”Journal of machine learning research, vol. 12, no. 7, 2011

    2011

  8. [16]

    Neural networks for machine learning lecture 6a overview of mini-batch gradient descent,

    G. Hinton, N. Srivastava, and K. Swersky, “Neural networks for machine learning lecture 6a overview of mini-batch gradient descent,”Cited on, vol. 14, no. 8, p. 2, 2012

    2012

  9. [17]

    Adadelta: an adaptive learning rate method,

    M. D. Zeiler, “Adadelta: an adaptive learning rate method,”arXiv preprint arXiv:1212.5701, 2012

    2012 arXiv

  10. [18]

    Incorporating nesterov momentum into adam,

    T. Dozat, “Incorporating nesterov momentum into adam,” 2016

    2016

  11. [19]

    On the convergence of adam and beyond,

    S. J. Reddi, S. Kale, and S. Kumar, “On the convergence of adam and beyond,” inInternational Conference on Learning Representations, 2018

    2018

  12. [20]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  13. [21]

    Adabelief optimizer: Adapting stepsizes by the belief in observed gradients,

    J. Zhuang, T. Tang, Y . Ding, S. C. Tatikonda, N. Dvornek, X. Pa- pademetris, and J. Duncan, “Adabelief optimizer: Adapting stepsizes by the belief in observed gradients,” inAdvances in neural information processing systems, 2020, pp. 18 795–18 806

    2020

  14. [22]

    Adafactor: Adaptive learning rates with sub- linear memory cost,

    N. Shazeer and M. Stern, “Adafactor: Adaptive learning rates with sub- linear memory cost,” inInternational Conference on Machine Learning, 2018, pp. 4596–4604

    2018

  15. [23]

    1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns,

    F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, “1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns,” inFifteenth annual conference of the international speech communication association, 2014

    2014

  16. [24]

    Signsgd: Compressed optimisation for non-convex problems,

    J. Bernstein, Y .-X. Wang, K. Azizzadenesheli, and A. Anandkumar, “Signsgd: Compressed optimisation for non-convex problems,” inIn- ternational Conference on Machine Learning, 2018, pp. 560–569

    2018

  17. [25]

    Momentum ensures convergence of SIGNSGD under weaker assumptions,

    T. Sun, Q. Wang, D. Li, and B. Wang, “Momentum ensures convergence of SIGNSGD under weaker assumptions,” inInternational Conference on Machine Learning, 2023, pp. 33 077–33 099

    2023

  18. [26]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training,

    H. Liu, Z. Li, D. Hall, P. Liang, and T. Ma, “Sophia: A scalable stochastic second-order optimizer for language model pre-training,” arXiv preprint arXiv:2305.14342, 2023

    2023 arXiv

  19. [27]

    A direct adaptive method for faster backpropagation learning: The rprop algorithm,

    M. Riedmiller and H. Braun, “A direct adaptive method for faster backpropagation learning: The rprop algorithm,” inIEEE international conference on neural networks. IEEE, 1993, pp. 586–591

    1993

  20. [28]

    1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs

    F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, “1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs.” inConference of the International Speech Communication Association, vol. 2014. Singapore, 2014, pp. 1058–1062

    2014

  21. [29]

    Lion secretly solves constrained optimization: As lyapunov predicts,

    L. Chen, B. Liu, K. Liang, and Q. Liu, “Lion secretly solves constrained optimization: As lyapunov predicts,”arXiv preprint arXiv:2310.05898, 2023

    2023 arXiv

  22. [30]

    Dissecting Adam: The sign, magnitude and variance of stochastic gradients,

    L. Balles and P. Hennig, “Dissecting Adam: The sign, magnitude and variance of stochastic gradients,” inInternational Conference on Machine Learning, 2018, pp. 404–413

    2018

  23. [31]

    Heavy- tailed class imbalance and why Adam outperforms gradient descent on language models,

    F. Kunstner, R. Yadav, A. Milligan, M. Schmidt, and A. Bietti, “Heavy- tailed class imbalance and why Adam outperforms gradient descent on language models,”arXiv preprint arXiv:2402.19449, 2024

    2024 arXiv

  24. [32]

    A method of solving a convex programming problem with convergence rate o\bigl(kˆ2\bigr),

    Y . Nesterov, “A method of solving a convex programming problem with convergence rate o\bigl(kˆ2\bigr),” inDoklady Akademii Nauk. Russian Academy of Sciences, 1983

    1983

  25. [33]

    Springer Science & Business Media, 2013, vol

    ——,Introductory lectures on convex optimization: A basic course. Springer Science & Business Media, 2013, vol. 87

    2013

  26. [34]

    Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models,

    X. Xie, P. Zhou, H. Li, Z. Lin, and S. Yan, “Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models,”IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2024

    2024

  27. [35]

    Win: Weight-decay-integrated nesterov acceleration for adaptive gradient algorithms,

    P. Zhou, X. Xie, and Y . Shuicheng, “Win: Weight-decay-integrated nesterov acceleration for adaptive gradient algorithms,” inInternational Conference on Learning Representations, 2023

    2023

  28. [36]

    Glm-130b: An open bilingual pre-trained model,

    A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y . Xu, W. Zheng, X. Xiaet al., “Glm-130b: An open bilingual pre-trained model,”arXiv preprint arXiv:2210.02414, 2022

    2022 arXiv

  29. [37]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  30. [38]

    Baichuan 2: Open large-scale language models,

    A. Yang, B. Xiao, B. Wang, B. Zhang, C. Bian, C. Yin, C. Lv, D. Pan, D. Wang, D. Yanet al., “Baichuan 2: Open large-scale language models,” arXiv preprint arXiv:2309.10305, 2023

    2023 arXiv

  31. [39]

    What language model to train if you have one million gpu hours?

    T. L. Scao, T. Wang, D. Hesslow, L. Saulnier, S. Bekman, M. S. Bari, S. Biderman, H. Elsahar, N. Muennighoff, J. Phanget al., “What language model to train if you have one million gpu hours?”arXiv preprint arXiv:2210.15424, 2022

    arXiv

  32. [40]

    A theory on adam insta- bility in large-scale machine learning,

    I. Molybog, P. Albert, M. Chen, Z. DeVito, D. Esiobu, N. Goyal, P. S. Koura, S. Narang, A. Poulton, R. Silvaet al., “A theory on adam insta- bility in large-scale machine learning,”arXiv preprint arXiv:2304.09871, 2023

    2023 arXiv

  33. [41]

    A mean field theory of batch normalization,

    G. Yang, J. Pennington, V . Rao, J. Sohl-Dickstein, and S. S. Schoen- holz, “A mean field theory of batch normalization,”arXiv preprint arXiv:1902.08129, 2019

    1902 arXiv

  34. [42]

    Understanding the difficulty of training transformers,

    L. Liu, X. Liu, J. Gao, W. Chen, and J. Han, “Understanding the difficulty of training transformers,”arXiv preprint arXiv:2004.08249, 2020

    2004 arXiv

  35. [43]

    On layer normalization in the trans- former architecture,

    R. Xiong, Y . Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y . Lan, L. Wang, and T. Liu, “On layer normalization in the trans- former architecture,” inInternational Conference on Machine Learning. PMLR, 2020, pp. 10 524–10 533

    2020

  36. [44]

    The lipschitz constant of self- attention,

    H. Kim, G. Papamakarios, and A. Mnih, “The lipschitz constant of self- attention,” inInternational Conference on Machine Learning, 2021, pp. 5562–5571

    2021

  37. [45]

    Lipsformer: In- troducing lipschitz continuity to vision transformers,

    X. Qi, J. Wang, Y . Chen, Y . Shi, and L. Zhang, “Lipsformer: In- troducing lipschitz continuity to vision transformers,”arXiv preprint arXiv:2304.09856, 2023

    2023 arXiv

  38. [46]

    Signal propagation in transformers: Theoretical perspectives and the role of rank collapse,

    L. Noci, S. Anagnostidis, L. Biggio, A. Orvieto, S. P. Singh, and A. Lucchi, “Signal propagation in transformers: Theoretical perspectives and the role of rank collapse,” inAdvances in Neural Information Processing Systems, 2022, pp. 27 198–27 211

    2022

  39. [47]

    On the variance of the adaptive learning rate and beyond,

    L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han, “On the variance of the adaptive learning rate and beyond,”arXiv preprint arXiv:1908.03265, 2019

    1908 arXiv

  40. [48]

    Catapults in sgd: spikes in the training loss and their impact on generalization through feature learning,

    L. Zhu, C. Liu, A. Radhakrishnan, and M. Belkin, “Catapults in sgd: spikes in the training loss and their impact on generalization through feature learning,”arXiv preprint arXiv:2306.04815, 2023

    2023 arXiv

  41. [49]

    Loss spike in training neural networks,

    Z. Zhang and Z.-Q. J. Xu, “Loss spike in training neural networks,” arXiv preprint arXiv:2305.12133, 2023

    2023 arXiv

  42. [50]

    Stochasticity of deterministic gradient descent: Large learning rate for multiscale objective function,

    L. Kong and M. Tao, “Stochasticity of deterministic gradient descent: Large learning rate for multiscale objective function,”Advances in Neural Information Processing Systems, vol. 33, pp. 2625–2638, 2020

    2020

  43. [51]

    On the convergence of a class of adam-type algorithms for non-convex optimization,

    X. Chen, S. Liu, R. Sun, and M. Hong, “On the convergence of a class of adam-type algorithms for non-convex optimization,”arXiv preprint arXiv:1808.02941, 2018. JOURNAL OF LATEX CLASS FILES, 2025 12

    2018 arXiv

  44. [52]

    A simple convergence proof of adam and adagrad,

    A. D ´efossez, L. Bottou, F. Bach, and N. Usunier, “A simple convergence proof of adam and adagrad,”arXiv preprint arXiv:2003.02395, 2020

    2003 arXiv

  45. [53]

    Convergence guarantees for rmsprop and adam in non-convex optimization and an empirical compar- ison to nesterov acceleration,

    S. De, A. Mukherjee, and E. Ullah, “Convergence guarantees for rmsprop and adam in non-convex optimization and an empirical compar- ison to nesterov acceleration,”arXiv preprint arXiv:1807.06766, 2018

    2018 arXiv

  46. [54]

    Adam can converge without any modification on update rules,

    Y . Zhang, C. Chen, N. Shi, R. Sun, and Z.-Q. Luo, “Adam can converge without any modification on update rules,”Advances in neural information processing systems, vol. 35, pp. 28 386–28 399, 2022

    2022

  47. [55]

    Convergence of adam under relaxed assumptions,

    H. Li, A. Rakhlin, and A. Jadbabaie, “Convergence of adam under relaxed assumptions,”Advances in Neural Information Processing Sys- tems, vol. 36, 2023

    2023

  48. [56]

    On convergence of adam for stochastic optimization under relaxed assumptions,

    Y . Hong and J. Lin, “On convergence of adam for stochastic optimization under relaxed assumptions,”arXiv preprint arXiv:2402.03982, 2024

    2024 arXiv

  49. [57]

    Lower bounds for non-convex stochastic optimization,

    Y . Arjevani, Y . Carmon, J. C. Duchi, D. J. Foster, N. Srebro, and B. Woodworth, “Lower bounds for non-convex stochastic optimization,” Mathematical Programming, vol. 199, no. 1-2, pp. 165–214, 2023

    2023

  50. [58]

    A stochastic approximation method,

    H. Robbins and S. Monro, “A stochastic approximation method,”The annals of mathematical statistics, pp. 400–407, 1951

    1951

  51. [59]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gellyet al., “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  52. [60]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, vol. 1, no. 8, p. 9, 2019. JOURNAL OF LATEX CLASS FILES, 2025 13 APPENDIX A. Proof of Theorem 1 Proof.Recalling Eq. (2), we know m(j) t = 1−β...

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.