AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training

Bohan Wang; Huishuai Zhang; Luoxin Chen

REVIEW 3 major objections 7 minor 48 references

AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AdamS replaces AdamW's second-moment estimate with momentum itself, halving optimizer memory while matching AdamW's loss curves on GPT-2 through Llama2-13B.

desk verdict A simple, plausibly useful memory-halving AdamW variant whose empirical parity claims are credible but whose convergence theorem does not cover the recommended hyperparameters. read the letter →

arxiv 2505.16363 v1 pith:IXSVI7SW submitted 2025-05-22 cs.LG cs.AIstat.ML

Huishuai Zhang , Bohan Wang , Luoxin Chen This is my paper

classification cs.LGcs.AIstat.ML

keywords AdamSoptimizermomentum-basednormalizationmemory-efficientLLMtraining(L0L1)-smoothnessadaptivelearningrateslanguagemodelpretrainingGRPOpost-trainingstatereduction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces AdamS, an optimizer that keeps AdamW's momentum but replaces its second-moment estimate—the exponential average of squared gradients that accounts for roughly half the optimizer state—with a denominator built from the momentum and the current gradient. The design rests on the $(L_0,L_1)$-smoothness of transformer objectives, where local curvature grows with gradient magnitude, so the learning rate should scale inversely with the gradient; the paper argues that momentum is a reliable proxy for that magnitude. AdamS claims to match or beat AdamW on GPT-2 up to Llama2-13B pretraining and on GRPO-based RL post-training, using AdamW's own hyperparameters unchanged. If the claim holds, LLM training can halve optimizer memory and communication with essentially a drop-in replacement.

What carries the argument

The central object is the AdamS denominator $\nu_t = \beta_2 m_{t-1}^{\odot 2} + (1-\beta_2) g_t^{\odot 2}$, a coordinate-wise weighted sum of squares of the momentum and the current gradient. It replaces AdamW's recursive second-moment estimate $\nu_t = \beta_2 \nu_{t-1} + (1-\beta_2) g_t^{\odot 2}$, removing the need to maintain a second-moment state. The argument that this works rests on the $(L_0,L_1)$-smoothness condition: for transformer objectives the local smoothness constant is bounded by $L_0 + L_1 \|\nabla f(w)\|$, so the per-coordinate learning rate should scale inversely with gradient magnitude; momentum, as an exponential average of past gradients, is claimed to track that magnitude reliably with far less state. This denominator is what halves memory and compute footprint while preserving AdamW's per-coordinate adaptivity.

What would settle it

Run a head-to-head pretraining comparison at 100B+ tokens with multiple seeds and compare final validation loss; if AdamS is consistently worse than AdamW by more than the run-to-run noise, the parity claim collapses. A more direct check: on a real training run, compute the per-coordinate ratio $\sqrt{\beta_2 m_{t-1}^{\odot 2}+(1-\beta_2)g_t^{\odot 2}}$ over AdamW's second-moment square root, and show it deviates far from 1 as gradient noise grows (e.g., small batch sizes).

Watch

Extended reading notes

Core claim

AdamS eliminates the need for the second-moment estimate in Adam-family optimizers by defining the per-coordinate normalizer $\nu_t = \beta_2 m_{t-1}^{\odot 2} + (1-\beta_2) g_t^{\odot 2}$, a weighted mix of the previous momentum squared and the current gradient squared, so the update becomes $w_t = (1 - \eta_t\lambda)w_{t-1} - \eta_t m_t/(\sqrt{\nu_t}+\epsilon)$. Because $\nu_t$ uses only the momentum and the current gradient, no second-moment buffer is stored, and optimizer memory matches SGD with momentum. The paper claims that on transformer pretraining—where $(L_0,L_1)$-smoothness makes local curvature depend on gradient magnitude—momentum is a sufficient surrogate for gradient magnitude, and that with the same hyperparameters as AdamW, AdamS matches or exceeds AdamW's loss trajectories on GPT-2 (125M to 770M parameters), Llama2-7B, and Llama2-13B, and matches AdamW on DeepSeek-R1-Zero-style GRPO post-training. It also proves a convergence rate $\widetilde{O}(T^{-1/4})$ for nonconvex objectives under $(L_0,L_1)$-smoothness and sub-gaussian noise, matching the known lower bound for gradient-based optimizers.

Load-bearing premise

Transformer loss surfaces obey the $(L_0,L_1)$-smoothness condition with reasonable constants, and the fixed $\beta_2=0.95$ momentum-weighted denominator stays a reliable per-coordinate normalizer throughout training.

Editorial extensions

If this is right

Pretraining and post-training runs can halve optimizer-state memory without changing model code or hyperparameters: AdamS uses AdamW's $(\beta_1,\beta_2)=(0.9,0.95)$, the same weight decay, and the same learning-rate schedule.
Communication volume in sharded training drops because second-moment buffers no longer need to be exchanged across GPUs; the paper reports roughly a 36% throughput improvement on GPT2-XL when memory is the bottleneck.
The proven nonconvex convergence rate $\widetilde{O}(T^{-1/4})$ matches the known lower bound for gradient-based optimizers, so AdamS is not slower than other adaptive methods in the worst case.
In the low-gradient-noise regime (large batch sizes), the denominator's distribution converges toward AdamW's, which is the regime of practical LLM pretraining.
The optimizer is model-agnostic: no Hessian-based parameter grouping or architecture-specific partitioning is required, unlike Adam-mini.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The fixed $\beta_2=0.95$ used in all experiments is not covered by the proof's schedule ($\beta_2=1-\Theta(1/T)$); the claimed parity would be tested more sharply with multiple seeds and runs beyond 32B tokens for Llama2-7B.
Because the denominator needs only momentum and the current gradient, AdamS could combine naturally with communication-compression schemes that exchange momentum but skip second-moment aggregates—an extension the paper does not explore.
If momentum is a dependable gradient-magnitude proxy, the same substitution could apply to factored adaptive methods (e.g., Adafactor-style denominators), potentially cutting memory further without per-coordinate second moments.
The $(L_0,L_1)$ motivation predicts the margin over AdamW may shrink or reverse in high-noise, small-batch regimes; the RL experiments' switch to $\beta_2=0.95$ hints at where that boundary lies.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The genuinely new thing here is the denominator: nu_t = beta2 * m_{t-1}^2 + (1 - beta2) * g_t^2. That is not in the cited optimizer literature, and it is a clean way to drop the second-moment buffer entirely. The paper does well on several fronts: the motivation from (L0, L1) smoothness is coherent, the design is genuinely model-agnostic and drop-in, the GPT-2 and Llama2-7B curves show AdamS tracking AdamW closely, and the code is released. If the empirical parity holds up, this is a practically useful optimizer for pretraining and post-training. I would use it as a baseline in my own work. The soft spots are real but addressable. The largest is the theory-practice gap. Theorem 3.2 assumes beta2 = 1 - Theta(1/T), which for T about 10^5 means beta2 around 0.99999, while Section 2.3 explicitly warns that beta2 cannot be too close to 1 and recommends 0.95. So the convergence guarantee applies to a regime the paper itself calls unstable, and the 0.95 used in every experiment is not covered. That is an internal inconsistency, not a deep flaw in the optimizer idea, but the authors should either extend the theory to fixed beta2 or clearly label Theorem 3.2 as analyzing a different, asymptotic variant. Also, all empirical comparisons are single runs with no error bars; the Llama2-13B run uses only 0.96B tokens, which is a lot less than a serious scale test; and the beta2=0.95 choice is tuned to minimize the denominator gap with AdamW under an iid Gaussian model, so some of the parity is built into the design. A direct comparison against Adafactor or SM3 would have strengthened the memory-efficiency claim; the authors skip it by citing Adam-mini's Figure 8, which is a bit of a dodge. None of these are load-bearing flaws. The core claim, that a momentum-gradient denominator can replace the second moment in LLM training, is plausible, well-motivated, and backed by several loss curves. This paper deserves a serious referee. I would send it out, asking for multi-seed runs, longer 13B training, and a reconciliation of the theory's hyperparameter regime with the recommended beta2=0.95.

Referee Report

3 major / 7 minor

Summary. Sections 1–5 of arXiv:2505.16363 propose AdamS, a drop-in replacement for AdamW whose adaptive denominator ν_t = β2 m_{t−1}^2 + (1−β2) g_t^2 uses only the momentum and the current gradient, thereby halving optimizer-state memory while keeping AdamW's hyperparameter interface. The motivation is that transformer objectives obey an (L0, L1)-smoothness condition (Assumption 2.1) under which the local smoothness is governed by gradient magnitudes, so a momentum-based normalizer should suffice in place of the second moment. Section 2.3 supports the design with a stationary-distribution comparison under an iid Gaussian model; Section 3 states Theorem 3.2, a non-convex convergence guarantee at rate O~(T^{−1/4}) under sub-Gaussian noise; Section 4 reports GPT-2 (125M–770M), Llama2-7B (32B tokens), Llama2-13B (0.96B tokens, Appendix B.1), and GRPO post-training experiments in which AdamS's loss curves closely track AdamW's; Table 3 reports final perplexities within roughly 0.005 of AdamW at 100K and 300K steps. The paper concludes that AdamS matches or exceeds AdamW at parameter counts up to 13B while eliminating second-moment storage, and claims a theoretically optimal convergence rate.

Significance. If the central parity claim holds, AdamS is an attractive contribution to memory-efficient LLM optimization: it is conceptually simple, model-agnostic, introduces no new hyperparameters beyond AdamW's, ships with released code, and its memory savings are structural rather than heuristic. The paper also deserves credit for an explicit Limitations section and for a theory attempt (Theorem 3.2) that goes beyond the usual bounded-smoothness analysis and follows a recognized stopping-time framework. The impact is nevertheless conditional: the headline 'matches or exceeds AdamW' rests entirely on single-seed runs, the 13B data point covers only 0.96B tokens, and — most importantly — the stated convergence guarantee applies to a hyperparameter regime (β2 → 1) that the paper itself warns is unstable and that no experiment uses. Because the abstract and Section 1 advertise rigorous theoretical convergence guarantees as a core contribution, closing that gap is necessary before the paper's claims can be taken at face value.

major comments (3)

[§3 (Theorem 3.2) vs §2.3 and §4.1] Theorem 3.2's convergence guarantee is stated for a hyperparameter regime that is inconsistent with the algorithm the paper recommends and tests. The theorem requires β2 = 1 − Θ(1/T), β1 = 1 − Θ(1/√T), and η_t = O~(1/√T); for the T ≈ 10^5 steps of the §4.1 GPT-2 runs, this means β2 ≈ 0.99999 and β1 ≈ 0.997, whereas every experiment in Sections 4.1–4.3 and Appendix B.1 uses fixed β1 = 0.9, β2 = 0.95 with a cosine or linear decay schedule. Moreover, Section 2.3 and Figure 3 explicitly warn that 'overly large β2 values can destabilize updates' and that 'β2 cannot be too close to 1,' so the theorem's regime is the regime the paper itself identifies as unstable. Consequently, the abstract's and Section 1's claims of rigorous theoretical grounding do not apply to the recommended configuration, and the convergence of the algorithm actually tested is unsupported by Theorem 3.2 as stated. I note that the Appendix D proof sketch does not appear to use β2 → 1 in an essential way (Lemma D.4 bounds √ν only from above, and Lemma D.3 depends on max{β1/√β2, (1−β1)/√(1−β2)}, which is bounded for any fixed β2), so the authors should either extend the theorem to any fixed β2 ∈ (0,1) or explicitly scope the theoretical claim to the regime analyzed.
[§4.1, Table 3; Appendix B.1] The headline claim that AdamS 'matches or exceeds' AdamW (Section 1) rests entirely on single-seed runs. In Table 3, the AdamS–AdamW differences at 100K and 300K steps (2.898 vs 2.902 and 2.866 vs 2.867) are of the same order as typical seed-to-seed variation in nanoGPT-scale training, so without multiple seeds or error bars the parity claim cannot be stated at this precision. The 13B data point (Appendix B.1) covers 0.96B tokens (30K steps at batch 16 × 2048), a small fraction of a standard pretraining budget, so the 'up to 13B' framing in Section 1 and the abstract overstates what the experiment supports; the Limitations section acknowledges the budget constraint, but the main text and abstract do not qualify the claim. Reporting three to five seeds for the inexpensive GPT-2 conditions and stating the token budgets explicitly for the 7B/13B runs would make the central claim appropriately supported.
[§2.3] The analytical thought experiment concludes that 'the denominators of AdamS and AdamW are quite close when μ ≫ σ,' but the formulas displayed in the same section do not support that conclusion for the fluctuations. With β1 = 0.9 and β2 = 0.95, the paper's own expressions give Var(S∞) ≈ 0.0512σ^4 + 0.1024μ^2σ^2 for AdamW and Var(V∞) = 0.01σ^4 + 0.2μ^2σ^2 for AdamS, so in the μ ≫ σ regime the AdamS denominator has roughly twice AdamW's variance; the value β = 0.95 minimizes the gap only in a limited sense and leaves a factor-of-two discrepancy at the minimizer. Since the update divides by √ν_t, this fluctuation gap propagates directly into step-size noise, so the 'resemblance' claim holds for the means but not for the denominator fluctuations. Please either quantify this gap and temper the design rationale, or justify why the factor-of-two gap is benign at the batch sizes used in Sections 4.1–4.3.

minor comments (7)

[§2.3] The symbol β2 denotes AdamW's decay in the 'Analytical comparison' paragraph while the AdamS weighting is renamed β; please use notation consistent with Algorithm 1 throughout.
[§2.1] There is a duplicated word in 'the exponential average of of the square of historical gradients' near the end of §2.1.
[§4.3] The RL comparison changes both the denominator and β2 (AdamS uses 0.95 while the AdamW baseline uses 0.999), so the comparison does not isolate the effect of the new denominator; an AdamS run with β2 = 0.999 or an AdamW run with β2 = 0.95 would clarify.
[§4.1] The token budgets implied by 100K and 300K iterations at batch 480 × context 1024 (≈49B and ≈147B tokens) are several times the size of OpenWebText; please state the number of epochs and note the multi-epoch evaluation regime for the validation numbers in Table 3.
[Figure 2] The right panel's legend does not indicate that it compares Adam-mini with AdamW; please add this to the caption or the panel labels.
[Appendix D] Several displays in Appendix D are garbled (for example, 'β2 1∥∆t∥2' should contain β1^2, and 'Õ(1/⁴√T)' should be typeset as O~(T^{−1/4})); the step from Eq. (8) to the telescoping sum also deserves a few lines of explanation with explicit constants.
[Section 5] The closing sentence that AdamS 'can serve as the newly default optimizer' for large-scale LLM training overstates the evidence presented; a more measured claim would sit better with the paper's own Limitations discussion.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: AdamS is independently defined and its convergence proof and empirical comparisons do not reduce to the design equations; the only by-construction element is the explicit calibration of β2=0.95 to AdamW's denominator variance, which is a design choice rather than a fitted prediction.

full rationale

The claimed derivation chain is not circular. AdamS's denominator ν_t = β2 m_{t-1}^2 + (1−β2) g_t^2 is a new update rule defined independently of the target result, and the convergence guarantee in Theorem 3.2 is proved from Assumptions 2.1 and 3.1 using standard stochastic-optimization arguments adapted from Li et al. (2023), not from the empirical parity claim. The empirical evaluation compares AdamS with AdamW under matched hyperparameters and reports raw training/validation curves, so the 'matches AdamW' claim is tested independently of the design equations. The only by-construction element is Section 2.3's choice of β2=0.95 to minimize the variance gap between AdamS's and AdamW's denominators under an iid Gaussian model ('The best β = 0.95 to minimize the difference between the variance of St and Vt'); this is an explicit design calibration rather than a fitted parameter, and the later experiments and theory are not logically derived from that variance-matching step. A separate validity concern, noted in the reader take and not a circularity issue, is that Theorem 3.2 assumes β2 = 1 − Θ(1/T) while Section 2.3 warns that 'β2 cannot be too close to 1' and the experiments use fixed β2=0.95; this is an internal theory-practice mismatch, not a reduction of the conclusion to the inputs. No load-bearing self-citation was found: the cited works on (L0,L1)-smoothness and momentum gradient-magnitude proxies are external prior results, and the proof itself relies on an external convergence framework. The Limitations section acknowledges resource constraints and benchmarking confounders, which bear on evidence strength but not on circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

AdamS introduces no new physical or model entities. Its assumptions are standard optimizer-theory assumptions plus a hand-chosen beta2 that is tuned to mirror AdamW's denominator. The central claim depends primarily on Assumptions 2.1 and 3.1 and on the transfer of AdamW's hyperparameters, which is an empirical conjecture, not a proven fact.

free parameters (3)

beta2 denominator weight = 0.95
Chosen in Section 2.3 to minimize the variance gap between AdamS and AdamW denominators in an iid Gaussian thought experiment; used in all experiments, while Theorem 3.2 requires beta2 = 1 - Theta(1/T).
beta1 momentum weight = 0.9
Inherited from AdamW hyperparameters; central to the update rule and to the momentum-as-proxy argument in Section 2.1.
epsilon regularizer = Not specified in the paper
Regularizer in the denominator sqrt(nu_t)+epsilon; needed for numerical stability, but its exact value is not reported.

assumptions (4)

domain assumption Assumption 2.1: f satisfies the (L0,L1)-smoothness condition
Used throughout the convergence proof in Appendix D; the paper does not verify the assumption empirically for the actual transformer objectives.
domain assumption Assumption 3.1: stochastic gradient noise is sub-Gaussian
Used to bound noise terms and stopping-time probabilities in Lemma D.6 and Theorem 3.2.
domain assumption Independence of X_t and M_{t-1} in the Section 2.3 thought experiment
Used to compute E[V_infinity] and Var[V_infinity]; gradients in actual training are correlated over time, so the variance-matching conclusion is approximate.
domain assumption Large-batch regime with mu >> sigma
Section 2.3 concludes AdamS resembles AdamW only when gradient noise is negligible; stated to hold in LLM pretraining but not in all post-training settings.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training." pith.science (2026). https://pith.science/paper/IXSVI7SW

@misc{pith2026250516363,
  author       = {Pith},
  title        = {Pith review of: AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IXSVI7SW}},
  note         = {Machine review of arXiv:2505.16363}
}

abstract

We introduce AdamS, a simple yet effective alternative to Adam for large language model (LLM) pretraining and post-training. By leveraging a novel denominator, i.e., the root of weighted sum of squares of the momentum and the current gradient, AdamS eliminates the need for second-moment estimates. Hence, AdamS is efficient, matching the memory and compute footprint of SGD with momentum while delivering superior optimization performance. Moreover, AdamS is easy to adopt: it can directly inherit hyperparameters of AdamW, and is entirely model-agnostic, integrating seamlessly into existing pipelines without modifications to optimizer APIs or architectures. The motivation behind AdamS stems from the observed $(L_0, L_1)$ smoothness properties in transformer objectives, where local smoothness is governed by gradient magnitudes that can be further approximated by momentum magnitudes. We establish rigorous theoretical convergence guarantees and provide practical guidelines for hyperparameter selection. Empirically, AdamS demonstrates strong performance in various tasks, including pre-training runs on GPT-2 and Llama2 (up to 13B parameters) and reinforcement learning in post-training regimes. With its efficiency, simplicity, and theoretical grounding, AdamS stands as a compelling alternative to existing optimizers.

Figures

Figures reproduced from arXiv: 2505.16363 by the authors.

**Figure 1.** Training and validation loss curves for pretraining LLaMA 2–7B models. The proposed AdamS achieves convergence comparable to or better than baseline methods under the same hyperparameter settings as LLaMA 2 (Touvron et al., 2023b), while eliminating the need to store AdamW’s second-moment estimates. Rae et al., 2021; Chowdhery et al., 2022; Du et al., 2021; Liu et al., 2024; Dubey et al., 2024; Yang et al., 2024). … view at source ↗

**Figure 2.** The cosine similarities between the update matrices of AdamS and AdamW (left), Adam-mini and AdamW (right) for all layers of GPT2-Small model. Across the training trajectory, the update direction of AdamS closely aligns with that of AdamW. 10 5 0 5 10 k=grad/momentum 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 AdamS update magnitude 2=0.9 2=0.95 2=0.99 2=0.999 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 4.** Validation loss curves for pretraining GPT-2 models. Across three different model sizes and with the same hyperparameters as AdamW, the proposed AdamS achieves convergence comparable to baseline methods—without the need to store AdamW’s second-moment estimates. to be 1024 tokens. We use a batch size of 480 and employ a cosine learning rate schedule, setting the final learning rate to 0.1 × lr as suggested by Rae et … view at source ↗

Figures from the paper (2 more)

**Figure 5.** Figure 5: Mean critic scores for reinforcement learning (RL) post-training using the GRPO algorithm on the CountDown task are presented for the Qwen2.5-3B and DeepSeek-R1- Distill-Llama-8B models. The proposed AdamS closely resembles AdamW’s performance trajectory, achieving si…

**Figure 6.** Figure 6: Training and validation loss curves for pretraining LLaMA 2–13B models. The proposed AdamS achieves convergence comparable to or better than baseline methods under the same hyperparameter settings as AdamW, while eliminating the need to store AdamW’s second-moment esti…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 13 canonical work pages

[1]

Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. 2019. Memory efficient adaptive optimization. Advances in Neural Information Processing Systems, 32

work page 2019
[2]

Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. 2022. Lower bounds for non-convex stochastic optimization. Mathematical Programming, pages 1--50

work page 2022
[3]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

arXiv doi:10.48550/arxiv.2005.14165 2020
[4]

Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al. 2023. Symbolic discovery of optimization algorithms. arXiv preprint arXiv:2302.06675

arXiv 2023
[5]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

arXiv doi:10.48550/arxiv.2204.02311 2022
[6]

Ashok Cutkosky and Harsh Mehta. 2020. Momentum improves normalized SGD . In International conference on machine learning, pages 2260--2268. PMLR

work page 2020
[7]

Alexandre D \'e fossez, Leon Bottou, Francis Bach, and Nicolas Usunier. 2022. A simple convergence proof of Adam and Adagrad . Transactions on Machine Learning Research

work page 2022
[8]

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2021. https://arxiv.org/abs/2103.10360 All NLP tasks are generation tasks: A general pretraining framework . CoRR, abs/2103.10360

arXiv 2021

Show all 48 references

[9]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

2024 arXiv
[10]

Matthew Faw, Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. 2023. Beyond uniform smoothness: A stopped analysis of adaptive sgd. arXiv preprint arXiv:2302.06570

2023 arXiv
[11]

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The P ile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027

2020 arXiv
[12]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

2025 arXiv
[13]

Meixuan He, Yuqing Liang, Jinlan Liu, and Dongpo Xu. 2023. Convergence of adam for non-convex objectives: Relaxed hyperparameters and non-ergodic case. arXiv preprint arXiv:2307.11782

2023 arXiv
[14]

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556

2022 arXiv
[15]

Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt Kusner. 2023. No train no gain: Revisiting efficient training algorithms for transformer-based language models. In Thirty-seventh Conference on Neural Information Processing Systems

2023
[16]

Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

2020 arXiv
[17]

Kingma and Jimmy Ba

Diederik P. Kingma and Jimmy Ba. 2014. https://doi.org/10.48550/ARXIV.1412.6980 Adam: A method for stochastic optimization . arXiv preprint

2014 doi arXiv
[18]

Haochuan Li, Ali Jadbabaie, and Alexander Rakhlin. 2023. Convergence of Adam under relaxed assumptions. arXiv preprint arXiv:2304.13972

2023 arXiv
[19]

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

2024 arXiv
[20]

Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. 2023. Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv preprint arXiv:2305.14342

2023 arXiv
[21]

Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations

2019
[22]

Yang Luo, Xiaozhe Ren, Zangwei Zheng, Zhuo Jiang, Xin Jiang, and Yang You. 2023. Came: Confidence-guided adaptive memory efficient optimization. arXiv preprint arXiv:2307.02047

2023 arXiv
[23]

Kai Lv, Hang Yan, Qipeng Guo, Haijun Lv, and Xipeng Qiu. 2023. Adalomo: Low-memory optimization with adaptive learning rate. arXiv preprint arXiv:2310.10195

2023 arXiv
[24]

Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, Binh Tang, Diana Liskovich, Puxin Xu, Yuchen Zhang, Melanie Kambadur, Stephen Roller, and Susan Zhang. 2023. https://arxiv.org/abs/23...

2023 arXiv
[25]

Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. 2025. Tinyzero. https://github.com/Jiayi-Pan/TinyZero. Accessed: 2025-01-24

2025
[26]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. P ytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Sy...

2019
[27]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

2019
[28]

Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, F

Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, F. Song, John Aslanides, Sarah Henderson, R. Ring, S. Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446

2021 arXiv
[29]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

2020
[30]

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2019. https://arxiv.org/abs/1910.02054 Zero: Memory optimization towards training A trillion parameter models . CoRR, abs/1910.02054

2019 arXiv
[31]

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

2024 arXiv
[32]

Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596--4604. PMLR

2018
[33]

Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

2024
[34]

Hugo Touvron, Thibault Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

2023 arXiv
[35]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

2023 arXiv
[36]

Bohan Wang, Jingwen Fu, Huishuai Zhang, Nanning Zheng, and Wei Chen. 2023 a . Closing the gap between the upper bound and lower bound of adam's iteration complexity. In Thirty-seventh Conference on Neural Information Processing Systems

2023
[37]

Bohan Wang, Huishuai Zhang, Zhiming Ma, and Wei Chen. 2023 b . Convergence of adagrad for non-convex objectives: Simple proofs and relaxed assumptions. In The Thirty Sixth Annual Conference on Learning Theory, pages 161--190. PMLR

2023
[38]

Bohan Wang, Yushun Zhang, Huishuai Zhang, Qi Meng, Zhi-Ming Ma, Tie-Yan Liu, and Wei Chen. 2022. Provable adaptivity in Adam . arXiv preprint arXiv:2208.09900

2022 arXiv
[39]

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

2024 arXiv
[40]

Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. 2021. Large scale private learning via low-rank reparametrization. In International Conference on Machine Learning, pages 12208--12218. PMLR

2021
[41]

Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. 2020. Improved analysis of clipping algorithms for non-convex optimization. Advances in Neural Information Processing Systems, 33:15511--15521

2020
[42]

Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. 2019. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Representations

2019
[43]

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Ming-Wei Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xiaodong Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

2022 arXiv
[44]

Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. 2024. Adam-mini: Use fewer learning rates to gain more. arXiv preprint arXiv:2406.16793

2024 arXiv
[45]

Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507

2024 arXiv
[46]

Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. 2023. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277

2023 arXiv
[47]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[48]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. 2019. Memory efficient adaptive optimization. Advances in Neural Information Processing Systems, 32

work page 2019

[2] [2]

Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. 2022. Lower bounds for non-convex stochastic optimization. Mathematical Programming, pages 1--50

work page 2022

[3] [3]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

arXiv doi:10.48550/arxiv.2005.14165 2020

[4] [4]

Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al. 2023. Symbolic discovery of optimization algorithms. arXiv preprint arXiv:2302.06675

arXiv 2023

[5] [5]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

arXiv doi:10.48550/arxiv.2204.02311 2022

[6] [6]

Ashok Cutkosky and Harsh Mehta. 2020. Momentum improves normalized SGD . In International conference on machine learning, pages 2260--2268. PMLR

work page 2020

[7] [7]

Alexandre D \'e fossez, Leon Bottou, Francis Bach, and Nicolas Usunier. 2022. A simple convergence proof of Adam and Adagrad . Transactions on Machine Learning Research

work page 2022

[8] [8]

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2021. https://arxiv.org/abs/2103.10360 All NLP tasks are generation tasks: A general pretraining framework . CoRR, abs/2103.10360

arXiv 2021