Pith. sign in

REVIEW 3 major objections 3 minor 7 cited by

Benchmarking Optimizers for Large Language Model Pretraining

T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims AdEMAMix beats AdamW in most LLM pretraining settings, with MARS close behind at larger scale.

desk verdict A careful, broad optimizer benchmark whose headline ranking is plausible but not robust to its own re-tuning evidence; still worth a serious referee. read the letter →

arxiv 2509.01440 v1 pith:ONWCAFRN submitted 2025-09-01 cs.LG

Andrei Semenov , Matteo Pagliardini , Martin Jaggi This is my paper · ORCID
classification cs.LG MSC 68T0790C26
keywords optimizerbenchmarkingLLMpretrainingAdEMAMixAdamWMARShyperparametertuninglearningrateschedulesmixture-of-experts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks a practical question: now that many optimizers claim to beat AdamW for large language models, which one should a pretraining run actually use? To answer it, the authors run a controlled benchmark of 11 optimizers on decoder-only transformers from 124M to 720M parameters, varying batch size and training duration up to 48B tokens, with each method tuned individually. Their central claim is that AdEMAMix, an AdamW variant with an extra slow momentum buffer, consistently reaches the lowest validation loss across settings, and that MARS is the strongest challenger at the largest scale tested. The paper also documents that rankings depend on training horizon and batch size, and that routine choices—weight decay, warmup, learning-rate decay, scheduler, and momentum-memory settings—can change which optimizer wins. If these results hold at larger scale, the practical upshot is that AdamW is no longer the automatic default, and the paper provides the tuning recipes to make the switch.

What carries the argument

The central object is the dual-momentum update of AdEMAMix: alongside AdamW's fast momentum buffer, it maintains a slower exponential moving average of gradients, with a blending coefficient and a memory parameter controlling the slow buffer; this slow EMA is what lets the optimizer exploit older gradient information and drives its lead. The other load-bearing instrument is the benchmark protocol itself: a standardized grid over model size (124M–720M plus a 520M mixture-of-experts model), batch size, and token budget, with hyperparameters tuned at a near-compute-optimal duration and then held fixed when extending training. That protocol is what makes the rankings comparable, and its re-tunin

What would settle it

Train AdEMAMix, SOAP, MARS, and AdamW on a 1B–3B parameter model for at least 100B tokens, re-tuning each optimizer's betas and learning rate at the full target duration rather than at a shorter proxy; if AdamW or SOAP matches or beats AdEMAMix at that horizon, the paper's headline ranking fails. The paper's own Figure 21 already shows SOAP overtaking AdEMAMix when the slow-momentum setting is not re-tuned, so the decisive test is with re-tuning at the target horizon.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a ranking, not a new algorithm: after extensive per-method tuning, AdEMAMix is the best optimizer for LLM pretraining across nearly all model sizes, batch sizes, and training horizons considered, and MARS is the best at 720M scale alongside it, with a noticeable gap over other methods. AdamW remains competitive but is surpassed; sign-based optimizers (Lion, Signum) and MARS become competitive only with large batches; Sophia converges for short runs but diverges when training extends past a certain horizon. The paper further shows that the benchmark setup itself matters: decaying the learning rate to 1% of its peak instead of 10%, using decoupled wei

Load-bearing premise

The ranking assumes hyperparameters tuned at a shorter, near-compute-optimal training length remain valid when training is extended; the paper itself shows that re-tuning the momentum-memory settings at 33.6B tokens reverses the SOAP/AdEMAMix ordering.

Editorial extensions

If this is right

  • A practitioner pretraining a model in the tested regime (up to 720M parameters, up to 48B tokens) can expect lower final validation loss with AdEMAMix than with AdamW, provided its slow-momentum and learning-rate settings are tuned for the intended horizon.
  • MARS is the main alternative at larger scale and large batch size; it improves with batch size and model size, making it the strongest candidate when AdEMAMix is unavailable or too costly.
  • Signum, Lion, and MARS become competitive with AdamW only at sufficiently large batch sizes; at small batches they fall behind, so batch size should be part of any optimizer comparison.
  • Sophia's early gains are not durable: it diverges in small-batch runs and again when training is extended well beyond the compute-optimal horizon in the large-batch setup.
  • Optimizer rankings found on dense models transfer to a 520M mixture-of-experts model when the same hyperparameters are reused, so MoE pretraining can adopt the same optimizer choice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the momentum-memory sensitivity the paper documents generalizes, then any fixed-horizon optimizer ranking is provisional: rankings should be reported with the tuning horizon matched to the deployment horizon, or with re-tuning at every target duration.
  • The AdEMAMix lead at 720M over 48B tokens is consistent with—but does not prove—a growing advantage at multi-billion-parameter scale; testing it at 1B+ parameters and 100B+ tokens would be a direct stress test.
  • Because the study tracks validation loss rather than downstream task performance, a complementary experiment would check whether the loss ranking translates to question-answering or reasoning benchmarks; the paper itself notes that loss and downstream gains can diverge.
  • The short-run benefit of large weight decay suggests a cheap heuristic for compute-limited training: raise weight decay toward 0.5 when the token budget is below the compute-optimal duration, then drop it to 0.1 for long runs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper benchmarks 11 optimizers (AdamW, ADOPT, AdEMAMix, Lion, Signum, Muon/D-Muon, SOAP, Sophia, SF-AdamW, Prodigy, MARS) for LLM pretraining under standardized settings: Llama-like models of 124M, 210M, 583M, 720M, and a 520M MoE; batch sizes from 16k to 2M tokens; and training horizons up to 48B tokens on FineWeb. Its central claims are that AdEMAMix consistently achieves state-of-the-art performance and scales robustly with training duration, that MARS is the second-best at 720M/48B, and that several widely used or recently proposed methods are competitive only in certain regimes. The paper also contributes extensive ablations on weight decay, warmup, learning-rate schedulers, beta parameters, initialization, and wall-clock overhead, and it releases the full toolkit and all run configurations.

Significance. If the headline ranking holds, this is a valuable reference for both practitioners and optimizer researchers: the study is large in scope (2900 runs, ~30k GPU hours), the code is released, and many ablations are useful independently of the ranking. The paper also documents several concrete implementation pitfalls, such as the Signum weight-decay bug and the discrepancy between the published Sophia update and its code. However, the central claim rests on an extrapolation protocol that the paper's own re-tuning evidence shows to be fragile, and the main ranking figures report no seed-level variance. These issues make the ranking conditional rather than established.

major comments (3)
  1. [§3, §4.3, Fig. 21, Takeaway 14] The central ranking is protocol-dependent in a way the paper itself documents. §3 states that hyperparameters are tuned near-Chinchilla (2.1B tokens for 124M, 16B for 720M) and then kept for longer runs, with only occasional re-tuning. Fig. 21 and Takeaway 14 show that at 124M/33.6B, SOAP with β2=0.999—the value selected at 16.8B—overtakes AdEMAMix with β3=0.999, and AdEMAMix only regains the lead after β3 is re-tuned to 0.9999. The 720M/48B runs are 3× beyond the 16B tuning point and use the same 'keep hyperparameters' protocol, yet MARS and SOAP/AdamW/D-Muon are not re-tuned at 48B. Without a 48B re-tuning sweep for all top contenders, or a sensitivity analysis showing the ranking is stable, the headline 'noticeable gap' in Fig. 1 is not established. This is load-bearing because the paper's main claim is that AdEMAMix and MARS reliably beat AdamW across scales and durations.
  2. [Figs. 1, 5, 13, 19] The main ranking figures plot single final validation loss values without error bars, seed counts, or variance information. Several adjacent methods differ by less than 0.02 loss at 124M, and no evidence is given that these differences exceed run-to-run noise. The wall-clock comparison (§4.3, Fig. 18) does use 5 seeds, but the ranking runs do not report any replication. For a paper whose primary output is a ranking, at minimum 3 seeds for the top-3 methods in each main setting, or an explicit statement that the reported differences are stable across seeds, is needed.
  3. [§4.1, Fig. 5(a)] Sophia is excluded from the small-batch ranking after 'we decided not to include Sophia at this stage' because it diverges. If divergence is the exclusion criterion, this should be a pre-defined rule applied uniformly and reported as such; as presented, it is a post-hoc selection that makes the comparison less transparent. The divergence is itself informative and should appear in the ranking figure (e.g., marked as diverged) rather than being omitted. This does not affect the top of the ranking, but it is a fairness issue in a benchmark paper.
minor comments (3)
  1. [Throughout] There are many typos and garbled labels, e.g., 'noticable' (Fig. 1), 'Suprisingly' (Takeaway 2), 'Cinchilla' (Appendix D.1), 'adoptation' (§A.2), and 'Ranking/uni00000003' appearing in figure labels. A careful copy-edit is needed.
  2. [§3] The text refers to 'Tables 3, 4, 5, 6, and 48'; Table 48 does not exist in the provided manuscript. The cross-reference likely points to a later appendix table and should be corrected.
  3. [Fig. 6 (right) and Fig. 21] The caption/text for Fig. 6 (right) says the SOAP/AdEMAMix gap 'narrow and, finally, increases', which is confusing given Fig. 21 shows the gap reverses for un-tuned betas. Please clarify whether the statement refers to re-tuned or un-tuned configurations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's ranking is an empirical benchmark generated by its own controlled runs, not a derivation that reduces to its inputs.

full rationale

The paper's central claims—that AdEMAMix and MARS achieve the best validation loss at 720M/48B tokens and that AdEMAMix scales robustly—are supported by the paper's own extensive hyperparameter sweeps and training runs (2900 models, ~30000 GPU hours). There is no equation in the paper that defines a predicted quantity in terms of a fitted parameter and then reports that quantity as an independent finding. The only remotely self-referential element is that AdEMAMix was introduced by one of the present authors (Pagliardini et al., [99]), and the paper follows that work's suggestion to increase beta3 for longer training. However, the paper independently verifies this through its own ablation in Figure 21 and Appendix D.1, showing that without re-tuning beta3, SOAP overtakes AdEMAMix at 33.6B tokens, and with re-tuning AdEMAMix regains the lead. The beta3 recommendation is thus an experimentally tested input, not an unexamined self-citation that forces the conclusion. The Prodigy effective-learning-rate analysis (Eq. 2) is explicitly presented as a post-hoc explanatory mechanism, not as a derivation of the benchmark ranking. The paper also discloses its limitation that hyperparameters tuned near the Chinchilla duration may not transfer to longer runs, and it reports the re-tuning sensitivity openly. No step in the claimed derivation chain is equivalent to its own inputs by construction. The benchmarking conclusions are empirical, falsifiable, and self-contained; any concerns about tuning-protocol sensitivity or single-seed variance are correctness or robustness issues, not circularity.

Assumptions & free parameters 7 free parameters · 3 assumptions · 0 invented entities

The central claim is an empirical ranking, not a mathematical derivation, so most of the ledger consists of tuned hyperparameters and domain assumptions about what makes a fair comparison. The paper does not introduce new physical or mathematical entities.

free parameters (7)
  • AdEMAMix beta3 = 0.999 for short runs; 0.9999 for 16.8B+ tokens
    Re-tuning beta3 flips the SOAP vs AdEMAMix ordering at 33.6B tokens (Figure 21), so the ranking depends on this tuned value.
  • SOAP beta2 = 0.999 at 2.1B; 0.9999 at 16.8B+
    Changing beta2 changes the gap to AdEMAMix (Figure 21); the paper argues beta2 should be re-tuned with training duration.
  • Signum / Lion peak learning rate = 1e-3 at 124M; 5e-4 at 210M and larger
    Sign-based methods diverge at larger learning rates (Takeaway 7, Figure 9), so the comparison is sensitive to this tuned value.
  • Weight decay = 0.1 for long runs; 0.5 for short runs
    Weight decay sweeps across optimizers show a regime-dependent optimum (Takeaway 3, Figure 23), affecting the ranking at different horizons.
  • Warmup duration = 2k steps for most; 8k for SF-AdamW, Signum, Sophia, Lion
    Warmup ablation (Takeaway 5) shows optimizer-dependent optima; the chosen warmup affects final loss for several methods.
  • MARS eta (variance reduction scaling) = Taken from [153] defaults and tuned in Appendix E
    The paper tunes this per method; its value affects MARS convergence and therefore the ranking.
  • Prodigy gamma = 1 (default)
    The effective learning rate of Prodigy emulates AdamW (Figure 40), and this default is what makes Prodigy competitive; the ranking is sensitive to this assumption.
assumptions (3)
  • domain assumption Validation loss on the FineWeb subset is a reliable proxy for optimizer quality at these scales.
    The entire ranking is based on final validation loss; the paper itself notes downstream performance can diverge from loss (Limitations section).
  • domain assumption Hyperparameters tuned at a near-Chinchilla duration transfer to longer training runs.
    Section 3 states this protocol, but Figure 21 shows it fails for beta parameters at 33.6B tokens, so the assumption is load-bearing and only partially valid.
  • domain assumption Chinchilla-optimal token counts define comparable training durations across model sizes.
    The paper uses Chinchilla ratios to choose training lengths and labels runs as 'beyond optimal', which sets the comparison grid for the ranking.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Optimizers for Large Language Model Pretraining." pith.science (2026). https://pith.science/paper/ONWCAFRN

@misc{pith2026250901440,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Optimizers for Large Language Model Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ONWCAFRN}},
  note         = {Machine review of arXiv:2509.01440}
}
read the original abstract

The recent development of Large Language Models (LLMs) has been accompanied by an effervescence of novel ideas and methods to better optimize the loss of deep learning models. Claims from those methods are myriad: from faster convergence to removing reliance on certain hyperparameters. However, the diverse experimental protocols used to validate these claims make direct comparisons between methods challenging. This study presents a comprehensive evaluation of recent optimization techniques across standardized LLM pretraining scenarios, systematically varying model size, batch size, and training duration. Through careful tuning of each method, we provide guidance to practitioners on which optimizer is best suited for each scenario. For researchers, our work highlights promising directions for future optimization research. Finally, by releasing our code and making all experiments fully reproducible, we hope our efforts can help the development and rigorous benchmarking of future methods.

Figures

Figures reproduced from arXiv: 2509.01440 by the authors.

Figure 1
Figure 1. Ranking of optimizers for 720M Llama-based models. We plot the final validation loss obtained by the best-tuned optimizers on the FineWeb dataset. We use a batch size of 1M tokens and train multiple methods beyond and below the Chinchilla optimal duration, which is 14.4B for model of this size. AdEMAMix and MARS are the best optimizers in this setup, with a noticable gap in performance compared to other methods. We … view at source ↗
Figure 2
Figure 2. Training dynamics of leading op￾timizers on 520M MoE model pretraining. We use a batch size of 131k tokens, and train models for both short runs, i.e., less than Chin￾chilla optimal duration, and for extended runs beyond this regime. The dashed blue lines cor￾respond to the final validation loss of AdamW baselines trained for both 42k and 336k steps. evaluation pipelines, and hyperparameter configura￾tions, to enabl… view at source ↗
Figure 3
Figure 3. Comparing optimizers for training a 124M parameter LLM. We plot the validation loss dynamics obtained by considered optimizers. In (a), we train methods with a “small” batch size of 16k tokens for {64, 128, 256, 384, 512, 1024}k iterations. In (b), we train methods with nearly 8× larger batch size of 131k tokens for {8, 16, 32, 48, 64, 128}k iterations. Thus, in both settings, we result in the same number of tokens … view at source ↗
Figures from the paper (41 more)
Figure 4
Figure 4. Figure 4: Weight decay in Muon & D-Muon. We compare two methods—basic Muon [59], and D-Muon [81] with a weight decay applied to all parameter groups. Across model sizes used in our benchmarking of dense LLMs, we observe a major improvement of D-Muon over Muon. We relate this obs…
Figure 5
Figure 5. Figure 5: Ranking of optimizers for 124M models with “small” and “large” batch sizes. In both (a) and (b), we show the final validation loss for different training durations, corresponding to different numbers of tokens. Above each token number, we write the number of training i…
Figure 6
Figure 6. Figure 6: Scaling batch size vs. scaling the number of iterations. Our results demonstrate that: (left) scaling the batch size significantly improves MARS, Signum, Lion and Prodigy making them as good as AdamW even for a long training for 16.8B tokens. Which was not the case in …
Figure 7
Figure 7. Figure 7: Larger weight decay achieves significantly better results when training on fewer tokens. In (a) we observe that runs of AdamW, Signum, and Lion with the large weight decay of 0.5 consistently outperform the baseline AdamW with weight decay of 0.1 for all training durat…
Figure 8
Figure 8. Figure 8: Importance of weight decay for Muon. We complement our weight decay ablation with comparison of two version of Muon: one that uses a weight decay for all parameters (D-Muon), and another, with weight decay being applied only to embeddings, scalar parameters, and the fi…
Figure 9
Figure 9. Figure 9: Optimal learning rate stability across optimizers. The optimal learning rate determined during tuning on 2.1B tokens remains consistent after a learning rate sweep on 16.8B tokens for most optimizers. In (a), we observe that sign-based methods and similar to them Sophi…
Figure 10
Figure 10. Figure 10: Warmup ablation. For 124M model trained on the batches of 256 × 512 tokens, we perform a sweep over the linear warmup durations of {1.56%, 6.25%, 25%} of the length of training, which corresponds to {2, 8, 32}k steps, respectively. Clearly, sign-based optimizers, Soph…
Figure 11
Figure 11. Figure 11: Comparisons between cosine, WSD, and the linear schedulers. Notably, schedulers behave differently with respect to optimizer. In (a), the Muon optimizer shows a preference for WSD across most training durations. Sophia exhibits an almost perfect match between all thre…
Figure 12
Figure 12. Figure 12: , and in [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 13
Figure 13. Figure 13: Ranking of optimizers for 210M models with the batch size of 256 × 512 tokens. Increasing a model size from 124M to 210M re￾sults in almost identical ranking of optimizers com￾pared to [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 14
Figure 14. Figure 14: Comparing optimizers for training a 210M parameter LLM. We plot the training dynamics of: (a,b) optimizers that underperform AdamW for pretraining a 210M model; (c) optimizers that outperform AdamW in this setup. A complete ranking of methods in this setting is in [P…
Figure 15
Figure 15. Figure 15: Decaying the learning rate down to 0.01×γmax and beyond, instead of only to 10%. We run a 210M Llama model, and observe a common pattern for different schedulers that decreasing the learning rate to moderate 0.01 × γmax value is a better choice than decreasing it down…
Figure 16
Figure 16. Figure 16: Ablation of z-loss regularization. Incorporating the z-loss regularizer does not improve the final loss or reduce the spikiness of the loss curves. Moreover, combining z￾loss with small weight decay and decaying γ down to 10%, further degrades overall perfor￾mance. No…
Figure 17
Figure 17. Figure 17: Comparing optimizers for training a 720M parameter LLM. We conduct runs with the batch size of 1M tokens. While previous ablations (see [PITH_FULL_IMAGE:figures/full_fig_p013_17.png]
Figure 18
Figure 18. Figure 18: Wall-clock time comparison. SOAP slows down the most as model size increases. Wall-clock time comparison. After conduct￾ing experiments for models of different sizes, we are ready to present the wall-time compar￾ison for each method. For this purposes, we use a single…
Figure 19
Figure 19. Figure 19: Ranking optimizers for 520M MoE models with 256 × 512 batch size. We report results for models trained for both 42k iterations (left), and 336k (right). MoE configuration correspond to one of the 124M dense model. Optimizer rankings closely mirror those in [PITH_FULL…
Figure 20
Figure 20. Figure 20: Comparing optimizers for training a 520M parameter MoE. Training dynamics of leading optimizers is in [PITH_FULL_IMAGE:figures/full_fig_p014_20.png]
Figure 21
Figure 21. Figure 21: Re-tuning beta parameters is significant for longer training. This experimental setup coincides with the one from [PITH_FULL_IMAGE:figures/full_fig_p039_21.png]
Figure 22
Figure 22. Figure 22: Warmup sweep for AdamW. We observe that the smaller yet reasonable warmup value is the best. However, this is not the case for other methods like Signum, Lion, Sophia, and SF-AdamW— see [PITH_FULL_IMAGE:figures/full_fig_p040_22.png]
Figure 23
Figure 23. Figure 23: Larger weight decay achieves significantly better results when training on fewer tokens. We observe that the majority of runs with the large weight decay of 0.5 consistently outperform those with weight decay of 0.1 for all training durations except for the long train…
Figure 24
Figure 24. Figure 24: Learning rate sensitivity. In the current setting, only SOAP, SF-AdamW, and D-Muon reach the better performance with the large learning rate of 0.002. Conversely, Sophia and all sign-based methods (Signum and Lion) diverge with this learning rate value. MARS and Prodi…
Figure 25
Figure 25. Figure 25: WSD scheduler underperforms both AdamW with cosine scheduler and SF-AdamW. This is the only experiment we conduct on the OpenWebText2 (OWT2) dataset. We follow the small-batch setup, and replicate the best hyperparameters of each optimizer found through our tuning pro…
Figure 26
Figure 26. Figure 26: Comparisons between cosine, WSD, and the linear schedulers. We complement results in [PITH_FULL_IMAGE:figures/full_fig_p043_26.png]
Figure 27
Figure 27. Figure 27: Gradient norm patterns for cosine, linear, and WSD γ-schedulers. We run all optimizers on 124M models and track the gradient norms (before clipping) for runs using different γ-schedulers. For most optimizers, we see that gradient norms tend to increase over the course…
Figure 28
Figure 28. Figure 28: Gradient norm patterns for weight decay sweep. We complement our weight decay ablation (Figures 7 and 23) by tracking the gradient norms for all the optimizers studied in our benchmark. To highlight the effect of changing the weight decay, we use the same cosine γ-sch…
Figure 29
Figure 29. Figure 29: Gradient norm patterns for learning rate sweep. In this experiment, we complement the result on the learning rate sweep for optimizers (Figures 9 and 24) by tracking the gradient norms. We follow the same setup as for the γ-sensitivity ablation, varying the learning r…
Figure 30
Figure 30. Figure 30: Do not decay the learning rate down to 10%: ablation on 124M models. We extend our ablation on learning rate decay from [PITH_FULL_IMAGE:figures/full_fig_p047_30.png]
Figure 31
Figure 31. Figure 31: Sophia diverges in the large-batch setup, when training for many iterations. In the small-batch setup, we observed that Sophia exhibited convergence issues. With batch size 256 × 512, Sophia initially converges reliably across all training durations for 124M models us…
Figure 32
Figure 32. Figure 32: Sophia diverges in the small-batch setting even with sufficiently small learn￾ing rate. We train 124M Llama models with batch size 32 × 512 tokens for T ∈ {64, 128, 256, 384, 512, 1024}k iterations. Sophia diverges with the typical learning rate γmax = 0.001, and even…
Figure 33
Figure 33. Figure 33: Clipping is significant for Schedule-Free. Contrary to the claims of Defazio et al. [27], we find that gradient clipping remains a critical hyperparameter for SF-AdamW. As shown in (a), disabling clipping causes severe training instabilities. To mitigate these undesir…
Figure 34
Figure 34. Figure 34: ADOPT still needs β2. One of the main theoretical claims of Taniguchi et al. [134]—that ADOPT converges with any β2. The authors ver￾ify those on a toy problem motivated by Reddi et al. [115]. However, in LLM training, the choice of β2 still matters significantly. Our…
Figure 35
Figure 35. Figure 35: Prodigy is sensitive to beta parameters in the small-batch setting. In this experiment, we follow our setup (§ 3) with a small batch size of 32 × 512 tokens, training 124M models with the best hyperparameters while sweeping β2. Although β2 = 0.999 yields the best resu…
Figure 36
Figure 36. Figure 36: Impact of beta parameters on Schedule-Free. We elaborate further on the question of the sensitivity of SF-AdamW to β2. For language modeling, Defazio et al. [27] initially suggested using default (β1 = 0.9, β2 = 0.95). Then, Hägele et al. [54] revisited hyperparameter…
Figure 37
Figure 37. Figure 37: Muon’s dependence on the number of Newton-Schulz iterations. We perform a short ablation targeting the final loss of Muon (Algorithm 8) by varying the number of Newton￾Schulz iterations. Training is done for 16k steps with a batch size of 256 × 512 tokens, sweeping TN…
Figure 38
Figure 38. Figure 38: Comparison of different update rules for Signum. We evaluate three variants of the Signum update: Nesterov (our default), dampening—which resembles an EMA of mt when the dampening parameter τ equals the mo￾mentum β—and the “plain” Signum without Nesterov momentum or d…
Figure 39
Figure 39. Figure 39: MARS family of optimizers. We study three MARS-based algorithms: MARS-AdamW (just MARS in our work), MARS-Lion, and MARS-Shampoo. In this ablation, our goal is to complement our MARS runs with experiments for other similar methods, and support findings for these optim…
Figure 40
Figure 40. Figure 40: (a) shows the dynamics of the effective learning rate γ eff t , while (b) illustrates the effect of applying scheduling to γ = 1. The starting points of the curves differ slightly due to variations in the final learning rate—cosine decays γt down to 0.01, whereas WSD …
Figure 41
Figure 41. Figure 41: ADOPT’s sensitivity to ε. Interest￾ingly, the suggested by the authors ε = 10−6 is the best hyperparameter for this method. There is not a noticeable difference in convergence for ε = {10−6 , 10−7 , 10−8 , 10−9 , 10−10}, but the values of 10−5 and above give a much mo…
Figure 42
Figure 42. Figure 42: Weight initialization with smaller std prefers longer warmup. We compare final loss of models trained with AdamW using two weight initializations: the conventional std = 0.02 and a smaller std = 0.006 as in DeepSeek. We vary the training horizon, warmup duration, and …
Figure 43
Figure 43. Figure 43: Wall-clock time performance: gathered. We report the wall-clock time (in seconds) for training each model for 100 iterations using a small batch size of 16 × 512 tokens on a single GPU, without gradient accumulation or torch.compile. Bars show the ranking of optimizer…
Figure 44
Figure 44. Figure 44: Wall-clock time performance: individual. Complementing Figures 18 and 43, this figure shows the evolution of wall-clock time per 100 iterations for each optimizer as model size increases. Optimizers already shown in the main part are omitted. To improve visualization,…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A preconditioned, product-aware spectral optimizer for LoRA reaches Adam's final loss in 1.2–1.7x fewer steps on instruction-tuning, with ~3% per-step overhead.

  2. (A)iSpy: Parasitic Trojans for Machine Learning Infrastructure

    cs.CR 2026-07 conditional novelty 6.0 of 10

    A runtime-extension Trojan turns a single poisoned sample into a 97%+ backdoor via replay/amplification and leaks training hyperparameters through watermarked weights or innocuous text codewords.

  3. OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.

  4. LoRDO: Distributed Low-Rank Optimization with Infrequent Communication

    cs.LG 2026-02 conditional novelty 6.0 of 10

    LoRDO combines global low-rank projections with full-rank quasi-hyperbolic momentum to let infrequent-synchronization distributed training match low-rank DDP at roughly 10x less communication.

  5. Fantastic Pretraining Optimizers and Where to Find Them

    cs.LG 2025-09 conditional novelty 6.0 of 10

    After careful tuning, alternative optimizers speed up LLM pretraining by only 1.1x to 1.4x over AdamW, and the advantage shrinks with model scale.

  6. CMuon: Accelerating and Stabilizing Diffusion Transformer Training via Chunked Momentum Orthogonalization

    cs.AI 2026-08 conditional novelty 5.0 of 10

    Chunked Muon, which orthogonalizes fused DiT weight matrices per functional block instead of jointly, reaches FID 1.18 on ImageNet 256 in 200 epochs, about 2× faster than AdamW.

  7. Scale Weight Decay and Train Better

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Muon with weight decay scaled by η/η_max reaches the same MoE validation loss ~30% faster than constant-decay Muon while preserving asymptotic stationarity of the unregularized objective.

Reference graph

Works this paper leans on

176 extracted references · 62 canonical work pages · cited by 7 Pith papers

  1. [1]

    Dion: A communication-efficient optimizer for large models, 2025

    Kwangjun Ahn and Byron Xu. Dion: A communication-efficient optimizer for large models, 2025

    2025

  2. [2]

    Essential AI, :, Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, Somanshu Singla, Tim Romanski, Yash Vanjani, and Ash...

    2025

  3. [3]

    Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The polar express: Optimal matrix sign methods and their application to the muon algorithm, 2025

    2025

  4. [4]

    ASGO: Adaptive structured gradient optimization, 2025

    Kang An, Yuxing Liu, Rui Pan, Yi Ren, Shiqian Ma, Donald Goldfarb, and Tong Zhang. ASGO: Adaptive structured gradient optimization, 2025

    2025

  5. [5]

    Dissecting adam: The sign, magnitude and variance of stochastic gradients, 2020

    Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients, 2020

    2020

  6. [6]

    Bekas, E

    C. Bekas, E. Kokiopoulou, and Y . Saad. An estimator for the diagonal of a matrix.Applied Numerical Mathematics, 57(11):1214–1229, 2007. Numerical Algorithms, Parallelism and Applications (2)

    2007

  7. [7]

    Straight to zero: Why linearly decaying the learning rate to zero works best for llms, 2025

    Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness. Straight to zero: Why linearly decaying the learning rate to zero works best for llms, 2025

    2025

  8. [8]

    Old optimizer, new norm: An anthology, 2024

    Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology, 2024

    2024

Show all 176 references
  1. [9]

    signsgd: Compressed optimisation for non-convex problems, 2018

    Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd: Compressed optimisation for non-convex problems, 2018

    2018

  2. [10]

    Pythia: A suite for analyzing large language models across training and scaling, 2023

    Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language mode...

    2023

  3. [11]

    Prince, Björn Deiseroth, Andres Felipe Cruz-Salinas, Carlo Luschi, Samuel Weinbach, and Douglas Orr

    Charlie Blake, Constantin Eichenberg, Josef Dean, Lukas Balles, Luke Y . Prince, Björn Deiseroth, Andres Felipe Cruz-Salinas, Carlo Luschi, Samuel Weinbach, and Douglas Orr. u-µp: The unit-scaled maximal update parametrization, 2025

    2025

  4. [12]

    Loss-to- loss prediction: Scaling laws for all datasets, 2024

    David Brandfonbrener, Nikhil Anand, Nikhil Vyas, Eran Malach, and Sham Kakade. Loss-to- loss prediction: Scaling laws for all datasets, 2024

    2024

  5. [13]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Je...

    2020

  6. [14]

    How to scale your ema, 2023

    Dan Busbridge, Jason Ramapuram, Pierre Ablin, Tatiana Likhomanenko, Eeshan Gunesh Dhekane, Xavier Suau, and Russ Webb. How to scale your ema, 2023. 16

    2023

  7. [15]

    Preconditioned spectral descent for deep learning

    David Edwin Carlson, Edo Collins, Ya-Ping Hsieh, Lawrence Carin, and V olkan Cevher. Preconditioned spectral descent for deep learning. In Neural Information Processing Systems, 2015

    2015

  8. [16]

    Communication-efficient language model training scales reliably and robustly: Scaling laws for diloco, 2025

    Zachary Charles, Gabriel Teston, Lucio Dery, Keith Rush, Nova Fallen, Zachary Garrett, Arthur Szlam, and Arthur Douillard. Communication-efficient language model training scales reliably and robustly: Scaling laws for diloco, 2025

    2025

  9. [17]

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V . Le. Symbolic discovery of optimization algorithms, 2023

    2023

  10. [18]

    Gradient clipping improves adagrad when the noise is heavy-tailed, 2024

    Savelii Chezhegov, Yaroslav Klyukin, Andrei Semenov, Aleksandr Beznosikov, Alexander Gasnikov, Samuel Horváth, Martin Takáˇc, and Eduard Gorbunov. Gradient clipping improves adagrad when the noise is heavy-tailed, 2024

    2024

  11. [19]

    Palm: Scaling language modeling with pathways, 2022

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, and Hyung Won. Palm: Scaling language modeling with pathways, 2022

    2022

  12. [20]

    George E. Dahl, Frank Schneider, Zachary Nado, Naman Agarwal, Chandramouli Shama Sastry, Philipp Hennig, Sourabh Medapati, Runa Eschenhagen, Priya Kasimbeg, Daniel Suo, Juhan Bae, Justin Gilmer, Abel L. Peirson, Bilal Khan, Rohan Anil, Mike Rabbat, Shankar Krishnan, Daniel Sni...

    2023

  13. [21]

    Why do we need weight decay in modern deep learning?, 2024

    Francesco D’Angelo, Maksym Andriushchenko, Aditya Varre, and Nicolas Flammarion. Why do we need weight decay in modern deep learning?, 2024

    2024

  14. [22]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022

    2022

  15. [23]

    Deepseek llm: Scaling open-source language models with longtermism, 2024

    DeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism, 2024

    2024

  16. [24]

    Deepseek-v3 technical report, 2024

    DeepSeek-AI. Deepseek-v3 technical report, 2024

    2024

  17. [25]

    Why gradients rapidly increase near the end of training, 2025

    Aaron Defazio. Why gradients rapidly increase near the end of training, 2025

    2025

  18. [26]

    Optimal linear decay learning rate schedules and further refinements, 2024

    Aaron Defazio, Ashok Cutkosky, Harsh Mehta, and Konstantin Mishchenko. Optimal linear decay learning rate schedules and further refinements, 2024

    2024

  19. [27]

    The road less scheduled, 2024

    Aaron Defazio, Xingyu Alice Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky. The road less scheduled, 2024

    2024

  20. [28]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    2019

  21. [29]

    The practitioner’s guide to the maximal update parameterization

    Nolan Dey, Quentin Anthony, and Joel Hestness. The practitioner’s guide to the maximal update parameterization. https://www.cerebras.ai/blog/ the-practitioners-guide-to-the-maximal-update-parameterization , Septem- ber 2024

    2024

  22. [30]

    Incorporating Nesterov Momentum into Adam, 2016

    Timothy Dozat. Incorporating Nesterov Momentum into Adam, 2016. ICLR 2016 Workshop

    2016

  23. [31]

    Understanding emergent abilities of language models from the loss perspective, 2025

    Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective, 2025

    2025

  24. [32]

    Adaptive subgradient methods for online learning and stochastic optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121–2159, 2011

    2011

  25. [33]

    A simple convergence proof of adam and adagrad, 2022

    Alexandre Défossez, Léon Bottou, Francis Bach, and Nicolas Usunier. A simple convergence proof of adam and adagrad, 2022. 17

    2022

  26. [34]

    Data movement bottlenecks to large-scale model training: Scaling past 1e28 flop,

    Ege Erdil. Data movement bottlenecks to large-scale model training: Scaling past 1e28 flop,

  27. [35]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

    2022

  28. [36]

    A stable whitening optimizer for efficient neural network training, 2025

    Kevin Frans, Sergey Levine, and Pieter Abbeel. A stable whitening optimizer for efficient neural network training, 2025

    2025

  29. [37]

    Dimakis, Gabriel Ilharco, Pang Wei Koh, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt

    Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Luca Soldaini, Alexandros G. Dimakis, Gabriel Ilharco, Pang W...

    2024

  30. [38]

    The Pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  31. [39]

    The pile: An 800gb dataset of diverse text for language modeling, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020

    2020

  32. [40]

    Gemini: A family of highly capable multimodal models, 2024

    Google Gemini Team. Gemini: A family of highly capable multimodal models, 2024

    2024

  33. [41]

    A loss curvature perspective on training instability in deep learning, 2021

    Justin Gilmer, Behrooz Ghorbani, Ankush Garg, Sneha Kudugunta, Behnam Neyshabur, David Cardoze, George Dahl, Zachary Nado, and Orhan Firat. A loss curvature perspective on training instability in deep learning, 2021

    2021

  34. [42]

    A minimalist optimizer design for llm pretraining, 2025

    Athanasios Glentis, Jiaxiang Li, Andi Han, and Mingyi Hong. A minimalist optimizer design for llm pretraining, 2025

    2025

  35. [43]

    Generating sequences with recurrent neural networks, 2014

    Alex Graves. Generating sequences with recurrent neural networks, 2014

    2014

  36. [44]

    Accelerating newton-schulz iteration for orthogonalization via chebyshev-type polynomials, 2025

    Ekaterina Grishina, Matvey Smirnov, and Maxim Rakhuba. Accelerating newton-schulz iteration for orthogonalization via chebyshev-type polynomials, 2025

    2025

  37. [45]

    AdaPlus: Integrating nesterov momentum and precise stepsize adjustment on adamw basis, 2023

    Lei Guan. AdaPlus: Integrating nesterov momentum and precise stepsize adjustment on adamw basis, 2023

    2023

  38. [46]

    Shampoo: Preconditioned stochastic tensor optimization, 2018

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization, 2018

    2018

  39. [47]

    Under- standing and minimising outlier features in neural network training, 2024

    Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, and Thomas Hofmann. Under- standing and minimising outlier features in neural network training, 2024

    2024

  40. [48]

    Deep residual learning for image recognition, 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015

    2015

  41. [49]

    Nicholas J. Higham. Functions of Matrices. Society for Industrial and Applied Mathematics, 2008

    2008

  42. [50]

    Neural networks for machine learning, lecture 6e rmsprop: Divide the gradient by a running average of its recent magnitude, 2012

    Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning, lecture 6e rmsprop: Divide the gradient by a running average of its recent magnitude, 2012. Coursera Lecture Notes

    2012

  43. [51]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022

  44. [52]

    Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai L...

    2024

  45. [53]

    Improving transformer opti- mization through better initialization

    Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Maksims V olkovs. Improving transformer opti- mization through better initialization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machi...

    2020

  46. [54]

    Scaling laws and compute-optimal training beyond fixed training durations, 2024

    Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro V on Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations, 2024

    2024

  47. [55]

    Averaging weights leads to wider optima and better generalization, 2019

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization, 2019

    2019

  48. [56]

    Intellect-1 technical report, 2024

    Sami Jaghouar, Jack Min Ong, Manveer Basra, Fares Obeid, Jannik Straube, Michael Keib- linger, Elie Bakouch, Lucas Atkins, Maziyar Panahi, Charles Goddard, Max Ryabinin, and Johannes Hagemann. Intellect-1 technical report, 2024

    2024

  49. [57]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024

  50. [58]

    modded-nanogpt: Speedrunning the nanogpt baseline, 2024

    Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, @fernbear.bsky.social, Boza Vlado, You Jiacheng, Franz Cesista, Braden Koszarsky, and @Grad62304977. modded-nanogpt: Speedrunning the nanogpt baseline, 2024

    2024

  51. [59]

    Muon: An optimizer for hidden layers in neural networks, 2024

    Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cecista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024

    2024

  52. [60]

    Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt J. Kusner. No train no gain: Revisiting efficient training algorithms for transformer-based language models, 2023

    2023

  53. [61]

    Why warmup the learning rate? underlying mechanisms and improvements, 2024

    Dayal Singh Kalra and Maissam Barkeshli. Why warmup the learning rate? underlying mechanisms and improvements, 2024

    2024

  54. [62]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020

    2020

  55. [63]

    Error feedback fixes signSGD and other gradient compression schemes

    Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signSGD and other gradient compression schemes. In ICML 2019 - International Conference on Machine Learning, pages 3252–3261. PMLR, 2019

    2019

  56. [64]

    Andrej Karpathy. NanoGPT. https://github.com/karpathy/nanoGPT, 2022

    2022

  57. [65]

    Priya Kasimbeg, Vincent Roulet, Naman Agarwal, Sourabh Medapati, Fabian Pedregosa, Atish Agarwala, and George E. Dahl. How far away are truly hyperparameter-free learning algorithms?, 2025

    2025

  58. [66]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017

    2017

  59. [67]

    Sign operator for coping with heavy-tailed noise in non-convex optimization: High probability bounds under (l0,l 1)-smoothness, 2025

    Nikita Kornilov, Philip Zmushko, Andrei Semenov, Mark Ikonnikov, Alexander Gasnikov, and Alexander Beznosikov. Sign operator for coping with heavy-tailed noise in non-convex optimization: High probability bounds under (l0,l 1)-smoothness, 2025

    2025

  60. [68]

    Analyzing & reducing the need for learning rate warmup in gpt training, 2024

    Atli Kosson, Bettina Messmer, and Martin Jaggi. Analyzing & reducing the need for learning rate warmup in gpt training, 2024. 19

    2024

  61. [69]

    Rotational equilibrium: How weight decay balances learning across neural networks, 2024

    Atli Kosson, Bettina Messmer, and Martin Jaggi. Rotational equilibrium: How weight decay balances learning across neural networks, 2024

    2024

  62. [70]

    Understanding gradient orthogonalization for deep learning via non-euclidean trust-region optimization, 2025

    Dmitry Kovalev. Understanding gradient orthogonalization for deep learning via non-euclidean trust-region optimization, 2025

    2025

  63. [71]

    Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan

    Tanishq Kumar, Zachary Ankner, Benjamin F. Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision, 2024

    2024

  64. [72]

    Why do machine learning optimizers that work, work? PhD thesis, University of British Columbia, 2024

    Frederik Kunstner. Why do machine learning optimizers that work, work? PhD thesis, University of British Columbia, 2024

    2024

  65. [73]

    Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be, 2023

    Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be, 2023

    2023

  66. [74]

    Heavy- tailed class imbalance and why adam outperforms gradient descent on language models, 2024

    Frederik Kunstner, Robin Yadav, Alan Milligan, Mark Schmidt, and Alberto Bietti. Heavy- tailed class imbalance and why adam outperforms gradient descent on language models, 2024

    2024

  67. [75]

    Farseer: A refined scaling law in large language models, 2025

    Houyi Li, Wenzhen Zheng, Qiufeng Wang, Zhenyu Ding, Haoying Wang, Zili Wang, Shijie Xuyang, Ning Ding, Shuigeng Zhou, Xiangyu Zhang, and Daxin Jiang. Farseer: A refined scaling law in large language models, 2025

    2025

  68. [76]

    Predictable scale: Part i – optimal hyperparameter scaling law in large language model pretraining, 2025

    Houyi Li, Wenzhen Zheng, Qiufeng Wang, Hanshan Zhang, Zili Wang, Shijie Xuyang, Yuantao Fan, Shuigeng Zhou, Xiangyu Zhang, and Daxin Jiang. Predictable scale: Part i – optimal hyperparameter scaling law in large language model pretraining, 2025

    2025

  69. [77]

    Datacomp-lm: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processin...

    2024

  70. [78]

    Pytorch distributed: Experiences on accelerating data parallel training, 2020

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. Pytorch distributed: Experiences on accelerating data parallel training, 2020

    2020

  71. [79]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training, 2024

    Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training, 2024

    2024

  72. [80]

    Same pre-training loss, better downstream: Implicit bias matters for language models, 2022

    Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better downstream: Implicit bias matters for language models, 2022

    2022

  73. [81]

    Muon is scalable for llm training, 2025

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Z...

    2025

  74. [82]

    The llama 3 herd of models, 2024

    AI @ Meta Llama Team. The llama 3 herd of models, 2024

    2024

  75. [83]

    Sgdr: Stochastic gradient descent with warm restarts, 2017

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, 2017

    2017

  76. [84]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019

    2019

  77. [85]

    SW AN: SGD with normalization and whitening enables stateless llm training, 2025

    Chao Ma, Wenbo Gong, Meyer Scetbon, and Edward Meeds. SW AN: SGD with normalization and whitening enables stateless llm training, 2025

    2025

  78. [86]

    Small batch size training for language models: When vanilla sgd works, and why gradient accumulation is wasteful, 2025

    Martin Marek, Sanae Lotfi, Aditya Somasundaram, Andrew Gordon Wilson, and Micah Goldblum. Small batch size training for language models: When vanilla sgd works, and why gradient accumulation is wasteful, 2025

    2025

  79. [87]

    New insights and perspectives on the natural gradient method, 2020

    James Martens. New insights and perspectives on the natural gradient method, 2020. 20

    2020

  80. [88]

    Effects of parameter norm growth during transformer training: Inductive bias from gradient descent, 2023

    William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, and Noah Smith. Effects of parameter norm growth during transformer training: Inductive bias from gradient descent, 2023

    2023

  81. [89]

    Mixed precision training, 2018

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training, 2018

    2018

  82. [90]

    Prodigy: An expeditiously adaptive parameter-free learner, 2024

    Konstantin Mishchenko and Aaron Defazio. Prodigy: An expeditiously adaptive parameter-free learner, 2024

    2024

  83. [91]

    Connections between schedule-free optimizers, ademamix, and accelerated sgd variants, 2025

    Depen Morwani, Nikhil Vyas, Hanlin Zhang, and Sham Kakade. Connections between schedule-free optimizers, ademamix, and accelerated sgd variants, 2025

    2025

  84. [92]

    Nemirovskii and Yu.E

    A.S. Nemirovskii and Yu.E. Nesterov. Optimal methods of smooth convex minimization. USSR Computational Mathematics and Mathematical Physics, 25(2):21–30, 1985

    1985

  85. [93]

    Nesterov and V

    Yu. Nesterov and V . Shikhman. Quasi-monotone Subgradient Methods for Nonsmooth Convex Minimization. Journal of Optimization Theory and Applications, 165(3):917–940, June 2015

    2015

  86. [94]

    A method for unconstrained convex minimization problem with the rate of convergenceo(1/k2), 1983

    Yurii Nesterov. A method for unconstrained convex minimization problem with the rate of convergenceo(1/k2), 1983

    1983

  87. [95]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024

    2024

  88. [96]

    Neural networks (maybe) evolved to make adam the best optimizer, 2020

    Francesco Orabona. Neural networks (maybe) evolved to make adam the best optimizer, 2020

    2020

  89. [97]

    Open problem: Parameter-free and scale-free online algorithms

    Francesco Orabona and Dávid Pál. Open problem: Parameter-free and scale-free online algorithms. In Vitaly Feldman, Alexander Rakhlin, and Ohad Shamir, editors, 29th Annual Conference on Learning Theory, volume 49 of Proceedings of Machine Learning Research, pages 1659–1664, Co...

    2016

  90. [98]

    In search of adam’s secret sauce, 2025

    Antonio Orvieto and Robert Gower. In search of adam’s secret sauce, 2025

    2025

  91. [99]

    The ademamix optimizer: Better, faster, older, 2024

    Matteo Pagliardini, Pierre Ablin, and David Grangier. The ademamix optimizer: Better, faster, older, 2024

    2024

  92. [100]

    Pytorch: An imperative style, high-performance deep learning library, 2019

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fan...

    2019

  93. [101]

    The fineweb datasets: Decanting the web for the finest text data at scale, 2024

    Guilherme Penedo, Hynek Kydlíˇcek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro V on Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024

    2024

  94. [102]

    Fineweb2: A sparkling update with 1000s of languages, December 2024

    Guilherme Penedo, Hynek Kydlí ˇcek, Vinko Sabol ˇcec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro von Werra, and Thomas Wolf. Fineweb2: A sparkling update with 1000s of languages, December 2024

    2024

  95. [103]

    Bowen Peng, Jeffrey Quesnelle, and Diederik P. Kingma. Demo: Decoupled momentum optimization, 2024

    2024

  96. [104]

    Training deep learning models with norm-constrained lmos, 2025

    Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and V olkan Cevher. Training deep learning models with norm-constrained lmos, 2025

    2025

  97. [105]

    Some methods of speeding up the convergence of iteration methods

    Boris Polyak. Some methods of speeding up the convergence of iteration methods. Ussr Computational Mathematics and Mathematical Physics, 4:1–17, 1964

    1964

  98. [106]

    New method of stochastic approximation type

    Boris Polyak. New method of stochastic approximation type. Automation and Remote Control, 1990, 01 1990

    1990

  99. [107]

    Resolving discrepancies in compute-optimal scaling of language models, 2024

    Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models, 2024. 21

    2024

  100. [108]

    Using the output embedding to improve language models, 2017

    Ofir Press and Lior Wolf. Using the output embedding to improve language models, 2017

    2017

  101. [109]

    Xiao, Maximilian Dax, Bernhard Schölkopf, and Weiyang Liu

    Zeju Qiu, Simon Buchholz, Tim Z. Xiao, Maximilian Dax, Bernhard Schölkopf, and Weiyang Liu. Reparameterized llm training via orthogonal equivalence transformation, 2025

    2025

  102. [110]

    Improving language understanding by generative pre-training

    Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018

    2018

  103. [111]

    Lan- guage models are unsupervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask learners. OpenAI, 2019

    2019

  104. [112]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023

    2023

  105. [113]

    Zero: Memory opti- mizations toward training trillion parameter models, 2020

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory opti- mizations toward training trillion parameter models, 2020

    2020

  106. [114]

    DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20...

    2020

  107. [115]

    Reddi, Satyen Kale, and Sanjiv Kumar

    Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond, 2019

    2019

  108. [116]

    Gluon: Making Muon & Scion great again! (bridging theory and practice of lmo-based optimizers for llms), 2025

    Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, and Peter Richtárik. Gluon: Making Muon & Scion great again! (bridging theory and practice of lmo-based optimizers for llms), 2025

    2025

  109. [117]

    A Stochastic Approximation Method

    Herbert Robbins and Sutton Monro. A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3):400 – 407, 1951

    1951

  110. [118]

    Efficient estimations from a slowly convergent robbins-monro process

    David Ruppert. Efficient estimations from a slowly convergent robbins-monro process. 1988

    1988

  111. [119]

    Beyond chinchilla-optimal: Accounting for inference in language model scaling laws, 2024

    Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws, 2024

    2024

  112. [120]

    The surprising agreement between convex optimization theory and learning-rate scheduling for large model training, 2025

    Fabian Schaipp, Alexander Hägele, Adrien Taylor, Umut Simsekli, and Francis Bach. The surprising agreement between convex optimization theory and learning-rate scheduling for large model training, 2025

    2025

  113. [121]

    Schmidt, Frank Schneider, and Philipp Hennig

    Robin M. Schmidt, Frank Schneider, and Philipp Hennig. Descending through a crowded valley - benchmarking deep learning optimizers, 2021

    2021

  114. [122]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024

    2024

  115. [123]

    Understanding Machine Learning: From Theory to Algorithms

    Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, USA, 2014

    2014

  116. [124]

    Glu variants improve transformer, 2020

    Noam Shazeer. Glu variants improve transformer, 2020

    2020

  117. [125]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017

    2017

  118. [126]

    Cox, and Rameswar Panda

    Yikang Shen, Matthew Stallone, Mayank Mishra, Gaoyuan Zhang, Shawn Tan, Aditya Prasad, Adriana Meza Soria, David D. Cox, and Rameswar Panda. Power scheduler: A batch size and token number agnostic learning rate scheduler, 2024

    2024

  119. [127]

    Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. 22

    2020

  120. [128]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    2024

  121. [129]

    Through the river: Under- standing the benefit of schedule-free methods for language model training, 2025

    Minhak Song, Beomhan Baek, Kwangjun Ahn, and Chulhee Yun. Through the river: Under- standing the benefit of schedule-free methods for language model training, 2025

    2025

  122. [130]

    Is your batch size the problem? revisiting the Adam-SGD gap in language modeling, 2025

    Teodora Sre´ckovi´c, Jonas Geiping, and Antonio Orvieto. Is your batch size the problem? revisiting the Adam-SGD gap in language modeling, 2025

    2025

  123. [131]

    GaLore 2: Large-scale llm pre-training by gradient low-rank projection, 2025

    DiJia Su, Andrew Gu, Jane Xu, Yuandong Tian, and Jiawei Zhao. GaLore 2: Large-scale llm pre-training by gradient low-rank projection, 2025

    2025

  124. [132]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023

    2023

  125. [133]

    On the importance of initialization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, Proceedings of Machine Le...

    2013

  126. [134]

    Adopt: Modified adam can converge with anyβ2 with the optimal rate, 2024

    Shohei Taniguchi, Keno Harada, Gouki Minegishi, Yuta Oshima, Seong Cheol Jeong, Go Na- gahara, Tomoshi Iiyama, Masahiro Suzuki, Yusuke Iwasawa, and Yutaka Matsuo. Adopt: Modified adam can converge with anyβ2 with the optimal rate, 2024

    2024

  127. [135]

    Scale efficiently: Insights from pre-training and fine-tuning transformers, 2022

    Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers, 2022

    2022

  128. [136]

    2 OLMo 2 furious, 2024

    Team OLMo. 2 OLMo 2 furious, 2024

    2024

  129. [137]

    OLMo: Accelerating the science of language models, 2024

    Team OLMo. OLMo: Accelerating the science of language models, 2024

    2024

  130. [138]

    On the stability of inverse problems

    Andrey Nikolayevich Tikhonov et al. On the stability of inverse problems. In Dokl. akad. nauk sssr, volume 39, pages 195–198, 1943

    1943

  131. [139]

    Understanding why adam outperforms sgd: Gradient heterogeneity in transformers, 2025

    Akiyoshi Tomihari and Issei Sato. Understanding why adam outperforms sgd: Gradient heterogeneity in transformers, 2025

    2025

  132. [140]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langua...

    2023

  133. [141]

    SOAP: Improving and stabilizing shampoo using Adam, 2024

    Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. SOAP: Improving and stabilizing shampoo using Adam, 2024

    2024

  134. [142]

    Closing the gap between the upper bound and the lower bound of adam’s iteration complexity, 2023

    Bohan Wang, Jingwen Fu, Huishuai Zhang, Nanning Zheng, and Wei Chen. Closing the gap between the upper bound and the lower bound of adam’s iteration complexity, 2023

    2023

  135. [143]

    GradPower: Powering gradients for faster language model pre-training, 2025

    Mingze Wang, Jinbo Wang, Jiaqi Zhang, Wei Wang, Peng Pei, Xunliang Cai, Weinan E, and Lei Wu. GradPower: Powering gradients for faster language model pre-training, 2025

    2025

  136. [144]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022

    2022

  137. [145]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

    2023

  138. [146]

    BLOOM: A 176b-parameter open-access multilingual language model, 2023

    BigScience Workshop. BLOOM: A 176b-parameter open-access multilingual language model, 2023. 23

    2023

  139. [147]

    Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D

    Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl- dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Small-scale proxies for larg...

    2023

  140. [148]

    Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models, 2024

    Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models, 2024

    2024

  141. [149]

    On layer normalization in the transformer architecture, 2020

    Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the transformer architecture, 2020

    2020

  142. [150]

    Unveiling downstream performance scaling of llms: A clustering-based perspective, 2025

    Chengyin Xu, Kaiyuan Chen, Xiao Li, Ke Shen, and Chenggang Li. Unveiling downstream performance scaling of llms: A clustering-based perspective, 2025

    2025

  143. [151]

    Baichuan 2: Open large-scale language models

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023

    2023 arXiv

  144. [152]

    Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao

    Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer, 2022

    2022

  145. [153]

    Mars: Unleashing the power of variance reduction for training large models, 2024

    Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. Mars: Unleashing the power of variance reduction for training large models, 2024

    2024

  146. [154]

    Adaptive methods for nonconvex optimization

    Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive methods for nonconvex optimization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 3...

    2018

  147. [155]

    Scaling vision trans- formers, 2022

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision trans- formers, 2022

    2022

  148. [156]

    Root mean square layer normalization, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019

    2019

  149. [157]

    How does critical batch size scale in pre-training?, 2024

    Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean Foster, and Sham Kakade. How does critical batch size scale in pre-training?, 2024

    2024

  150. [158]

    Why are adaptive methods good for attention models?Advances in Neural Information Processing Systems, 33:15383–15393, 2020

    Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models?Advances in Neural Information Processing Systems, 33:15383–15393, 2020

    2020

  151. [159]

    Why are adaptive methods good for attention models?, 2020

    Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank J Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models?, 2020

    2020

  152. [160]

    Why transformers need adam: A hessian perspective, 2024

    Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. Why transformers need adam: A hessian perspective, 2024

    2024

  153. [161]

    Decon- structing what makes a good optimizer for language models

    Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham Kakade. Decon- structing what makes a good optimizer for language models. ICLR, 2025

    2025

  154. [162]

    Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on sc...

    2023

  155. [163]

    Ronny Huang, and Tom Goldstein

    Chen Zhu, Renkun Ni, Zheng Xu, Kezhi Kong, W. Ronny Huang, and Tom Goldstein. Gradinit: Learning to initialize neural networks for stable and efficient training, 2021

    2021

  156. [164]

    Pan, Zhangyang Wang, and Jinwon Lee

    Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z. Pan, Zhangyang Wang, and Jinwon Lee. APOLLO: SGD-like memory, Adamw-level performance, 2025

    2025

  157. [165]

    St-moe: Designing stable and transferable sparse expert models, 2022

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022. 24 Contents 1 Introduction 1 2 Background & Related Work 2 3 Experimental Setup 4 4 Results 5 4.1 Be...

    2022

  158. [167]

    Adam-like methods: AdamW (Algorithm 1), ADOPT (Algorithm 2), and AdEMAMix (Algorithm 3)

  159. [168]

    Sign-based methods: Lion (Algorithm 4), Signum (Algorithms 5 and 6)

  160. [169]

    Approximate second-order optimizers: Muon (Algorithm 8), SOAP (Algorithm 10), and Sophia (Algorithm 11)

  161. [170]

    Learning rate / scheduler-free learning algorithms: Schedule-Free AdamW (Algorithm 12), Prodigy (Algorithm 13)

  162. [171]

    Invalid le ar nin g rate : { lr }

    MARS methods: (Algorithms 14, 15, 16). Notation. In our work, we denote vectors and matrices in bold, and scalars in regular type. Let L :D→ R be an empirical loss function parameterized byx and mapping a batch of inputsξ⊂D to R. Asg =∇xL (x,ξ), we denote a stochastic gradient...

  163. [172]

    Given the slowly changing coordinate basis provided by eigenvectorsl andr, SOAP updates its second moment estimates in this basis; that is to say, it runs AdamW in another, a rotated space

  164. [173]

    In Algorithm 9, setting bothql andqr to the identity matrix would result in AdamW

    To update the eigenvectors of l andr, SOAP runs QR decomposition with the preconditioning frequencyϕ. In Algorithm 9, setting bothql andqr to the identity matrix would result in AdamW. The overall SOAP algorithm can be formalized as Algorithm 10. Sophia. Despite being named a ...

  165. [174]

    cleanliness

    ▷ Optional: use bias correction 8: rt←√β2rt−1 + (1−√β2)γtd2 t⟨gt,x0−xt⟩ 9: st←√β2st−1 + (1−√β2)γtd2 tgt 10: dt+1← max { dt, rt ∥st∥1 } 11: xt+1←xt−γtdt ( mt/ (√vt +dtε ) +λxt ) 12: end for 13: Return:xT A.5 MARS Very recently, Yuan, Liu et al. [153] introduced MARS—a family of...

    1920

  166. [175]

    Dampening update: {mt←βmt−1 + (1−τ)gt, xt+1←xt−γt (sign (mt) +λxt). 50

  167. [176]

    implicit warmup

    The “plain” update of Signum without Nesterov momentum: {mt←βmt−1 +gt, xt+1←xt−γt (sign (mt) +λxt). 2:1 16:8 Tokens (B) 22 24 26 28 30 32 Final Validation Perplexity Nesterov Dampening Plain Figure 38: Comparison of different update rules for Signum. We evaluate three variants...

    1984

  168. [2024]

    Accessed: 2025-01-19

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.