Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants

Depen Morwani; Hanlin Zhang; Nikhil Vyas; Sham Kakade

REVIEW 4 major objections 6 minor 2 cited by

Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Schedule-Free SGD and AdEMAMix sit inside a single accelerated-SGD framework, the paper argues.

desk verdict Clean algebraic unification of Schedule-Free SGD with accelerated SGD, but the AdEMAMix connection and the new optimizer's claims rest on a special case that the paper's own experiments show is brittle. read the letter →

arxiv 2502.02431 v1 pith:J7FIK7AB submitted 2025-02-04 cs.LG cs.AI

Depen Morwani , Nikhil Vyas , Hanlin Zhang , Sham Kakade This is my paper · ORCID

classification cs.LGcs.AI

keywords acceleratedSGDSchedule-FreeAdEMAMixmomentumweightaveragingnoise-dominatedregimeAdamWdeeplearningoptimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that several recently proposed deep-learning optimizers—Schedule-Free SGD/AdamW, Lion, MARS, and AdEMAMix—are not separate inventions but instances of one design: accelerated stochastic gradient descent, optionally with preconditioning and weight averaging. The central proof is algebraic: rewriting the Schedule-Free SGD update in terms of a momentum variable shows that its evaluation-iterate sequence follows the standard accelerated-SGD form, and its reported iterate is a weighted average of that sequence. The same lens places AdEMAMix closest to theoretically grounded accelerated SGD, which the paper says explains why it performs best in a noise-dominated small-batch regime. It then proposes Simplified-AdEMAMix, which keeps AdEMAMix-like performance with a single momentum buffer. If the connection is right, hyperparameter choices for these optimizers reduce to choosing a momentum strength, a current-gradient weight, and a weight-averaging schedule.

What carries the argument

The load-bearing object is the generalized accelerated-SGD update template $m_t=\beta_{a,t}m_{t-1}+g_t$ and $w_{t+1}=w_t-\eta_{a,t}m_t-\alpha_{a,t}g_t$, which separates the momentum coefficient from the weight put on the current gradient. The paper proves that Schedule-Free SGD lands inside this template through the substitution $m_{t+1}=(x_t-z_{t+1})/\gamma$, producing the exact coefficients $\beta_{a,t}=1-c_t$, $\eta_{a,t}=\gamma\beta c_{t+1}$, and $\alpha_{a,t}=\gamma(1-\beta)$, and that $x_t$ is a weighted average of $y_t$. That identity is what lets the paper compare optimizers term by term and port acceleration schedules between them.

What would settle it

Train the 150m model at 32k-token batch with full AdEMAMix ($\beta_1=0.9$) and with $\beta_1=0$ using the paper's sweeps; if the $\beta_1=0$ run's validation loss separates from the full run by more than run-to-run noise, the central AdEMAMix claim is confined to a special case. Alternatively, record $x_t,z_t,y_t,g_t$ from a Schedule-Free SGD run and check the identity $y_{t+1}=y_t-\gamma[\beta c_{t+1}m_t+(1-\beta)g_t]$ with $m_t=(x_{t-1}-z_t)/\gamma$; a persistent mismatch would falsify the claimed equivalence.

Watch

Extended reading notes

Core claim

The paper establishes an exact algebraic equivalence between Schedule-Free SGD and accelerated SGD followed by weight averaging. Defining $m_{t+1}=(x_t-z_{t+1})/\gamma$ in the Schedule-Free update yields $m_t=(1-c_t)m_{t-1}+g_t$ and $y_{t+1}=y_t-\gamma[\beta c_{t+1}m_t+(1-\beta)g_t]$, so the sequence $y_t$ on which gradients are evaluated obeys the general accelerated-SGD template $m_t=\beta_{a,t}m_{t-1}+g_t$, $w_{t+1}=w_t-\eta_{a,t}m_t-\alpha_{a,t}g_t$ with $\beta_{a,t}=1-c_t$, $\eta_{a,t}=\gamma\beta c_{t+1}$, and $\alpha_{a,t}=\gamma(1-\beta)$; the reported iterate $x_t$ is then an exponential average of $y_t$. The same template, read after setting $\beta_1=0$ in AdEMAMix, makes AdEMAMix's slow momentum term line up with the momentum schedule $\beta_{a,t}=1-k/t$ prescribed for accelerated SGD in noise, which the paper uses to explain its experimental ordering and to motivate Simplified-AdEMAMix.

Load-bearing premise

The load-bearing premise is that AdEMAMix with $\beta_1=0$ behaves almost identically to the full AdEMAMix in the small-batch regime where the comparison is made; the paper relies on a cited result and one set of small-batch experiments for this, while its large-batch figure shows the proxy failing.

Editorial extensions

If this is right

If the equivalence is right, Schedule-Free SGD and accelerated SGD with weight averaging are the same algorithm, so any difference between their empirical results is an implementation artifact, not a new method.
AdEMAMix's small-batch advantage is explained by its slow momentum term matching the accelerated-SGD schedule $\beta_{a,t}=1-k/t$; the advantage should shrink as the batch grows and the noise term weakens.
Large-batch degradation of Schedule-Free AdamW is caused by the coupling of momentum and weight-averaging coefficients, not by the order of preconditioning and momentum.
Simplified-AdEMAMix, with one momentum buffer and a fixed weight $\alpha$ on the current gradient, matches AdEMAMix in both regimes; at $\alpha=0$ it reduces to standard Adam with momentum scheduling.
Adam with a scheduled momentum coefficient can match AdEMAMix at large batch sizes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the mapping is generic, hyperparameter searches over these optimizers could be replaced by tuning three numbers—momentum strength, current-gradient weight, and averaging window—which is a testable claim for architectures beyond the 150m decoder-only transformer.
The paper's reading implies that any optimizer keeping a fixed weight on the current gradient will lose to a scheduled-weight version in low-noise (large-batch) settings, so applying accelerated-SGD schedules to existing Adam-family optimizers is a promising route to batch-size-robust training.
The AdEMAMix conclusion is derived through the $\beta_1=0$ simplification; if that proxy fails outside the tested regime, the practical takeaway is closer to 'use Adam with momentum scheduling' than to 'use two momentum terms.'

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The part of this paper worth remembering is Section 4.1. The rewrite of Schedule-Free SGD as accelerated SGD followed by weight averaging (Eqs. 2-11) is clean, self-contained, and, as far as I know, not in the earlier literature. It makes the relationship to Defazio (2021) precise and gives the separate works on accelerated SGD a concrete bridge to practical optimizers. That derivation will survive review and is a real contribution. The citation pattern is honest: Defazio (2021) is credited for anticipating some connections.

The rest of the paper is thinner. The AdEMAMix correspondence in Section 4.4 is explicitly for the beta1 = 0 variant. That's a legitimate special case to analyze, but the paper leans on it to say AdEMAMix "most closely resembles" accelerated SGD and therefore performs best. The justification for beta1 = 0 being faithful is a citation plus one visual comparison, with no repeated-seed measurements. The paper's own Figure 3 shows that the beta1 = 0 proxy degrades sharply at large batch size, and the authors' explanation is that the fast momentum term matters precisely there. So the practical-AdEMAMix story is at best heuristic, and in the small-batch regime where it is claimed to hold, the evidence is one unseeded curve.

The proposed Simplified-AdEMAMix is also undersold by its own results. At large batch, the best configuration has alpha = 0, which makes it standard Adam with a momentum schedule. Matching AdEMAMix by essentially becoming AdamW with the right beta1 schedule is an interesting observation, but it is not the "new optimizer" the abstract implies.

Empirically, everything is best runs with no seeds or error bars, so the comparative claims in Figures 1-3 should be treated as suggestive. That is a soft spot, not a fatal one, because the theoretical core does not depend on the experiments.

Who should read this: optimization theorists and people designing optimizers for LLM training. It reframes Schedule-Free SGD and gives a usable bridge to accelerated SGD. It will not change practice overnight, but the unification is useful and likely citable. I would send it to review. The authors should be asked to add seeded runs, to state plainly that the AdEMAMix connection holds only for beta1 = 0, and to frame Simplified-AdEMAMix as "Adam with scheduled momentum matches AdEMAMix" rather than a new algorithm. With those changes it is a solid workshop or conference paper.

Referee Report

4 major / 6 minor

Summary. The paper claims three main things. First, it derives an algebraic equivalence showing that Schedule-Free SGD's iterate y_t satisfies the accelerated-SGD update of Eq. (1), with the evaluation iterate x_t being an exponentially weighted average of y_t. Second, it argues that Lion, MARS, and AdEMAMix can also be viewed as preconditioned accelerated-SGD variants, with AdEMAMix (via a β1=0 proxy) most closely matching the accelerated-SGD template. Third, it proposes Simplified-AdEMAMix, an optimizer with a single momentum buffer plus a fixed current-gradient weight α, and reports 150M-parameter language-model experiments claiming that Simplified-AdEMAMix matches AdEMAMix in both small- and large-batch regimes. The paper's central conceptual contribution is the precise connection between Schedule-Free SGD and accelerated SGD.

Significance. If the claims held up, the paper would provide a useful unification: Schedule-Free optimizers, Lion, MARS, and AdEMAMix would be seen as instances of a common accelerated-SGD design space. The Section 4.1 derivation is self-contained, correct, and a genuine contribution. However, the empirical evidence is currently too weak to support the comparative claims, and the AdEMAMix connection is established only for a special case. The paper is honest about the β1=0 proxy's failure at large batch, but it does not resolve the mismatch between the theory (which applies to the proxy) and the headline empirical claims (which are about the full algorithm).

major comments (4)

[Section 4.4, Algorithm 1, Eq. (1)] The connection between AdEMAMix and accelerated SGD is derived only for the special case β1=0. With β1>0, the AdEMAMix update is θ_t ← θ_{t-1} - η(\hat m_1^{(t)} + α m_2^{(t)})/√\hat ν_t, which contains two momentum accumulators plus the current gradient and does not reduce to the single-momentum form of Eq. (1) by any transformation shown in the paper. The paper's justification that the β1=0 variant is nearly equivalent relies on a citation to Pagliardini et al. (2024) and on one unseeded visual comparison, while the paper's own Figure 3 shows that the proxy fails at batch size 1M tokens. Consequently, the statement that the practical AdEMAMix algorithm 'most closely resembles accelerated SGD' is not established; it is established only for a special case. Please provide either an exact folding of the two momentum terms into Eq. (1) or repeated-seed comparisons demonstrating that the β1=0 proxy faithfully represents full AdEMAMix in the small-batch regime.
[Section 5.1, Figure 3] The large-batch result for Simplified-AdEMAMix in Figure 3 is obtained at α=0.0, as stated in Section 5.1. At α=0.0, Algorithm 2 reduces to standard Adam with a scheduled momentum coefficient, so the experiment demonstrates that Adam with momentum scheduling can match AdEMAMix at large batch (the paper's contribution 3) rather than validating the proposed current-gradient weight α. The small-batch experiments use α∈{10,20,50,100} (Appendix A), so the configuration that achieves the claimed match differs qualitatively between the two regimes. The claim that Simplified-AdEMAMix 'maintains the same performance as AdEMAMix across both large and small batch-size settings' should be restricted to α=0 at large batch, or large-batch runs with nonzero α should be reported.
[Section 5, Figures 1-3] All empirical results are reported as best runs with no seeds or error bars. The paper's headline comparative claims—that AdEMAMix 'exhibits superior performance' and that Simplified-AdEMAMix 'matches' it—are therefore supported only by single trajectories. Since the differences between the optimizers in Figure 1 appear small (e.g., a spread on the order of 0.02 in validation loss), repeated-seed experiments are needed to establish that the ordering is not noise. Even one seed count per condition with a variance estimate across, say, three seeds would substantially strengthen the claims.
[Section 4.4, 'prescribed schedules' claim] The statement that 'the prescribed schedules for β3 and α in AdEMAMix closely match theoretical schedules proposed for accelerated SGD (Gupta et al., 2023)' is not backed by a derivation. The paper notes that β3 approximates 1−k/t and that α scales proportionally to t, but it does not specify the required k or the proportionality constant, nor does it show how AdEMAMix's schedule maps to the specific coefficients in Gupta et al.'s algorithm. As this is one of the two 'precise theoretical connections' promised in the introduction, the connection needs a formal statement (or at least a precise table of coefficient mappings) rather than a qualitative parallel.

minor comments (6)

[Section 4.1, Eq. (2)-(3)] Equation (2) defines m_{t+1} = (x_t − z_{t+1})/γ, but Eq. (3) and subsequent equations use m_t with the same right-hand side; please reindex so that the momentum variable in Eq. (7) and Eq. (11) is consistent.
[Title] The title is typeset as 'A DE-MAM IX' in the header; please correct to 'AdEMAMix'.
[Appendix A, item 4 and Figure 1] Figure 1's legend 'AdamW-wavg' is ambiguous: it could refer to item 3 (AdamW with constant fraction weight averaging) or item 4 (AdamW with cosine decay and weight averaging). Please clarify which configuration is plotted.
[Section 3.2] The definition γ_t ≈ 1 − 1/(δt) with '0 ≤ δ ≤ 1' is problematic for δ=0; please state the allowed range of δ (e.g., δ∈(0,1]).
[Section 5] The small-batch description says 'a batch size of 32' while the introduction says 'batch size of 32k tokens'; please make the units consistent (presumably 32 sequences of length 1024).
[Appendix B] The subsection title 'M ASS' should be 'MASS', and the method 'AGNES' from Gupta et al. (2023) is not defined in the main text or reference list; please add a brief description or citation for the reader.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central Schedule-Free SGD equivalence is derived self-containedly, and the AdEMAMix β1=0 proxy concern is an empirical faithfulness issue, not a definitional reduction.

full rationale

The central algebraic claim, that Schedule-Free SGD's y_t obeys the accelerated SGD update (Eq. 1) with βa,t = 1 − c_t, ηa,t = γβc_{t+1}, αa,t = γ(1 − β), and that x_t is a weighted average of y_t, is derived directly from the update equations in Section 4.1 with no fitted constants and no import of external results; it is therefore self-contained. The same holds for the rewriting of Lion and MARS in Sections 4.2–4.3. The AdEMAMix connection in Section 4.4 is not circular: the paper explicitly analyzes the β1 = 0 special case, cites external empirical evidence (Pagliardini et al. 2024) for the faithfulness of that special case at small batch size, and gives its own Figure 1 as corroboration. Whether that proxy is faithful at large batch is an empirical and correctness concern, not a definitional reduction; indeed, the paper's own Figure 3 shows the proxy degrades at large batch, and the paper does not claim the equivalence holds for full AdEMAMix in that regime. The self-citations present (Jain et al. 2018 and Kidambi et al. 2018, both co-authored by Kakade) are background references for known accelerated-SGD results and are not load-bearing for the paper's new derivations; the load-bearing schedule requirement is attributed to Gupta et al. (2023), which is external. No fitted parameter is renamed as a prediction, and the empirical claims are presented as observations rather than as forced consequences of the algebra. Overall, the derivation chain is independent and the remaining concerns are about empirical validity, not circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The central derivations use only elementary algebra and no fitted constants. The assumptions that matter are domain-level: the beta1=0 AdEMAMix proxy, the relevance of convex noise-dominated accelerated-SGD theory to transformer training, and the claim that the small-batch run is noise-dominated. The only hand-set values entering the proposed algorithm are its hyperparameters, especially alpha, which the experiments tune.

free parameters (2)

alpha (Simplified-AdEMAMix current-gradient weight) = 0.0 (large-batch best); 10-100 range swept (small-batch)
Algorithm 2 assigns a fixed weight alpha to the current gradient. The paper does not derive alpha from theory; in large-batch experiments the best value is 0.0, at which point Algorithm 2 is Adam with momentum scheduling.
beta1 schedule in Simplified-AdEMAMix and beta1 in AdamW baselines = swept values, e.g., beta1 in [0.9, 0.95, 0.99] for large batch and [0.99, 0.999, 0.9999] for small batch
Momentum coefficients are hyperparameters chosen by grid search, not derived from the theory; the performance claims depend on these choices.

assumptions (3)

domain assumption Accelerated SGD theory for smooth convex noisy problems transfers to non-convex transformer training.
Sections 3.3 and 4.4 use Gupta et al. schedule advice to explain AdEMAMix and to predict large-batch degradation; no proof is given for transformers.
domain assumption AdEMAMix with beta1=0 is a faithful proxy for full AdEMAMix.
Section 4.4 sets beta1=0 to align with accelerated SGD, citing Pagliardini et al.; the connection to the practical AdEMAMix depends on this proxy.
domain assumption The 32k-token batch, 15b-token run is noise-dominated.
Section 5 asserts this without measuring gradient noise or justifying the threshold.

invented entities (1)

Simplified-AdEMAMix independent evidence
purpose: Reduces AdEMAMix's two momentum buffers to one by using theory-style momentum plus a fixed current-gradient weight alpha.
It is a new algorithm, not a physical entity. Its independent evidence is the released code and the reported validation-curve comparisons; those comparisons are best-run only and at large batch size the best alpha=0 makes it Adam with momentum scheduling.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants." pith.science (2026). https://pith.science/paper/J7FIK7AB

@misc{pith2026250202431,
  author       = {Pith},
  title        = {Pith review of: Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J7FIK7AB}},
  note         = {Machine review of arXiv:2502.02431}
}

read the original abstract

Recent advancements in deep learning optimization have introduced new algorithms, such as Schedule-Free optimizers, AdEMAMix, MARS and Lion which modify traditional momentum mechanisms. In a separate line of work, theoretical acceleration of stochastic gradient descent (SGD) in noise-dominated regime has been achieved by decoupling the momentum coefficient from the current gradient's weight. In this paper, we establish explicit connections between these two lines of work. We substantiate our theoretical findings with preliminary experiments on a 150m language modeling task. We find that AdEMAMix, which most closely resembles accelerated versions of stochastic gradient descent, exhibits superior performance. Building on these insights, we introduce a modification to AdEMAMix, termed Simplified-AdEMAMix, which maintains the same performance as AdEMAMix across both large and small batch-size settings while eliminating the need for two different momentum terms. The code for Simplified-AdEMAMix is available on the repository: https://github.com/DepenM/Simplified-AdEMAMix/.

Figures

Figures reproduced from arXiv: 2502.02431 by the authors.

**Figure 2.** Comparison of the best runs of AdamW with cosine decay, schedule free AdamW and LAProp at higher [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

**Figure 3.** Comparison of the best runs of AdEMAMix (with and without [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Dimension-adapted Momentum Outscales SGD
stat.ML 2025-05 conditional novelty 7.0 of 10

DANA, with dimension- and time-dependent momentum, provably outscales SGD on power-law random features when 2α>1, improving loss exponents and compute-optimal curves.
OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
cs.LG 2026-07 conditional novelty 6.0 of 10

A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages · cited by 2 Pith papers

[1]

The optimal values of β1 and β2 were .9 and .999 respectively matching the default values

AdamW with cosine decay - 51.2k warmup - learning rate in [3.16e-4, 1e-3, 3.16e-3], β1 in [0.9, 0.95], β2 in [0.99, 0.999, 0.99968, 0.9999]. The optimal values of β1 and β2 were .9 and .999 respectively matching the default values. We note that for larger batch sizes it is common to useβ2 = .95, the benfit of higher β2 at smaller batch sizes has also been...

work page 2024
[2]

we fix β1, β2 to be the optimal values from the previous sweep

AdamW with cosine decay - 10k warmup - learning rate in [3.16e-4, 1e-3, 3.16e-3], β1 = 0.9, β2 = 0.999 i.e. we fix β1, β2 to be the optimal values from the previous sweep. This performed worse that warmup of 51.2k steps

work page
[3]

AdamW constant fraction weight averaging: - learning rate in [3.16e-4, 1e-3, 3.16e-3], β1 = 0.9, β2 in [0.99, 0.997, 0.999, 0.9997], δ in [0.05, 0.1, 0.2]

work page
[4]

AdamW with cosine decay and weight averaging - learning rate in [3.16e-4, 1e-3, 3.16e-3], β1 = 0 .9, β2 = 0.999, δ in [0.025, 0.05, 0.1]

work page
[5]

Accelerated SGD based AdamW with cosine decay - learning rate in [3.16e-4, 1e-3, 3.16e-3], β1 in [0.999, 0.99968, 0.9999], β2 in [0.99, 0.9968, 0.999], β3 = 0.9

work page
[6]

Accelerated SGD based AdamW with constant learning rate and weight averaging - learning rate in [3.16e-4, 1e-3, 3.16e-3], β1 in [0.99684, 0.999], β2 in [0.999], β3 = 0.9, δ in [0.05, 0.1]

work page
[7]

Accelerated SGD based AdamW with cosine decay and weight average - learning rate in [3.16e-4, 1e-3, 3.16e-3], β1 in [0.99684, 0.999], β2 = 0.999, δ in [0.05, 0.1], β3 = 0.9

work page
[8]

Schedulefree AdamW with constant learning rate - learning rate in [3.16e-4, 1e-3, 3.16e-3, 1e-2], β1 in [0.8, 0.9, 0.95], β2 = 0.999

work page

Show all 17 references

[9]

Schedulefree AdamW with cosine decay - [3.16e-4, 1e-3, 3.16e-3, 1e-2], β1 in [0.8, 0.9, 0.95], β2 = 0.999
[10]

MARS - [3.16e-4, 1e-3, 3.16e-3, 1e-2], β1 in [0.9, 0.95 0.99], β2 in [0.99, 0.999], γ in [0.0, 0.01, 0.02, 0.03, 0.04, 0.05], precondition 1d was set to True
[11]

AdEMAMix - [3.16e-4, 1e-3, 3.16e-3], β1 in [0.0, 0.9], β2 = 0 .999, β3 in [0.99, 0.999, 0.9999], α in [2,4,8,16]
[12]

Sim-AdEMAMix - [1e-6, 3.16e-6, 1e-5, 3.16e-5], β1 in [0.99, 0.999, 0.9999], β2 = 0.999, α in [10, 20, 50, 100] The hyperparameter sweeps for the large batch experiments are provided below:
[13]

Schedule-Free AdamW: [1e-3, 3.16e-3, 1e-2], β in [0.8,0.9,0.95], β2 in [0.9,0.95], r in [0.0, 5.0, 9.0, 50.0]
[14]

AdamW: [1e-3, 3.16e-3, 1e-2], β1 in [0.9,0.95], β2 in [0.9, 0.95]
[15]

LAProp: [1e-3, 3.16e-3, 1e-2], β1 in [0.9,0.95], β2 in [0.9, 0.95]
[16]

AdEMAMix: [1e-3, 3.16e-3, 1e-2], β1 in [0.0, 0.9], β2 = 0.95, β3 in [0.9, 0.95, 0.99], α in [2,4,8,16]
[17]

In this section, we will show that all the methods in the works Jain et al

Sim-AdEMAMix: [1e-4, 3.16e-4, 1e-3], β1 in [0.9, 0.95, 0.99], β2 = 0.95, α in [0.0, 0.5, 1.0] B E QUIVALENCE OF PREVIOUS ACCELERATION METHODS The general accelerated SGD form is provided in Equation (1). In this section, we will show that all the methods in the works Jain et a...

2018

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

The optimal values of β1 and β2 were .9 and .999 respectively matching the default values

AdamW with cosine decay - 51.2k warmup - learning rate in [3.16e-4, 1e-3, 3.16e-3], β1 in [0.9, 0.95], β2 in [0.99, 0.999, 0.99968, 0.9999]. The optimal values of β1 and β2 were .9 and .999 respectively matching the default values. We note that for larger batch sizes it is common to useβ2 = .95, the benfit of higher β2 at smaller batch sizes has also been...

work page 2024

[2] [2]

we fix β1, β2 to be the optimal values from the previous sweep

AdamW with cosine decay - 10k warmup - learning rate in [3.16e-4, 1e-3, 3.16e-3], β1 = 0.9, β2 = 0.999 i.e. we fix β1, β2 to be the optimal values from the previous sweep. This performed worse that warmup of 51.2k steps

work page

[3] [3]

AdamW constant fraction weight averaging: - learning rate in [3.16e-4, 1e-3, 3.16e-3], β1 = 0.9, β2 in [0.99, 0.997, 0.999, 0.9997], δ in [0.05, 0.1, 0.2]

work page

[4] [4]

AdamW with cosine decay and weight averaging - learning rate in [3.16e-4, 1e-3, 3.16e-3], β1 = 0 .9, β2 = 0.999, δ in [0.025, 0.05, 0.1]

work page

[5] [5]

Accelerated SGD based AdamW with cosine decay - learning rate in [3.16e-4, 1e-3, 3.16e-3], β1 in [0.999, 0.99968, 0.9999], β2 in [0.99, 0.9968, 0.999], β3 = 0.9

work page

[6] [6]

Accelerated SGD based AdamW with constant learning rate and weight averaging - learning rate in [3.16e-4, 1e-3, 3.16e-3], β1 in [0.99684, 0.999], β2 in [0.999], β3 = 0.9, δ in [0.05, 0.1]

work page

[7] [7]

Accelerated SGD based AdamW with cosine decay and weight average - learning rate in [3.16e-4, 1e-3, 3.16e-3], β1 in [0.99684, 0.999], β2 = 0.999, δ in [0.05, 0.1], β3 = 0.9

work page

[8] [8]

Schedulefree AdamW with constant learning rate - learning rate in [3.16e-4, 1e-3, 3.16e-3, 1e-2], β1 in [0.8, 0.9, 0.95], β2 = 0.999

work page