Pith. sign in

REVIEW 5 cited by

Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.13960 v1 pith:5IM3U267 submitted 2023-04-27 cs.LG math.OC

Frederik Kunstner , Jacques Chen , Jonathan Wilder Lavington , Mark Schmidt This is my paper
classification cs.LGmath.OC
keywords adamdescentgradientnoisebatchbecausehypothesisoutperform
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The success of the Adam optimizer on a wide array of architectures has made it the default in settings where stochastic gradient descent (SGD) performs poorly. However, our theoretical understanding of this discrepancy is lagging, preventing the development of significant improvements on either algorithm. Recent work advances the hypothesis that Adam and other heuristics like gradient clipping outperform SGD on language tasks because the distribution of the error induced by sampling has heavy tails. This suggests that Adam outperform SGD because it uses a more robust gradient estimate. We evaluate this hypothesis by varying the batch size, up to the entire dataset, to control for stochasticity. We present evidence that stochasticity and heavy-tailed noise are not major factors in the performance gap between SGD and Adam. Rather, Adam performs better as the batch size increases, while SGD is less effective at taking advantage of the reduction in noise. This raises the question as to why Adam outperforms SGD in the full-batch setting. Through further investigation of simpler variants of SGD, we find that the behavior of Adam with large batches is similar to sign descent with momentum.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Overcoming the Communication-Performance Tradeoff in LLM Pretraining

    cs.LG 2025-08 conditional novelty 7.0 of 10

    SparseLoCo combines error feedback with Top-k sparsification and 2-bit quantization to send 1-3% of the pseudo-gradient during LLM pre-training while matching or beating DiLoCo's dense updates.

  2. Adaptive Momentum and Nonlinear Damping for Neural Network Training

    cs.LG 2026-01 conditional novelty 6.0 of 10

    Per-parameter kinetic-energy friction and cubic damping in momentum optimizers close much of the Adam–mSGD gap on transformer training, with deterministic exponential-convergence guarantees for strongly convex losses.

  3. Simple Convergence Proof of Adam From a Sign-like Descent Perspective

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Adam's O(1/T^1/4) convergence is proven from a sign-like descent perspective, but the dimension-free claim depends on restrictive coordinate-wise assumptions.

  4. On the Performance of Differentially Private Optimization with Heavy-Tail Class Imbalance

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Under heavy-tail class imbalance, subtracting the DP noise variance from Adam's second moment (DP-AdamBC) substantially improves learning of rare classes compared with DP gradient descent.

  5. SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam

    cs.LG 2025-07 reject novelty 5.0 of 10

    S3, an optimizer with a p-th order momentum denominator, equal EMA coefficients, and Nesterov acceleration, is claimed to match AdamW's 100k-step perplexity at 50k steps while avoiding loss spikes.

Pith tools