Pith. sign in

REVIEW 5 cited by

A Theory on Adam Instability in Large-Scale Machine Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.09871 v2 pith:SO76PEDC submitted 2023-04-19 cs.LG cs.AImath.OC

Igor Molybog , Peter Albert , Moya Chen , Zachary DeVito , David Esiobu , Naman Goyal , Punit Singh Koura , Sharan Narang
show 9 more authors
Andrew Poulton Ruan Silva Binh Tang Diana Liskovich Puxin Xu Yuchen Zhang Melanie Kambadur Stephen Roller Susan Zhang
This is my paper
classification cs.LGcs.AImath.OC
keywords trainingbillionadamlanguagelargetheoryargueartifact
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present a theory for the previously unexplained divergent behavior noticed in the training of large language models. We argue that the phenomenon is an artifact of the dominant optimization algorithm used for training, called Adam. We observe that Adam can enter a state in which the parameter update vector has a relatively large norm and is essentially uncorrelated with the direction of descent on the training loss landscape, leading to divergence. This artifact is more likely to be observed in the training of a deep model with a large batch size, which is the typical setting of large-scale language model training. To argue the theory, we present observations from the training runs of the language models of different scales: 7 billion, 30 billion, 65 billion, and 546 billion parameters.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. One QK Channel, Many Sources: Guarding Low-Precision Attention Collapse

    cs.LG 2026-08 conditional novelty 7.0 of 10

    Different low-precision errors converge on the same query-key spectral runaway, entry is gated by temporal sign-coherence, and a dormant query-key normalization guard contains it.

  2. Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Weight decay on scale-invariant weights creates a norm-dependent sharpness boundary; crossing it predicts loss spikes in normalized networks.

  3. SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models

    cs.CV 2025-10 conditional novelty 5.0 of 10

    A unified multimodal model can improve its own text-to-image generation by using its understanding module as a rewarder in a global-plus-local reward-weighted training loop.

  4. SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam

    cs.LG 2025-07 reject novelty 5.0 of 10

    S3, an optimizer with a p-th order momentum denominator, equal EMA coefficients, and Nesterov acceleration, is claimed to match AdamW's 100k-step perplexity at 50k steps while avoiding loss spikes.

  5. BlueLM-2.5-3B Technical Report

    cs.AI 2025-07 conditional novelty 5.0 of 10

    BlueLM-2.5-3B is a small multimodal model with a switchable thinking mode that reportedly matches larger models like Qwen3-4B and comes close to Kimi-VL-A3B-16B on many benchmarks.

Pith tools