REVIEW 7 cited by
Why Transformers Need Adam: A Hessian Perspective
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
SGD performs worse than Adam by a significant margin on Transformers, but the reason remains unclear. In this work, we provide an explanation through the lens of Hessian: (i) Transformers are "heterogeneous": the Hessian spectrum across parameter blocks vary dramatically, a phenomenon we call "block heterogeneity"; (ii) Heterogeneity hampers SGD: SGD performs worse than Adam on problems with block heterogeneity. To validate (i) and (ii), we check various Transformers, CNNs, MLPs, and quadratic problems, and find that SGD can perform on par with Adam on problems without block heterogeneity, but performs worse than Adam when the heterogeneity exists. Our initial theoretical analysis indicates that SGD performs worse because it applies one single learning rate to all blocks, which cannot handle the heterogeneity among blocks. This limitation could be ameliorated if we use coordinate-wise learning rates, as designed in Adam.
Forward citations
Cited by 7 Pith papers
-
Muse: Representation Geometry of Muon Beyond Normalized Momentum
The matrix shape given to Muon's orthonormalization step is a genuine optimizer axis: square-ish reshapes match native Muon, skinnier reshapes interpolate toward normalized SGD with momentum.
-
Sketched Gaussian Mechanism for Private Federated Learning
A sketched Gaussian mechanism is shown to have privacy loss that shrinks as the sketch dimension grows, giving communication-efficient federated learning with stronger privacy per noise budget.
-
Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling
SGD with momentum can match Adam's performance in language modeling when trained with small batches and careful tuning, a result that contradicts several popular explanations for the optimizer gap.
-
Spectral-factorized Positive-definite Curvature Learning for NN Training
The paper derives a Riemannian update rule for the spectral factors of a positive-definite preconditioner, making arbitrary matrix roots fast and numerically stable for low-precision NN training.
-
PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflicts
PiKE adaptively re-weights pretraining data sources by gradient magnitude and variance, exploiting low gradient conflicts to speed up convergence and improve downstream accuracy in LLM pretraining.
-
Mechanistic Insights into Grokking from the Embedding Layer
Trainable embeddings in a simple MLP cause delayed generalization (grokking) on modular arithmetic, and a higher embedding learning rate plus balanced sampling accelerates it.
-
Improving Adaptive Moment Optimization via Preconditioner Diagonalization
Rotating gradients into their SVD coordinate system before Adam-style updates can roughly halve the number of steps LLaMA models need to reach a given perplexity.
Discussion (0). Continue with ORCID to comment.