Pith. sign in

REVIEW 4 cited by

Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.19449 v2 pith:COB7HH6H submitted 2024-02-29 cs.LG cs.CLmath.OCstat.ML

Frederik Kunstner , Robin Yadav , Alan Milligan , Mark Schmidt , Alberto Bietti This is my paper
classification cs.LGcs.CLmath.OCstat.ML
keywords descentadamclassgradientimbalancelanguagelossmodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Adam has been shown to outperform gradient descent on large language models by a larger margin than on other tasks, but it is unclear why. We show that a key factor in this performance gap is the heavy-tailed class imbalance found in language tasks. When trained with gradient descent, the loss of infrequent words decreases more slowly than the loss of frequent ones. This leads to a slow decrease on the average loss as most samples come from infrequent words. On the other hand, Adam and sign-based methods are less sensitive to this problem. To establish that this behavior is caused by class imbalance, we show empirically that it can be reproduced across architectures and data types, on language transformers, vision CNNs, and linear models. On a linear model with cross-entropy loss, we show that class imbalance leads to imbalanced, correlated gradients and Hessians that have been hypothesized to benefit Adam. We also prove that, in continuous time, gradient descent converges slowly on low-frequency classes while sign descent does not.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Simple Convergence Proof of Adam From a Sign-like Descent Perspective

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Adam's O(1/T^1/4) convergence is proven from a sign-like descent perspective, but the dimension-free claim depends on restrictive coordinate-wise assumptions.

  2. Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling

    cs.LG 2025-06 conditional novelty 6.0 of 10

    SGD with momentum can match Adam's performance in language modeling when trained with small batches and careful tuning, a result that contradicts several popular explanations for the optimizer gap.

  3. On the Performance of Differentially Private Optimization with Heavy-Tail Class Imbalance

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Under heavy-tail class imbalance, subtracting the DP noise variance from Adam's second moment (DP-AdamBC) substantially improves learning of rare classes compared with DP gradient descent.

  4. SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam

    cs.LG 2025-07 reject novelty 5.0 of 10

    S3, an optimizer with a p-th order momentum denominator, equal EMA coefficients, and Nesterov acceleration, is claimed to match AdamW's 100k-step perplexity at 50k steps while avoiding loss spikes.

Pith tools