REVIEW 9 cited by
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
We empirically demonstrate that full-batch gradient descent on neural network training objectives typically operates in a regime we call the Edge of Stability. In this regime, the maximum eigenvalue of the training loss Hessian hovers just above the numerical value $2 / \text{(step size)}$, and the training loss behaves non-monotonically over short timescales, yet consistently decreases over long timescales. Since this behavior is inconsistent with several widespread presumptions in the field of optimization, our findings raise questions as to whether these presumptions are relevant to neural network training. We hope that our findings will inspire future efforts aimed at rigorously understanding optimization at the Edge of Stability. Code is available at https://github.com/locuslab/edge-of-stability.
Forward citations
Cited by 9 Pith papers
-
A Defense of the Quadratic Model
Local Taylor-expanded quadratic models reproduce a 150M-parameter LLM's validation loss for up to 10% of training late in the run, and LLM pretraining operates within a factor of 2 of a stochastic or deterministic edg...
-
Limitations of SGD for Multi-Index Models Beyond Statistical Queries
Vanilla SGD provably fails to learn periodic and low-information-exponent single/multi-index targets unless the input dimension is small or the number of iterations is large.
-
Universal One-third Time Scaling in Learning Peaked Distributions
Softmax + cross-entropy on peaked targets yields loss ∼ t^{−1/3}, giving an architecture-driven explanation for power-law LLM training time without power-law data.
-
Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks
Compute-optimally trained networks of different sizes show loss curves that collapse onto one universal curve after normalization; with learning rate decay, the collapse is tighter than seed-to-seed noise, providing a...
-
The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions
A single-weight perturbation at the very start of training makes otherwise identical neural networks diverge to different loss basins, and this sensitivity drops sharply within the first fraction of training.
-
Multigrade Neural Network Approximation
For every continuous function on a hypercube, there exists a fixed-width multigrade ReLU network whose residuals decrease monotonically pointwise and converge uniformly to zero.
-
Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning
Loss deceleration, a piecewise-linear break in log-log loss curves, is attributed to zero-sum learning where per-example gradients oppose one another, and scaling helps by mitigating it.
-
Models of Heavy-Tailed Mechanistic Universality
A new random matrix model with one structure parameter explains heavy-tailed spectra in trained networks, and yields scaling laws, optimizer-tail behavior, and a description of the five-plus-one phases of training.
-
EMA Without the Lag: Bias-Corrected Iterate Averaging Schemes
A bias-corrected exponential moving average (BEMA) is claimed to remove the lag of standard EMA weight averaging during LLM fine-tuning, improving convergence and final performance over EMA and vanilla training.
Discussion (0). Continue with ORCID to comment.