REVIEW 7 cited by
Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This study investigates how weight decay affects the update behavior of individual neurons in deep neural networks through a combination of applied analysis and experimentation. Weight decay can cause the expected magnitude and angular updates of a neuron's weight vector to converge to a steady state we call rotational equilibrium. These states can be highly homogeneous, effectively balancing the average rotation -- a proxy for the effective learning rate -- across different layers and neurons. Our work analyzes these dynamics across optimizers like Adam, Lion, and SGD with momentum, offering a new simple perspective on training that elucidates the efficacy of widely used but poorly understood methods in deep learning. We demonstrate how balanced rotation plays a key role in the effectiveness of normalization like Weight Standardization, as well as that of AdamW over Adam with L2-regularization. Finally, we show that explicitly controlling the rotation provides the benefits of weight decay while substantially reducing the need for learning rate warmup.
Forward citations
Cited by 7 Pith papers
-
SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
SOAP and Muon, stabilized by per-step QR eigenbasis updates and KL-Shampoo covariance accumulation, beat AdamW on large-batch LLM pretraining up to 100M-token batches.
-
Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors
Splitting weight matrices into a fixed-norm direction and learnable per-row/column magnitudes improves LLM training over AdamW/Muon, removes weight decay and warmup, and transfers the optimal LR across width.
-
Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler
Cooldown shapes in WSD training trade off bias and variance, with sqrt and lowered-linear-0.7 shapes best for single runs, mirror-cosine best for averaging, and higher beta_2 during cooldown matching shape-selection gains.
-
Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
Lion and Muon with weight decay are shown to be instances of one stochastic Frank-Wolfe algorithm, and clipped and variance-reduced variants get the first high-probability convergence rates for nonconvex Frank-Wolfe u...
-
Scale Weight Decay and Train Better
Muon with weight decay scaled by η/η_max reaches the same MoE validation loss ~30% faster than constant-decay Muon while preserving asymptotic stationarity of the unregularized objective.
-
Taming Transformer Without Using Learning Rate Warmup
A spectral-norm ratio rule on the optimizer shrinks per-layer learning rates when weight updates are too large, letting Transformers train without warmup.
-
Energy Considerations for Large Pretrained Neural Networks
Fine-tuning pretrained CNNs on ImageNet used far less electricity when weights were quantized to about 8 to 13 bits, while pruning and low-rank factorization did not reliably reduce energy.
Discussion (0). Continue with ORCID to comment.