REVIEW 15 cited by
On the Variance of the Adaptive Learning Rate and Beyond
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Here, we study its mechanism in details. Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate (i.e., it has problematically large variance in the early stage), suggest warmup works as a variance reduction technique, and provide both empirical and theoretical evidence to verify our hypothesis. We further propose RAdam, a new variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate. Extensive experimental results on image classification, language modeling, and neural machine translation verify our intuition and demonstrate the effectiveness and robustness of our proposed method. All implementations are available at: https://github.com/LiyuanLucasLiu/RAdam.
Forward citations
Cited by 15 Pith papers
-
Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks
Bounded trajectories of a broad class of GD optimizers (Adam, RMSprop, NAG, Adan, etc.) converge with polynomial rates to critical points of KL objectives with locally Lipschitz gradients, covering analytic-activation...
-
HealthCAT: An Interpretable Encoder-only Transformer Framework for Health Indicator Prediction and Temporal Interpretation of Wearable Sensor Data
A transformer with AttentiveCAT yields class-specific, time-step importance scores for wearable health data, beats deep-learning baselines, and beats random time-step selection in masking tests.
-
Amortized Moment Matching for Visual Generation
Amortized Fréchet Distance uses neural nets to match conditional means and covariances, yielding stronger one-step visual generators than explicit FD-loss or multi-step teachers.
-
Payne4GAIN: NLTE Corrections for Red Giants in Milky Way Mapper using H-Band Neural Network Emulators
Applying NLTE physics to APOGEE H-band spectra shifts red-giant abundances by ~0.1 dex for Al, Mn, and Ti; the paper provides a 360k-star correction catalog.
-
A machine-learned probability distribution in the phase space of turbulent channel flow for synthetic turbulence and flow reconstruction
A flow-matching generative model trained on minimal conditional flow units approximates the invariant phase-space distribution of turbulent channel flow at Re_tau=180, enabling synthetic turbulence generation and flow...
-
OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.
-
Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors
Splitting weight matrices into a fixed-norm direction and learnable per-row/column magnitudes improves LLM training over AdamW/Muon, removes weight decay and warmup, and transfers the optimal LR across width.
-
TABX: A High-Throughput Sandbox Battle Simulator for Multi-Agent Reinforcement Learning
TABX is a JAX-based, GPU-accelerated, configurable multi-agent battle simulator that lets researchers vary units, terrain, and physics to benchmark cooperative MARL algorithms.
-
From Next Token Prediction to (STRIPS) World Models
Transformers trained via next-token prediction on action traces can learn STRIPS action models that support planning over exponentially many unseen initial states and goals.
-
Deep Potential: Recovering the gravitational potential and local pattern speed in the solar neighborhood with GDR3 using normalizing flows
Using normalizing flows and a neural network on Gaia DR3 data, the authors recover a local pattern speed of 28.2 km/s/kpc and a total matter density of 0.086 solar masses per cubic parsec within 1 kpc of the Sun.
-
GCR Spectra Reconstructed with Neutron Monitor Yield Function and Artificial Neural Networks: Comparison of Two Methods
Neural networks trained on worldwide neutron-monitor counts plus solar indices reconstruct daily proton and helium cosmic-ray spectra for 2006–2022, matching PAMELA and AMS-02 data and beating a yield-function/force-f...
-
Why Do We Need Warm-up? A Theoretical Perspective
Under the proposed (H0,H1)-smoothness condition, gradient descent with a warm-up-style adaptive step-size provably converges faster than with any fixed step-size.
-
Criticality analysis of nuclear binding energy neural networks
On a two-input nuclear binding energy network, the paper validates ANNFT predictions for variance, kurtosis, and an optimal depth-to-width ratio r*=0.034 under SGD, while adaptive optimizers obscure criticality.
-
SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam
S3, an optimizer with a p-th order momentum denominator, equal EMA coefficients, and Nesterov acceleration, is claimed to match AdamW's 100k-step perplexity at 50k steps while avoiding loss spikes.
-
Feature-Enhanced TResNet for Fine-Grained Food Image Classification
A TResNet variant with style recalibration and criss-cross-style attention reports modest Top-1 accuracy gains on two Chinese food benchmarks.
Discussion (0). Sign in to comment.