REVIEW 3 cited by
Adam with model exponential moving average is effective for nonconvex optimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In this work, we offer a theoretical analysis of two modern optimization techniques for training large and complex models: (i) adaptive optimization algorithms, such as Adam, and (ii) the model exponential moving average (EMA). Specifically, we demonstrate that a clipped version of Adam with model EMA achieves the optimal convergence rates in various nonconvex optimization settings, both smooth and nonsmooth. Moreover, when the scale varies significantly across different coordinates, we demonstrate that the coordinate-wise adaptivity of Adam is provably advantageous. Notably, unlike previous analyses of Adam, our analysis crucially relies on its core elements -- momentum and discounting factors -- as well as model EMA, motivating their wide applications in practice.
Forward citations
Cited by 3 Pith papers
-
The Convergence Behavior of Adam under Heavy-Tailed Noise
Under heavy-tailed noise with bounded p-th moments, vector-form Adam converges to (ρ,ε)-stationary points at rate O(ε^{-(5p/(3p-4)+3/2)}) for p∈(4/3,2]; with known-radius clipping the rate is optimal O(ε^{-(p/(p-1)+3/2)}).
-
Differentially Private Natural Gradient Descent
DP-NGD enables second-order optimization under differential privacy by decoupling curvature estimation onto public data, performing isotropic DP operations in a whitened space, and dynamically clamping curvature eigen...
-
PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning
PADAM runs K differently averaged Adam trajectories in parallel, selects the one with the smallest test error, and achieves the best optimization error in nearly all of 13 tested scientific machine learning problems w...
Discussion (0). Continue with ORCID to comment.