REVIEW 3 cited by
Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Although ADAM is a very popular algorithm for optimizing the weights of neural networks, it has been recently shown that it can diverge even in simple convex optimization examples. Several variants of ADAM have been proposed to circumvent this convergence issue. In this work, we study the ADAM algorithm for smooth nonconvex optimization under a boundedness assumption on the adaptive learning rate. The bound on the adaptive step size depends on the Lipschitz constant of the gradient of the objective function and provides safe theoretical adaptive step sizes. Under this boundedness assumption, we show a novel first order convergence rate result in both deterministic and stochastic contexts. Furthermore, we establish convergence rates of the function value sequence using the Kurdyka-Lojasiewicz property.
Forward citations
Cited by 3 Pith papers
-
Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks
Bounded trajectories of a broad class of GD optimizers (Adam, RMSprop, NAG, Adan, etc.) converge with polynomial rates to critical points of KL objectives with locally Lipschitz gradients, covering analytic-activation...
-
Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer
For quadratic losses with two-point random data, stochastic Adam converges to the true minimizer if and only if the data is symmetrically distributed; for non-symmetric data it provably converges to a wrong point.
-
Sharp higher order convergence rates for the Adam optimizer
Adam can achieve the accelerated momentum convergence rate locally on smooth strongly convex problems when its momentum and step size are tuned to the condition number, while RMSprop is shown to converge at the slower...
Discussion (0). Continue with ORCID to comment.