REVIEW 5 cited by
On exponential convergence of SGD in non-convex over-parametrized learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large over-parametrized models learned via stochastic gradient descent (SGD) methods have become a key element in modern machine learning. Although SGD methods are very effective in practice, most theoretical analyses of SGD suggest slower convergence than what is empirically observed. In our recent work [8] we analyzed how interpolation, common in modern over-parametrized learning, results in exponential convergence of SGD with constant step size for convex loss functions. In this note, we extend those results to a much broader non-convex function class satisfying the Polyak-Lojasiewicz (PL) condition. A number of important non-convex problems in machine learning, including some classes of neural networks, have been recently shown to satisfy the PL condition. We argue that the PL condition provides a relevant and attractive setting for many machine learning problems, particularly in the over-parametrized regime.
Forward citations
Cited by 5 Pith papers
-
Linear Convergence of Adaptive Stochastic Gradient Descent
AdaGrad-Norm provably reaches ε error in O(log 1/ε) iterations for strongly convex and PL objectives from any initial step size, under new RUIG and zero-noise-at-optimum assumptions.
-
Sharp First-Order Lower Bounds under $\alpha$-Polyak-Lojasiewicz Conditions
Proves minimax lower bounds for first-order methods under sublevel alpha-PL conditions that match gradient descent and SGD upper bounds after showing global alpha-PL plus smoothness forces constant functions.
-
Structure Before Collapse: Transient semantic geometry in next-token prediction
Semantic geometry emerges transiently early in next-token prediction training before collapsing to Neural Collapse symmetry in synthetic settings with latent semantic factors.
-
A Stochastic Gradient Descent Method for Globally Minimizing Nearly Convex Functions
A noisy gradient descent with adaptive Gaussian noise converges linearly to the global minimizer of nearly convex functions when a sharp lower bound is known.
-
A Generalized Energy-Based Adaptive Gradient Method for Optimization
A generalized energy-based adaptive gradient method achieves unconditional energy stability and optimal O(1/ε) stationary-point convergence for any smooth concave energy function; the log-energy variant ALEGD converge...
Discussion (0). Continue with ORCID to comment.