Pith. sign in

REVIEW 5 cited by

Methods for Convex $(L_0,L_1)$-Smooth Optimization: Clipping, Acceleration, and Adaptivity

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.14989 v2 pith:GJVJ2WLH submitted 2024-09-23 math.OC cs.LG

Eduard Gorbunov , Nazarii Tupitsa , Sayantan Choudhury , Alen Aliev , Peter Richtárik , Samuel Horváth , Martin Takáč This is my paper · ORCID
classification math.OCcs.LG
keywords gradientconvergenceconvexderivedescentoptimizationratessmooth
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Due to the non-smoothness of optimization problems in Machine Learning, generalized smoothness assumptions have been gaining a lot of attention in recent years. One of the most popular assumptions of this type is $(L_0,L_1)$-smoothness (Zhang et al., 2020). In this paper, we focus on the class of (strongly) convex $(L_0,L_1)$-smooth functions and derive new convergence guarantees for several existing methods. In particular, we derive improved convergence rates for Gradient Descent with (Smoothed) Gradient Clipping and for Gradient Descent with Polyak Stepsizes. In contrast to the existing results, our rates do not rely on the standard smoothness assumption and do not suffer from the exponential dependency from the initial distance to the solution. We also extend these results to the stochastic case under the over-parameterization assumption, propose a new accelerated method for convex $(L_0,L_1)$-smooth optimization, and derive new convergence rates for Adaptive Gradient Descent (Malitsky and Mishchenko, 2020).

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under $(L_0, L_1)$-Smoothness

    math.OC 2025-02 conditional novelty 7.0 of 10

    First high-probability bounds for SignSGD with batching or majority voting under (L0, L1)-smoothness and heavy-tailed noise, with near-optimal epsilon-dependencies.

  2. Nonlinearly Preconditioned Gradient Methods under Generalized Smoothness

    math.OC 2025-02 conditional novelty 6.0 of 10

    The paper introduces (L,\bar L)-anisotropic smoothness and proves O(1/K) convergence rates for nonlinearly preconditioned gradient methods, unifying gradient clipping, Adam, and Adagrad under one theory.

  3. Normalized First-Order Methods for Convex (L0, L1)-Smooth Optimization with Inexact Gradients

    math.OC 2026-07 conditional novelty 5.0 of 10

    Comparison-oracle variants of NGD and Polyak GD converge for convex (L0, L1)-smooth objectives when the normalized-gradient error δ is bounded by explicit O(√ε)-scale thresholds.

  4. Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Rates

    math.OC 2025-11 conditional novelty 5.0 of 10

    Non-Euclidean SGD variants (SignSGD, Muon) provably match adaptive optimizers' convergence rates under structured smoothness and noise assumptions.

  5. Why Do We Need Warm-up? A Theoretical Perspective

    cs.LG 2025-10 conditional novelty 5.0 of 10

    Under the proposed (H0,H1)-smoothness condition, gradient descent with a warm-up-style adaptive step-size provably converges faster than with any fixed step-size.

Pith tools