Pith. sign in

REVIEW 10 cited by

Why gradient clipping accelerates training: A theoretical justification for adaptivity

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1905.11881 v2 pith:ENLPZKPT submitted 2019-05-28 math.OC cs.LG

Jingzhao Zhang , Tianxing He , Suvrit Sra , Ali Jadbabaie This is my paper · ORCID
classification math.OCcs.LG
keywords gradientsmoothnesstrainingneuralclippingtheoreticalalgorithmscondition
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We provide a theoretical explanation for the effectiveness of gradient clipping in training deep neural networks. The key ingredient is a new smoothness condition derived from practical neural network training examples. We observe that gradient smoothness, a concept central to the analysis of first-order optimization algorithms that is often assumed to be a constant, demonstrates significant variability along the training trajectory of deep neural networks. Further, this smoothness positively correlates with the gradient norm, and contrary to standard assumptions in the literature, it can grow with the norm of the gradient. These empirical observations limit the applicability of existing theoretical analyses of algorithms that rely on a fixed bound on smoothness. These observations motivate us to introduce a novel relaxation of gradient smoothness that is weaker than the commonly used Lipschitz smoothness assumption. Under the new condition, we prove that two popular methods, namely, \emph{gradient clipping} and \emph{normalized gradient}, converge arbitrarily faster than gradient descent with fixed stepsize. We further explain why such adaptively scaled gradient methods can accelerate empirical convergence and verify our results empirically in popular neural network training settings.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Convergence Behavior of Adam under Heavy-Tailed Noise

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Under heavy-tailed noise with bounded p-th moments, vector-form Adam converges to (ρ,ε)-stationary points at rate O(ε^{-(5p/(3p-4)+3/2)}) for p∈(4/3,2]; with known-radius clipping the rate is optimal O(ε^{-(p/(p-1)+3/2)}).

  2. Enroll-on-Wakeup: A First Comparative Study of Target Speech Extraction for Seamless Interaction in Real Noisy Human-Machine Dialogue Scenarios

    eess.AS 2026-02 conditional novelty 6.0 of 10

    Using the wake-up word as enrollment degrades current target-speech-extraction models; TTS cleanup improves perceived quality but not ASR accuracy.

  3. Sailing Towards Zero-Shot State Estimation using Foundation Models Combined with a UKF

    eess.SY 2025-09 conditional novelty 6.0 of 10

    A transformer dynamics model plugged into an unscented Kalman filter gives zero-shot state estimates on unseen simulated container ships and outperforms classical baselines when velocity sensors are missing.

  4. Revisiting Convergence: Shuffling Complexity Beyond Lipschitz Smoothness

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Shuffling gradient methods converge without Lipschitz smoothness under a sub-quadratic ℓ-smoothness condition, matching Lipschitz-case rates when ℓ is constant.

  5. Normalized First-Order Methods for Convex (L0, L1)-Smooth Optimization with Inexact Gradients

    math.OC 2026-07 conditional novelty 5.0 of 10

    Comparison-oracle variants of NGD and Polyak GD converge for convex (L0, L1)-smooth objectives when the normalized-gradient error δ is bounded by explicit O(√ε)-scale thresholds.

  6. Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Rates

    math.OC 2025-11 conditional novelty 5.0 of 10

    Non-Euclidean SGD variants (SignSGD, Muon) provably match adaptive optimizers' convergence rates under structured smoothness and noise assumptions.

  7. Why Do We Need Warm-up? A Theoretical Perspective

    cs.LG 2025-10 conditional novelty 5.0 of 10

    Under the proposed (H0,H1)-smoothness condition, gradient descent with a warm-up-style adaptive step-size provably converges faster than with any fixed step-size.

  8. Revisiting Glorot Initialization for Long-Range Linear Recurrences

    cs.LG 2025-05 reject novelty 5.0 of 10

    Glorot initialization is unstable for long-range linear RNNs because the spectral radius exceeds 1, with a proposed rescaling that helps in practice, but the theoretical claim that explosion starts at t = O(√n) is not...

  9. The Ball-Proximal (="Broximal") Point Method: a New Algorithm, Convergence Theory, and Applications

    math.OC 2025-02 conditional novelty 5.0 of 10

    A ball-constrained minimization oracle yields an idealized optimization method with finite and linear convergence for nonsmooth convex problems, plus a tailored 'ball-convex' nonconvex class.

  10. Generative Adversarial Networks Bridging Art and Machine Intelligence

    cs.LG 2025-02 unverdicted novelty 1.0 of 10

    This paper is a textbook-style review of generative adversarial networks, covering theory, classic variants, training methods, and applications; no new architecture, theorem, or experimental result is introduced.

Pith tools