pith. sign in

arxiv: 1811.08888 · v3 · pith:MJ2WA57Dnew · submitted 2018-11-21 · 💻 cs.LG · cs.AI· math.OC· stat.ML

Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks

Difan Zou , Yuan Cao , Dongruo Zhou , Quanquan Gu This is my paper
classification 💻 cs.LG cs.AImath.OCstat.ML
keywords descentgradientdeepstochasticnetworksrelutrainingloss
0
0 comments X
read the original abstract

We study the problem of training deep neural networks with Rectified Linear Unit (ReLU) activation function using gradient descent and stochastic gradient descent. In particular, we study the binary classification problem and show that for a broad family of loss functions, with proper random weight initialization, both gradient descent and stochastic gradient descent can find the global minima of the training loss for an over-parameterized deep ReLU network, under mild assumption on the training data. The key idea of our proof is that Gaussian random initialization followed by (stochastic) gradient descent produces a sequence of iterates that stay inside a small perturbation region centering around the initial weights, in which the empirical loss function of deep ReLU networks enjoys nice local curvature properties that ensure the global convergence of (stochastic) gradient descent. Our theoretical results shed light on understanding the optimization for deep learning, and pave the way for studying the optimization dynamics of training modern deep neural networks.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Limitations of Lazy Training of Two-layers Neural Networks

    stat.ML 2019-06 unverdicted novelty 8.0

    For quadratic targets in d dimensions, two-layer quadratic networks achieve lower risk when fully trained than in random features or neural tangent regimes if hidden units < d.

  2. Mild Over-Parameterization Benefits Asymmetric Tensor PCA

    cs.LG 2026-04 unverdicted novelty 7.0

    A three-phase alternating-update method for asymmetric tensor PCA achieves d to the power of k-minus-2 sample complexity with d-squared memory and improves when signal vectors align.

  3. Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

    cs.LG 2024-01 unverdicted novelty 6.0

    SPIN lets weak LLMs become strong by self-generating training data from previous model versions and training to prefer human-annotated responses over its own outputs, outperforming DPO even with extra GPT-4 data on be...

  4. On Symmetry and Initialization for Neural Networks

    cs.LG 2019-07 unverdicted novelty 5.0

    For symmetric target functions, chosen initial conditions in one-hidden-layer networks enable SGD to produce generalization guarantees, unlike random initialization.

  5. Convergence rates for gradient descent in the training of overparameterized artificial neural networks with piecewise affine activation

    cs.LG 2021-02 unverdicted novelty 4.0

    Batch gradient descent achieves linear convergence to zero MSE with high probability for sufficiently wide shallow NNs with non-affine piecewise affine activations and distinct inputs.

  6. Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization

    cs.LG 2019-07 unverdicted novelty 4.0

    Provides Hessian-based theoretical characterizations of SGD dynamics and a scale-invariant generalization bound for deep nets, backed by experiments on synthetic data, MNIST, and CIFAR-10.

  7. Two-block vs. Multi-block ADMM: An empirical evaluation of convergence

    stat.ML 2019-07 unverdicted novelty 4.0

    Empirical study finds multi-block ADMM outperforms two-block ADMM on optimization and prediction in multi-task learning across all tested datasets and dual step sizes.