REVIEW 6 cited by
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
A longstanding goal in deep learning research has been to precisely characterize training and generalization. However, the often complex loss landscapes of neural networks have made a theory of learning dynamics elusive. In this work, we show that for wide neural networks the learning dynamics simplify considerably and that, in the infinite width limit, they are governed by a linear model obtained from the first-order Taylor expansion of the network around its initial parameters. Furthermore, mirroring the correspondence between wide Bayesian neural networks and Gaussian processes, gradient-based training of wide neural networks with a squared loss produces test set predictions drawn from a Gaussian process with a particular compositional kernel. While these theoretical results are only exact in the infinite width limit, we nevertheless find excellent empirical agreement between the predictions of the original network and those of the linearized version even for finite practically-sized networks. This agreement is robust across different architectures, optimization methods, and loss functions.
Forward citations
Cited by 6 Pith papers
-
Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Product
In double asymptotic limits, the squared singular value process of non-square matrix products obeys geometric Dyson Brownian motion whose T-transform solves a Burgers equation, producing the free log-normal law via fr...
-
Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
Under strong regularity assumptions, neural natural policy gradient converges to a global optimum at rate O(1/sqrt(T)), and neural vanilla policy gradient converges to a stationary point at the same rate.
-
Quantitative Understanding of PDF Fits and their Uncertainties
After an initial transient, a PDF-fitting neural network's output obeys f_t = U(t) f_0 + V(t) Y, a linear blend of the initial network and the data with explicit time-dependent operators.
-
Pre-Strings Lectures on Artificial Intelligence
Lecture notes define neural-network field theory and survey how it recovers known QFT/string results plus applied AI techniques for string problems.
-
Assessing Quantum Advantage for Gaussian Process Regression
Quantum algorithms for Gaussian process regression lose their exponential speedup because kernel matrix condition numbers grow at least linearly with dataset size.
-
Finite size corrections for neural network Gaussian processes
For finite single-hidden-layer networks with symmetric weight initialization, the output distribution is a Gaussian with an O(1/N) fourth-Hermite correction, an instance of the classical Edgeworth expansion.
Discussion (0). Continue with ORCID to comment.