Pith. sign in

REVIEW 3 major objections 4 minor 92 references

Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One symmetry reduces MLPs, attention, and convolutions to a single quadratic model.

desk verdict NQF is a clean local normal form with real new content; the transfer to full-trajectory sudden learning and scaling laws is conditional on an unverified truncation bound, so read the derivation and the scaling claims separately. read the letter →

arxiv 2608.13335 v1 pith:RJ33B5N2 submitted 2026-08-13 cs.LG cond-mat.dis-nncond-mat.stat-mech

Liu Ziyin , Yizhou Xu , Tomaso Poggio , Isaac Chuang This is my paper · ORCID
classification cs.LGcond-mat.dis-nncond-mat.stat-mech
keywords neuralquadraticformpermutationsymmetrysuddenlearningscalinglawsLotka-Volterradynamicsfeaturestructurematrixsmallinitialization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the many architectures used in deep learning—perceptrons, attention heads, mixtures of experts, and convolutions—share one leading-order mathematical description when training starts from small weights. The description is a neural quadratic form, in which the only architecture-specific object is a structure matrix $A(x)$. The paper argues that this single model produces both sudden, plateau-like learning and smooth power-law scaling, with exponents predicted from the spectrum of $A(x)$.

What carries the argument

The central object is the neural quadratic form plus the structure matrix $A(x)$, which encodes the architecture. The argument then uses the order parameter $M=WW^\top$ and, under a commutative-eigenbasis assumption, reduces training to a generalized Lotka–Volterra system for the eigenvalues $z_k$ of $M$.

What would settle it

Train a two-layer MLP from a large initialization (say $\sigma=0.2$ or larger) on a low-rank target and check whether the loss trajectory diverges from the NQF prediction over hundreds of steps; the paper's own Figure 6 indicates divergence after about 100 steps in a teacher-student setting.

Watch

Extended reading notes

Core claim

Theorem 1 states that any three-times differentiable module with $S_d$-symmetric interchangeable components and zero gradient at zero obeys $f_x(W)=f_x(0)+\sum_i\mathrm{Tr}[w_iw_i^\top A(x)]+O(\|W\|^3)$. This forces a universal quadratic leading term where all architecture-specific detail is confined to $A(x)$. The paper computes $A(x)$ for specific architectures and shows how training dynamics close on the order parameter $M=WW^\top$, reduce to a Lotka–Volterra equation under commuting data, and generate sharp feature-wise or sample-wise transitions whose aggregate produces power-law scaling with predicted exponents.

Load-bearing premise

The quadratic truncation remains accurate throughout training, which requires the weights to stay small compared with the feature length scale set by the data; the paper shows empirically that this fails for large initialization.

Editorial extensions

If this is right

  • The same NQF predicts plateau-and-drop learning and smooth power-law loss decay within one framework.
  • Architectures with the same structure matrix have identical training dynamics at leading order.
  • Sudden learning is a singular limit in initialization scale; smaller initialization separates feature activation times as $\ln(1/\epsilon)$.
  • The scaling-law exponent is determined by the spectral decay of $A(x)$ and the target alignment.
  • A network can be compressed to a width proportional to the data subspace dimension without changing its learning dynamics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The theory implies that spectral properties of $A(x)$, rather than detailed parameter counts, govern which model sizes show a given learning curve.
  • One could test whether the predicted exponent holds for real datasets by estimating the spectrum of $A(x)$ from architecture and data alone.
  • Extending NQF to cubic and quartic terms (as the paper suggests) might explain additional plateaus in training curves.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper introduces the 'neural quadratic form' (NQF), a local normal form for neural network layers with exchangeable units. Theorem 1 states that any C^3 module that is S_d-symmetric and has zero gradient at the origin satisfies f_x(W)=f_x(0)+\sum_i Tr[w_i w_i^T A(x)]+O(||W||^3), with all architecture-specific information in A(x). The authors compute A(x) for MLPs, CNNs, multi-head attention, query-key-only attention, mixtures of experts, and several matrix factorization models. They show that SGD dynamics of the truncated quadratic model close on M=WW^T and \mu=\sum_i w_i (Theorem 2), that the model is compressible (Theorem 3), and that under commutativity or isotropy assumptions the dynamics reduce to Lotka-Volterra equations solvable in four regimes (Theorems 4-7). From these solutions they derive saddle-to-saddle plateaus with t_k^* ~ (1/\zeta_k) ln(1/\epsilon) and power-law decay of excess loss with exponents (\alpha_1-1)/\alpha_2 and (2\gamma_1-1)/\gamma_2. Experiments on synthetic data confirm the surrogate dynamics; a Fourier MLP experiment shows a power-law training loss. The paper candidly lists limitations: small initialization, smoothness, and the assumption of power-law spectra.

Significance. If the central transfer step were established, the paper would provide a genuinely unifying explanation of sudden learning and scaling laws across architectures, with quantitative, testable exponent predictions and explicit architecture-specific matrices A(x). The local expansion theorem and the exact solution theorems are derived in appendices and appear algebraically correct; the code for reproducing the figures is provided. The main weakness is that the solved dynamics are those of the truncated quadratic surrogate, and the O(||W||^3) remainder is not controlled along the trajectory up to feature saturation; the paper's own Figure 6 shows the approximation failing at finite times. The scaling-law exponents are also conditional on assumed power-law spectra. For these reasons the work is best regarded as a significant and useful surrogate-model theory whose transfer to real networks is not yet established.

major comments (3)
  1. [§2, Eq. (2); §5, Eq. (39); Appendix C, Fig. 6] Theorem 1 is a local statement at W=0: the remainder O(||W||^3) in Eq. (2) is controlled only while the weights remain near the expansion point. In the predicted ignition events, z_k(t) evolves from \Theta(\epsilon) to \Theta(\zeta_k/C_k)=\Theta(1) (Eqs. (39) and (42)), so ||W(t)|| grows from \Theta(\sqrt{\epsilon}) to \Theta(1), and the cubic remainder becomes O(1) precisely in the regime the theory is about. No trajectory-length bound is provided, and Appendix C Figure 6 shows that in a teacher-student MLP and in a query-key-only attention model the NQF and the original network agree for roughly 100 steps and then diverge. Limitation 1 (small initialization) does not rescue the claim because the divergence occurs at the timescale on which the theory's predicted signatures appear. Consequently, Theorems 8-11 establish the plateau times and exponent relations only for the truncated quadratic surrogate, not for the original architectures; the transfer to actual networks is an unverified uniformity assumption. I ask the authors to provide a trajectory error bound or to reframe the main claims as statements about the surrogate.
  2. [§5, Assumptions 1 and 2; §6, Eq. (46)] Section 5's exact solutions and Section 6's quantitative predictions require Assumption 1 (mutually commuting data matrices) or Assumption 2 (isotropic second moment), and the feature-wise results additionally require orthogonal features. These are strong, non-generic conditions. For example, for the two-layer MLP of Proposition 1, A(x) is proportional to [[0,x],[x^T,0]], and A(x)A(x') is not equal to A(x')A(x) for generic x and x', so Assumption 1 already fails on typical data; the empirical demonstrations in Figures 3 and 4 are engineered to satisfy the assumptions exactly. The paper gives no bound on the error when Assumptions 1 or 2 fail and does not show that plateaus or power laws persist for generic structure matrices. This scope restriction should be stated prominently in the introduction or abstract.
  3. [§7, Figures 2-6] The empirical support for the abstract's claim that sudden learning and scaling laws are confirmed 'across training methods and architectures' is limited. Figure 2 trains the original modules and their NQF approximations on an NQF teacher, which is a favorable setting; Figures 3 and 4 validate the Lotka-Volterra dynamics of the surrogate, not of original networks; and Figure 6, in the teacher-student setting, shows the NQF approximation diverging from the original after about 100 steps. The only original-network power-law experiment is the Fourier MLP in Figure 5. I recommend either adding experiments that verify the plateau times and the exponent predictions on original architectures under small initialization, or explicitly limiting the paper's claims to the NQF surrogate.
minor comments (4)
  1. [§6, Eq. (43); Appendix C] The initialization conventions are inconsistent: Theorem 8 assumes z_k(0)=\Theta(\epsilon), Figure 3 sets W(0)=\sqrt{\epsilon} I so that z_k(0)=\epsilon, while Figure 5 initializes W from N(0,\epsilon^2), giving z_k(0)=\Theta(\epsilon^2). This changes the prefactor of t_k^* by a factor of two; please standardize the definition of \epsilon or state the relation between \epsilon and z_k(0).
  2. [§7, Figure 2] The text uses \sigma and \epsilon interchangeably for the initialization scale; please use one symbol throughout and define it precisely.
  3. [Appendix C, Figure 6] The text says the NQF matches the original for the first 100 steps, but the agreement window is not marked in the figure; please annotate the divergence onset.
  4. [§9, Limitation 3] The limitation 'Other plateaus' states that the experiments show plateaus not accounted for by the NQF, but no specific experiment or figure is cited; please add a reference to the relevant panel.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the universal quadratic form is a self-contained Taylor expansion, and the scaling-law exponents follow from explicitly stated spectral assumptions rather than from fitted quantities.

full rationale

The paper's central derivation is self-contained. Theorem 1 is an explicit multivariate Taylor expansion: S_d-symmetry plus the ZGZ condition kills the first-order and cross-neuron second-order terms, and forces the remaining per-neuron Hessians to share one structure matrix A(x) (Appendix A.3, Eqs. (51)-(57)). No external or self-cited result is used to obtain this form; the citations to the authors' own prior work, such as [77] and [80], appear only as background about the role of permutation symmetry and are not load-bearing. The dynamics results (Theorems 2, 4-7) are derived from the NQF equations themselves and are not imported from earlier work. The sudden-learning times and scaling-law exponents in Section 6 are obtained analytically from the logistic feature equations under the explicit hypothesis of power-law spectra, and the paper states this plainly in Limitation 4: 'Power-law spectra are an assumption, not a prediction... Whether it arises generically, for realistic data and architectures, is the question that would have to be settled in future works.' Similarly, the Fourier-feature experiment constructs s_k and b_k with chosen power-law decays and then verifies the predicted exponent, so the exponent is not fitted from the measured loss curve. The residual concern, that the O(||W||^3) remainder of Theorem 1 is not controlled along the trajectory and that Figure 6 shows divergence after about 100 steps, is a validity and scope issue about transferring NQF dynamics to the original networks, not a circularity: the quadratic model's own predictions are derived from its own equations. No step reduces by construction to its inputs, and no load-bearing self-citation chain forces the claimed results.

Assumptions & free parameters 5 free parameters · 8 assumptions · 1 invented entities

The central normal form rests on symmetry and smoothness axioms; the dynamic and scaling predictions additionally require commuting data matrices, small initialization, and power-law spectra. The latter two are openly acknowledged by the authors. The per-experiment construction of power-law spectra means the scaling-law validation is a check of the derived exponent mapping under the assumptions, not a discovery that real datasets have such spectra.

free parameters (5)
  • Initialization scale epsilon = 10^-5 in Figure 3, 10^-4 in Figure 5
    Central asymptotics take epsilon to 0; NQF validity and plateau separation depend on small epsilon. The paper does not derive epsilon from data.
  • Spectral decay exponents alpha1, alpha2 = alpha1=2, alpha2=0.8 in Figure 3 right; alpha1=2*beta, alpha2=theta+beta in Figure 5
    The scaling-law exponent formula depends on these assumed power-law tails; they are inputs, not predicted quantities.
  • Initialization profile exponent beta0 = z_k(0)=epsilon*c_k*k^-1.2 in the Figure 3 experiment
    Theorem 9 assumes z_k(0)=epsilon*c_k*k^-beta0 with beta0 at least alpha1-alpha2; this profile is chosen, not derived.
  • Sample-wise exponents gamma1, gamma2 = assumed, not estimated from data
    Theorem 11 assumes y_mu ~ mu^-gamma1 and zeta_mu ~ mu^-gamma2; the resulting exponent depends on them.
  • Feature growth rates r_k and target strengths V_k = r_k=k^-0.8, V_k=k^-2 in Figure 3 right; from s_k=k^-theta and b_k=k^-beta in Figure 5
    These coefficients set activation times and exponents; in experiments they are constructed rather than measured from a natural dataset.
assumptions (8)
  • domain assumption The layer function is S_d-symmetric: relabeling the d interchangeable components leaves the output unchanged.
    Definition 1 states this as the starting point; it is the symmetry that forces the universal quadratic form.
  • domain assumption The module is three times continuously differentiable and satisfies ZGZ: each component's gradient vanishes when that component's weights are zero.
    Theorem 1 and Property 1; this excludes ReLU at the origin, bias terms, normalization, and post-aggregation nonlinearities, as Limitation 2 concedes.
  • ad hoc to paper The O(||W||^3) truncation error remains small along the entire training trajectory to feature saturation.
    Used to apply NQF dynamics to genuine networks; no theorem controls this error, and Figure 6 shows divergence after about 100 steps in a teacher-student run.
  • domain assumption The per-sample data matrices A(x_mu) commute and share a common eigenbasis.
    Assumption 1 in Section 5; the Lotka-Volterra reduction of z_k depends on it, and it fails for generic data.
  • ad hoc to paper The spectra of growth rates and target strengths have power-law tails, e.g., zeta_k ~ k^-alpha2 and V_k ~ k^-alpha1.
    Theorems 9 and 11 assume these laws; Section 9, Limitation 4 explicitly states they are assumptions, not predictions, and the experiments construct them.
  • domain assumption The initialization scale epsilon tends to zero and the widths or sample counts go to infinity in the relevant limits.
    Theorems 8 through 11 take epsilon to 0; the plateau separation and power-law limits depend on these limits.
  • domain assumption The training is full-batch or minibatch SGD with MSE loss and real-valued labels.
    The dynamics and exact solutions are derived for this setting; Adam and momentum are only empirically tracked.
  • standard math Standard Taylor expansion, ODE solvability, and summation limit theorems hold.
    Background results used throughout the appendix proofs.
invented entities (1)
  • Structure matrix A(x) independent evidence
    purpose: Encodes all architecture-specific information in the quadratic leading term; computed in Propositions 1 through 6 for MLP, CNN, attention variants, and mixtures of experts.
    It is defined directly from the architecture and input, and its predictions, such as query and key matrices of multi-head attention appearing only at quartic order, are falsifiable by comparing NQF dynamics to original training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws." pith.science (2026). https://pith.science/paper/RJ33B5N2

@misc{pith2026260813335,
  author       = {Pith},
  title        = {Pith review of: Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RJ33B5N2}},
  note         = {Machine review of arXiv:2608.13335}
}
abstract

Neural networks trained by gradient descent on a smooth cost function can nevertheless learn in steps: the cost holds on long plateaus and then drops abruptly. Meanwhile, training losses instead follow smooth power laws. Variants of both behaviors occur in architectures with very different microscopic structures, which is the signature of a few relevant collective variables. We show that a symmetry fixes what those variables are: a network layer is a sum over interchangeable units, so relabeling the units leaves it unchanged; given smoothness and the condition that a unit's gradient vanish at the origin, symmetry then enforces a universal leading form for the expansion about the near-zero weights present at the start of training, the quadratic $\Tr[WW^{\top}A(x)]$, in which every architectural detail is confined to a single ``structure matrix" $A(x)$ that we compute for each architecture. Perceptrons, attention layers, mixtures of experts, and convolutions become one model at different $A$. Its training dynamics then close on the ``order parameter" $M=WW^{\top}$ and, whenever the data matrices share an eigenbasis, reduce to a Lotka--Volterra equation whose modes switch on one after another. The smaller the initial weights, the further apart the switch-on times, and the plateaus appear as a singular limit of a smooth flow; when many modes are unresolved the same events merge into a power law in training time whose exponent the theory predicts. We confirm both numerically across training methods and architectures.

Figures

Figures reproduced from arXiv: 2608.13335 by the authors.

Figure 1
Figure 1. Permutation symmetry of the units allows us to characterize the model with structure matrices A. (a) Perceptrons, attention layers, mixtures of experts and convolutions are all sums over interchangeable components, so a permutation leaves the represented function unchanged. (b) Training begins with all weights near zero, so each module may be Taylor expanded about that point. Given smoothness and the condition that … view at source ↗
Figure 2
Figure 2. NQF approximation across initialization, optimizer, and architecture. Two initialization scales are [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Sudden learning in a solvable NQF, the phenomenon the theory of this paper is built to explain. The experi [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The experiment is under the setting of the sample-wise descent in Section [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Curves show the normalized MSE L(t)/L(0); dotted lines are the slopes predicted in Section 6. Left: For θ = 1, the Fourier MLP (blue) descends as the predicted power law t −(2β−1)/(θ+β) (see Appendix C) and is tracked by its NQF approximation (black); a standard tanh M…
Figure 6
Figure 6. Figure 6: Left: Comparison between a two-layer MLP and its NQF approximation (Proposition 1) under a teacher￾student setting. Right: Comparison between a query-key-only attention model and its NQF approximation (Proposi￾tion 3). Models are trained via online SGD (learning rate 0…
Figure 7
Figure 7. Figure 7: Comparison of training dynamics between a two-layer MLP ( [PITH_FULL_IMAGE:figures/full_fig_p050_7.png]
Figure 8
Figure 8. Figure 8: Training dynamics of 4−layer MLPs and 2−layer attention models compared to their NQF approximations under different optimizers and initialization scales. The black line represents the original model. The orange line shows the 2-layer NQF approximations. The purple line…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

92 extracted references · 63 canonical work pages

  1. [80]

    Parameter symmetry potentially unifies deep learning theory, 2025

    Liu Ziyin, Yizhou Xu, Tomaso Poggio, and Isaac Chuang. Parameter symmetry potentially unifies deep learning theory, 2025. 20 Contents 1 Introduction 1 2 Neural Quadratic Forms 3 3 NQF for Different Architectures 4 4 Learning Dynamics of the NQF 7 5 Exactly Solvable Cases of Learning Dynamics 8 6 Sudden Learning and Neural Scaling Laws 10 7 Experiments 11 ...

    work page 2025

  2. [1]

    Sgd learning on neural networks: Leap com- plexity and saddle-to-saddle dynamics

    Emmanuel Abbe, Enric Boix Adser `a, and Theodor Misiakiewicz. Sgd learning on neural networks: Leap com- plexity and saddle-to-saddle dynamics. InProceedings of the Thirty Sixth Conference on Learning Theory, volume 195 ofProceedings of Machine Learning Research, pages 2552–2623. PMLR, 2023

    2023

  3. [2]

    Birkh ¨auser, 2012

    Hisham Abou-Kandil, Gerhard Freiling, Vlad Ionescu, and Gerhard Jank.Matrix Riccati equations in control and systems theory. Birkh ¨auser, 2012

    2012

  4. [3]

    Advani, Andrew M

    Madhu S. Advani, Andrew M. Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks.Neural Networks, 132:428–446, 2020

    2020

  5. [4]

    Intrinsic dimensionality explains the effectiveness of language model fine-tuning

    Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. InProceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 7319–7328, 2021

    2021

  6. [5]

    Implicit regularization in deep matrix factorization

    Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems, volume 32, 2019

    2019

  7. [6]

    Max-margin token selection in attention mechanism.Advances in neural information processing systems, 36:48314–48362, 2023

    Davoud Ataee Tarzanagh, Yingcong Li, Xuechen Zhang, and Samet Oymak. Max-margin token selection in attention mechanism.Advances in neural information processing systems, 36:48314–48362, 2023

    2023

  8. [7]

    Explaining neural scaling laws

    Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):e2311878121, 2024

    2024

Show all 92 references
  1. [8]

    Statistical mechanics of deep learning.Annual Review of Condensed Matter Physics, 11:501–528, 2020

    Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Sohl-Dickstein, and Surya Ganguli. Statistical mechanics of deep learning.Annual Review of Condensed Matter Physics, 11:501–528, 2020

    2020

  2. [9]

    VICReg: Variance-invariance-covariance regularization for self- supervised learning

    Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regularization for self- supervised learning. InInternational Conference on Learning Representations, 2022

    2022

  3. [10]

    Mechanism of feature learning in convolutional neural networks.arXiv preprint arXiv:2309.00570, 2023

    Daniel Beaglehole, Adityanarayanan Radhakrishnan, Parthe Pandit, and Mikhail Belkin. Mechanism of feature learning in convolutional neural networks.arXiv preprint arXiv:2309.00570, 2023

    2023 arXiv

  4. [11]

    Erdogdu, Nuri Mert Vural, and Denny Wu

    G ´erard Ben Arous, Murat A. Erdogdu, Nuri Mert Vural, and Denny Wu. Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws. InAdvances in Neural Information Processing Systems, volume 38, 2025

    2025

  5. [12]

    Incremental learning in diagonal linear networks.Journal of Machine Learning Research, 24(171):1–26, 2023

    Rapha ¨el Berthier. Incremental learning in diagonal linear networks.Journal of Machine Learning Research, 24(171):1–26, 2023

    2023

  6. [13]

    Single-head attention in high dimensions: A theory of generalization, weights spectra, and scaling laws

    Fabrizio Boncoraglio, Vittorio Erba, Emanuele Troiani, Yizhou Xu, Florent Krzakala, and Lenka Zdeborov ´a. Single-head attention in high dimensions: A theory of generalization, weights spectra, and scaling laws. In Proceedings of the 43rd International Conference on Machine Le...

    2026

  7. [14]

    A dynamical model of neural scaling laws

    Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. In Proceedings of the 41st International Conference on Machine Learning, volume 235 ofPMLR, 2024

    2024

  8. [15]

    Spectrum dependent learning curves in kernel re- gression and wide neural networks

    Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel re- gression and wide neural networks. InProceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, pages 102...

    2020

  9. [16]

    Ecological communities with Lotka–Volterra dynamics.Phys

    Guy Bunin. Ecological communities with Lotka–Volterra dynamics.Phys. Rev. E, 95:042414, 2017

    2017

  10. [17]

    Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks.Nature Communications, 12(2914), 2021

    Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks.Nature Communications, 12(2914), 2021. 16

    2021

  11. [18]

    Cand `es, Xiaodong Li, and Mahdi Soltanolkotabi

    Emmanuel J. Cand `es, Xiaodong Li, and Mahdi Soltanolkotabi. Phase retrieval via wirtinger flow: Theory and algorithms.IEEE Transactions on Information Theory, 61(4):1985–2007, 2015

    1985

  12. [19]

    Tight sample complexity of learning one-hidden-layer convolutional neural net- works.Advances in Neural Information Processing Systems, 32, 2019

    Yuan Cao and Quanquan Gu. Tight sample complexity of learning one-hidden-layer convolutional neural net- works.Advances in Neural Information Processing Systems, 32, 2019

    2019

  13. [20]

    Machine learning and the physical sciences.Rev

    Giuseppe Carleo, Ignacio Cirac, Kyle Cranmer, Laurent Daudet, Maria Schuld, Naftali Tishby, Leslie V ogt- Maranto, and Lenka Zdeborov ´a. Machine learning and the physical sciences.Rev. Mod. Phys., 91:045002, 2019

    2019

  14. [21]

    Chaikin and Tom C

    Paul M. Chaikin and Tom C. Lubensky.Principles of Condensed Matter Physics. Cambridge University Press, Cambridge, 1995

    1995

  15. [22]

    On lazy training in differentiable programming

    L ´ena¨ıc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. InAdvances in Neural Information Processing Systems, volume 32, 2019

    2019

  16. [23]

    Scaling laws and spectra of shallow neural networks in the feature learning regime

    Leonardo Defilippis, Yizhou Xu, Julius Girardin, Vittorio Erba, Emanuele Troiani, Lenka Zdeborov ´a, Bruno Loureiro, and Florent Krzakala. Scaling laws and spectra of shallow neural networks in the feature learning regime. InThe Fourteenth International Conference on Learning ...

    2026

  17. [24]

    Gradient descent learns one-hidden- layer cnn: Don’t be afraid of spurious local minima

    Simon Du, Jason Lee, Yuandong Tian, Aarti Singh, and Barnabas Poczos. Gradient descent learns one-hidden- layer cnn: Don’t be afraid of spurious local minima. InInternational Conference on Machine Learning, pages 1339–1348. PMLR, 2018

    2018

  18. [25]

    Cambridge University Press, Cambridge, 2001

    Andreas Engel and Christian Van den Broeck.Statistical Mechanics of Learning. Cambridge University Press, Cambridge, 2001

    2001

  19. [26]

    Bilinear sequence regression: A model for learning from long sequences of high-dimensional tokens.Physical Review X, 15(2):021092, 2025

    Vittorio Erba, Emanuele Troiani, Luca Biggio, Antoine Maillard, and Lenka Zdeborov ´a. Bilinear sequence regression: A model for learning from long sequences of high-dimensional tokens.Physical Review X, 15(2):021092, 2025

    2025

  20. [27]

    (S)GD over diagonal linear networks: Implicit bias, large stepsizes and edge of stability

    Mathieu Even, Scott Pesme, Suriya Gunasekar, and Nicolas Flammarion. (S)GD over diagonal linear networks: Implicit bias, large stepsizes and edge of stability. InAdvances in Neural Information Processing Systems, volume 36, 2023

    2023

  21. [28]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. InInternational Conference on Learning Representations, 2019

    2019

  22. [29]

    A regularity condition of the information matrix of a multilayer perceptron network.Neural Networks, 9(5):871–879, 1996

    Kenji Fukumizu. A regularity condition of the information matrix of a multilayer perceptron network.Neural Networks, 9(5):871–879, 1996

    1996

  23. [30]

    Matrix completion has no spurious local minimum.Advances in neural information processing systems, 29, 2016

    Rong Ge, Jason D Lee, and Tengyu Ma. Matrix completion has no spurious local minimum.Advances in neural information processing systems, 29, 2016

    2016

  24. [31]

    word2vec explained: Deriving mikolov et al.’s negative-sampling word- embedding method, 2014

    Yoav Goldberg and Omer Levy. word2vec explained: Deriving mikolov et al.’s negative-sampling word- embedding method, 2014

    2014

  25. [32]

    Addison-Wesley, Reading, MA, 1992

    Nigel Goldenfeld.Lectures on Phase Transitions and the Renormalization Group. Addison-Wesley, Reading, MA, 1992

    1992

  26. [33]

    Implicit regularization in matrix factorization

    Suriya Gunasekar, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Implicit regularization in matrix factorization. InAdvances in Neural Information Processing Systems, volume 30, pages 6151–6159. Curran Associates, Inc., 2017

    2017

  27. [34]

    Gradient descent happens in a tiny subspace.arXiv preprint arXiv:1812.04754, 2018

    Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace.arXiv preprint arXiv:1812.04754, 2018

    2018 arXiv

  28. [35]

    Cambridge university press, 1998

    Josef Hofbauer and Karl Sigmund.Evolutionary games and population dynamics. Cambridge university press, 1998. 17

    1998

  29. [36]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Milli- can, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Os...

    2022

  30. [37]

    Position: The platonic representation hypoth- esis

    Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic representation hypoth- esis. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 20617–20642. PMLR, 2024

    2024

  31. [38]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Cl ´ement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. InAdvances in Neural Information Processing Systems, volume 31, 2018

    2018

  32. [39]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  33. [40]

    The universal weight subspace hypothesis, 2025

    Prakhar Kaushik, Shravan Chaudhari, Ankit Vaidya, Rama Chellappa, and Alan Yuille. The universal weight subspace hypothesis, 2025

    2025

  34. [41]

    Matrix factorization techniques for recommender systems

    Yehuda Koren, Robert Bell, and Chris V olinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30–37, 2009

    2009

  35. [42]

    Imagenet classification with deep convolutional neural networks.Advances in neural information processing systems, 25, 2012

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks.Advances in neural information processing systems, 25, 2012

    2012

  36. [43]

    Alternating gradient flows: A theory of feature learning in two-layer neural net- works.Advances in Neural Information Processing Systems, 38:4377–4424, 2025

    Daniel Kunin, Giovanni Luca Marchetti, Feng Chen, Dhruva Karkada, James Simon, Michael Deweese, Surya Ganguli, and Nina Miolane. Alternating gradient flows: A theory of feature learning in two-layer neural net- works.Advances in Neural Information Processing Systems, 38:4377–4...

    2025

  37. [44]

    Elsevier, 2013

    Lev Davidovich Landau and Evgenii Mikhailovich Lifshitz.Statistical Physics: Volume 5, volume 5. Elsevier, 2013

    2013

  38. [45]

    Measuring the intrinsic dimension of objective landscapes

    Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. InInternational Conference on Learning Representations, 2018

    2018

  39. [46]

    Towards under- standing grokking: An effective theory of representation learning.Advances in Neural Information Processing Systems, 35:34651–34663, 2022

    Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards under- standing grokking: An effective theory of representation learning.Advances in Neural Information Processing Systems, 35:34651–34663, 2022

    2022

  40. [47]

    Phase retrieval in high dimensions: Statistical and computational phase transitions.Advances in Neural Information Processing Systems, 33:11071– 11082, 2020

    Antoine Maillard, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborov ´a. Phase retrieval in high dimensions: Statistical and computational phase transitions.Advances in Neural Information Processing Systems, 33:11071– 11082, 2020

    2020

  41. [48]

    Bayes-optimal learning of an extensive-width neural network from quadratically many samples.Advances in Neural Information Processing Systems, 37:82085–82132, 2024

    Antoine Maillard, Emanuele Troiani, Simon Martin, Lenka Zdeborov ´a, and Florent Krzakala. Bayes-optimal learning of an extensive-width neural network from quadratically many samples.Advances in Neural Information Processing Systems, 37:82085–82132, 2024

    2024

  42. [49]

    Roberts, and James Sully

    Alexander Maloney, Daniel A. Roberts, and James Sully. A solvable model of neural scaling laws.arXiv preprint arXiv:2210.16859, 2022

    2022 arXiv

  43. [50]

    Attention-based clustering.Advances in Neural Infor- mation Processing Systems, 38:66455–66506, 2025

    Rodrigo Maulen Soto, Pierre Marion, and Claire Boyer. Attention-based clustering.Advances in Neural Infor- mation Processing Systems, 38:66455–66506, 2025

    2025

  44. [51]

    Robert M. May. Will a large complex system be stable?Nature, 238:413–414, 1972

    1972

  45. [52]

    Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit

    Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. InConference on learning theory, pages 2388–2464. PMLR, 2019

    2019

  46. [53]

    A defense of the quadratic model, 2026

    Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade, and Alex Damian. A defense of the quadratic model, 2026. 18

    2026

  47. [54]

    The quantization model of neural scaling.Advances in Neural Information Processing Systems, 36:28699–28722, 2023

    Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling.Advances in Neural Information Processing Systems, 36:28699–28722, 2023

    2023

  48. [55]

    Corrado, and Jeff Dean

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. InAdvances in Neural Information Processing Systems, volume 26, 2013

    2013

  49. [56]

    An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem.Advances in Neural Information Processing Systems, 37:39632–39693, 2024

    Yoonsoo Nam, Nayara Fonseca, Seok H Lee, Chris Mingard, and Ard A Louis. An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem.Advances in Neural Information Processing Systems, 37:39632–39693, 2024

    2024

  50. [57]

    Sigmoid gating is more sample efficient than softmax gating in mixture of experts.Advances in Neural Information Processing Systems, 37:118357–118388, 2024

    Huy Nguyen, Nhat Ho, and Alessandro Rinaldo. Sigmoid gating is more sample efficient than softmax gating in mixture of experts.Advances in Neural Information Processing Systems, 37:118357–118388, 2024

    2024

  51. [58]

    Dissecting query-key interaction in vision transformers

    Xu Pan, Aaron Philip, Ziqian Xie, and Odelia Schwartz. Dissecting query-key interaction in vision transformers. InAdvances in Neural Information Processing Systems, volume 37, 2024

    2024

  52. [59]

    Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity.Advances in Neural Information Processing Systems, 34:29218–29230, 2021

    Scott Pesme, Loucas Pillaud-Vivien, and Nicolas Flammarion. Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity.Advances in Neural Information Processing Systems, 34:29218–29230, 2021

    2021

  53. [60]

    Pope.Turbulent Flows

    Stephen B. Pope.Turbulent Flows. Cambridge University Press, Cambridge, 2000

    2000

  54. [61]

    Mechanism for feature learning in neural networks and backpropagation-free machine learning models.Science, 383(6690):1461–1467, 2024

    Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Mechanism for feature learning in neural networks and backpropagation-free machine learning models.Science, 383(6690):1461–1467, 2024

    2024

  55. [62]

    Saxe, James L

    Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. InInternational Conference on Learning Representations, 2014

    2014

  56. [63]

    H. S. Seung, H. Sompolinsky, and N. Tishby. Statistical mechanics of learning from examples.Phys. Rev. A, 45:6056–6091, 1992

    1992

  57. [64]

    Le, Geoffrey E

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V . Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InInternational Con- ference on Learning Representations, 2017

    2017

  58. [65]

    Maximum-margin matrix factorization.Advances in neural information processing systems, 17, 2004

    Nathan Srebro, Jason Rennie, and Tommi Jaakkola. Maximum-margin matrix factorization.Advances in neural information processing systems, 17, 2004

    2004

  59. [66]

    Dominik St ¨oger and Mahdi Soltanolkotabi. Small random initialization is akin to spectral learning: Optimiza- tion and generalization guarantees for overparameterized low-rank matrix reconstruction.Advances in Neural Information Processing Systems, 34:23831–23843, 2021

    2021

  60. [67]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  61. [68]

    Self-supervised learning with data augmentations provably isolates content from style

    Julius V on K ¨ugelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Sch ¨olkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:164...

    2021

  62. [69]

    A universal compression theory for lottery ticket hypothesis and neural scaling laws

    Hong-Yi Wang, Di Luo, Tomaso Poggio, Isaac L Chuang, and Liu Ziyin. A universal compression theory for lottery ticket hypothesis and neural scaling laws. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  63. [70]

    Lee, and Denny Wu

    Zixuan Wang, Eshaan Nichani, Alberto Bietti, Alex Damian, Daniel Hsu, Jason D. Lee, and Denny Wu. Learning compositional functions with transformers from easy-to-hard data. InProceedings of Thirty Eighth Conference on Learning Theory, volume 291 ofProceedings of Machine Learni...

    2025

  64. [71]

    Chuang, and Max Tegmark

    Tailin Wu, Ian Fischer, Isaac L. Chuang, and Max Tegmark. Learnability for the information bottleneck.Entropy, 21(10):924, 2019. 19

    2019

  65. [72]

    Fundamental limits of matrix sensing: Exact asymptotics, universality, and applications

    Yizhou Xu, Antoine Maillard, Lenka Zdeborov ´a, and Florent Krzakala. Fundamental limits of matrix sensing: Exact asymptotics, universality, and applications. InProceedings of Thirty Eighth Conference on Learning Theory, volume 291 ofProceedings of Machine Learning Research, p...

    2025

  66. [73]

    Three mechanisms of feature learning in a linear network

    Yizhou Xu and Liu Ziyin. Three mechanisms of feature learning in a linear network. InInternational Conference on Learning Representations, 2025

    2025

  67. [74]

    Statistical physics of inference: Thresholds and algorithms.Advances in Physics, 65(5):453–552, 2016

    Lenka Zdeborov ´a and Florent Krzakala. Statistical physics of inference: Thresholds and algorithms.Advances in Physics, 65(5):453–552, 2016

    2016

  68. [75]

    Yedi Zhang, Andrew Saxe, and Peter E. Latham. Saddle-to-saddle dynamics explains a simplicity bias across neural network architectures. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  69. [76]

    Quadratic models for under- standing catapult dynamics of neural networks

    Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan, and Mikhail Belkin. Quadratic models for under- standing catapult dynamics of neural networks. InInternational Conference on Learning Representations, 2024

    2024

  70. [77]

    Symmetry induces structure and constraint of learning

    Liu Ziyin. Symmetry induces structure and constraint of learning. InForty-first International Conference on Machine Learning, 2024

    2024

  71. [78]

    What shapes the loss landscape of self-supervised learning? InInternational Conference on Learning Representations, 2023

    Liu Ziyin, Ekdeep Singh Lubana, Masahito Ueda, and Hidenori Tanaka. What shapes the loss landscape of self-supervised learning? InInternational Conference on Learning Representations, 2023

    2023

  72. [79]

    Parameter symmetry and noise equilibrium of stochastic gradient descent

    Liu Ziyin, Mingze Wang, Hongchao Li, and Lei Wu. Parameter symmetry and noise equilibrium of stochastic gradient descent. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  73. [81]

    Smoothness: Since the activation functionϕ∈C 3,f x is at least three times continuously differentiable

  74. [82]

    Permutation Symmetry: Letσbe an arbitrary permutation of the indices{1, . . . , d}. fx(wσ(1), . . . , wσ(d))= d ∑ i=1 vσ(i)ϕ(u⊺ σ(i)x)= d ∑ i=1 viϕ(u⊺ i x)=f x(w1, . . . , wd).(58)

  75. [83]

    Finally let us computeA(x)= 1 2 Hii, whereH ii =∇ 2 wi fx∣wi=0 is the Hessian matrix with respect to thei-th neuron’s parametersw i =[u ⊺ i , vi]⊺

    ZGZ: The gradient with respect to the combined neuron parameterw i consists of ∇ui fx =v iϕ′(u⊺ i x)x,∇ vi fx =ϕ(u ⊺ i x).(59) Settingu i =w i =0gives ∇ui fx∣ui=0,vi=0 =0⋅ϕ ′(0)x=0,∇ vi fx∣ui=0,vi=0 =ϕ(0)=0.(60) Therefore, the two-layer MLP satisfies all assumptions in Theorem...

  76. [84]

    Thus,f x isC ∞

    Smoothness: Thesoftmaxfunction is infinitely differentiable everywhere in its domain. Thus,f x isC ∞

  77. [85]

    Permutation Symmetry: Letσbe an arbitrary permutation of the head indices{1, . . . , d}. fx(wσ(1), . . . , wσ(d))= d ∑ i=1 Headσ(i)(x, X)= d ∑ i=1 Headi(x, X)=f x(w1, . . . , wd)(66)

  78. [86]

    ZGZ: LetA i ∈R N denote the softmax attention weights. Evaluating the gradients atw i =0impliesv i =0, W V i =0,W Q i =0, andW K i =0: ∇vi fx∣wi=0 =W V i XA i∣W V i =0 =0(67) ∇W V i fx∣wi=0 =v iA⊺ i X⊺∣vi=0 =0(68) For the query and key matrices, the gradients involve the deriv...

  79. [87]

    ˜fx =f x for all inputsx∈Xat initialization; 2.˜µ(l) t =V lV⊺ l µ(l) t and ˜M (l) t = dl d′ l VlV⊺ l M (l) t VlV⊺ l for all layersland all training stepst≥0, whereV l ∈R pl×kl is the orthogonal basis ofV (l)

  80. [88]

    The output and learning dynamics of the smaller model are identical to the original model: ˜fx(t)=f x(t)for all t≥0. B.5 Compression Error Corollary 3.For any original NQF withdneurons and input dimensionp, and any target widthd ′ ≤p, there exists a compressed NQF withd′ neuro...

  81. [89]

    The NTK remains invariant while the loss decreases byO(1)

    Lazy Training (Constant NTK): Ifα A >0andα B >1, thenlim d→∞ ˙Θ=0. The NTK remains invariant while the loss decreases byO(1). The NQF effectively behaves as a linear model

  82. [90]

    The NTK changes on the same timescale as the loss, allowing the model to learn data-dependent representations

    Feature Learning (Evolving NTK): Ifα A =0or 1 2 ≤α B ≤1, then ˙Θ=O(1). The NTK changes on the same timescale as the loss, allowing the model to learn data-dependent representations. Proof.By the chain rule under gradient flow ˙W=−∇ W L, the dynamics of the function output are ...

  83. [91]

    Sinceα g =1/2andα B ≥1/2, we haveu=g+2Bµ=O(d −1/2)

    Term 1:d⋅u ⊺B˙µ≈−d 2u⊺Bu. Sinceα g =1/2andα B ≥1/2, we haveu=g+2Bµ=O(d −1/2). Thus, this term scales asd 2(d−1/2)2d−αB =O(d 1−αB)

  84. [92]

    Momentum

    Term 2:−d⋅u ⊺Au. This term scales asd(d −1/2)2d−αA =O(d −αA). Other terms, such asTr(A 2 ˙M), scale asd −3αA or lower and are dominated. Therefore, the overall drift in NTK is bounded by ˙Θ=O(d 1−αB +d−αA). For the NTK to remain constant (while the loss decreases byO(1)), the ...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.