pith. sign in

arxiv: 2606.28242 · v1 · pith:CQ4NHSASnew · submitted 2026-06-26 · 💻 cs.LG · cond-mat.dis-nn· cs.AI· stat.ML

How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks

Julius Girardin , Emanuele Troiani , Yizhou Xu , Vittorio Erba , Florent Krzakala , Lenka Zdeborov\'a This is my paper

Pith reviewed 2026-06-29 04:20 UTC · model grok-4.3

classification 💻 cs.LG cond-mat.dis-nncs.AIstat.ML
keywords generalization scaling lawsquadratic neural networksphase diagramspectral structureinterpolation thresholdfinite widthfinite sample
0
0 comments X

The pith

Generalization error follows data-dependent power laws controlled by target spectral structure in quadratic two-layer networks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper studies l2-regularized minimization in a quadratic two-layer network with finite width and samples. It derives an exact closed-form for the test error that depends on width, sample count, and regularization strength. This formula produces a phase diagram of scaling regimes whose power-law exponents are set by the eigenvalues of the target. The work also locates the transitions between regimes, including the interpolation point. The result supplies concrete predictions for how width and data jointly affect performance when the model can learn features.

Core claim

The generalization error of the quadratic two-layer network admits an exact closed-form expression in terms of the number of samples, model width, and regularization. This expression exhibits distinct scaling regimes whose exponents are governed by the spectral structure of the target function, together with explicit transitions between regimes that include the onset of interpolation.

What carries the argument

The exact closed-form expression for the generalization error of the quadratic two-layer network, which encodes the dependence on width, sample size, regularization, and the spectral properties of the target.

If this is right

  • Generalization error obeys power laws in width whose exponents depend on the target's spectrum.
  • The interpolation threshold marks a transition where further width increase changes from improving to possibly harming generalization.
  • Regularization strength shifts the locations of all regime boundaries.
  • Different spectral decays of the target produce different families of scaling exponents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same spectral mechanism could control scaling in deeper networks whenever a local quadratic approximation remains accurate.
  • Estimates of the target spectrum extracted from data might be used to select near-optimal widths before training.
  • The derived phase diagram supplies a candidate explanation for width-dependent double-descent curves observed in other models.

Load-bearing premise

The quadratic two-layer architecture permits an exact closed-form expression for the generalization error that remains valid across the full range of widths and sample sizes considered.

What would settle it

Numerical evaluation of test error on quadratic networks trained on data whose covariance spectrum is known in advance, checking whether the measured scaling exponents versus width and sample size match the predicted values in each regime.

Figures

Figures reproduced from arXiv: 2606.28242 by Emanuele Troiani, Florent Krzakala, Julius Girardin, Lenka Zdeborov\'a, Vittorio Erba, Yizhou Xu.

Figure 1
Figure 1. Figure 1: Excess test error vs width p (left) and number of samples n (right), for d = 400, γ = 1.25 and λ˜ = p n/100d 2. Lines are theoretical predictions from Result 1, dots are numerical simulations by LBFGS on (1), see Appendix C. As a function of the width, we highlight the low-p data-dependent decay of the error with exponent 1 − 2γ (dashed black). As a function of the number of samples, we highlight that netw… view at source ↗
Figure 2
Figure 2. Figure 2: Scaling laws as a function of the number of training samples [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (Left) Test error scaling behavior in the over-regularized regime for different values of [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Phase diagram of the noiseless estimation task. The dashed line and the blue lines represent the [PITH_FULL_IMAGE:figures/full_fig_p028_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Test error scaling in the over-regularized regime for multiple data exponents [PITH_FULL_IMAGE:figures/full_fig_p032_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: (Left) Test error scaling behavior in the under-regularized noiseless regime for different values of [PITH_FULL_IMAGE:figures/full_fig_p033_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Test error (left) and training error (right) as functions of the network width for different regular [PITH_FULL_IMAGE:figures/full_fig_p033_7.png] view at source ↗
read the original abstract

Understanding how performance scales jointly with model size and data is a central problem in modern machine learning. Existing theoretical works on scaling laws typically describe generalization as a function of data or compute, often in fixed-feature or infinite-width regimes and for online SGD. Here, we instead study how generalization scales with the number of trainable parameters and the number of samples in a feature-learning model. We analyze $\ell_2$-regularized empirical test error minimization in a quadratic two-layer network in a finite-sample setting with structured data. This setting allows for an explicit characterization of the generalization error as a function of the number of samples, model width, and regularization. Our results reveal a phase diagram with distinct scaling regimes as the number of parameters varies. In particular, the generalization error follows data-dependent power laws controlled by the spectral structure of the target. We further characterize the transitions between regimes, including the onset of interpolation, and their impact on generalization.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper analyzes ℓ₂-regularized empirical risk minimization in a quadratic two-layer network with structured data in the finite-sample regime. It derives an explicit characterization of the generalization error in terms of the number of samples n, model width p, and regularization strength, which yields a phase diagram of distinct scaling regimes. The generalization error is shown to obey data-dependent power laws governed by the spectral structure of the target function, with explicit transitions between regimes including the onset of interpolation.

Significance. If the explicit closed-form characterization holds without hidden limits, the work provides a rare exact, finite-width and finite-sample handle on joint scaling of width and data in a feature-learning model. This is a strength relative to typical infinite-width or fixed-feature analyses, as it directly ties power-law exponents and interpolation transitions to the data covariance spectrum and could guide empirical scaling predictions.

major comments (1)
  1. [Abstract and main theoretical derivation] The central claim of an explicit closed-form for the ℓ₂-regularized test error valid for arbitrary finite p and n (stated in the abstract and used to define the phase diagram) is load-bearing. The derivation (via quadratic loss and eigen-decomposition of the data covariance) must be checked to ensure it does not implicitly require p → ∞ or a p-independent spectral decay; otherwise the reported data-dependent power laws and regime transitions become regime-specific rather than universal.
minor comments (2)
  1. [Introduction] Notation for the quadratic network and the precise definition of the target spectrum should be introduced earlier to improve readability of the phase-diagram claims.
  2. [Figures] Figure captions for the phase diagrams could more explicitly link plotted curves to the derived closed-form expression.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their careful review and for highlighting the importance of verifying the scope of our closed-form result. We respond to the major comment below.

read point-by-point responses

  1. Referee: [Abstract and main theoretical derivation] The central claim of an explicit closed-form for the ℓ₂-regularized test error valid for arbitrary finite p and n (stated in the abstract and used to define the phase diagram) is load-bearing. The derivation (via quadratic loss and eigen-decomposition of the data covariance) must be checked to ensure it does not implicitly require p → ∞ or a p-independent spectral decay; otherwise the reported data-dependent power laws and regime transitions become regime-specific rather than universal.

    Authors: The derivation begins from the quadratic loss and solves the ℓ₂-regularized ERM problem in closed form for any finite p and n. We project onto the eigenbasis of the data covariance matrix Σ, which is fixed by the input distribution and therefore independent of model width p. The resulting expression for the test error is obtained by direct algebraic computation of the bias and variance contributions; no limit p → ∞ is taken, nor is any p-dependent approximation introduced in the spectral decomposition. The formula holds for an arbitrary spectrum of the target function. Power-law decay is invoked only when specializing the exact expression to obtain explicit scaling exponents and to delineate the phase diagram; the transitions themselves (including the interpolation threshold) follow from the finite-p,n formula without further assumptions. We therefore maintain that the abstract statement is accurate as written. revision: no

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper derives an explicit closed-form characterization of the ℓ₂-regularized generalization error for the quadratic two-layer model as a function of n, p, and regularization, directly from the loss structure and data covariance eigen-decomposition. This yields data-dependent power laws and phase transitions without any quoted reduction to fitted parameters, self-definitional loops, or load-bearing self-citations. The abstract and reader's assessment confirm the result is mathematically self-contained rather than constructed by renaming inputs as outputs.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The analysis rests on the quadratic architecture permitting exact solvability and on standard linear-algebra facts about the spectrum of the data matrix; no new entities are introduced.

axioms (2)
  • domain assumption The network is exactly quadratic, allowing the loss to be written in closed form in terms of the Gram matrix and target projections.
    Invoked to obtain the explicit generalization error formula stated in the abstract.
  • standard math Standard matrix concentration or eigenvalue bounds from random matrix theory apply to the finite-sample covariance.
    Required for locating the interpolation threshold as a function of width and samples.

pith-pipeline@v0.9.1-grok · 5718 in / 1303 out tokens · 37782 ms · 2026-06-29T04:20:28.327889+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 1 canonical work pages

  1. [1]

    Statistical mechanics of complex neural systems and high dimensional data.Journal of Statistical Mechanics: Theory and Experiment, 2013(03):P03014,

    Madhu Advani, Subhaneil Lahiri, and Surya Ganguli. Statistical mechanics of complex neural systems and high dimensional data.Journal of Statistical Mechanics: Theory and Experiment, 2013(03):P03014,

    2013

  2. [2]

    Scaling and renormalization in high- dimensional regression.Journal of Statistical Mechanics: Theory and Experiment, 2026(4):043404,

    Alexander Atanasov, Jacob A Zavatone-Veth, and Cengiz Pehlevan. Scaling and renormalization in high- dimensional regression.Journal of Statistical Mechanics: Theory and Experiment, 2026(4):043404,

    2026

  3. [3]

    The lasso risk for gaussian matrices.IEEE Transactions on Infor- mation Theory, 58(4):1997–2017,

    Mohsen Bayati and Andrea Montanari. The lasso risk for gaussian matrices.IEEE Transactions on Infor- mation Theory, 58(4):1997–2017,

    1997

  4. [4]

    Fabrizio Boncoraglio, Vittorio Erba, Emanuele Troiani, Yizhou Xu, Florent Krzakala, and Lenka Zdeborová

    ISSN 2049-8764, 2049-8772. Fabrizio Boncoraglio, Vittorio Erba, Emanuele Troiani, Yizhou Xu, Florent Krzakala, and Lenka Zdeborová. Single-head attention in high dimensions: A theory of generalization, weights spectra, and scaling laws. InICML,

    2049

  5. [5]

    Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan

    arxiv:2509.24914. Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. InInternational Conference on Machine Learning, pages 1024–1034. PMLR,

    Pith/arXiv arXiv

  6. [6]

    A dynamical model of neural scaling laws

    Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. In Proceedings of the 41st International Conference on Machine Learning, pages 4345–4382, 2024a. Blake Bordelon, Lorenzo Noci, Mufan Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit. In1...

    2024

  7. [7]

    Deriving neural scaling laws from the statistics of natural language.arXiv preprint arXiv:2602.07488,

    Francesco Cagnetta, Allan Raventós, Surya Ganguli, and Matthieu Wyart. Deriving neural scaling laws from the statistics of natural language.arXiv preprint arXiv:2602.07488,

    arXiv

  8. [8]

    Resnets of all shapes and sizes: Convergence of training dynamics in the large-scale limit.arXiv preprint arXiv:2603.18168,

    Louis-Pierre Chaintron, Lénaïc Chizat, and Javier Maas. Resnets of all shapes and sizes: Convergence of training dynamics in the large-scale limit.arXiv preprint arXiv:2603.18168,

    arXiv

  9. [9]

    Optimal scaling laws in learning hierarchical multi-index models.arXiv preprint arXiv:2602.05846, 2026a

    Leonardo Defilippis, Florent Krzakala, Bruno Loureiro, and Antoine Maillard. Optimal scaling laws in learning hierarchical multi-index models.arXiv preprint arXiv:2602.05846, 2026a. Leonardo Defilippis, Yizhou Xu, Julius Girardin, Emanuele Troiani, Vittorio Erba, Lenka Zdeborová, Bruno Loureiro, and Florent Krzakala. Scaling laws and spectra of shallow ne...

    arXiv

  10. [10]

    The approximation of one matrix by another of lower rank

    doi: 10.1007/BF02288367. Vittorio Erba, Emanuele Troiani, Lenka Zdeborová, and Florent Krzakala. The nuclear route: Sharp asymp- totics of erm in overparameterized quadratic networks.Advances in Neural Information Processing Sys- tems, 38:88862–88901,

    work page doi:10.1007/bf02288367

  11. [11]

    Stationary points of shallow neural networks with quadratic activation function.arXiv preprint arXiv:1912.01599,

    David Gamarnik, Eren C Kızıldağ, and Ilias Zadik. Stationary points of shallow neural networks with quadratic activation function.arXiv preprint arXiv:1912.01599,

    arXiv 1912

  12. [12]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Pith/arXiv arXiv 2001

  13. [13]

    A solvable model of neural scaling laws.arXiv preprint arXiv:2210.16859,

    Alexander Maloney, Daniel A Roberts, and James Sully. A solvable model of neural scaling laws.arXiv preprint arXiv:2210.16859,

    arXiv

  14. [14]

    High-dimensional analysis of gradient flow for extensive-width quadratic neural networks.arXiv preprint arXiv:2601.10483,

    Simon Martin, Giulio Biroli, and Francis Bach. High-dimensional analysis of gradient flow for extensive-width quadratic neural networks.arXiv preprint arXiv:2601.10483,

    arXiv

  15. [15]

    A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and GCV estimator.arXiv preprint arXiv:2403.08938,

    Theodor Misiakiewicz and Basil Saeed. A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and GCV estimator.arXiv preprint arXiv:2403.08938,

    arXiv

  16. [16]

    Asymptotic learning curves of kernel methods: empirical data versus teacher–student paradigm.Journal of Statistical Mechanics: Theory and Experiment, 2020 (12):124001,

    Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods: empirical data versus teacher–student paradigm.Journal of Statistical Mechanics: Theory and Experiment, 2020 (12):124001,

    2020

  17. [17]

    Asymptotics of non-convex generalized linear models in high-dimensions: A proof of the replica formula.arXiv preprint arXiv:2502.20003,

    Matteo Vilucchio, Yatin Dandi, Matéo Pirio Rossignol, Cedric Gerbelot, and Florent Krzakala. Asymptotics of non-convex generalized linear models in high-dimensions: A proof of the replica formula.arXiv preprint arXiv:2502.20003,

    arXiv

  18. [18]

    (2026) to power-law targets and width-constrained students

    13 A Derivation of Result 1: analytical test error characterization We derive the characterization in Result 1 by adapting the derivation of Erba et al. (2026) to power-law targets and width-constrained students. For the sake of the derivation, as discussed in the main text, we first assume thatn= Θ(d 2),p= Θ(d), ˜λ= Θ(1), thatS ⋆has a limiting spectral d...

    2026

  19. [19]

    •AMP and its analysis:The asymptotic behavior of ERM in Gaussian linear models has been stud- ied extensively (Bayati et al., 2010; Montanari et al., 2012; Rangan et al., 2016; Loureiro et al., 2021; Vilucchio et al., 2025), in particular through Approximate Message Passing (AMP). This framework allows to write an algorithm, the mentioned AMP, whose fixed...

    2010

  20. [20]

    Algebraic manipulations of (21) under the change of variableδ=√ˆq/ˆmandϵ= 2/ˆmleads to (9). We thus see that the rank constraint on the original problem induces an additional spectral cut in (30) (the sum going up top<d, and not up todas it would in the non-rank-constrained case). This concludes the raw derivation of (9). The expression of the training lo...

    2026

  21. [21]

    When fixingn,d and increasingp, the error decreases asp2γ−1and becomes a constant afterp= n d

    For the under regularized case (Figure 4(a)), when fixingp,dand increasingn, the error decreases untiln=pd, and then becomes a constant. When fixingn,d and increasingp, the error decreases asp2γ−1and becomes a constant afterp= n d. Thus large width and lower regularization is always preferred. B.3 Derivation of the excess test error for pruning Now we con...

    2026

  22. [22]

    explicitly in terms of eigenvalues. In practice, we computeE ˆS,S⋆ by takingS ⋆deterministic with power-law diagonal (as defined in the main text), by samplingn empirical GOE noise matricesZ∼GOE(d), computing for each of the GOE matrices( ˆS,νi,˜νi), and taking the empirical means of the resulting(m,q,Σ)variables. In practice, we used nempirical = 16in ou...

    2019