Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation

Francesco Camilli; Jean Barbier; Mauro Pastore; Minh-Toan Nguyen; Rudy Skerk

REVIEW 2 major objections 6 minor 1 cited by

Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation

T0 review · 2 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For a two-layer Bayesian network trained with $n=\Theta(d^2)$ samples, the paper derives an effective theory of the Bayes-optimal generalisation error and predicts a discontinuous transition at a computable sampling rate $\alpha_{\rm sp}$…

desk verdict A solid heuristic extension of the quadratic-activation theory to generic activations and priors, but the specialisation transition rests on an unvalidated entropic ansatz and needs direct testing. read the letter →

arxiv 2501.18530 v3 pith:UM24OJY7 submitted 2025-01-30 stat.ML cond-mat.dis-nncond-mat.stat-mechcs.ITcs.LGmath.IT

Jean Barbier , Francesco Camilli , Minh-Toan Nguyen , Mauro Pastore , Rudy Skerk This is my paper · ORCID

classification stat.MLcond-mat.dis-nncond-mat.stat-mechcs.ITcs.LGmath.IT MSC 82B4460B2068T0762F15

keywords two-layerneuralnetworksteacher-studentmodelBayes-optimalgeneralisationerrorinterpolationregimereplicamethodGaussianequivalencephasetransitionHermiteexpansion

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper considers a two-layer teacher–student network in which the input dimension $d$ and hidden width $k$ are both large with $k/d\to\gamma$, and the number of training samples $n$ is of order $d^2$, i.e. the regime where the number of trainable parameters matches the dataset size. Its central claim is that the Bayes-optimal generalisation error has a sharp, discontinuous phase transition at a computable sampling rate $\alpha_{\rm sp}(\gamma)$: below it the student learns only certain nonlinear combinations of the teacher weights and the error is universal in the teacher weight prior; above it the student's hidden weights align with the teacher's and the error drops quickly but becomes prior-dependent. For activations with Hermite coefficients beyond the quadratic one, the transition occurs even for Gaussian weights; for a purely quadratic activation with Gaussian weights it does not. The same effective theory predicts that the superior specialisation solution is often algorithmically hard to find, so near interpolation the best performance reachable by practical algorithms can lie strictly above the Bayes-optimal value.

What carries the argument

The engine is a replica calculation of the quenched free entropy. Its core object is the covariance $K^{ab}=\sum_{\ell\ge1}\mu_\ell^2 Q^{ab}_\ell/\ell!$ of the replicated post-activations $(\lambda_a)_{a\ge0}$, which under the replica-symmetric ansatz concentrates to $q_K^*+(r_K-q_K^*)\delta_{ab}$. The calculation rests on two substitutions: joint Gaussianity of the post-activations, and Eq. (15), which replaces Hadamard powers $(\Omega^{ab}_{ij})^\ell$ of the overlap matrix by the diagonal $\delta_{ij}(Q^{ab}_W)^\ell$ for $\ell\ge3$. The two phases correspond to two ansatze for the law of the order-2 tensor $S_2^a$: a rotationally invariant one (universal) and a Gaussian one aligned with $Q_W$ (specialisation). From these follow the free entropies $f_{\rm uni}$ and $f_{\rm sp}$, and transition criterion (7) compares the two scalars to locate $\alpha_{\rm sp}$.

What would settle it

Compute the fourth-order cumulant $\kappa_4 = \mathbb{E}[\lambda_0\lambda_1^3] - 3K^{01}K^{11}$ of posterior-sampled post-activations on a fresh test input in the universal phase at, say, $d=1000$; the Gaussian ansatz predicts $\kappa_4\to0$, so a clearly nonzero value would falsify Result 3.2 and criterion (7) with it.

Watch

Extended reading notes

Core claim

On the paper's own terms: in the limit $d,k,n\to\infty$ with $k/d\to\gamma$ and $n/d^2\to\alpha$, the Bayes-optimal generalisation error of a fully trained two-layer student with a centred activation is given by Result 3.2. The single input needed is the extremal overlap $q_K$: for $\alpha<\alpha_{\rm sp}(\gamma)$ it comes from the universal free entropy $f_{\rm uni}$, and for $\alpha>\alpha_{\rm sp}(\gamma)$ from the specialisation free entropy $f_{\rm sp}$; the transition point $\alpha_{\rm sp}(\gamma)$ is the first $\alpha$ for which $f_{\rm sp}\ge f_{\rm uni}$, Eq. (7). In the universal phase the overlap $q_W$ between student and teacher inner weights is zero, so the student recovers only the combinations $S_1^0=k^{-1/2}v^{0\top}W^0$ and $S_2^0=k^{-1/2}W^{0\top}{\rm diag}(v^0)W^0$, and the generalisation error is independent of the weight law. In the specialisation phase $q_W$ is positive, the student synchronises with the teacher, and the error decreases faster with explicit prior dependence, including a contribution $g(1)-g(q_W)$ from Hermite degrees $\ell\ge3$. The paper validates both branches numerically and reports that the specialisation branch is metastable or exponentially slow to reach for standard algorithms in many cases.

Load-bearing premise

The load-bearing premise is that the replicated post-activations $(\lambda_a)_{a\ge0}$ are jointly Gaussian, an ansatz the paper states it cannot prove; if that fails for some activation or weight prior, the free entropies, the transition point $\alpha_{\rm sp}$, and the generalisation-error formula (Result 3.2) all collapse with it.

Editorial extensions

If this is right

For any activation with a nonzero Hermite coefficient beyond degree two, the paper predicts a threshold $\alpha_{\rm sp}(\gamma)$ where the Bayes-optimal generalisation error drops discontinuously; the value is computable from Eq. (7) alone.
Below the transition the generalisation error is universal: only the mean and variance of the teacher's inner-weight prior matter, so a student with binary or Gaussian prior performs identically.
Above the transition the error is prior-dependent and decays faster; for Rademacher weights the overlap $q_W$ saturates exponentially fast in $\alpha$, while for Gaussian weights the saturation is algebraic.
The polynomial-time approximate message-passing predictor adapted in the paper tracks the universal branch for all $\alpha$, both before and after the transition, and therefore provides a tractable learner that may be strictly suboptimal.
For odd activations with $\mu_2=0$ the error is flat in the universal phase and drops only at the transition, because the order-2 tensor is absent and higher-order tensors are learned only through specialisation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the central claim is right, adding a small third-degree Hermite term to a quadratic activation should create an $\alpha_{\rm sp}$ for Gaussian weights, with the threshold decreasing as the third coefficient grows; this can be checked with the paper's own saddle-point equations.
If the central claim is right, the same universal-to-specialisation transition should appear in deeper or structured architectures whenever the sample count matches the parameter count, with layer-wise overlaps replacing the single matrix $Q_W$.
If the central claim is right, any estimator that is effectively rotationally invariant on the order-2 tensor is confined to the universal branch, which would explain a statistical-to-computational gap for certain teacher targets near interpolation.
If the central claim is right, the sharp error drop at $\alpha_{\rm sp}$ is a quantitative candidate mechanism for grokking-like sudden generalisation improvements in networks trained near interpolation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know. This paper generalises the Maillard et al. theory to arbitrary Hermite-expandable activations and generic iid weight priors in the n~d^2 regime, and it predicts a specialisation phase for Gaussian inner weights when the activation has Hermite degree above two. That is genuinely new. The second thing is less comforting: the entire specialisation branch, including the transition location alpha_sp, depends on a Gaussian ansatz on the conditional law of the hidden-layer tensor S2 given the overlap Q_W (ansatz (22)). That ansatz is distinct from the post-activation Gaussian ansatz and from the Hadamard-power approximation (15), and the paper does not test it directly. If it fails, the determinant in Eq. (94), the saddle-point system, and hence alpha_sp all shift.

What the paper does well: the replica calculation is careful and internally consistent; the authors are transparent about what is ansatz and what is computed. The universal phase is on firmer ground: it follows from Gaussian universality and matches extensive MCMC and GAMP-RIE numerics. The Hadamard-power approximation (15) is checked directly in Fig. 3, which is good. The algorithmic side—extending GAMP-RIE to generic activations and showing it tracks the universal branch—is a useful contribution. The treatment of the mutual information and the link to matrix denoising is also coherent.

Where it is soft. The stress-test concern holds up. Ansatz (22) is chosen as the simplest law matching the asymptotic second-moment property of S2; it is not derived and no experiment probes the fourth-order cumulants of the conditional law. The a-posteriori match of generalisation curves is a weak test in the narrow specialisation window, and the paper itself reports a ~1% miss for Gaussian readouts (App. I) and small deviations near transitions. The post-activation Gaussian ansatz is also unproven, but that is a known, widely used heuristic in this literature; the specialisation ansatz is the one that needs stronger evidence, because the main new quantitative claim—the existence and location of the transition—sits on it.

Bottom line: this is a plausible and potentially important paper, more heuristic than the authors' confidence suggests. Send it to a serious referee; the referee should ask for either a derivation of (22) from a more controlled approximation or a direct MCMC test of the conditional law of S2, plus a commit-hash-pinned code release. With that, a conditional accept.

Referee Report

2 major / 6 minor

Summary. The paper studies Bayes-optimal learning in a teacher-student two-layer network with i.i.d. Gaussian inputs, extensive width k=Theta(d), and n=Theta(d^2) samples. The authors use the replica method under a Gaussian ansatz for the replicated post-activations and a novel ansatz for the conditional law of the second-order tensor S2 to derive free-entropy expressions for two candidate solutions: a 'universal' phase with trivial weight overlap and a 'specialisation' phase in which the hidden weights align with the teacher. They predict a first-order phase transition at alpha_sp(gamma) via free-entropy comparison, provide saddle-point equations for the overlaps, and derive the limiting Bayes-optimal generalisation error. Numerical experiments with MCMC, HMC, and an extended GAMP-RIE are compared with the theory for binary and Gaussian weights, polynomial and ReLU/ELU activations, and the algorithmic hardness of reaching the specialisation solution is investigated.

Significance. The paper is a serious and detailed replica-method analysis. Its strongest contribution is a coherent effective theory for the Bayes-optimal generalisation error of two-layer networks in the quadratic-sampling regime, extending previous work that was restricted to quadratic activation and Gaussian weights. The derivation is self-contained, the saddle-point systems (Suni) and (Ssp) are explicit and reproducible, and the numerical experiments cover several activations, weight priors, and sampling rates, including a publicly available code. The theory makes falsifiable quantitative predictions, in particular the transition criterion (7) and the universal/specialisation error curves, and the paper is honest about the heuristic status of its key ansaetze. The main concern is that the entropic ansatz (22) for the specialisation phase is a second, untested distributional assumption on which the predicted transition and errors rest; the a-posteriori match of generalisation curves is a necessary but not sufficient check.

major comments (2)

[Section 5, Eq. (22)] The specialisation phase free entropy f_sp relies on the entropic ansatz (22), which postulates that, conditionally on Q_W, the off-diagonal entries of the tensors S^a_2 are zero-mean jointly Gaussian with covariance (Q_W^{circ 2}) and that diagonal entries concentrate. This is a separate distributional assumption from the Gaussian ansatz on the post-activations lambda_a and from the Hadamard-power approximation (15). It enters f_sp through the Gaussian integral leading to the determinant term in Eq. (94), and the saddle-point system (Ssp), hence alpha_sp in Eq. (7) and the overlaps used in Result 3.2, are direct consequences of this functional form. The paper validates Eq. (16) in Fig. 3 and checks the generalisation curves, but no experiment tests the conditional law of S_2 itself. The authors report small but non-negligible deviations near transitions (Sec. 6) and a ~1% miss of the predicted specialisation overlap for Gaussian read-outs (App. I). I request a direct test: from HMC samples, compute the empirical conditional distribution of the off-diagonal S^a_2 entries given the empirical Q_W, compare its covariance and fourth-order cumulants with the Gaussian (22), and verify the determinant expression in Eq. (94) by Monte Carlo evaluation of both sides. This is currently the main unverified load-bearing input to the predicted transition.
[Eq. (15), Section 5] The simplification (Omega^ab)^ell approximately delta_ij (Q^ab_W)^ell for ell >= 3 is used to write the covariance K in Eq. (57) and hence the g(q_W) terms in Results 3.1 and 3.2. It is only checked numerically in Fig. 3 for a single polynomial activation and for the few overlaps appearing in that simulation; it is not verified for ReLU, ELU, or for off-equilibrium MCMC trajectories. Since g(q_W) contributes directly to q_K(q2, q_W) and to the free entropies f_uni and f_sp, I ask the authors to include a quantitative test of (15), for example by comparing both sides of the quadratic form (10) for a set of activations and for intermediate values of q_W, or to prove a bound that justifies the approximation in the relevant limit. Without such a test, the quantitative predictions of alpha_sp and of the specialisation-phase generalisation error rest on an assumption that is only spot-checked.

minor comments (6)

[Appendix E.3] The sentence after Eq. (60), 'combined with the Nishimori identities , combined with the Nishimori identities mW = qW, ...', contains a duplicated phrase; please remove the repetition.
[Eq. (22)] The notation S^a_{2;alpha alpha} (with a semicolon) is not defined; please introduce a consistent notation for the entries of S^a_2, for example S^a_2(alpha, beta).
[Abstract] The abstract claims the theory applies to 'any activation function'; the body requires the activation to admit a Hermite expansion and the main text assumes mu_0 = 0 (App. F relaxes this to non-centred activations). Please qualify the abstract to reflect these conditions.
[Fig. 1 caption] The sentence about MCMC points following the specialisation curve before the transition should state clearly that these points are obtained with informative initialisation and therefore sample the metastable specialisation branch, not the equilibrium Gibbs distribution.
[Section 6] The statement 'we observe small but non-negligible deviations ... see for instance Fig. 2 around alpha = 4' should specify whether the deviation is within the finite-size error bars and, if not, discuss its possible origin, such as lack of equilibration or a systematic effect of the ansaetze.
[Table 1] Please note in the caption that the Hermite coefficients are defined with respect to probabilist's Hermite polynomials, since the numerical values depend on the normalization convention.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: generalisation error and α_sp are saddle-point outputs, not fitted inputs; the stated ans\u00e4tze are explicit limitations, not self-referential predictions.

full rationale

The central claims (Result 3.1 and Result 3.2) are derived from replica free-entropy computations. The order parameters q2, qW, and qK are obtained by extremizing the free entropies through the saddle-point systems (Suni) and (Ssp), and no generalisation-error datum is inserted as a fitted parameter. The transition location α_sp is defined as the crossing of the two free-entropy branches, not as a fit to the empirical error curves. The main assumptions are explicit: the joint Gaussian ansatz on replicated post-activations (Section 5), the Hadamard-power diagonalisation (15), and the conditional Gaussian ansatz on S2 in (22). These are heuristics that the paper cannot prove and validates only a posteriori; if any of them fails, the quantitative predictions fail with them. That is a correctness or verification risk, not circularity, because the predictions are not equivalent to the ansatz inputs by construction. The empirical MCMC points are obtained by plugging measured overlaps into the exact covariance formula (45), while the theoretical curves are obtained from the solved saddle-point overlaps; the comparison is therefore informative rather than tautological. Self-citations to Barbier and Panchenko, Barbier and Macris, and Barbier et al. (2024) are either rigorous external theorems or side consistency remarks; they do not carry the derivation of α_sp or the generalisation error. The paper itself flags small deviations near transitions and the ~1% miss of q_sp for Gaussian read-outs in Appendix I, which further shows the validation is not a circular fit. Overall, no load-bearing step reduces to its own input by definition or by fitted-parameter construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim depends on four unproved assumptions, most importantly the Gaussian ansatz and the Hadamard-power diagonalisation. None of them is a fitted parameter; all are structural ansatze of the replica calculation.

assumptions (4)

ad hoc to paper The replicated post-activations {lambda_a} are jointly Gaussian under the posterior and teacher randomness.
Invoked in Sec. 5 and App. C to obtain the covariance formula (10) and the free entropies; explicitly stated to be unproven and validated only a posteriori by numerical agreement.
ad hoc to paper Higher-order Hadamard powers of the overlap matrix become diagonal at leading order: (Omega^ab)^ell is approximately delta_ij (Q^ab_W)^ell for ell >= 3.
Eq. (15) in Sec. 5; verified numerically in Fig. 3 for a few overlaps but assumed for all ell and for all posterior samples.
domain assumption Replica method and replica-symmetric saddle point are asymptotically exact for this Bayes-optimal problem, with the limits s to 0 and d,n,k to infinity commuting.
Used throughout Sec. 5 and App. E; standard in statistical physics of learning but not proven for this model; references Barbier and Panchenko for general Bayes-optimal inference.
ad hoc to paper Q1 converges to the all-ones matrix and Qv converges to the all-ones matrix in the specialisation phase, that is first-order tensor and read-out weights are perfectly learnable when the inner weights synchronise.
Assumed in Sec. 5 before Eq. (22); justified by counting arguments but not proven.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation." pith.science (2026). https://pith.science/paper/UM24OJY7

@misc{pith2026250118530,
  author       = {Pith},
  title        = {Pith review of: Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UM24OJY7}},
  note         = {Machine review of arXiv:2501.18530}
}

abstract

We consider a teacher-student model of supervised learning with a fully-trained two-layer neural network whose width $k$ and input dimension $d$ are large and proportional. We provide an effective theory for approximating the Bayes-optimal generalisation error of the network for any activation function in the regime of sample size $n$ scaling quadratically with the input dimension, i.e., around the interpolation threshold where the number of trainable parameters $kd+k$ and of data $n$ are comparable. Our analysis tackles generic weight distributions. We uncover a discontinuous phase transition separating a "universal" phase from a "specialisation" phase. In the first, the generalisation error is independent of the weight distribution and decays slowly with the sampling rate $n/d^2$, with the student learning only some non-linear combinations of the teacher weights. In the latter, the error is weight distribution-dependent and decays faster due to the alignment of the student towards the teacher network. We thus unveil the existence of a highly predictive solution near interpolation, which is however potentially hard to find by practical algorithms.

Figures

Figures reproduced from arXiv: 2501.18530 by the authors.

**Figure 1.** Top: Theoretical prediction (solid curves) of the Bayesoptimal mean-square generalisation error for binary inner weights and polynomial activations: σ1(x) = He2(x)/ √ 2, σ2(x) = He3(x)/ √ 6, σ3(x) = He2(x)/ √ 2 + He3(x)/6, with γ = 0.5, d = 150, Gaussian label noise with ∆ = 1.25, and fixed read-outs v = v 0 = 1. Dots are obtained by plugging the overlaps obtained from MCMC into Eq. (45) in App. C, which neglects … view at source ↗

**Figure 2.** Theoretical prediction (solid curves: blue for the universal branch, red for the specialisation one) of the Bayes-optimal mean-square generalisation error for Gaussian inner weights and ReLU activation, d = 150, γ = 0.5, ∆ = 0.1, fixed read-outs v = v 0 = 1. Here the specialisation transition is at αsp ≈ 5.54. The numerical points are obtained with Hamiltonian Monte Carlo with informative/uninformative initialisati… view at source ↗

**Figure 3.** Hamiltonian Monte Carlo dynamics of the overlaps qW = Q 01 W and qℓ = Q 01 ℓ (ℓ = 1, . . . , 5) between student and teacher weights, with activation function σ(x) = He1(x) + He2(x)/ √ 2 + He3(x)/6, d = 200, γ = 0.5, ∆ = 1.25 and two different choices of sample rate: α = 0.5 (Top), α = 5 (Bottom). The inner weights W0 of the teacher are Gaussian, while the readouts v 0 binary. The dynamics is initialised informative… view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Mutual information in the universal (dashed) and specialisation (dotted) phase, for the activation functions in [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]

**Figure 5.** Figure 5: Left: Mutual information for Gaussian prior over the inner weights of the universal (dashed) and specialisation (dotted) solutions, for the activation functions in [PITH_FULL_IMAGE:figures/full_fig_p028_5.png]

**Figure 6.** Figure 6 [PITH_FULL_IMAGE:figures/full_fig_p030_6.png]

**Figure 7.** Figure 7: Same as in [PITH_FULL_IMAGE:figures/full_fig_p031_7.png]

**Figure 8.** Figure 8: Trajectories of the generalisation error of neural networks trained with ADAM at fixed batch size B = ⌊n/4⌋, for ReLU activation with parameters ∆ = 10−4 , γ = 0.5 and α = 5.0 > αsp. Left: Constant read-outs. Centre: Rademacher read-outs. Right: Gaussian read-outs. Rea…

**Figure 9.** Figure 9 [PITH_FULL_IMAGE:figures/full_fig_p032_9.png]

**Figure 10.** Figure 10: Trajectories of the overlap q2 in HMC runs for the polynomial activation σ3(x) = He2(x)/ √ 2 + He3(x)/6 with parameters ∆ = 0.1, γ = 0.5 and α = 1.0 > αsp (= 0.790, 0.678, 0.933 for constant, Rademacher and Gaussian read-outs respectively), as explained in the text. L…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Microscopic and collective signatures of feature learning in neural networks
cond-mat.dis-nn 2025-08 conditional novelty 6.0 of 10

In over-parameterized Bayesian one-hidden-layer networks, class-manifold separation becomes nonmonotonic in temperature and hidden weights develop data-dependent correlations, signatures of feature learning despite Ga...

Reference graph

Works this paper leans on

21 extracted references · 12 canonical work pages · cited by 1 Pith paper

[1]

Specifically, declining the saddle point equations (104) to Rademacher prior it is easy to see that qW = E tanh( p ˆqW ξw + ˆqW ) (122) with ˆqW containing a global factor α, and increasing with it. The presence of the hyperbolic tangent is what yields the characteristic exponential saturation to 1 of qW when α grows, and thus the exponential decrease of ...

work page 2024
[2]

Notice that we have re-normalised Sa 2 by √ d in order to work with matrices with O(1) eigenvalues

exp h d 4 sX a,b=0 ˆQab 2 Tr Sa 2√ d Sb⊺ 2√ d i (77) up to vanishing corrections. Notice that we have re-normalised Sa 2 by √ d in order to work with matrices with O(1) eigenvalues. Using the Bayes-optimality of the setting, we can perform an additional simplifying RS ansatz on the saddle point optimisation: Qaa 2 = r2, 0 ≤ a ≤ s, and Qab 2 = q2, a ̸= b, ...

work page
[6]

Pacelli, R., Ariosto, S., Pastore, M., Ginelli, F., Gher- ardi, M., and Rotondo, P

URL https://www.sciencedirect.com/ science/article/pii/S0304414910002917. Pacelli, R., Ariosto, S., Pastore, M., Ginelli, F., Gher- ardi, M., and Rotondo, P. A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit. Nature Machine Intelligence , 5(12):1497–1507, 12 2023. ISSN 2522-5839. doi: 10.1038/s42256-023-00...

work page arXiv 2023
[7]

URL https://doi

doi: 10.1214/21-AOS2133. URL https://doi. org/10.1214/21-AOS2133. Hu, H. and Lu, Y . M. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory, 69(3):1932–1964, 2023. doi: 10. 1109/TIT.2022.3217698. URL https://doi.org/ 10.1109/TIT.2022.3217698. Hu, H., Lu, Y . M., and Misiakiewicz, T. Asymptotics of ...

work page arXiv 1932
[14]

cc/paper_files/paper/2020/file/ 9b8b50fb590c590ffbf1295ce92258dc-Paper

URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ 9b8b50fb590c590ffbf1295ce92258dc-Paper. pdf. Schwarze, H. and Hertz, J. Generalization in a large commit- tee machine. Europhysics Letters, 20(4):375, Oct 1992. doi: 10.1209/0295-5075/20/4/015. URL https://dx. doi.org/10.1209/0295-5075/20/4/015. Schwarze, H. and Hertz, J. Generalization in f...

work page arXiv 2020
[18]

Formally the integration is over the imaginary axis iR

exp − d(ˆr2 + ˆq2) 4 sX a=0 Tr h Sa 2√ d 2i + dˆq2 4 Tr h sX a=0 Sa 2√ d 2i + d2 4ns (s + 1)ˆr2r2 − s(s + 1)ˆq2q2 o , (80) 1In this manuscript, we often represent the delta function using its Fourier representation δ(x − c) = 1 2π R iR dˆx exp(ˆx(x − c)). Formally the integration is over the imaginary axis iR. The complex-valued Fourier conjugates ˆx asso...

work page
[19]

The high-dimensional integral that remains is the free entropy per datum of a Bayes-optimal matrix denoising problem: Y(ˆq2) = p ˆq2 ¯S0 2 + Z, (82) with a rotationally invariant prior on ¯S0

work page
[20]

(84) Here I(¯S0 2; Y(ˆq2)) is the MI related to the channel (82), µY(ˆq2) is the asymptotic spectral law of the observation matrix Y(ˆq2)

Therefore, we can directly import the results from (Pourkamali et al., 2024; Maillard et al., 2024b): Jn,0(q2) = 1 αextr n ˆq2(r2 − q2) 4 − ι(ˆq2) o (83) where we remind the reader that r2 = 1 + γ(Ev0)2, and ι(ˆq2) := lim d→∞ 1 d2 I(¯S0 2; Y(ˆq2)) = 1 8 + 1 2 Z ln |x − y| dµY(ˆq2)(x)dµY(ˆq2)(y). (84) Here I(¯S0 2; Y(ˆq2)) is the MI related to the channel ...

work page 1988

Show all 21 references

[39]

Aiudi, R., Pacelli, R., Baglioni, P., Vezzani, A., Burioni, R., and Rotondo, P

URL https://scipost.org/10.21468/ SciPostPhys.18.1.039. Aiudi, R., Pacelli, R., Baglioni, P., Vezzani, A., Burioni, R., and Rotondo, P. Local kernel renormalization as a mechanism for feature learning in overparametrized convolutional neural networks. Nature Communications, 16...
[271]

com/doi/abs/10.1142/0271

URL https://www.worldscientific. com/doi/abs/10.1142/0271. Monasson, R. Properties of neural networks storing spa- tially correlated patterns. Journal of Physics A: Math- ematical and General , 25(13):3701, Jul 1992. doi: 10.1088/0305-4470/25/13/019. URL https://dx. doi.org/10...

1992 doi
[1980]

doi: 10.1063/1.524438

ISSN 0022-2488. doi: 10.1063/1.524438. URL https://doi.org/10.1063/1.524438. Kabashima, Y ., Krzakala, F., M´ezard, M., Sakata, A., and Zdeborov´a, L. Phase transitions and sample complexity in Bayes-optimal matrix factorization. IEEE Transactions on Information Theory, 62(7):...
[1996]

doi: 10.1007/ 978-1-4612-0745-0 2

ISBN 978-1-4612-0745-0. doi: 10.1007/ 978-1-4612-0745-0 2. URL https://doi.org/10. 1007/978-1-4612-0745-0_2 . Nishimori, H. Statistical Physics of Spin Glasses and In- formation Processing: An Introduction . Oxford Uni- versity Press, 07 2001. ISBN 9780198509417. doi: 10.1093/...
[1998]

URL https: //doi.org/10.1088/0305-4470/31/38/012

doi: 10.1088/0305-4470/31/38/012. URL https: //doi.org/10.1088/0305-4470/31/38/012. 14 Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation A. Hermite basis and Mehler’s formula Recall the Hermite expansion of the activat...

2024 doi
[2018]

cc/paper_files/paper/2018/file/ 84f0f20482cde7e5eacaf7364a643d33-Paper

URL https://proceedings.neurips. cc/paper_files/paper/2018/file/ 84f0f20482cde7e5eacaf7364a643d33-Paper. pdf. Baglioni, P., Pacelli, R., Aiudi, R., Di Renzo, F., Vezzani, A., Burioni, R., and Rotondo, P. Predictive power of a Bayesian effective action for fully connected one h...

2018 doi
[2020]

Camilli, F., Tieplova, D., and Barbier, J

URL https://proceedings.mlr.press/ v119/bordelon20a.html. Camilli, F., Tieplova, D., and Barbier, J. Fundamental limits of overparametrized shallow neural networks for super- vised learning, 2023. URL https://arxiv.org/ abs/2307.05635. Canatar, A., Bordelon, B., and Pehlevan, ...

2023 arXiv
[2021]

cc/paper_files/paper/2021/file/ b24d21019de5e59da180f1661904f49a-Paper

URL https://proceedings.neurips. cc/paper_files/paper/2021/file/ b24d21019de5e59da180f1661904f49a-Paper. pdf. Neal, R. M. Priors for Infinite Networks , pp. 29–53. Springer New York, New York, NY ,

2021
[2022]

Guionnet, A

URL https://proceedings.mlr.press/ v145/goldt22a.html. Guionnet, A. and Zeitouni, O. Large deviations asymptotics for spherical integrals. Journal of Functional Analysis , 188(2):461–515, 2002. ISSN 0022-1236. doi: 10.1006/jfan.2001.3833. URL https://www.sciencedirect.com/ sci...

2002
[2023]

Del Giudice, P., Franz, S., and Virasoro, M

URL https://proceedings.mlr.press/ v202/cui23b.html. Del Giudice, P., Franz, S., and Virasoro, M. A. Per- ceptron beyond the limit of capacity. J. Phys. France, 50(2):121–134, 1989. doi: 10.1051/jphys: 01989005002012100. URL https://doi.org/10. 1051/jphys:01989005002012100. Di...

1989
[2024]

URL https://proceedings.mlr.press/ v238/martin24a.html. Mato, G. and Parga, N. Generalization properties of multi- layered neural networks. Journal of Physics A: Math- ematical and General , 25(19):5047, Oct 1992. doi: 10.1088/0305-4470/25/19/017. URL https://dx. doi.org/10.10...

1992
[2025]

van Meegen, A

URL https://openreview.net/forum? id=Mwzui5H0VN. van Meegen, A. and Sompolinsky, H. Coding schemes in neural networks learning classification tasks, 2024. URL https://arxiv.org/abs/2406.16689. Venturi, L., Bandeira, A. S., and Bruna, J. Spurious valleys in one-hidden-layer neu...

2024 arXiv
[8772]

URL https: //doi.org/10.1093/imaiai/iaaa008

doi: 10.1093/imaiai/iaaa008. URL https: //doi.org/10.1093/imaiai/iaaa008. Barbier, J. and Macris, N. The adaptive interpolation method: a simple scheme to prove replica formulas in bayesian inference. Probability Theory and Related Fields, 174(3):1133–1185, Aug 2019. ISSN 1432...

arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

Specifically, declining the saddle point equations (104) to Rademacher prior it is easy to see that qW = E tanh( p ˆqW ξw + ˆqW ) (122) with ˆqW containing a global factor α, and increasing with it. The presence of the hyperbolic tangent is what yields the characteristic exponential saturation to 1 of qW when α grows, and thus the exponential decrease of ...

work page 2024

[2] [2]

Notice that we have re-normalised Sa 2 by √ d in order to work with matrices with O(1) eigenvalues

exp h d 4 sX a,b=0 ˆQab 2 Tr Sa 2√ d Sb⊺ 2√ d i (77) up to vanishing corrections. Notice that we have re-normalised Sa 2 by √ d in order to work with matrices with O(1) eigenvalues. Using the Bayes-optimality of the setting, we can perform an additional simplifying RS ansatz on the saddle point optimisation: Qaa 2 = r2, 0 ≤ a ≤ s, and Qab 2 = q2, a ̸= b, ...

work page

[3] [6]

Pacelli, R., Ariosto, S., Pastore, M., Ginelli, F., Gher- ardi, M., and Rotondo, P

URL https://www.sciencedirect.com/ science/article/pii/S0304414910002917. Pacelli, R., Ariosto, S., Pastore, M., Ginelli, F., Gher- ardi, M., and Rotondo, P. A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit. Nature Machine Intelligence , 5(12):1497–1507, 12 2023. ISSN 2522-5839. doi: 10.1038/s42256-023-00...

work page arXiv 2023

[4] [7]

URL https://doi

doi: 10.1214/21-AOS2133. URL https://doi. org/10.1214/21-AOS2133. Hu, H. and Lu, Y . M. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory, 69(3):1932–1964, 2023. doi: 10. 1109/TIT.2022.3217698. URL https://doi.org/ 10.1109/TIT.2022.3217698. Hu, H., Lu, Y . M., and Misiakiewicz, T. Asymptotics of ...

work page arXiv 1932

[5] [14]

cc/paper_files/paper/2020/file/ 9b8b50fb590c590ffbf1295ce92258dc-Paper

URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ 9b8b50fb590c590ffbf1295ce92258dc-Paper. pdf. Schwarze, H. and Hertz, J. Generalization in a large commit- tee machine. Europhysics Letters, 20(4):375, Oct 1992. doi: 10.1209/0295-5075/20/4/015. URL https://dx. doi.org/10.1209/0295-5075/20/4/015. Schwarze, H. and Hertz, J. Generalization in f...

work page arXiv 2020

[6] [18]

Formally the integration is over the imaginary axis iR

exp − d(ˆr2 + ˆq2) 4 sX a=0 Tr h Sa 2√ d 2i + dˆq2 4 Tr h sX a=0 Sa 2√ d 2i + d2 4ns (s + 1)ˆr2r2 − s(s + 1)ˆq2q2 o , (80) 1In this manuscript, we often represent the delta function using its Fourier representation δ(x − c) = 1 2π R iR dˆx exp(ˆx(x − c)). Formally the integration is over the imaginary axis iR. The complex-valued Fourier conjugates ˆx asso...

work page

[7] [19]

The high-dimensional integral that remains is the free entropy per datum of a Bayes-optimal matrix denoising problem: Y(ˆq2) = p ˆq2 ¯S0 2 + Z, (82) with a rotationally invariant prior on ¯S0

work page

[8] [20]

(84) Here I(¯S0 2; Y(ˆq2)) is the MI related to the channel (82), µY(ˆq2) is the asymptotic spectral law of the observation matrix Y(ˆq2)

Therefore, we can directly import the results from (Pourkamali et al., 2024; Maillard et al., 2024b): Jn,0(q2) = 1 αextr n ˆq2(r2 − q2) 4 − ι(ˆq2) o (83) where we remind the reader that r2 = 1 + γ(Ev0)2, and ι(ˆq2) := lim d→∞ 1 d2 I(¯S0 2; Y(ˆq2)) = 1 8 + 1 2 Z ln |x − y| dµY(ˆq2)(x)dµY(ˆq2)(y). (84) Here I(¯S0 2; Y(ˆq2)) is the MI related to the channel ...

work page 1988