Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theories

Haobo Zhang; Jianfa Lai; Jun S. Liu; Qian Lin; Yicheng Li

REVIEW 3 major objections 5 minor 1 cited by

Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theories

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes that feature learning in neural networks can be studied through an over-parameterized Gaussian sequence model, where training is gradient flow on an orthogonal rotation and a diagonal scaling that align with the true…

desk verdict Solid survey, genuine new prototype, but the bridge to neural networks is a hypothesis that fails in the high-dimensional regime where it matters most. read the letter →

arxiv 2412.18756 v1 pith:MYC3LECP submitted 2024-12-25 cs.LG math.STstat.TH

Haobo Zhang , Jianfa Lai , Yicheng Li , Qian Lin , Jun S. Liu This is my paper · ORCID

classification cs.LGmath.STstat.TH

keywords neuralnetworksgeneralizationabilityfeaturelearningkernelregressiontangentover-parameterizedGaussiansequencemodeladaptivegradientflow

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the statistical essence of feature learning in neural networks can be captured by a much simpler object: an over-parameterized Gaussian sequence model, in which data are noisy coefficients of the truth in a fixed basis and training evolves an orthogonal rotation and a diagonal scaling alongside a linear readout. The supporting claim is that kernel gradient flow—the infinite-width limit of neural-network training—is strongly equivalent to gradient flow in a classical Gaussian sequence model once the empirical feature matrix is close to orthonormal. If that equivalence holds, changing the network's features corresponds exactly to multiplying the sequence model by a learnable orthogonal-diagonal pair, so feature learning reduces to tracking how that pair aligns with the true parameter vector. The paper reviews why fixed-kernel theory cannot explain this alignment, presents supporting rate-matching calculations and simulations, and leaves the rigorous high-dimensional equivalence as an open problem. A sympathetic reader would take the proposal as a new prototype for turning the question 'how do networks learn features?' into a tractable statistical question about rotation-and-scale alignment.

What carries the argument

The central object is the over-parameterized Gaussian sequence model: noisy coefficients $z_j = \theta^*_j + \xi_j$ are explained by a product $A_t D_t \alpha_t$, with $A_t$ orthogonal (rotation of the basis), $D_t$ diagonal (rescaling of eigenvalues), and $\alpha_t$ the readout. The work it does is to convert the intractable coupled dynamics of an adaptive feature model into a structured matrix-factorization flow: rotating and scaling the frame is exactly what a neural network does when it learns features, and the model reduces to vanilla Gaussian sequence flow when $A_t$ and $D_t$ are frozen. Carrying the argument is the hypothesized equivalence between kernel gradient flow and Gaussian sequence gradient flow, made plausible by the formal identity of the coordinate-wise dynamics and by the matching learning-curve rates in the regularized regime.

What would settle it

Compute the empirical feature covariance $n^{-1}\Psi(X)\Psi(X)^\top$ for an inner-product kernel on the $d$-sphere with $n \asymp d^\gamma$ and a source-condition truth, then compare the actual kernel-gradient-flow generalization error with the Gaussian-sequence-model prediction $n^{-s\alpha} + n^{-(1-\alpha/\beta)}$ for the same regularization schedule; if the two diverge as the covariance deviates from $I$, the equivalence is falsified at that scaling.

Watch

Extended reading notes

Core claim

The paper's central proposal is the over-parameterized Gaussian sequence model (Definition 4.3.1): observations $z_j = \theta^*_j + \xi_j$ with $\xi_j \sim N(0,n^{-1})$ are fitted by gradient flow on the loss $L(A,D,\alpha) = \|Z - AD\alpha\|_2^2$, where $A_t$ stays orthogonal, $D_t$ stays diagonal, and $\alpha_0 = 0$. The authors hypothesize that this model is the adaptive-feature counterpart of the classical Gaussian sequence model, in the same way that the adaptive feature model $\Phi_{\eta_t}(x)^\top \beta_t$ is the adaptive counterpart of kernel regression. Concretely, they argue that replacing a kernel's eigenfunctions and eigenvalues by a new feature is equivalent to multiplying the sequence model by an orthogonal-diagonal matrix pair, so learning features means learning $A_t$ and $D_t$. They support the equivalence by showing that kernel gradient flow and the Gaussian sequence gradient flow obey the same coordinate-wise dynamics under the approximation $\frac{1}{n}\Psi(X)\Psi(X)^\top \approx I$ and $\frac{1}{n}\Psi(X)Y \approx \Lambda^{1/2}\beta^* + \text{noise}$, and by showing the two models produce identical learning-curve rates $n^{-s\alpha} + n^{-(1-\alpha/\beta)}$ in the regularized regime. The intended payoff is that feature learning and its generalization benefits can be studied through the alignment of $A_t$ and $D_t$ with $\theta^*$, a question the paper poses as the next step and partly illustrates by simulation.

Load-bearing premise

The whole argument rests on treating the data's feature vectors as nearly uncorrelated and the labels as nearly equal to the true coefficients plus independent noise; the paper itself notes that a rigorous justification of this concentration, especially in high dimensions, remains open.

Editorial extensions

If this is right

Feature learning becomes a quantitative alignment problem: the key observable is how the projections $u_j^\top \theta^*$ (columns of $A_t$) concentrate on the large entries of $D_t$, and generalization should improve as this concentration increases.
The over-parameterized model reproduces kernel-gradient-flow rates when $A_t$ and $D_t$ are frozen, so it generalizes NTK and kernel theory as a special case rather than contradicting it.
The model predicts that allowing $A_t$ and $D_t$ to move yields faster convergence than vanilla sequence flow, consistent with the one-step feature-learning gains and the over-parameterized regression results reviewed in Section 4.4.
If the equivalence is made rigorous, the polynomial approximation barrier for fixed kernels becomes a statement about frozen $A$ and $D$; adaptive features should beat it whenever the truth is aligned with a low-dimensional learnable subspace.
The framework suggests concrete comparisons between architectures: different network structures induce different families of orthogonal and diagonal updates, so their feature-learning efficiency can be compared through the resulting alignment dynamics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The model can be read as a nonconvex matrix factorization $Z \approx A_t D_t \alpha_t$; existing results on gradient-flow implicit regularization in factorization problems could give the first rigorous statements about which structures of $\theta^*$ the flow aligns with, a direction the paper does not pursue.
A direct way to test the equivalence is to simulate kernel gradient flow on an inner-product kernel in high dimensions where the empirical covariance is known not to concentrate, then compare the trajectory against the Gaussian sequence flow; any rate gap would delimit where the prototype stops being faithful.
If the equivalence holds, the paper's prototype reframes the benign-overfitting debate: interpolation in the over-parameterized model corresponds to late-time growth of $D_t$, so the model could explain when interpolation is benign versus catastrophic as a function of $\theta^*$ alignment, which the paper only touches implicitly.
The alignment measure used in the simulations (percentage of first-$p$ projections) is a testable statistic; applying the same measure to real networks with varying width and depth would provide a scaling-law check of whether the prototype's rotation dynamics match practice.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about arXiv:2412.18756. First, the survey parts (Sections 2–3) are well done and genuinely useful: they organize the NTK and kernel regression literature around learning curves, the polynomial approximation barrier, multiple descent, interpolation phase diagrams, and recent feature-learning results. Second, the new contribution is Definition 4.3.1, the over-parameterized Gaussian sequence model, in which both an orthogonal A_t and a diagonal D_t evolve under gradient flow. That is a real, if small, extension of Li and Lin (2024), and the intuition—that feature learning should be about aligning A_t and D_t with θ*—is sensible and cleanly stated.

The honest reading is that the paper proposes a direction rather than proving a theorem. The bridge from kernel gradient flow to the Gaussian sequence model in Section 4.2 rests on the approximation (1/n)Ψ(X)Ψ(X)^T ≈ I. Entrywise, for fixed coordinates, a law of large numbers gives something like this, but the gradient flow requires a uniform approximation over the evolving β_t, and when the active feature dimension exceeds n, the empirical Gram matrix has rank at most n, so the operator-norm distance to I is at least 1. That is not a technicality: the high-dimensional phenomena reviewed in §2.4 are driven by the non-identity structure of the empirical kernel matrix. The authors know this—Section 5 explicitly says a rigorous equivalence, especially in high dimensions, is open. So the central prototype is conditional on a hypothesis that is unsupported in exactly the regime where feature learning is supposed to matter.

The simulation in §4.3.3 is illustrative: it shows the expected concentration of projections, but there is no code, no error bars, and the orthogonality-preserving update for A_t is not specified. Those are addressable, but they make the empirical support thin.

Who should read this? People who want a compact, accurate survey of kernel regression and feature learning, and people working on sequence-model prototypes. The paper deserves a serious referee: the survey alone is worth publishing, and the prototype is well-defined enough to invite follow-up work. A referee should insist on a precise statement of the equivalence conditions and on code or at least a full description of the simulation.

My recommendation: send it out for peer review, but expect heavy revision. I would cite it as the reference for the over-parameterized Gaussian sequence model.

Referee Report

3 major / 5 minor

Summary. This paper is a review-and-proposal article. It first reviews the neural tangent kernel theory and recent kernel regression results in both fixed and high dimensions, then discusses limitations of fixed-feature theories and recent work on feature learning. Its original contribution is the over-parameterized Gaussian sequence model (Definition 4.3.1), in which an orthogonal matrix A_t and a diagonal matrix D_t evolve together with a coefficient vector alpha_t under gradient flow; the paper argues that this model is a prototype for the adaptive feature model of neural networks. The link between kernel gradient flow and the Gaussian sequence model is explicitly presented as a hypothesis (Section 4.2), with supporting heuristics and a review of matching convergence rates, while the rigorous equivalence in high dimensions is listed as an open problem in the discussion (Section 5).

Significance. The review component is competent and genuinely useful: it synthesizes a large body of recent work on kernel regression learning curves, interpolation, and high-dimensional phenomena, and it frames feature learning through kernel alignment in a way that connects several threads of the literature. The paper's main value is conceptual: the over-parameterized Gaussian sequence model is a clean, tractable object that isolates the possible effect of learnable features and eigenvectors on gradient-flow dynamics, and the authors are transparent about the unproven nature of the kernel-to-sequence equivalence. The proposal is falsifiable in the sense that its usefulness depends on a specific conjectured equivalence, and the simulation in Section 4.3.3 demonstrates a qualitative alignment effect. However, the manuscript does not establish the model's connection to actual neural networks in the high-dimensional regime where feature learning matters, and it lacks any quantitative demonstration that the full (A_t, D_t) model improves upon simpler variants already in the literature.

major comments (3)

[Section 4.2.1, Eq. (22)] The approximation (1/n)Psi(X)Psi(X)^T = I used to pass from kernel gradient flow (19) to the sequence-model dynamics (22) is asserted to hold 'as the sample size n increases', but the claimed equality of the two gradient flows requires uniform control over the evolving beta_t. When the number of active features exceeds n, the empirical feature covariance has rank at most n, so its operator norm distance to the identity is at least 1; moreover, the non-identity structure of the empirical kernel is precisely what drives the high-dimensional phenomena reviewed in Section 2.4 (polynomial approximation barrier, multiple descent, phase diagrams). Section 5 lists the rigorous equivalence, especially in high dimensions, as an open problem. Since Definition 4.3.1 is motivated through this equivalence, the paper's central prototype is at present a model of the Gaussian sequence model rather than of neural-network feature learning in the high-dimensional regime. Please either restrict the claimed scope, provide quantitative evidence (for example, simulations comparing kernel gradient flow and the sequence model in high dimensions) that the finite-rank effect does not qualitatively change the dynamics, or state the conjectured equivalence with the precise conditions under which it is expected to hold.
[Section 4.3.2, Definition 4.3.1 and Eq. (28)] The gradient flow in (28) is written as unconstrained Euclidean gradient descent, yet the model requires A_t to remain orthogonal and D_t to remain diagonal. The text states that these properties are 'kept' during gradient flow, but it does not specify the mechanism (for example, projection onto the Stiefel manifold, reparametrization, or a natural-gradient update). The simulation in Section 4.3.3 also does not describe the update rule used for A_t and D_t. Without this specification the model is not fully well-defined and the simulation is not reproducible; please define the constrained dynamics explicitly, or clarify that the gradient-flow statement is a formal abbreviation for a constrained gradient flow.
[Section 4.3.3 and Section 4.4] The simulation in Section 4.3.3 shows that the top-projection percentage increases with iterations, but it does not quantify generalization error or compare the over-parameterized model's risk against the vanilla gradient flow (21) or the D-only model of Li and Lin (2024). The expectation stated in Section 4.4 that the additional A_t dynamics yield 'greater improvements' is therefore not supported by the paper's own experiments. To substantiate the claimed benefit of the orthogonal component, the paper should provide at least one direct comparison of the three models' risks (for instance, as a function of n or t) or, alternatively, explicitly mark this as an open conjecture rather than a demonstrated advantage.

minor comments (5)

[Section 2.3.2, Eq. (9)] The notation Omega-tilde is defined in the text but is not included in the global notation list at the end of Section 1; adding it would help readers. Also, the sentence before Eq. (9) states 'with beta > 1, s > 0' but the embedding index assumption is invoked without being displayed again; a brief reminder would improve readability.
[Figure 1] The caption mentions 'dashed lines' and 'Figure 1 (a)', but the displayed figure has no panel label (a); please label the panel or update the caption so the reference is meaningful.
[Section 4.1, Figure 5] The relation between the feature dimension m = 500 used in the projection percentage and the network widths described in the experimental details is not stated explicitly. Please clarify how m is obtained from the fully connected network (for MNIST) and the convolutional network (for CIFAR-10).
[Section 4.3.2] In the sentence beginning 'As in Section 3.2, we hope that projections...', the notation u_j is introduced as the j-th column vector of A_t, but if N is infinite this is informal; please clarify whether N is assumed finite in Definition 4.3.1 or use an index-set formulation for the infinite case.
[Section 4.2] The inner product notation <f*, psi_j>_{L2} is used before the underlying space L^2(X, mu) is explicitly identified in the context of Mercer's decomposition; a one-line definition at first use would avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proposed over-parameterized Gaussian sequence model is explicitly presented as a hypothesis-driven prototype, and its supporting calculations are derived directly rather than being forced by the reviewed results.

full rationale

The paper's central contribution, Definition 4.3.1, is a proposed prototype model, not a theorem derived from its inputs. Section 4.2 explicitly labels the kernel-gradient-flow to Gaussian-sequence-model relation as a hypothesis ('We hypothesize that there is a strong equivalence between the kernel gradient flow (19)... and the corresponding gradient flow for the Gaussian sequence model (21)...'), and Section 5 acknowledges that 'a rigorous proof and the high-dimensional case remain open problems.' The convergence-rate comparison in Section 4.2.2 computes the Gaussian sequence model risk directly by solving the ODE and applying eigenvalue-decay and source-condition assumptions that the paper states as analogous to Assumptions 1 and 2; it does not use the kernel learning-curve result (9) as an input, so the agreement is a consistency check rather than a fitted prediction. The self-citations in the review sections (e.g., Li et al. 2024a, Zhang et al. 2024a,b,c) supply published background results and are not load-bearing for the proposed model, which is self-contained. The simulation in Section 4.3.3 is illustrative and involves no parameter fitting to a prediction target. No step in the paper reduces a claimed result to its own inputs by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central proposal relies on standard RKHS and kernel-regression background, plus the unproven concentration heuristic that connects kernel gradient flow to a Gaussian sequence model. No new physical or mechanistic entities are introduced. Free parameters appear only in the illustrative simulation.

free parameters (2)

Simulation coefficients θ*_j and λ_j = θ*_j = 1/(N-j+2), λ_j = 1/(j+5)^2
Hand-chosen in the Section 4.3.3 simulation to create an initial misalignment between features and target; they are illustrative and not fitted to real data, and they do not enter any theoretical derivation.
Simulation hyperparameters = N=500, n=4000, learning rate=0.5
Configuration of the illustrative simulation in Section 4.3.3; no claim of optimality or generality is attached to these values.

assumptions (4)

standard math Mercer decomposition of the kernel and the RKHS representation (Section 2.2, Eq. (4))
Used to define the feature map Φ(x) = Λ^{1/2}Ψ(x) and to state kernel regression equivalences; standard functional analysis background.
domain assumption Eigenvalue decay rate (Assumption 1) and source condition (Assumption 2) in fixed dimensions
Used to state the learning-curve rates for kernel gradient flow (Section 2.3.2) and to derive the analogous rates in the Gaussian sequence model (Section 4.2.2).
domain assumption Embedding index assumption for interpolation spaces (Section 2.3.2)
Invoked for the kernel gradient flow learning curve result in Eq. (9); a technical condition on the kernel that is standard in this literature.
ad hoc to paper Approximation (1/n)Ψ(X)Ψ(X)^T ≈ I for large n (Section 4.2.1)
This concentration heuristic underpins the hypothesized equivalence between kernel gradient flow and the Gaussian sequence model. It is not proven and may fail in high dimensions, and the paper flags the rigorous equivalence as an open problem.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theories." pith.science (2026). https://pith.science/paper/MYC3LECP

@misc{pith2026241218756,
  author       = {Pith},
  title        = {Pith review of: Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MYC3LECP}},
  note         = {Machine review of arXiv:2412.18756}
}

read the original abstract

A primary advantage of neural networks lies in their feature learning characteristics, which is challenging to theoretically analyze due to the complexity of their training dynamics. We propose a new paradigm for studying feature learning and the resulting benefits in generalizability. After reviewing the neural tangent kernel (NTK) theory and recent results in kernel regression, which address the generalization issue of sufficiently wide neural networks, we examine limitations and implications of the fixed kernel theory (as the NTK theory) and review recent theoretical advancements in feature learning. Moving beyond the fixed kernel/feature theory, we consider neural networks as adaptive feature models. Finally, we propose an over-parameterized Gaussian sequence model as a prototype model to study the feature learning characteristics of neural networks.

Figures

Figures reproduced from arXiv: 2412.18756 by the authors.

**Figure 2.** Best convergence rates of KRR and corresponding minimax lower rate (w.r.t. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Phase diagram about the consistency and the optimality of kernel interpolation. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: The inclusion relationships of the adaptive feature theories, neural networks theories and kernel regression [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

**Figure 5.** Figure 5: x-axis: number of training iterations; y-axis: the percentage of the first p projections: Pp j=1 f 2 j / Pm j=1 f 2 j , p = 1, 100, 300, m = 500. The projections fj concentrate on top eigenspaces as the training proceeds. 4.2 Relation between kernel regression and the …

**Figure 6.** Figure 6: x-axis: number of training iterations; y-axis: the percentage of the first p projections: Pp j=1 f 2 j,t/ PN j=1 f 2 j,t, p = 10, 100, 300, N = 500. The projections fj concentrate on top eigenspaces as the training proceeds. where the noise {ξj}∞ j=1 are ϵ 2 -sub-Gauss…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Breaking the Curse with BAND: Nonparametric Distribution Estimation in High Dimensions
stat.ML 2026-07 conditional novelty 6.0 of 10

Sparse Bayesian-network factorization plus sparsity-aware regression yields polynomial TV rates for high-dimensional mixed-type distribution estimation, beating classical histogram rates under sparsity.

Reference graph

Works this paper leans on

28 extracted references · 7 canonical work pages · cited by 1 Pith paper

[5]

Asymptotics of feature learning in two-layer networks after one gradient-step

Hugo Cui, Luca Pesce, Yatin Dandi, Florent Krzakala, Yue M Lu, Lenka Zdeborová, and Bruno Loureiro. Asymptotics of feature learning in two-layer networks after one gradient-step. arXiv preprint arXiv:2402.04980,

arXiv
[8]

A geometrical analysis of kernel ridge regression and its applications

Georgios Gavrilopoulos, Guillaume Lecué, and Zong Shang. A geometrical analysis of kernel ridge regression and its applications. arXiv preprint arXiv:2404.07709,

arXiv
[9]

Disentangling feature and lazy training in deep neural networks

Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020(11):113301,

work page 2020
[10]

URL https: //doi.org/10.1214/20-AOS1990

doi: 10.1214/20-AOS1990. URL https: //doi.org/10.1214/20-AOS1990. 21 Nikhil Ghosh, Song Mei, and Bin Yu. The three stages of learning dynamics in high-dimensional kernel methods. In International Conference on Learning Representations,

doi:10.1214/20-aos1990
[11]

Finite depth and width corrections to the neural tangent kernel

Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. arXiv preprint arXiv:1909.05989,

arXiv 1909
[12]

URL https://doi.org/10.1214/21-AOS2133

doi: 10.1214/21-AOS2133. URL https://doi.org/10.1214/21-AOS2133. Hong Hu and Yue M Lu. Sharp asymptotics of kernel ridge regression beyond the linear regime. arXiv preprint arXiv:2205.06798, 2022a. Hong Hu and Yue M Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory, 69(3):1932–1964, 2022b. We...

arXiv
[16]

Improving Adaptivity via Over-Parameterization in Sequence Models

Yicheng Li and Qian Lin. Improving adaptivity via over-parameterization in sequence models. arXiv preprint arXiv:2409.00894,

work page Pith review arXiv
[18]

URL https://doi.org/10.1214/19-AOS1849

doi: 10.1214/19-AOS1849. URL https://doi.org/10.1214/19-AOS1849. Tengyuan Liang, Alexander Rakhlin, and Xiyu Zhai. On the multiple descent of minimum-norm interpolants and restricted lower isometry of kernels. In Conference on Learning Theory, pages 2683–2711. PMLR,

doi:10.1214/19-aos1849

Show all 28 references

[19]

Optimal rate of kernel regression in large dimensions

Weihao Lu, Haobo Zhang, Yicheng Li, Manyun Xu, and Qian Lin. Optimal rate of kernel regression in large dimensions. arXiv preprint arXiv:2309.04268,

arXiv
[20]

Benign, tempered, or catastrophic: A taxonomy of overfitting

Neil Mallinar, James B Simon, Amirhesam Abedsoltan, Parthe Pandit, Mikhail Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: A taxonomy of overfitting. arXiv preprint arXiv:2207.06569,

arXiv
[21]

doi: https://doi.org/10.1016/j.acha.2021.12.003

ISSN 1063-5203. doi: https://doi.org/10.1016/j.acha.2021.12.003. URL https://www.sciencedirect.com/ science/article/pii/S1063520321001044. Theodor Misiakiewicz. Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge r...

2021 arXiv
[22]

A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and gcv estimator

Theodor Misiakiewicz and Basil Saeed. A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and gcv estimator. arXiv preprint arXiv:2403.08938,

arXiv
[23]

A theory of non-linear feature learning with one gradient step in two-layer neural networks

Behrad Moniri, Donghwan Lee, Hamed Hassani, and Edgar Dobriban. A theory of non-linear feature learning with one gradient step in two-layer neural networks. arXiv preprint arXiv:2310.07891,

arXiv
[24]

Generalization guarantees for neural networks via harnessing the low-rank structure of the jacobian

Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi. Generalization guarantees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint arXiv:1906.05392,

1906 arXiv
[26]

A theory of neural tangent kernel alignment and its influence on training

Haozhe Shan and Blake Bordelon. A theory of neural tangent kernel alignment and its influence on training. arXiv preprint arXiv:2105.14301,

arXiv
[28]

Feature learning in infinite-width neural networks

Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14522,

2011 arXiv
[29]

On the optimality of misspecified spectral algorithms

Haobo Zhang, Yicheng Li, and Qian Lin. On the optimality of misspecified spectral algorithms. Journal of Machine Learning Research, 25(188):1–50, 2024a. Haobo Zhang, Yicheng Li, Weihao Lu, and Qian Lin. Optimal rates of kernel ridge regression under source condition in large d...

arXiv
[1996]

Jianqing Fan, Cong Ma, and Yiqiao Zhong

doi: 10.1017/CBO9780511662201. Jianqing Fan, Cong Ma, and Yiqiao Zhong. A selective overview of deep learning. Statistical science: a review journal of the Institute of Mathematical Statistics, 36(2):264,

doi
[2001]

Neural spectrum alignment: Empirical study

Dmitry Kopitkov and Vadim Indelman. Neural spectrum alignment: Empirical study. In Artificial Neural Networks and Machine Learning–ICANN 2020: 29th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 15–18, 2020, Proceedings, Part II 29, pag...

2020
[2007]

On the inconsistency of kernel ridgeless regression in fixed dimensions

Daniel Beaglehole, Mikhail Belkin, and Parthe Pandit. On the inconsistency of kernel ridgeless regression in fixed dimensions. SIAM Journal on Mathematics of Data Science, 5(4):854–872, 2023a. Daniel Beaglehole, Adityanarayanan Radhakrishnan, Parthe Pandit, and Mikhail Belkin....

arXiv
[2009]

A survey on statistical theory of deep learning: Approximation, training dynamics, and generative models

Namjoon Suh and Guang Cheng. A survey on statistical theory of deep learning: Approximation, training dynamics, and generative models. arXiv preprint arXiv:2401.07187,

arXiv
[2010]

URL https://doi.org/10.1214/08-AOS648

doi: 10.1214/08-AOS648. URL https://doi.org/10.1214/08-AOS648. Dmitry Kobak, Jonathan Lomond, and Benoit Sanchez. The optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization. Journal of Machine Learning Resear...

doi
[2018]

Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn features

Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn features. arXiv preprint arXiv:2212.13881,

arXiv
[2019]

Generalization ability of wide neural networks on R

Jianfa Lai, Manyun Xu, Rui Chen, and Qian Lin. Generalization ability of wide neural networks on R. arXiv preprint arXiv:2302.05933,

arXiv
[2021]

Generalization in kernel regression under realistic assumptions

Daniel Barzilai and Ohad Shamir. Generalization in kernel regression under realistic assumptions. arXiv preprint arXiv:2312.15995,

arXiv
[2022]

Learning two-layer neural networks, one (giant) step at a time

Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic Stephan. Learning two-layer neural networks, one (giant) step at a time. arXiv preprint arXiv:2305.18270,

arXiv
[2023]

Ridgeless

ISSN 1464-3510. doi: 10.1093/biomet/asad048. URL https://doi.org/10.1093/biomet/asad048. Yicheng Li, Weiye Gan, Zuoqiang Shi, and Qian Lin. Generalization error curves for analytic spectral algorithms under power-law decay. arXiv preprint arXiv:2401.01599, 2024a. Yicheng Li, Q...

arXiv
[2024]

Tin Sum Cheng, Aurelien Lucchi, Anastasis Kratsios, and David Belius

URL https://arxiv.org/pdf/2410.05626. Tin Sum Cheng, Aurelien Lucchi, Anastasis Kratsios, and David Belius. Characterizing overfitting in kernel ridgeless regression through the eigenspectrum. arXiv preprint arXiv:2402.01297,

arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [5]

Asymptotics of feature learning in two-layer networks after one gradient-step

Hugo Cui, Luca Pesce, Yatin Dandi, Florent Krzakala, Yue M Lu, Lenka Zdeborová, and Bruno Loureiro. Asymptotics of feature learning in two-layer networks after one gradient-step. arXiv preprint arXiv:2402.04980,

arXiv

[2] [8]

A geometrical analysis of kernel ridge regression and its applications

Georgios Gavrilopoulos, Guillaume Lecué, and Zong Shang. A geometrical analysis of kernel ridge regression and its applications. arXiv preprint arXiv:2404.07709,

arXiv

[3] [9]

Disentangling feature and lazy training in deep neural networks

Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020(11):113301,

work page 2020

[4] [10]

URL https: //doi.org/10.1214/20-AOS1990

doi: 10.1214/20-AOS1990. URL https: //doi.org/10.1214/20-AOS1990. 21 Nikhil Ghosh, Song Mei, and Bin Yu. The three stages of learning dynamics in high-dimensional kernel methods. In International Conference on Learning Representations,

doi:10.1214/20-aos1990

[5] [11]

Finite depth and width corrections to the neural tangent kernel

Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. arXiv preprint arXiv:1909.05989,

arXiv 1909

[6] [12]

URL https://doi.org/10.1214/21-AOS2133

doi: 10.1214/21-AOS2133. URL https://doi.org/10.1214/21-AOS2133. Hong Hu and Yue M Lu. Sharp asymptotics of kernel ridge regression beyond the linear regime. arXiv preprint arXiv:2205.06798, 2022a. Hong Hu and Yue M Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory, 69(3):1932–1964, 2022b. We...

arXiv

[7] [16]

Improving Adaptivity via Over-Parameterization in Sequence Models

Yicheng Li and Qian Lin. Improving adaptivity via over-parameterization in sequence models. arXiv preprint arXiv:2409.00894,

work page Pith review arXiv

[8] [18]

URL https://doi.org/10.1214/19-AOS1849

doi: 10.1214/19-AOS1849. URL https://doi.org/10.1214/19-AOS1849. Tengyuan Liang, Alexander Rakhlin, and Xiyu Zhai. On the multiple descent of minimum-norm interpolants and restricted lower isometry of kernels. In Conference on Learning Theory, pages 2683–2711. PMLR,

doi:10.1214/19-aos1849