Pith. sign in

REVIEW 3 major objections 4 minor 9 references

Revisiting the Neural Tangent Kernel: the role of large width and depth

T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that the kernel-regression output of infinitely wide ReLU networks converges to a bounded, non-trivial limit as depth grows, even though the normalized neural tangent kernel itself collapses toward a matrix of ones.

desk verdict The kernel-recursion half is clean, but Theorem 3's rough-path proof rests on a false smoothness claim and an unproven determinant ratio, so the central claim is not established. read the letter →

arxiv 2511.07272 v2 pith:6UAPCI67 submitted 2025-11-10 cs.LG stat.ML

William St-Arnaud , Margarida Carvalho , Golnoosh Farnadi This is my paper
classification cs.LGstat.ML MSC 60L2046E2268T05
keywords neuraltangentkerneldeepReLUnetworksinfinitewidthlazytrainingregressionroughpaththeorysphericaldatadepthscaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper re-examines a standard story about very wide neural networks: that as depth increases, the neural tangent kernel (NTK) degenerates into an all-ones matrix and the closed-form approximation to the trained output becomes trivial and data-independent. It claims this conclusion is wrong in the deterministic infinite-width regime when depth grows more slowly than width. For ReLU networks with data on a sphere, the normalized NTK indeed converges entrywise to one, but the key kernel-regression vector kappa_x^T kappa^-1 converges to a bounded, meaningful limit; the output does not collapse. The argument uses rough path theory and identifies a short list of kernel properties under which the same limiting behavior holds. A reader should care because depth is the dividing line where kernel descriptions of neural networks were thought to fail.

What carries the argument

The central object is the normalized NTK, whose recursion is driven by the function h(z) = z arcsin(z)/pi + sqrt(1-z^2)/pi + z/2. The proof constructs a smooth interpolation path A^(L+1)_n(t) between consecutive normalized Gram matrices using a bump function psi_D with scale D equal to the ratio of their determinants, then applies Cramer's rule and the Ito-Lyons map of rough path theory. The normalization removes the exponential decay of the raw kernel, allowing the inverse to have a non-trivial limit when the driving path converges to zero.

What would settle it

Take a two-point dataset on the sphere with non-colinear points and compute D_L = det(normalizedNTK^(L+1)(XX^T)) / det(normalizedNTK^(L)(XX^T)) for increasing L using the closed-form ReLU kernel. If D_L fails to tend to 0, or tends to 0 too slowly for the interpolation derivative to stay controlled, then the rough-path convergence of u^(L) is unproved.

Watch

Extended reading notes

Core claim

For a fixed spherical dataset X and input x, define u^(L)(x) = normalizedNTK^(L)(x^T X) * (normalizedNTK^(L)(X^T X))^-1. Theorem 3 claims that when every hidden-layer width goes to infinity and depth L = o(width), u^(L)(x) converges as L goes to infinity to a limit bounded by C(x)1_n, with C continuous and hence bounded on the sphere. Consequently, although the normalized NTK tends to a matrix of ones, the kernel-regression solution does not become trivial. The paper also supports an experimental and theoretical distinction: deviations of individual hidden-neuron activations vanish while the aggregate norm of deviations does not, so activations of the last hidden layer do not stay constant.

Load-bearing premise

Everything rests on the unstated requirement that the determinant ratio between consecutive normalized NTK Gram matrices tends to zero as depth grows; if this ratio does not vanish, the convergence claim collapses.

Editorial extensions

If this is right

  • Infinite-width ReLU networks trained in the lazy regime have a well-defined, non-trivial NTK-regression output at arbitrarily large depths, provided depth grows slower than width.
  • The all-ones limit of the normalized kernel is not fatal for the closed-form solution: normalization cancels in kappa_x^T kappa^-1, and the inverses align to a bounded limit.
  • Last-hidden-layer activations fluctuate collectively even though pointwise activation deviations vanish, complicating a simple frozen-features picture.
  • The same convergence holds for any kernel sequence on the sphere that is diagonally dominant, eventually positive definite, and has determinants tending to zero, for example iterated logistic-like kernels.
  • For data outside the sphere, the results extend by projecting to the sphere when no two points are colinear, or via stereographic projection otherwise.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The proof's real requirement is that the determinant ratio D_L = det(normalizedNTK^(L+1)(XX^T)) / det(normalizedNTK^(L)(XX^T)) tends to zero as depth grows; this is the natural quantity to test, and no spectral condition guaranteeing it is stated.
  • Because the paper's experiments show the kernel's convergence to one is extremely slow, finite-depth networks may need impractically large depth before the limiting behavior is observable.
  • The listed kernel conditions suggest a broader principle: any kernel family whose normalized Gram matrices become degenerate in determinant while keeping well-behaved principal minors can yield non-trivial kernel-regression limits.
  • If the rough-path argument holds, it hints at a deterministic pointwise NTK limit at large depth that could connect lazy and mean-field descriptions of very deep networks, though such a connection is not established here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies the infinite-width ReLU neural tangent kernel (NTK) as the depth L grows, with L ∈ o(width). It proves convergence of the correlation ρ^{(L)} to 1 (Lemma 1), gives a recurrence for the normalized kernel \barΘ^{(L)}∞ (Proposition 4), and states convergence of the normalized kernel to the all-ones matrix (Theorem 2). The central result (Theorem 3) claims that the closed-form kernel-regression weight vector u^{(L)} = \barΘ^{(L)}∞(x^⊤X)(\barΘ^{(L)}∞(XX^⊤))^{-1} converges to a bounded, non-trivial limit as L→∞, via a rough-path argument built on a smooth interpolation ψ_D and a determinant ratio D. Experiments with L=1,...,10 and n0=128 are used to illustrate convergence rates.

Significance. If Theorem 3 were correct, it would provide a meaningful extension of NTK theory to large depth in the deterministic regime, contradicting the intuition that the NTK solution becomes trivial as depth grows. The paper's explicit recursions (Proposition 4) and the proposed generalization criteria (Section 6) are potentially useful. There are no fitted parameters, and the proof attempts a rigorous rough-path framework. However, the central convergence claim is not supported by the proof as written: a key smoothness property is false and a critical spectral premise is unproven. The result may be true, but this manuscript does not establish it.

major comments (3)
  1. [Proposition 5, property (4) and Theorem 3 proof] Property (4) of ψ_d is false as stated. A direct computation gives ψ_d'(0)=1/(2d), which diverges as d→0. The proof of Theorem 3 uses property (4) to conclude that the Cramer's-rule ratios v^{(i,j)} vanish in 1-variation, and the interpolation A^{(L+1)}_n(t) crosses z=0 at t=1/2 exactly when 2t−1=0. At that point the derivative of ψ_D is unbounded as D→0, so the term Z_A = −A' diag(u) in Eq. (5) is not controlled. Hence the rough-path argument does not establish convergence of u^{(L)}.
  2. [Theorem 3 proof and Section 6] The proof requires D = det(\barΘ^{(L+1)}∞(XX^⊤))/det(\barΘ^{(L)}∞(XX^⊤)) to tend to 0, but this is never proven. Since Theorem 2 states \barΘ^{(L)}∞ → J (the all-ones matrix), the determinant decays like c δ_L^{n−1}, where δ_L is the spectral gap. Nothing in the paper shows δ_{L+1}/δ_L → 0; for polynomial decay of δ_L, D → 1, not 0. Section 6 merely lists determinant decay as a sufficient condition for other kernels; it is not verified for the ReLU NTK. Without D→0, the inequalities involving ψ_D in the proof fail, so the convergence of u^{(L)} is unproven.
  3. [Theorem 3 statement] The theorem is not a well-posed convergence statement. It asserts existence of a sequence of paths v^{(L)} such that the solution u^{(L)} satisfies u'^{(L)}_i=0 for each L and yet u^{(L)}_i(1) equals the L-dependent NTK expression. If u'=0, then u^{(L)}(1)=u^{(L)}(0); no limiting object or initial-condition convergence is specified. The sentence '\barΘ^{(L)}∞(x^⊤X)(\barΘ∞(X^⊤X))^{-1} < C(x)1_n ... ∈ O(n)' is also dimensionally and notationally unclear. This makes the claim of convergence difficult to interpret, let alone verify.
minor comments (4)
  1. [Definition 6 / Proposition 5] The notation in property (4), 'lim_{d→0+} d^k/dz^k ψ_d(z) dj = 0', is malformed; it should specify which derivative with respect to d is taken. This ambiguity is not merely cosmetic because the property is used to justify the vanishing of the Cramer's-rule ratios.
  2. [Section 3 and Theorem 3] Notation for the data matrix is inconsistent: the theorem writes X^⊤X while the surrounding text and Definition 4 use XX^⊤. This should be unified.
  3. [Abstract and Section 6] The abstract mentions 'properly scaling the depth and stopping time,' but the proof of Theorem 3 does not involve a stopping time; Proposition 6 is only sketched and not connected to the main theorem. The experiments use only L=1,...,10 and n0=128, so the claim of 'large dataset sizes' is not supported.
  4. [Theorem 2 proof] The proof of Theorem 2 contains informal approximations (e.g., '(1+x)^α ≈ e^{αx}') without rigorous error bounds. While Theorem 2 itself is plausible, the derivation as written is too loose to serve as a foundation for the spectral assumptions needed in Theorem 3.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: Theorem 3's issues are unproved spectral-rate assumptions, not self-referential derivation.

full rationale

I looked for the enumerated circularity patterns. There are no fitted parameters that are later re-predicted: the normalization in Definition 4 cancels in the closed-form expression and is not fitted to data. The only self-referential pointer is Table 1's '(Our work 2025)', which is a summary of contributions, not a load-bearing argument. The central derivation (Proposition 4 and Theorem 2) uses standard, external NTK recursion results (Jacot et al., Arora et al.) plus an elementary fixed-point argument; the rough-path tool (Lyons 1998) is also external. The closest candidate is Theorem 3's proof, which uses an interpolation with ψ_D and invokes the Universal Limit Theorem. The proof is mathematically fragile: it requires determinant ratio D to tend to 0 and property (4) of ψ_D to hold, but D→0 is never established and ψ'_D(0) ~ 1/(2D) actually diverges as D→0; moreover, the initial condition of the interpolated system is itself a kernel-inverse quantity of the same type whose convergence the theorem asserts. These are genuine correctness and rigor gaps, but they are not cases where the conclusion is equivalent to the inputs by construction or where a fitted value is renamed as a prediction. The paper's claims are not reduced to a self-citation chain, and no ansatz is smuggled in via prior work by the same authors. I therefore find no significant circularity; the concerns raised are analytical/spectral, not circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 2 invented entities

The paper introduces no fitted free parameters. The load-bearing inputs are all structural: the deterministic infinite-width regime, kernel invertibility on the sphere, the behavior of the newly introduced ψ_d function (which fails at the transition point as rendered), and the applicability of the Universal Limit Theorem to a sequence whose initial conditions are the claimed limit. The η^{(L)} kernel family is an illustration whose required properties are only sketched.

assumptions (5)
  • domain assumption Deterministic infinite-width NTK limit (Jacot et al. 2018, Theorem 1): Θ^{(L)} → Θ^{(L)}∞ as widths → ∞ with L ∈ o(min width)
    The entire analysis operates inside the deterministic kernel regime; the depth scaling L ∈ o(min_l n_l) is asserted (Section 5) to keep the NTK deterministic, separating from Hanin–Nica (2020) where L/n → ∞ gives stochasticity. If this premise fails, Theorems 2 and 3 are not about the network's actual kernel.
  • domain assumption PSD/invertibility of the depth-L limiting kernel on the sphere for L ≥ 2 under the stated dataset conditions (cases a–c, Section 4)
    Cramer's rule in Theorem 3's proof requires det Θ^{(L)} > 0; Proposition 3 requires κ invertible. The paper asserts invertibility for sphere data with no colinear pairs; the main proof's inequalities rely on determinants lying in (0,1).
  • ad hoc to paper Property (4) of ψ_d: all derivatives vanish in the d→0 limit
    Introduced for Theorem 3. As rendered it is false at z=0 (ψ_d′(0) = −1/(2d) → −∞); the proof needs ψ_D′·(Θ^{(L+1)}−Θ^{(L)}) → 0 at t=1/2, which requires the unproven premise D = det Θ^{(L+1)}/det Θ^{(L)} → 0 together with a rate. This is the load-bearing gap.
  • ad hoc to paper Lyons' Universal Limit Theorem applies to the sequence of interpolated systems with the same initial data as the claimed limit
    The RDE step takes as input convergence of the initial conditions u^{(L)}(0) = κx^{(L)}(κ^{(L)})^{-1} — the very quantity the theorem claims converges. Without a priori convergence of the initial data, ULT only yields a statement conditional on it.
  • domain assumption Canonical projection preserves the analysis for non-compact domains (no colinear pairs)
    Extension in Sections 5 and 7 to general R^{n0} relies on canonical projection to the sphere; datasets with colinear pairs require stereographic projection into one extra dimension (Section 4, footnote 2).
invented entities (2)
  • ψ_d smoothing path (Definition 6)
    purpose: Interpolates A(t) between Θ^{(L)} and Θ^{(L+1)} to run the rough-path argument in Theorem 3
    Technical device; its regularity properties near z=0 control the whole proof but are only asserted (property (4)), and fail at z=0 as rendered.
  • η^{(L)} sigmoid-squared kernel family (Section 6 / Proposition 7)
    purpose: Example family satisfying the generalization criteria so Theorem 3's technique extends beyond the ReLU NTK
    PSD claim on the sphere is a proof sketch via Gneiting (2013) coefficient positivity; convergence to β>0 follows from contraction of h, but the det → 0 requirement for Theorem 3's machinery is not checked for η.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revisiting the Neural Tangent Kernel: the role of large width and depth." pith.science (2026). https://pith.science/paper/6UAPCI67

@misc{pith2026251107272,
  author       = {Pith},
  title        = {Pith review of: Revisiting the Neural Tangent Kernel: the role of large width and depth},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6UAPCI67}},
  note         = {Machine review of arXiv:2511.07272}
}
read the original abstract

Overparameterized fully-connected neural networks have been shown to behave like kernel models when trained with gradient descent, assuming standard scaling conditions on the width, the learning rate, and the parameter initialization. In the limit of infinitely large widths and infinitesimal learning rate, the obtained kernel provides a description of the learned model's output via a closed-form solution dependent on the architecture and the activation function. The Neural Tangent Kernel, central to this description, remains constant throughout training, a phenomenon that is referred to as ``lazy training'' or within the ``lazy regime''. Prior works show that the ``lazy regime'' leads to non-varying hidden neuron activations in infinitely-wide networks. Moreover, as infinitely-wide networks increase in depth, the Neural Tangent Kernel induces a closed-form solution that is data-independent, hence trivial. The Neural Tangent Kernel seemingly fails to describe the complexity of overparameterized neural networks on two distinct axes: large widths and large depths. In this work, we challenge these two conclusions and open the door to re-evaluating the Neural Tangent Kernel's role in describing the output of overparameterized neural networks. Specifically, we show experimentally that while deviations in the activations of individual hidden neurons vanish, the aggregate norm of these deviations does not. We support this finding with a theoretical result showing that the activations of the last hidden layer do not remain constant. Furthermore, we demonstrate that properly scaling the depth and stopping time in infinitely-wide ReLU networks yields a well-behaved, non-trivial output at large dataset sizes. We empirically evaluate the stability of this behavior on large datasets, and we describe the essential properties that enable the generalization of our results to other kernels.

Figures

Figures reproduced from arXiv: 2511.07272 by the authors.

Figure 1
Figure 1. Convergence rate of κ on X and point x. 7 Conclusion In this article, we provide a detailed analysis of behaviour of the deterministic kernel Θ (L) ∞ as L → ∞. We observe that the under the conditions of Jacot et al. (2018) and with L ∈ o(minl=1,...,L−1 nl), a fully-connected ReLU neural network approaches a limiting solution for any x given a fixed dataset X. While our results are for data in S n0−1 , we can extend… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

9 extracted references · 1 linked inside Pith

  1. [1]

    LetX , Ybe continuous maps∆ 0,1 →T (⌊p⌋) Rd and letX j (resp.Y j) denote the projection ofX(resp.Y) onto itsj-tensor component. Thep-variation metricd p is defined by dp (X,Y) = max j=1,...,⌊p⌋ sup 0=t0<t1<···<tm=1 m−1X i=0 ∥Xj −Y j∥ p j ! j p , where the supremum is taken over all finite partitions{0 =t 0 < t1 <· · ·< tm = 1} of [0,1]. Definition 10(Roug...

    1998

  2. [2]

    Theorem 2(Convergence of ¯Θ(L) ∞ ).For any x, x′ ∈S n0−1, the value ¯Θ(L) ∞ (x, x′)strictly increases to1asL→ ∞. Proof.We have a system describing¯Θ(L+1) ∞ , ¯Θ(L+1) ∞ 1 = L L+1 h′ ρ(L)(x, x′) 1 L+1 h ρ(L)(x, x′) 0 1 ¯Θ(L) ∞ (x, x′) 1 where we define the2 × 2matrix on the right-hand side to beA(L). For now, fixL and observe the product A(L+K) . . . A(L), ...

    1998

  3. [3]

    Proposition 6(closed-form for fτ; section 5 from Jacot et al

    We include a proof sketch for the reader, although we wish to underscore that this result is already known in the literature. Proposition 6(closed-form for fτ; section 5 from Jacot et al. (2018)).Givenτ <∞ and the spectrumΛ(κ), the outputf τ is given by fτ (x) =f 0(x) +κ ⊤ x κ−1 ¯Λ(y0 −y ∗), whereκ x, y0, y∗ are as in proposition 3, and ¯Λij = exp(−λiτ)−1...

    2018

  4. [6]

    Moreover, the kernelsη(L) are positive definite onSn0−1 and satisfyη (L)(x, x)≥η (L)(x1, x2)

    Proposition 7(Convergence of η(L)).The values η(L)(x, x′)converge to a unique β >0 for all x, x′ ∈S n0−1 as L→ ∞. Moreover, the kernelsη(L) are positive definite onSn0−1 and satisfyη (L)(x, x)≥η (L)(x1, x2). Proof sketch. As L→ ∞, all values converge to the same limit since the derivative ofh is strictly smaller than1on [−1,1]. The kernels η(L) also satis...

    2013

  5. [9]

    • Spectrum characterized via Hermite expan- sion (Nguyen et al., 2021; Murray et al., 2023; Li et al., 2024)

    15 Architecture Activation Results and relevant works Wide, fixed depth, fully-connected General • Convergence to NTK on the sphere and general do- main; Hessian is approximately zero (Jacot et al., 2018; Belkin, 2021; Liu et al., 2020; 2022). • Spectrum characterized via Hermite expan- sion (Nguyen et al., 2021; Murray et al., 2023; Li et al., 2024). • P...

    2018

  6. [2009]

    Tilmann Gneiting

    URL https://proceedings.neurips.cc/paper_files/paper/2009/file/ 5751ec3e9a4feab575962e78e006250d-Paper.pdf. Tilmann Gneiting. Strictly and non-strictly positive definite functions on spheres.Bernoulli, 19(4):1327 – 1349,

    2009

  7. [2013]

    URLhttps://doi.org/10.3150/ 12-BEJSP06

    doi: 10.3150/12-BEJSP06. URLhttps://doi.org/10.3150/ 12-BEJSP06. Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. In8th International Conference on Learning Representations, ICLR 2020,

    doi:10.3150/12-bejsp06 2020

  8. [2018]

    URL https://www.science.org/doi/abs/10.1126/science

    doi: 10.1126/science.aar6404. URL https://www.science.org/doi/abs/10.1126/science. aar6404. Russell Tsuchida, Fred Roosta, and Marcus Gallagher. Invariance of weight distributions in rectified mlps. InInternational Conference on Machine Learning, pp. 4995–5004. PMLR,

    doi:10.1126/science.aar6404

Show all 9 references
  1. [2021]

    Deep equals shallow for relu networks in kernel regimes

    Alberto Bietti and Francis Bach. Deep equals shallow for relu networks in kernel regimes. arXiv preprint arXiv:2009.14397,

    2009 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.