REVIEW 3 major objections 5 minor 17 references
On the Multiple Descent of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Multiple descent: kernel interpolant risk rises and falls as n grows
desk verdict A promising framework with a real gap: the key invertibility bound in Section 5 ignores the kernel's constant term, so the main multiple-descent theorem is unproven as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a restricted lower isometry property for the empirical kernel matrix on the filtration of polynomial eigenspaces of the population covariance operator. The paper proves that, with probability at least $1-e^{-\Omega(n/d^\iota)}$, the degree-$\iota$ truncated kernel matrix $K^{[\le\iota]}$ has all its $\binom{\iota+d}{\iota}$ nonzero eigenvalues at least $C' d^{-\iota}$. The proof represents the kernel matrix as $\Phi\Phi^\top$ with polynomial features, changes basis using Gram-Schmidt orthogonal polynomials so the features become weakly correlated, applies the small-ball method (Paley-Zygmund plus a covering net) to control the smallest eigenvalue, and then splits the variance of the interpolant into a low-degree part controlled by this isometry and a high-degree Taylor tail that contributes the $n/d^{\iota+1}$ term.
What would settle it
For a kernel whose Taylor coefficients vanish after degree $\iota$ (an exact polynomial kernel), the theorem predicts risk of order $d^\iota/n$ with no second peak; observing a second rise in risk near $n\asymp d^{\iota+1}$ for such a kernel would falsify the proposed mechanism.
Extended reading notes
Core claim
For any integer $\iota\ge1$, under product distributions on the coordinates, a smooth inner-product kernel $k(x,z)=h(x^\top z/d)$ with nonnegative Taylor coefficients, and a target function lying in the RKHS, the risk of the minimum-norm interpolant satisfies $$\mathbb{E}[\|\hat f-f_*\|^2_{P_X}\mid X]\le C\left(\frac{d^\iota}{n}+\frac{n}{$d^{{\iota+1}}$}\right)\asymp $n^{{-\beta}}$,\quad \$\beta$=\min\{(\iota+1)\$\alpha$-1,1-\iota\$\alpha$\},$$ with high probability on the design, whenever $\alpha\in[1/(\iota+1),1/\iota)$. The bound is minimized at $d=n^{1/(\iota+1/2)}$ with rate $n^{-1/(2\iota+1)}$, and it degenerates near $d=n^{1/\iota}$, producing peaks. The same variance bound is shown to hold for polynomial kernels without the second term, and for neural-tangent-type kernels up to logarithmic factors.
Load-bearing premise
The peaks rely on the assumption that, besides the first $\iota$ Taylor coefficients being positive, at least one Taylor coefficient beyond degree $2\iota+3$ is positive; if the kernel is a finite polynomial of degree $\iota$, the term that creates the peaks vanishes and only monotone decay remains.
Editorial extensions
If this is right
- At the valley scaling $d=n^{1/(\iota+1/2)}$, the risk bound improves at rate $n^{-1/(2\iota+1)}$, which slows as $\iota$ grows.
- At the peak scalings $d=n^{1/\iota}$, the upper bound is of constant order, so the rate guarantee disappears exactly where the experiments locate peaks.
- For a fixed dimension $d$, the bound alternates between decreasing and increasing in $n$: larger sample sizes can push the interpolant from a valley toward a peak.
- If the kernel is a polynomial of exact degree $\iota$, only the $d^\iota/n$ term remains, so the multiple-descent shape disappears.
- Through the neural tangent kernel, the same variance bound gives estimation guarantees for wide randomly initialized neural networks trained by gradient flow.
Reading between the lines
- The proof attributes the height of each peak to the Taylor tail above degree $\iota$; a testable extension is that truncating or shrinking coefficients $\alpha_k$ for $k>\iota$ should lower the peaks even when the low-degree coefficients are unchanged.
- If the upper bound is tight, kernel interpolation in high dimensions violates the usual monotone bias-variance picture on long stretches of the scaling axis, not just at a single double-descent point.
- The restricted lower isometry result concerns random kernel matrices alone and could be reused for other spectral questions, such as uncertainty quantification or the behavior of kernel spectral algorithms under interpolation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies the out-of-sample risk of minimum-norm interpolants in reproducing kernel Hilbert spaces when the input dimension d and sample size n satisfy d = n^α, α ∈ (0,1). The main object is a restricted lower isometry property for kernel matrices restricted to low-degree polynomial subspaces (Proposition 1). This is used to prove an upper bound on the variance term (Theorem 2) and on the bias term (Theorem 3), leading to the informal Theorem 1 bound risk ≲ d^ι/n + n/d^{ι+1}, which has a multiple-descent shape as a function of α. The authors also extend the result to neural-tangent-type kernels and provide simulation evidence that the predicted peak locations are observed empirically.
Significance. If the central claims are correct, the paper makes a genuinely interesting contribution to the interpolation literature: it provides a rigorous mechanism for multiple descent in kernel ridgeless regression, identifies the critical scalings α = 1/i, and gives novel generalization guarantees for wide neural networks through the NTK connection. The restricted lower isometry of kernel matrices on a filtration of polynomial eigenspaces is a technically useful tool that may be of independent interest. The paper is also careful in stating that only upper bounds are proven and that lower bounds exhibiting multiple descent remain open. The proof is derived from first principles with no fitted constants, and the predicted peak locations are compared with simulation rather than used to fit the theory, which are strengths. However, as detailed below, several load-bearing steps in the proof are currently not established as written, so the central result requires substantial repair.
major comments (3)
- [Section 5, Eq. (13)] The bound ||B||_{1→1} ≤ O(n(√log n/√d)^{ι'}) is not valid under the paper's own assumptions. For i ≠ j, the off-diagonal entries of nK[ι'] equal Σ_{l=0}^{ι'} α_l (x_iᵀx_j/d)^l, and when α0 > 0 — which Proposition 1 explicitly requires — the leading term is α0, a positive constant, not O((√log n/√d)^{ι'}). Hence ||B||_{1→1} is of order n·α0, and the displayed inequality cannot imply nK ≽ cI_n. Since the global lower bound nK ≻ cI is used in Eq. (17) to obtain the tail term n/d^{ι+1}, the proof of Theorem 2(i) is incomplete. A rank-one decomposition nK = α0 J + (nK − α0 J) might repair the argument, but the paper does not supply it.
- [Theorem 2(i) and Proposition 1] There is a mismatch between the assumptions of Theorem 2(i) and those of Proposition 1. Theorem 2(i) assumes α1,...,αι > 0 and some α_{ι'} > 0, but Proposition 1 requires α0,...,αι0 > 0 to conclude that K[≤ι] has (ι+d choose ι) nonzero eigenvalues. If α0 = 0, the constant column of the feature matrix Φ in Eq. (23) is identically zero, so the rank assertion of Proposition 1 fails and the bound in Eq. (18) for the i = 0 term is unsupported. The proof therefore needs either an explicit assumption α0 > 0 in Theorem 2(i) or a separate treatment of the constant term.
- [Section 8.4, Step 3, Eq. (76)] The probability calculation for the covering argument contains a serious error. With L = exp(ι(n − d^ι log d)) n^{ι−1/2}, we have log(1/r) ≈ ι(n − d^ι log d), so the net-size term in Eq. (76) is exp(Ω(d^ι(n − d^ι log d))). For n ≫ d^ι log d this is not dominated by e^{−cn}; the claimed bound ≤ exp(−c'' n/d^ι) does not follow. This appears to be a typo — perhaps L should be exp(ι(n/d^ι − log d)) n^{ι−1/2} — but as written the small-ball covering argument does not establish Proposition 1's probability guarantee over the required regime.
minor comments (5)
- [Figure 2] The simulation figure shows no error bars or repeated-trial variability; adding them would strengthen the empirical claim that the multiple-peak shape is not an artifact of a single draw.
- [Theorem 3 statement] The phrase 'bounded in 𝓁2-sense: EX[C1(X)]2, EX[C2(X)]2 ≼ 1' is nonstandard and should be rewritten, e.g., as E_X[C1(X)^2] ≼ 1 and E_X[C2(X)^2] ≼ 1, to match the proof in Proposition 4.
- [Section 5, Eq. (18)] The step 'v⊤(K|Φ)^{-1}v ≤ v⊤(K[≤i])^+v' deserves a short justification; the preceding text sketches it, but the displayed inequality is not immediate because the pseudo-inverse of K[≤i] is taken on a subspace of the range of K[≤i].
- [Section 8.2, Lemma 4] The proof of Lemma 4 asserts uniqueness of b in Eq. (34), but it may be helpful to state explicitly that this follows from linear independence of the q_{r_1...r_d} basis; as written the existence of Λ is clear, while uniqueness is implicit.
- [Section 5, Eq. (12)] The factor involving Hölder's inequality is correct for symmetric matrices, but calling it Hölder's inequality is slightly unusual; Riesz–Thorin or simply the bound ||B||_2 ≤ sqrt(||B||_1||B||_∞) would be clearer.
Circularity Check
No significant circularity: the risk bound follows from an independent spectral analysis, and the self-citations are contextual rather than load-bearing.
full rationale
I found no step in which a claimed prediction or first-principles result reduces by construction to its own inputs. The main bound in Theorem 1 is stated conditional on Theorems 2 and 3, and those theorems are proved from the closed-form interpolant, a bias-variance decomposition, and the restricted lower isometry of the empirical kernel matrix. Proposition 1 is not assumed: Section 8 constructs the polynomial features, performs Gram-Schmidt, establishes the small-ball property, and gives a covering-net argument to lower-bound the eigenvalues. The constants are explicit, and no parameter is fitted to the predicted multiple-descent curve; the experimental peak locations are compared with the theory after the fact. The cited prior work by the authors is used for motivation and for the separate alpha=0 non-consistency result, not as an input to the new upper bound. The multiple-descent shape is a direct consequence of the two competing terms d^iota/n and n/d^{iota+1} in the proved variance bound, not a renaming of a known empirical pattern. The only questionable passage, Eq. (13), concerns the size of the off-diagonal block in the invertibility proof; that is a possible mathematical gap in the proof as written, not a circular reduction, so it does not raise the circularity score.
Assumptions & free parameters
assumptions (7)
- domain assumption Coordinate distribution P has zero mean, polynomial tail with exponent ν > ι0, and P(z∈S) < 1 for every finite set S (Assumption 1).
- domain assumption Kernel h is C∞ with Taylor coefficients α_i ≥ 0 and, for the variance theorem, α_1,...,α_ι > 0 and some α_{ι'} > 0 for ι' ≥ 2ι+3.
- domain assumption The condition P is sub-Gaussian holds for Theorem 2.
- domain assumption Target function admits f*(x) = ∫ k(x,z)ρ*(z)PX(dz) with ∫ ρ*^4 dPX ≤ C.
- domain assumption Data dimension satisfies d^ι log d = o(n) and n is sufficiently large.
- standard math Small-ball lower bounds via Paley-Zygmund and Hoeffding, as in Koltchinskii-Mendelson (2015) and Mendelson (2014).
- domain assumption Gradient flow on sufficiently wide randomly initialized networks converges to the minimum-norm NTK interpolant, per Du et al. (2018).
Cite this review
Pith. "Pith review of On the Multiple Descent of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels." pith.science (2026). https://pith.science/paper/RMTWZXCO
@misc{pith2026190810292,
author = {Pith},
title = {Pith review of: On the Multiple Descent of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels},
year = {2026},
howpublished = {\url{https://pith.science/paper/RMTWZXCO}},
note = {Machine review of arXiv:1908.10292}
}
abstract
We study the risk of minimum-norm interpolants of data in Reproducing Kernel Hilbert Spaces. Our upper bounds on the risk are of a multiple-descent shape for the various scalings of $d = n^{\alpha}$, $\alpha\in(0,1)$, for the input dimension $d$ and sample size $n$. Empirical evidence supports our finding that minimum-norm interpolants in RKHS can exhibit this unusual non-monotonicity in sample size; furthermore, locations of the peaks in our experiments match our theoretical predictions. Since gradient flow on appropriately initialized wide neural networks converges to a minimum-norm interpolant with respect to a certain kernel, our analysis also yields novel estimation and generalization guarantees for these over-parametrized models. At the heart of our analysis is a study of spectral properties of the random kernel matrix restricted to a filtration of eigen-spaces of the population covariance operator, and may be of independent interest.
Figures
Reference graph
Works this paper leans on
-
[1]
Benign overfitting in linear regression
Peter L Bartlett, Philip M Long, G \'a bor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. arXiv preprint arXiv:1906.11300, 2019
arXiv 1906
-
[2]
Reconciling modern machine learning and the bias-variance trade-off
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning and the bias-variance trade-off. arXiv preprint arXiv:1812.11118, 2018 a
arXiv 2018
-
[3]
Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
Mikhail Belkin, Daniel Hsu, and Partha Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. arXiv preprint arXiv:1806.05161, 2018 b
work page Pith review arXiv 2018
-
[4]
To understand deep learning we need to understand kernel learning
Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. arXiv preprint arXiv:1802.01396, 2018 c
arXiv 2018
-
[5]
Does data interpolation contradict statistical optimality?
Mikhail Belkin, Alexander Rakhlin, and Alexandre B Tsybakov. Does data interpolation contradict statistical optimality? arXiv preprint arXiv:1806.09471, 2018 d
work page Pith review arXiv 2018
-
[6]
Optimal rates for the regularized least-squares algorithm
Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7 0 (3): 0 331--368, 2007
2007
-
[7]
Gradient descent provably optimizes over-parameterized neural networks
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054, 2018
arXiv 2018
-
[8]
Does learning require memorization? a short tale about a long tail
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. arXiv preprint arXiv:1906.05271, 2019
arXiv 1906
Show all 17 references
-
[9]
Linearized two-layers neural networks in high dimension
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension. arXiv preprint arXiv:1904.12191, 2019
1904 arXiv
-
[10]
Surprises in high-dimensional ridgeless least squares interpolation
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. arXiv preprint arXiv:1903.08560, 2019
1903 arXiv
-
[11]
Neural tangent kernel: Convergence and generalization in neural networks
Arthur Jacot, Franck Gabriel, and Cl \'e ment Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pages 8571--8580, 2018
2018
-
[12]
Bounding the smallest singular value of a random matrix without concentration
Vladimir Koltchinskii and Shahar Mendelson. Bounding the smallest singular value of a random matrix without concentration. International Mathematics Research Notices, 2015 0 (23): 0 12991--13008, 2015
2015
-
[13]
ridgeless
Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel" ridgeless" regression can generalize. The Annals of Statistics, to appear, 2018
2018
-
[14]
The generalization error of random features regression: Precise asymptotics and double descent curve
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and double descent curve. arXiv preprint arXiv:1908.05355, 2019
1908 arXiv
-
[15]
Learning without concentration
Shahar Mendelson. Learning without concentration. In Conference on Learning Theory, pages 25--39, 2014
2014
-
[16]
Consistency of interpolation with laplace kernels is a high-dimensional phenomenon
Alexander Rakhlin and Xiyu Zhai. Consistency of interpolation with laplace kernels is a high-dimensional phenomenon. arXiv preprint arXiv:1812.11167, 2018
2018 arXiv
-
[17]
Tsybakov
Alexander Rakhlin, Karthik Sridharan, and Alexandre B. Tsybakov. Empirical entropy, minimax regret and minimax risk. Bernoulli, 23 0 (2): 0 789--824, May 2017. doi:10.3150/14-bej679. URL https://doi.org/10.3150/14-bej679
2017 doi
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.