Pith. sign in

REVIEW 2 major objections 4 minor 21 references

The Double Descent Behavior in Two Layer Neural Network for Binary Classification

T0 review · 2 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A two-layer ReLU network trained by square loss has an asymptotic test-error curve that peaks exactly when the number of training samples equals the model dimension, then descends again.

desk verdict The CGMT machinery is real, but one dropped factor of r* in the square-loss fixed point invalidates every double-descent curve in the paper. read the letter →

arxiv 2504.19351 v1 pith:YT4DBPCC submitted 2025-04-27 stat.ML cs.LG

Chathurika S Abeykoon , Aleksandr Beknazaryan , Hailin Sang This is my paper · ORCID
classification stat.MLcs.LG MSC 68T0762H30
keywords doubledescentgeneralizationerrortwo-layerneuralnetworkReLUactivationbinaryclassificationConvexGaussianMin-MaxTheoremsquarelosshigh-dimensionalasymptotics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper derives a closed-form asymptotic formula for the test error of a two-layer ReLU network trained for binary classification, as a function of the ratio $\alpha = n/d$ between training samples and model dimension. Working in a teacher–student setup where labels come from two Gaussian clusters, the authors show that under low $\ell_2$ regularization the test-error curve has a U-shape in the over-parameterized regime, reaches its maximum exactly at $\alpha = 1$, and then decreases monotonically as more data become available. The derivation avoids simulating training: it uses the Convex Gaussian Min-Max Theorem to replace the empirical risk minimizer by the minimizer of an auxiliary optimization problem, whose solution collapses to a scalar fixed-point system. If the identification is right, the paper turns double descent from an empirical curiosity into a computable curve whose peak location, sweet spot, and regularization sensitivity can be read off directly.

What carries the argument

The central object is the CGMT lower bound $\omega_\lambda^{(d)}(r,s) = \lambda r/2 + (\alpha/n)\sum_{i=1}^n l(v_i)$, obtained by rewriting the ReLU risk through $\sigma(z)=(z+|z|)/2$, applying the Legendre transformation to the margin loss, and passing to the auxiliary Gaussian optimization problem. Minimizing this bound under $s^2\le r$ and sending $n,d\to\infty$ reduces the high-dimensional empirical risk minimization to the four scalar equations (5.26)–(5.29) for $r^*,s^*,b^*,\gamma^*$. This scalar fixed-point system is what carries the argument: it turns an ERM over $\mathbb{R}^d$ into a one-dimensional curve, and for square loss it closes explicitly in (8.27), giving $\gamma^*$ as the positive root of the quadratic in (8.21).

What would settle it

Simulate the teacher–student model for a fixed small regularization, say $d=200$ and $n=\alpha d$ with $\alpha$ ranging from 0.2 to 5, train the two-layer ReLU student with square loss by running an optimizer from many restarts (or a global solver), and compare the empirical test-error curve to (8.28). The central claim fails if the empirical peak does not occur at $\alpha\approx1$ or if the curve deviates systematically from the closed form beyond finite-sample fluctuations. A sharper check is to verify the Section 8.4 claim that $\lim_{n,d\to\infty}\sup_{s^2\le r}|\tilde L_\lambda(r,s)-\omega_\lambda^{(d)}(r,s)|=0$ using the boundedness of $\|u\|_2/d$; if that limit is nonzero for square loss, the lower-bound identification collapses.

Watch

Extended reading notes

Core claim

For the student model $f(x)=\sigma(x^{\mathsf T}\beta/\sqrt{d}+b)$ with ReLU activation, trained on data generated from the teacher model $x_i = \eta y_i/\sqrt{d} + \epsilon_i$, the paper claims that as $n,d\to\infty$ with $\alpha=n/d$ fixed, the limiting test error under square loss is $R^*(\hat\beta)=1-\rho_1\Phi((s^*+b^*)/\sqrt{r^*})-\rho_{-1}\Phi((s^*-b^*)/\sqrt{r^*})$, where $r^*,s^*,b^*$ solve the fixed-point system (8.27). The resulting curve exhibits ratio-wise double descent: error first falls, peaks at $\alpha=1$, then falls again as the model enters the under-parameterized region. Theorems 4.1 and 5.1 are the two load-bearing results, with (8.28) giving the explicit square-loss specialization.

Load-bearing premise

The load-bearing premise is that the auxiliary lower bound produced by the Convex Gaussian Min-Max Theorem converges to the true global training loss in the limit, so the minimizer of the easy problem can stand in for the empirical-risk minimizer. The proof of that identification is a sketch: it invokes boundedness of $\|u\|_2/d$ and sup-norm convergence without fully verifying all CGMT conditions.

Editorial extensions

If this is right

  • Under low regularization the test-error peak is located exactly at $\alpha=1$, the interpolation threshold where $n=d$; the best model lies in the under-parameterized region $\alpha>1$.
  • Raising the $\ell_2$ regularization coefficient $\lambda$ flattens the peak: for balanced classes with sufficiently large $\lambda$ the test error decreases monotonically, matching the classical idea that more data are always better.
  • For uneven class probabilities (such as $\rho_1=0.7$) strong regularization holds the test error flat until a large $\alpha$, then lets it decline; the paper recommends a moderate range $0.1\le\lambda\le1$ for best performance.
  • Because the fixed-point equations before specialization are derived for any margin-based convex loss, the same two-theorem framework would produce test-error curves for hinge or logistic loss by substituting the corresponding loss function.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The closed-form curve in (8.28) can be differentiated with respect to $\alpha$ and $\lambda$ to locate the sweet-spot minimum as a function of regularization, a quantity the paper does not isolate.
  • Since the derivation fixes one dimension $d$ and sweeps $n$, the ratio-wise curve can be read as a smoothed theoretical version of sample-wise double descent for this teacher–student model, though the paper does not make that connection.
  • The same CGMT argument should yield analogous fixed-point systems for other convex activations, but the paper does not test that extension, so it remains an open possibility rather than a claim.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper analyzes the generalization error of a two-layer neural network with a single ReLU hidden unit for binary classification, with data generated from a Gaussian mixture teacher model. The authors derive an exact asymptotic expression for the test error as a function of three quantities r, s, and b (Theorem 4.1). They then use the Convex Gaussian Min-Max Theorem (CGMT) to characterize the asymptotic minimizers of the regularized empirical risk, obtaining a system of fixed-point equations in Theorem 5.1. Specializing to square loss, they solve this system in Section 8.6 and use the resulting formulas to plot the test error against alpha = n/d in Section 6. The reported curves exhibit a double-descent peak at alpha = 1, which the authors interpret as ratio-wise double descent. The central claim of the paper is that these curves are exact consequences of the derived theory.

Significance. The paper's central claim is the existence of ratio-wise double descent for this two-layer ReLU classification model, with the test-error peak occurring exactly at alpha = 1. If the derivation were correct, the paper would provide a rigorous CGMT-based asymptotic analysis of double descent in a simple neural network, which would be a useful contribution to the theoretical double-descent literature. The derivation of Theorem 4.1 is clean and correct, and the manuscript provides reproducible R code for the figures. However, as detailed in the major comments, the fixed-point solution in Section 8.6 contains an algebraic error that invalidates the numerical results; the paper's own equations do not support the claimed peak at alpha = 1. Since the plotted curves are the primary evidence for the paper's main conclusion, the significance of the manuscript is critically undermined.

major comments (2)
  1. [Section 8.6, Eqs. (5.26) and (8.21)] The solution for gamma* in equation (8.21) does not follow from the paper's own equation (5.26). The limit computation in Section 8.6 correctly yields alpha * sqrt(r*) * [gamma* * sqrt(r*) / (1 + gamma*)] = 1 - 4 lambda gamma*, which simplifies to alpha * gamma* * r* / (1 + gamma*) = 1 - 4 lambda gamma*. Solving this quadratic for gamma* gives gamma* = [-(alpha r* + 4 lambda - 1) ± sqrt((alpha r* + 4 lambda - 1)^2 + 16 lambda)] / (8 lambda). The published equation (8.21) instead contains alpha in place of alpha r*, which is equivalent to silently setting r* = 1 with no justification. Direct substitution shows that the values in (8.27) do not solve (5.26): for alpha = 1, lambda = 10^-5, rho_1 = 0.5, the formulas in (8.27) give gamma* approximately 158 and r* approximately 158, so the left-hand side of (5.26) is approximately 157 while the right-hand side is approximately 0.99. Because Figures 4-8 are computed from (8.27), the claimed test-error peak at alpha = 1, and with it the central double-descent conclusion, does not follow from the stated theory. This algebraic error is load-bearing and invalidates the paper's main numerical claim.
  2. [Theorem 8.1 and Section 8.4] The proof that the global training loss L*_lambda is asymptotically equal to the infimum of the lower bound omega_lambda^(d)(r,s) is only a sketch and leaves CGMT regularity conditions unverified. In particular, inequality (8.10) is merely asserted, and the text itself concedes that the right-hand side may be infinite when only omega_lambda^(d) is bounded. The boundedness of ||u||_2/d and the sup-norm convergence needed to apply CGMT are not rigorously established. Since Theorem 5.1 inherits this identification, the fixed-point system that underlies all subsequent calculations is not fully proved. This is a serious gap, although it is secondary to the algebraic error in Major Comment 1.
minor comments (4)
  1. [Section 2, p.4] The text says "CIFAR-10 and MINST" but should say "MNIST."
  2. [Section 3.2, Eq. (3.1)] The sentence following equation (3.1) contains malformed notation: "xT_iβ√d+b,∈Rd" should be written as "x_i^T β / sqrt(d) + b in R."
  3. [Section 9.1, R code] In the loop used to generate data for the Section 6 plots, the line "x <- (eta*y)/sqrt(features) + epsilon" overwrites the entire matrix x on each iteration instead of assigning to the row x[i, ]; the intended data matrix is therefore not stored. The theoretical curves in Section 6 do not actually use the simulated x, so this does not affect those plots, but the code is misleading as written.
  4. [Section 6.2] The statement that "when λ ≥ 1, the test error keeps unchanged until a specific α value is reached" is vague; Figure 6 suggests the error stays near the Bayes-optimal value for small alpha, but the text should state this explicitly and relate it to the formulas in (8.27).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the fixed-point equations are derived from CGMT with alpha, lambda, and rho as external inputs, and the test-error curve is not used as an input to the derivation.

full rationale

The paper's derivation chain is self-contained rather than circular. The asymptotic equations in Theorem 5.1 are obtained from a CGMT-based lower bound, with the parameters alpha, lambda, rho_1 (and the teacher-model constants) as inputs; the resulting r*, s*, b* are solved from the fixed-point system and then inserted into the closed-form test-error formula of Theorem 4.1. No fitted parameter is relabeled as a prediction, and the claimed double-descent curve is not used to define the equations that produce it. The citations to CGMT ([19], [20]) are external mathematical results, not self-citations by the present authors, and no uniqueness claim is imported from the authors' own prior work. The most serious defect identified—the transition from equation (5.26) to (8.21) in Section 8.6, where the factor r* is silently dropped so that gamma* is solved with alpha rather than alpha*r*—is an algebraic inconsistency with the paper's own equations, not a circular reduction: the target generalization error is not an input to that calculation. Similarly, the proof sketch of Theorem 8.1 and the assertion of inequality (8.10) with an explicitly conceded possibly infinite right-hand side are rigor gaps, not cases where the conclusion is equivalent to an assumption by construction. Therefore the appropriate circularity finding is no significant circularity, score 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no fitted constants and no invented physical entities. It relies on the Gaussian teacher model, the CGMT theorem, and the asymptotic equivalence of the lower bound to the true loss; the last item is the least established.

assumptions (5)
  • domain assumption Training data are generated from the two-Gaussian teacher model xi = eta yi / sqrt(d) + epsilon_i with eta and epsilon_i having i.i.d. standard normal entries.
    Introduced in Section 3.3; all subsequent calculations of test error and the CGMT analysis depend on this data model.
  • standard math The Convex Gaussian Min-Max Theorem applies to the empirical risk problem (3.3) and yields both lower and upper tail bounds on the optimal value.
    Invoked in Section 5.1 citing [19,20]; the convexity-concavity and boundedness conditions required by CGMT are asserted rather than verified.
  • ad hoc to paper The candidate omega_lambda^(d)(r,s), the lower bound of the auxiliary problem, is asymptotically equal to the global training loss.
    This is the paper's own Theorem 8.1; its proof in Sections 8.3 and 8.4 is sketchy and not fully rigorous.
  • domain assumption The limiting regime n,d -> infinity with n/d = alpha fixed, and eta^T h / d -> 0 by the law of large numbers.
    Asymptotic framework in Section 3.5; used to simplify equations (5.18) through (5.20) to (5.21) through (5.24).
  • domain assumption The constraint s^2 <= r holds, with s = beta^T eta / d and r = ||beta||^2 / d.
    Section 5 uses Cauchy-Schwarz to justify s^2 <= r in high dimension, defining the domain of the global training loss and keeping sqrt(r - s^2) real.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Double Descent Behavior in Two Layer Neural Network for Binary Classification." pith.science (2026). https://pith.science/paper/YT4DBPCC

@misc{pith2026250419351,
  author       = {Pith},
  title        = {Pith review of: The Double Descent Behavior in Two Layer Neural Network for Binary Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YT4DBPCC}},
  note         = {Machine review of arXiv:2504.19351}
}
read the original abstract

Recent studies observed a surprising concept on model test error called the double descent phenomenon, where the increasing model complexity decreases the test error first and then the error increases and decreases again. To observe this, we work on a two layer neural network model with a ReLU activation function designed for binary classification under supervised learning. Our aim is to observe and investigate the mathematical theory behind the double descent behavior of model test error for varying model sizes. We quantify the model size by the ratio of number of training samples to the dimension of the model. Due to the complexity of the empirical risk minimization procedure, we use the Convex Gaussian Min Max Theorem to find a suitable candidate for the global training loss.

Figures

Figures reproduced from arXiv: 2504.19351 by the authors.

Figure 1
Figure 1. The double descent phenomenon in highly over-parameterized models. With the double descent behavior, the test error first decreases and then increases tracing the U-shaped curve, and decreases again after the model complexity goes through a certain threshold value. The orig￾inal concept in [3] analyzes the test error with respect to function class sizes or specif￾ically by the number of parameters needed (p) and the… view at source ↗
Figure 2
Figure 2. Test error showing the double descent behavior when a two-layer ReLu model is [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Test error not showing the double descent behavior when a two-layer ReLu model is [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Test error of model (3.1) with square loss as a function of α with low regularization (λ = 10−5 ) and with ρ1 = 0.5 [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Test error of model (3.1) for square loss as a function of α with ρ1 = 0.5 and with varying regularization [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Test error of model (3.1) for square loss as a function of α with ρ1 = 0.7, and with varying regulariza￾tion values [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: shows the same curves in [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Test error of model (3.1) as a function of regularization. We fix the values [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 18 canonical work pages

  1. [1]

    S., Andrew, M

    Advani, M. S., Andrew, M. S., and Haim, S. (2020). High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132 , 428-446

    work page 2020

  2. [2]

    Amir, I., Koren, T., and Livni, R. (2021). SGD generalizes better than GD (and regular- ization doesn’t help). Conference on Learning Theory, 63-92

    work page 2021

  3. [3]

    and Mandal, S

    Belkin, M., Hsu, D., Ma, S. and Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116 (32), 15849-15854

    work page 2019

  4. [4]

    Bhavsar, H., and Ganatra, A. (2012). A comparative study of training algorithms for supervised machine learning. International Journal of Soft Computing and Engineering (IJSCE), 2(04), 2231-2307

    work page 2012

  5. [5]

    Bonaccorso, G. (2017). Machine Learning Algorithms . Packt Publishing Ltd. Birming- ham, UK

    work page 2017

  6. [6]

    D’Ascoli, S., Refinetti, M., Biroli, G., and Krzakala, F. (2020). Double trouble in double descent: Bias and variance (s) in the lazy regime. International Conference on Machine Learning, 2280-2290

    work page 2020

  7. [7]

    Deng, Z., Kammoun, A., and Thrampoulidis, C. (2022). A model of double descent for high-dimensional binary linear classification. Information and Inference: A Journal of the IMA, 11 (02), 435-495

    work page 2022

  8. [8]

    Geiger, M., Jacot, A., Spigler, S., Gabriel, F., Sagun, L., d’Ascoli, S., Biroli, G., Hongler, C., and Wyart, M. (2020). Scaling description of generalization with number of param- eters in deep learning. Journal of Statistical Mechanics: Theory and Experiment , (02), 023401

    work page 2020

Show all 21 references
  1. [9]

    Hutter, F., Kotthoff, L., and Vanschoren, J. (2019). Automated machine learning: meth- ods, systems, challenges. Springer Nature, 219

    2019

  2. [10]

    Kini, G. R. and Thrampoulidis, C. (2020). Analytic study of double descent in binary classification: The impact of loss. 2020 IEEE International Symposium on Information Theory (ISIT), 2527-2532. 17

    2020

  3. [11]

    and Cherkassky, V., 2024

    Lee, E.H. and Cherkassky, V., 2024. Understanding Double Descent Using VC- Theoretical Framework. IEEE Transactions on Neural Networks and Learning Systems

    2024

  4. [12]

    Mahesh, B. (2020). Machine learning algorithms-a review. International Journal of Sci- ence and Research (IJSR), 381-386

    2020

  5. [13]

    and Zdeborova, L., (2020), The role of regularization in classification of high-dimensional noisy gaussian mixture

    Mignacco, F., Krzakala, F., Lu, Y., Urbani, P. and Zdeborova, L., (2020), The role of regularization in classification of high-dimensional noisy gaussian mixture. International conference on machine learning, 6874-6883

    2020

  6. [14]

    Nakkiran, P. (2019). More data can hurt for linear regression: Sample-wise double de- scent. arXiv preprint arXiv: 1912.07242

    2019 arXiv

  7. [15]

    Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. (2021). Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment , (12), 124003

    2021

  8. [16]

    and Ma, T

    Nakkiran, P., Venkat, P., Kakade, S. and Ma, T. (2020). Optimal regularization can mitigate double descent. arXiv preprint arXiv: 2003.01897

    2020 arXiv

  9. [17]

    Simon, C. P. and Blume, L. (1994). Mathematics for Economists, (7), New York: Norton

    1994

  10. [18]

    and Wyart, M

    Spigler, S., Geiger, M., d’Ascoli, S., Sagun, L., Biroli, G. and Wyart, M. (2019). A jamming transition from under-to over-parametrization affects generalization in deep learning. Journal of Physics A: Mathematical and Theoretical, 52 (47), 474001

    2019

  11. [19]

    and Hassibi, B

    Thrampoulidis, C., Oymak, S. and Hassibi, B. (2014). The Gaussian min-max theorem in the presence of convexity. arXiv preprint arXiv: 1408.4837

    2014 arXiv

  12. [20]

    conjugate

    Thrampoulidis, C., Oymak, S. and Hassibi, B. (2015). Regularized linear regression: A precise analysis of the estimation error. Conference on Learning Theory, 1683-1709. 18 8 Supplementary material 1 8.1 Legendre transformation Given a function l : R→ R, its Legendre transform...

    2015

  13. [21]

    (8.16) Differentiating (5.17) with respect to s gives the following expression

    Rearranging the terms will yield, l′′(vi)dvi ds = 1 γ 1 2− dvi ds−l′(vi)dγ ds . (8.16) Differentiating (5.17) with respect to s gives the following expression. 8αγ||l′(v)||2 2 dγ ds + 8αγ2 nX i=1 l′(vi)l′′(vi)dvi ds = 2n p r−s2− sηT h d − s√ r−s2−ηT h d . We letG = √ r−s2ηT h ...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.