{"id":"d8c4d88f-bdb6-4595-9a53-1df376ea96d6","arxiv_id":"2411.18806","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A one-step early stopping rule for underparameterized two-layer networks, with an NTK- and Rademacher-based upper bound on population loss that decreases after one gradient step.","lead":"This paper derives a formula for stopping the training of a two-layer neural network after a single gradient descent step, based on the top eigenvalue of the neural tangent kernel and the initial training error. It provides a theoretical upper bound on the population loss, demonstrated on a small network imitating a model predictive controller for a Van der Pol oscillator.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Linearized dynamics assumption (Eq. 7) is falsified by the paper's own Table I: predicted one-step training loss ≈0.008, reported 0.19. Without a finite-width residual bound, Theorem 4 does not apply to the tested network.","rationale":"The reader's weakest assumption is exactly the use of Eq. (7) for finite-width networks, and the strongest claim (Theorem 4) depends on it. The paper's own Table I contains an internal contradiction: the empirical ||v(t1)||^2/n = 0.19 is far from the linearized projection bound (A1')^2/n + (B1')^2/n ≈ 0.008. This is not a minor numerical issue; it shows the central mechanism of the proof — that the NTK contraction along the top eigenvector bounds the post-step empirical loss — is absent in the tested configuration. Without any finite-width residual estimate, Theorem 4 cannot be applied to the example, and the method's claim of being well-suited to the underparameterized regime is unsupported. The formal theorem may be salvageable by adding explicit finite-width assumptions or by verifying the linearization error empirically, but as written the central argument does not hold for the demonstrated network. The reader's conditional verdict should be tightened to reject, because the primary numerical evidence contradicts the core assumption rather than merely requiring a fix to a proof detail.","tokens_in":10934,"tokens_out":10871,"duration_ms":89270,"concrete_test":"Reproduce Example 1 and compute the normalized linearization residual R = ||v(t1) − (I − η H[W(t0)]) v0|| / ||v0||, together with the empirical ||v(t1)||^2/n and the linearized prediction ((1−β)A1)^2/n + B1^2/n. If R > 0.05, or if the empirical one-step training loss is not within 10% of the linearized prediction, then Eq. (7) is violated and Theorem 4 does not apply to this network. Run the same check for m = 10 (as in the paper) and, for sensitivity, for m = 100 and m = 1000.","verdict_should_be":"REJECT","load_bearing_attack":"Theorem 4 and the intermediate upper bound ω1 on L*_D rely on the exact linearized error update v(t1) = (I − η H[W(t0)]) v0 (Eq. 7). This identity is exact only in the infinite-width or linearized-NTK regime; for a finite-width network with m = 10, the higher-order terms are never quantified. The paper's own Example 1 gives a quantitative refutation: with A1 = 13.9, B1 = 0.35, ρ = 1 − β = 0.18, the linearized dynamics predict a one-step training loss of (ρA1)^2/n + B1^2/n = (0.18·13.9)^2/800 + 0.35^2/800 ≈ 0.0078, whereas Table I reports ||v(t1)||^2/n = 0.19 — a roughly 24-fold discrepancy. Therefore ω1 = ν1 + ((A1')^2 + (B1')^2)/n does not upper-bound the true L*_D for the actual network. The reported Ω1 = 0.31 only manages to exceed the observed L*_D ≈ 0.30 because the claimed linearized contraction did not occur; the decrease by |Δ1| in Theorem 3 is not realized in the experiment. Consequently, Theorem 4 is not established for finite-width networks, and the paper's central premise fails in its own demonstration.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a one-step early stopping rule for a single-hidden-layer neural network trained by gradient descent on a finite sample. Using Rademacher complexity to bound the generalization gap and the neural tangent kernel (NTK) to describe the evolution of the training error, it derives an upper bound Ω1 on the population loss after one GD step (Theorem 4, Eq. (28)) and a condition (Eq. (18)) under which this bound strictly decreases. The method is illustrated on a neural network simulating MPC control of a Van der Pol oscillator, with reported values in Table I. The central claim is that the bound is computable from the initial error vector v0 and the NTK matrix H, and that it decreases after one step at t1 = t0 + β/λ1^-.","tokens_in":11220,"tokens_out":10069,"duration_ms":80391,"significance":"If the result were valid, it would extend NTK-based generalization analysis to the underparameterized regime (m moderate compared with n), where most NTK eigenvalues vanish, and would provide an analytically computable, width-independent stopping time in the normalized-output case. The paper also sketches a speculative but interesting connection to benign overfitting. These are valuable directions. However, the main theorem relies critically on the exactness of the linearized error dynamics and on a projection bound whose proof contains a serious gap; moreover, the paper's own example contradicts the theory. As presented, the central claim is not established.","major_comments":[{"comment":"The identity v(t1) = (I − ηH[W(t0)])v0 is exact only in the infinite-width or linearized-NTK regime. The manuscript applies it to a finite-width network with m = 10 (Example 1) without quantifying the omitted higher-order terms. Example 1 itself shows the approximation is poor: with A1 = 13.9, B1 = 0.35, β = 0.82, the linearized dynamics predict ||v(t1)||^2/n ≈ ((0.18·13.9)^2 + 0.35^2)/800 ≈ 0.008, whereas Table I reports ||v(t1)||^2/n = 0.19, a roughly 24-fold discrepancy. Therefore Proposition 4's projection bounds, and consequently the upper bound Ω1 in Theorem 4, are not established for the tested network, and the claimed decrease |Δ1| is not realized in the experiment.","section":"Section II-A, Eq. (7); Theorem 4"},{"comment":"There is an internal contradiction in the reported numbers. The text states that ω1 − ν1 ≈ ((γ1A1)^2 + B1^2)/n = 0.008, which uses the linearized projections, but Table I reports ||v(t1)||^2/n = 0.19. Since ω1 is defined to satisfy ν1 + ||v(t1)||^2/n ≤ ω1 (Eq. (23)), the value ω1 ≈ 0.118 (ν1 = 0.11 plus 0.008) is smaller than the actual empirical loss at t1, contradicting the inequality chain (27). In addition, Ω1 = 0.31 in Table I is not reproduced by the formula Ω1 = ω0 + Δ1 with the given values, which yields approximately 0.23; the origin of the discrepancy is unexplained.","section":"Section III, Example 1"},{"comment":"The proof of Proposition 4 asserts that because all entries of I_n − ηΛ are non-negative and not larger than those of I_n − ηΛ_abs, the vector inequality v(t1) ≤ P(I_n − ηΛ_abs)P^T v(t0) follows. This inference is valid only if P and P^T have nonnegative entries. P is an orthogonal eigenvector matrix of the NTK matrix and is not generally nonnegative, so the entrywise comparison does not survive conjugation by P. The later reduction to a 2×2 rotation matrix P_2 in Eq. (32) is also not justified by the preceding eigendecomposition. Since Proposition 4 underpins Eq. (20) and Theorem 2, the main theorem does not follow from the proof as written.","section":"Appendix VI, Eq. (31)"}],"minor_comments":[{"comment":"The discussion of benign overfitting is explicitly informal; it would be helpful to state clearly that it is a heuristic argument and not a proved consequence of Theorem 4.","section":"Section IV, Overparameterized case"},{"comment":"When M1 is estimated by a Monte Carlo method with probability ε, Theorem 4's probability statement should explicitly combine δ and ε in a single displayed formula; the text only mentions it in passing after Eq. (28).","section":"Remark 4"},{"comment":"The matrix P is first described as a transition matrix between basis vectors and then used as the eigenvector matrix of H; please clarify the notation and its role in Eq. (30)–(32).","section":"Appendix VI, first paragraph"}],"recommendation":"reject","confidential_remarks":"The paper addresses an interesting and timely problem, and the algebraic derivation from Theorem 3 to Theorem 4 is internally consistent under the linearized-dynamics assumption. However, the central theorem is not valid for finite-width networks as stated because Eq. (7) is only approximate outside the NTK limit, and the paper's own example (m = 10) exhibits a 24-fold discrepancy between the predicted and actual one-step training error. The proof of Proposition 4 also contains an unjustified matrix-inequality step. The internal numerical inconsistency between Table I and the computed ω1 suggests that the experimental validation was not cross-checked against the theoretical formulas. These are load-bearing issues that cannot be resolved by minor revisions within the manuscript's current scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is genuinely new: using the leading eigenvalue of the NTK and Rademacher complexity to design a one-step early stopping rule for underparameterized two-layer networks, with no width assumption in the normalized output case. The algebra from Theorem 3 to Theorem 4 is internally consistent if you accept the exact linearized error update v(t1) = (I - ηH[W(t0)])v0. That assumption is the problem.\n\nEquation (7) is exact only in an infinite-width or linearized-NTK limit. For the finite network in Example 1 (m=10, n=800), the paper never quantifies the higher-order terms. Worse, the example itself refutes the assumption: the linearized projection bound gives ||v(t1)||²/n ≈ 0.008, while Table I reports 0.19, a roughly 24-fold gap. So the predicted contraction along the top eigenvector did not happen in the actual training run. The reported Ω1 = 0.31 only exceeds the observed population loss because the linearized decrease |Δ1| is not realized. Theorem 4, as stated, is not established for finite-width networks, and the paper's own demonstration fails.\n\nThe written proof of Proposition 4 also contains an invalid step: for i≥2, the entries of I - ηΛ are larger than those of the rank-one abstraction, not smaller, so the entrywise comparison in Equation (31) goes the wrong way. This is fixable, but as written the projection bounds do not follow. The confidence bound in Example 1 has a wrong constant too: Theorem 4 gives 3M1² sqrt(log(2/δ)/(2n)) ≈ 0.0047 sqrt(log(2/δ)) for M1=0.25, n=800, not 0.19 sqrt(log(2/δ)). The overparameterized discussion at the end is informal and does not affect the main result, but it should be labeled as heuristic.\n\nWhat the paper does well: the combination of NTK spectral information with Rademacher complexity for early stopping is a fresh angle, and the formulas for γ1, Δ1, and t1 are explicit and checkable. The authors also cite the relevant underparameterized and NTK literature, and the limitation to single hidden layer and scalar output is stated clearly.\n\nThe bottom line: this is a conditional theoretical result that is not validated by its own experiment. A revision that either (a) adds a finite-width residual bound and then tests in a regime where linearization is plausible, or (b) reframes the contribution as a bound for the idealized linearized dynamics and evaluates the nonlinear network separately, could be worth a second look. As it stands, the central claim is not solid enough to rely on.\n\nFor peer review: I would send this to a serious referee, because the idea has substance and the flaws are identifiable and potentially repairable, but it needs major revision before publication.","headline":"Novel idea for NTK-based one-step early stopping, but the main theorem is contradicted by the paper's own finite-width experiment, and the proof of Proposition 4 has a backwards inequality.","tokens_in":11769,"tokens_out":2848,"would_cite":false,"duration_ms":25404,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68Q32"],"pacs":[],"model":"deepseek-v4-flash","headline":"After one gradient step, an upper bound on population loss provably drops","keywords":["early stopping","neural tangent kernel","Rademacher complexity","generalization bound","underparameterized networks","gradient descent","population loss","model predictive control"],"falsifier":"Train a fixed single-hidden-layer network with m = 10 and n = 800 as in the paper's example, measure the actual training error ||v(t1)||^2/n after one gradient step, and compare it with the linearized prediction (ρ1 A1)^2/n. If the measured value is substantially larger than the prediction, as it already appears to be in Table I (0.19 versus about 0.006), then the linearized dynamics fail and the decrease bound in Theorem 4 cannot be concluded.","tokens_in":10710,"feed_emoji":"⏱","tokens_out":5164,"duration_ms":53626,"temperature":0.7,"pith_summary":"This paper gives a formula for when to stop training a neural network after a single gradient-descent step, in the setting where the number of parameters is not large compared with the number of training samples. The formula uses only the initial training error vector and the eigenvalues of the neural tangent kernel, and it comes with a high-probability upper bound on the population loss. Under a condition on a computable scalar γ1, this upper bound strictly decreases after one step, so the bound itself selects the stopping time. The authors demonstrate the bound on a 10-neuron network simulating the MPC control of a Van der Pol oscillator.","feed_headline":"One gradient step can provably cut the population-loss bound","feed_subtitle":"A bound that uses only the initial error and the neural tangent kernel picks a stopping time—no width assumption needed.","key_machinery":"The argument rests on the linearized error dynamics v(t1) = (I − η H[W(t0)]) v0, where H is the n × n neural tangent kernel matrix. The proof projects the initial error v0 onto the leading eigenvector u1(t0), tracks the contraction along that direction with factor ρ1 = 1 − β, and bounds the orthogonal component via a rotation term; Rademacher complexity supplies the generalization-gap term. The scalar γ1 defined in Equation (17) measures the relative size of the generalization-gap term against the contraction, and the condition γ1 < 1 − β/2 is what makes the overall bound decrease.","core_discovery":"The central claim is that, for a single-hidden-layer network with fixed output weights trained by gradient descent on the quadratic loss, the population loss LD after one step at t1 = t0 + β/λ1^- is controlled by an explicit bound that decreases under the condition γ1 < 1 − β/2. More precisely, Theorem 4 gives, with probability at least 1 − δ, LD ≤ Ω1 + $3M1^{2}$ $\\sqrt$(log(2/δ)/(2n)), where Ω1 = ω0 + Δ1 and Δ1 = −($2A1^{2}$/n) β (1 − γ1 − β/2)(1 + α1)^2 + D1/n. Because the main term of Δ1 is negative, the upper bound on the population loss is smaller at t1 than at t0, so one observationally informed step is a provably safe place to stop. In the underparameterized setting that motivates the paper, most eigenvalues of the neural tangent kernel are zero, and the relevant eigenvalue is the largest positive one, λ1. In the normalized-output-weight case the whole computation is independent of the hidden-layer width m and depends only on the initial error vector v0 and the NTK matrix H.","pith_inferences":["The paper leaves open how large the finite-width error in Equation (7) is; a natural extension is to add an explicit remainder term that depends on the size of the higher-order terms in the NTK expansion, making the bound honest for finite m.","The reported numerical example may already probe this limitation: the measured one-step training error of 0.19 is much larger than the linearized projection prediction (A1')^2/n ≈ 0.006, suggesting the exact linearization is not satisfied for the 10-neuron network.","The benign-overfitting argument suggests a practical diagnostic: compute the spectrum of the NTK; if γ1 stays small over many steps, the bound predicts no overfitting, which could be compared empirically against test-loss curves."],"forward_implications":["The optimal one-step stopping time is t1 = t0 + β/λ1^-; choosing β = 1 − γ1 maximizes the guaranteed decrease |Δ1|.","In the normalized-output case the whole computation uses only the initial error vector and the NTK matrix, so the user does not need to know or control the hidden-layer width.","If the condition γ1 < 1 − β/2 fails, the theorem does not certify a decrease after one step, meaning the method itself signals when early stopping is not justified.","In the overparameterized limit m → ∞ the paper shows γ1 → 0 and iterating the one-step argument over n steps drives the bound to zero, offering a new explanation of benign overfitting."],"supporting_citations":[{"why":"Defines the neural tangent kernel matrix H that the whole bound is built on.","marker":"[2]"},{"why":"Provides the linearized gradient-descent error dynamics (Equation (7)) and the near-zero initialization assumption.","marker":"[3]"},{"why":"Supplies the overparameterized generalization framework and the connection between empirical and population loss for two-layer networks.","marker":"[4]"},{"why":"Previous underparameterized training and generalization study whose Van der Pol MPC example is reused for the numerical illustration.","marker":"[7]"},{"why":"Source of Proposition 1, the Rademacher-complexity bound on the generalization gap.","marker":"[8]"},{"why":"Gives the Rademacher complexity bound for one-hidden-layer networks used in Proposition 2.","marker":"[9]"},{"why":"Also cited for the same Rademacher complexity bound (Theorem 5.7).","marker":"[10]"},{"why":"Used for the Rademacher complexity of linear hypothesis sets in the proof of Proposition 2.","marker":"[13]"}],"fun_headline_variants":["One gradient step provably shrinks the population-loss bound","NTK-based early stopping: one step cuts loss bound","Stop after one step: proven bound improvement","Early stop in one step with NTK and Rademacher","Rademacher and NTK bound shrinks after a single step"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that the one-step error update v(t1) = (I − ηH[W(t0)])v0 is exact for the finite-width network being trained, although this identity is only exact in the infinite-width linearized limit.","fun_headline_variants_meta":{"raw":{"variants":["One gradient step provably shrinks the population-loss bound","NTK-based early stopping: one step cuts loss bound","Stop after one step: proven bound improvement","Early stop in one step with NTK and Rademacher","Rademacher and NTK bound shrinks after a single step"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1464,"prompt_tokens":930,"completion_tokens":534,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":453}},"tokens_in":546,"tokens_out":534,"duration_ms":4765,"temperature":1.0,"reasoning_tokens":453,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:54:40.101785+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a fixed single-hidden-layer network with m = 10 and n = 800 as in the paper's example, measure the actual training error ||v(t1)||^2/n after one gradient step, and compare it with the linearized prediction (ρ1 A1)^2/n. If the measured value is substantially larger than the prediction, as it already appears to be in Table I (0.19 versus about 0.006), then the linearized dynamics fail and the decrease bound in Theorem 4 cannot be concluded.","supporting_citations":[{"cited_title":"Neural tangent kernel: Con- vergence and generalization in neural networks,","cited_arxiv_id":null,"evidence_quote":"Defines the neural tangent kernel matrix H that the whole bound is built on."},{"cited_title":"Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the overparameterized generalization framework and the connection between empirical and population loss for two-layer networks."},{"cited_title":"Training and generalization errors for underparameterized neural networks,","cited_arxiv_id":null,"evidence_quote":"Previous underparameterized training and generalization study whose Van der Pol MPC example is reused for the numerical illustration."},{"cited_title":"Founda- tions of machine learning,","cited_arxiv_id":null,"evidence_quote":"Source of Proposition 1, the Rademacher-complexity bound on the generalization gap."},{"cited_title":"Rademacher and gaussian complex- ities: Risk bounds and structural results,","cited_arxiv_id":null,"evidence_quote":"Gives the Rademacher complexity bound for one-hidden-layer networks used in Proposition 2."},{"cited_title":"Lecture notes for machine learning theory,","cited_arxiv_id":null,"evidence_quote":"Also cited for the same Rademacher complexity bound (Theorem 5.7)."}],"review_version":1}