{"id":"0d7a9a19-78d7-40da-841b-c966b236a9a7","arxiv_id":"2506.07088","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"For local minimizers of l2-regularized nonlinear least squares, the prediction gap around its expected value is bounded by a Fisher-information-like weighted norm, yielding pointwise confidence intervals that widen outside the training data.","lead":"The paper derives a non-asymptotic, pointwise confidence bound for predictions of nonlinear least squares models at any fixed test input, with width controlled by a Hessian-based norm that grows when the input is far from the training data. It also gives a conjugate-gradient method that computes the norm at roughly the cost of a gradient evaluation, and tests the intervals on neural networks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 4 mishandles the sub-gamma assumption: the mgf of ||∇F||² is applied at argument λ²π²/8, so the correction is λ⁴v²/(1−cλ²), not λ²v²/(1−cλ); Theorem 1's second term is not established by the proof.","rationale":"The central claim of the paper is the finite-sample δ-level bound in Theorem 1, and the proof of that theorem rests entirely on Lemma 4. The step where the sub-gamma assumption is combined with Pisier's inequality is not correct as written: the mgf of ||∇F||² is evaluated at t=λ²π²/8, so the variance-factor contribution is of order λ⁴v² with scale cλ²π²/8, not λ²v² with scale cλ. This is a concrete algebraic error in the main proof, not a missing estimate or an unverified constant. Even if v and c were known, and even if the ReLU networks were replaced by smooth networks, the claimed interval would not follow from the assumptions. The reader's weakest assumption correctly identifies that v and c are not estimated and that ReLU violates differentiability, but my check shows a more fundamental problem: the bound's own tail dependence on the sub-gamma parameters is not derived correctly. The experiments, which set v=c=1 and report coverage of the true function, cannot validate a level claim in the absence of a corrected proof, especially since the theorem does not control bias. I therefore recommend that the theorem not be accepted as stated; a corrected proof may salvage a related bound, but that is not the paper currently before us.","tokens_in":23029,"tokens_out":19213,"duration_ms":203566,"concrete_test":"Re-derive the proof of Lemma 4 by substituting t=λ²π²/8 into Definition 1 and re-optimizing the Chernoff bound; then compare the resulting tail with the claimed second term. If the exponent is λ⁴v² rather than λ²v², compute the exact tail and the paper's claimed level for a generic F with known sub-gamma gradient norm, e.g. F(S)=||S||²/√n with S∼N(0,I_n), whose ||∇F||² is a scaled χ²_n with explicit v,c. Check at δ=0.05 and n=10,100,1000 whether the Lemma 4 interval covers; undercoverage confirms that Theorem 1's level is not a consequence of the stated assumptions.","verdict_should_be":"REJECT","load_bearing_attack":"Section 4.2's Lemma 4 is the proof engine for Theorem 1, and its combination of Pisier's inequality with Definition 1 is algebraically wrong. Pisier's inequality gives E exp(λ|F(S)−EF|) ≤ 2 E exp((λ²π²/8)||∇F(S)||²). If X=||∇F||² is sub-gamma with variance factor v² and scale c, Definition 1 at t=λ²π²/8 gives log E exp(t(X−EX)) ≤ t²v²/[2(1−ct)], i.e. an exponent of order λ⁴v²/(1−cλ²), with scale cλ²π²/8. The proof instead writes λ²(v²/2)/(1−cλ), a linear-in-λ use of the sub-gamma mgf. Consequently, inverting the Chernoff bound does not produce the theorem's second term sqrt(2ln(2/δ)v + (2/3)ln(2/δ)c)/n; a posteriori, the theorem even writes v where Definition 1 uses v². This is not a question of estimating v,c and is not merely a calibration issue: the stated finite-sample level is unsupported even when v,c are known exactly. The empirical coverage results do not repair the proof gap, and they evaluate coverage against the true function while the theorem controls only the bias-free variance term.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a non-asymptotic, high-probability pointwise confidence bound for predictions of a local minimizer of an l2-regularized non-linear least-squares objective with fixed design and Gaussian noise. The bound is expressed through a weighted norm ||∇θ f(x; θhat)||²_{M(θhat)} involving the inverse regularized Hessian, which the authors connect to the Fisher information matrix and to classical asymptotic MLE confidence intervals. They also give a conjugate-gradient based algorithm for computing the weighted norm with cost close to that of a gradient evaluation, a uniform bound over the unit sphere, and experiments comparing the method with bootstrap confidence intervals. The central theoretical result is Theorem 1, whose proof relies on an implicit differentiation lemma and a concentration lemma (Lemma 4) based on Pisier's Gaussian concentration inequality and a sub-gamma tail assumption on the weighted norm.","tokens_in":23341,"tokens_out":14178,"duration_ms":155786,"significance":"If the main theorem were correct, the paper would make a useful contribution: it gives a concrete, computable uncertainty estimate that grows away from the training data, without requiring re-training as in bootstrap; the efficient Hessian-inverse-vector product computation is a practical strength; and the fixed-design, non-asymptotic formulation is a worthwhile complement to asymptotic MLE intervals. The paper also contains a clearly stated limitation of not controlling bias. However, these strengths are tied to the validity of Theorem 1, and the proof of that theorem has a load-bearing algebraic error that is not merely a presentation issue.","major_comments":[{"comment":"Lemma 4 is the proof engine for Theorem 1, and its derivation is incorrect. Pisier's inequality (Lemma 2) requires bounding E exp(λ²π²/8 ∥∇F(S)∥²), so the sub-gamma mgf from Definition 1 must be applied at argument t = λ²π²/8. That substitution gives an exponent of order λ⁴v²/(1 − cλ²π²/8), not λ²(v²/2)/(1 − cλ) as written in the proof. The subsequent Chernoff inversion therefore does not produce the term sqrt(2 ln(2/δ)v + (2/3)ln(2/δ)c)/n in Theorem 2; the stated finite-sample level is unsupported even when v and c are known exactly. The v/v² mismatch between Definition 1 and the statement of Theorem 2 is a symptom of the same error.","section":"Section 4.2, Lemma 4"},{"comment":"The theorem conditions on unknown sub-gamma parameters v² and c, and the paper offers no estimation or certification procedure for them. Section 5 states that the proposed method is 'the bound in Theorem 1 with ν=c=1'; this does not yield the claimed δ-level guarantee unless the true parameters are known to be no larger than 1. Moreover, the empirical coverage is measured against the ground-truth function f(x;θ⋆), whereas Theorem 1 bounds |f(x;θhat)−E[f(x;θhat)]| and explicitly does not control the bias term E[f(x;θhat)]−f(x;θ⋆). The reported coverage numbers therefore cannot be read as a validation of the theorem's probability statement.","section":"Section 3, Theorem 1 and Section 5"},{"comment":"The experiments use ReLU networks trained with AdamW. Lemma 1, on which Theorem 1 rests, requires Lλ to be differentiable in θ and the regularized Hessian at the local minimizer to be invertible; ReLU networks are not differentiable at the activation kinks, and AdamW is not shown to reach an exact local minimizer. As a result, the experimental setting lies outside the theorem's assumptions, so the empirical claims about neural-network confidence intervals are not supported by the theory.","section":"Section 5 and Lemma 1"},{"comment":"The covering argument for the uniform bound over S^{d−1} treats bound(ln(1/δ)) as though it depends only on δ, but the bound in Theorem 1 depends on x through E∥∇θf(x;θhat)∥²_{M(θhat)}. Passing from the finite cover to the whole sphere requires a sup over x of that expectation, or a separate union bound for each x; this step is not written in the proof.","section":"Appendix C.3"}],"minor_comments":[{"comment":"The σ-dependence of the second term is inconsistent: Theorem 1 has an extra σ² in front of the sub-gamma term while Theorem 2 does not, and M(θ) is defined without σ² even though the noise scale σ enters the gradient with respect to the noise variable. These definitions should be aligned.","section":"Theorem 1 and Theorem 2"},{"comment":"Definition 1 uses a variance factor v², but several theorem statements and the experimental section write v in the bound; the notation should be made consistent throughout.","section":"Definition 1 and Section 3"},{"comment":"The claimed CG iteration count k = Ω(sqrt(1 + B²/λ) ln(1/ε)) is stated without proof; please either prove it or describe it as a heuristic complexity estimate.","section":"Section 3.1"},{"comment":"The Lipschitz bound in Lemma 8 contains (Lλ(θ)/(λK))^{K/2} and therefore requires λ > 0, so the discussion of the interpolation regime with λ = 0 in Section 3.1 should be explicitly qualified.","section":"Lemma 8 and Section 3.1"}],"recommendation":"reject","confidential_remarks":"The paper has an appealing idea and a useful computational scheme, but the main theorem is not proven: Lemma 4 misapplies the sub-gamma mgf, and the claimed concentration rate does not follow. The experimental section also does not test the theorem's actual object, since it measures coverage of the ground truth rather than of the expectation, and uses ReLU networks under differentiability assumptions. These are load-bearing issues rather than local presentation problems, so I cannot recommend revision as the path forward; a corrected concentration result would likely change the theorem's form and would need a fresh review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a genuinely appealing idea — a pointwise non-asymptotic confidence bound for local minimizers of ℓ2-regularized nonlinear least squares, with an inverse-Hessian weighted norm that grows outside the training data. The efficient CG/HVP computation is a real practical contribution, and the discussion connecting the bound to Fisher information and the NTK is informative. The reader's conditional verdict is more charitable than the text supports.\n\nThe soft spots are serious. The proof of the main theorem goes through Lemma 4, and that lemma's proof applies the sub-gamma mgf at the wrong argument. Pisier's inequality gives E exp(λ|F−EF|) ≤ 2 E exp((λ²π²/8)||∇F||²). If X=||∇F||² is sub-gamma with variance factor v² and scale c, its mgf at argument t=λ²π²/8 yields an exponent of order λ⁴v²/(1−cλ²), not the λ²(v²/2)/(1−cλ) the paper uses. The theorem's second term, with its √(2 ln(2/δ)v + …), is therefore not established even if v and c were known exactly. This is a load-bearing proof gap, not a tuning issue.\n\nThe experiments do not repair it. They set v=c=1 without estimation, which is a heuristic, and the claimed δ-level guarantee is not delivered in practice. They also evaluate coverage against the true function, while the theorem controls only the variance around E[f(x;θhat)]. And the ReLU networks used in the experiments are outside the assumptions: Lλ is not twice differentiable with invertible regularized Hessian at the solution.\n\nWhat is actually new: the weighted-norm representation of uncertainty, the non-asymptotic formulation, and an efficient way to compute the norm. These are worth taking seriously. The paper reads like an honest attempt, and the error is localized and likely fixable — a corrected Lemma 4 may give a weaker bound or require a different concentration argument.\n\nWho this is for: researchers in UQ for neural networks who are looking for alternatives to asymptotic MLE/Laplace intervals. If the proof is repaired, it will be a useful paper. In its current form I would not cite it, but I would send it to a serious referee: the idea is important enough to justify chasing down the proof and forcing a clean revision.","headline":"A promising weighted-norm confidence idea for nonlinear least squares, but Lemma 4's sub-gamma argument is wrong and the main theorem does not follow as stated.","tokens_in":23855,"tokens_out":6109,"would_cite":false,"duration_ms":60542,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For ℓ2-regularized non-linear least squares, this paper proves a non-asymptotic pointwise confidence bound whose width is controlled by a Hessian-weighted gradient norm that grows outside the training data.","keywords":["pointwise confidence estimation","non-linear least squares","ℓ2 regularization","sub-gamma concentration","weighted norm","inverse Hessian","fixed design regression","uncertainty quantification"],"falsifier":"Resample the noise many times for a fixed model and dataset, compute the centered weighted norm $W = \\|\\nabla_{\\theta} f(x;\\hat{\\theta})\\|^2_{M(\\hat{\\theta})} - E[\\|\\nabla_{\\theta} f(x;\\hat{\\theta})\\|^2_{M(\\hat{\\theta})}]$ for each draw, and check whether the empirical moment-generating function satisfies the sub-gamma inequality $\\ln E[e^{\\lambda W}] \\le \\lambda^2 v^2/(2(1-c\\lambda))$ for the chosen $v,c$ and all $\\lambda \\in (0,1/c)$; if it fails, or if the empirical coverage of the interval over many datasets falls persistently below the nominal $1-\\delta$, the load-bearing premise collapses.","tokens_in":22801,"feed_emoji":"📊","tokens_out":19761,"duration_ms":158892,"temperature":0.7,"pith_summary":"To know how much to trust a learned non-linear predictor at a single test point, one needs a confidence interval that does not rely on asymptotic arguments. This paper derives such an interval for $\\ell^2$-regularized non-linear least squares under Gaussian noise: for any fixed test input $x$, the prediction error is controlled by a weighted norm of the predictor's gradient, with a weight matrix built from the regularized Hessian and the empirical Fisher information. The norm plays the role of $x^{\\top} \\Sigma^{-1} x$ from linear ridge regression, so the interval automatically widens when $x$ is poorly represented by the training data. The result holds for any local minimizer of the regularized loss and comes with an efficient conjugate-gradient algorithm that avoids forming the Hessian explicitly.","feed_headline":"Finite-sample confidence bounds for nonlinear least squares","feed_subtitle":"Its width is a Hessian-weighted norm that grows outside training data, matching ridge bounds in the linear case.","key_machinery":"The load-bearing object is the positive semidefinite matrix $M(\\theta) = (\\nabla^2 L(\\theta) + \\lambda I)^{-1} (1/n \\sum_i \\nabla_{\\theta} f(x_i;\\theta)\\nabla_{\\theta} f(x_i;\\theta)^{\\top}) (\\nabla^2 L(\\theta) + \\lambda I)^{-1}$, a non-linear generalization of the inverse covariance that asymptotically approaches the inverse Fisher information. The proof differentiates the stationary-point condition to get $\\partial \\hat{\\theta}/\\partial z_i = (1/n)(\\lambda I + \\nabla^2 L(\\hat{\\theta}))^{-1} \\nabla_{\\theta} f(x_i;\\hat{\\theta})$, which turns the prediction's sensitivity to each noise variable into the weighted norm of the gradient. A Gaussian concentration inequality for differentiable functions of normal vectors then controls $|f(x;\\hat{\\theta}) - E[f(x;\\hat{\\theta})]|$ by the expected norm, and a sub-gamma tail assumption on the norm converts that into the explicit confidence interval. Computationally, the norm is evaluated by conjugate gradient with Hessian-vector products from automatic differentiation, at cost $\\tilde{O}((n+p)/\\sqrt{\\lambda} + np)$ rather than $O(p^3)$.","core_discovery":"The central claim is Theorem 1: if the centered weighted norm $\\|\\nabla_{\\theta} f(x;\\hat{\\theta})\\|^2_{M(\\hat{\\theta})}$ is sub-gamma with variance factor $v^2$ and scale $c$, then for every fixed test input $x$ and every $\\delta \\in (0,1)$, with probability at least $1-\\delta$, $|f(x;\\hat{\\theta}) - E[f(x;\\hat{\\theta})]| \\le \\sigma \\sqrt{(\\pi^2/2) \\ln(2/\\delta) / n \\cdot E[\\|\\nabla_{\\theta} f(x;\\hat{\\theta})\\|^2_{M(\\hat{\\theta})}]} + \\sigma^2 \\sqrt{2 \\ln(2/\\delta) v + (2/3) \\ln(2/\\delta) c}/n$, where $M(\\theta) = (\\nabla^2 L(\\theta) + \\lambda I)^{-1} (1/n \\sum_i \\nabla_{\\theta} f(x_i;\\theta) \\nabla_{\\theta} f(x_i;\\theta)^{\\top}) (\\nabla^2 L(\\theta) + \\lambda I)^{-1}$. The bound requires only that the regularized loss be differentiable and that $\\hat{\\theta}$ be a local minimizer; it recovers the ridge-regression interval as a sanity check, its asymptotic shape matches the delta-method interval for the MLE, and the weighted norm is computable without ever forming the $p \\times p$ inverse Hessian.","pith_inferences":["If $v$ and $c$ were estimated from data (for example by subsampling or empirical tail bounds) and plugged into the interval, the $\\delta$-level guarantee would become fully empirical; the paper leaves that as an open engineering step.","The polynomial-tail variant in the appendix removes the sub-gamma assumption at the cost of $1/\\delta$ rather than $\\ln(1/\\delta)$, so a practical implementation could switch between the two bounds depending on whether a certified tail is available.","Because ReLU networks are not differentiable everywhere, the formal theorem does not literally cover the experiments; a smoothed-activation or subgradient version of the proof would likely preserve the same interval shape and is a natural next step.","Testable extension: for a dataset with known ground truth, estimate the empirical coverage of the interval with $v=c=1$ across many noise draws; the gap from the nominal level quantifies how sensitive the guarantee is to the unverified sub-gamma parameters."],"forward_implications":["A single trained non-linear model yields pointwise confidence bands that automatically widen in regions with few or no training observations; in the paper's experiments this is the behavior that bootstrap-based intervals fail to deliver.","For linear models the bound reduces to the standard ridge-regression interval with the known $x^{\\top}\\Sigma^{-1} x$ weighting, so the non-linear result is a strict generalization of a classical guarantee.","Under consistency the bound's asymptotic form matches the delta-method interval from asymptotic normality of the MLE, giving a finite-sample analogue rather than a separate heuristic.","Uniform confidence over a whole input sphere follows for ReLU networks with only a mild covering-factor cost once the training procedure controls the predictor's Lipschitz constant.","The weighted norm is computed by conjugate gradient with complexity roughly $\\tilde{O}((n+p)/\\sqrt{\\lambda} + np)$ and $O(p)$ memory, so it scales to large parameter counts."],"supporting_citations":[{"why":"Supplies the Gaussian concentration inequality that bounds a differentiable function of normal vectors by its gradient norm, the central tool in the proof of Theorem 1.","marker":"Pisier (1986)"},{"why":"States the same Gaussian concentration inequality in textbook form; the paper cites it alongside Pisier in its Lemma 2.","marker":"Wainwright (2019, Display below Lemma 2.27)"},{"why":"Provides the sub-gamma rate-function inversion used to turn the exponential-moment bound into the explicit confidence interval.","marker":"Boucheron, Lugosi, and Massart (2003, Lemma 11)"},{"why":"Supplies the asymptotic normality of the MLE and the delta method that the paper's bound is presented as a non-asymptotic counterpart of.","marker":"Van der Vaart (2000)"},{"why":"Gives the ridge-regression confidence interval with the $x^{\\top}\\Sigma^{-1}x$ weighted norm that the bound recovers in the linear sanity check.","marker":"Lattimore and Szepesvári (2020, Chap. 20)"},{"why":"Defines the bootstrap method used as the empirical baseline whose coverage/width trade-off the proposed intervals are compared against.","marker":"Efron and Tibshirani (1994)"},{"why":"Defines the neural tangent kernel used to contextualize the proof's time-varying kernel interpretation of gradient descent at the stationary point.","marker":"Jacot et al. (2018)"}],"fun_headline_variants":["Confidence intervals that widen outside training data","Pointwise confidence with Hessian-weighted scaling","Non-asymptotic confidence for nonlinear least squares","Confidence that grows with input novelty","Hessian-weighted bounds for nonlinear least squares"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee rests on the centered weighted norm $\\|\\nabla_{\\theta} f(x;\\hat{\\theta})\\|^2_{M(\\hat{\\theta})}$ being sub-gamma with known variance factor $v^2$ and scale $c$; the paper gives no way to estimate or certify $v$ and $c$, and the experiments simply set both to 1, so the announced $\\delta$-level coverage is conditional on an unverified tail assumption.","fun_headline_variants_meta":{"raw":{"variants":["Confidence intervals that widen outside training data","Pointwise confidence with Hessian-weighted scaling","Non-asymptotic confidence for nonlinear least squares","Confidence that grows with input novelty","Hessian-weighted bounds for nonlinear least squares"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000967,"raw_usage":{"total_tokens":4195,"prompt_tokens":1110,"completion_tokens":3085,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":726,"completion_tokens_details":{"reasoning_tokens":3018}},"tokens_in":726,"tokens_out":3085,"duration_ms":21334,"temperature":1.0,"reasoning_tokens":3018,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:42:45.499720+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Resample the noise many times for a fixed model and dataset, compute the centered weighted norm $W = \\|\\nabla_{\\theta} f(x;\\hat{\\theta})\\|^2_{M(\\hat{\\theta})} - E[\\|\\nabla_{\\theta} f(x;\\hat{\\theta})\\|^2_{M(\\hat{\\theta})}]$ for each draw, and check whether the empirical moment-generating function satisfies the sub-gamma inequality $\\ln E[e^{\\lambda W}] \\le \\lambda^2 v^2/(2(1-c\\lambda))$ for the chosen $v,c$ and all $\\lambda \\in (0,1/c)$; if it fails, or if the empirical coverage of the interval over many datasets falls persistently below the nominal $1-\\delta$, the load-bearing premise collapses.","supporting_citations":[{"cited_title":"Probabilistic methods in the geometry of banach spaces","cited_arxiv_id":null,"evidence_quote":"Supplies the Gaussian concentration inequality that bounds a differentiable function of normal vectors by its gradient norm, the central tool in the proof of Theorem 1."},{"cited_title":"Asymptotic statistics, volume 3","cited_arxiv_id":null,"evidence_quote":"Supplies the asymptotic normality of the MLE and the delta method that the paper's bound is presented as a non-asymptotic counterpart of."},{"cited_title":"An introduction to the bootstrap","cited_arxiv_id":null,"evidence_quote":"Defines the bootstrap method used as the empirical baseline whose coverage/width trade-off the proposed intervals are compared against."}],"review_version":1}