{"id":"2f9b10b1-9f00-46e7-b123-8e58ab0e7bcb","arxiv_id":"2501.02353","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Weighted ERM with margin or inverse-variance weights tightens conditional risk bounds on high-confidence subregions by a constant factor, and yields an O(1/n) rate for estimating heteroscedastic variance.","lead":"This paper studies weighted empirical risk minimization, where each training sample is reweighted by its confidence before fitting. It proves that reweighting by the margin in classification, or by the inverse noise variance in regression, improves conditional risk bounds on high-confidence subregions compared with standard ERM. The claimed improvement is a constant factor, not a rate change, but it comes with a new fast rate for estimating the variance function.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Classification weight-estimation theorem targets the wrong conditional mean under the paper's ±1 label model, so Corollary 1's classification end-to-end guarantee does not follow as written.","rationale":"The reader's weakest assumption identifies the same load-bearing defect: Theorem 4.3's squared-loss estimator over ±1 labels does not recover η*, and the derived bω is not ε-close to the margin ω*. I added the concrete arithmetic check (η*=0.8 gives bω=0.1 vs ω*=0.6) and the factor-of-two inconsistency in the paper's two definitions of ω* (Section 4.1 states both ω*=2P(y=f*(x)|x)-1 and ω*≡|η*-1/2|, which differ by a factor of 2). This is a genuine correctness risk in a central advertised application, not a stylistic issue, and it is fixable by coding labels as 0/1 (or regressing 2y-1) and rescaling bω. The oracle-weight conditional bounds and the regression-side variance learning rate appear sound and are not affected by this defect. Since the reader already assigned CONDITIONAL and my analysis supports that verdict without moving it, the appropriate recommendation is UNCHANGED.","tokens_in":34487,"tokens_out":6998,"duration_ms":63405,"concrete_test":"Take X to be a single point, η*(x)=0.8, with F and G containing constant functions and labels drawn as y=1 with probability 0.8, y=-1 with probability 0.2, per DGP (5). The population squared-loss minimizer over constant η∈[0,1] is E[y]=0.6, so bω=|0.6-0.5|=0.1 while ω*=2|0.8-0.5|=0.6, giving Ex[(bω-ω*)²]=0.25, which violates the ε-approximation condition for any ε<0.25 used in Theorem 4.3 and Corollary 1. Independently re-derive the key display in Appendix A.5, replacing the assertion 'η*(x) ≜ E[y]' with the correct E[y|x]=2η*(x)-1; the equality ¯P∆R,bη = Ex[|bη-η*|²] becomes Ex[(bη-(2η*-1))²], showing that the theorem's conclusion measures distance to the wrong function.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The end-to-end classification claim (Corollary 1) combines Theorem 4.1 with Theorem 4.3, which asserts that the squared-loss minimizer bη = argmin (1/n)Σ(η(x_i)-y_i)² over η∈G satisfies Ex[(bη-η*)²]≤ε. Under the paper's own DGP (5), y|x ∈ {-1,1} with P(y=1|x)=η*(x), so the population squared-loss minimizer over η∈[0,1] is the projection of E[y|x]=2η*(x)-1 onto [0,1], not η*(x). The proof in Appendix A.5 asserts 'η*(x) ≜ E[y]', which is false for ±1 labels. Consequently bω=|bη-1/2| does not approximate the margin ω*(x)=2P(y=f*(x)|x)-1=2|η*(x)-1/2|. For a constant η*=0.8, bη→0.6 and bω→0.1, whereas ω*=0.6; the gap is not a constant factor but a change in which points are upweighted. Even after a rescaling of bω by 2, the estimated weight would be |2bη-1|, which converges to |2(2η*-1)-1|=|4η*-3|, not to the margin. Therefore the hypothesis Ex[(bω-ω*)²]≤ε required by Theorem 4.1 is not met, and Corollary 1's conclusion does not follow. The oracle-weight conditional bounds and the regression-side variance estimation are not affected by this defect, but the classification path advertised in the abstract and Table 1 is a central application.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies weighted empirical risk minimization in which each sample's loss is multiplied by a data-dependent weight, and it proves conditional excess-risk bounds on sub-regions where the weight is large. In binary classification the weight is the margin function and the sub-region is the large-margin set; in heteroscedastic regression the weight is the inverse conditional variance and the sub-region is the low-variance set. The main theoretical results are oracle-weight bounds (Theorems 4.1 and 4.5), weight-estimation results (Theorems 4.3 and 4.6), a general balanceable-Bernstein-condition result (Theorem 4.7), a lower bound for classification (Theorem 4.4), and an ERM-versus-weighted-ERM construction (Theorem 4.2). The paper reports an improved constant by a factor of order gamma/c in the selected regions and an O(1/n) rate for learning the variance function, together with synthetic experiments.","tokens_in":34651,"tokens_out":13834,"duration_ms":135845,"significance":"If the results were fully valid, the paper would make a useful contribution: the local Rademacher analysis under an epsilon-additive Bernstein condition is a reasonable technical tool, the oracle-weight inequalities in (6) and (13) are clean algebraic verifications, and an O(1/n) sample complexity for estimating the inverse variance function would improve on the earlier O(1/sqrt(n)) result in [65]. The conditional-risk improvement in high-margin and low-variance regions is a genuinely interesting refinement of standard ERM bounds. However, the advertised end-to-end classification guarantee currently rests on a weight-estimation theorem that does not estimate the object it claims to estimate under the paper's own +/-1 label model. This makes the classification application, which is central to the abstract and Table 1, not established as written, while the oracle-weight and regression-side analyses remain credible.","major_comments":[{"comment":"Theorem 4.3 estimates the wrong conditional mean. Under DGP (5), y takes values in {-1,1} with P(y=1|x)=eta*(x), so E[y|x]=2eta*(x)-1, not eta*(x). The squared-loss population minimizer over eta in [0,1] is therefore the projection of 2eta*(x)-1 onto [0,1], not eta*(x). The proof in Appendix A.5 asserts 'as eta*(x) := E[y]', which is false for this label model. Consequently the claimed bound Ex[(b_eta-eta*)^2] <= epsilon is not valid, and b_omega=|b_eta-1/2| does not approximate the margin. For a constant eta*=0.8, b_eta converges to 0.6, so b_omega converges to 0.1, whereas the margin omega*=2|eta*-1/2| is 0.6. Corollary 1, which combines Theorem 4.1 with Theorem 4.3, therefore does not follow from the paper's own equations, and the classification entries in the abstract and Table 1 are not supported.","section":"Section 4.1, Theorem 4.3, Corollary 1, Appendix A.5"},{"comment":"The definition of the margin function is internally inconsistent by a factor of two. Under the +/-1 label model in (5), the correct margin is omega*(x)=2P(y=f*(x)|x)-1=2|eta*(x)-1/2|=|2eta*(x)-1|, and this is what the derivation in Appendix A.1 actually yields. However, the text after (5) and in Theorem 4.3 defines omega*=|eta*-1/2|, and Theorem 4.3 constructs b_omega=|b_eta-1/2|. The two conventions are not equivalent. Even if the label-encoding issue in Theorem 4.3 were repaired by using {0,1} labels, the proposed b_omega=|b_eta-1/2| would converge to |eta*-1/2|, which is half of the margin stated in Table 1. The paper needs a single consistent definition of omega* and a weight estimator that targets that definition.","section":"Section 4.1 and Table 1; Appendix A.1"}],"minor_comments":[{"comment":"The proof of Theorem 4.3 also asserts Var[Delta R,eta] <= 2E[Delta R,eta] without a displayed calculation. Under the +/-1 label model, |y-eta(x)| can be as large as 2, and the variance-to-mean ratio is not bounded by 2 in the way claimed; this is part of the same label-encoding problem and should be corrected explicitly.","section":"Appendix A.5"},{"comment":"In the displayed decomposition of the excess risk, the first term should be multiplied by P[x in X1], not P[x in X2]; as printed, the decomposition is arithmetically wrong even before the subsequent lower-bound argument.","section":"Theorem 4.4 proof, Appendix A.6"},{"comment":"The classification experiments estimate the margin using cross-entropy, whereas Theorem 4.3 is stated for squared-loss minimization. The experiments therefore do not directly validate the theoretical weight-estimation result, and the mismatch should be acknowledged or the theory extended to the cross-entropy procedure.","section":"Section 5.2"},{"comment":"The final step of the proof of Theorem 4.6, converting the excess negative-log-likelihood bound into the claimed L2 bound on (1/b_sigma^2 - 1/sigma*^2), is only sketched with the phrase 'one can leverage' an inequality. Since the sample-complexity statement is advertised as an O(1/n) improvement over [65], the conversion should be written out with all constants, including the dependence on c3, gamma, and the bound on xi.","section":"Theorem 4.6 and Appendix A.9"},{"comment":"The displayed sample complexity contains the artifact 'T1T 3 2', which appears to mean T1 T2^3, and the statement mixes eF, eG with the stated dP(F), dP(G) without defining the resulting constants. This should be cleaned up.","section":"Notation in Theorem 4.6"}],"recommendation":"major_revision","confidential_remarks":"The classification end-to-end guarantee is the advertised headline of the paper, and its failure under the paper's own label model is a load-bearing defect. That said, the defect is localized: the oracle-weight bounds and the regression-side analysis are separable, and the classification path can likely be repaired by changing the label encoding to {0,1}, or by estimating E[y|x] and defining b_omega from the corrected mean. I would not recommend rejection if the authors fix the label-encoding/target mismatch and the margin-definition inconsistency; the remaining results would then be publishable. The factor-of-two issue in the margin definition is not merely cosmetic because it changes the claimed weight approximation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core idea is sound and worth engaging with. The balanceable Bernstein condition is a nice device: weighting by margin or inverse variance removes the 1/γ factor, and the conditional-risk improvement on large-margin/low-variance regions is a clean constant-factor win over plain ERM. The oracle-weight bounds in Theorems 4.1, 4.5, and 4.7 follow from a competent local Rademacher argument; the algebra in the appendix checks out for the chosen weight. The regression-side variance-learning rate in Theorem 4.6, O(1/n) versus the earlier O(1/√n), is a genuine improvement and appears to be correctly derived.\n\nThe soft spot is real and load-bearing for the classification application. Theorem 4.3 claims that the squared-loss minimizer bη over η ∈ [0,1] estimates η*(x) = P(y=1|x). But with labels in {-1,1} as in the paper's own DGP (5), the population minimizer of E[(η(x)-y)^2] is E[y|x] = 2η*(x)-1, not η*(x). The proof in Appendix A.5 literally asserts 'η*(x) ≜ E[y]', which is false. Consequently bω = |bη - 1/2| does not approximate the margin; for a constant η* = 0.8 it converges to 0.1 while the margin is 0.6, and the discrepancy changes which points are upweighted. Corollary 1, the end-to-end classification guarantee advertised in the abstract and Table 1, does not follow from the paper's own equations. This is fixable — regress on 0/1 labels or use a proper loss, and rescale the weight to match the 2|η* - 1/2| margin — but as written it is a broken step, not a cosmetic one.\n\nThere is also a smaller consistency issue: the margin is defined as 2P(y=f*(x)|x)-1 in the intro and Table 1, but Section 4.1 states ω*(x) ≡ |η*(x)-1/2|, which differs by a factor of two. That should be cleaned up.\n\nThe regression path and the oracle-weight conditional bounds are unaffected, and the synthetic experiments are honest. This paper deserves a serious referee: the core message is defensible and the variance-learning result is useful on its own, but the classification weight-estimation proof must be corrected before the stated claims are supported. I would not cite the classification end-to-end result in its current form, but I would cite the variance-learning rate and the conditional-risk framework once fixed.","headline":"The conditional-risk bound and O(1/n) variance-learning rate are real contributions, but the classification weight-estimation theorem targets the wrong conditional mean, so Corollary 1 does not follow as written.","tokens_in":35405,"tokens_out":3422,"would_cite":true,"duration_ms":32219,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","62G08","62G05"],"pacs":[],"model":"deepseek-v4-flash","headline":"Weighted ERM provably improves conditional risk bounds on large-margin and low-variance regions by a factor gamma/c.","keywords":["weighted empirical risk minimization","conditional excess risk","Bernstein condition","margin condition","heteroscedastic regression","selective classification","local Rademacher complexity"],"falsifier":"For the classification pipeline, fit $\\hat\\eta$ by squared loss on $\\pm 1$ labels under the DGP of Section 5.2 and compute the population minimizer: if it equals $2\\eta^*-1$, then $|\\hat\\eta-1/2|$ does not converge to $\\omega^*$, and $\\mathbb{E}[(\\hat\\eta-\\eta^*)^2]$ will not shrink to zero as $n$ grows. For the main reweighting bound, compare weighted and unweighted ERM conditional excess risk on $\\{\\omega^*>c\\}$ at fixed sample size: the theory predicts a $\\gamma/c$ improvement that should be visible in the risk gap.","tokens_in":34060,"feed_emoji":"⚖️","tokens_out":7556,"duration_ms":75172,"temperature":0.7,"pith_summary":"This paper argues that empirical risk minimization can be improved on selected sub-regions by multiplying each loss term by a data-dependent weight: the classification margin in classification, and the inverse conditional variance in regression. Under a \"balanceable\" Bernstein condition, weighted ERM replaces the usual conservative constant $B=1/\\gamma$ by $B=1$, yielding conditional excess-risk bounds of order $\\varepsilon/(c\\,P(\\omega^*(x)>c))$ instead of $\\varepsilon/(\\gamma\\,P(\\omega^*(x)>c))$. These sub-regions are the large-margin sets and the low-variance sets, and the improvement is a factor $\\gamma/c$. The paper also derives an $O(1/n)$ rate for learning the variance function, improving an earlier $O(1/\\sqrt n)$ guarantee, and gives a lower-bound construction showing the conditional bound is tight.","feed_headline":"Reweighting tightens risk bounds in high-confidence regions","feed_subtitle":"Weighted ERM with margin or inverse-variance weights cuts conditional error by a factor gamma/c.","key_machinery":"The balanceable Bernstein condition (Assumption 1 of the paper):\n$$\\mathrm{Var}[\\omega(x)(\\ell(f;z)-\\ell(f^*;z))] \\le \\mathbb{E}[\\omega(x)(\\ell(f;z)-\\ell(f^*;z))].$$\nThe weight is designed so that the variance-to-mean ratio is $1$, eliminating the conservative $B=1/\\gamma$ factor used in standard Bernstein-type analyses. The proofs re-run the local Rademacher complexity machinery of [6] under an $\\varepsilon$-additive version of this condition, so that a plug-in estimate $\\hat\\omega$ can be used in place of $\\omega^*$.","core_discovery":"The central claim is that, for losses satisfying a balanceable Bernstein condition, the weighted ERM estimator that minimizes $\\sum_i \\omega(x_i)\\ell(f;z_i)$ achieves conditional excess risk\n$$\\mathbb{E}_z[\\ell(\\hat f;z)-\\ell(f^*;z)\\mid \\omega^*(x)>c] \\le \\frac{\\varepsilon}{c\\,P(\\omega^*(x)>c)}$$\nwith high probability, improving on the standard ERM bound by the factor $\\gamma/c$. The weight $\\omega^*$ is the margin $2P(y=f^*(x)\\mid x)-1$ in classification and $1/\\sigma^{2*}(x)$ in regression. The improvement comes from balancing the Bernstein inequality as $\\mathrm{Var}[\\omega^* h] \\le \\mathbb{E}[\\omega^* h]$, which removes the $1/\\gamma$ multiplier from the variance-to-mean bound. In classification the paper proves the conditional bound is minimax-optimal via a matching lower bound; in regression it proves the variance function can be estimated at rate $O(1/n)$.","pith_inferences":["The balanceable-condition template suggests a soft-abstention view: instead of hard-excluding low-margin points, continuous down-weighting by the margin could yield similar selective gains while keeping some coverage; the paper draws this analogy but does not analyze the coverage-versus-risk trade-off in detail.","The claimed $O(1/n)$ variance-learning rate should extend to other NLL-based heteroscedastic models satisfying the same bounded-residual condition; a natural stress test is to add misspecification and check whether the rate degrades to $O(1/\\sqrt n)$.","A corrected margin estimator for the classification pipeline would regress on $(y+1)/2\\in\\{0,1\\}$, whose population minimizer is exactly $\\eta^*(x)$, or would directly form $|2\\hat\\eta(x)-1|$ from a calibrated probability estimate; using such an estimator would make the end-to-end classification result follow from the paper's own equations."],"forward_implications":["In classification, with $\\tilde\\Theta(1/\\varepsilon)$ samples, the weighted ERM bound on the large-margin region $\\{\\omega^*(x)>c\\}$ is $\\varepsilon/(c\\,P(\\omega^*(x)>c))$ versus $\\varepsilon/(\\gamma\\,P(\\omega^*(x)>c))$ for ERM; whenever $c>\\gamma$ this is a strict improvement.","Under the low-margin diminishing condition $P(\\omega^*(x)\\le c)\\,c^2\\le\\varepsilon$, the unconditional excess risk bound becomes $\\varepsilon/c$, a $\\gamma/c$ improvement over standard ERM.","In heteroscedastic regression, weighted ERM attains $\\mathbb{E}_x[(1/\\sigma^{2*}(x))(\\hat f(x)-f^*(x))^2]\\le\\varepsilon$ with $\\tilde\\Theta(1/(\\gamma\\varepsilon))$ samples, whereas ERM needs $\\tilde\\Theta(1/(\\gamma^2\\varepsilon))$ samples; the gain is again a factor $1/\\gamma$.","The variance function $\\sigma^{2*}(x)$ can be learned at rate $O(1/n)$ through negative log-likelihood estimation, improving over the earlier $O(1/\\sqrt n)$ bound and making the variance-estimation step no harder than learning the mean function.","The lower-bound construction shows that the conditional excess-risk bound for weighted ERM on $\\{\\omega^*(x)>c\\}$ is tight among all estimators, so the improvement over ERM is not an artifact of a loose analysis."],"supporting_citations":[{"why":"Supplies the local Rademacher complexity theorem (Theorem 3.3) and the Bernstein-type analysis that the weighted ERM proofs adapt with an epsilon-additive condition.","marker":"[6]"},{"why":"Provides the standard ERM conditional risk bound and the minimax lower-bound framework that the paper compares against and uses in its lower-bound construction.","marker":"[39]"},{"why":"Prior work by the authors that established the $O(1/\\sqrt n)$ rate for variance-function learning, which Theorem 4.6 improves to $O(1/n)$.","marker":"[65]"},{"why":"Gives the derivation connecting classification excess risk to the margin-weighted disagreement $\\mathbb{E}_x[\\omega^*(x)\\mathbf{1}\\{f\\neq f^*\\}]$, a step used in the main classification argument.","marker":"[7]"},{"why":"Establishes fast classification rates with a rejection option, which the paper positions as the hard-decision counterpart to its soft reweighting schema.","marker":"[8]"},{"why":"Provides the covering-number bound for VC/pseudo-dimension classes used in the entropy-integral estimates of the proof.","marker":"[24]"}],"fun_headline_variants":["Weighted ERM sharpens conditional risk bounds in key regions","Margin and variance weights tighten risk guarantees","Reweighting reduces conditional error in key subregions","Adaptive weights improve conditional risk bounds","Weighted ERM tightens conditional risk for margin and low variance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The end-to-end result assumes a weight estimate $\\hat\\omega$ with $\\mathbb{E}[(\\hat\\omega-\\omega^*)^2]\\le\\varepsilon$ can be obtained at sample size $O(d\\log(1/\\varepsilon)/\\varepsilon)$; for classification, the paper constructs $\\hat\\omega=|\\hat\\eta-1/2|$ where $\\hat\\eta$ minimizes squared loss on $\\pm 1$ labels, and that estimator actually converges to $|2\\eta^*-3/2|$, not to the margin $\\omega^*=2|\\eta^*-1/2|$, so the stated derivation of the classification weight estimate is the load-bearing step that needs repair.","fun_headline_variants_meta":{"raw":{"variants":["Weighted ERM sharpens conditional risk bounds in key regions","Margin and variance weights tighten risk guarantees","Reweighting reduces conditional error in key subregions","Adaptive weights improve conditional risk bounds","Weighted ERM tightens conditional risk for margin and low variance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000484,"raw_usage":{"total_tokens":2352,"prompt_tokens":867,"completion_tokens":1485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":1409}},"tokens_in":483,"tokens_out":1485,"duration_ms":9878,"temperature":1.0,"reasoning_tokens":1409,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:15:57.513301+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For the classification pipeline, fit $\\hat\\eta$ by squared loss on $\\pm 1$ labels under the DGP of Section 5.2 and compute the population minimizer: if it equals $2\\eta^*-1$, then $|\\hat\\eta-1/2|$ does not converge to $\\omega^*$, and $\\mathbb{E}[(\\hat\\eta-\\eta^*)^2]$ will not shrink to zero as $n$ grows. For the main reweighting bound, compare weighted and unweighted ERM conditional excess risk on $\\{\\omega^*>c\\}$ at fixed sample size: the theory predicts a $\\gamma/c$ improvement that should be visible in the risk gap.","supporting_citations":[{"cited_title":"L., Bousquet, O., and Mendelson, S","cited_arxiv_id":null,"evidence_quote":"Supplies the local Rademacher complexity theorem (Theorem 3.3) and the Bernstein-type analysis that the weighted ERM proofs adapt with an epsilon-additive condition."},{"cited_title":"and N´ed´elec, ´E","cited_arxiv_id":null,"evidence_quote":"Provides the standard ERM conditional risk bound and the minimax lower-bound framework that the paper compares against and uses in its lower-bound construction."},{"cited_title":"Risk bounds on aleatoric uncertainty recovery","cited_arxiv_id":null,"evidence_quote":"Prior work by the authors that established the $O(1/\\sqrt n)$ rate for variance-function learning, which Theorem 4.6 improves to $O(1/n)$."},{"cited_title":"Theory of classification: A survey of some recent advances","cited_arxiv_id":null,"evidence_quote":"Gives the derivation connecting classification excess risk to the margin-weighted disagreement $\\mathbb{E}_x[\\omega^*(x)\\mathbf{1}\\{f\\neq f^*\\}]$, a step used in the main classification argument."},{"cited_title":"and Zhivotovskiy, N","cited_arxiv_id":null,"evidence_quote":"Establishes fast classification rates with a rejection option, which the paper positions as the hard-decision counterpart to its soft reweighting schema."},{"cited_title":"Sphere packing numbers for subsets of the boolean n-cube with bounded Vapnik- Chervonenkis dimension","cited_arxiv_id":null,"evidence_quote":"Provides the covering-number bound for VC/pseudo-dimension classes used in the entropy-integral estimates of the proof."}],"review_version":1}