{"id":"5cf3373e-49d5-4a6e-8a7d-fef2f86fe463","arxiv_id":"2505.23225","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A geometric probability called epsilon-VCP rises as models overfit, and the paper offers it as a label-free overfitting diagnostic.","lead":"This paper introduces a number, the epsilon-valid counterfactual probability, that measures how easily a small perturbation can flip a model's prediction, and argues it goes up when the model overfits. The authors propose using this number as a label-free signal for detecting overfitting.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical estimator measures the full-ball crossing fraction, not the shell-conditional probability of Definition 4.1, so Figures 3–4 do not validate the theoretical quantity.","rationale":"The reader's weakest assumption lists two gaps: the unmeasured train-test generalization gap and the mismatch between the shell-based definition and the ball-based estimator. I consider the estimator mismatch the most load-bearing for the central claim, because the empirical curves are presented as validating Theorem 4.2, yet they compute a different quantity that differs by a factor epsilon^n/(epsilon^n - gamma_i^n). This factor is unbounded as gamma_i approaches epsilon, so even a perfect linear-classifier experiment would not reproduce the theoretical formula. The missing test accuracy is a related weakness, but the proxy claim would remain unsubstantiated even if a generalization gap were measured, as long as the measured quantity is not the one defined. A shell-restricted Monte Carlo recomputation with known or estimated gamma_i would settle whether the observed trend is an artifact of the full-ball estimator. I also note in passing that Theorem 4.3's stated local expansion appears dimensionally incorrect (gamma_i^{(n-1)/2} delta^{(n-1)/2} instead of (delta/gamma_i)^{(n-1)/2}), suggesting an algebra error in Appendix A.1; this does not affect the main proxy claim but adds to the need for revision. My recommendation remains CONDITIONAL, matching the reader's verdict, hence UNCHANGED.","tokens_in":14658,"tokens_out":16085,"duration_ms":153864,"concrete_test":"Re-run the Section 5 experiments with shell-restricted Monte Carlo sampling. For logistic regression, gamma_i is available in closed form; draw r with radius distributed proportional to r^{n-1} on [gamma_i, epsilon] and uniform direction, then record the fraction of label flips. For the MLPs, estimate gamma_i (e.g., by the Wu-Yu method) and apply the same shell sampling. Compare the resulting average epsilon-VCP curves to Figures 3b and 4a. If the trends no longer increase monotonically with training accuracy or no longer separate plain vs. dropout models, the empirical validation fails; if the trends match, the proxy claim survives but Section 5.1 still needs correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Definition 4.1 defines p_i^epsilon as the probability that a perturbation drawn from phi with supp(phi) = S(x_i, gamma_i, epsilon) yields a label change. Theorem 4.2 computes this shell-conditional probability. Section 5.1, however, estimates each p_i^epsilon as 'the fraction of the n-ball B(x_i, epsilon) that crosses the decision boundary,' i.e., P_{r~Uniform(B(x_i,epsilon))}[f(x_i+r) != f(x_i)]. These are different quantities: for a linear classifier the ball-based fraction is (1/2) I(1-(gamma_i/epsilon)^2; (n+1)/2, 1/2), while the shell-based value in Theorem 4.2 carries the extra factor epsilon^n/(epsilon^n - gamma_i^n), which diverges as gamma_i approaches epsilon. The empirical curves in Figures 3 and 4 therefore do not measure the quantity for which the paper derives a closed form. Because the central claim that average epsilon-VCP rises with overfitting is validated only through this mismatched estimator, the empirical support for the quantitative proxy is missing even if the geometric formula is correct.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the ε-valid counterfactual probability (ε-VCP), defined as the probability that a random perturbation drawn from a distribution supported on the shell S(x_i, γ_i, ε) between the geometric margin γ_i and radius ε flips the model's prediction. For linear classifiers under uniform shell perturbations, Theorem 4.2 derives a closed-form expression for p_i^ε as a decreasing function of γ_i. Theorem 4.3 offers a local asymptotic approximation for nonlinear models. The paper then connects average ε-VCP to overfitting through margin theory, claiming that overfitting reduces average margins and therefore increases average ε-VCP, and presents experiments on two tabular datasets with logistic regression and MLPs that plot average ε-VCP against training accuracy and epochs. The central assertion is that average ε-VCP can serve as a quantitative proxy for model overfitting.","tokens_in":14940,"tokens_out":6766,"duration_ms":67521,"significance":"If the central claim were fully established, the paper would make a novel connection between counterfactual explainability and generalization, providing a concrete geometric quantity with a clean closed form for linear classifiers and an open-source implementation. The derivation of Theorem 4.2 is a correct geometric calculation for the shell-conditional probability, and the paper is commendable for making the dependence of counterfactual validity on margin quantitative. However, the empirical validation does not measure the derived quantity and does not tie the observed trends to an actual generalization drop, so the significance is currently prospective rather than demonstrated.","major_comments":[{"comment":"The empirical estimator described in Section 5.1 computes the fraction of the full n-ball B(x_i, ε) that crosses the decision boundary, whereas Definition 4.1 and Theorem 4.2 define p_i^ε as a probability over the shell S(x_i, γ_i, ε). For a linear classifier, these two quantities differ by the factor ε^n/(ε^n - γ_i^n), which diverges as γ_i approaches ε. Consequently, Figures 3 and 4 do not validate the closed-form formula of Theorem 4.2, and the paper's empirical support for the proxy claim rests on a different quantity than the one analyzed theoretically. Please reconcile the estimator with the definition, for example by sampling only from the shell or by redefining ε-VCP and rederiving the theorem accordingly.","section":"§5.1 vs §4.1/§4.3.1"},{"comment":"The central claim that average ε-VCP is a quantitative indicator of model overfitting is not supported by the reported experiments because no test accuracy or train-test generalization gap is measured. Figures 3 and 4 only show that ε-VCP rises with training accuracy; without a measured overfitting signal, the trend cannot be attributed to overfitting rather than to ordinary improvement in data fitting. The paper should either report test performance for the same checkpoints/model pairs or explicitly limit the empirical claim to a margin-based trend rather than an overfitting proxy.","section":"§4.4 and §5.2"},{"comment":"The exponent of γ_i in Theorem 4.3 appears to have the wrong sign. Expanding the exact expression from Theorem 4.2 along the lines of Appendix A.1 yields p_i^ε ~ const · γ_i^{-(n-1)/2} δ^{(n-1)/2}, not γ_i^{(n-1)/2} δ^{(n-1)/2} as stated. The stated form also gives p_i^ε dimensions of length^{n-1}, which is inconsistent for a probability. Please correct Theorem 4.3 and its proof, and check the implications for the claimed limiting behavior.","section":"Theorem 4.3 and Appendix A.1"},{"comment":"The proof of monotonicity of g(γ̄) is not rigorous: it only examines endpoint limits, it uses the derivative dx/dγ̄ = -1/ε even though x = 1 - (γ̄/ε)^2 gives dx/dγ̄ = -2γ̄/ε^2, and it asserts without proof that the negative term dominates throughout the interior of the interval. Since Jensen's inequality in Equation (13) and the subsequent overfitting argument depend on the monotonicity direction of g, a complete proof or a precise reference is needed.","section":"Appendix A.2"}],"minor_comments":[{"comment":"The text refers to 'Appendix ??' when discussing the choice of ε and to 'Appendix ??' for full experimental details; these cross-references should be filled in.","section":"§5.1"},{"comment":"There is a typo: 'decreasining' should be 'decreasing'.","section":"Appendix A.2"},{"comment":"The phrase 'the the tangent hyperplane' contains a duplicated article and should be corrected.","section":"Theorem 4.3 proof sketch"},{"comment":"The columns for 'Exp. Water' and 'Exp. Air Quality' both list 'Fig. 4a'; the second should presumably be 'Fig. 4b'.","section":"Table 1"},{"comment":"The terms 'Plain ε-VCP' and 'Regularized ε-VCP' are used in the figure but not defined in the text; please clarify which model configuration each corresponds to.","section":"Figure 4 caption"},{"comment":"The description of X_i as indicating 'whether there exists an ε-valid counterfactual example' conflates deterministic existence with the probability of finding one under random perturbation; the Bernoulli framing would be clearer if it referred to the event that a random perturbation drawn from φ yields a label change.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is built on two load-bearing assumptions that are currently not validated: the shell-based definition versus the ball-based estimator, and the margin-theory bridge to overfitting. The geometric theorem appears correct in isolation, and the intended contribution is publishable if the authors re-run experiments with a shell-based estimator and report generalization measures. I would also recommend double-checking the exponent in Theorem 4.3, as it may indicate a deeper slip in the asymptotic expansion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know about arXiv:2505.23225: it offers a clean geometric quantity, ε-VCP, with a correct closed form for linear classifiers and a plausible local asymptotic for non-linear ones. The paper's real value is the theory; its empirical support is currently measuring the wrong quantity.\n\nThe ε-VCP is well-defined: probability that a random perturbation drawn from the shell S(x_i, gamma_i, epsilon) flips the label. Theorems 4.1 and 4.2 are correct for uniform perturbations on the shell, and the regularized incomplete beta expression is new as far as I can tell. Theorem 4.3's local expansion is also plausible and the appendix gives a real derivation. That is worth having in the literature.\n\nThe soft spots are not subtle. In Section 5.1 the estimator is described as the fraction of the full n-ball B(x_i, epsilon) that crosses the decision boundary, i.e., random perturbations uniformly inside the ball, not on the shell. For a linear classifier the ball-based fraction is (1/2) I(1 - gamma_i^2/eps^2; ...) without the ε^n/(ε^n - γ_i^n) factor. So Figures 3 and 4 validate a different object from the one Theorem 4.2 derives. The distinction matters; as γ_i approaches ε the two diverge substantially. The paper never reconciles this.\n\nThe second weakness is that the proxy claim is not tested against actual generalization. They show training accuracy rises along with ε-VCP, and cite margin theory to equate that with overfitting. But there is no test accuracy, no train-test gap, and the dropout comparison in Figure 4 does not show that the regularized model generalizes better on these datasets. So the headline \"average ε-VCP is a proxy for overfitting\" is supported only by the geometric monotonicity, which is fairly unsurprising: points closer to the boundary have more crossing directions.\n\nMinor: the monotonicity proof in Appendix A.2 only checks derivative signs at the endpoints and then asserts the interior behaves likewise. It is probably true, but the argument is not complete.\n\nOverall, this is a solid geometric note that overclaims its empirical reach. With a corrected estimator (sampling the shell) and a real look at generalization gaps, it could be a useful paper. I'd send it to review, but the referee should require those fixes before acceptance.","headline":"Useful closed-form geometry for a new counterfactual-ease metric, but the experiments measure a different quantity than the theory and never tie the trend to an actual generalization gap.","tokens_in":15419,"tokens_out":4865,"would_cite":true,"duration_ms":43582,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the average probability of finding a valid counterfactual within a fixed radius rises as models overfit, so that this probability can serve as a quantitative overfitting proxy.","keywords":["counterfactual examples","epsilon-valid counterfactual probability","overfitting","geometric margin","decision boundary","model generalization","margin theory","explainability"],"falsifier":"Train a model with a held-out test set and record the average ε-VCP alongside the train–test accuracy gap at regular checkpoints: if the average ε-VCP rises while the test gap stays flat or shrinks, the proxy is not tracking overfitting. The paper's plots show training accuracy and margins but not test accuracy, so this measurement is the decisive missing check.","tokens_in":14466,"feed_emoji":"🎯","tokens_out":8358,"duration_ms":79289,"temperature":0.7,"pith_summary":"The paper introduces the ε-valid counterfactual probability (ε-VCP): the chance that a random perturbation of a data point, sampled within a shell of radius ε, flips the model's prediction. It argues that this probability, averaged over the training set, goes up as training accuracy rises and geometric margins shrink, and therefore that the average ε-VCP can serve as a quantitative indicator of overfitting. If the claim holds, practitioners could track overfitting with a measure that is local, explanation-aware, and computable during training, without waiting for test-set errors. The supporting analysis is geometric: a closed-form formula ties ε-VCP to the margin of a point under a linear decision boundary, and a local version extends this to nonlinear boundaries. Experiments on two tabular datasets with logistic regression and multi-layer perceptrons show the average ε-VCP increasing through training and being larger in unregularized than in dropout-regularized models.","feed_headline":"Counterfactuals get easier as models overfit","feed_subtitle":"A new measure, ε-VCP, rises when training margins shrink, giving a quantitative overfitting signal.","key_machinery":"The central object is the ε-valid counterfactual shell $S(x_i,\\gamma_i,\\varepsilon)=B(x_i,\\varepsilon)\\setminus B(x_i,\\gamma_i)$, the thin layer of points at distance between the margin and ε from $x_i$, which is the only region where valid counterfactuals can appear. The argument runs through the ratio of two volumes: the spherical cap of directions that cross the decision boundary divided by the shell volume, expressed in Theorem 4.2 as a regularized incomplete $\\beta$ function of $1-(\\gamma_i/\\varepsilon)^2$. That formula converts boundary geometry into probability, and Jensen's inequality transfers monotonicity from individual margins to the average margin, which is the step that turns geometry into an overfitting indicator.","core_discovery":"On the paper's own terms, the central discovery is that ε-VCP is controlled by the distance from a point to the decision boundary. For a linear classifier with uniform perturbations, a point at margin $\\gamma_i$ has $\\varepsilon$-VCP $p_i^\\varepsilon = \\frac{1}{2}\\frac{\\varepsilon^n}{\\varepsilon^n-\\gamma_i^n} I\\left(1-(\\gamma_i/\\varepsilon)^2; \\frac{n+1}{2},\\frac{1}{2}\\right)$, where $I$ is the regularized incomplete $\\beta$ function; this is the volume fraction of the $\\varepsilon$-shell that lies across the separating hyperplane. The probability tends to $1/2$ as the margin goes to zero and vanishes as the shell collapses. Averaging over training points and applying Jensen's inequality gives a lower bound in which the average ε-VCP is a monotonically decreasing function of the average margin, and the paper cites margin theory to equate shrinking average margins with overfitting. For nonlinear boundaries, a tangent-hyperplane approximation yields an asymptotic form proportional to $\\gamma_i^{(n-1)/2}(\\varepsilon-\\gamma_i)^{(n-1)/2}$. The empirical sections record average ε-VCP rising with training epochs and accuracy, and lower values for dropout-regularized networks.","pith_inferences":["Beyond the paper, any training intervention known to increase average margins—weight decay, early stopping, adversarial training—should also suppress average ε-VCP; this prediction is testable and would separate margin effects from raw accuracy effects.","On data lying on a low-dimensional manifold, the same shell-volume argument should hold with intrinsic instead of Euclidean volumes, suggesting a manifold-aware version of the proxy.","Applied to test or out-of-distribution points, high ε-VCP could double as a local predictive-instability or uncertainty score, since boundary proximity is a standard correlate of low confidence.","If ε-VCP is to become a general overfitting proxy, the missing direct measurement for the paper's own models is the train–test accuracy gap; a study that tracks ε-VCP alongside test error would establish or refute the proxy on its own terms."],"forward_implications":["During training, the average ε-VCP can be monitored as an overfitting signal without waiting for a held-out test set.","Regularization that smooths the decision boundary, such as dropout, measurably lowers the average ε-VCP, so the measure tracks boundary complexity rather than raw accuracy.","For linear classifiers the closed-form identity makes the proxy a direct margin statistic, computable without Monte Carlo sampling.","The trade-off implies that easy counterfactual generation is not an unmixed good: it can signal that a model has wrapped itself tightly around the training data."],"supporting_citations":[{"why":"Supplies the counterfactual-example framing the paper builds on: a minimal perturbation that changes the prediction.","marker":"Wachter et al. [2017]"},{"why":"Anchor of the margin-theory bridge: overfitting is tied to reduced training margins and margin optimization improves generalization.","marker":"Mason et al. [1998]"},{"why":"Supports the claim that overfitting in deep networks is accompanied by shrinking margins.","marker":"Neyshabur et al. [2017]"},{"why":"Provides both the average-margin shrinkage result and the margin-estimation method used for the MLP training curves.","marker":"Wu and Yu [2019]"},{"why":"Adds the dimpled-manifold picture of adversarial examples, further grounding the margin-overfitting link.","marker":"Shamir et al. [2021]"},{"why":"Gives the hyperspherical-cap volume formula needed for the closed-form ε-VCP expression.","marker":"Li [2010]"},{"why":"Supplies the Water Potability dataset used in the empirical validation.","marker":"Kadiwal [2020]"},{"why":"Supplies the Air Quality dataset used in the empirical validation.","marker":"Vito [2008]"}],"fun_headline_variants":["Overfitting boosts counterfactual success","Easier counterfactuals signal overfitting","Shrinking margins, easier counterfactuals","ε-VCP: a quantitative overfitting signal","Generalization vs explainability: a real trade-off"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The overfitting conclusion rests on the imported margin-theory claim that overfitting shrinks average training margins, and on treating the empirical fraction of the ε-ball that crosses the boundary as a faithful estimate of the shell-based ε-VCP; if either fails, a rising average ε-VCP may only track training accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Overfitting boosts counterfactual success","Easier counterfactuals signal overfitting","Shrinking margins, easier counterfactuals","ε-VCP: a quantitative overfitting signal","Generalization vs explainability: a real trade-off"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1600,"prompt_tokens":930,"completion_tokens":670,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":614}},"tokens_in":546,"tokens_out":670,"duration_ms":6821,"temperature":1.0,"reasoning_tokens":614,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:50:56.988581+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a model with a held-out test set and record the average ε-VCP alongside the train–test accuracy gap at regular checkpoints: if the average ε-VCP rises while the test gap stays flat or shrinks, the proxy is not tracking overfitting. The paper's plots show training accuracy and margins but not test accuracy, so this measurement is the decisive missing check.","supporting_citations":[],"review_version":1}