{"id":"496be44c-8653-4282-a795-b8ea75437130","arxiv_id":"2505.05242","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new active learning algorithm, FCCM, maximizes factual and counterfactual coverage and reduces treatment effect estimation error under limited labeling budgets.","lead":"When labeling treatment outcomes is expensive, this paper shows which data points to label in order to estimate treatment effects most accurately. It introduces a coverage-based selection rule, FCCM, that outperforms existing active learning methods on three benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.4's proof drops the empirical training loss via a zero-training-loss assumption that the DUE estimators used in the experiments do not satisfy, so the covering-radius bound is not connected to the reported risk reductions.","rationale":"The strongest claim is Theorem 3.4: that covering radii control the subset generalization gap. Reading the proof closely, the step from Eq. (13f) to Eq. (13g) is the linchpin: it asserts the left-hand side equals (1/n)Σξ(x_i; f_S) because the empirical training loss is zero. This is the same assumption introduced in Section 3 before Eq. (5), following Sener and Savarese. It is not an innocent technical condition. The experimental estimators are regularized deep kernel models with inducing points, dropout, and spectral normalization, so exact interpolation is not expected. If the training-loss term is positive, the theorem bounds only one of the two nonnegative terms in the gap, and the reducible covering-radius terms are not proven to move the PEHE. I therefore agree with the reader's weakest_assumption. I also note two additional proof gaps that are real but secondary: Lemma A.9's 'identical set' assertion under strong ignorability is unjustified because Algorithm 1's nearest-neighbor step does not produce an identical proxy set, and Eq. (47b) in Theorem 4.4 equates the mean of four normalized coverages with half the union coverage, which requires equal group sizes and consistent denominators that the paper does not establish. These concern the approximation guarantees of the specific algorithms, whereas the zero-loss assumption undermines the foundational risk bound itself. The empirical evaluation is solid and the code is released, which supports the claim that FCCM performs well in practice, but it does not repair the theory. Hence the reader's CONDITIONAL verdict should stand, pending either a repaired proof or an empirical demonstration that the dropped training-loss term is negligible and Eq. (6) holds on their benchmarks.","tokens_in":33686,"tokens_out":12607,"duration_ms":130654,"concrete_test":"Rerun the Section 5 acquisition loops with the released code and record, at each budget step, the training MSE of DUE-DNN/DUE-CNN on the selected subset S at the checkpoint used for evaluation. On the TOY benchmark, where potential outcomes are known, also compute the empirical subset generalization gap (the LHS of Eq. 6) and compare it with the RHS using measured covering radii and Lipschitz/IPM constants inferred from the fitted model. If training MSE is not below a small tolerance (e.g., 1e-6) and the LHS exceeds the RHS at any step, the equality in Eq. (13g) is load-bearing; if the inequality still holds with slack despite nonzero training loss, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The linchpin of Theorem 3.4 is the zero-training-loss assumption, introduced in Section 3 before Eq. (5) and used again as Eq. (13g) in Appendix A.1. To pass from the subset generalization gap to (1/n)Σξ(x_i; f_S), the proof relies on the equality (1/|S|)Σ l(x_j,y_j,t_j; f_S) = 0. Since the gap is an absolute value of two nonnegative terms, this equality holds only if the trained estimator exactly interpolates every selected outcome. That is a premise about the trained model, not about the geometry of the pool. The paper's own experiments train DUE-DNN/DUE-CNN, deep kernel models with inducing points, dropout, and spectral norm; exact interpolation of thousands of labeled points at the evaluation checkpoint is not expected. If the training-loss term is positive, Eq. (6) is not established as an upper bound on the actual subset generalization gap, and the four covering radii are not proven to control the reducible part of the PEHE. This is load-bearing because it is the step that converts the covering objective into a guarantee on treatment effect estimation risk.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies active learning for treatment effect estimation under a labeling budget, formalizing the subset selection problem as minimization of a PEHE risk upper bound. Its central theoretical contribution is Theorem 3.4, which bounds the subset generalization gap by the sum of four data-dependent quantities: factual covering radii δ(t,t) and counterfactual covering radii δ(t,1−t), plus a distributional discrepancy term and a finite-sample term. Based on this bound, the authors propose Algorithm 1, a greedy radius-reduction method with a 2-OPT guarantee, and Algorithm 2 (FCCM), a coverage-maximization method claimed to give a (1−1/e)-approximation. They evaluate FCCM on TOY, IBM, and CMNIST datasets against general and treatment-effect-specific active learning baselines, reporting lower PEHE in most settings and including ablation and sensitivity studies.","tokens_in":33874,"tokens_out":6236,"duration_ms":65204,"significance":"If the theoretical claims were sound, the paper would contribute a useful model-independent principle for data-efficient treatment effect estimation: acquisition should reduce factual and counterfactual covering radii rather than merely reduce uncertainty or diversity. The empirical evaluation is reasonably thorough, with multiple datasets, standard baselines, ablations, and released code, and the visualization of acquisition behavior supports the qualitative explanation of FCCM's gains. However, the main theoretical bridge from covering radii to PEHE risk, Theorem 3.4, relies on a zero-training-loss assumption that the experimental estimators do not satisfy, and the approximation guarantees in Theorems 4.1 and 4.4 contain gaps. These issues are central to the paper's stated contributions and need to be repaired before the theoretical framing can be accepted.","major_comments":[{"comment":"The proof of Theorem 3.4 drops the empirical training loss through the zero-training-loss assumption introduced before Eq. (5) and used again in Eq. (13g). In particular, the subset generalization gap is equated to (1/n)Σ_i ξ(x_i; f_S) only when (1/|S|)Σ_{j∈S} l(x_j,y_j,t_j; f_S) = 0, and the transition to Eq. (13c) also drops the expected subset losses ε_S^t. If the trained estimator does not interpolate the labeled outcomes, both terms are positive and the displayed expression is no longer an upper bound on the actual subset generalization gap. The estimators used in the experiments, DUE-DNN and DUE-CNN, are deep kernel models with inducing points, dropout, and spectral norm regularization, for which exact interpolation of the selected training points is not expected. Since Theorem 3.4 is the step that converts the covering-radius objective into a guarantee on treatment effect estimation risk, the paper needs either an argument that these estimators interpolate the selected data, an explicit training-loss term carried through the bound, or an alternative derivation that does not rely on zero training loss.","section":""},{"comment":"The proof of Lemma A.9 assumes that the proxy collection S-tilde_{1−t} has an identical counterpart in D_t, stating that strong ignorability guarantees an 'identical set' S_{(t,1−t)} ∈ D_t matching the proxy collection. Strong ignorability gives overlap of the covariate supports of the two treatment groups; it does not imply that every proxy point has an exact copy in the opposite treatment group. If the nearest factual point satisfies d(a^t, a^{1−t}) > 0, the equality d(u^{1−t}, S_{(t,1−t)}) = d(u^{1−t}, S-tilde_{1−t}) in Eq. (43a) fails, and the subsequent chain does not yield the claimed 2-OPT bound for the counterfactual radius. The theorem should state the exact-match condition explicitly or incorporate the approximation error of the nearest factual point into the bound.","section":""},{"comment":"The equality P(A) = (1/2)P(A_F^{t=1} ∪ A_CF^{t=1}) + (1/2)P(A_F^{t=0} ∪ A_CF^{t=0}) does not follow from the definition of P(A) in Eq. (9). For each group t, the identity P(A_F^t) + P(A_CF^t) = 2 P(A_F^t ∪ A_CF^t) requires that the factual and counterfactual cover sets are disjoint and that |D_t| = |D_{1−t}|, conditions that are neither stated nor generally true for the datasets used. Without this equality, the proof does not establish that maximizing the four individual coverage fractions in Eq. (9) is equivalent to maximizing the two union-coverage terms that the greedy analysis actually addresses.","section":""},{"comment":"The approximation guarantee in Theorem 4.4 is stated for an equally weighted graph with unscaled out-degree, but Algorithm 2 as defined uses counterfactual edge weight α and a scaled out-degree coefficient c(x) = ζ(x)(1−ζ(x)). The proof does not track either α or the scaling coefficient, so the (1−1/e) guarantee does not apply to the algorithm as implemented and evaluated. The paper should either prove the guarantee for the weighted, scaled objective actually optimized by FCCM or clearly state that the theorem covers a simplified variant and justify empirically that the modification preserves the approximation behavior.","section":""}],"minor_comments":[{"comment":"The symbol D is used both for the full pool set in Eq. (1) and for the training dataset in the definition of the estimator f_D at the start of Section 2; this conflation should be resolved with separate notation for the labeled and unlabeled pools.","section":""},{"comment":"The Hoeffding inequality is written as an equality; it should be an upper bound, P(·) ≤ exp(−2nε²/L_l²), and the variable ε here is overloaded with the risk notation used elsewhere.","section":""},{"comment":"The assumption states that the squared loss l is bounded by L_l, but a squared loss on unbounded outcomes is not bounded; the boundedness condition should be stated as an explicit restriction on the support of the outcomes or on the hypothesis class.","section":""},{"comment":"The comment in the sketch version that 'a is not labeled' is confusing because the algorithm outputs S as the set to be labeled; the distinction between the proxy point a′ and the factual query a should be stated more clearly in both the sketch and the full pseudocode.","section":""},{"comment":"Several figure captions are inconsistent, including two figures numbered 'Figure 4' and a caption for the CMNIST sensitivity plot that says 'TOY'; these should be corrected.","section":""}],"recommendation":"major_revision","confidential_remarks":"The empirical portion of the paper is solid and reproducible, and the covering-radius perspective is likely of interest to the active learning and causal inference communities. My main concern is that the central theoretical claims are currently not established as stated: the zero-training-loss assumption breaks the connection between covering radii and PEHE risk for the estimators used, and the proofs of the approximation guarantees have identifiable gaps. These appear fixable, but the revision needs to address them substantively rather than cosmetically."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this paper has a genuinely new idea and a solid empirical evaluation, but the theoretical guarantee that frames the algorithm is not actually established. The counterfactual covering radius is a natural extension of the core-set bound to treatment effect estimation, and FCCM's joint factual/counterfactual coverage objective performs consistently well on three benchmarks. That part is worth taking seriously.\n\nThe problems are in the proofs. Theorem 3.4's derivation drops the empirical training loss by assuming zero training loss in the same way as Sener & Savarese. That is a premise about the trained estimator, and the DUE models used in the experiments (deep kernel learning with inducing points, dropout, spectral norm) are not interpolating the labeled set. If the training loss is positive, the bound in Eq. (6) does not upper-bound the subset generalization gap, and the covering radii are not proven to control the reducible part of PEHE. This is load-bearing for the theory, though the heuristic could still work.\n\nTwo further proof gaps. Lemma A.9 claims strong ignorability lets the proxy collection be an 'identical set' to the queried factual points. Strong ignorability gives overlapping support in population, not exact duplicates in a finite sample, so the 2-OPT guarantee for counterfactual radii is not justified. And Eq. (47b) in the proof of Theorem 4.4 equates the mean of per-group coverage fractions with union coverage for each treatment group; that equality only holds in special cases (disjoint sets, common denominator), not in general. So the (1-1/e) approximation result as stated doesn't follow.\n\nMinor but relevant: no random acquisition baseline. Since the main claim is that the acquisition strategy matters, a random selector would make the comparison sharper. Sensitivity analysis is there, the ablated counterfactual term is useful, and code is public—those are real credits.\n\nWho should read this: anyone working on active learning for causal effect estimation. The empirical findings and the covering perspective are useful even if the proofs need repair. I'd send it to a serious referee, not desk reject, but the right decision is major revision: fix the zero-training-loss step or state it as a separate assumption and test its plausibility, correct the approximation proofs, and add the random baseline.","headline":"Genuinely new covering perspective with solid empirical gains, but the theory as written doesn't hold together: the core bound assumes zero training loss, and the approximation proofs have gaps.","tokens_in":34451,"tokens_out":4587,"would_cite":true,"duration_ms":45611,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The error of active treatment-effect estimation is governed by factual and counterfactual covering radii, so label acquisition is a covering problem.","keywords":["active learning","treatment effect estimation","counterfactual covering radius","factual covering radius","PEHE","coverage maximization","core-set","causal inference"],"falsifier":"On the TOY setting, run FCCM with a deliberately non-interpolating estimator (for example, a regularized neural network with non-negligible weight decay or early stopping). Record the sum of the four covering radii, the training loss on the labeled set, and the PEHE at each acquisition step; if radii decrease substantially while training loss stays well above zero and PEHE does not follow the bound's reductions, the claim that covering radius controls risk would be refuted in a concrete case.","tokens_in":33457,"feed_emoji":"🎯","tokens_out":12473,"duration_ms":110452,"temperature":0.7,"pith_summary":"When treatment outcomes are expensive to label, the paper claims, the error of a treatment-effect estimator trained on a small actively chosen subset is governed, with high probability, by how well the labeled points cover both treatment groups in covariate space. The reducible part of the risk bound is a sum of two kinds of covering radii: a factual radius within each treatment group and a counterfactual radius from each group to the other group's points. This reframes label acquisition as a covering problem: spend the labeling budget on points that shrink the largest radius. The paper proves this bound, gives a greedy radius-reduction algorithm with a factor-two approximation under idealized conditions, and proposes FCCM, a coverage-maximization variant for partially overlapping distributions. The stakes are practical: the same inequality says what information a labeling budget should buy to improve individualized treatment-effect estimates.","feed_headline":"Two covering radii govern active-learning treatment-effect error","feed_subtitle":"A new bound says choosing which patients to label is a covering problem; the FCCM algorithm acts on it.","key_machinery":"The load-bearing objects are the factual covering radius $\\delta(t,t)$ and the counterfactual covering radius $\\delta(t,1-t)$: the radius of the smallest ball centered at the labeled samples from treatment group $t$ whose union covers all pool samples of group $t$ (factual) or of the opposite group $1-t$ (counterfactual). These radii enter linearly in Theorem 3.4, so shrinking them is the only model-independent lever on the risk bound; the $\\kappa_H$ discrepancy term is fixed once the pool is given. The second mechanism is the greedy reduction procedure, which at each step selects the point that reduces the largest of the four radii, reducing to the k-center problem for the factual radii and to a cross-group covering problem for the counterfactual radii. FCCM replaces radius minimization with coverage maximization on a weighted directed graph whose edges connect each candidate center to points inside its factual and counterfactual balls; it repeatedly picks the highest-degree vertex and removes covered edges, which is the max-coverage machinery that yields the $1-1/e$ guarantee.","core_discovery":"The paper's central claim is Theorem 3.4: for an i.i.d. pool of $n$ covariate samples, with probability at least $1-\\gamma$, the subset generalization gap — the difference between the population PEHE risk and the empirical training loss on the selected labeled subset — is at most $\\sum_{t\\in\\{0,1\\}} \\kappa_t(\\delta(t,t)+\\delta(t,1-t)) + 2\\kappa_H + \\sqrt{L_l^2\\log(1/\\gamma)/(2n)}$, where $\\kappa_H$ is a distributional discrepancy (an integral probability metric) between the treated and control groups. Since the four covering radii decrease as the labeled set grows, the gap is $O(\\delta(1,1)) + O(\\delta(1,0)) + O(\\delta(0,0)) + O(\\delta(0,1))$. The paper therefore claims that the model-independent, reducible part of treatment-effect estimation risk under active learning is exactly a covering problem: label points so every pool point is close to a labeled point of its own group and close to a labeled point of the opposite group. It further proves that a greedy k-center-style radius-reduction algorithm is within a factor of two of the optimal sum of radii under strong ignorability, and that FCCM, which maximizes coverage under fixed radii, approximates full coverage to within $1-1/e$ under its assumptions.","pith_inferences":["One could apply the same radius objective to more than two treatments or to continuous treatments by defining $\\delta(t,t')$ for every pair of treatment regions; the paper only develops the binary case.","A testable extension is to track the four covering radii on the pool during acquisition and check whether they track PEHE; the paper does not implement such a diagnostic.","Because the bound assumes interpolating models, a natural follow-up is an acquisition rule that adds a weighted empirical training-loss term for non-interpolating estimators; the paper does not derive one.","For non-overlapping treatment groups, FCCM's acquisition is challenged; a latent-space variant that aligns the groups before covering would sacrifice some model independence, and the paper flags this as an open limitation."],"forward_implications":["If Theorem 3.4 is correct, any acquisition strategy that only maximizes diversity inside one treatment group leaves $\\delta(t,1-t)$ uncontrolled, so the risk bound need not shrink even as the labeled set grows.","Corollary 3.5 implies that, with a fixed pool, the subset generalization gap is $O(\\delta(1,1)) + O(\\delta(1,0)) + O(\\delta(0,0)) + O(\\delta(0,1))$; the largest radius is the right next acquisition target.","Theorem 4.1 says that under strong ignorability the greedy radius-reduction algorithm returns a sum of radii within a factor of two of optimal, so an idealized balanced distribution admits a near-optimal k-center-style acquisition rule.","Theorem 4.4 says FCCM is a $(1-1/e)$-approximation to full factual and counterfactual coverage under its assumptions, providing a tractable surrogate for radius reduction on realistic partially overlapping distributions.","In the paper's experiments, FCCM reaches lower PEHE than both general active learning baselines and treatment-effect-specific baselines on TOY, IBM, and CMNIST, and its acquisitions concentrate on high-density overlap regions."],"supporting_citations":[{"why":"Supplies the potential-outcome PEHE setting and the factual/counterfactual loss decomposition with the IPM distributional discrepancy that Theorem 3.4 extends.","marker":"Shalit et al., 2017"},{"why":"Supplies the core-set active-learning bound and the zero-training-loss assumption used to drop the empirical training loss in Appendix A.1.","marker":"Sener & Savarese, 2018"},{"why":"Gives the budgeted heterogeneous treatment effect estimation framework that the paper generalizes by adding the counterfactual covering radius.","marker":"Qin et al., 2021"},{"why":"Provides the Causal-BALD baselines (µBALD, ρBALD, µρBALD) and the uncertainty-based acquisition alternative compared in the experiments.","marker":"Jesson et al., 2021b"},{"why":"Provides the MACAL baseline and the experimental protocol; MACAL's pair-query constraint motivates FCCM's single-acquisition design.","marker":"Wen et al., 2025"},{"why":"Supplies the max-coverage view of active learning whose graph-covering construction FCCM adapts to two treatment groups with counterfactual edges.","marker":"Yehuda et al., 2022"},{"why":"Identifies the core-set/k-center formulation and its hardness, grounding the greedy radius-reduction strategy and its approximation guarantee.","marker":"Tsang et al., 2005"},{"why":"Provides the greedy approximation proof technique (cluster/pigeonhole argument) used in the proofs of Theorem 4.1 and Theorem 4.4.","marker":"Dinitz, 2019"},{"why":"Defines the PEHE metric that the paper uses as the risk criterion for individualized treatment effect estimation.","marker":"Hill, 2011"}],"fun_headline_variants":["Active learning for causal effects: it's a covering problem","Two covering radii determine active-learning treatment-effect error","FCCM: cover factual and counterfactual to cut treatment-effect error","Theoretical bound: labeling for effects is a counterfactual covering task","Why data-efficient effect estimation reduces to covering radii"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bound assumes the trained estimator achieves zero training loss on the selected labeled subset; if the model does not interpolate the labeled outcomes, the covering-radius terms no longer control the subset generalization gap, so the acquisition objective is not proven to reduce risk.","fun_headline_variants_meta":{"raw":{"variants":["Active learning for causal effects: it's a covering problem","Two covering radii determine active-learning treatment-effect error","FCCM: cover factual and counterfactual to cut treatment-effect error","Theoretical bound: labeling for effects is a counterfactual covering task","Why data-efficient effect estimation reduces to covering radii"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000309,"raw_usage":{"total_tokens":1807,"prompt_tokens":1028,"completion_tokens":779,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":705}},"tokens_in":644,"tokens_out":779,"duration_ms":7734,"temperature":1.0,"reasoning_tokens":705,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:10:40.919055+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the TOY setting, run FCCM with a deliberately non-interpolating estimator (for example, a regularized neural network with non-negligible weight decay or early stopping). Record the sum of the four covering radii, the training loss on the labeled set, and the PEHE at each acquisition step; if radii decrease substantially while training loss stays well above zero and PEHE does not follow the bound's reductions, the claim that covering radius controls risk would be refuted in a concrete case.","supporting_citations":[{"cited_title":"D., and Sontag, D","cited_arxiv_id":null,"evidence_quote":"Supplies the potential-outcome PEHE setting and the factual/counterfactual loss decomposition with the IPM distributional discrepancy that Theorem 3.4 extends."},{"cited_title":"and Savarese, S","cited_arxiv_id":null,"evidence_quote":"Supplies the core-set active-learning bound and the zero-training-loss assumption used to drop the empirical training loss in Appendix A.1."},{"cited_title":"Budgeted heterogeneous treatment effect estimation","cited_arxiv_id":null,"evidence_quote":"Gives the budgeted heterogeneous treatment effect estimation framework that the paper generalizes by adding the counterfactual covering radius."},{"cited_title":"K., Sadiq, S., and Yin, H","cited_arxiv_id":null,"evidence_quote":"Provides the MACAL baseline and the experimental protocol; MACAL's pair-query constraint motivates FCCM's single-acquisition design."},{"cited_title":"Active learning through a covering lens","cited_arxiv_id":null,"evidence_quote":"Supplies the max-coverage view of active learning whose graph-covering construction FCCM adapts to two treatment groups with counterfactual edges."},{"cited_title":"W., Kwok, J","cited_arxiv_id":null,"evidence_quote":"Identifies the core-set/k-center formulation and its hardness, grounding the greedy radius-reduction strategy and its approximation guarantee."},{"cited_title":"Lecture 4: Approximation algorithms - vertex cover and set cover","cited_arxiv_id":null,"evidence_quote":"Provides the greedy approximation proof technique (cluster/pigeonhole argument) used in the proofs of Theorem 4.1 and Theorem 4.4."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the PEHE metric that the paper uses as the risk criterion for individualized treatment effect estimation."}],"review_version":1}