{"id":"88cdd845-39f9-4f59-9936-557f2a38828a","arxiv_id":"2411.11256","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A distance-based batch active learning criterion that simultaneously promotes diversity and treated-control similarity reduces PEHE in causal effect estimation under limited labeling budgets, per experiments on IHDP, IBM, and CMNIST.","lead":"This paper proposes an active learning method to choose which patients or users to label when training a causal effect model under a small labeling budget. The method pairs similar treated and untreated samples while favoring diverse, high-uncertainty points, and the authors report lower prediction error than existing baselines on three datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's Eq. (15) rests on an unjustified greedy decomposition: per-step bound shrinkages depend on the accumulated dataset and model, so the union of per-step argmaxes is not generally the global argmax.","rationale":"The reader's weakest_assumption names exactly the right failure point: Eq. (15) treats per-step shrinkages as independent and unions as commutative, which is false in this setting. I agree with the REJECT verdict. The concern is load-bearing because Theorem 1 is the paper's advertised theoretical foundation; without it, MACAL is a reasonable Euclidean-distance heuristic but not a 'principled' algorithm with a proven optimal risk-reduction guarantee. I also note supporting weaknesses that reinforce the same conclusion without replacing it: Lemma 2's variance lower bound assumes the maximum predictive variance in a finite pool equals the GP signal variance σ_f^2, which is not guaranteed; Lemma 3 requires 'identical samples from two groups can always be found', an idealization the algorithm only approximates with nearest pairs, and its permutation-invariance step is not generally valid. Section C candidly admits the convergence analysis is limited to extreme C_φ regimes, which is consistent with the theoretical gap. Credit is due for open-sourced code and a broad empirical comparison, but the core theoretical claim is not established. A small exhaustive-search counterexample would settle the issue; if instead no greedy-failure sequence exists in the tested instances, that would only be evidence for those instances, since no condition in the paper rules out failure in general.","tokens_in":29203,"tokens_out":5908,"duration_ms":59255,"concrete_test":"Build a minimal finite-pool instance of the paper's GP variance-only setting (e.g., 6 pool points, batch size 1, 2 query steps, fixed RBF kernel) and enumerate all acquisition sequences exhaustively, updating the GP posterior after each step and computing ΔB_overall = B0 - B2 for each sequence. If there exist first-step batches A and B with ΔB_1(A) > ΔB_1(B) but ΔB_1(A) + max_{second|A} ΔB_2 < ΔB_1(B) + max_{second|B} ΔB_2, then Eq. (15) is false and Theorem 1 fails. Report the exact point coordinates and kernel parameters so the counterexample is reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 1 claims arg max_{D_overall} ΔB_overall = ∪_{i=1}^I arg max_{D_i} ΔB_i. The proof's step Eq. (15) is invalid. The telescoping identity ΔB_overall = Σ ΔB_i (Eq. 14d) holds for a fixed trajectory, but it does not license swapping the global argmax with the union of per-step argmaxes. Each ΔB_i = Σ_t Δ^t_Var,i + C_φ ΔIPM_i depends on D_{i-1} and on the model f_i trained on D_i = D_{i-1} ∪ D_i (Eqs. 13f-13h), so the objective at step i is a function of all previous acquisition choices, not just of the current batch D_i. Moreover, even if the terms were separable, greedy per-step maximization is globally optimal only under a monotonicity, submodularity, or exchange condition that is neither stated nor proved. The theorem also conflates constrained maximization over |D_overall| ≤ M with a union of independently unconstrained per-batch maximizations; the feasible set of later batches shrinks as earlier batches are chosen. Hence the central claim that MACAL's per-step criterion provably maximizes total risk-bound reduction is unsupported. The empirical results may still show MACAL to be a useful heuristic, but the paper's theoretical justification for the headline claim collapses at Eq. (15).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies active learning for causal effect estimation (CEE) under a limited labeling budget, proposing a batch-mode acquisition algorithm named MACAL. The central theoretical claim is Theorem 1, which asserts that the maximum reduction of the Shalit et al. PEHE upper bound over the entire labeling budget is achieved by greedily maximizing the per-query bound shrinkage at each step. MACAL operationalizes this idea by scoring candidate pairs with a distance-based criterion that combines within-group diversity/uncertainty and cross-group similarity, regularized by a hyperparameter α. Experiments on IHDP, IBM, and CMNIST compare MACAL with general and causal active learning baselines, reporting consistent empirical gains.","tokens_in":29480,"tokens_out":5640,"duration_ms":58206,"significance":"If Theorem 1 were valid, the paper would contribute a provably near-optimal, model-agnostic acquisition rule with O(N^2) complexity for an important practical problem. The empirical study is a real strength: the implementation is open-sourced, the comparisons span three datasets and multiple baselines, and the paper includes visualizations and an ablation of the symmetry regularizer. However, the theoretical foundation is unsound: the key step in the proof of Theorem 1 is invalid, and the supporting lemmas rely on idealized assumptions that are neither stated as formal conditions nor satisfied in the paper's own experiments. The contribution therefore reduces to an empirical heuristic, which does not match the paper's advertised theoretical framing.","major_comments":[{"comment":"The proof of Theorem 1 contains an invalid interchange of argmax and union. Equation (14d) is a telescoping identity for a fixed acquisition trajectory: ΔB_overall = Σ_i ΔB_i. This does not imply that arg max_{D_overall} Σ_i ΔB_i equals ∪_i arg max_{D_i} ΔB_i. Each ΔB_i depends on the accumulated dataset D_{i-1} and the model trained on D_i (Equations 13f–13h), so the objective at step i is a function of all previous acquisition choices. No monotonicity, submodularity, or exchange condition is stated to make greedy per-step maximization globally optimal, and the budget constraint |D_overall| ≤ M further couples the per-step feasible sets. This invalid step is the entirety of the theoretical justification for MACAL's design.","section":"Appendix A.1, Eq. (15)"},{"comment":"The lower bound in Lemma 2 assumes that each of the b0 acquired most-uncertain samples has predictive variance equal to the prior signal variance σ^2_f and that, after acquisition, its posterior variance is zero. In a Gaussian process, the maximum prior variance is approached only by points far from all training data; there may be fewer than b0 such points in a finite pool, and the posterior variance of an observed point is zero only at that exact input. The manuscript does not establish that the necessary number of maximum-variance points exists, so the claimed Ω(β^i) rate is not guaranteed by the argument.","section":"Appendix A.2, Eqs. (21h)–(21i)"},{"comment":"Lemma 3's proof assumes that identical cross-treatment samples can be found at every query step and that the optimal Wasserstein matching after adding these identical pairs leaves the matching of the previous points unchanged. The second assertion is not generally true: adding points can re-optimize the transport plan, so the equality in Eq. (27h)–(27i) is unsupported. The first assertion is violated in the paper's own IHDP experiment, where the t=1 pool is exhausted and MACAL switches to one-sided acquisition. Consequently, the O(1/(i+γ0)) convergence rate is not established.","section":"Appendix A.3, Eq. (27)"},{"comment":"Even if Theorem 1 were valid, the paper does not show that the implemented criterion (7) maximizes the bound shrinkage ΔB_i or the combinatorial objective (4). The transition from the entropy-weighted objective (4) to the greedy distance-based pair selection is described as an approximation, but no error bound is given, and the hyperparameter α is not connected to the constant C_φ in the bound. The algorithmic claim is therefore a heuristic approximation of an already unproven optimality statement.","section":"Section 3.2, Eq. (7)"}],"minor_comments":[{"comment":"The acronym is rendered as 'SUTV A' instead of 'SUTVA' in the assumption statement; please correct the typo.","section":"Assumption 1"},{"comment":"The word 'fucntion' should be 'function'.","section":"Appendix A.5, Proposition 2 proof"},{"comment":"In the else branch for the exhausted t=1 group, the update writes to D^{t=1}_train; it should update D^{t=0}_train.","section":"Algorithm 1, line 14"},{"comment":"The text refers to 'Appendix 5.3' when discussing the α ablation; this should be 'Section 5.3'.","section":"Section 3.2 and 5.3"},{"comment":"The subfigure labels 'IHDP-CausalAL' and 'IHDP-GeneralAL' are not defined in the text; please clarify which baselines are causal-specific and which are general.","section":"Figure 3"},{"comment":"Claimed performance gains are reported without confidence intervals or significance tests across the multiple outcome simulations; providing such statistics would strengthen the empirical claim.","section":"Section 5.1"}],"recommendation":"reject","confidential_remarks":"The central theoretical result is invalid as stated, and the paper's framing depends heavily on that result. The empirical study is competently executed and could support a resubmission that reframes MACAL as a heuristic, drops the optimality claim, and either proves a near-optimality guarantee under explicit assumptions (e.g., submodularity or regret bounds) or positions the contribution as purely empirical. As it stands, the mismatch between the advertised theory and the sound part of the contribution is too large for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Main thing you should know: the paper's theoretical centerpiece does not hold, but the empirical study is a reasonable first look at an under-explored problem and the code is public. The acquisition rule in Eq. (7) — combine farthest-point diversity with a penalty for cross-treatment distance — is a sensible heuristic, and the experiments span three datasets with several AL baselines. That part earns attention.\n\nThe problem is Theorem 1. The proof's step (15) treats the union of per-query argmaxes as the global argmax of the sum of bound shrinkages. That is only valid if each ΔB_i is independent of earlier acquisitions. But ΔB_i depends on D_{i-1} and the model trained on it (Eqs. 13f–13h). Choices at step 1 change the objective at step 2. No submodularity, exchange, or diminishing-returns condition is stated. The telescoping identity (14d) holds for a fixed trajectory; it doesn't license swapping argmax and sum. This is a load-bearing flaw: without a valid Theorem 1, the abstract's and Section 3.1's 'provably' language is overreach.\n\nThe two convergence lemmas are also idealized. Lemma 3 assumes identical samples from both treatment groups can always be found; the actual algorithm only approximates that with nearest-neighbor distance. Lemma 2's rate statement is garbled — 'lower-bounded by Ω(β^i)' with β<1 — and the proof's variance-reduction bound assumes the acquired points attain the maximum GP predictive variance, which is exactly what MACAL doesn't do (it uses Euclidean distance as a proxy).\n\nOn the plus side, the authors are transparent about limits: Appendix C states that full-bound convergence for intermediate C_φ is not obtained. The empirical comparison is broad and the code is open-sourced, so the heuristic results are checkable. What's missing is error bars or significance tests; some claimed gains (IHDP after the turning point) look like noise.\n\nNet: this is a useful heuristic paper wrapped in a theory that doesn't work. Practitioners working on active CEE will get value from the algorithm and the benchmarks; anyone relying on the theory should be careful. I'd send it to review, but the revision needs to either prove a valid greedy guarantee under stated conditions or honestly reframe MACAL as empirically motivated, demoting the theory to intuition. Experiments need error bars before any claim of demonstrable gain.","headline":"The theoretical centerpiece does not survive contact with the proof, but the heuristic and the open-sourced experiments are worth a look.","tokens_in":30027,"tokens_out":2326,"would_cite":false,"duration_ms":25175,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Greedily shrinking the causal-effect risk bound at each labeling step is provably optimal over the whole budget, and a simple pairing rule, MACAL, achieves it in practice.","keywords":["causal effect estimation","active learning","PEHE","generalization risk bound","positivity assumption","batch-mode active learning","label acquisition","treatment effect estimation"],"falsifier":"Run a two-step acquisition on a small synthetic pool and compare MACAL's greedy per-step choice against a one-step-lookahead oracle that retrains the model on every candidate batch and picks the batch minimizing the final $\\sqrt{\\epsilon_{\\mathrm{PEHE}}}$ after both steps; since $\\Delta B_i$ depends on the model state left by the previous step, the oracle should sometimes beat the union of per-step greedy maximizers, which would falsify the equality in Theorem 1 as a statement about real models. A second check: compute Lemma 2's variance-reduction lower bound on a finite pool where the most uncertain points have predictive variance below the prior cap $\\sigma^2_f$, and test whether the $b_0 \\sigma^2_f / N_{\\mathrm{pool}}$ step still holds.","tokens_in":28954,"feed_emoji":"🎯","tokens_out":9611,"duration_ms":85437,"temperature":0.7,"pith_summary":"Estimating individual treatment effects from observational data usually assumes a large labeled dataset, but in high-stakes domains labels are scarce and expensive. This paper studies which unlabeled samples to buy labels for, and claims that the right objective is to shrink the upper bound on the PEHE generalization risk at every acquisition step: the bound's per-step shrinkage decomposes into a model-variance term and a treatment-group distributional-discrepancy term, and Theorem 1 asserts that greedily maximizing each step's shrinkage maximizes the total risk reduction over the whole budget. To realize this, the paper proposes MACAL, a model-agnostic batch acquisition rule that iteratively pairs one treated and one control sample, rewarding distance from the already-labeled data (diversity and uncertainty) while penalizing the pair's mutual distance (preserving positivity). If the claim holds, a cheap $O(N^2)$ acquisition rule makes causal-effect models data-efficient under expensive labeling, and the experiments on IHDP, IBM, and CMNIST show lower $\\sqrt{\\epsilon_{\\mathrm{PEHE}}}$ than general and causal-aware active learning baselines across two deep-kernel backbones.","feed_headline":"Greedy pairing rule cuts causal-estimation error fastest","feed_subtitle":"Per-step risk-bound shrinkage adds up, so matched treatment pairs make small label budgets count.","key_machinery":"The engine of the paper is the Shalit et al. PEHE upper bound $B_i = \\sum_{t\\in\\{0,1\\}} \\epsilon^t_{i,F} + C_\\phi\\,\\mathrm{IPM}(\\rho^{t=1}_\\phi, \\rho^{t=0}_\\phi)$, rewritten as bias plus variance plus noise plus discrepancy, whose between-step difference $\\Delta B_i = \\sum_t \\Delta^t_{\\mathrm{Var},i} + C_\\phi\\,\\Delta\\mathrm{IPM}_i$ is the object Theorem 1 tells one to maximize at every query. Two lemmas carry the convergence claims: Lemma 2 uses Gaussian-process predictive variance, capped by the signal variance $\\sigma^2_f$ and non-growing as the training set expands, to produce the $\\Omega(\\beta^i)$ variance rate; Lemma 3 uses the 1-Wasserstein distance and the existence of identical paired samples to produce the $O(1/(i+\\gamma_0))$ discrepancy rate. The practical criterion is Eq. (7), the MACAL pair score, which combines group-wise nearest-neighbor distances with a pairing-penalty term $\\alpha$.","core_discovery":"The central claim is that the PEHE risk of a causal effect estimation model can be reduced as fast as possible by treating every acquisition step as a maximization of the shrinkage of the Shalit et al. upper bound, $\\Delta B_i = \\sum_{t\\in\\{0,1\\}} \\Delta^t_{\\mathrm{Var},i} + C_\\phi\\,\\Delta\\mathrm{IPM}_i$, with bias and noise terms cancelling between consecutive steps; Theorem 1 states that the union of the per-step optimal batches is the global optimum over the entire labeling budget. The convergence behaviors rest on Lemma 2, which gives an $\\Omega(\\beta^i)$ lower bound for variance reduction under Gaussian-process uncertainty sampling, and Lemma 3, which gives an $O(1/(i+\\gamma_0))$ rate for the 1-Wasserstein discrepancy when matched pairs can be labeled. MACAL implements the combined objective through the pair-acquisition criterion $(\\tilde{x}^{t=1}, \\tilde{x}^{t=0})^* = \\arg\\max \\sum_{t\\in\\{0,1\\}} \\min_{x'\\in D^t_{\\mathrm{train}}} d(\\tilde{x}^t, x') - \\alpha\\, d(\\tilde{x}^{t=1}, \\tilde{x}^{t=0})$, acquiring $|\\tilde{D}_i|/2$ pairs per query in $O(N_{\\mathrm{pool}}^2)$ time. The paper reports that MACAL achieves lower empirical $\\sqrt{\\epsilon_{\\mathrm{PEHE}}}$ than Random, BADGE, BAIT, LCMD, QHTE, and the $\\mu$BALD, $\\rho$BALD, and $\\mu\\rho$BALD variants on IHDP, IBM, and CMNIST with DUE backbones, and that its post-acquisition training sets show visibly improved treated-control overlap in t-SNE projections.","pith_inferences":["The paper does not prove the greedy step is globally optimal in the realistic case where $\\Delta B_i$ depends on the model state; a lookahead or submodularity analysis of the acquisition objective would be needed to close that gap, and the empirical gains could survive even if the theorem's separability fails.","The paper's own ablation ($\\alpha \\in \\{0, 1, 2.5, 5, 10\\}$) shows the best regularization strength drifts downward as the pool is consumed, suggesting an annealed schedule for $\\alpha$, stronger early and weaker late, is a natural extension the paper leaves implicit.","The pairing principle transfers beyond binary treatment: continuous treatments, multiple treatment arms, or unequal labeling costs per group would each need a modified pair score, but the variance-plus-discrepancy shrinkage view supplies the template.","The $\\Omega(\\beta^i)$ variance-rate argument is tied to Gaussian-process predictive variance; testing MACAL with deep ensembles or other uncertainty estimators would show whether the rate is a property of the acquisition rule or an artifact of the Gaussian-process assumption."],"forward_implications":["If Theorem 1 holds, any labeling budget is best spent by greedily acquiring the batch that maximizes the per-step bound shrinkage; no batch that looks worse at one step but better later can beat it.","In the variance-dominated regime the risk bound converges no slower than $\\Omega(\\beta^i)$, and in the discrepancy-dominated regime no faster than $O(1/(i+\\gamma_0))$; the two rates justify combining uncertainty and pairing in one criterion.","MACAL turns the NP-hard combinatorial batch-selection problem into $O(N_{\\mathrm{pool}}^2)$ distance computations per query step, so the acquisition rule remains usable with large pools.","Empirically, MACAL sets the lower envelope of $\\sqrt{\\epsilon_{\\mathrm{PEHE}}}$ across IHDP, IBM, and CMNIST under two deep-kernel backbones, and its advantage on IHDP lasts until the treated pool is exhausted.","Because the criterion is model-agnostic, it can be dropped into any causal effect estimation backbone without re-deriving acquisition-specific uncertainties."],"supporting_citations":[{"why":"Supplies the PEHE upper bound $B_i$ that the paper decomposes into variance and IPM terms and then shrinks step by step.","marker":"[33]"},{"why":"Gives the Gaussian-process predictive-variance expression and the signal-variance cap used in Lemma 2.","marker":"[44]"},{"why":"Provides the lemma that predictive variance does not grow as the training set expands, the inequality making the variance-reduction lower bound non-negative.","marker":"[45]"},{"why":"Establishes the Wasserstein metric's triangle inequality used to justify the optimal-permutation argument in Lemma 3.","marker":"[5]"},{"why":"The main causal-aware baseline (Causal-BALD variants) and the source of the DUE backbone setup used in the experiments.","marker":"[17]"},{"why":"The QHTE baseline whose loosened risk bound the paper contrasts with its own decomposition.","marker":"[28]"},{"why":"Defines the PEHE metric and provides the IHDP dataset used in the experiments.","marker":"[11]"},{"why":"Provides the IBM benchmark dataset and its potential-outcome simulations.","marker":"[34]"},{"why":"Provides the CMNIST dataset and the latent-manifold outcome simulation used for image-input experiments.","marker":"[15]"},{"why":"Supplies the DUE deep-kernel backbones (DUE-DNN and DUE-CNN) shared by all compared acquisition methods.","marker":"[38]"}],"fun_headline_variants":["Pair-based acquisition shrinks causal effect error","MACAL: active learning for causal estimation on a budget","Risk-bound shrinkage guides label pairing in causal ML","Match treatment pairs to reduce PEHE with few labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof that maximizing each step's shrinkage is globally optimal assumes the per-step shrinkages are independent, so that choices made at step 1 cannot change what step 2 can achieve, yet every acquired batch retrains the model and changes the data, and the paper itself states in Appendix C that convergence is only established in two extreme regimes, not the realistic middle one where variance and discrepancy are comparably important.","fun_headline_variants_meta":{"raw":{"variants":["Pair-based acquisition shrinks causal effect error","MACAL: active learning for causal estimation on a budget","Risk-bound shrinkage guides label pairing in causal ML","Match treatment pairs to reduce PEHE with few labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000654,"raw_usage":{"total_tokens":3151,"prompt_tokens":1256,"completion_tokens":1895,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":872,"completion_tokens_details":{"reasoning_tokens":1834}},"tokens_in":872,"tokens_out":1895,"duration_ms":12862,"temperature":1.0,"reasoning_tokens":1834,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:44:51.902330+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a two-step acquisition on a small synthetic pool and compare MACAL's greedy per-step choice against a one-step-lookahead oracle that retrains the model on every candidate batch and picks the batch minimizing the final $\\sqrt{\\epsilon_{\\mathrm{PEHE}}}$ after both steps; since $\\Delta B_i$ depends on the model state left by the previous step, the oracle should sometimes beat the union of per-step greedy maximizers, which would falsify the equality in Theorem 1 as a statement about real models. A second check: compute Lemma 2's variance-reduction lower bound on a finite pool where the most uncertain points have predictive variance below the prior cap $\\sigma^2_f$, and test whether the $b_0 \\sigma^2_f / N_{\\mathrm{pool}}$ step still holds.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the Gaussian-process predictive-variance expression and the signal-variance cap used in Lemma 2."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the lemma that predictive variance does not grow as the training set expands, the inequality making the variance-reduction lower bound non-negative."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the Wasserstein metric's triangle inequality used to justify the optimal-permutation argument in Lemma 3."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The main causal-aware baseline (Causal-BALD variants) and the source of the DUE backbone setup used in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The QHTE baseline whose loosened risk bound the paper contrasts with its own decomposition."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CMNIST dataset and the latent-manifold outcome simulation used for image-input experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DUE deep-kernel backbones (DUE-DNN and DUE-CNN) shared by all compared acquisition methods."}],"review_version":1}