{"id":"5ae28951-e467-4d25-b2b8-90353ff9c942","arxiv_id":"2607.05768","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Balanced designs that balance covariates (especially blocking) are asymptotically variance-optimal for binary ATE under Neyman's nonparametric model, and a CMH-based variance estimator is finite-sample conservative and asymptotically tight under local alternatives.","lead":"This paper shows how to design randomized experiments with binary outcomes so the treatment-effect estimate is as precise as possible, while still giving honest confidence intervals. It matters for clinical trials and A/B tests where success/failure is the outcome and covariates are known before assignment.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged Condition (15) and Lipschitz assumptions.","rationale":"The paper's strongest claim is a clean package of exact variance + design optimality + conservative/tight variance estimator under Neyman's nonparametric model for binary outcomes. The reader's identification of Condition (15) and the Lipschitz assumption as the weakest points is accurate and already reflected in the CONDITIONAL verdict. Those conditions are stated explicitly (Theorem 1, Theorem 3, §2.4) and are not papered over; when they fail the paper itself notes that tightness can fail. Simulations are limited to one logistic family but are consistent with the predicted U-shape and CMH advantage. No public code is a practical limitation already noted by the reader, not a correctness flaw. Because the load-bearing math checks out under the stated assumptions and no stronger internal concern surfaces, the verdict should remain CONDITIONAL with no adjustment.","tokens_in":23158,"tokens_out":572,"duration_ms":6527,"concrete_test":"Independently re-derive the difference E[V_CMH]-Var[τ̂] from Eq. 5 and the expansion of E[YᵀΣ_W Y] in the proof of Theorem 2 (supplement A.3) without using the Schur-product step; confirm the non-negative remainder is exactly (4/n^{2})ηᵀΣ_W⊕2 η. If the identity holds, the conservativeness claim is secure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (asymptotic optimality of balanced designs meeting the covariate-balance condition, blocking satisfaction under Lipschitz, and finite-sample conservativeness plus local-alternative tightness of V_CMH under bounded λ_max) is internally consistent with the provided derivations. Theorems 1–3 and the exact variance formula (Eq. 5) follow from standard matrix algebra, Schur product, and the suboptimal-cube blocking argument in the supplement; the difference term (4/n^{2})ηᵀΣ_W⊕2 η is non-negative by construction. The reader's weakest_assumption already isolates the two places where the claim can fail outside its stated regime: Condition (15) (1/n)‖η‖^{2}\to0 and Lipschitz continuity of h_T,h_C. No additional hidden inconsistency or load-bearing gap in the proofs or simulation design appears to undermine the claim inside those regimes.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper studies randomized experiments with binary outcomes under Neyman’s nonparametric model (fixed covariates, random independent Bernoulli potential outcomes). It derives the exact variance of the balanced difference-in-means estimator (Eq. 5), shows that any balanced design satisfying the covariate-balance condition (7) is asymptotically variance-optimal, and proves that optimal uniform blocking with B\to∞ meets (7) under Lipschitz response probabilities (Theorem 1). Because unbiased variance estimation is impossible, the authors introduce a CMH-based conservative estimator V_CMH for general balanced designs (Theorem 2: E[V_CMH]−Var[τ̂]=(4/n^{2})ηᵀΣ_W^{⊙2}η) and an extension of Robins’ estimator for blocking designs. Under local alternatives with the second-moment condition (15) and bounded λ_max(Σ_W), V_CMH is asymptotically tight and consistent (Theorem 3), yielding asymptotically valid CIs. Simulations (N_sim=10 000) compare designs and inference procedures and support the claim that designs achieving both covariate balance and sufficient randomness perform well with CMH inference.","tokens_in":23400,"tokens_out":1163,"duration_ms":12502,"significance":"If the results hold, the paper supplies a coherent design-and-inference package for binary (incidence) outcomes under the practically relevant Setting (b): asymptotic variance optimality via a transparent balance condition, finite-sample conservative variance estimation that is not restricted to blocking, and local-alternative tightness that justifies CMH-based confidence intervals. The explicit variance formula, the Schur-product conservativeness argument, the suboptimal-cube bound for blocking, and the term-by-term covariance analysis for consistency are clean and reusable. The simulation design (fixed X, many designs including optimal and naïve blocking, rerandomization, greedy matching hybrids) is thorough and illustrates the balance–randomness trade-off. The work therefore advances both the theory of optimal randomization for binary responses and the practical toolkit for conservative yet asymptotically tight inference.","major_comments":[{"comment":"§2.4, Condition (15) and Theorem 3: asymptotic tightness of V_CMH (and therefore asymptotic validity of the CMH CIs) requires not only √n τ\toτ_∞ but also (1/n)‖η‖^{2}\to0. The manuscript notes that the condition can fail when individual effects remain large while averaging to a small τ, yet the main claims and the simulation design (logistic-linear model with fixed ‖β‖=3, β_T=0.5) stay inside the regime. A short discussion or numerical illustration of how large the difference term remains when (15) is violated would clarify the practical scope of the “asymptotically tight” claim.","section":null},{"comment":"§2.3.1–2.3.2 and Proposition 1: the comparison of V_CMH and V_Robbins-ext is informative for BCRD, but the paper’s recommendation of CMH for general balanced designs rests on finite-sample conservativeness (Theorem 2) while Robbins-ext is only asymptotically conservative and restricted to blocking. The simulations show Robbins-ext badly missized for small blocks; a clearer statement of when (if ever) Robbins-ext is preferred, or an explicit recommendation to prefer CMH except in large-block settings, would strengthen the practical takeaway.","section":null}],"minor_comments":[{"comment":"Abstract and §1: “incidence outcomes” is used without definition; a brief parenthetical would help non-epidemiology readers.","section":null},{"comment":"§2.2: the bound after Theorem 1 assumes Lipschitz constant 1 and unit-cube support; stating that the argument scales with the Lipschitz constant and diameter would make the result more transparent.","section":null},{"comment":"§3.1: the Monte-Carlo approximation (18) for Σ_W of GreedyMD / BinaryMatchThenGreedyMD is correct but computationally heavy; a short remark on whether a closed-form or low-rank approximation is feasible would be useful.","section":null},{"comment":"Figures 1–2 and Table 1: the U-shape over block size is clearest for p=1; a sentence noting that the pattern is attenuated (but still visible) for p=5,10 would help readers interpret the higher-dimensional panels.","section":null},{"comment":"References: a few self-citations (Azriel et al. 2024, 2026; Kapelner et al. 2023, 2025) supply earlier variance expressions; ensuring that the present paper is self-contained for the key formulas (especially Eq. 5) is already largely achieved, but a one-sentence pointer would be helpful.","section":null},{"comment":"Typographical: “Subjets” (§3.1, OptimalB), “Robbins” vs “Robins” inconsistency in a few places, and “hormetic U-shape” may be unfamiliar to some readers.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The manuscript is a solid, technically careful contribution that fits a methods journal well. The self-citation density is noticeable but the new results (general-design CMH, local-alternative tightness, blocking optimality under Lipschitz) are genuine extensions. I see no integrity or scope issues; minor revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a clean methods paper that actually delivers a usable package. Under Neyman nonparametric Setting (b) (covariates fixed, binary potential outcomes random), they give the exact variance of the difference-in-means estimator for any balanced design, show that any design meeting the simple covariate-balance condition lim (1/n)(p_T + p_C)^T Σ_W (p_T + p_C) = 0 is asymptotically optimal, prove that optimal uniform blocking with B \to ∞ meets it under Lipschitz response surfaces, and supply a CMH-style variance estimator that works for general balanced designs (not just blocks). That estimator is finite-sample conservative by the Schur product theorem, and they prove it is asymptotically tight under local alternatives when λ_max(Σ_W) is bounded. The Robins extension is a useful secondary tool for blocks. Simulations (10k reps, several designs, n up to 256) line up with the theory: CMH beats Wald and the extended Robins on power/size/CI length, and you see the expected U-shape in block size from the balance–randomness trade-off.\n\nWhat is new is the combination: exact variance under Setting (b), the general-design CMH form, the local-alternative tightness theorem, and the clean optimality characterization for blocking. The proofs (supplement) are standard matrix algebra and are fully written out; the self-citations are mostly prior variance expressions and power results that the paper builds on, not circular. No public code is a real but minor annoyance for a methods paper of this type.\n\nSoft spots are exactly the ones the reader flagged and are already stated in the paper: tightness needs the second-moment condition (1/n)‖η‖^{2} \to 0 (heterogeneous individual effects that cancel in the mean can break it), Lipschitz for the blocking result, and asymptotic normality for the CIs leans on their earlier work. Simulations use one logistic family. None of these sink the central claims inside the stated regimes.\n\nThis is for people who design or analyze binary-endpoint experiments with observed covariates (clinical trials, A/B tests, etc.). It is worth a serious referee and worth citing if you work in that area. I would engage.","headline":"Solid design-plus-inference package for binary outcomes under Neyman nonparametric Setting (b): exact variance, blocking optimality, and a usable CMH variance that is finite-sample conservative and locally tight.","tokens_in":24049,"tokens_out":584,"would_cite":true,"duration_ms":6782,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62K05","62K10","62G05"],"pacs":[],"model":"grok-4.5","headline":"For binary outcomes under fixed covariates, balanced designs that cancel covariate imbalance make the difference-in-means estimator asymptotically variance-optimal, and a CMH variance estimator is conservative yet tight under local alternat","keywords":["binary treatment effects","Neyman nonparametric model","optimal experimental design","blocking designs","Cochran-Mantel-Haenszel variance","conservative inference","local alternatives","covariate balance"],"falsifier":"Simulate binary outcomes under a local alternative in which half the units have treatment effect +c and half have -c (so average τ is small but (1/n)‖η‖^{2} stays order 1) using optimal blocking with growing B; if n(E[V_CMH] − Var[τ̂]) fails to vanish, the tightness claim is false.","tokens_in":24011,"feed_emoji":"🎲","tokens_out":800,"duration_ms":7816,"temperature":0.7,"pith_summary":"The paper studies randomized two-arm experiments with binary responses when the covariates are already observed and fixed, but the potential outcomes remain random (Neyman's nonparametric model). It derives the exact variance of the usual difference-in-means estimator and shows that any balanced assignment whose covariance matrix drives the quadratic form in the average success probabilities to zero is asymptotically optimal. A broad class of blocking designs meets that condition once the number of blocks grows and the success probabilities are mildly smooth functions of the covariates. Because the exact variance depends on unknown probabilities, unbiased estimation is impossible; the authors therefore introduce a conservative Cochran–Mantel–Haenszel variance estimator that works for any balanced design and prove it becomes asymptotically exact under local alternatives when the design is sufficiently random. Simulations confirm that intermediate blocking (or other designs that combine balance with randomness) paired with this estimator yields higher power and shorter intervals than Wald or extended-Robins procedures while remaining properly sized.","feed_headline":"Blocking plus CMH gives optimal binary ATE inference","feed_subtitle":"Variance-optimal designs stay conservative yet become tight under local alternatives.","key_machinery":"The exact variance formula Var[τ̂] = (1/n^{2})[(p_T + p_C)^T Σ_W (p_T + p_C) + 2(p_T^T(1-p_T) + p_C^T(1-p_C))] together with the CMH estimator V_CMH = (4/n^{2}) Y^T Σ_W Y, whose expectation exceeds the true variance by a non-negative quadratic form in the individual treatment effects.","core_discovery":"Any balanced design satisfying lim (1/n)(p_T + p_C)^T Σ_W (p_T + p_C) = 0 is asymptotically variance-optimal for the difference-in-means estimator; optimal uniform blocking with diverging block number satisfies the condition under Lipschitz response probabilities, and the CMH estimator V_CMH is finite-sample conservative with excess equal to (4/n^{2}) η^T (Σ_W ⊙ Σ_W) η and asymptotically tight under local alternatives whenever λ_max(Σ_W) stays bounded.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Balanced designs with covariate condition are asymptotic variance-optimal","Blocking plus CMH estimator delivers tight binary ATE inference","Any balanced design meeting lim condition optimizes binary DIM variance","Uniform blocking with diverging blocks satisfies optimality under Lipschitz","CMH variance is finite-sample conservative yet tight under local alternatives"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"Asymptotic tightness of the CMH variance estimator requires that the average squared individual treatment-effect differences also go to zero, not merely that their average shrinks like 1 over square-root n; if large positive and negative individual effects cancel, the estimator stays conservative even asymptotically.","fun_headline_variants_meta":{"raw":{"variants":["Balanced designs with covariate condition are asymptotic variance-optimal","Blocking plus CMH estimator delivers tight binary ATE inference","Any balanced design meeting lim condition optimizes binary DIM variance","Uniform blocking with diverging blocks satisfies optimality under Lipschitz","CMH variance is finite-sample conservative yet tight under local alternatives"]},"model":"grok-4.5","effort":"low","cost_usd":0.004556,"raw_usage":{"total_tokens":1346,"prompt_tokens":784,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":45560000,"prompt_tokens_details":{"text_tokens":784,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":497,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":784,"tokens_out":65,"duration_ms":5856,"temperature":1.0,"reasoning_tokens":497,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T02:14:05.347525+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Simulate binary outcomes under a local alternative in which half the units have treatment effect +c and half have -c (so average τ is small but (1/n)‖η‖^{2} stays order 1) using optimal blocking with growing B; if n(E[V_CMH] − Var[τ̂]) fails to vanish, the tightness claim is false.","supporting_citations":[],"review_version":1}