REVIEW 2 major objections 4 minor 11 references
On the Foundations of the Design-Based Approach
T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper argues that SUTVA's real work is not just ruling out interference within a study; it is what licenses generalizing causal claims from one design to another, and the weaker NURVA assumption cannot do that.
desk verdict A clear conceptual distinction between SUTVA and NURVA, with a real but conditional claim about external validity; the paper deserves a serious referee, though the design-space choice needs tightening. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the exposure mapping $g_i:\mathcal{Z}\to\mathbb{R}$, which assigns each unit an exposure for every intervention in the design space, together with the raw potential outcomes $y_i(z)$ for $z\in\mathcal{Z}$. NURVA (Condition 5.1) demands that when $g_i(z)=g_i(z')$ for two interventions in $\mathrm{Supp}(Z)$, the outcomes agree; SUTVA (Condition 5.2) demands the same across all of $\mathcal{Z}$. The machinery works by making the support of the design the boundary of what can be learned: NURVA can hold while outcomes under assignments outside the support vary freely, and that hidden variation is exactly what blocks the AEED from generalizing to other designs.
What would settle it
Ask whether NURVA alone determines the AEED under a design $Z'$ whose support strictly contains the support of the realized design $Z$. If two raw outcome schedules both satisfy NURVA under $Z$, are observationally identical under $Z$, and yet give different AEEDs under $Z'$, the paper's claim is demonstrated; a proof that such schedules cannot exist would refute it.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that SUTVA, once NURVA is separated out, is a generalizing assumption rather than a mere no-interference/no-hidden-variations restriction. For any fixed design, NURVA and SUTVA are observationally equivalent, so all within-design estimation and inference proceeds as in the standard paradigm. But NURVA is design-dependent: it only requires stability of potential outcomes on the support of the realized design. Consequently, AEEDs identified under NURVA describe the contrast induced by using the realized design to place a unit in exposure $d$ rather than $d'$, not the contrast under an arbitrary intervention that assigns exposures differently. The paper's two-unit examples show the AEED can change sign when the design's support is extended to uniform treatment or uniform control even though NURVA holds under the realized design. The paper then reconstructs the standard paradigm, with SUTVA placed at the end as the assumption that carries causal conclusions beyond the realized design.
Load-bearing premise
The paper's main negative claim applies only when the researcher wants conclusions about interventions whose possible assignments reach beyond those of the realized design; for conclusions strictly about the realized design, NURVA is enough and the claimed insufficiency does not arise.
Editorial extensions
If this is right
- For a fixed randomized or sampling design, replacing SUTVA with NURVA changes nothing observable: finite-sample unbiased Horvitz–Thompson estimation of AEPO/AEED, conservative variance estimation, and standard asymptotics all go through.
- An AEED estimated under NURVA does not identify the corresponding contrast under a design with different support, such as treating everyone versus treating no one.
- Claims that an experimental effect will persist under a scaled-up or differently targeted intervention require SUTVA or an equivalent assumption, and the realized design's data cannot certify them.
- SUTVA's role shifts from a technical condition for unbiasedness to the assumption that converts design-specific exposure contrasts into treatment effects as usually understood.
- A researcher can trim the population or redefine exposures to restore individual positivity, which keeps the AEPO and AEED well-defined in settings with interference.
Reading between the lines
- External-validity debates may be better framed around whether the target intervention lies in the support of the realized assignment mechanism than around effect heterogeneity, since NURVA cannot bridge a support gap.
- A field experiment that compares AEEDs under two designs on the same population, one with a support nested in the other, could serve as a diagnostic: agreement would suggest SUTVA approximately holds, disagreement would reveal hidden variation that NURVA misses.
- When generalization to a policy intervention is the goal, the design principle implied by the paper is to include the policy intervention in the design's support—for example, whole-community treatment arms—rather than relying on SUTVA to extrapolate from partial treatment.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops a design-based framework for causal inference and survey sampling that does not begin from SUTVA. It defines potential outcomes at the level of the full assignment vector, introduces exposure mappings, and then defines expected potential outcomes (EPO), average expected potential outcomes (AEPO), expected exposure differences (EED), and average expected exposure differences (AEED). These quantities are well-defined even under arbitrary interference. The paper contrasts a new assumption, NURVA (stability of potential outcomes on the support of the realized design), with SUTVA (stability on the entire design space), and argues that NURVA is practically insufficient for identifying substantively interesting quantities because it cannot support generalization to designs whose supports lie outside the realized design. The paper also provides estimation theory for AEPO and AEED, including Horvitz-Thompson type estimators, conservative variance estimators, covariate adjustment, and asymptotic results, with proofs in an appendix. Its central claims are illustrated with fully specified toy examples, including a household voter-turnout example and general-equilibrium job-training and campaign-ad examples.
Significance. If the central claim is accepted with appropriate qualifications, the paper makes a useful conceptual contribution: it isolates precisely what SUTVA adds over a design-specific stability assumption, and it offers a coherent set of estimands that remain meaningful under interference. The paper's strengths include its explicit formal definitions, the hand-checkable toy examples, the observation that SUTVA and NURVA are observationally equivalent for a given realized design, and the demonstration that HT-type estimators can be unbiased for the new targets without SUTVA. The estimation appendix is internally consistent and extends the Aronow and Samii (2017) framework. The main significance is foundational: it clarifies that SUTVA is a generalizing assumption rather than merely a regularity condition for design-based estimation.
major comments (2)
- [Sections 3.1, 5.1, 5.2, and 6] The central claim of the paper, that NURVA is practically insufficient for identifying substantively interesting quantities, depends critically on how the design space Z is specified, and the paper never formalizes that choice. Condition 5.1 (NURVA) and Condition 5.2 (SUTVA) differ only in the quantifier domain: Supp(Z) versus Z. In Section 3.1, Z is defined only as the image of a bijective map from all conceivable experimental interventions, which is not a well-defined criterion, and Section 5 notes that Z may include assignments outside the design. If an analyst adopts the standard design-based convention that the design is the actual randomization distribution and sets Z = Supp(Z), then NURVA and SUTVA coincide, and the failure of extrapolation illustrated in Tables 1, 2, and 4 does not arise. In that case the household voter-turnout example satisfies SUTVA vacuously on the support, and the AEED is the same under every design whose support lies in Z. Thus the claimed practical insufficiency of NURVA is not an intrinsic property of the assumption; it is a consequence of the analyst's decision to broaden Z to include out-of-support interventions and to define inferential targets under those interventions. The paper is transparent about this in some passages, but the abstract and conclusion present the insufficiency as unconditional. The manuscript should state explicitly that the insufficiency claim is relative to a design space Z that is closed under the researcher's causal queries (for example, the union of supports of all designs under consideration), and should adjust the abstract and conclusion accordingly. Without such a qualification, the central claim is an artifact of an unformalized modeling choice.
- [Section 7 and Appendix A.4] The paper claims, in Section 7, that finite-sample unbiasedness can be attained for covariate-adjusted estimators even if NURVA does not hold. The proof in Appendix A.4, however, relies on the condition f_d(X_i, beta-hat) being independent of D_i, which is not stated in the main text. This is a substantive assumption: it fails when beta-hat is estimated from data that depend on the treatment or exposure assignment, which is the typical case in regression-adjusted estimation. The main-text claim as written is therefore stronger than what is proved. Please state the independence condition explicitly in Section 7, and discuss how it relates to standard practice, such as cross-fitting or using auxiliary data, so that readers do not infer that covariate adjustment is unbiased under NURVA without any additional condition.
minor comments (4)
- [Section 5, toy example] The sentence above Table 1 reads the AEPO under the treatment exposure is 1, y(1) = 0, and the AEPO under the control exposure is 0, y(0) = 1, which is internally inconsistent with the subsequent computation of AEED = -1. The text should say that the AEPO under treatment exposure is 0 and the AEPO under control exposure is 1.
- [Section 3.1] The notation for the design space Z and the random vector Z is confusing: the sentence We define the design space Z = Z(Omega), where Z is a bijective random vector Z : Omega to R^N uses the same symbol for both the design space and the vector. Consider using a different symbol for one of the two objects.
- [Appendix A.3] The proof of Proposition 1 is only a sketch: it states that the result follows from plugging terms into the variance formula and using Chebyshev's inequality, but it does not show how Conditions A.1 through A.3 jointly control the variance. Since the appendix is the only place where the asymptotic claims are developed, a more detailed derivation would increase confidence in the consistency result.
- [Section 8] The final sentence claims that inference on AEEDs is asymptotically equivalent to inference on the average direct effect if both statistical dependence in D and unmodeled interference are sufficiently local. This is an informal, unproven claim; either state the precise regularity conditions or flag it as a conjecture.
Circularity Check
No significant circularity: the NURVA/SUTVA distinction follows from explicit definitions, and the examples and estimation results are self-contained.
full rationale
The paper's derivation is self-contained. NURVA (Condition 5.1) and SUTVA (Condition 5.2) are explicitly defined over Supp(Z) and Z, respectively; the observational equivalence of the two for a fixed design and the failure of NURVA to constrain behavior on Z\Supp(Z) are direct consequences of these definitions, not assumptions disguised as conclusions. The central examples (household voter turnout, job training, campaign ad) are constructed tables of raw potential outcomes with fully specified designs, and the AEED values are computed directly from those tables, not fitted or predicted. The estimation results in the appendix are proven algebraically—for instance, the Horvitz-Thompson estimator is shown to be unbiased for the AEPO by direct expectation calculations—and do not require importing the paper's own previous results as load-bearing evidence. Citations to Aronow and Samii (2017) provide terminology such as the exposure mapping and prior estimation theory, but the new claims do not reduce to those citations; no fitted parameter is renamed as a prediction, and no uniqueness theorem from the authors is imported to exclude alternatives. The only point that could appear definitional—that NURVA cannot support claims about interventions outside the support of the design—is explicitly conditional on the researcher's choice of design space Z, and the paper is transparent about this conditionality. That is a scope limitation, not circular reasoning.
Assumptions & free parameters
assumptions (6)
- domain assumption Finite sample space and known design: the cardinality of the sample space is finite and the researcher knows the assignment probabilities and support (Section 3.1).
- domain assumption Existence of raw potential outcomes yi(z) for all feasible interventions z in the design space Z (Section 3.2).
- domain assumption Exposure mapping gi is determined by the researcher and defines the meaningful treatment contrasts (Section 3.3).
- domain assumption Target design Z' with extended support is the quantity of interest (Section 6).
- domain assumption Individual positivity: each unit has positive probability of each relevant exposure (Condition 4.1).
- standard math Regularity conditions A.1-A.3 for consistency: bounded outcomes, restricted exposure dependence, and an interaction condition.
invented entities (2)
-
No Unmodeled Revealable Variation Assumption (NURVA)
-
Average Expected Potential Outcome (AEPO) and Average Expected Exposure Difference (AEED)
Cite this review
Pith. "Pith review of On the Foundations of the Design-Based Approach." pith.science (2026). https://pith.science/paper/7COMZNGH
@misc{pith2026250510519,
author = {Pith},
title = {Pith review of: On the Foundations of the Design-Based Approach},
year = {2026},
howpublished = {\url{https://pith.science/paper/7COMZNGH}},
note = {Machine review of arXiv:2505.10519}
}
read the original abstract
The design-based paradigm may be adopted in causal inference and survey sampling when we assume Rubin's stable unit treatment value assumption (SUTVA) or impose similar frameworks. While often taken for granted, such assumptions entail strong claims about the data generating process. We develop an alternative design-based approach: we first invoke a generalized, non-parametric model that allows for unrestricted forms of interference, such as spillover. We define a new set of inferential targets and discuss their interpretation under SUTVA and a weaker assumption that we call the No Unmodeled Revealable Variation Assumption (NURVA). We then reconstruct the standard paradigm, reconsidering SUTVA at the end rather than assuming it at the beginning. Despite its similarity to SUTVA, we demonstrate the practical insufficiency of NURVA for identifying substantively interesting quantities. In so doing, we provide clarity on the nature and importance of SUTVA for applied research.
Reference graph
Works this paper leans on
-
[1]
With restrictions on variation on reweighted revealable potential outcomes, exposure dependence among units, and their interaction, (Conditions A.1, A.2, A.3), as well as NURVA (Condition 5.1), the Horvitz-Thompson estimator ˆy(d) is consistent for the 14In the asymptotic regime we could also index observations of, e.g., yi and πi(d) by (N) but we leave t...
work page 2021
-
[4]
(10) 29 The Horvitz-Thompson estimator can be seen as a generalization of the sample mean
We use the Horvitz-Thompson inverse probability weighted estimator (Horvitz and Thomp- son 1952), as defined in Equation 10, ˆy(d) = 1 N NX i=1 YiI [Di =d] πi(d) . (10) 29 The Horvitz-Thompson estimator can be seen as a generalization of the sample mean. Sup- pose that exposure is individualistic, and we have a simple random sampling strategy of n of N un...
work page 1952
-
[6]
When we reintroduce notational dependence on Z, we use the fact that Yi = yi(Z), from the definition of raw potential outcomes given in Section 3.2. 13 A key take-away from the unbiasedness of the Horvitz-Thompson estimator is that even without NURVA, in standard (simple random sampling without replacement) surveys, the sample mean is unbiased for the AEP...
work page 2014
-
[1934]
“On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection.” Journal of the Royal Statistical Society 97 (4): 558–625. . 1990 [1923]. “On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9 (reprint).” Statistical Science 5 (4): 465–472. ...
work page 1990
-
[2004]
Normal approximation under local dependence
“Normal approximation under local dependence.” The Annals of Probability 32 (3A): 1985–2028. 25 Chernozhukov, Victor, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins
work page 1985
-
[2010]
Design-based Estimation Theory for Complex Experiments
“The spread of behavior in an online social network experiment.” Science 329 (5996): 1194–1197. Chang, Haoge. 2023a. Design-based Estimation Theory for Complex Experiments. arXiv: 2311.06891 [econ.EM]. https://arxiv.org/abs/2311.06891. . 2023b. “Essays on Design-Based Inference.” PhD diss., Yale University. Chen, Louis Hy, and Qi-Man Shao
-
[2017]
“Estimating average causal effects under general interference, with application to a social network experiment.” The Annals of Applied Statistics 11 (4): 1912–1947. Bickel, Peter J, and David A Freedman
work page 1912
-
[2018]
A Unified Theory of Regression Adjustment for Design-based Inference
A Unified Theory of Regression Adjustment for Design-based In- ference. arXiv: 1803.06011 [math.ST]. https://arxiv.org/abs/1803.06011
Show all 11 references
-
[2021]
https://arxiv.org/abs/2109.09236
Unifying Design-based Inference: A New Variance Estimation Principle.arXiv: 2109.09236 [stat.ME]. https://arxiv.org/abs/2109.09236. 27 Miguel, Edward, and Michael Kremer
-
[2023]
arXiv: 2106.15074 [stat.ME]
Causal Inference with Panel Data under Temporal and Spatial Interference. arXiv: 2106.15074 [stat.ME]. https://arxiv.org/abs/2106.15074. Wu, Edward, and Johann A Gagnon-Bartsch
-
[2024]
arXiv: 2112
Optimized variance estimation under interference and complex experimental designs. arXiv: 2112 . 01709 [stat.ME]. https://arxiv.org/abs/2112.01709. Hern´ an, Miguel A, and James M Robins
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.