REVIEW 3 major objections 5 minor 1 cited by
Estimating Representative Causal Effects with Double Machine Learning
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Residuals-on-residuals regression, a popular double-machine-learning estimator, converges to a variance-weighted average of derivatives evaluated at pseudo-treatment values that need not appear in the observed data, so it does not…
desk verdict RORR bias for many-valued treatments is real and cleanly derived; the proposed AIPW fix is promising but the theory and the zero-inflated application both need tightening. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the mean-value pseudo-treatment $T_i^*$, the point between the realized treatment $T_i$ and its conditional mean $h(X_i)$ at which RORR effectively evaluates the dose-response derivative; the paper shows this point need not be an observed or even realizable treatment level. The argument is carried by the bias decomposition $\tilde\theta-\mathbf{E}[f'(T_i)] = A + \mathrm{Cov}(\omega_i,f'(T_i))$, where $A$ is bounded by $L\kappa$ and $\omega_i$ is the normalized conditional variance weight. The proposed estimator is Coarsened AIPW: partition the treatment support into $K$ bins, estimate bin-level propensity scores $p_k(X_i)$ and outcome regressions $m_k(X_i)$, form AIPW estimates $\hat\psi_k$ of each bin's counterfactual mean, and aggregate $\hat\psi=\sum_{k=1}^{K-1} w_k(\hat\psi_{k+1}-\hat\psi_k)/(t_{k+1}-t_k)$ with weights $w_k$ proportional to observed bin frequencies; consistency follows from a lemma showing $\int f(t) r_k(t|x)dt = f(\bar t_k)+O(\ell^2)$.
What would settle it
Simulate a large sample from the model (5) with a known concave dose-response function such as $f(t)=\log(t+1)$ and conditionally Poisson treatment, compute the closed-form RORR plim (17) and the true ACD (16), and check that they differ whenever $f$ is non-affine. The paper predicts the gap is systematic and bounded by (15); observing equality for a non-affine $f$ with no exact cancellation of $A$ and $B$ would refute Proposition 1.
Extended reading notes
Core claim
For the structural model $Y_i=f(T_i)+g(X_i)+e_i$, $T_i=h(X_i)+u_i$ with conditionally ignorable treatment and a twice continuously differentiable dose-response function $f$, the paper shows that the RORR plim is $\tilde\theta = \mathbf{E}[(T_i-h(X_i))f(T_i)]/\mathbf{E}[(T_i-h(X_i))^2]$. By the mean value theorem this equals $\mathbf{E}[\omega_i f'(T_i^*)]$, where $\omega_i = (T_i-h(X_i))^2/\mathbf{E}[(T_i-h(X_i))^2]$ is the normalized conditional variance weight and $T_i^*$ lies between $T_i$ and $h(X_i)$. The bias relative to the ACD $\mathbf{E}[f'(T_i)]$ decomposes into $A=\mathbf{E}[\omega_i(f'(T_i^*)-f'(T_i))]$, bounded by $L\kappa$ with $L$ the Lipschitz constant of $f'$ and $\kappa=\mathbf{E}|T_i-h(X_i)|^3/\mathbf{E}[(T_i-h(X_i))^2]>0$, plus $B=\mathrm{Cov}(\omega_i,f'(T_i))$. Both vanish if $f$ is affine, so the RORR estimand equals the ACD in general only in that case. The paper's alternative, Coarsened AIPW, replaces the residual-on-residual step with bin-level AIPW estimates of counterfactual means and differences adjacent bins weighted by observed bin frequencies, which re-centers the estimand on the actual treatment distribution.
Load-bearing premise
The consistency of the proposed Coarsened AIPW estimator relies on uniform positivity of the bin-level treatment probabilities and a high-level condition that bin-level AIPW components are uniformly consistent under cross-fitting, and these conditions are not derived from primitive assumptions; they can fail for zero-inflated or highly skewed treatments.
Editorial extensions
If this is right
- For any many-valued treatment with a nonlinear dose-response function, a RORR coefficient should be read as a variance-weighted derivative at pseudo-treatment values, not as the average causal derivative, even when nuisances are perfectly estimated.
- Ranking features or interventions by RORR coefficients is not the same as ranking by representative causal effects, because the variance weights and the pseudo-treatment evaluation shift the target quantity.
- The Coarsened AIPW estimator is consistent for the ACD under positivity and uniform nuisance consistency, giving large-scale observational studies a computationally practical alternative to generalized propensity score estimation.
- When the treatment is right-skewed and the dose-response curve is concave, RORR will tend to understate the average causal derivative, because it up-weights the flatter high-dose region.
Reading between the lines
- A direct experimental test of the paper's logic would compare RORR and Coarsened AIPW against a known dose-response curve from a randomized continuous-treatment study; the prediction is that RORR's error grows with the curvature of $f$ and with heteroskedasticity of the treatment residuals.
- If uniform positivity fails for the bins that matter, the AIPW consistency result no longer applies, so practitioners should report sensitivity of the coarsened estimate to bin count, propensity trimming, and outcome-model extrapolation.
- The pseudo-treatment decomposition is a property of two-step partialling-out generally, so similar bias should appear in other residualized estimators (e.g., residualized instrumental-variable or panel regressions) when the first-stage residual is used as a regressor.
- Because the bias term $A$ is quantifiable from data through $\kappa$ and an estimated Lipschitz constant of $f'$, one could construct an explicit diagnostic that reports how far RORR is from the ACD for a given dataset.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper studies the probability limit of residuals-on-residuals regression (RORR) for many-valued treatments under treatment-effect heterogeneity. The authors show that, under the structural model Y_i = f(T_i) + g(X_i) + e_i, the RORR plim is E[omega_i f'(T*_i)], where omega_i are normalized conditional-variance weights and T*_i lies between T_i and E[T_i | X_i]; this does not generally equal the average causal derivative E[f'(T_i)]. They propose a coarsened Augmented Inverse Propensity Weighting (AIPW) estimator that bins the treatment, estimates bin-level counterfactual means, and aggregates first differences to estimate the ACD, proving consistency under high-level conditions. A numerical example illustrates the RORR bias, and an application to Netflix usage data reports conflicting RORR and AIPW estimates.
Significance. The RORR bias result is a useful and accessible warning for practitioners: for nonlinear dose-response functions, even when nuisances are consistently estimated, RORR weights derivatives at pseudo-treatment values not observed in the sample. The decomposition into the two bias components in Proposition 1 is clean, and the numerical example is transparent, with replication code provided. If the coarsened AIPW estimator could be justified under primitive conditions, it would be a practical contribution for large-scale observational causal inference. At present, however, the theoretical support for the proposed estimator is incomplete: Proposition 2 assumes a uniform consistency condition rather than deriving it, and the empirical application uses a zero-inflated treatment that violates the positivity and smoothness assumptions used in the proof. These gaps are load-bearing because the paper's headline claim that the proposed estimator is a consistent remedy for RORR is not established in the settings where the estimator is demonstrated.
major comments (3)
- [Section 4.2, Proposition 2] The hypothesis that max_k |psi_hat_k - psi_k| = o_p(1) under cross-fitting is assumed rather than derived. This is a uniform consistency condition over K growing bins, with effective sample size N/K per bin; standard DML results for a fixed-dimension parameter do not imply it. The proof needs primitive conditions on the estimation errors of p_k and m_k (for example, rate conditions and empirical-process or entropy bounds) and must show that the weighted first-difference aggregation error is o_p(1) after division by ell, rather than citing reference [17] for the last step. Without such conditions, Proposition 2 is a consistency result conditional on a high-level assumption, not a theorem about the proposed estimator.
- [Section 5, Assumption (23)] The empirical application bins the Feature A usage rate into a zero bin and quartiles of nonzero values. The conditional treatment distribution therefore has a point mass at zero, so the conditional density p(t|x) is not twice continuously differentiable and inequality (23), Pr(T_i in S_k | X_i = x) >= epsilon*ell, cannot hold for all k and x with ell > 0; the zero bin has no positive length. Proposition 2 consequently does not cover the reported AIPW estimate, which is used as evidence for the favorable properties of the proposed estimator. The paper should either extend the theory to atomic or zero-inflated treatments, for example through an Average Incremental Effect formulation with a separate treatment of the mass at zero, or present the Netflix result as an illustrative heuristic rather than an application of Proposition 2.
- [Section 4.2, proof of Proposition 2] After equation (24), the proof asserts that the bin-level AIPW estimators converge to the displayed doubly robust expression and that the nuisance-estimation remainders are o_p(1). This step is not established for K growing with N: the condition N ell^2 -> infinity does not by itself control the accumulation of estimation errors in the aggregated sum sum_k w_k (psi_hat_{k+1} - psi_hat_k)/ell. The variance calculation O(1/(N ell^2)) is for the oracle estimator, and the remainder terms from estimated p_k and m_k require separate rate conditions and a demonstration that the aggregated remainder is o_p(1). The proof should provide explicit rates for the nuisance estimates and check the aggregation step.
minor comments (5)
- [Section 2.2, Proposition 1] The statement that the converse holds 'except in special cases where the bias terms cancel exactly' is not proved and the special cases are not characterized. The proof establishes only that f affine is sufficient; the necessity claim would require showing that A and B are nonzero for all non-affine f, which is not shown. Suggest rewording to 'f affine is a sufficient condition, and the bias is generically nonzero otherwise.'
- [Section 4.2, equation (24)] The notation psi_k = lim psi_hat_k is imprecise; define psi_k as the probability limit of the bin-level AIPW estimator under the stated model and clarify that it depends on the bin S_k and on the conditional treatment distribution within that bin.
- [Appendix B] In the MSE calculation, the line 'Suppose psi_hat_k - m_k = o_p((N/K)^{-1/2})' mixes the AIPW estimator psi_hat_k with the outcome regression m_k; clarify what exactly is being assumed about the nuisance estimators and their convergence rates.
- [Section 4.2 and Table 2] Table 2 reports simulation results for an AIE estimator, but Section 4.2 mainly describes an estimator of the ACD; clarify how the estimator is adapted for the integer-valued treatment and how the AIE target is defined with bins equal to individual treatment values.
- [Figure 3] The y-axis labels are hidden for confidentiality, which makes it difficult for readers to assess the magnitudes and scales of the estimated dose-response curve, bin-to-bin effects, and bin proportions; consider unblinding the labels or providing a more detailed description in the caption.
Circularity Check
No significant circularity: the RORR bias result is derived from stated structural assumptions, and the Coarsened AIPW consistency proof assumes high-level regularity conditions rather than redefining the estimand.
full rationale
The paper's central theoretical claim (Proposition 1) follows from the structural model (5), conditional ignorability, smoothness of f, and consistency of the nuisance estimators. The RORR estimand θ̃ = E[(T−h(X))f(T)]/E[(T−h(X))^2] is not defined in terms of the ACD, and the bias decomposition into A and B is algebraic manipulation rather than a restatement of the target. The Coarsened AIPW consistency proof (Proposition 2) assumes, rather than derives, uniform consistency of cross-fitted bin-level AIPW components, max_k|ψ̂_k−ψ_k|=o_p(1), and uniform positivity (23). These are high-level regularity conditions; they do not presuppose the conclusion ψ̂→E[f'(T)], because the population bin means still have to be converted into a derivative estimate via Lemma 1 and the subsequent Lipschitz and Riemann-sum argument. The only author self-citation ([19], Lal 2024) supports a corollary about treatment-effect ranking and is not load-bearing for the main derivation. The empirical application's zero-inflated treatment likely violates the smoothness and positivity assumptions of Proposition 2, and the converse in Proposition 1 is asserted rather than fully proven; however, these are correctness or robustness gaps, not circular reductions. No step in the claimed derivation chain reduces by construction to its own inputs.
Assumptions & free parameters
free parameters (2)
- number of bins K =
5 (Netflix application); O(N^{1/7}) recommended
- bin cutpoints =
zero bin plus quartiles of nonzero usage rates
assumptions (6)
- domain assumption Conditional ignorability: E[e_i|X_i]=0, E[u_i|X_i]=0, E[e_i u_i|X_i]=0
- domain assumption Positivity: Pr(T_i in S_k | X_i=x) >= epsilon*ell uniformly in x and k
- domain assumption Smoothness: f and p(t|x) twice continuously differentiable, f' Lipschitz
- ad hoc to paper Uniform nuisance consistency: max_k |psi_hat_k - psi_k| = o_p(1) under cross-fitting
- domain assumption Structural model Y_i = f(T_i) + g(X_i) + e_i with a common f for all units
- standard math Non-degeneracy: E[(T_i - h(X_i))^2] > 0
Cite this review
Pith. "Pith review of Estimating Representative Causal Effects with Double Machine Learning." pith.science (2026). https://pith.science/paper/AK234S23
@misc{pith2026250607462,
author = {Pith},
title = {Pith review of: Estimating Representative Causal Effects with Double Machine Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/AK234S23}},
note = {Machine review of arXiv:2506.07462}
}
read the original abstract
Double Machine Learning is widely used to estimate treatment effects from non-experimental data. The "residuals-on-residuals" regression (RORR) is especially popular for its simplicity and computational tractability. However, with heterogeneous treatment effects, the proper interpretation of RORR may not be well understood. We show that, for non-binary treatments with continuous dose-response functions, RORR estimates a conditional variance-weighted average of derivatives evaluated at treatment values not in the observed dataset. This estimand does not equal the Average Causal Derivative (ACD) in general. Hence, even if all units share the same dose-response function, RORR does not estimate an average treatment effect in the population represented by the sample. We propose an alternative estimator for the ACD that is well suited to the large datasets found in applied data science settings. We demonstrate the pitfalls of RORR and the favorable properties of the proposed estimator through an illustrative numerical example and with real-world data from Netflix. Our methodology is used by default in Netflix's observational causal inference platform, where it regularly powers causal research and decision-making at scale.
Figures
Forward citations
Cited by 1 Pith paper
-
A Human-Augmenting Agentic Workflow for Observational Causal Inference
A scaffolded, human-in-the-loop agentic workflow outperforms less structured LLM baselines and matches hand-tuned methods on several observational causal inference benchmarks.
Reference graph
Works this paper leans on
-
[17]
Edward H Kennedy. 2024. Semiparametric doubly robust targeted double machine learning: a review.Handbook of statistical methods for precision medicine(2024), 207–236
work page 2024
-
[1]
Joshua D. Angrist. 1998. Estimating the Labor Market Impact of Voluntary Military Service Using Social Security Data on Military Applicants. 66 (1998), 249–288. Issue 2. http://www.jstor.org/stable/2998558 Conference’17, July 2017, Washington, DC, USA Apoorva Lal and Winston Chou Figure 3: Treatment Effects of Feature A Usage on Netflix Visits After AIPW ...
-
[2]
Joshua D. Angrist and Alan B. Krueger. 1999.Chapter 23 - Empirical Strategies in Labor Economics. Vol. 3. Elsevier, 1277–1366. http://www.sciencedirect.com/ science/article/pii/S1573446399030047
work page 1999
-
[3]
Peter M. Aronow and Cyrus Samii. 2016. Does Regression Produce Representative Estimates of Causal Effects? 60 (2016), 250–267. Issue 1. https://onlinelibrary. wiley.com/doi/abs/10.1111/ajps.12185
-
[4]
Peter C Austin and Elizabeth A Stuart. 2015. Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies.Statistics in medicine34, 28 (2015), 3661–3679
work page 2015
-
[5]
Anna Baiardi and Andrea A. Naghi. 2024. The Effect of Plough Agriculture on Gender Roles: A Machine Learning Approach.Journal of Applied Econometrics 39, 7 (2024), 1396–1402. doi:10.1002/jae.3083
-
[6]
Peter J. Bickel, Chris A. J. Klaassen, Ya’Acov Ritov, and Jon A. Wellner. 1993. Efficient and adaptive estimation for semiparametric models. Vol. 4. Johns Hopkins University Press
work page 1993
-
[7]
2024.Round 2: A Survey of Causal Inference Applications at Netflix
Netflix Technology Blog. 2024.Round 2: A Survey of Causal Inference Applications at Netflix. https://netflixtechblog.com/round-2-a-survey-of-causal-inference- applications-at-netflix-fd78328ee0bb Accessed: 2025-10-06
work page 2024
Show all 26 references
-
[8]
Burden and J
Richard L. Burden and J. Douglas Faires. 2011. Numerical Analysis
2011
-
[9]
Cattaneo
Matias D. Cattaneo. 2010. Efficient semiparametric estimation of multi-valued treatment effects under ignorability. 155 (2010), 138–154. Issue 2. https://www. sciencedirect.com/science/article/pii/S030440760900236X
2010
-
[10]
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. 2018. Double/Debiased Machine Learning for Treatment and Structural Parameters. 21 (2018), C1–C68. Issue 1. http://doi.wiley.com/10.1111/ectj.12097
2018 doi
-
[11]
Victor Chernozhukov, Hiroyuki Kasahara, and Paul Schrimpf. 2020. Causal impact of masks, policies, behavior on early COVID-19 pandemic in the US. Journal of Econometrics220, 1 (2020), 23
2020
-
[12]
Hochachka, Shawn Ligocki, Lauren O
Daniel Fink, Alison Johnston, Matt Strimas-Mackey, Tom Auer, Wesley M. Hochachka, Shawn Ligocki, Lauren O. Jaromczyk, Orin Robinson, Chris Wood, Steve Kelling, and Amanda D. Rodewald. 2023. A Double Machine Learning Trend Model for Citizen Science Data.Methods in Ecology and E...
2023 doi
-
[13]
David Holtz, Michael Zhao, Seth G Benzell, Cathy Y Cao, Mohammad Amin Rahimian, Jeremy Yang, Jennifer Allen, Avinash Collis, Alex Moehring, Tara Sowrirajan, et al. 2020. Interdependence and the cost of uncoordinated responses to COVID-19.Proceedings of the National Academy of ...
2020
-
[14]
Guido W. Imbens. 2000. The role of the propensity score in estimating dose- response functions. 87 (2000), 706–710. Issue 3. https://academic.oup.com/ biomet/article/87/3/706/284466
2000
-
[15]
Guido W. Imbens. 2004. Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review. 86 (2004), 4–29. Issue 1. https://doi.org/10.1162/ 003465304323023651
2004
-
[16]
Jake Alton Jares and Neil Malhotra. 2025. Policy impact and voter mobilization: Evidence from farmers’ trade war experiences.American Political Science Review 119, 2 (2025), 847–869
2025
-
[18]
Kennedy, Zongming Ma, Matthew D
Edward H. Kennedy, Zongming Ma, Matthew D. McHugh, and Dylan S. Small
-
[19]
Apoorva Lal. 2024. Does Regression Produce Representative Causal Rankings? arXiv:2411.02675 [econ.EM] https://arxiv.org/abs/2411.02675
2024 arXiv
-
[20]
Maya L Petersen, Kristin E Porter, Susan Gruber, Yue Wang, and Mark J Van Der Laan. 2012. Diagnosing and responding to violations in the positivity as- sumption.Statistical methods in medical research21, 1 (2012), 31–54
2012
-
[21]
Robins, Andrea Rotnitzky, and Lue Ping Zhao
James M. Robins, Andrea Rotnitzky, and Lue Ping Zhao. 1994. Estimation of regression coefficients when some regressors are not always observed.J. Amer. Statist. Assoc.89, 427 (1994), 846–866
1994
-
[22]
P. M. Robinson. 1988. Root-N-Consistent Semiparametric Regression. 56 (1988), 931–954. Issue 4. http://www.jstor.org/stable/1912705
1988
-
[23]
Tymon Słoczyński. 2022. Interpreting OLS Estimands When Treatment Effects Are Heterogeneous. 104 (2022), 501–509. Issue 3. https://direct.mit.edu/rest/ article/104/3/501/97692/Interpreting-OLS-Estimands-When-Treatment-Effects
2022
-
[24]
Xinyu Wei, Mingwang Cheng, Kaifeng Duan, and Xiangxing Kong. 2024. Effects of Big Data on PM2.5: A Study Based on Double Machine Learning.Land13, 3 (2024), 327. doi:10.3390/land13030327
2024 doi
-
[25]
Shlomo Yitzhaki. 1996. On Using Linear Regressions in Welfare Economics. 14 (1996), 478–486. Issue 4. http://www.jstor.org/stable/1392256 Does Residuals-on-Residuals Regression Produce Representative Estimates of Causal Effects? Conference’17, July 2017, Washington, DC, USA A ...
1996
-
[2017]
Non-parametric methods for doubly robust estimation of continuous treatment effects.Journal of the Royal Statistical Society, Series B79, 4 (2017), 1229–1245
2017
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.