REVIEW 2 major objections 6 minor 5 references
Honesty in Causal Forests: When It Helps and When It Hurts
T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Honest estimation—the default in causal-forest software—can reduce the accuracy of individual treatment effect estimates by limiting the data used to detect heterogeneity, and the cost can reach 27% more training data.
desk verdict Useful large-scale confirmation of an existing finding, with a clean bias-variance story and a 27% data-cost number that deserves scrutiny before you repeat it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a bias–variance decomposition of causal forest predictions. Each tree's error splits into approximation error—the gap between the leaf-level subpopulation average treatment effect and the individual's true CATE—and estimation error, which under adaptive estimation carries a selection bias because the same sampling error that makes a split look attractive also inflates its leaf-level estimates. Honesty eliminates estimation bias by construction, because separating the splitting and estimation samples makes leaf assignment and effect estimation independent, while increasing approximation bias because smaller splitting samples detect heterogeneity less well. At the forest level, variance decomposes into target coupling, noise overlap, and target–estimation spillover, and honesty reduces variance by removing the spillover and shrinking noise overlap at the price of higher bias. A stylized single-split example with exactly one informative binary feature shows adaptive estimation selecting the informative feature more often and, despite a positive selection bias, ending with smaller overall bias. The empirical analysis is organized around the signal-to-noise ratio $\text{SNR} = \operatorname{Var}(\beta(X))/\operatorname{Var}(\xi)$, the $S^2$ metric $1 - \text{MSE}_{\text{te}}/\operatorname{Var}_{\text{te}}(\beta)$, and tuning on transformed outcomes with a one-standard-error selection rule that defaults to honesty.
What would settle it
Re-run the ACIC comparison with both forests tuned on the benchmark's ground-truth CATEs instead of the transformed-outcome proxy, holding everything else fixed: if honest forests then match or beat adaptive forests, the claimed cost of honesty is an artifact of the tuning protocol, and if the gap persists, the central claim survives its most direct challenge.
Extended reading notes
Core claim
The central claim is that honest estimation degrades rather than protects the accuracy of individual treatment effect estimates in causal forests. Because honesty splits the data, trees have less information to detect where effects differ, and the resulting approximation bias—summarizing heterogeneous individuals into poorly aligned leaves—typically outweighs the estimation bias and selection effects that honesty removes, so overall bias is lower under adaptive estimation. Empirically, across 7,004 benchmark datasets with detectable heterogeneity, adaptive forests capture a larger share of explainable effect variance ($S^2$) than honest forests in most signal-to-noise deciles, and the cost of defaulting to honesty grows with signal strength, reaching the equivalent of 27% more data in the strongest-heterogeneity settings. The paper reads honesty as a regularization choice that trades bias for variance: honest forests have lower variance in every regime, but the variance gap closes faster than the bias gap as the signal strengthens, which is why adaptive estimation wins where heterogeneity is detectable. The authors do not declare adaptive estimation universally better: when the signal is weak, honesty can match adaptive performance with roughly 7% less data, but in that regime the signal is often too weak for either forest to model heterogeneity reliably.
Load-bearing premise
The comparison assumes that tuning both methods on transformed outcomes with a one-standard-error rule that defaults to honesty gives the two approaches fairly matched model complexity; if that tuning proxy systematically disadvantages honest forests, the headline performance gaps and the 27% data-cost estimate could be tuning artifacts rather than properties of honesty itself.
Editorial extensions
If this is right
- Honesty should be treated as a hyperparameter: practitioners should tune over it rather than accept the software default, because its value depends on signal strength, sample size, and the goal of the analysis.
- Adaptive estimation is the more defensible default for point prediction of individual treatment effects: it outperforms honesty in most settings, and its advantage grows as effect heterogeneity becomes stronger and easier to detect.
- Defaulting to honesty carries a concrete data cost: in the strongest-heterogeneity settings an honest forest needs up to 27% more training data to match an adaptive forest's accuracy, a number that belongs in experimental design conversations.
- Honesty keeps a distinct role for uncertainty quantification: its theoretical guarantees support confidence intervals and hypothesis tests, and the paper argues that point estimation and inference need not use the same model—an adaptive forest for estimates and an honest forest for testing can coexist.
- The bias–variance reading carries beyond forests: the same tension between separating selection from estimation and preserving flexibility appears in variable selection and meta-learners, and the paper reproduces the pattern with an adaptive-versus-honest Lasso on the same benchmarks.
Reading between the lines
- The paper's own regularization account implies a middle ground it does not explore: the fraction of data reserved for estimation could be tuned continuously, and if a small split captures most of honesty's variance reduction at a fraction of its approximation cost, a partially honest forest would dominate both endpoints on the same benchmarks.
- The headline metric is point-prediction accuracy ($S^2$); downstream uses such as budget-constrained targeting or policy learning optimize a decision value rather than an error, and the honest-versus-adaptive ranking could differ under that loss even where $S^2$ favors adaptivity.
- The observed crossover between methods is sample-size dependent, so in deployed systems the honest/adaptive decision should be revisited as data accumulates rather than fixed once; the paper's learning-curve analysis illustrates the crossover but leaves this operational implication implicit.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies whether the standard practice of honest estimation in causal forests—separating the sample used to build leaves from the sample used to estimate leaf-level effects—improves point prediction of conditional average treatment effects. It contrasts honest estimation (HE) with adaptive estimation (AE), argues that honesty acts as a form of regularization in a bias-variance trade-off, and supports this with a stylized analytical example, a covariance decomposition, and a large empirical study on the ACIC benchmark (7,500 datasets) plus a MegaFon learning-curve analysis. The main empirical claims are that AE is often more accurate, that an empirical selector combining the two dominates both defaults, and that defaulting to honesty can cost up to 27% more data in high-signal regimes. The paper concludes that honesty should be treated as a tunable design choice rather than a universal default.
Significance. If the empirical results are validated, the paper makes a practically important point: the default use of honesty in software such as grf and EconML may be counterproductive for individual-level CATE prediction in high-signal settings. The paper's strengths include a large public semi-synthetic benchmark with ground-truth CATEs, a clean stylized model that isolates selection bias versus approximation bias, a transparent decomposition of forest variance into target coupling, noise overlap, and spillover, and a quantitative falsifiable claim (the 27% data-cost figure). The analytical section is self-contained, and the empirical workflow is reproducible in principle from public data and libraries. The main risk is that the headline numbers rest on an unvalidated tuning protocol; this is addressable and does not undermine the value of the paper if the authors can provide the requested sensitivity analysis.
major comments (2)
- [Section 4.2, Eq. (35); Figures 4–6] The tuning protocol is not validated against oracle tuning, and this is load-bearing for the central empirical claims. The HE min_samples_leaf grid is fixed at exactly half the AE grid (10...320 for AE versus 5...160 for HE), and both methods are selected by 5-fold cross-validation on transformed-outcome MSE, Eq. (35). The paper never verifies that the selected leaf counts for HE and AE actually match, nor that the transformed-outcome proxy selects the same model complexity as would be selected on the true CATEs—even though true CATEs are available in the ACIC benchmark. The Section 4.4 leaf-matched regimes use the same assumed 2m/m equivalence, so they do not independently resolve the issue. If the proxy or the grid matching systematically disadvantages HE, the AE advantage in Table 1, the 27% data-cost estimate in Figure 5, and the bias-variance comparisons in Figure 6 could be tuning artifacts rather than intrinsic properties of honesty. The one-standard-error rule defaults to HE and therefore cannot explain AE's advantage, which makes the tuning stage the unvalidated link in the argument. I ask the authors to report the actual leaf-count distributions, validate the transformed-outcome proxy against oracle tuning on the known CATEs, and recompute the headline quantities under both tuning rules.
- [Appendix A, Eq. (9)/(49)] The proposition that the splitting criterion is an unbiased estimator of the MSE reduction is not correct in finite samples. For independent child samples, E[(\hat{\tau}_1 - \hat{\tau}_2)^2] = (\tau_1 - \tau_2)^2 + Var(\hat{\tau}_1) + Var(\hat{\tau}_2), so the squared difference contains additive variance terms. The random weight n_{\ell_1} n_{\ell_2}/(n_{\ell_1}+n_{\ell_2})^2 does not remove these terms and introduces additional sampling randomness. Thus Eq. (49) is at best a consistent or heuristic proxy for the MSE gain, not an unbiased estimate as stated. The authors should either correct the proposition, add the additional assumptions under which unbiasedness holds, or reframe the claim as an asymptotic or ranking-equivalent criterion.
minor comments (6)
- [Section 3.4] The sentence 'Theory predicts it helps when SNR is low and hurts when it is high' is stronger than the derivations support: Appendix B derives a bias ordering and Appendix C derives component-wise variance effects, but no complete MSE dominance theorem is proved. I suggest softening this to 'the analytical decomposition suggests' or adding a formal statement if intended.
- [Section 3.3.1 and Figure 2] The main text says that empirically HE produces lower target coupling than AE even at high SNR, while Figure 2's caption and the surrounding discussion say HE increases target coupling in high-SNR settings. Please clarify that Figure 2 is the theoretical prediction and Figure 7 is the empirical result, and reconcile the wording so the contrast is explicit.
- [Appendix B, Eq. (64)] The analytical example assumes the split rule \hat{\ell} = \arg\max_j |\Delta_j|, which is not the EconML splitting criterion of Eq. (9). The equivalence holds for balanced binary splits, but for the general algorithm it is only heuristic; please state this explicitly when interpreting the empirical results through the theory.
- [Section 4.3, Figure 5] The calculation of the 'extra data honesty needs' is described only verbally. Please specify how the required training size is interpolated or selected when, at a given sample size, neither method significantly outperforms the other at the 5% level, and state whether the 27% figure is a decile mean or a maximum.
- [Throughout] There are minor typographical issues, including 'ashonest estimation' in Section 1 and the spaced author name 'Y anfang Hou' in the header; these should be corrected in the final version.
- [Introduction and Appendix H] The text cites two recent Bachelor's theses (Havelka 2022; Prodan 2025) that already report context-dependent advantages of adaptive versus honest estimation. The novelty statement in the introduction should more clearly position the present paper relative to these works, emphasizing the scale, the 27% quantitative estimate, and the variance decomposition rather than claiming the trade-off itself is new.
Circularity Check
No significant circularity: the analytical derivation is self-contained and the empirical claims rest on an external benchmark; the unvalidated tuning proxy is an experimental-design concern, not a circular reduction.
full rationale
The paper's central claims are not circular. The analytical bias result (Section 3.2, Appendix B) derives from explicit distributional assumptions and a stated splitting rule; the conclusion that AE selects the informative feature more often than HE follows from the larger splitting sample in AE, not from an input disguised as a result. The variance decomposition (Section 3.3, Appendix C) is a definitional identity plus a stated 'forest honesty' abstraction; it is used to interpret, not to manufacture, the empirical findings. The empirical study (Section 4) tunes AE and HE separately via 5-fold cross-validation on transformed outcomes, then evaluates on ground-truth CATEs from the external ACIC benchmark. No fitted parameter is renamed as a prediction, and the 27% data-cost figure is measured from these tuned models rather than derived from an input. The one-standard-error selection rule defaults to HE, so it cannot by construction produce the observed AE advantage; indeed Table 1 shows AE has lower regret even under this conservative rule. The halved min_samples_leaf grid (Section 4.2) and the m/2m leaf-matching equivalence (Section 4.4) are heuristic complexity-matching assumptions, and the paper does not validate the transformed-outcome proxy against oracle tuning on true CATEs; this is a legitimate experimental-design concern, but it is not a circular reduction: the tuning criterion and the evaluation metric are distinct. Self-citations (Fernández-Loría and Provost, 2025) appear only in the broader discussion and in the Appendix H survey, not as load-bearing support for the paper's own empirical or analytical conclusions. No uniqueness theorem, fitted-input-as-prediction step, or ansatz-smuggling-via-citation is present. The paper is self-contained against an external benchmark and its derivation chain does not collapse into its inputs.
Assumptions & free parameters
assumptions (6)
- domain assumption Ignorability and random treatment assignment: Y(1),Y(0) independent of T given X, with treatment assigned uniformly at random.
- standard math The splitting criterion in Eq. (9) is an unbiased estimate of the MSE gain, as proved in Appendix A, which presumes balanced sampling and unbiased leaf SPATE estimates.
- ad hoc to paper In the stylized example, exactly one binary feature X1 is informative and independent of all other features, outcomes are homoscedastic, and SPATE estimates are approximately jointly normal.
- standard math The transformed outcome z_i has E[z_i | X_i] equal to the CATE, so minimizing transformed-outcome MSE is a valid model selection criterion.
- ad hoc to paper Forest honesty: the idealized condition that E[epsilon_b | leaves of all trees] = 0, so estimation errors are mean-zero conditional on all leaf assignments.
- ad hoc to paper SPATE estimates are sample means of i.i.d. transformed outcomes, and estimation subsamples are drawn uniformly and independently of outcomes under honesty.
Cite this review
Pith. "Pith review of Honesty in Causal Forests: When It Helps and When It Hurts." pith.science (2026). https://pith.science/paper/FE6FEC3M
@misc{pith2026250613107,
author = {Pith},
title = {Pith review of: Honesty in Causal Forests: When It Helps and When It Hurts},
year = {2026},
howpublished = {\url{https://pith.science/paper/FE6FEC3M}},
note = {Machine review of arXiv:2506.13107}
}
read the original abstract
Causal forests estimate how treatment effects vary across individuals, guiding personalized interventions in areas like marketing, operations, and public policy. A standard practice is honest estimation: dividing the data into two samples, one to define subgroups and another to estimate treatment effects within them. This is intended to reduce overfitting and is the default in many software packages. But is it the right choice? We show that honest estimation can reduce the accuracy of estimates of individual treatment effects, especially when effect heterogeneity is substantial and datasets are large enough to detect it. The reason is a bias-variance trade-off: honesty lowers the risk of overfitting but increases the risk of underfitting by limiting the data available to detect and model heterogeneity. Across more than 7,000 benchmark datasets, we find that the cost of using honesty by default can be as high as requiring 27% more data to match the performance of models trained without it. Honesty is best understood as a form of regularization. Whether to adopt it should depend on the goals of the application and its empirical performance, not on reflexive default use.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[2]
across SNR deciles for adaptive, cross-fit honest, and single-fit honest Lasso models.AE underperforms HE in the lowest SNR decile and is preferable when SNR is higher. This highlights the regularization effect of honesty. The weak performance of the single-fit HE emphasizes the efficiency loss of sample splitting, which is mitigated via cross-fitting. Fi...
work page 2016
-
[2015]
Stochastic optimization forests
Nathan Kallus and Xiaojie Mao. Stochastic optimization forests. Management Science, 69(4): 1975–1994,
work page 1975
-
[2019]
Consistency of Honest Decision Trees and Random Forests
Version 0.x. Martin Bladt and Rasmus Frigaard Lemvig. Consistency of honest decision trees and random forests. arXiv preprint arXiv:2601.14991,
-
[2022]
Accuracy Limits of Causal Trees for Individualized Treatment Effects
Matias D Cattaneo, Jason M Klusowski, and Ruiqi Rae Yu. The honest truth about causal trees: Accuracy limits for heterogeneous treatment effect estimation. arXiv preprint arXiv:2509.11381,
-
[2026]
Matias D Cattaneo, Jason M Klusowski, and Peter M Tian. On the pointwise behavior of recur- sive partitioning and its implications for heterogeneous causal effect estimation. arXiv preprint arXiv:2211.10805,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.