REVIEW 4 major objections 6 minor 36 references
Incorporating structural uncertainty in causal decision making
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Uncertainty about which way causation runs should be averaged over, not selected away, in causal decision-making.
desk verdict Clean decision-theoretic framing of when structural uncertainty matters, but the empirical support overreaches because the bootstrap structural probabilities are never calibrated and Proposition 3 doesn't apply to the simulated loss. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing objects are the posterior structure probabilities $P(G_i\mid D)$ and the averaging rule of Equation (15) that weights each structure's conditional expected loss by those probabilities. A third piece is Proposition 3's lower bound, which decomposes the benefit of averaging into the action gap $\Delta=|a_1^*-a_2^*|$, the loss sensitivity $\kappa$, the minimum structural probability, and the model-selection error rate. The bound does the work of saying when structural uncertainty is worth taking seriously.
What would settle it
Run repeated simulations with known true direction and, for each, record the bootstrap $P(G_1\mid D)$; check calibration by seeing whether the empirical frequency of $G_1$ among cases with $P(G_1\mid D)\approx p$ equals $p$. A calibration failure there, or deliberately overconfident structure probabilities plugged into the decision rule, would predict that the positive $\Delta L$ found in Section 6.6.1 shrinks or reverses.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that the action rule $a^{MA}=\arg\min_a \sum_i \mathbb{E}[L(a,G_i,\theta_i)\mid G_i,D]\,P(G_i\mid D)$, which weights each structure's expected loss by its posterior probability, is the decision-theoretically correct response to structural uncertainty. Theorems 1 and 2 state that under a well-specified hierarchical model this rule minimizes posterior expected loss and frequentist risk over all measurable decision rules. Proposition 3 quantifies the advantage over model selection: the expected loss saving is at least $\kappa\,\Delta\,\min_i P(G_i\mid D)\,P_{\mathrm{err}}$, where $\Delta$ is the gap between optimal actions, $\kappa$ is loss sensitivity, and $P_{\mathrm{err}}$ is the probability that selection picks the wrong structure. The simulation study then shows the rule in action: averaging outperforms selection on average across heteroskedastic and nonlinear DGPs, with the paper noting that the practical bottleneck is the quality of $P(G\mid D)$.
Load-bearing premise
The practical gain depends on $P(G\mid D)$ being a well-calibrated posterior probability over causal structures; the paper computes it as the frequency of bootstrap samples favoring a direction, offers no calibration check, and concedes that bootstrapping is only a heuristic, so miscalibration can flip the conclusion and make model selection win.
Editorial extensions
If this is right
- A decision maker facing two plausible causal directions should use the posterior-weighted action rather than the action of the most probable structure, unless the probabilities are known to be miscalibrated.
- The biggest gains from averaging should appear in small samples, where discovery is uncertain, and in settings where the two structures recommend very different interventions.
- If the loss function is insensitive to action deviations or the optimal actions under the two structures nearly coincide, structural uncertainty does not matter and averaging buys little.
- Because the simulation evidence depends on causal discovery methods exploiting nonlinear or heteroskedastic data, averaging will not deliver its advertised benefit on data where the direction is not identifiable.
Reading between the lines
- A practical next step would be to recalibrate bootstrap-derived structure probabilities on data with known direction before using them as decision weights; the paper's own caveat suggests this is where the method would stand or fall.
- The same logic implies a cheap diagnostic for practitioners: estimate the gap between optimal actions under candidate structures; if it is tiny, skip structural uncertainty quantification entirely.
- For multivariate problems, exact averaging over all DAGs is infeasible, and an approximate posterior over a pruned set would inherit the same calibration risk; the optimality theorems would need re-examination rather than automatic extension.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies whether and when uncertainty about the direction of a bivariate causal relation (X → Y versus Y → X) should be incorporated into decisions, and it proposes Bayesian model averaging over the two candidate structures as an alternative to model selection. It formalizes a decision-theoretic setup, states conditions under which averaging helps (moderate structural uncertainty, differing optimal actions, sensitive loss functions), proves Bayesian and frequentist optimality of averaging under a well-specified hierarchical model, and reports simulations in which bootstrap-derived structural probabilities are used to weight the two structures. The headline simulation result is an average loss reduction of ΔL = 0.103 across 6,400 runs, which the authors interpret as direct empirical support for Theorems 1 and 2. The paper closes with a discussion of limitations and extensions to multivariate settings.
Significance. The topic is relevant: structural uncertainty is often ignored in applied causal inference, and a practical recipe that connects causal discovery output to decision-making would be a useful addition. The paper is clearly written, and it is honest about several limitations, especially in Sections 7 and 8 where it concedes that the bootstrap is not a Bayesian posterior and that calibration of P(G|D) remains an open problem. If the results were fully established, the paper would provide a simple justification for averaging over causal structures. However, the theoretically novel content is thin: Theorem 1 restates the definition of the proposed estimator, Theorem 2 is a standard Bayes-risk-minimization argument, and Proposition 3, which is the main characterization of when averaging helps, is stated without proof and is applied to a loss function that does not satisfy its key condition. The empirical section does not calibrate the bootstrap structural probabilities and uses plug-in effect estimates, so the headline ΔL does not directly validate the optimality theorems. The framework is salvageable, but the main practical claims need substantial additional support.
major comments (4)
- [Section 6.6.1; Sections 6.2, 6.3, 7.1] The claim that ΔL = 0.103 'directly supporting Theorems 1 and 2' is not warranted. Theorems 1 and 2 are conditional on Assumption 2, under which P(G_i|D) are posterior probabilities from a well-specified hierarchical model and the action minimizes posterior expected loss over parameters as well as structures. In the simulations, P(G1|D) is the proportion of bootstrap samples in which a heuristic score favors X → Y (Section 6.2), and Section 7.1 concedes that bootstrapping 'is not a direct Bayesian posterior.' Moreover, Section 6.3 uses point estimates of the causal effect rather than posterior parameter expectations. No calibration check, reliability diagram, or proper-scoring-rule evaluation for P(G1|D) is reported anywhere in Section 6. The observed positive ΔL therefore demonstrates only that this particular bootstrap-weighting scheme beats hard thresholding on these specific DGPs; it does not provide empirical validation of the optimality theorems.
- [Section 5.1, Eqs. (15) and (18)] Theorem 1 is true by construction: Eq. (15) defines a_MA as the minimizer of the posterior expected loss, so Eq. (18) is a restatement of that definition. Theorem 2 then follows immediately from the standard Bayes-risk argument in Eqs. (21)–(22), and the proof given is correct. Presenting these as the paper's 'optimality results' overstates their content. The authors should either reframe them as formal observations that connect the proposed estimator to standard decision theory, or add a substantive result such as finite-sample regret bounds or conditions under which the two-stage model selection rule is approximately optimal.
- [Section 4.3, Proposition 3; Section 6.5] Proposition 3 is stated without proof, and its application to the simulation loss is invalid. Definition 1 requires |L(a1, Gi, θi) − L(a2, Gi, θi)| ≥ κδ for all pairs with |a1 − a2| ≥ δ, but the quadratic loss L(a) = 0.5(E − a)^2 + λa^2 is not κ-sensitive for any κ > 0: for E = 0, a1 = −ε, and a2 = ε, the loss difference is zero while |a1 − a2| = 2ε. The quantity 1 + 2λ is the second derivative of the loss, not the sensitivity constant of Definition 1. Consequently inequality (17) cannot be invoked for the main example used in the simulation study, and the paper's characterization of loss sensitivity as a driver of averaging benefits is not established for its own primary loss function.
- [Section 4.2, Proposition 2] Proposition 2 is stated without proof and is not generally true for arbitrary loss functions satisfying only Assumption 1. The minimizer of a weighted average of two nonconvex loss functions need not lie in the interval between the individual minimizers, so |a*_1 − a*_2| ≤ ε does not by itself imply |a_MA − a_MS| ≤ ε. A proof, or an explicit convexity assumption on the loss, is needed before this result can support the paper's tripartite characterization of when structural uncertainty matters.
minor comments (6)
- [Sections 4.1 and 5.2] Two different results are both labeled 'Proposition 1': the extreme-certainty result in Section 4.1 and the suboptimality-under-misspecification result in Section 5.2. These should be renumbered to avoid confusion.
- [Section 6.5] The text refers to 'Proposition 4.1' and 'Proposition 4.2' when discussing predictions about sample size and effect size effects; these should be the proposition numbers from Section 4, not section numbers.
- [Section 3.1.3] There is a grammatical error in the sentence 'the causal relationships is:' which should read 'the causal relationships are:'.
- [Table 3 and Table 4] The column header 'n' is used for the number of simulation runs, which conflicts with the sample size n used in Table 2 and throughout the paper; renaming this column 'N' would improve clarity.
- [Section 6.3] The description of causal effect estimation is ambiguous for model averaging runs in which the bootstrap favors G2: the MA action formula requires an estimate of the effect under G1, but the text only describes fitting G1-based models when the estimated direction is G1. Please clarify how Ä E(x) is obtained in those runs.
- [Section 6.4] In the derivation of the optimal action, the line after taking the derivative has a sign inconsistency: the equation should read −(E_true − a) + 2λa = 0, which then gives a(1 + 2λ) = E_true.
Circularity Check
Theorem 1 is a definitional tautology: a_MA is defined as the minimizer of posterior expected loss, and the theorem only restates that definition; the simulation is independent but cannot directly validate the conditional optimality theorem without calibrated P(G|D).
-
self definitional
[Section 5.1, Theorem 1 and its proof; Eqs. (15), (18)-(19)]
"The action aM A defined in Equation (15) is the minimizer of this expression by construction. Hence, it minimizes the posterior expected loss."
Equation (15) defines a_MA as the argmin over a in A of sum_i E[L(a,Gi,theta_i)|Gi,D] P(Gi|D). Theorem 1 asserts that this same action minimizes the posterior expected loss, and the proof shows the posterior expected loss equals exactly that sum by the law of total expectation, then concludes by construction. The optimality conclusion is therefore a restatement of the definition rather than a derived property. The substantive content is only the standard Bayes-rule identity; it does not by itself establish optimality of the practical bootstrap-weighted procedure used in the simulations.
full rationale
The only step that reduces by construction is Theorem 1. Theorem 2 is a standard Bayes-risk consequence of Theorem 1, so it inherits the same definitional status. The qualitative propositions in Sections 4.1-4.3 are non-circular, and the simulation study is an independent computational experiment: it generates data from specified DGPs, estimates structural weights by bootstrap frequencies, and compares model averaging with model selection. I did not count the bootstrap P(G|D) issue as circular: Section 7.1 concedes 'the bootstrapping approach, while providing a heuristic for uncertainty, is not a direct Bayesian posterior,' and no calibration check is reported, so the observed Delta L = 0.103 in Section 6.6.1 is evidence for a heuristic weighting scheme rather than direct support for the conditional optimality theorems; this is a validity and calibration gap, not a reduction of the simulation to its inputs. No load-bearing self-citations, imported uniqueness theorems, or renamed known results were found. Because the paper's central formal optimality claim is a tautology, the score is 6 rather than 0-2, even though the simulation work has independent content.
Assumptions & free parameters
assumptions (5)
- domain assumption Assumption 1: the action space A is compact, the loss function is jointly measurable and finite a.s., and continuous in a for each structure/parameter.
- domain assumption Assumption 2: the data-generating process follows the hierarchical model within the candidate set, and the analyst's priors match the true generative process.
- domain assumption Under structure G2 (Y to X), the causal effect of intervening on X on Y is exactly zero.
- ad hoc to paper The bootstrap proportion in Section 6.2 approximates the posterior structure probability P(Gi|D).
- ad hoc to paper The quadratic loss used in simulations satisfies the kappa-sensitivity condition of Definition 1 with kappa = 1 + 2*lambda.
Cite this review
Pith. "Pith review of Incorporating structural uncertainty in causal decision making." pith.science (2026). https://pith.science/paper/Y3SUJ76A
@misc{pith2026250723495,
author = {Pith},
title = {Pith review of: Incorporating structural uncertainty in causal decision making},
year = {2026},
howpublished = {\url{https://pith.science/paper/Y3SUJ76A}},
note = {Machine review of arXiv:2507.23495}
}
abstract
Practitioners making decisions based on causal effects typically ignore structural uncertainty. We analyze when this uncertainty is consequential enough to warrant methodological solutions (Bayesian model averaging over competing causal structures). Focusing on bivariate relationships ($X \rightarrow Y$ vs. $X \leftarrow Y$), we establish that model averaging is beneficial when: (1) structural uncertainty is moderate to high, (2) causal effects differ substantially between structures, and (3) loss functions are sufficiently sensitive to the size of the causal effect. We prove optimality results of our suggested methodological solution under regularity conditions and demonstrate through simulations that modern causal discovery methods can provide, within limits, the necessary quantification. Our framework complements existing robust causal inference approaches by addressing a distinct source of uncertainty typically overlooked in practice.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Causal inference in statistics, social, and biomedical sciences
Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. 2015
work page 2015
-
[2]
P Richard Hahn, Jared S Murray, and Carlos M Carvalho. Bayesian regression tree models for causal inference: Regularization, confounding, and heterogeneous effects. Bayesian Analysis, 15(3):965–1056, 2020
work page 2020
-
[3]
Causation, pre- diction, and search
Peter Spirtes, Clark N Glymour, and Richard Scheines. Causation, pre- diction, and search . MIT Press, 2nd edition, 2000
work page 2000
-
[4]
Review of causal discov- ery methods based on graphical models
Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discov- ery methods based on graphical models. Frontiers in Genetics , 7:524, 2016
work page 2016
-
[5]
Statistical Decision Theory and Bayesian Analysis
James O Berger. Statistical Decision Theory and Bayesian Analysis . Springer-Verlag, 2nd edition, 1985
work page 1985
-
[6]
The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation
Christian P Robert. The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation . Springer-Verlag, 2nd edition, 2007
work page 2007
-
[7]
Jennifer L. Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217–240, 2011. doi: 10.1198/jcgs.2010.08162
arXiv 2011
-
[8]
Jiji Zhang. On the completeness of orientation rules for causal discovery in the presence of latent confounders and selection bias. Artificial Intel- ligence, 172(16–17):1873–1896, 2008. doi: 10.1016/j.artint.2008.08.001
Show all 36 references
-
[9]
Optimal structure identification with greedy search
David M Chickering. Optimal structure identification with greedy search. Journal of Machine Learning Research , 3:547–572, 2002
2002
-
[10]
Learning bayesian networks
David Heckerman. Learning bayesian networks. Communications of the ACM, 38(3):31–36, 1995
1995
-
[11]
Nonlinear causal discovery with additive noise models
Patrik O Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Sch¨ olkopf. Nonlinear causal discovery with additive noise models. Advances in Neural Information Processing Systems , 21:689– 696, 2009. 35
2009
-
[12]
Causal infer- ence in statistics: A primer
Jonas Peters, Dominik Janzing, and Bernhard Sch¨ olkopf. Causal infer- ence in statistics: A primer . MIT Press, 2017
2017
-
[13]
Information-geometric approach to inferring causal directions
Dominik Janzing, Joris Mooij, Kun Zhang, Jan Lemeire, Jakob Zscheis- chler, Povilas Daniusis, Bastian Steudel, and Bernhard Sch¨ olkopf. Information-geometric approach to inferring causal directions. Artifi- cial Intelligence , 182:1–31, 2012
2012
-
[14]
Bayesian graphical models for discrete data
David Madigan, Jeremy C York, and David Allard. Bayesian graphical models for discrete data. International Statistical Review, 63(2):215–232, 1995
1995
-
[15]
Daphne Eaton and Kevin P. Murphy. Exact bayesian structure learning from uncertain interventions. Journal of Machine Learning Research, 8: 627–660, 2007
2007
-
[16]
Bayesian model averaging: A tutorial
Jennifer A Hoeting, David Madigan, Adrian E Raftery, and Chris T Volinsky. Bayesian model averaging: A tutorial. Statistical Science, 14 (4):382–401, 1999
1999
-
[17]
Bayesian Model Selection in Social Research , vol- ume 25
Adrian E Raftery. Bayesian Model Selection in Social Research , vol- ume 25. American Sociological Association, 1995
1995
-
[18]
Optimal predictive model selection
Maria Maddalena Barbieri and James O Berger. Optimal predictive model selection. Annals of Statistics , 32(3):870–897, 2004
2004
-
[19]
Bayesian adap- tive sampling for variable selection and model averaging
Merlise A Clyde, Joyee Ghosh, and Michael L Littman. Bayesian adap- tive sampling for variable selection and model averaging. Journal of Computational and Graphical Statistics , 20(1):80–101, 2011
2011
-
[20]
Rosenbaum
Paul R. Rosenbaum. Observational Studies. Springer Series in Statistics. Springer, 2nd edition, 2002
2002
-
[21]
VanderWeele and Peng Ding
Tyler J. VanderWeele and Peng Ding. Sensitivity analysis in observa- tional research: introducing the e-value. Annals of Internal Medicine , 167(4):268–274, 2017. doi: 10.7326/M16-2607
2017 doi
-
[22]
Tchetgen Tchetgen
Matthew Masten, Bruno Poirier, and Eric J. Tchetgen Tchetgen. Sal- vaging a forced-choice instrumental variable design with sensitivity anal- ysis. Journal of the American Statistical Association , 115(531):242–254, 2020. 36
2020
-
[23]
Sensitivity analysis under unmeasured confound- ing: bounds on the average causal effect
Matteo Bonvini et al. Sensitivity analysis under unmeasured confound- ing: bounds on the average causal effect. Biometrika, 109(2):419–435, 2022
2022
-
[24]
Heejung Bang and James M. Robins. Doubly robust estimation in miss- ing data and causal inference models. Biometrics, 61:962–972, 2005. doi: 10.1111/j.1541-0420.2005.00377.x
2005
-
[25]
Joseph D. Y. Kang and Joseph L. Schafer. Demystifying double robust- ness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science, 22(4):523–539, 2007. doi: 10.1214/07-STS227
2007 doi
-
[26]
Double/debiased machine learning for treat- ment and structural parameters
Victor Chernozhukov et al. Double/debiased machine learning for treat- ment and structural parameters. The Econometrics Journal , 21(1):C1– C68, 2018
2018
-
[27]
Edward H. Kennedy. Optimal doubly robust estimation of heterogeneous causal effects. Journal of Machine Learning Research , 21:1–39, 2020
2020
-
[28]
Causal effect estimation under multi- ple candidate causal graphs
Amit Sharma and Emre Kiciman. Causal effect estimation under multi- ple candidate causal graphs. Advances in Neural Information Processing Systems, 34:15435–15447, 2021
2021
-
[29]
Learning causal structure under model uncertainty
Kyungah Jung et al. Learning causal structure under model uncertainty. Journal of Causal Inference , 9(2):1–23, 2021
2021
-
[30]
Charles F. Manski. Social choice with partial knowledge of treatment response. Journal of Econometrics , 65(1):137–158, 2004
2004
-
[31]
Dehejia and Sadek Wahba
Rajeev H. Dehejia and Sadek Wahba. Practical propensity score match- ing: a discriminator function approach. Journal of Econometrics , 125 (1–2):225–238, 2005
2005
-
[32]
Calibrating sensitivity analyses for unmeasured con- founding in observational studies
Zhi Wang et al. Calibrating sensitivity analyses for unmeasured con- founding in observational studies. Journal of the Royal Statistical Soci- ety: Series A , 175(2):513–533, 2012
2012
-
[33]
Doubly robust estimation of heterogeneous causal effects
Sebastian Dorn, Yunjie Guo, and Mark van der Laan. Doubly robust estimation of heterogeneous causal effects. Journal of Causal Inference , 9(1):1–30, 2021. 37
2021
-
[34]
Causality: Models, Reasoning, and Inference
Judea Pearl. Causality: Models, Reasoning, and Inference . Cambridge University Press, 2nd edition, 2009
2009
-
[35]
Causal discovery using regression-based conditional independence tests
Kun Zhang, Yu Li, Peng Zhang, Bo Huang, and Bernhard Sch¨ olkopf. Causal discovery using regression-based conditional independence tests. In Proceedings of the AAAI Conference on Artificial Intelligence , vol- ume 31, 2017
2017
-
[36]
An introduction to the boot- strap
Bradley Efron and Robert J Tibshirani. An introduction to the boot- strap. Monographs on statistics and applied probability , 57, 1994. 38
1994
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.