REVIEW 3 major objections 6 minor 32 references
Estimating Probabilities of Causation with Machine Learning Models
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read An MLP with the Mish activation function predicts the bounds of the probability of necessity and sufficiency for 32,768 subpopulations with a mean absolute error of about 0.02, after training on about 2,000 subpopulations with sufficient…
desk verdict Plausible ML transfer-learning pilot for PNS bounds, but the 'first' claims are overstated and the missing train/test split makes the headline MAE unverifiable as stated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the tight bounds on PNS: the lower bound is the maximum of 0, $P(y_x)-P(y_{x'})$, $P(y)-P(y_{x'})$, and $P(y_x)-P(y)$, while the upper bound is the minimum of $P(y_x)$, $P(y_{x'})$, $P(x,y)+P(x',y')$, and $P(y_x)-P(y_{x'})+P(x,y')+P(x',y)$. The paper converts these formulas into regression labels: for each subpopulation it computes the experimental and observational distributions from the known structural causal model, plugs them into the bound equations, and trains models to map the 15 observed binary features to those labels. The best-performing configuration is a multilayer perceptron with layers 15, 64, 32, 16, and 1, using the Mish activation $s\cdot\tanh(\ln(1+e^s))$, a smooth non-monotonic function chosen because the PNS lower bound has many exact zeros and Mish remains differentiable at zero.
What would settle it
Recompute the true PNS bounds from the appendix's fixed structural causal model for the subpopulations that fell below the 1,300-sample threshold, and compare the MLP(Mish) predictions against those values; if the mean absolute error on that held-out rare-subpopulation slice is substantially above 0.02, the generalization claim is refuted.
Extended reading notes
Core claim
The center of the paper is a supervised regression claim: the pair of tight PNS bounds is a function of a subpopulation's observable characteristics, and that function can be learned from a subset of subpopulations that have enough data. The paper fixes a structural causal model, computes the true bounds for all $2^{15}$ observed feature combinations, filters out subpopulations with fewer than 1,300 experimental or observational samples, and trains five model classes on the remaining about 2,000 entries. The reported outcome is that an MLP with the Mish activation function reaches mean squared error 0.0011 on the lower bound and 0.0010 on the upper bound, and mean absolute error 0.0225 and 0.0247, substantially better than support vector regression, random forest, gradient boosting, and the Transformer. The authors take this as evidence that machine learning is a feasible tool for estimating probabilities of causation when direct estimation is impractical.
Load-bearing premise
The roughly 2,000 subpopulations that have at least 1,300 experimental and observational samples are representative of all 32,768 subpopulations, so a model trained only on them can predict bounds for the rare subpopulations it never sees.
Editorial extensions
If this is right
- Subpopulation-level causal bounds can be obtained without collecting 1,300 experimental and observational samples for every subgroup, as long as some data-rich subgroups exist and share the same feature space.
- The reported 0.02 mean absolute error suggests the learned bounds are accurate enough to rank subgroups by PNS, for example in unit-selection tasks that look for subgroups with high lower bounds or low upper bounds.
- Because PN and PS have bounds of the same structural form, the same machine learning pipeline should extend to those probabilities, even though the experiments here report PNS only.
- Model and activation choice matter: the smooth Mish activation outperforms ReLU and LeakyReLU on a target with many zero labels, indicating that tailoring the architecture to the causal quantity's shape is important.
Reading between the lines
- The method's success on one synthetic structural causal model suggests a broader amortized-identification principle: a neural network can internalize the mapping from population features to causal bounds, effectively compressing experimental and observational distributions into a feature lookup.
- A stress test the paper does not run is to vary the number of unobserved confounders; as the gap between observed features and the true causal mechanism widens, the achievable accuracy should degrade, and the rate of degradation would tell practitioners how many confounders can safely be ignored.
- For real deployments, true PNS bounds are unknown, so the practical version of this claim would be tested by collecting a small set of high-quality subgroups as a validation set and checking whether predicted bounds agree with directly computed bounds on those held-out groups.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a supervised machine learning pipeline that predicts the Tian-Pearl bounds of the probability of necessity and sufficiency (PNS) for subpopulations whose own experimental and observational data are too sparse for direct estimation. The authors generate a synthetic structural causal model (SCM) with 15 observed and 5 unobserved binary features, compute exact PNS bounds for all 2^15 = 32,768 observed subpopulations from the SCM equations, and then train five models (SVM, random forest, GBDT, Transformer, and MLP with several activation functions) on the roughly 2,000 subpopulations that have at least 1,300 experimental and observational samples. The central claim is that an MLP with the Mish activation function achieves a mean absolute error of about 0.02 in predicting the PNS bounds across the 32,768 subpopulations, and the paper concludes that machine learning is a feasible tool for estimating probabilities of causation when subgroup-level data are insufficient.
Significance. If the claimed result holds as a genuine held-out generalization, the paper makes a useful feasibility contribution: it shows that a purely supervised map from observable subgroup features to causal-effect bounds can be learned from well-measured subpopulations and applied to poorly measured ones. The explicit SCM, the exact informer-data computation, the inclusion of code, and the comparison of several model families are strengths that make the result reproducible and testable. However, the significance is currently tempered by two issues: the evaluation protocol is under-specified, so the headline MAE is ambiguous, and the experiment is a single synthetic SCM with one fixed coefficient draw, so the external validity of the claim is unknown. The paper is best read as a proof-of-concept rather than a general claim about real-world probability-of-causation estimation.
major comments (3)
- [Section 4.3] The train/test split is never stated. Section 4.3 reports 2,054 lower-bound and 2,065 upper-bound training entries, while the abstract claims an MAE of about 0.02 for 32,768 subpopulations. If the reported errors were computed on a set that includes the training subpopulations, the result would be optimistically biased and would not directly support the generalization claim. Notably, the confusion matrices in Figure 4 sum to about 30,713 (lower bound) and 30,702 (upper bound), which is very close to 32,768 minus 2,054 and 32,768 minus 2,065, respectively. This suggests that the evaluation was actually performed on the complement of the training subpopulations, i.e., on exactly the subpopulations with insufficient data. The paper must state this explicitly, report the MAE/MSE separately on the held-out complement, and clarify whether the reported 32,768 figure includes the training subpopulations. This is load-bearing for the central claim.
- [Section 4.3 and Section 6] The training set is selected by requiring at least 1,300 experimental and observational samples per subpopulation, whereas the target population is subpopulations with insufficient data. The paper provides no analysis of the distribution shift between the training subpopulations and the test subpopulations, such as the range of features, the fraction of zero lower bounds, or the spread of PNS values in each group. If the held-out complement is indeed the test set, the reported MAE would already constitute an empirical answer for this SCM, but the paper should still describe the coverage of the training distribution and acknowledge the possibility that the learned map might not transfer to subpopulations outside the support of the training features. This is relevant to the generalization claim and should be addressed explicitly.
- [Section 3 and Appendix A] The experimental evidence rests on a single SCM instance with one fixed draw of the coefficient vectors MX, MY, CY and one fixed set of Bernoulli parameters. The MLP(MLP-Mish) result is therefore a statement about this particular data-generating process, not about PNS prediction in general. The paper acknowledges in the Discussion that the causal structure is simple and that future work should consider more complex SCMs, which is appropriate. However, the conclusion that 'machine learning is a powerful tool for causal inference' goes beyond what a single configuration can support. I would ask the authors to either soften the conclusion to the specific SCM studied or add a small sensitivity analysis (e.g., multiple random SCM draws) to show that the MAE of about 0.02 is not an artifact of the chosen coefficients.
minor comments (6)
- [Section 1.1] The paper states that 'no prior work has applied machine learning to the problem of predicting probabilities of causation', but the same introduction cites Li et al. [2022a,b] as demonstrating 'the potential of machine learning models to achieve accurate estimations for (sub)populations'. These claims should be reconciled to avoid an overstatement of novelty.
- [Section 5.5] The MLP architecture is described as '15 → 64 → 32 → 16 → 1, utilizing ReLU-like functions and Sigmoid as activation functions', but the experiments compare ReLU, LeakyReLU, and Mish. Please specify which activation is used in each hidden layer and which is used at the output, and clarify how Mish fits the 'ReLU-like' description.
- [Figures 3 and 4] The plots labeled 'confusion matrices' are joint histograms of binned true and predicted values for a regression task, not classification confusion matrices. Consider renaming them to 'prediction-error histograms' or 'agreement matrices' to avoid confusion.
- [Section 4.3] The lower-bound data has 2,054 entries while the upper-bound data has 2,065 entries, even though both bounds are defined for the same subpopulations. Please explain this discrepancy; if it is due to a duplication-removal step or to subpopulations for which the bound is degenerate, state that explicitly.
- [Abstract and Conclusion] The abstract and conclusion say the model predicts 'PNS' with an MAE of about 0.02, but the experiments actually predict the lower and upper bounds of PNS, not PNS itself. Rephrase to 'PNS bounds' for accuracy.
- [Table 1] Table 1 reports point estimates of MSE and MAE without confidence intervals or repeated-run variance. Since the reported differences among models (e.g., MLP-ReLU vs MLP-LeakyReLU) are small, providing standard deviations over multiple seeds would strengthen the comparison.
Circularity Check
No significant circularity: PNS-bound prediction is a genuine supervised generalization task with labels computed from a fully specified SCM, not from the model inputs.
full rationale
The paper's derivation chain is a supervised regression from 15 observable subpopulation features to PNS bounds that are computed from a fully specified SCM with all coefficients and exogenous distributions given in the appendix. Training labels are derived from the SCM equations and are not used as inputs at test time, so the predictions are not fitted values renamed as predictions. The train/test separation is not explicitly described, but the confusion matrices sum to approximately 30,713 entries, matching the complement of the roughly 2,054 training subpopulations (32,768 - 2,054 = 30,714), which indicates that the reported MAE of about 0.02 is evaluated on held-out subpopulations with insufficient data. The mapping from observed features to PNS bounds is a well-defined deterministic function of the SCM and the distributions of the hidden variables, so learning that mapping from a subset of subpopulations is a genuine generalization task rather than a tautology. The paper cites prior work by the same authors for the SCM template and for the 1,300-sample adequacy heuristic, but those citations are not load-bearing for the numerical result because the SCM and all parameter values are fully specified in the paper itself. The absence of an explicit train/test split statement is a reporting ambiguity and a correctness risk, but it is not circular reasoning: no equation reduces to its own input, no fitted parameter is presented as a prediction, and no uniqueness claim is imported from the authors' prior work. The benchmark is self-generated, which limits external generalizability, but that is a limitation of scope, not circularity.
Assumptions & free parameters
free parameters (4)
- SCM coefficient vectors MX, MY and scalar CY =
20 coefficients each; CY = -0.7795 (Appendix A)
- Bernoulli parameters for exogenous variables UZi, UX, UY =
22 probabilities listed in Appendix A
- MLP architecture and training hyperparameters =
15-64-32-16-1, Adam lr=0.01, 1000 epochs
- Minimum sample threshold for training subpopulations =
1,300
assumptions (4)
- domain assumption For each subpopulation, the probabilities of causation are determined by the observed characteristics Z1,...,Z15.
- standard math The bounds for PNS, PS, PN from Tian and Pearl (2000) are correct and tight.
- domain assumption The selected subpopulations with sufficient samples are representative of all subpopulations.
- domain assumption A standard SCM with binary treatment/effect and exogenous Bernoulli noise captures the target setting.
Cite this review
Pith. "Pith review of Estimating Probabilities of Causation with Machine Learning Models." pith.science (2026). https://pith.science/paper/PIEQ2DAS
@misc{pith2026250208858,
author = {Pith},
title = {Pith review of: Estimating Probabilities of Causation with Machine Learning Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/PIEQ2DAS}},
note = {Machine review of arXiv:2502.08858}
}
read the original abstract
Probabilities of causation play a crucial role in modern decision-making. This paper addresses the challenge of predicting probabilities of causation for subpopulations with insufficient data using machine learning models. Tian and Pearl first defined and derived tight bounds for three fundamental probabilities of causation: the probability of necessity and sufficiency (PNS), the probability of sufficiency (PS), and the probability of necessity (PN). However, estimating these probabilities requires both experimental and observational distributions specific to each subpopulation, which are often unavailable or impractical to obtain with limited population-level data. We assume that the probabilities of causation for each subpopulation are determined by its characteristics. To estimate these probabilities for subpopulations with insufficient data, we propose using machine learning models that draw insights from subpopulations with sufficient data. Our evaluation of multiple machine learning models indicates that, given sufficient population-level data and an appropriate choice of machine learning model and activation function, PNS can be effectively predicted. Through simulation studies, we show that our multilayer perceptron (MLP) model with the Mish activation function achieves a mean absolute error (MAE) of approximately 0.02 in predicting PNS for 32,768 subpopulations using data from around 2,000 subpopulations.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Probabilistic counterfactuals: semantics, computation, and applications
Alexander Abraham Balke. Probabilistic counterfactuals: semantics, computation, and applications. University of California, Los Angeles, 1995
work page 1995
-
[2]
Random search for hyper-parameter optimization
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of machine learning research, 13 0 (2), 2012
work page 2012
-
[3]
Support-vector networks
Corinna Cortes. Support-vector networks. Machine Learning, 1995
1995
-
[4]
Philip Dawid, Monica Musio, and Rossella Murtas. The probability of causation. Law, Probability and Risk, 16: 0 163--179, 2017
work page 2017
- [5]
-
[6]
An axiomatic characterization of causal counterfactuals
David Galles and Judea Pearl. An axiomatic characterization of causal counterfactuals. Foundations of Science, 3 0 (1): 0 151--182, 1998
work page 1998
-
[7]
Joseph Y Halpern. Axiomatizing causal reasoning. Journal of Artificial Intelligence Research, 12: 0 317--337, 2000
work page 2000
-
[8]
Causal analysis after haavelmo
James Heckman and Rodrigo Pinto. Causal analysis after haavelmo. Econometric Theory, 31 0 (1): 0 115--151, 2015
work page 2015
Show all 32 references
-
[9]
Random decision forests
Tin Kam Ho. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, volume 1, pages 278--282. IEEE, 1995
1995
-
[10]
Causal inference in statistics, social, and biomedical sciences
Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge university press, 2015
2015
-
[11]
Unit selection based on counterfactual logic
Ang Li and Judea Pearl. Unit selection based on counterfactual logic. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, 2019
2019
-
[12]
Probabilities of causation: Role of observational data
Ang Li and Judea Pearl. Probabilities of causation: Role of observational data. In International Conference on Artificial Intelligence and Statistics, pages 10012--10027. PMLR, 2023
2023
-
[13]
Probabilities of causation with nonbinary treatment and effect
Ang Li and Judea Pearl. Probabilities of causation with nonbinary treatment and effect. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20465--20472, 2024 a
2024
-
[14]
Unit selection with nonbinary treatment and effect
Ang Li and Judea Pearl. Unit selection with nonbinary treatment and effect. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38(18), pages 20473--20480, 2024 b
2024
-
[15]
Chen, Jingzheng Qin, and Zhen Qin
Ang Li, Suming J. Chen, Jingzheng Qin, and Zhen Qin. Training machine learning models with causal logic. In Companion Proceedings of the Web Conference 2020, pages 557--561, 2020
2020
-
[16]
Learning probabilities of causation from finite population data
Ang Li, Song Jiang, Yizhou Sun, and Judea Pearl. Learning probabilities of causation from finite population data. arXiv preprint arXiv:2210.08453, 2022 a
2022 arXiv
-
[17]
Unit selection: Learning benefit function from finite population data
Ang Li, Song Jiang, Yizhou Sun, and Judea Pearl. Unit selection: Learning benefit function from finite population data. In NeurIPS 2022 Workshop on Causality for Real-world Impact, 2022 b
2022
-
[18]
Probabilities of causation: Adequate size of experimental and observational samples
Ang Li, Ruirui Mao, and Judea Pearl. Probabilities of causation: Adequate size of experimental and observational samples. In NeurIPS 2022 Workshop on Neuro Causal and Symbolic AI (nCSI), 2022 c
2022
-
[19]
Rectifier nonlinearities improve neural network acoustic models
Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, page 3. Atlanta, GA, 2013
2013
-
[20]
Mish: A self regularized non-monotonic activation function
Diganta Misra. Mish: A self regularized non-monotonic activation function. arXiv preprint arXiv:1908.08681, 2019
1908 arXiv
-
[21]
Perspective on `harm' in personalized medicine -- an alternative perspective
Scott Mueller and Judea Pearl. Perspective on `harm' in personalized medicine -- an alternative perspective. Technical Report R-530, Department of Computer Science, University of California, Los Angeles, CA, 2023. Forthcoming, American Journal of Epidemiology
2023
-
[22]
Causes of effects: Learning individual responses from population data
Scott Mueller, Ang Li, and Judea Pearl. Causes of effects: Learning individual responses from population data. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI-22), pages 2712--2718, 2022
2022
-
[23]
Rectified linear units improve restricted boltzmann machines
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807--814, 2010
2010
-
[24]
Aspects of graphical models connected with causality
Judea Pearl. Aspects of graphical models connected with causality. Proceedings of the 49th Session of the international Statistical Institute, Italy, pages 399--401, 1993
1993
-
[25]
Causal diagrams for empirical research
Judea Pearl. Causal diagrams for empirical research. Biometrika, 82 0 (4): 0 669--688, 1995
1995
-
[26]
Probabilities of causation: Three counterfactual interpretations and their identification
Judea Pearl. Probabilities of causation: Three counterfactual interpretations and their identification. Synthese, pages 93--149, 1999
1999
-
[27]
Causality
Judea Pearl. Causality. Cambridge university press, 2nd edition, 2009
2009
-
[28]
Causal fairness analysis
Drago Plecko and Elias Bareinboim. Causal fairness analysis. arXiv preprint arXiv:2207.11385, 2022
2022 arXiv
-
[29]
Rumelhart, Geoffrey E
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature, 323 0 (6088): 0 533--536, 1986. doi:10.1038/323533a0
1986 doi
-
[30]
Probabilities of causation: Bounds and identification
Jin Tian and Judea Pearl. Probabilities of causation: Bounds and identification. Annals of Mathematics and Artificial Intelligence, 28 0 (1-4): 0 287--313, 2000
2000
-
[31]
Attention is all you need
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017
2017
-
[32]
Causal ai framework for unit selection in optimizing electric vehicle procurement
Chi Zhang, Ang Li, Scott Mueller, and Rumen Iliev. Causal ai framework for unit selection in optimizing electric vehicle procurement. In 2nd Workshop on Sustainable AI, 2024
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.