Pith. sign in

REVIEW 2 major objections 3 minor 1 cited by

Are you doing better than random guessing? A call for using negative controls when evaluating causal discovery algorithms

T0 review · 2 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that standard causal discovery metrics like precision and recall can be matched by random guessing, and it proposes negative controls as an evaluation baseline.

desk verdict A clean, correct, and practically useful argument that causal discovery evaluations should report negative control baselines; the hypergeometric derivations are right and the warning about interpreting precision and recall is well supported. read the letter →

arxiv 2412.10039 v2 pith:ERKT3QS6 submitted 2024-12-13 stat.ME stat.ML

Anne Helby Petersen This is my paper · ORCID
classification stat.MEstat.ML
keywords causaldiscoverynegativecontrolsrandomguessingbaselinehypergeometricdistributionskeletonestimationevaluationmetricsprecisionandrecallexacttest
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard causal discovery evaluation metrics—precision, recall, and F1—can be deceptively high under random guessing, so reported numbers mean little without a random baseline. It proves that if an estimated skeleton places its edges uniformly at random, the true-positive count follows an exact hypergeometric distribution, giving expected precision equal to the true graph's density and expected recall equal to the estimate's density. As a result, dense true graphs let a completely uninformative estimate earn high precision, and adding many edges inflates recall and F1. The paper proposes negative controls as a required benchmark, constructs an exact test of skeleton fit, and provides a simulation-based pipeline for metrics without exact null distributions.

What carries the argument

The load-bearing object is the hypergeometric null for true positives, $\mathrm{HyperGeom}(m_{\max}, m_{\mathrm{true}}, m_{\mathrm{est}})$, obtained by conditioning on the fixed margins of the adjacency confusion matrix when edges are placed uniformly at random. This single distribution carries the argument because every adjacency metric considered is a linear function of the true-positive count, so expectations, quantiles, confidence intervals, and the one-sided skeleton-fit test all follow from it. For metrics outside this class—orientation precision and recall, structural Hamming distance, structural intervention distance—the paper substitutes a simulation-based negative-control pipeline: draw random graphs matched to the algorithm's edge-count distribution, evaluate the metric, and compare pairwise with a one-sided test.

What would settle it

Simulate many random true DAGs with a fixed number of edges and an estimator that always outputs the same fixed skeleton, such as a star graph, with the same edge count; the observed true-positive counts will not follow $\mathrm{HyperGeom}(m_{\max}, m_{\mathrm{true}}, m_{\mathrm{est}})$, and the proposed exact test will not hold its nominal level, showing the null applies only to genuinely uniform random placement.

Watch

Extended reading notes

Core claim

Conditional on the maximal number of possible edges $m_{\max}$, the true edge count $m_{\mathrm{true}}$, and the estimated edge count $m_{\mathrm{est}}$, the paper shows that the number of true positives in an estimated skeleton placed uniformly at random is exactly distributed as $\mathrm{HyperGeom}(m_{\max}, m_{\mathrm{true}}, m_{\mathrm{est}})$. Because precision, recall, F1, negative predictive value, and specificity are linear functions of the true-positive count given these margins, the paper derives their exact expectations and quantiles under this null: expected precision is $m_{\mathrm{true}}/m_{\max}$, expected recall is $m_{\mathrm{est}}/m_{\max}$, and expected F1 is $2 m_{\mathrm{est}} m_{\mathrm{true}} / (m_{\max}(m_{\mathrm{est}}+m_{\mathrm{true}}))$. This yields a warning that apparently strong metric values can arise from chance, an exact one-sided test for whether a skeleton is better than random, and, for metrics that cannot be expressed through the adjacency confusion matrix, a simulation-based negative-control pipeline matching the algorithm's edge-count distribution.

Load-bearing premise

The load-bearing premise is that a random guess places its $m_{\mathrm{est}}$ edges uniformly among all $m_{\max}$ possible edges, so conditioning on the edge counts fully describes guessing behavior; algorithms whose outputs have extra structure, such as preferred degrees, clustering, or acyclicity constraints, may not be faithfully benchmarked by that null.

Editorial extensions

If this is right

  • Report a negative-control value whenever precision, recall, or F1 is used to evaluate a causal discovery algorithm; without it, high values may reflect graph density rather than skill.
  • When comparing two algorithms, the comparison is interpretable only if the outputs have similar edge counts or if each is judged against its own negative control, since dense outputs can look better without being more informative.
  • The exact hypergeometric test gives a calibrated p-value for 'better than random' skeleton recovery, and the paper demonstrates both rejection (TPC on the Metropolit cohort, p = 0.002) and failure to reject for several algorithms on the Sachs data.
  • For metrics beyond adjacency, the simulation-based pipeline provides an empirical chance distribution, and the paper's PC simulation shows that in dense settings even a sound algorithm may not separate from random guessing.
  • Ranking algorithms by negative-control p-values can differ from ranking by raw SHD, because the latter rewards sparsity regardless of correctness, as the Sachs example shows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The negative-control principle transfers to any structured-output prediction task—such as community detection, network reconstruction, or feature-set selection—where precision and recall are reported; a random baseline should become part of the evaluation protocol.
  • For small graphs with small $m_{\max}$, the exact test is conservative and borderline p-values should not be over-read; this follows from the discreteness the paper notes, but the practical warning is our emphasis.
  • A reader can use the paper's formulas to compute a chance ceiling for any previously reported precision; some existing benchmark claims may not outperform random guessing once graph density is accounted for.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper argues that evaluations of causal discovery algorithms should include negative controls (random guessing) as a common baseline, and it provides the mathematical and methodological tools to do so. For the task of graph skeleton estimation, it derives exact distributional results: conditional on the maximal possible number of edges mmax, the true number of edges mtrue, and the estimated number of edges mest, the number of true positives under random edge placement follows a hypergeometric distribution. From this it obtains expected values and quantiles for precision, recall, F1, negative predictive value, and specificity, showing that these metrics can take surprisingly favorable values under random guessing when the true and/or estimated graphs are dense. The paper proposes an exact test of overall skeleton fit and applies it to a temporal PC analysis of a real cohort dataset. It further proposes a simulation-based negative control pipeline for more general metrics (e.g., structural Hamming distance, orientation measures, SID), demonstrated on a PC simulation study and on the Sachs protein-signaling data.

Significance. If the claims are correct, this is a valuable and practically relevant contribution to the causal discovery evaluation literature. It gives a simple, principled baseline and quantifies the often unappreciated fact that raw precision/recall values are uninterpretable without a random-guessing reference. The hypergeometric derivation is correct, clearly presented, and exact under the stated model; the paper is transparent about the key assumptions (fixed mmax, mtrue, mest and uniform edge placement) and about the limitations of the Erdős–Rényi-type null. The emphasis on reporting distributional summaries and on comparing algorithms only after matching edge count is sound advice that is likely to influence evaluation practice. The main weakness is in the presentation of the simulation-based p-value in Section 6, which is internally inconsistent and needs to be fixed before the pipeline can be reproduced or interpreted reliably.

major comments (2)
  1. [Section 6 and Supplementary C] The definition of the p-value for the simulation-based negative control is internally inconsistent and cannot reproduce the values reported in Tables 5 and 6. In Section 6, the p-value is defined as the proportion p = (1/b) * sum_i 1{f(G_i_true, Ghat_i_algo) <= f(G_i_true, Ghat_i_NC)} for metrics where small values are favorable, while Supplementary C uses the opposite inequality for the same metric (SHD) and reports p = 0.202. The reported values such as p = 0.002 in the sparse case and p = 0.001 for PC in Table 6 are not proportions of pairs where the algorithm performs at least as well as the negative control; if the algorithm is better, that proportion would be near 1, not 0.001. These numbers are instead consistent with a binomial tail probability P(T >= t_obs) under a null of exchangeability, but that is not what the displayed formula gives. Moreover, the conclusion in Section 6.1 that PC is 'not significantly different' in the dense case does not follow from the formula as written, since a proportion of 0.202 would be far below 0.5 and would indicate the opposite direction. The authors should state explicitly whether the reported p is a test statistic or a p-value, give the correct formula for the p-value (e.g., the binomial tail probability under the null), and verify that the code and the Supplementary Materials use the same definition.
  2. [Section 6, Step 2] The procedure says to draw 'a negative control random DAG' but does not specify the distribution over DAGs with a given edge count. To be consistent with the exact results of Sections 3–5, the negative control graph should place its mest edges uniformly at random among all possible edges (that is, a uniform random skeleton conditional on the edge count, with an acyclic orientation). If instead one draws uniformly from the set of all DAGs with mest edges, the skeleton is not uniformly distributed over all m-edge subsets, and the negative control distribution will differ from the hypergeometric null even when the algorithm has no information. The paper should state explicitly how the negative control graphs are generated and, if a different graph-generating process is intended, clarify that the exact distributional results of Sections 3–5 apply only to the uniform-edge-placement null.
minor comments (3)
  1. [Table 5, notes] The intervals labelled '95% CI' are empirical quantile intervals for the distribution of each metric, not confidence intervals for a parameter; consider relabelling them as 'empirical 95% intervals' or similar to avoid ambiguity.
  2. [Section 6.2] There is a typographical error in the phrase 'how often the discovery algortihm' — 'algortihm' should be 'algorithm'.
  3. [Section 4] The sentence 'the expected precision does not depend on the number of edges in the estimated graph' is correct under the conditional hypergeometric model but could be misread as an unconditional statement; a short reminder that this holds conditional on (mmax, mtrue, mest) would improve clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the negative-control distribution is derived from a stated random-edge null model, and the metric expectations follow by direct linearity calculations.

full rationale

The paper's central derivation is self-contained. Section 3 defines random guessing as placing mest edges uniformly among the mmax possible edges and conditions on (mmax, mtrue, mest); from that definition TP follows a hypergeometric distribution, which is a standard combinatorial fact rather than a fitted input. Table 3 expectations for precision, recall, F1, NPV, and specificity are then obtained by writing each metric as a linear function of TP and taking expectations; no parameter is fitted to data and no target result is assumed. The Section 5 skeleton-fit test uses the same stated null hypothesis and is conservative by construction, not circularly validated. Self-citations (Petersen et al. 2021; 2023a; 2023b) appear only as application examples, algorithm implementations, or background about PC's sparsity bias, and none of them supplies the load-bearing distributional argument. The uniform-edge null is an explicit assumption and a limitation for structured graphs, but the paper also offers a simulation-based negative-control pipeline for such cases; this does not make the derivation circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities and uses no fitted parameters. The central claims rest on a small set of explicitly stated modeling assumptions: uniform random edge placement under the null, a priori fixing of the estimated edge count, and the correctness of the expert- or literature-derived ground truth in the real data applications.

assumptions (4)
  • domain assumption The estimated graph under random guessing places mest edges uniformly at random among the mmax possible edges, independently of the true graph.
    Invoked in Section 3 to derive the hypergeometric distribution for TP. The Discussion acknowledges this as the Erdős-Rényi assumption.
  • domain assumption The number of edges in the estimated graph, mest, is fixed a priori (e.g., by a pre-specified tuning parameter).
    Stated in Section 3 and discussed in Section 7. If mest is data-dependent, the hypergeometric result does not hold, and the simulation pipeline is recommended instead.
  • domain assumption In the Metropolit cohort application, the expert model is treated as the true DAG.
    Section 5.1: 'we assume that the expert model is correct'. This is needed to define mtrue and the observed true positive count.
  • domain assumption In the Sachs data application, the 20-edge ground truth graph is correct.
    Supplementary D: the 20-edge truth version is used because it was the one used in the NOTEARS evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are you doing better than random guessing? A call for using negative controls when evaluating causal discovery algorithms." pith.science (2026). https://pith.science/paper/ERKT3QS6

@misc{pith2026241210039,
  author       = {Pith},
  title        = {Pith review of: Are you doing better than random guessing? A call for using negative controls when evaluating causal discovery algorithms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ERKT3QS6}},
  note         = {Machine review of arXiv:2412.10039}
}
read the original abstract

New proposals for causal discovery algorithms are typically evaluated using simulations and a few selected real data examples with known data generating mechanisms. However, there does not exist a general guideline for how such evaluation studies should be designed, and therefore, comparing results across different studies can be difficult. In this article, we propose to use negative controls as a common evaluation baseline by posing the question: Are we doing better than random guessing? For the task of graph skeleton estimation, we derive exact distributional results under random guessing for the expected behavior of a range of typical causal discovery evaluation metrics, including precision and recall. We show that these metrics can achieve very favorable values under random guessing in certain scenarios, and hence warn against using them without also reporting negative control results, i.e., performance under random guessing. We also propose an exact test of overall skeleton fit, and showcase its use on a real data application. Finally, we propose a general pipeline for using negative controls beyond the skeleton estimation task, and apply it both in a simulated example and a real data application.

Figures

Figures reproduced from arXiv: 2412.10039 by the authors.

Figure 1
Figure 1. The True Underlying DAG (a) and an Estimated [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Expected Adjacency F1 Scores Under Random [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Score-Based Causal Discovery with Temporal Background Information

    stat.ME 2025-02 conditional novelty 6.0 of 10

    TGES extends greedy equivalence search with tiered background knowledge and is proven sound and complete in the large sample limit, while improving finite-sample recall over temporal PC.

Reference graph

Works this paper leans on

30 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    Comparison of statistical methods for finding network motifs

    Vanna Albieri and Vanessa Didelez. Comparison of statistical methods for finding network motifs. Statistical Applications in Genetics and Molecular Biology, 13 0 (4): 0 403--422, 2014. doi:doi:10.1515/sagmb-2013-0017. URL https://doi.org/10.1515/sagmb-2013-0017

    work page doi:10.1515/sagmb-2013-0017 2014

  2. [2]

    Learning high-dimensional directed acyclic graphs with mixed data-types

    Bryan Andrews, Joseph Ramsey, and Gregory F Cooper. Learning high-dimensional directed acyclic graphs with mixed data-types. In The 2019 ACM SIGKDD Workshop on Causal Discovery, pages 4--21. PMLR, 2019

    work page 2019

  3. [3]

    Fast scalable and accurate discovery of dags using the best order score search and grow shrink trees

    Bryan Andrews, Joseph Ramsey, Ruben Sanchez Romero, Jazmin Camchong, and Erich Kummerfeld. Fast scalable and accurate discovery of dags using the best order score search and grow shrink trees. Advances in neural information processing systems, 36: 0 63945--63956, 2023

    work page 2023

  4. [4]

    Improving finite sample performance of causal discovery by exploiting temporal structure

    Christine W Bang, Janine Witte, Ronja Foraita, and Vanessa Didelez. Improving finite sample performance of causal discovery by exploiting temporal structure. arXiv preprint arXiv:2406.19503, 2024

    arXiv 2024

  5. [5]

    Tuning causal discovery algorithms

    Konstantina Biza, Ioannis Tsamardinos, and Sofia Triantafillou. Tuning causal discovery algorithms. In International Conference on Probabilistic Graphical Models, pages 17--28. PMLR, 2020

    work page 2020

  6. [6]

    Optimal structure identification with greedy search

    David Maxwell Chickering. Optimal structure identification with greedy search. Journal of machine learning research, 3 0 (Nov): 0 507--554, 2002

    2002

  7. [7]

    Toward falsifying causal graphs using a permutation-based test

    Elias Eulig, Atalanti A Mastakouri, Patrick Bl \"o baum, Michaela Hardt, and Dominik Janzing. Toward falsifying causal graphs using a permutation-based test. arXiv preprint arXiv:2305.09565v2, 2024

    arXiv 2024

  8. [8]

    The case for evaluating causal models using interventional measures and empirical data

    Amanda Gentzel, Dan Garant, and David Jensen. The case for evaluating causal models using interventional measures and empirical data. Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

Show all 30 references
  1. [9]

    Can algorithms replace expert knowledge for causal inference? a case study on novice use of causal discovery

    Rajesh Gururaghavendran and Eleanor J Murray. Can algorithms replace expert knowledge for causal inference? a case study on novice use of causal discovery. American Journal of Epidemiology, page kwae338, 2024

    2024

  2. [10]

    Adjustment identification distance: A gadjid for causal structure learning

    Leonard Henckel, Theo W \"u rtzen, and Sebastian Weichwald. Adjustment identification distance: A gadjid for causal structure learning. In Uncertainty in Artificial Intelligence, pages 1569--1598. PMLR, 2024

    2024

  3. [11]

    The (mis) use of overlap of confidence intervals to assess effect modification

    Mirjam J Knol, Wiebe R Pestman, and Diederick E Grobbee. The (mis) use of overlap of confidence intervals to assess effect modification. European journal of epidemiology, 26: 0 253--254, 2011

    2011

  4. [12]

    Gradient-based neural dag learning

    S \'e bastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient-based neural dag learning. In International Conference on Learning Representations, 2020

    2020

  5. [13]

    Greedy relaxations of the sparsest permutation algorithm

    Wai-Yin Lam, Bryan Andrews, and Joseph Ramsey. Greedy relaxations of the sparsest permutation algorithm. In Uncertainty in Artificial Intelligence, pages 1052--1062. PMLR, 2022

    2022

  6. [14]

    Supervised whole dag causal discovery

    Hebi Li, Qi Xiao, and Jin Tian. Supervised whole dag causal discovery. arXiv preprint arXiv:2006.04697, 2020

    2006 arXiv

  7. [15]

    SID: Structural Intervention Distance, 2023

    Jonas Peters. SID: Structural Intervention Distance, 2023. URL https://CRAN.R-project.org/package=SID. R package version 1.1

    2023

  8. [16]

    Structural intervention distance for evaluating causal graphs

    Jonas Peters and Peter B \"u hlmann. Structural intervention distance for evaluating causal graphs. Neural computation, 27 0 (3): 0 771--799, 2015

    2015

  9. [17]

    Data-driven model building for life-course epidemiology

    Anne H Petersen, Merete Osler, and Claus T Ekstr m. Data-driven model building for life-course epidemiology. American Journal of Epidemiology, 190 0 (9): 0 1898--1907, 2021

    1907

  10. [18]

    causalDisco: Tools for Causal Discovery on Observational Data, 2022

    Anne Helby Petersen. causalDisco: Tools for Causal Discovery on Observational Data, 2022. URL https://cran.r-project.org/web/packages/causalDisco/index.html. R package version 0.9.1

    2022

  11. [19]

    Constructing causal life-course models: Comparative study of data-driven and theory-driven approaches

    Anne Helby Petersen, Claus Thorn Ekstr m, Peter Spirtes, and Merete Osler. Constructing causal life-course models: Comparative study of data-driven and theory-driven approaches. American Journal of Epidemiology, 192 0 (11): 0 1917--1927, 2023 a

    1917

  12. [20]

    Causal discovery for observational sciences using supervised machine learning

    Anne Helby Petersen, Joseph Ramsey, Claus Thorn Ekstrøm, and Peter Spirtes. Causal discovery for observational sciences using supervised machine learning. Journal of Data Science, 21 0 (2), 2023 b

    2023

  13. [21]

    Tetrad—a toolbox for causal discovery

    Joseph D Ramsey, Kun Zhang, Madelyn Glymour, Ruben Sanchez Romero, Biwei Huang, Imme Ebert-Uphoff, Savini Samarasinghe, Elizabeth A Barnes, and Clark Glymour. Tetrad—a toolbox for causal discovery. In 8th international workshop on climate informatics, pages 1--4, 2018

    2018

  14. [22]

    Causal protein-signaling networks derived from multiparameter single-cell data

    Karen Sachs, Omar Perez, Dana Pe'er, Douglas A Lauffenburger, and Garry P Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science, 308 0 (5721): 0 523--529, 2005

    2005

  15. [23]

    A linear non-gaussian acyclic model for causal discovery

    Shohei Shimizu, Patrik O Hoyer, Aapo Hyv \"a rinen, Antti Kerminen, and Michael Jordan. A linear non-gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 7 0 (10), 2006

    2006

  16. [24]

    An algorithm for fast recovery of sparse causal graphs

    Peter Spirtes and Clark Glymour. An algorithm for fast recovery of sparse causal graphs. Social science computer review, 9 0 (1): 0 62--72, 1991

    1991

  17. [25]

    Causation, prediction, and search

    Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search. MIT press, 2000

    2000

  18. [26]

    The max-min hill-climbing bayesian network structure learning algorithm

    Ioannis Tsamardinos, Laura E Brown, and Constantin F Aliferis. The max-min hill-climbing bayesian network structure learning algorithm. Machine learning, 65: 0 31--78, 2006

    2006

  19. [27]

    Separation-based distance measures for causal graphs

    Jonas Wahl and Jakob Runge. Separation-based distance measures for causal graphs. In The 28th International Conference on Artificial Intelligence and Statistics, 2025

    2025

  20. [28]

    Causal structure learning with one-dimensional convolutional neural networks

    Chuanyu Xu and Wei Xu. Causal structure learning with one-dimensional convolutional neural networks. IEEE Access, 9: 0 162147--162155, 2021

    2021

  21. [29]

    Dag-gnn: Dag structure learning with graph neural networks

    Yue Yu, Jie Chen, Tian Gao, and Mo Yu. Dag-gnn: Dag structure learning with graph neural networks. In International conference on machine learning, pages 7154--7163. PMLR, 2019

    2019

  22. [30]

    Dags with no tears: Continuous optimization for structure learning

    Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. Advances in neural information processing systems, 31, 2018

    2018

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.