REVIEW 4 major objections 5 minor 3 references
Do LLMs Act as Repositories of Causal Knowledge?
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Asking commercial LLMs to label confounders in a well-studied clinical trial produces only slightly better-than-chance discrimination against non-confounders, and answers flip with prompt wording and option order.
desk verdict A useful, reproducible empirical check on LLMs as causal-knowledge repositories; the inconsistency results are solid, but the headline non-confounder comparison needs a cleaner negative control before it is quoted quantitatively. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The test apparatus is a ground-truth benchmark built from the Coronary Drug Project placebo arm, where adherence affects mortality and confounding is well documented. Three expert sources supply confounder lists — the original 1980 trial report, a 2016 reanalysis, and a 2024 expert-curated causal diagram — and the authors add a list of 60 variables considered non-confounders. The models are queried directly ('is this variable a confounder?') and indirectly (first asking whether the variable affects adherence, then whether it affects mortality), with variants that add or remove step-by-step reasoning and that shuffle the answer options. A confounder is a pre-treatment variable that, if left unadjusted, distorts the estimated effect of treatment on outcome; the benchmark measures whether LLMs can reproduce expert judgments about which variables play that role.
What would settle it
Run the same confounder-labeling prompts with the full text of CDPRG (1980), Murray and Hernán (2016), and Debertin et al. (2024) inserted into the context window, then measure discrimination between expert confounders and the non-confounder list. If the models are repositories of causal knowledge, giving them the answer key should produce near-perfect and stable labels; if discrimination stays weak and answers still flip when the multiple-choice order is changed, the repository theory fails.
Extended reading notes
Core claim
The central claim is that current mass-market LLMs, when used as recall devices rather than reasoners, cannot reliably retrieve expert causal judgments. Even though the models can produce text showing that the Coronary Drug Project and its confounder discussions are in their training data, their confounder labels discriminate only weakly between expert-chosen confounders and expert-designated non-confounders. The paper finds that the apparent success of some configurations is driven by a tendency to label almost everything a confounder rather than by genuine agreement with experts, and that designations change substantially when the prompt is direct versus indirect, when reasoning is encouraged versus suppressed, and when the multiple-choice options are reordered. In the authors' words, LLMs do not yet have the ability to automate the reporting of causal links.
Load-bearing premise
The expert-selected confounder lists from the three studies, and the authors' list of non-confounders, are treated as the correct ground truth for the adherence–mortality relationship; if those lists are wrong or context-dependent, the measured performance is disagreement with particular experts rather than absence of causal knowledge.
Editorial extensions
If this is right
- A researcher cannot yet delegate confounder selection to an off-the-shelf LLM: in the best-performing configuration, 65–74% of expert-rejected variables were still labeled confounders.
- A prompt that looks good on one model or dataset is not portable, because answers shift with option order and with direct versus indirect questioning.
- Reported success rates overstate true causal knowledge, since much of the agreement comes from a general tendency to say 'confounder' rather than from discriminating between confounders and non-confounders.
- The CDP setting gives future models a ready-made benchmark: the same code and ground-truth lists can be rerun as LLMs improve to test whether the gap closes.
- If the inconsistency persists after the models have clearly absorbed more causal literature, that is evidence the failure is structural, not just a matter of training data coverage.
Reading between the lines
- If this pattern extends beyond the CDP, then any observational study that auto-generates adjustment sets from current LLMs risks systematically biased effect estimates, because false-positive confounders and false negatives enter the model in ways that are hard to detect without expert review.
- The option-order flips suggest the models are choosing plausible continuations of the prompt rather than consulting a stable internal representation of causal structure; a testable extension would be to see whether supplying the relevant study text in the prompt removes the instability.
- The authors note that experts might not be more correct than the LLMs in cases of disagreement; an editorial inference is that a multi-panel expert elicitation on the same variable set would separate 'LLMs lack causal knowledge' from 'experts disagree among themselves.'
- A natural next experiment is to apply the same benchmark to other well-studied questions with published confounder sets, such as postmenopausal hormone therapy and cardiovascular disease, to see whether the mediocre discrimination is specific to the CDP or general.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper tests whether three commercial LLMs (GPT-4o, GPT-o1-preview, Claude 3.5 Sonnet) can identify confounders for the adherence-mortality relationship in the Coronary Drug Project, using expert-selected confounder lists from CDPRG (1980), Murray and Hernán (2016), and Debertin et al. (2024), plus an author-constructed list of 60 supposed non-confounders. Across direct and indirect prompting, with and without encouraged reasoning, and with shuffled multiple-choice options, the authors report moderate true-positive rates, high false-positive rates on the non-confounder set, and substantial sensitivity to iteration, model, prompt, and option order. They conclude that LLMs cannot yet be used to automate the reporting of causal links for this kind of task.
Significance. If the empirical claims hold, the paper offers a useful, reproducible benchmark for evaluating whether LLMs can recall rather than reason about causal structure. The strengths are considerable: the positive ground truth comes from published expert lists rather than from the LLMs themselves; the study uses multiple models and prompt variants; repeated sampling reveals response distributions rather than single outputs; and the code is promised on OSF, providing a concrete falsifiable test for future models. The inconsistency and option-order results are robust and practically important regardless of any disagreement with the expert lists. However, the headline quantitative claim about the small separation between expert confounders and non-confounders depends on a negative control whose validity is questionable, and the effect sizes are presented without uncertainty quantification.
major comments (4)
- [Methods, Appendix C] The 'Non-confounder' negative control is not expert-derived and appears contaminated with plausible confounders. In a post-myocardial-infarction cohort, variables such as 'Shortness of breath at night', 'Swelling ankles', 'Rales', 'Palpable liver', 'Gouty arthritis', and 'Nervous system abnormalities' are prognostic markers that can plausibly influence both adherence and mortality; several items in the 'anticipated side-effect' category can likewise affect continued adherence and downstream mortality. If a substantial share of these 60 variables are true confounders, the reported LLM false-positive rates are overstated and the abstract's 'only slightly more likely' separation is artificially compressed. Because this list was constructed by the authors rather than elicited from experts, it cannot serve as an unvalidated benchmark for the headline claim. Please recompute the separation using only administrative and sub-study variables that are unlikely to be confounders, or obtain independent expert validation of the negative set.
- [Abstract, Results, Table 1] The central claim that expert confounders are 'only slightly more likely' to be labeled as confounders than non-confounders is presented without any inferential quantification. No confidence intervals, standard errors, or tests are given for the mean designation rates or their differences, even though the repeated prompting design provides the raw material for such estimates. As written, the reader cannot assess whether the observed separation is statistically distinguishable from zero for any model-prompt combination. Please add variable-level means with confidence intervals, or a compact table of differences with uncertainty, for at least the direct and indirect with-reasoning conditions.
- [Data, Table 1, Figures 1-2] The GPT-o1-preview non-confounder data are incomplete and were collected under different conditions: a content-flagging workaround was added to direct prompts, follow-up variables were re-prompted as baseline measurements, and indirect-method prompts for non-confounders were apparently unavailable. Nevertheless, Table 1 reports a single row for GPT-o1 across all variable sets, and the figures plot non-confounder rates without indicating which cells are missing or based on modified prompts. This can distort cross-model comparisons and should be explicitly flagged, with affected cells either excluded from aggregate tables or analyzed separately.
- [Introduction, Coronary Drug Project] The claim that 'text about the ground truth is in their training data' is supported only by asking the LLMs themselves whether they know about the Coronary Drug Project. Such self-reports are weak evidence of training-data inclusion, since models can produce plausible text without having seen a specific document. Please either verify inclusion by external means (for example, retrieval of distinctive phrases or documentation of publication dates in known corpora) or soften the claim to 'likely in the training data'.
minor comments (5)
- [Appendix A] There are several typos in the variable-set headings: 'Counfounders' should be 'Confounders', and 'abnormalitites' should be 'abnormalities'.
- [Results, Consistency of Confounder Designations] In the paragraph on option-order sensitivity, 'GPA-4o' should be 'GPT-4o'.
- [Appendix C] In the sub-study category, 'Alpha-lipoprotein cholecystitis' appears to be a copy-paste error and should read 'Alpha-lipoprotein cholesterol'.
- [Results, Figure 7] The footnote explaining that 'Mixed' pools all intermediate percentages is important and should appear in the main text, because the claim that reasoning made 'no difference at all' for indirect prompting is an artifact of this coarse binning.
- [Abstract] The phrase 'only slightly more likely' is informal and underspecified; please replace it with a numeric difference or range once the uncertainty analysis is added.
Circularity Check
No circularity: the LLM benchmark is anchored to external published expert lists; the co-authored ground-truth papers and the author-built non-confounder set are validity caveats, not constructional equivalences.
full rationale
This paper is an empirical benchmark rather than a derivation chain with predictions or first-principles results. The ground-truth confounder labels come from three published expert sources (CDPRG 1980, Murray and Hernán 2016, and Debertin et al. 2024), not from the LLMs being tested, so the central comparison is externally anchored. Two of those sources are co-authored by the current second author, which is a self-citation worth noting, but it is not load-bearing in a circular sense: the paper's conclusion does not reduce to the truth of those papers, and the CDPRG (1980) list plus the response-inconsistency results provide independent content even if the co-authored lists were set aside. The 'Non-confounder' group was constructed by the authors rather than elicited from experts, which raises a genuine construct-validity concern about the negative control, but that is a benchmark-quality problem, not circularity: the LLM ratings are not defined in terms of that list, and a contaminated negative set would bias the separation estimate rather than make it true by construction. Similarly, the check that CDP discussion is in LLM training data uses the LLMs themselves as probes, but this is not a fitted parameter or a predicted quantity; the conclusion does not equate to the probe's output. No equation reduces to an input, no fitted value is relabeled as a prediction, and no uniqueness theorem is imported from the authors' prior work. The paper even acknowledges the limits of the expert ground truth ('we cannot prove that in the case of a deviation the experts are more correct'), which further shows the claim is an empirical comparison rather than a self-fulfilling derivation.
Assumptions & free parameters
free parameters (3)
- temperature =
0.7
- number of repeated prompts =
10
- indirect confounder coding rule =
A or B on both adherence and mortality questions
assumptions (4)
- domain assumption Expert confounder lists from CDPRG (1980), Murray and Hernán (2016), and Debertin et al. (2024) are a correct ground truth for true confounders of adherence and mortality.
- domain assumption The LLM training data contain the ground-truth text about confounding in the Coronary Drug Project.
- domain assumption Variables in the Non-confounder group are genuinely non-confounders.
- domain assumption The final multiple-choice letter in each LLM response reflects the model's causal judgment.
Cite this review
Pith. "Pith review of Do LLMs Act as Repositories of Causal Knowledge?." pith.science (2026). https://pith.science/paper/EKRQAF6D
@misc{pith2026241210635,
author = {Pith},
title = {Pith review of: Do LLMs Act as Repositories of Causal Knowledge?},
year = {2026},
howpublished = {\url{https://pith.science/paper/EKRQAF6D}},
note = {Machine review of arXiv:2412.10635}
}
read the original abstract
Large language models (LLMs) offer the potential to automate a large number of tasks that previously have not been possible to automate, including some in science. There is considerable interest in whether LLMs can automate the process of causal inference by providing the information about causal links necessary to build a structural model. We use the case of confounding in the Coronary Drug Project (CDP), for which there are several studies listing expert-selected confounders that can serve as a ground truth. LLMs exhibit mediocre performance in identifying confounders in this setting, even though text about the ground truth is in their training data. Variables that experts identify as confounders are only slightly more likely to be labeled as confounders by LLMs compared to variables that experts consider non-confounders. Further, LLM judgment on confounder status is highly inconsistent across models, prompts, and irrelevant concerns like multiple-choice option ordering. LLMs do not yet have the ability to automate the reporting of causal links.
Reference graph
Works this paper leans on
-
[2]
Mining Causality: AI-Assisted Search for Instrumental Variables
“Synthesizing Subject-Matter Expertise for Variable Selection in Causal Effect Estimation: A Case Study.”Epidemiology 35 (5): 642–53. https://doi.org/10.1097/ede.00 00000000001758. Gururaghavendran, Rajesh, and Eleanor J. Murray. 2024. “Can Algorithms Replace Expert Knowledge for Causal Inference? A Case Study on Novice Use of Causal Discovery.” American ...
-
[3]
Large Language Models and Causal Inference in Collaboration: A Comprehensive Survey
“Large Language Models and Causal Inference in Collaboration: A Comprehensive Survey.” https://doi.org/10.48550/ARXIV.2403.09606. Long, Stephanie, Tibor Schuster, and Alexandre Piché. 2023. “Can Large Language Models Build Causal Graphs?” https://doi.org/10.48550/ARXIV.2303.05279. Murray, Eleanor J., and Miguel A. Hernán. 2016. “Adherence Adjustment in th...
-
[2024]
Cause and Effect: Can Large Language Models Truly Understand Causality?
“Cause and Effect: Can Large Language Models Truly Understand Causality?” https://doi.org/10.48550/ARXIV.2402.18139. Cai, Hengrui, Shengjie Liu, and Rui Song. 2024. “Is Knowledge All Large Language Models Needed for Causal Reasoning?” https://doi.org/10.48550/ARXIV.2401.00139. Catalogue of Bias Collaboration, David Nunan, Jeffrey Aronson, and Clare Bankhe...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.