LLM Explainability with Counterfactual Chains and Causal Graphs

Liat Ein-Dor; Nirit Nussbaum-Hoffer; Nitay Calderon; Roi Reichart

arxiv: 2606.05972 · v1 · pith:4TTCKMTXnew · submitted 2026-06-04 · 💻 cs.LG

LLM Explainability with Counterfactual Chains and Causal Graphs

Nirit Nussbaum-Hoffer , Nitay Calderon , Liat Ein-Dor , Roi Reichart This is my paper

Pith reviewed 2026-06-28 03:07 UTC · model grok-4.3

classification 💻 cs.LG

keywords causal graphsLLM explainabilitycounterfactual augmentationconcept discoverycausal discoverymachine learning interpretability

0 comments

The pith

Causal graphs can represent how LLMs organize high-level concepts to reach predictions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a four-phase procedure that turns an LLM's responses on textual examples into a causal graph over human-interpretable concepts. First it extracts class-discriminative concepts, then assigns each input a perceived concept state according to the target model. An MCMC-style procedure then generates chains of counterfactual examples to enlarge the data set, after which σ-CG recovers the causal structure among those concepts. The resulting graphs are evaluated on predictive fidelity and structural stability across disease diagnosis, sentiment analysis, and LLM-as-a-judge tasks, showing dependencies that align with the models' observed behavior.

Core claim

Causal graphs recovered from LLM-perceived concept states, after MCMC-inspired counterfactual augmentation, capture meaningful dependencies that are consistent with the LLMs' reasoning on the studied tasks.

What carries the argument

Four-phase pipeline that maps inputs to LLM-perceived concept states, augments them via MCMC-inspired counterfactual chains, and applies σ-CG to obtain class-discriminative causal graphs.

If this is right

The graphs achieve both predictive fidelity to the original LLM and structural stability across random seeds.
The augmentation procedure converges and improves downstream utility of the discovered graphs.
The same pipeline produces informative graphs for disease diagnosis, sentiment analysis, and LLM-as-a-judge classification.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the graphs prove stable, they could serve as an editable interface for intervening on an LLM's concept-level reasoning without retraining.
The method may extend to settings where only black-box access to the LLM is available, provided the concept extractor remains reliable.
Comparing graphs across models on the same task could quantify how different LLMs organize the same concepts.

Load-bearing premise

The MCMC-inspired augmentation enlarges the data without changing the LLM's underlying concept perceptions.

What would settle it

If the recovered graphs assign zero probability to a direct causal link that is required to reproduce the LLM's accuracy on held-out examples, or if the graphs change substantially when the augmentation step is removed, the claim would be falsified.

Figures

Figures reproduced from arXiv: 2606.05972 by Liat Ein-Dor, Nirit Nussbaum-Hoffer, Nitay Calderon, Roi Reichart.

**Figure 2.** Figure 2: Overview of our four-phase pipeline for constructing causal graphs. The “papaya” running example is [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Illustrations of four causal graphs. All causal graphs are provided in Appendix [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** Figure 4: KL-divergence convergence during MCMCinspired data expansion on Sentiment Analysis for GPTOSS-20B. The empirical curve is compared against two bounds: perfect overlap (samples fall within the covered regions, lower bound) and orthogonal expansion (samples occupy unseen regions, upper bound). 5.2 Analysis of the Causal Graphs We next discuss the learned concept sets and causal graphs. On the synthetic Di… view at source ↗

**Figure 5.** Figure 5: Impact of MCMC: Mean accuracy across DD and SA datasets and all concept combinations, under three training regimes: All Original (full original dataset), Original Seed (subset used as a seed for the MCMC), and Seed + Counterfactuals. For each target v ∈ C ∪ {yˆ}, a logistic regression is trained over all possible concept combinations. The blue bars show the accuracy when the input consists of the parents b… view at source ↗

**Figure 6.** Figure 6: Box plots detailing the classification accuracy across 10 rounds of cross-validation for each model and [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗

**Figure 7.** Figure 7: Concept predictability accuracy comparison across training set variations (‘mcmc+seed‘, ‘filtered seed‘, [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗

**Figure 8.** Figure 8: The plots illustrates the convergence of our MCMC data expansion algorithm. The x-axis represents the [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗

**Figure 9.** Figure 9: Structural consistency analysis of the learned causal models across different stages of the MCMC [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗

**Figure 10.** Figure 10: Extracted graphs of the SA and DD task across the different models [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗

**Figure 11.** Figure 11: Representative causal graphs for the LAJ classification task across different models. Since our framework [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗

read the original abstract

Causal graphs provide a high-level language for making mechanisms transparent. Recent work uses Large Language Models (LLMs) to recover causal graphs of external-world processes. Instead, in this paper, we use causal graphs to model LLM inference itself, providing stakeholders with a transparent view of how the model perceives and organizes high-level concepts to produce a prediction. We propose a four-phase method for constructing such graphs. Given a target LLM and a set of textual examples, our method discovers class-discriminative, human-interpretable concepts and maps each input to LLM-perceived concept states. We then introduce an MCMC-inspired counterfactual augmentation procedure that expands the sparse observational data through chains of counterfactuals. This enables stable causal discovery with $\sigma$-CG, yielding informative, interpretable graphs. We apply our method to three LLMs across disease diagnosis, sentiment analysis, and LLM-as-a-judge classification tasks. We evaluate the learned graphs for predictive fidelity and structural stability, and the MCMC-inspired augmentation for convergence and downstream utility. Our results show that the discovered causal graphs capture meaningful dependencies consistent with LLMs' reasoning. Together, this paper provides a foundation for concept-level explainability of LLMs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper's new angle is modeling the LLM's own concept dependencies with causal graphs via MCMC-augmented data, but the abstract supplies no metrics to check whether the graphs actually track the model's reasoning.

read the letter

The core idea here is using causal graphs to capture how an LLM organizes concepts during inference, rather than recovering external-world causality. The four-phase pipeline starts with discovering class-discriminative concepts, maps inputs to the LLM's perceived states, expands the data with MCMC-inspired counterfactual chains, and runs σ-CG discovery on the result. They run this on disease diagnosis, sentiment analysis, and LLM-as-judge tasks, then check the graphs for predictive fidelity and structural stability.

The framing is distinct from earlier work that has LLMs recover real-world graphs. Treating the model itself as the system to be explained at the concept level makes sense for high-stakes use cases where stakeholders need to see the dependencies the model relies on. The augmentation step is a practical response to sparse observational data, and evaluating both the graphs and the augmentation procedure shows they thought about downstream utility.

The main weakness is that the abstract asserts the graphs capture meaningful dependencies consistent with the LLMs' reasoning but reports no numbers, no baselines, no error bars, and no concrete details on how fidelity or stability were measured. Without those, it is hard to judge whether the results support the claim. The stress-test concern about the counterfactual chains possibly assigning states the LLM would not endorse is reasonable on the current evidence; if the augmentation distorts the model's actual perceptions, the discovered edges could be artifacts. The abstract also leaves open whether validation creates circular dependence on the fitted quantities.

This is for researchers working on concept-level explainability for LLMs who are already comfortable with causal discovery methods. A reader looking for a worked example of the pipeline would find the high-level description useful, but anyone needing quantitative support for deployment would have to wait for the full experiments.

The paper deserves peer review because the pipeline is not in the cited prior literature and the goal is well-motivated, even though the current write-up needs stronger evidence to be convincing.

Referee Report

2 major / 0 minor

Summary. The paper proposes a four-phase pipeline to construct causal graphs that model an LLM's own inference process: (1) discover class-discriminative human-interpretable concepts, (2) map inputs to LLM-perceived concept states, (3) expand the resulting sparse observational data via an MCMC-inspired counterfactual augmentation procedure, and (4) run σ-CG causal discovery. The method is demonstrated on three LLMs across disease diagnosis, sentiment analysis, and LLM-as-a-judge tasks; the authors claim the resulting graphs exhibit predictive fidelity and structural stability and capture dependencies consistent with the LLMs' reasoning.

Significance. If the central claim holds after proper validation, the work would supply a concrete mechanism for concept-level, mechanism-transparent explainability of LLMs that goes beyond post-hoc feature attribution. The combination of counterfactual chain augmentation with causal discovery is a distinctive technical contribution whose utility would be high if the augmentation step can be shown to preserve the target model's concept perceptions.

major comments (2)

[Abstract] Abstract: the abstract asserts that the learned graphs achieve 'predictive fidelity and structural stability' and that 'our results show that the discovered causal graphs capture meaningful dependencies consistent with LLMs' reasoning,' yet supplies no numerical metrics, baselines, error bars, or description of how σ-CG was applied or how fidelity/stability were quantified. This absence makes it impossible to assess support for the central claim.
[Four-phase method] Four-phase method (paragraph describing Phase 3): the MCMC-inspired counterfactual augmentation is asserted to expand sparse data 'without distorting the LLM's actual concept perceptions,' but the manuscript provides no quantitative check (e.g., agreement rate between LLM outputs on original versus augmented inputs, or a held-out consistency metric) that the generated chains preserve the model's own concept states. Because this step is load-bearing for the claim that the discovered edges reflect the LLM's reasoning rather than augmentation artifacts, the missing validation is a material gap.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which highlight opportunities to strengthen the clarity and validation in our manuscript. We address each major comment below and will make the indicated revisions.

read point-by-point responses

Referee: [Abstract] Abstract: the abstract asserts that the learned graphs achieve 'predictive fidelity and structural stability' and that 'our results show that the discovered causal graphs capture meaningful dependencies consistent with LLMs' reasoning,' yet supplies no numerical metrics, baselines, error bars, or description of how σ-CG was applied or how fidelity/stability were quantified. This absence makes it impossible to assess support for the central claim.

Authors: We agree that the abstract would benefit from quantitative support. In the revised version we will incorporate specific metrics for predictive fidelity (e.g., graph-based prediction accuracy on held-out examples), structural stability (e.g., edge overlap across multiple runs), and a concise description of the σ-CG procedure and its parameterization. revision: yes
Referee: [Four-phase method] Four-phase method (paragraph describing Phase 3): the MCMC-inspired counterfactual augmentation is asserted to expand sparse data 'without distorting the LLM's actual concept perceptions,' but the manuscript provides no quantitative check (e.g., agreement rate between LLM outputs on original versus augmented inputs, or a held-out consistency metric) that the generated chains preserve the model's own concept states. Because this step is load-bearing for the claim that the discovered edges reflect the LLM's reasoning rather than augmentation artifacts, the missing validation is a material gap.

Authors: The manuscript reports evaluations of the augmentation procedure for convergence and downstream utility. We acknowledge, however, that an explicit agreement-rate comparison between LLM concept states on original versus counterfactual inputs is not provided. We will add this metric (percentage agreement on a held-out sample of chains) in the revision to directly confirm that the augmentation preserves the target model's perceptions. revision: yes

Circularity Check

0 steps flagged

No circularity: pipeline derives graphs from LLM states then evaluates independently

full rationale

The provided abstract and method description outline a four-phase procedure that first extracts LLM-perceived concept states from textual examples, applies an MCMC-inspired augmentation, runs σ-CG discovery, and then separately evaluates the graphs on predictive fidelity and structural stability. No equation, definition, or quoted claim reduces the final consistency result to the input data by construction, renames a fit as a prediction, or relies on a self-citation chain for a uniqueness theorem. The central claim that the graphs capture meaningful dependencies is presented as an empirical outcome of the pipeline rather than a tautology, satisfying the criteria for a self-contained derivation.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no identifiable free parameters, axioms, or invented entities; the method appears to rely on standard causal discovery assumptions whose details are not stated.

pith-pipeline@v0.9.1-grok · 5746 in / 1115 out tokens · 43660 ms · 2026-06-28T03:07:35.158289+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

24 extracted references · 3 canonical work pages

[1]

Linguis- tics, 47(2):333–386

Causalm: Causal model explanation through counterfactual language models.Comput. Linguis- tics, 47(2):333–386. Tao Feng, Lizhen Qu, Niket Tandon, Zhuang Li, Xiaoxi Kang, and Gholamreza Haffari. 2025. On the relia- bility of large language models for causal discovery. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics...

work page arXiv 2025
[2]

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E

Probing internal representations of multi- word verbs in large language models.CoRR, abs/2502.04789. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Ef- ficient memory management for large language model serving with pagedattention.Preprint, arXiv:2309.06180. Tian Lan, Ji...

work page arXiv 2023
[3]

Association for Computational Linguistics. Jing Ma. 2025. Causal inference with large language model: A survey. InFindings of the Association for Computational Linguistics: NAACL 2025, Albu- querque, New Mexico, USA, April 29 - May 4, 2025, Findings of ACL, pages 5886–5898. Association for Computational Linguistics. Andrew L. Maas, Raymond E. Daly, Peter ...

2025
[4]

The quest for the right mediator: Surveying mechanistic interpretability through the lens of causal mediation analysis

Learning word vectors for sentiment analysis. InProceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA. Association for Computational Lin- guistics. Aaron Mueller, Jannik Brinkmann, Millicent L. Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna...

work page arXiv 2024
[5]

coarse expansion

(N= 1448 ), predicting among Migraine, Sinusitis, and Influenza. For Sentiment Analy- sis, we use raw movie review texts from IMDB (N= 2096 ), classified as Positive or Negative. For LLM-as-a-Judge, we retain the primary user query and two candidate responses sourced from Reddit (Calderon et al., 2025) (N= 395). Handling Positional Bias in LAJThe LAJ task...

2096
[6]

Item” + Original Index (e.g., “Item98

JSON Output: A single valid JSON object. Item keys are “Item” + Original Index (e.g., “Item98”). JSON Structure: { "Item_Original_Index_Here" : { "User Query": "Original user query", "Original Index": "Original Index from input", "Chosen Response": "Final Chosen Response", "Rejected Response": "Final Rejected Response", "ChangeFlag": boolean } } END JSON
[7]

Reasoning Output(After JSON, one block per item): Reasoning for Item [Original Index]:
[8]

Original Index: [Original Index]
[9]

Target Concept: [*Target Concept* Name]
[10]

Current Label for Target Concept: [1, 0, 2, or 3]
[11]

*wanted label*

Target Label: "*wanted label*"
[12]

Labels are identical (Current Label is

Critical Check Outcome: ["Labels are identical (Current Label is "*wanted label*")" OR "Labels are NOT identical (Current Label is [actual_label], not "*wanted label*")"]
[13]

Scenario A applied (because Current Label is

Scenario Applied: ["Scenario A applied (because Current Label is "*wanted label*")" OR "Scenario B applied (because Current Label is [actual_label], not "*wanted label*")"]
[14]

No modifications made as required by Scenario A. ChangeFlag is False

Final Action & ChangeFlag Value Details: - If Scenario A: "No modifications made as required by Scenario A. ChangeFlag is False." - If Scenario B: "Modifications were mandatory under Scenario B. [Describe changes to Chosen and Rejected responses.]" ChangeFlag is True
[15]

Conciseness

If you were given the reason for current labeling by the LLM, add how you included this reason in the modified responses. EXAMPLE (Keep this output structure and reasoning detail): Let’s assume the Target Concept is “Conciseness”. Input Item 1: Original Index:98 User Query:“What is the capital of France?” Chosen Response: “Ah, you’re asking about the capi...
[17]

Current Label for Target Concept: 3
[18]

Critical Check Outcome: Labels are NOT identical (Current Label is 3, not 1)
[19]

Scenario Applied: Scenario B applied (because Current Label is 3, not 1)
[20]

Paris.") was made significantly less concise by adding verbiage. The original Chosen Response (a long sentence) was made more concise. These changes aim to ensure

Final Action & ChangeFlag Value Details: Modifications were mandatory under Scenario B. The original Rejected Response ("Paris.") was made significantly less concise by adding verbiage. The original Chosen Response (a long sentence) was made more concise. These changes aim to ensure "Conciseness" now better represents Chosen. ChangeFlag is True. Reasoning...
[21]

Target Concept: Conciseness
[22]

Current Label for Target Concept: 1
[23]

Critical Check Outcome: Labels are identical (Current Label is 1)
[24]

Scenario Applied: Scenario A applied (because Current Label is 1)
[25]

migraine

Final Action & ChangeFlag Value Details: No modifications made as required by Scenario A. ChangeFlag is False. Dataset: Prompt E.6: Prompt for MCMC Inspired Expansion stage dataset : LIBERTY Role: You are a medical data augmentation and modification expert. Core Task: Your goal is to modify a patient description so that a symptom:*SYMPTOM* becomes *DX* al...

2025

[1] [1]

Linguis- tics, 47(2):333–386

Causalm: Causal model explanation through counterfactual language models.Comput. Linguis- tics, 47(2):333–386. Tao Feng, Lizhen Qu, Niket Tandon, Zhuang Li, Xiaoxi Kang, and Gholamreza Haffari. 2025. On the relia- bility of large language models for causal discovery. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics...

work page arXiv 2025

[2] [2]

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E

Probing internal representations of multi- word verbs in large language models.CoRR, abs/2502.04789. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Ef- ficient memory management for large language model serving with pagedattention.Preprint, arXiv:2309.06180. Tian Lan, Ji...

work page arXiv 2023

[3] [3]

Association for Computational Linguistics. Jing Ma. 2025. Causal inference with large language model: A survey. InFindings of the Association for Computational Linguistics: NAACL 2025, Albu- querque, New Mexico, USA, April 29 - May 4, 2025, Findings of ACL, pages 5886–5898. Association for Computational Linguistics. Andrew L. Maas, Raymond E. Daly, Peter ...

2025

[4] [4]

The quest for the right mediator: Surveying mechanistic interpretability through the lens of causal mediation analysis

Learning word vectors for sentiment analysis. InProceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA. Association for Computational Lin- guistics. Aaron Mueller, Jannik Brinkmann, Millicent L. Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna...

work page arXiv 2024

[5] [5]

coarse expansion

(N= 1448 ), predicting among Migraine, Sinusitis, and Influenza. For Sentiment Analy- sis, we use raw movie review texts from IMDB (N= 2096 ), classified as Positive or Negative. For LLM-as-a-Judge, we retain the primary user query and two candidate responses sourced from Reddit (Calderon et al., 2025) (N= 395). Handling Positional Bias in LAJThe LAJ task...

2096

[6] [6]

Item” + Original Index (e.g., “Item98

JSON Output: A single valid JSON object. Item keys are “Item” + Original Index (e.g., “Item98”). JSON Structure: { "Item_Original_Index_Here" : { "User Query": "Original user query", "Original Index": "Original Index from input", "Chosen Response": "Final Chosen Response", "Rejected Response": "Final Rejected Response", "ChangeFlag": boolean } } END JSON

[7] [7]

Reasoning Output(After JSON, one block per item): Reasoning for Item [Original Index]:

[8] [8]

Original Index: [Original Index]

[9] [9]

Target Concept: [*Target Concept* Name]

[10] [10]

Current Label for Target Concept: [1, 0, 2, or 3]

[11] [11]

*wanted label*

Target Label: "*wanted label*"

[12] [12]

Labels are identical (Current Label is

Critical Check Outcome: ["Labels are identical (Current Label is "*wanted label*")" OR "Labels are NOT identical (Current Label is [actual_label], not "*wanted label*")"]

[13] [13]

Scenario A applied (because Current Label is

Scenario Applied: ["Scenario A applied (because Current Label is "*wanted label*")" OR "Scenario B applied (because Current Label is [actual_label], not "*wanted label*")"]

[14] [14]

No modifications made as required by Scenario A. ChangeFlag is False

Final Action & ChangeFlag Value Details: - If Scenario A: "No modifications made as required by Scenario A. ChangeFlag is False." - If Scenario B: "Modifications were mandatory under Scenario B. [Describe changes to Chosen and Rejected responses.]" ChangeFlag is True

[15] [15]

Conciseness

If you were given the reason for current labeling by the LLM, add how you included this reason in the modified responses. EXAMPLE (Keep this output structure and reasoning detail): Let’s assume the Target Concept is “Conciseness”. Input Item 1: Original Index:98 User Query:“What is the capital of France?” Chosen Response: “Ah, you’re asking about the capi...

[16] [17]

Current Label for Target Concept: 3

[17] [18]

Critical Check Outcome: Labels are NOT identical (Current Label is 3, not 1)

[18] [19]

Scenario Applied: Scenario B applied (because Current Label is 3, not 1)

[19] [20]

Paris.") was made significantly less concise by adding verbiage. The original Chosen Response (a long sentence) was made more concise. These changes aim to ensure

Final Action & ChangeFlag Value Details: Modifications were mandatory under Scenario B. The original Rejected Response ("Paris.") was made significantly less concise by adding verbiage. The original Chosen Response (a long sentence) was made more concise. These changes aim to ensure "Conciseness" now better represents Chosen. ChangeFlag is True. Reasoning...

[20] [21]

Target Concept: Conciseness

[21] [22]

Current Label for Target Concept: 1

[22] [23]

Critical Check Outcome: Labels are identical (Current Label is 1)

[23] [24]

Scenario Applied: Scenario A applied (because Current Label is 1)

[24] [25]

migraine

Final Action & ChangeFlag Value Details: No modifications made as required by Scenario A. ChangeFlag is False. Dataset: Prompt E.6: Prompt for MCMC Inspired Expansion stage dataset : LIBERTY Role: You are a medical data augmentation and modification expert. Core Task: Your goal is to modify a patient description so that a symptom:*SYMPTOM* becomes *DX* al...

2025