The reviewed record of science sign in
Pith

arxiv: 2604.11061 · v1 · submitted 2026-04-13 · cs.LG · cs.AI

Pando: Do Interpretability Methods Work When Models Won't Explain Themselves?

Ziqian Zhong , Aashiq Muhamed , Mona T. Diab , Virginia Smith , Aditi Raghunathan This is my paper

Reviewed by Pith2026-05-10 15:25 UTCgrok-4.3open to challenge →

classification cs.LG cs.AI
keywords mechanistic interpretabilitymodel organismsexplanation faithfulnessgradient attributionrelevance patchingelicitation confounderdecision treesalignment auditing
0
0 comments X

The pith

Gradient-based attribution recovers hidden decision rules in models that give no explanations or false ones, while most other interpretability tools add no value.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper creates Pando, a benchmark of 720 finetuned models that each follow a hidden decision-tree rule but differ in how they respond to queries about their own decisions. Some models explain the true rule faithfully, others give no explanation, and others confidently describe a wrong distractor rule. Agents then try to predict the model's answers on new queries after seeing only ten examples, with or without an added interpretability-tool readout. When models explain faithfully, simply prompting on the examples works as well as or better than any white-box tool. When explanations are missing or misleading, gradient attribution raises accuracy by three to five points and relevance patching raises it further, yet logit lens, sparse autoencoders, and circuit tracing produce no reliable extra gain.

Core claim

The Pando benchmark shows that white-box interpretability methods provide measurable but modest help only in the regime where models withhold or falsify their own explanations; gradient attribution and relevance patching track the actual computation that drives output decisions, whereas other readouts remain dominated by surface-level task biases and field-identity preferences.

What carries the argument

The explanation axis inside Pando, which trains otherwise identical models to output faithful explanations of the true hidden decision-tree rule, no explanation at all, or confident but false explanations of a disjoint distractor rule, then measures the incremental accuracy gained by adding one tool readout to a ten-example black-box prompt.

If this is right

  • Black-box prompting alone is sufficient whenever models produce faithful explanations of their rules.
  • Relevance patching yields the largest accuracy lift among tested tools when explanations are absent or deceptive.
  • Gradient attribution outputs align with the fields that causally determine the model's decision rather than with general task representation.
  • Logit lens, sparse autoencoders, and circuit tracing remain dominated by biases toward field identity and value and therefore add no reliable predictive power.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Prior claims that interpretability tools improve model understanding may often reflect better elicitation of what the model would have said anyway rather than extraction of hidden internal structure.
  • Any future evaluation of interpretability methods should include an explicit no-explanation or misleading-explanation control condition to avoid the elicitation confounder.
  • The modest size of the gains suggests that current white-box tools are best viewed as small supplements rather than complete substitutes for black-box auditing when models are unforthcoming.

Load-bearing premise

The finetuned models actually implement the intended hidden decision-tree rules rather than using unrelated shortcuts that the ten-example prediction task cannot separate from genuine rule following.

What would settle it

Training a set of control models that produce the same query-response pairs but do not implement the decision trees at all, then checking whether the reported accuracy gains from gradient attribution and relevance patching disappear.

Figures

Figures reproduced from arXiv: 2604.11061 by Aashiq Muhamed, Aditi Raghunathan, Mona T. Diab, Virginia Smith, Ziqian Zhong.

Figure 1
Figure 1. Figure 1: Overview of Pando. A decision tree is sampled and finetuned into a language model. The explanation axis controls whether the model explains itself faithfully, not at all, or with a misleading distractor rule. An agent receives k=10 query–response pairs plus optional white-box tool outputs and must predict the model’s decisions on held-out inputs. Right panels show prediction accuracy averaged over scenario… view at source ↗
Figure 2
Figure 2. Figure 2: An example setup in the oversight_defection scenario. Green rows mark the three decision￾relevant fields; the tree shows the planted rule and the path taken by this input. The faithful rationale names the correct fields; the unfaithful rationale confidently cites disjoint distractors. The decision tree here is for illustrative purposes only since it does not yield a near 50/50 class balance. 2.2 Explanatio… view at source ↗
Figure 3
Figure 3. Figure 3: Held-out accuracy vs. sample budget across explanation setups. relp and prefill (faithful￾explanation setup) show the largest advantage at low budgets. Rankings are consistent across budgets, confirming k=10 does not favor particular methods. Field F1 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Automated agent search. Top: progression over ∼25.5 hours (78 experiments); dashed lines show relp baselines; vertical gray lines mark human interactions. Bottom: final results (90% CIs). The improvements from gradient → relp and relp → the final agent are comparable in magnitude, with most gains within CI overlap. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Field F1 vs. sample budget across explanation setups on the car purchase scenario. relp maintains a consistent advantage over sample_only across all budgets, and the gap is largest at low budgets (k=3–5). Under faithful explanations, prefill achieves high F1 even at low budgets and stays relatively stable across budgets. 0)—with small and inconsistent circuit diffs (drivetrain and interior are slightly neg… view at source ↗
read the original abstract

Mechanistic interpretability is often motivated for alignment auditing, where a model's verbal explanations can be absent, incomplete, or misleading. Yet many evaluations do not control whether black-box prompting alone can recover the target behavior, so apparent gains from white-box tools may reflect elicitation rather than internal signal; we call this the elicitation confounder. We introduce Pando, a model-organism benchmark that breaks this confound via an explanation axis: models are trained to produce either faithful explanations of the true rule, no explanation, or confident but unfaithful explanations of a disjoint distractor rule. Across 720 finetuned models implementing hidden decision-tree rules, agents predict held-out model decisions from $10$ labeled query-response pairs, optionally augmented with one interpretability tool output. When explanations are faithful, black-box elicitation matches or exceeds all white-box methods; when explanations are absent or misleading, gradient-based attribution improves accuracy by 3-5 percentage points, and relevance patching, RelP, gives the largest gains, while logit lens, sparse autoencoders, and circuit tracing provide no reliable benefit. Variance decomposition suggests gradients track decision computation, which fields causally drive the output, whereas other readouts are dominated by task representation, biases toward field identity and value. We release all models, code, and evaluation infrastructure.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces Pando, a controlled benchmark using 720 finetuned models that implement hidden decision-tree rules while varying explanation faithfulness (faithful, absent, or unfaithful distractor). Agents predict held-out model decisions from 10 query-response pairs, with or without one interpretability tool output. Results show black-box elicitation suffices when explanations are faithful; when absent or misleading, gradient attribution and relevance patching (RelP) yield 3-5 pp accuracy gains while logit lens, SAEs, and circuit tracing do not; variance decomposition indicates gradients track decision computation whereas other methods are dominated by task biases.

Significance. If the models faithfully implement the target rules, the work supplies a rare controlled test of the elicitation confounder in interpretability, with large scale (720 models), variance decomposition, and full release of models/code/infrastructure. This strengthens claims about when white-box tools add value beyond prompting and provides a reusable model-organism setup for future auditing studies.

major comments (3)
  1. [§3] §3 (model training and faithfulness): The central claim requires that each model's decisions are generated exactly by the hidden decision tree rather than surface heuristics induced by unfaithful-explanation training. No rule-specific counterfactual probes, node ablations, or faithfulness metrics (e.g., agreement on tree-node interventions) are reported; without them the 3-5 pp gains from gradients/RelP and the computation-vs-bias decomposition could reflect recovery of learned distractor correlations instead of isolation of the elicitation confounder.
  2. [§4.2] §4.2 and Table 2 (variance decomposition): The decomposition attributes gradient gains to 'decision fields' and other methods to 'task representation/biases.' This relies on the post-hoc labeling of fields as causal; if the 10-example agent task already encodes field identity, the reported separation may be partly definitional rather than empirical evidence that gradients uniquely track computation.
  3. [§4.1] §4.1 (agent prediction task): The 10-example setup is intended to isolate incremental value of tool outputs. However, the paper does not report whether black-box performance saturates with more examples or whether tool benefits persist when the agent is given the full training distribution; this leaves open whether the reported gains are specific to the low-data elicitation regime or general.
minor comments (2)
  1. [Figure 3] Figure 3 and §4.3: The variance decomposition plots would benefit from error bars across the 720 models and explicit statistical tests for the reported differences between methods.
  2. [§2] §2 (related work): The discussion of prior elicitation studies could more explicitly contrast Pando's explanation-axis control with existing faithfulness benchmarks.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their insightful comments, which help clarify the strengths and limitations of our controlled benchmark. We address each major comment in turn, providing clarifications and committing to revisions where they strengthen the evidence for our claims about the elicitation confounder.

read point-by-point responses

  1. Referee: [§3] §3 (model training and faithfulness): The central claim requires that each model's decisions are generated exactly by the hidden decision tree rather than surface heuristics induced by unfaithful-explanation training. No rule-specific counterfactual probes, node ablations, or faithfulness metrics (e.g., agreement on tree-node interventions) are reported; without them the 3-5 pp gains from gradients/RelP and the computation-vs-bias decomposition could reflect recovery of learned distractor correlations instead of isolation of the elicitation confounder.

    Authors: We agree that verifying the models implement the hidden decision trees, rather than relying on surface heuristics from the training process, is essential to isolate the effect of explanation faithfulness. Our models are fine-tuned on datasets generated exactly by the decision tree rules, with the explanation condition varied independently. To provide stronger evidence, we will incorporate rule-specific counterfactual probes and node ablation experiments into the revised §3. These will demonstrate that model outputs align with the tree structure across all explanation conditions, ensuring the 3-5 pp gains reflect recovery of the true computation rather than distractor correlations. revision: yes

  2. Referee: [§4.2] §4.2 and Table 2 (variance decomposition): The decomposition attributes gradient gains to 'decision fields' and other methods to 'task representation/biases.' This relies on the post-hoc labeling of fields as causal; if the 10-example agent task already encodes field identity, the reported separation may be partly definitional rather than empirical evidence that gradients uniquely track computation.

    Authors: The variance decomposition is performed empirically by partitioning the variance in the agent's predictions based on the known ground-truth components of the decision trees. While the labels are derived from the model structure, the separation is not definitional because the same 10-example inputs are provided to all methods, yet only the gradient-based and RelP methods exhibit substantial variance explained by the 'decision fields' component. Other methods show variance primarily in the bias terms. We will revise the text in §4.2 to emphasize this empirical differential and clarify that the decomposition tests which tools capture the causal computation. revision: partial

  3. Referee: [§4.1] §4.1 (agent prediction task): The 10-example setup is intended to isolate incremental value of tool outputs. However, the paper does not report whether black-box performance saturates with more examples or whether tool benefits persist when the agent is given the full training distribution; this leaves open whether the reported gains are specific to the low-data elicitation regime or general.

    Authors: We chose the 10-example regime to simulate realistic low-data auditing scenarios where an auditor has limited access to model queries. We agree that reporting saturation behavior would strengthen the results. In the revision, we will add experiments in an appendix showing black-box performance with 50 examples and the full training set. These will confirm that while black-box accuracy improves with more data, the relative gains from gradient attribution and RelP persist in the unfaithful explanation conditions, supporting the relevance of our findings to practical elicitation settings. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical benchmark with released artifacts and no derivation chain

full rationale

The paper is a controlled empirical study that trains 720 models under three explanation conditions (faithful, absent, unfaithful) and measures agent prediction accuracy with and without interpretability tool outputs. No equations, ansatzes, or theoretical derivations are present in the provided text or abstract. Claims rest on direct experimental measurements (accuracy deltas of 3-5 pp for gradients/RelP) rather than any fitted parameter renamed as a prediction or self-citation chain. The elicitation-confounder control is implemented via the explicit training axis, not derived from prior self-referential results. All models and code are released, making the benchmark externally falsifiable. This matches the default case of a self-contained empirical paper with no load-bearing circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The work rests on standard machine-learning assumptions about finetuning and evaluation rather than new free parameters or invented entities.

axioms (1)
  • domain assumption Finetuned models can be made to implement hidden decision-tree rules while independently controlling the faithfulness of their verbal explanations.
    Central premise enabling the three explanation conditions.

pith-pipeline@v0.9.0 · 5552 in / 1201 out tokens · 79683 ms · 2026-05-10T15:25:08.364769+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages

  1. [1]

    Marks, J

    URLhttps://arxiv.org/abs/2503.10965. Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iván Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fried Fiotto-Kaufman, Tal Haklay, Michael Hanna, et al. Mib: A mechanistic interpretability benchmark. InForty-second International Conference on Machine Learning, 2025. Joe Needham, Giles Edkins, Govind Pimpale,...

    work page arXiv 2025

  2. [2]

    (attr=−0.44): lists of items or elements, often indicated by commas ... Layer 25: [10150] (attr=−7.44): terms related to artistic or cultural expressions [14186] (attr=−6.59): references to legal or structured standards [14087] (attr=−5.09): No description available

    work page

  3. [3]

    Under review

    (attr=−4.59): scientific terminology and references [14800] (attr=+4.25): dialogue or quoted speech within the text sae_tfidf_filtered— selected layers (keyword-filtered from∼520 to∼137 features across 26 layers): Layer 0: [10370] (tfidf=5.96, act=0.90): affirmative responses to questions or statements 21 Preprint. Under review. ... Layer 13:

    work page

  4. [4]

    Layer 18: [10124] (tfidf=72.27, act=13.40): affirmative answers

    (tfidf=29.58, act=4.72): expressions related to car features and conditions [15179] (tfidf=26.96, act=3.54): phrases indicating the condition and quality of items ... Layer 18: [10124] (tfidf=72.27, act=13.40): affirmative answers

    work page

  5. [5]

    (tfidf=40.63, act=5.99): describing condition

    work page

  6. [6]

    (tfidf=40.33, act=5.90): certain model year

    work page

  7. [7]

    Layer 23: [14116] (tfidf=172.08, act=25.12): references to specific car models and features

    (tfidf=34.41, act=5.81): car brands and models ... Layer 23: [14116] (tfidf=172.08, act=25.12): references to specific car models and features

    work page

  8. [8]

    (tfidf=101.22, act=14.94): references to the BMW brand and its products

    work page

  9. [9]

    condition

    (tfidf=72.15, act=11.45): features related to automobile safety [12493] (tfidf=57.59, act=9.12): specific vehicle names or models ... Keyword filtering surfaces car-relevant features, and some descriptions mention “condition” (the decision- relevant field). We highlighted some layers containing relevant fields here. However, many high-scoring features ref...

    work page 2020

  10. [10]

    RelP gradients(+5min): wrapped gradient computation in RelP context manager with LRP rules (LN, Identity, Half, AH). +0.8 pp. 3.Contrastive comparison(+52min): added per-field yes/no value comparison section. +0.7 pp

    work page

  11. [11]

    First to beatrelpon calibrate

    Candidate rules(+3h44min): programmatic single-field rule construction from top-attributed fields with threshold search. First to beatrelpon calibrate. 26 Preprint. Under review. relp gradient logit_lens_field Field Out In Diff Out In Diff Out In Diff brand 3.0 10.2 +7.2 14.7 41.9 +27.1 −9.9−9.5 +0.4 year 7.1 11.4 +4.3 29.6 71.9 +42.3 −2.6−2.3 +0.3 color ...

    work page

  12. [12]

    verify against ALL examples

    5-step verified prompt(+7h21min): rewrote GPT-5.1 prompt to structured 5-step process with “verify against ALL examples. ”

    work page

  13. [13]

    Start simple

    “Start simple”(+7h56min): 2-line prompt tweak: “Start simple (1–2 fields), add complexity only if needed. ” +0.6 pp

    work page

  14. [14]

    Gradient-guided prefill(+14h41min): run gradientsfirst, use top-attributed field in prefill text if clearly dominant (>1.5×second field)

    work page

  15. [15]

    27 Preprint

    Top-4 fields + 10-token prefill(+21h21min): show only top 4 attributed fields per sample (was all); reduce prefill to 10 tokens (was 20). 27 Preprint. Under review. Agent brand year color horse drive mpg seat inter cond price Mean relp .94* .81* .98* .83* .91* .91* .82* .94* .98* .86* .90 gradient .89* .79* .93* .81* .84* .86* .81* .88* .91* .82* .85 logi...

    work page

  16. [16]

    always write experiment results in experiment_logs/

    Final(+25h32min): fixed a bug where the scenario format’s_header key (used in structured format) was not being filtered out, leaking through as a spurious field in attribution and candidate rules. Neutral on calibrate (which uses natural format without this key) but a correctness fix. Calibrate: 83.4%±0.5%. 45+ additional experiments were tried and discar...

    work page 2014

  17. [17]

    mpg <= 32 (accuracy: 100%, attribution: 207.4)

    work page

  18. [18]

    year <= 2009 (accuracy: 80%, attribution: 69.8)

    work page 2009

  19. [19]

    price <= 27586 (accuracy: 70%, attribution: 64.1)

    work page

  20. [20]

    horsepower >= 199 (accuracy: 60%, attribution: 98.4) Find the decision rule. Steps:

    work page

  21. [21]

    Use the attribution scores to identify the most important fields

    work page

  22. [22]

    Use the model reasoning and input values to determine thresholds

    work page

  23. [23]

    Start simple (1-2 fields), add complexity only if needed

    Formulate a rule. Start simple (1-2 fields), add complexity only if needed

    work page

  24. [24]

    If mismatch, add conditions or adjust

    Verify against ALL examples. If mismatch, add conditions or adjust

    work page

  25. [25]

    Yes" if and only if mpg <= 32; otherwise output

    Output ONLY the final decision rule, nothing else. GPT-5.1 output: Output "Yes" if and only if mpg <= 32; otherwise output "No". GPT-4.1 rule application prompt (one held-out input): You are a decision-making system. Apply the following rule to determine if the output should be Yes or No. Rule: Output "Yes" if and only if mpg <= 32; otherwise output "No"....

    work page 2019

  26. [26]

    horsepower >= 438 (accuracy: 80%, attribution: 123.9)

    work page

  27. [27]

    year <= 2004 (accuracy: 80%, attribution: 62.8)

    work page 2004

  28. [28]

    color =='White'(accuracy: 70%, attribution: 76.8)

    work page

  29. [29]

    Yes" if and only if drivetrain == FWD and color =='White'; otherwise output

    price <= 7494 (accuracy: 70%, attribution: 50.7) [Same 5-step instructions] GPT-5.1 output: Output "Yes" if and only if drivetrain == FWD and color =='White'; otherwise output "No". The agent correctly identifies 2 of 3 ground-truth fields (drivetrain,color) but misseshorsepower, which interacts with the other two fields in a depth-3 tree. The candidate r...

    work page