Pith. sign in

REVIEW 4 major objections 6 minor 45 references

Correcting Gradient-Based Circuit Localization via Interaction-Aware Backpropagation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Attention self-repair erases gradients; a temperature fix restores them

desk verdict A broad, well-executed empirical study of a real attention self-repair phenomenon, wrapped in a theory that is thinner than advertised and a method (TSG) whose causal link to that theory is explicitly unproven. read the letter →

arxiv 2505.17630 v4 pith:ZBRVHMNJ submitted 2025-05-23 cs.CL cs.LG

Joakim Edin , Casper L. Christensen , Róbert Csordás , Tuukka Ruotsalo , Zhengxuan Wu , Maria Maistro , Jing Huang , Lars Maal{o}e This is my paper · ORCID
classification cs.CLcs.LG
keywords attentionself-repairgradient-basedattributionmechanisticinterpretabilitycircuitlocalizationfeaturesoftmaxgradientslayernormalizationlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that attention self-repair—softmax redistributing weight to similar-value positions—makes both perturbation-based and gradient-based interpretability methods systematically underestimate how much attention scores matter. Building on a proof that the gradient of the output logit with respect to an attention score vanishes when all high-weight positions contribute similarly to the output, the authors introduce Gradient Interaction Modifications (GIM), which changes backpropagation in three ways: temperature-adjusted softmax gradients, layernorm freeze, and gradient normalization. GIM is reported to reach state-of-the-art faithfulness on the circuit localization track of the Mechanistic Interpretability Benchmark and to outperform five existing gradient-based feature attribution methods across seven LLMs and six datasets. If the claim is right, existing mechanistic analyses of transformers may have missed or mis-ranked exactly the attention components that matter most.

What carries the argument

The load-bearing object is the softmax gradient identity for attention scores, Equation (3), together with the cancellation that occurs under Equation (5). Attention self-repair is named for the situation where several high-weight attention positions carry similar value information, so the softmax renormalizes to compensate when any one score is perturbed; the corresponding gradient $\partial z/\partial a_j$ is shown to be near zero. The proposed countermeasure, temperature-adjusted softmax gradients (TSG), replaces the softmax used in the backward pass with a higher-temperature version ($\tau > 1$), which flattens the attention distribution, enlarges the set $I_\epsilon$ of positions that receive meaningful gradient weight, and breaks the uniformity that causes cancellation. Two further modifications—layernorm freeze, which treats the normalization scale as constant during backpropagation to block a previously identified layer-norm self-repair, and grad norm, which divides gradients by the number of multiplied inputs—handle other interaction effects. The three modifications together define GIM, and the paper isolates each one's contribution in an ablation study.

What would settle it

Take any input where the paper's self-repair detector fires (multiple high attention weights on similarly contributing values) and compute the joint effect of ablating the top two attention scores together. If GIM's attributions do not rank those jointly ablating scores substantially above the scores whose individual ablations already change the output, or if a model with zero such self-repair cases still shows GIM's full faithfulness gain, then the proposed mechanism is not what drives the improvement.

Watch

Extended reading notes

Core claim

The discovery is a previously unidentified failure mode inside the attention mechanism, formalized as follows. Let $a_j$ be an attention score and $s_j$ its softmax weight; when the value vectors at all positions with substantial weights contribute approximately equally to the output, so $\partial z/\partial s_j \approx c$ for all $j$ in $I_\epsilon$, the softmax gradient identity $\partial z/\partial a_j = s_j(\partial z/\partial s_j(1-s_j)-\sum_{k\neq j}\partial z/\partial s_k s_k)$ collapses to zero because $1-s_j \approx \sum_{k\neq j}s_k$. The paper calls this attention self-repair and shows empirically that jointly ablating the largest attention scores produces much larger output changes than the sum of individual ablations. GIM counteracts the cancellation by recomputing the softmax at a higher temperature during backpropagation, which widens the set of positions with non-negligible weights and lets the two canceling terms separate, and it adds layernorm freeze and gradient normalization for other interaction-driven distortions. The paper's claim is that these modifications make GIM the most consistently faithful gradient-based method for both circuit identification and feature attribution across the tested models.

Load-bearing premise

The load-bearing premise is that running softmax at a higher temperature during backpropagation really approximates jointly ablating several attention scores; the authors state they developed TSG empirically and cannot conclusively establish the causal link.

Editorial extensions

If this is right

  • Existing gradient-based circuit maps likely downgrade any attention head whose attended positions carry redundant values; re-ranking with GIM should change those maps on exactly the inputs where joint ablation differs from individual ablation.
  • Faithfulness gains concentrate in early layers, consistent with evidence that LLMs use early-layer attention heads most; circuit localization should weight early-layer attributions accordingly.
  • A temperature sweet spot exists per model and dataset, so practical deployment of GIM requires tuning $\tau$; the paper's default of 2 was chosen on two datasets and performance degrades at high temperatures.
  • Because GIM is a backpropagation modification, it can be dropped into any gradient-based attribution baseline, not only GradientXInput, and the ablations show grad norm plus layernorm freeze already give most of the gain.
  • Joint perturbation, not single-component perturbation, is the right causal target for importance when self-repair is present; evaluation protocols that only ablate one component will understate importance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not claim this, but if attention self-repair is as frequent as reported (65–1200 occurrences per input), published circuit analyses built on single-component activation patching may have systematically omitted redundant-but-important attention edges; GIM offers a way to re-screen those circuits.
  • The paper's own limitations note that TSG's improved faithfulness is not causally linked to its self-repair mechanism. An unclaimed corollary is that TSG's empirical value may survive even if the firing-squad explanation is wrong, and other attribution methods could adopt higher-temperature backward softmax without endorsing that rationale.
  • The joint-versus-individual perturbation question the paper raises is general: any gradient method on a model with redundant, interacting substructures faces the same cancellation, so the temperature idea could transfer beyond attention to other softmax-based routing modules such as mixture-of-experts gates.
  • If modified gradients are better causal estimates, they could be used not only for interpretation but for training signal or structured pruning; the paper only speculates about training, so this remains an inference.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper identifies a failure mode it calls "attention self-repair": when several attention weights are large and the corresponding value vectors contribute similarly to the output, the gradient of the final logit with respect to each attention score is near zero, so gradient-based attribution methods underestimate the importance of those attention scores. The authors prove this in Section 3.1 (Eqs. 3–7) and propose GIM, a backpropagation method that combines temperature-adjusted softmax gradients (TSG), layer-normalization freezing, and gradient normalization. GIM is evaluated on feature attribution across seven LLMs and six datasets, on layer-wise circuit identification faithfulness, and on the Mechanistic Interpretability Benchmark (MIB) circuit-localization track, where it reports state-of-the-art CPR scores. An ablation study attributes part of the improvement to each modification, and code is released.

Significance. The empirical contribution is potentially valuable: GIM is a simple, cheap modification to gradient-based attribution that reports strong faithfulness gains over established baselines across a wide range of models and datasets, plus competitive MIB results. The paper also honestly discusses residual weaknesses, including per-model variance and the lack of a proven causal mechanism for TSG. However, the theoretical novelty is modest: the zero-gradient result in Eq. (7) is a direct consequence of the standard softmax shift-invariance property, and the paper's central explanation of why TSG works is explicitly conceded to be unproven. If the empirical results are reproducible, the method deserves attention, but the manuscript currently overstates the theoretical contribution and the causal explanation. The broad evaluation, released code, and the inclusion of confidence intervals in the circuit-identification figures are strengths.

major comments (4)
  1. [3.1, Eq. (7)] The derivation in Eqs. (3)–(7) is mathematically correct, but it reduces to the standard softmax constant-shift invariance: for any set of logits with exactly uniform upstream gradients c, the softmax Jacobian yields ∂z/∂a_j = s_j(c − c Σ_k s_k) = 0 because the weights sum to 1. The result does not depend on the value vectors being similar, on attention-specific structure, or on the particular definition of I_ϵ beyond the uniform-gradient assumption. The paper should either provide a theorem or empirical test that distinguishes attention self-repair from generic softmax behavior, or substantially soften the claim that this is a "novel form of self-repair" (Abstract, Section 3).
  2. [4.1, Limitations] The causal link between TSG and the theoretical mechanism is not established, and the paper itself states in the Limitations that "we cannot conclusively establish a causal link between these observations." Specifically, under exactly uniform gradients, increasing the temperature changes s_k but does not break the cancellation in Eq. (7); the argument that expanding I_ϵ to include smaller-weight positions breaks uniformity is informal and unquantified, and newly included positions have proportionally small softmax weights. To support the central claim that TSG works by counteracting attention self-repair, the authors should add an experiment that compares TSG on attention heads with detected self-repair against heads without self-repair, and ideally compare TSG against a control modification (e.g., adding isotropic noise to the softmax gradients) to rule out unrelated effects.
  3. [Table 1, Section 5.2] Table 1 reports point estimates without confidence intervals or significance tests, yet the text makes statements such as "GIM significantly outperforms" and "GIM achieves the highest scores for the majority of combinations." Given that many differences between adjacent entries in Table 1 are small (e.g., 0.68 vs. 0.67 in several cells), the absence of uncertainty quantification undermines the comparative claim. The authors should provide bootstrap confidence intervals or pairwise significance tests, at least for the main feature-attribution table.
  4. [Section 5.2 (temperature selection)] The TSG temperature T=2 is chosen based on results on Gemma-2 2B on the FEVER and HateXplain datasets, which are also part of the main evaluation. This is a form of selection on the evaluation data for those dataset–model pairs. Although Table 7 shows that the results are reasonably stable across temperatures, the main claims would be cleaner if T were selected on a separate validation set not used in the reported table, or if the headline results were averaged over a range of temperatures.
minor comments (6)
  1. [Abstract / Reproducibility statement] The Abstract gives the code URL as https://github.com/corticph/gim, while Section 9 gives https://github.com/JoakimEdin/gim; these should be reconciled.
  2. [References] There are two separate references with identical titles for Kramár et al. (2024a, 2024b); this looks like a duplicate entry and should be collapsed into one.
  3. [Eq. (2)] Equation (3) is derived for the standard softmax without temperature, but Eq. (2) defines softmax with a temperature parameter τ. The default τ=1 should be stated explicitly before Eq. (3).
  4. [Table 1] AttnLRP is listed as "-" for Gemma models; the authors should explain in the caption or text whether the method is not applicable to those models or was not run.
  5. [Figure 4] The caption refers to "layer norm freeze and gradient normalization" in blue and "also including TSG" in orange, but the figure and text do not clearly report the baseline for the relative improvement; please clarify that the baseline is GradientXInput.
  6. [Section 5.2] The text says "We used a temperature of 2 for TSG, which we chose based on the results on Gemma-2 2B on the FEVER and HateXplain datasets," but Table 7 was run after this selection; this ordering should be stated more transparently to avoid the appearance of post-hoc selection.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: Eq. (7) follows algebraically from stated assumptions, TSG is explicitly empirical, and the headline claims are tested against external benchmarks; only minor disclosed tuning on two datasets prevents a clean 0.

full rationale

The paper's core derivation, Eqs. (3)-(7), is a standard softmax Jacobian identity: if the upstream gradients ∂z/∂s_k are approximately uniform over the high-weight set I_epsilon, the constant-shift cancellation in the softmax makes ∂z/∂a_j approximately zero. This is a valid algebraic consequence of the stated condition (5), not a fitted parameter or an imported result; the condition itself is operationalized as low coefficient of variation of value contributions, and the accompanying ablation experiments (Figure 2a) provide an independent behavioral check on the joint-effect claim. TSG is not derived from Eq. (7); the authors state they developed it empirically and, in the Limitations, concede they 'cannot conclusively establish a causal link' between temperature adjustment and faithfulness, which is an explanatory gap rather than circular reasoning. The main empirical claims are evaluated on the external MIB benchmark, with baseline scores copied from Mueller et al. (2025), and on comprehension/sufficiency faithfulness metrics over seven models and six datasets, so the central results are not self-referential. The only mild concern is that the temperature hyperparameter T=2 was selected on Gemma-2 2B for FEVER and HateXplain before the main comparison, and those same cells appear in Table 1; the paper discloses this and Table 7 shows scores are stable across T=1 to 100, so this is minor tuning rather than a forced equivalence. Self-citations (Edin et al. 2025, Csordas et al. 2025) are contextual and not load-bearing for the method's derivation or evaluation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper's central derivation relies only on softmax calculus plus an empirical uniformity condition. The method itself adds one fitted hyperparameter (T=2) and borrows two prior backpropagation rules. No new physical entities are introduced; attention self-repair is a behavioral pattern, not a new object.

free parameters (2)
  • TSG temperature T = 2
    Chosen by the authors based on Gemma-2 2B results on FEVER and HateXplain (Section 5.2), then applied to all models; Table 7 shows robustness across temperatures but the choice is still fitted to the evaluation suite.
  • Self-repair classification threshold (coefficient of variation < 0.1) = 0.1
    Used to count self-repair instances and to select cases for Figure 2; not a parameter of GIM itself, but it defines the empirical phenomenon being measured.
assumptions (4)
  • standard math Softmax gradient identity: ∂z/∂a_j = s_j[(1-s_j)∂z/∂s_j - Σ_{k≠j}s_k ∂z/∂s_k]
    Standard differentiation of softmax, used in Eq. 3 to derive the cancellation result.
  • domain assumption Uniformity condition for self-repair: ∂z/∂s_j ≈ c for all positions with large attention weight
    Formalized in Eq. 5; the paper detects it via coefficient of variation <0.1 over ∂z/∂o·V_j, an empirical proxy rather than a theorem.
  • ad hoc to paper TSG with higher temperature breaks the uniformity condition and approximates joint ablations
    Developed empirically (Section 4.1); the authors admit the causal link is not established in the Limitations section.
  • domain assumption Grad norm divides the gradient by the number of multiplied variables, with optimality from Taylor/Shapley in Achtibat et al.
    Borrowed from AttnLRP and assumed correct without re-derivation in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Correcting Gradient-Based Circuit Localization via Interaction-Aware Backpropagation." pith.science (2026). https://pith.science/paper/ZBRVHMNJ

@misc{pith2026250517630,
  author       = {Pith},
  title        = {Pith review of: Correcting Gradient-Based Circuit Localization via Interaction-Aware Backpropagation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZBRVHMNJ}},
  note         = {Machine review of arXiv:2505.17630}
}
read the original abstract

Circuit localization methods aim to identify the subset of model components responsible for specific behaviors in large language models, enabling detailed mechanistic analysis. Most existing methods assume components act independently and estimate importance by perturbing each component in isolation. However, components in neural networks interact, and ignoring these interactions leads to systematic misestimation of component importance. We find that one particularly problematic interaction is attention self-repair, in which softmax redistribution causes gradients for influential attention scores to vanish as other positions with similar values compensate. We introduce Gradient Interaction Modifications (GIM), a technique that explicitly accounts for feature interactions during backpropagation. GIM achieves state-of-the-art performance on the circuit localization track of the Mechanistic Interpretability Benchmark and outperforms existing gradient-based methods on feature attribution across diverse tasks. By accounting for interaction effects and explaining why prior methods underestimate component importance, GIM enables more faithful mechanistic analysis of large language models. GIM is available as a Python package at https://github.com/corticph/gim.

Figures

Figures reproduced from arXiv: 2505.17630 by the authors.

Figure 1
Figure 1. Attention self-repair. When multiple values associated with large attention weights [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The attention self-repair effect and how temperature-adjusted softmax gradients approximate [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Comparison of circuit identification methods for LLAMA-3.2 1B (95% CI). The top row [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Relative improvement of adding modifications to GradientXInput. We compare adding [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 3
Figure 3. Figure 3: TSG is a naive solution to the self-repair problem; future work should investigate methods to [PITH_FULL_IMAGE:figures/full_fig_p009_3.png]
Figure 5
Figure 5. Figure 5: Faithfulness per layer for LLAMA-3.2 1B. The top row depicts comprehensiveness per layer, where higher is better. The bottom row depicts sufficiency, where lower is better. A.10 Self-repair results In [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Faithfulness per layer for Gemma-2 2B. The top row depicts comprehensiveness per layer, where higher is better. The bottom row depicts sufficiency, where lower is better. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Faithfulness per layer for LLAMA-3.2 3B. The top row depicts comprehensiveness per layer, where higher is better. The bottom row depicts sufficiency, where lower is better. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Self-repair for Gemma-2 2B and how TSG increases the attributions for the attention scores with the strongest self-repair effects 21 [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Self-repair for LLAMA-3.2 1B and how TSG increases the attributions for the attention scores with the strongest self-repair effects 22 [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Self-repair for LLAMA-3.2 3B and how TSG increases the attributions for the attention scores with the strongest self-repair effects 23 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Self-repair for Qwen-2.5 1.5B and how TSG increases the attributions for the attention scores with the strongest self-repair effects 24 [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Self-repair for Qwen-2.5 3B and how TSG increases the attributions for the attention scores with the strongest self-repair effects 25 [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 30 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    AttnLRP : Attention-Aware Layer-Wise Relevance Propagation for Transformers

    Reduan Achtibat, Sayed Mohammad Vakilzadeh Hatefi, Maximilian Dreyer, Aakriti Jain, Thomas Wiegand, Sebastian Lapuschkin, and Wojciech Samek. AttnLRP : Attention-Aware Layer-Wise Relevance Propagation for Transformers . In Forty-First International Conference on Machine Learning , June 2024

    work page 2024

  3. [3]

    XAI for Transformers : Better Explanations through Conservative Propagation

    Ameen Ali, Thomas Schnake, Oliver Eberle, Gr \'e goire Montavon, Klaus-Robert M \"u ller, and Lior Wolf. XAI for Transformers : Better Explanations through Conservative Propagation . In Proceedings of the 39th International Conference on Machine Learning , pp.\ 435--451. PMLR, June 2022

    work page 2022

  4. [4]

    On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation

    Sebastian Bach, Alexander Binder, Gr \'e goire Montavon, Frederick Klauschen, Klaus-Robert M \"u ller, and Wojciech Samek. On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation . PLOS ONE, 10 0 (7): 0 e0130140, July 2015. ISSN 1932-6203. doi:10.1371/journal.pone.0130140

    doi:10.1371/journal.pone.0130140 2015

  5. [5]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss , Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwi...

    arXiv 2005

  6. [6]

    BoolQ : Exploring the Surprising Difficulty of Natural Yes / No Questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ : Exploring the Surprising Difficulty of Natural Yes / No Questions . In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics : ...

    2019

  7. [7]

    Towards Automated Circuit Discovery for Mechanistic Interpretability

    Arthur Conmy, Augustine N Mavor-Parker , Aengus Lynch, Stefan Heimersheim, and Adri \`a Garriga-Alonso . Towards Automated Circuit Discovery for Mechanistic Interpretability

    work page

  8. [8]

    Manning, and Christopher Potts

    R \'o bert Csord \'a s, Christopher D. Manning, and Christopher Potts. Do Language Models Use Their Depth Efficiently ?, May 2025

    work page 2025

Show all 45 references
  1. [9]

    Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER : A Benchmark to Evaluate Rationalized NLP Models . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp.\ 4443...

    2020 doi

  2. [10]

    An Unsupervised Approach to Achieve Supervised-Level Explainability in Healthcare Records

    Joakim Edin, Maria Maistro, Lars Maal e, Lasse Borgholt, Jakob Drachmann Havtorn, and Tuukka Ruotsalo. An Unsupervised Approach to Achieve Supervised-Level Explainability in Healthcare Records . In Yaser Al-Onaizan , Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2...

    2024 doi

  3. [11]

    Christensen, Tuukka Ruotsalo, Lars Maal e, and Maria Maistro

    Joakim Edin, Andreas Geert Motzfeldt, Casper L. Christensen, Tuukka Ruotsalo, Lars Maal e, and Maria Maistro. Normalized AOPC : Fixing Misleading Faithfulness Metrics for Feature Attribution Explainability . In Proceedings of the 63rd Annual Meeting of the Association for Comp...

    2025

  4. [12]

    Information Flow Routes : Automatically Interpreting Language Models at Scale , April 2024

    Javier Ferrando and Elena Voita. Information Flow Routes : Automatically Interpreting Language Models at Scale , April 2024

    2024

  5. [13]

    Hartshorn

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle , Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, and et al. Hartshorn. The Llama 3 Herd of Models , November 2024

    2024

  6. [14]

    Have Faith in Faithfulness : Going Beyond Circuit Overlap When Finding Model Mechanisms , July 2024

    Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have Faith in Faithfulness : Going Beyond Circuit Overlap When Finding Model Mechanisms , July 2024

    2024

  7. [15]

    AtP *: An efficient and scalable method for localizing LLM behaviour to components, March 2024 a

    J \'a nos Kram \'a r, Tom Lieberum, Rohin Shah, and Neel Nanda. AtP *: An efficient and scalable method for localizing LLM behaviour to components, March 2024 a

    2024

  8. [16]

    AtP *: An efficient and scalable method for localizing LLM behaviour to components, March 2024 b

    J \'a nos Kram \'a r, Tom Lieberum, Rohin Shah, and Neel Nanda. AtP *: An efficient and scalable method for localizing LLM behaviour to components, March 2024 b

    2024

  9. [17]

    Towards Faithful Explanations for Text Classification with Robustness Improvement and Explanation Guided Training

    Dongfang Li, Baotian Hu, Qingcai Chen, and Shan He. Towards Faithful Explanations for Text Classification with Robustness Improvement and Explanation Guided Training . In Anaelia Ovalle, Kai-Wei Chang, Ninareh Mehrabi, Yada Pruksachatkun, Aram Galystan, Jwala Dhamala, Apurv Ve...

    2023

  10. [18]

    Optimal ablation for interpretability

    Maximilian Li and Lucas Janson. Optimal ablation for interpretability. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp.\ 109233--109282. Curran Associates, Inc., 2024....

    2024

  11. [19]

    A Unified Approach to Interpreting Model Predictions

    Scott M Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predictions . In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017

    2017

  12. [20]

    Towards Faithful Model Explanation in NLP : A Survey

    Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch . Towards Faithful Model Explanation in NLP : A Survey . Computational Linguistics, 50 0 (2): 0 657--723, June 2024. doi:10.1162/coli_a_00511

    2024 doi

  13. [21]

    Tweet sentiment extraction, 2020

    Maggie , Phil Culliton, and Wei Chen. Tweet sentiment extraction, 2020

    2020

  14. [22]

    HateXplain : A Benchmark Dataset for Explainable Hate Speech Detection

    Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. HateXplain : A Benchmark Dataset for Explainable Hate Speech Detection . Proceedings of the AAAI Conference on Artificial Intelligence, 35 0 (17): 0 14867--14875, May 2021. ISSN 2...

    2021 doi

  15. [23]

    Copy Suppression : Comprehensively Understanding an Attention Head , October 2023

    Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy Suppression : Comprehensively Understanding an Attention Head , October 2023

    2023

  16. [24]

    The Hydra Effect : Emergent Self-repair in Language Model Computations , July 2023

    Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The Hydra Effect : Emergent Self-repair in Language Model Computations , July 2023

    2023

  17. [25]

    Locating and Editing Factual Associations in GPT

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT

  18. [26]

    Locating and Editing Factual Associations in GPT

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT . Advances in Neural Information Processing Systems, 35: 0 17359--17372, December 2022

    2022

  19. [27]

    Mib: A mechanistic interpretability benchmark, 2025

    Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iván Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fiotto-Kaufman, Tal Haklay, Michael Hanna, Jing Huang, Rohan Gupta, Yaniv Nikankin, Hadas Orgad, Nikhil Prakash, Anja Reusch, Aruna Sankaranarayanan, Shun Shao, Alessand...

    2025 arXiv

  20. [28]

    Causality

    Judea Pearl. Causality. Cambridge University Press, September 2009. ISBN 978-0-521-89560-6

    2009

  21. [29]

    Qwen2.5 Technical Report , January 2025

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le ...

    2025

  22. [30]

    Explorations of Self-Repair in Language Models , May 2024

    Cody Rushing and Neel Nanda. Explorations of Self-Repair in Language Models , May 2024

    2024

  23. [31]

    Learning important features through propagating activation differences

    Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML '17, pp.\ 3145--3153, Sydney, NSW, Australia, August 2017 ...

    2017

  24. [32]

    Not Just a Black Box : Learning Important Features Through Propagating Activation Differences , April 2017 b

    Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not Just a Black Box : Learning Important Features Through Propagating Activation Differences , April 2017 b

    2017

  25. [33]

    Deep Inside Convolutional Networks : Visualising Image Classification Models and Saliency Maps , April 2014

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks : Visualising Image Classification Models and Saliency Maps , April 2014

    2014

  26. [34]

    Axiomatic Attribution for Deep Networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Deep Networks . In Proceedings of the 34th International Conference on Machine Learning , pp.\ 3319--3328. PMLR, July 2017

    2017

  27. [35]

    Attribution Patching Outperforms Automated Circuit Discovery , November 2023

    Aaquib Syed, Can Rager, and Arthur Conmy. Attribution Patching Outperforms Automated Circuit Discovery , November 2023

    2023

  28. [36]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , Johan Ferret, Peter Liu, and et al. Tafti. Gemma 2: Improving Open Language Models at a Practical Size...

    2024

  29. [37]

    FEVER : A Large-scale Dataset for Fact Extraction and VERification

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER : A Large-scale Dataset for Fact Extraction and VERification . In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Assoc...

    2018 doi

  30. [38]

    How does This Interaction Affect Me ? Interpretable Attribution for Feature Interactions

    Michael Tsang, Sirisha Rambhatla, and Yan Liu. How does This Interaction Affect Me ? Interpretable Attribution for Feature Interactions . In Advances in Neural Information Processing Systems , volume 33, pp.\ 6147--6159. Curran Associates, Inc., 2020

    2020

  31. [39]

    Fact or Fiction : Verifying Scientific Claims

    David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen , Arman Cohan, and Hannaneh Hajishirzi. Fact or Fiction : Verifying Scientific Claims . In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Method...

    2020 doi

  32. [40]

    SuperGLUE : A Stickier Benchmark for General-Purpose Language Understanding Systems

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. SuperGLUE : A Stickier Benchmark for General-Purpose Language Understanding Systems . pp.\ 30

  33. [41]

    Interpretability in the Wild : A Circuit for Indirect Object Identification in GPT-2 small, November 2022

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the Wild : A Circuit for Indirect Object Identification in GPT-2 small, November 2022

    2022

  34. [42]

    Using `` Annotator Rationales '' to Improve Machine Learning for Text Categorization

    Omar Zaidan, Jason Eisner, and Christine Piatko. Using `` Annotator Rationales '' to Improve Machine Learning for Text Categorization . In Candace Sidner, Tanja Schultz, Matthew Stone, and ChengXiang Zhai (eds.), Human Language Technologies 2007: The Conference of the North Am...

    2007

  35. [43]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  36. [44]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  37. [45]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.