Pith. sign in

REVIEW 5 major objections 5 minor 4 cited by

Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Self-influence functions trace GPT-2's reasoning to a three-stage path.

desk verdict A promising combination of circuit finding and self-influence, but the central step that turns parameter-level influence into per-token scores is never derived, so the reasoning-path conclusions rest on ungrounded numbers. read the letter →

arxiv 2502.09022 v2 pith:6ONUAHDO submitted 2025-02-13 cs.AI

Lin Zhang , Lijie Hu , Di Wang This is my paper · ORCID
classification cs.AI
keywords mechanisticinterpretabilityinfluencefunctionsself-influencecircuitanalysisGPT-2indirectobjectidentificationreasoningpathstransformercircuits
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that the reasoning behind a language model's answer can be read directly from the model, token by token and layer by layer, rather than being inferred from its outputs. The authors combine automatic circuit finding with influence-function self-influence scores to measure how much each input word contributes inside the small sub-network that performs a task. Applying this to GPT-2 on indirect object identification, they report a three-step hierarchy in which early layers identify the names, middle layers register the action verb, and final layers combine these clues to name the recipient. If correct, this would give interpretability researchers a relatively cheap way to expose the 'thought process' of a transformer on a specific task.

What carries the argument

The mechanism that carries the argument is the layer-wise self-influence score $I_H(x,x) = -\nabla_\theta L(x)^\top H^{-1}\nabla_\theta L(x)$, computed with respect to the parameters of the circuit identified by automatic circuit finding, together with the Hessian-vector-product approximation and the Taylor expansion $H^{-1}=\sum_{i=0}^\infty (I-H)^i$ that make the computation tractable. The paper assigns these scores to individual tokens and reads the largest scores at each layer as the model's current reasoning step.

What would settle it

Take a single IOI sentence, compute the self-influence ranking of tokens at each layer as the paper does, then independently measure token importance by ablating each token's embedding one at a time and recording the change in logit difference; if the two methods rank the key tokens differently or the claimed early-entity / middle-verb / final-recipient pattern disappears, the self-influence story is not causally real.

Watch

Extended reading notes

Core claim

The central claim is that the computation a transformer performs for a reasoning task can be decomposed into a human-interpretable sequence by watching how self-influence scores shift across layers of the task's circuit. For GPT-2 on the IOI task, the paper claims this sequence is: identify the candidate entities in the early layers, detect the action verb 'gave' in the middle layers, and integrate that information in the final layers to select the indirect object. The paper further claims that the circuits used for this task are small (1-2% of edges) yet recover at least 85% of model performance, and that circuits found by EAP-IG and EAP-IG-KL are more faithful than those found by EAP, while all three methods concentrate key nodes in the first and last few layers.

Load-bearing premise

The entire token-level reading rests on an unstated move: the paper computes one self-influence score per layer over circuit parameters and then reports numbers for individual tokens, but it never gives the formula that connects the two, so the per-token rankings are only as sound as that implicit decomposition.

Editorial extensions

If this is right

  • The SICAF framework can in principle be applied to any model and task where a circuit can be found, turning self-influence scores into a map of the model's reasoning path.
  • EAP-IG and EAP-IG-KL produce circuits that are more faithful and more balanced across layers, which supports their use over plain EAP for future mechanistic interpretability studies.
  • If the three-stage hierarchy is real, then probing early-layer representations for entity markers and middle-layer representations for verb semantics would offer a direct check of the claimed computation.
  • The small-circuit result suggests that influence-function computations, which are normally infeasible on full models, become affordable when restricted to the 1-2% of edges that matter for a task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step would be to replace the unstated token-level decomposition with an explicit attribution rule, such as projecting the parameter gradient onto per-token gradients, and test whether the reported path survives.
  • If the method generalizes, it could be used to compare reasoning strategies across model families and scales, and to check whether fine-tuning or prompt style changes the reasoning path before the final answer changes.
  • A direct ablation, such as swapping the action verb in the IOI sentence and monitoring whether the self-influence of the recipient token rises earlier, would separate genuine mechanism from post-hoc narrative.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a framework (SICAF) that combines circuit analysis with influence-function techniques to trace a token-level "reasoning path" in GPT-2 on the indirect object identification (IOI) task. The method first selects a circuit using EAP/EAP-IG/EAP-IG-KL, then computes a self-influence score at each layer, and finally reads off a human-interpretable three-stage reasoning narrative (early entity identification, middle verb interpretation, final recipient synthesis) from per-token self-influence scores reported in Tables 5–7. The paper also reports circuit-faithfulness experiments showing that EAP-IG and EAP-IG-KL identify more faithful circuits than EAP, and that nodes concentrate in the first and last layers.

Significance. If the central methodological step were sound, the paper would offer a new tool for mechanistic interpretability: using self-influence within an identified circuit to attribute token-level importance per layer and thereby infer a model's reasoning strategy. The circuit-faithfulness comparisons (Section 5.2, Table 1) and the observation that EAP-IG variants find more faithful circuits are reasonable and consistent with existing literature. However, the paper's headline claim—that self-influence scores reveal the model's reasoning path—rests entirely on an underspecified and, as written, incorrect bridge between the parameter-level scalar of Eq. (7) and the per-token scores in Tables 5–7. The derivational gap is not a presentation issue; it invalidates the main empirical conclusion as currently stated. The paper also introduces an unanalyzed scaling factor and an unjustified Hessian-vector-product formulation, further undermining the numerical results. These issues are load-bearing, and the conclusion in Section 6 is not supported by the mathematics in Section 4.2.

major comments (5)
  1. [§4.2, Eq. (7); Algorithm 1 Phase 3; Tables 5–7] Equation (7) defines I_H(x,x) = -∇_θ L(x)^T H^{-1} ∇_θ L(x) as a scalar with no token index. Yet Algorithm 1 Phase 3 analyzes a distribution {I_ℓ(x, x_i)} over tokens, and Tables 5–7 report per-token values for every layer. No equation or algorithm in the paper defines I_ℓ(x, x_i) or derives a token-wise decomposition of ∇_θ L(x) through the embedding chain rule. The central claim that self-influence scores measure each token's impact is therefore not derivable from the stated formula, and the three-stage reasoning narrative in Section 5.4 is read directly off these undefined quantities.
  2. [§4.2, Eq. (8)] Equation (8) is presented as a Hessian-vector product for approximating H^{-1}v, but it computes a second derivative with respect to input tokens x, whereas the Hessian H in Eq. (7) is defined with respect to circuit parameters θ. These are mathematically different objects, and the paper offers no justification that one can substitute for the other. This is not a minor notational slip; the stated algorithm cannot compute the quantity claimed by Eq. (7).
  3. [§4.2, Hessian scaling and Taylor expansion] The recursive Taylor expansion H^{-1} = Σ (I - H)^i requires ||H|| ≤ 1 for convergence, and the paper addresses this by introducing a scaling factor c ∈ R+ without specifying how c is chosen or how ||H|| is measured. The approximation is therefore uncontrolled, and no convergence diagnostics or sensitivity analysis are reported. Since every numerical result in Tables 5–7 depends on this approximation, the quantitative claims are not reproducible from the information given.
  4. [§5.4 and §6] Even if a token-level decomposition were supplied, the inference of distinct roles (entity identification vs. verb interpretation vs. recipient synthesis) rests on small differences in the reported scores. For example, Table 5 (EAP-IG-KL) at L8 shows Christina at 0.146 and gave at 0.187, a difference smaller than the variation attributable to the unquantified approximation error. The paper provides no statistical test, ablation, or causal validation linking these score differences to the claimed reasoning stages. The narrative in Section 6 is therefore not established by the reported experiments.
  5. [§4.1 and §4.2] There is a methodological circularity risk: the circuit is selected using gradient-based attribution (EAP and variants), and then self-influence, also gradient-based, is computed within that circuit. The per-token importance values may largely reflect the same attribution signal used to choose the circuit, rather than an independent measure of reasoning. This does not by itself invalidate the approach, but the paper neither discusses nor controls for it, and it is another reason the current results cannot support the strong interpretative conclusions.
minor comments (5)
  1. [§5.2 and Figure captions] The text refers to 'Figure 1 (left plot), (center and right plots)' and to Figures 2–5 with captions that partly duplicate each other; the referencing is inconsistent and should be corrected.
  2. [Table 6] The EAP row for 'Andrew' at L4 lists '1456' rather than '1.456'; this typographical error needs correction.
  3. [§3, last sentence before Influence Functions] The sentence 'run EAP-IG with KL divergence as the loss' is a duplicated fragment and should be removed or integrated.
  4. [Table 4] The term 'Resample' in the Value and Direction columns is undefined; the table should specify what resampling means for each method.
  5. [§5.1] The paper does not state whether the code for SICAF is released; providing code would materially help reproducibility, especially given the underspecified self-influence computation.

Circularity Check

2 steps flagged · score 4.0 of 10

Token-level self-influence is asserted rather than derived from Eq. (7), and the same IOI loss drives both circuit selection and the token-importance narrative, making the central reasoning-path claim partially circular.

  1. self definitional [Section 4.2, Eq. (7) and Algorithm 1, Phase 3]
    "I_H (x, x) = −∇_θ L(x)^⊤ H^{−1} ∇_θ L(x) (7) ... Calculating self-influence allows us to measure each token’s impact on the parameter updates ... [Algorithm 1, Phase 3:] Analyze distribution {I_ℓ(x, x_i)}^n_{i=1} across tokens; Identify significant contributions to model’s decision pathway at layer ℓ."

    Equation (7) is a scalar per layer: ∇_θ L(x) is a gradient with respect to circuit parameters θ, and the quadratic form contains no token index. Algorithm 1 and Tables 5–7 nevertheless treat I_ℓ(x, x_i) as a per-token quantity. No chain-rule decomposition through token embeddings, and no alternate token-indexed definition, is supplied. The paper moves from 'self-influence of the input' to 'self-influence of each token' purely by notation, so the per-token conclusions are true by the name assigned to the scalar rather than by a derived result. The claimed three-stage reasoning path is read directly off these undefined token scores.

  2. fitted input called prediction [Sections 4.1 and 4.2; Eqs. (1) and (7)]
    "EAP identifies important edges by measuring the change in the loss function when each edge is perturbed ... (z′_u − z_u)^⊤ ∇_v L(s) (1) ... We compute the self-influence I_H (x, x) for each layer within the circuit ... where ∇_θ L(x) is the gradient of the loss function L(x)."

    The same task loss L(x) is used both to select the circuit and to score token influence inside it. The circuit is fixed to contain the edges most sensitive to the IOI loss, and then Eq. (7) measures influence of that same loss on those same parameters. Any token that causally drives the IOI logit difference is therefore expected to have high self-influence within the selected circuit. The finding that 'Christina', 'Amy', and 'gave' are the important tokens is largely a restatement of the circuit-selection objective and of the IOI task template (two names, a giving verb, and a target recipient), rather than an independent confirmation of a discovered reasoning process.

full rationale

The paper does not rely on a load-bearing self-citation chain, and it does not import a uniqueness theorem from its own authors; the circuit-faithfulness comparisons against EAP/EAP-IG baselines are externally meaningful. The partial circularity lies elsewhere. First, Eq. (7) defines a parameter-level scalar, while Algorithm 1 and the results tables use per-token scores without any connecting derivation; this is a definitional gap that makes the central 'thought process' claim an assertion rather than a derived consequence. Second, the same IOI loss is used both to discover the circuit (Eq. 1) and to compute self-influence within it (Eq. 7), so the conclusion that names and the verb are influential is partly a restatement of the selection criterion and the task definition. The paper's own limitations section acknowledges the GPT-2-only and IOI-only scope, but it does not address either issue. Because the layer-wise distribution of scores is not fully determined by the circuit-selection step, the circularity is partial (score 4) rather than total.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central claim rests on three unproven premises: convergence of the Hessian inversion, faithfulness of the extracted circuit, and the per-token interpretation of a parameter-level self-influence score. The scaling factor and truncation order are free parameters not specified or justified. No invented physical entities are introduced; the 'reasoning tree' is a conceptual construct without independent evidence.

free parameters (3)
  • Hessian scaling factor c = unspecified
    Introduced in Section 4.2 to ensure ||H|| <= 1 for the Taylor expansion of H^{-1}; the value is never specified or justified, and the scaling direction is ambiguous (H <- cH vs. H <- H/c).
  • Taylor expansion truncation order = unspecified
    Equation (9) approximates H^{-1} by an infinite series; the paper does not state how many terms are used or how approximation error is controlled.
  • Number of circuit edges k = 30 to 1000, main results at 100-1000
    Circuit size is chosen by greedy search over a range; faithfulness and downstream self-influence results depend on this choice, with no principled selection criterion.
assumptions (3)
  • standard math Hessian H is invertible and can be scaled so that the Neumann series for H^{-1} converges.
    Equation (9) requires ||I - H|| < 1 after scaling; the paper assumes this without verifying positive definiteness or spectral radius.
  • domain assumption The circuit extracted by EAP/EAP-IG/EAP-IG-KL faithfully represents the full model's task-relevant computation.
    Circuits recover at least 85% of model performance, but the remaining 15% could carry the reasoning steps the paper claims to trace; the fidelity threshold is arbitrary.
  • ad hoc to paper Self-influence scores computed within the circuit reflect the model's actual token-level decision-making process.
    This is the core interpretive leap in Section 4.2; no causal validation or comparison against baseline attribution methods is provided.
invented entities (1)
  • model's 'thought process' / reasoning tree
    purpose: Explanatory construct claimed to be revealed by self-influence traces across layers.
    The reasoning path (entities -> action -> recipient) is asserted from self-influence heatmaps without external validation, e.g., intervention tests or comparison to known IOI circuit mechanisms.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning." pith.science (2026). https://pith.science/paper/6ONUAHDO

@misc{pith2026250209022,
  author       = {Pith},
  title        = {Pith review of: Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6ONUAHDO}},
  note         = {Machine review of arXiv:2502.09022}
}
read the original abstract

Transformer-based language models have achieved significant success; however, their internal mechanisms remain largely opaque due to the complexity of non-linear interactions and high-dimensional operations. While previous studies have demonstrated that these models implicitly embed reasoning trees, humans typically employ various distinct logical reasoning mechanisms to complete the same task. It is still unclear which multi-step reasoning mechanisms are used by language models to solve such tasks. In this paper, we aim to address this question by investigating the mechanistic interpretability of language models, particularly in the context of multi-step reasoning tasks. Specifically, we employ circuit analysis and self-influence functions to evaluate the changing importance of each token throughout the reasoning process, allowing us to map the reasoning paths adopted by the model. We apply this methodology to the GPT-2 model on a prediction task (IOI) and demonstrate that the underlying circuits reveal a human-interpretable reasoning process used by the model.

Figures

Figures reproduced from arXiv: 2502.09022 by the authors.

Figure 1
Figure 1. (a) A simplified illustration of circuits within the model. (b) An example of how a language model [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of normalized faithfulness, number of nodes, and parameter percentage for circuits identified [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Heatmap of node importance across layers for EAP, EAP-IG, and EAP-IG-KL methods. The x-axis shows [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Self-influence scores of key tokens across model layers for the EAP, EAP-IG, and EAP-IG-KL methods [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Heatmap of node importance across layers for EAP, EAP-IG, and EAP-IG-KL methods. The x-axis [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. COMPKE: Complex Question Answering under Knowledge Editing

    cs.CL 2025-06 conditional novelty 6.0 of 10

    COMPKE is a new benchmark with 11,924 complex questions that tests knowledge editing through one-to-many relations and logical operations, where existing editing methods often fail.

  2. The Compositional Architecture of Regret in Large Language Models

    cs.CL 2025-06 reject novelty 5.0 of 10

    The paper claims that regret in LLMs is encoded by interacting neuron groups detectable in the final hidden layer, using new S-CDI, RDS, and GIC metrics.

  3. Attributing Data for Sharpness-Aware Minimization

    cs.LG 2025-07 reject novelty 4.0 of 10

    SAM-HIF and SAM-GIF are proposed as data attribution scores for SAM-trained models, but SAM-GIF is TracIn with SAM gradients and SAM-HIF's derivation contains a load-bearing error.

  4. Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images

    cs.AI 2025-06 reject novelty 4.0 of 10

    SHE lowers behavioral hallucination scores by about 10 percent by detecting low visual-textual similarity and projecting out the hallucinated direction in embedding space.

Reference graph

Works this paper leans on

63 extracted references · 31 canonical work pages · cited by 4 Pith papers

  1. [1]

    Naman Agarwal, Brian Bullins, and Elad Hazan. 2017. Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research, 18(116):1--40

    work page 2017

  2. [2]

    S Basu, P Pope, and S Feizi. 2021. Influence functions in deep learning are fragile. In International Conference on Learning Representations (ICLR)

    work page 2021

  3. [3]

    Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207--219

    2022

  4. [4]

    L Bereska and E Gavves. 2024. Mechanistic interpretability for ai safety--a review. arXiv preprint arXiv:2404.14082

    arXiv 2024

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  6. [6]

    Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ashton Anderson, and Richard Zemel. 2019. Understanding the origins of bias in word embeddings. In International conference on machine learning, pages 803--811. PMLR

    work page 2019

  7. [7]

    Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2023. Discovering latent knowledge in language models without supervision. In International Conference on Learning Representations (ICLR)

    work page 2023

  8. [8]

    Giuseppe Casalicchio, Christoph Molnar, and Bernd Bischl. 2019. Visualizing the feature importance for black box models. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10--14, 2018, Proceedings, Part I, pages 655--670. Springer International Publishing

    work page 2019

Show all 63 references
  1. [9]

    Hongge Chen, Si Si, Yang Li, Ciprian Chelba, Sanjiv Kumar, Duane Boning, and Cho-Jui Hsieh. 2020. Multi-stage influence function. Advances in Neural Information Processing Systems, 33:12732--12742

    2020

  2. [10]

    Zeming Chen, Gail Weiss, Eric Mitchell, Asli Celikyilmaz, and Antoine Bosselut. 2023. https://arxiv.org/abs/2305.06349 Reckoning: Reasoning through dynamic knowledge encoding . CoRR, abs/2305.06349

    2023 arXiv

  3. [11]

    Keyuan Cheng, Gang Lin, Haoyang Fei, Lu Yu, Muhammad Asif Ali, Lijie Hu, Di Wang, et al. 2024. Multi-hop question answering under temporal knowledge editing. arXiv preprint arXiv:2404.00492

    2024 arXiv

  4. [12]

    Aidan Conmy, Alex Mavor-Parker, Anthony Lynch, et al. 2023. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16318--16352

    2023

  5. [13]

    R Dennis Cook. 2000. Detection of influential observation in linear regression. Technometrics, 42(1):65--68

    2000

  6. [14]

    R Dennis Cook and Sanford Weisberg. 1980. Characterizations of an empirical influence function for detecting influential cases in regression. Technometrics, 22(4):495--508

    1980

  7. [15]

    Ian Covert, Scott Lundberg, and Su-In Lee. 2021. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 22(209):1--90

    2021

  8. [16]

    Antonia Creswell and Murray Shanahan. 2022. https://arxiv.org/abs/2208.14271 Faithful reasoning using large language models . CoRR, abs/2208.14271

    2022 arXiv

  9. [17]

    Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022. https://arxiv.org/abs/2205.09712 Selection-inference: Exploiting large language models for interpretable logical reasoning . CoRR, abs/2205.09712

    2022 arXiv

  10. [18]

    Hwang, Antoine Bosselut, Jackie Chi Kit Cheung, and Yejin Choi

    Yue Dong, Chandra Bhagavatula, Ximing Lu, Jena D. Hwang, Antoine Bosselut, Jackie Chi Kit Cheung, and Yejin Choi. 2021. On-the-fly attention modulation for neural generation. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-...

    2021

  11. [19]

    Andreas Geiger, Hongjing Lu, Thomas Icard, et al. 2021. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574--9586

    2021

  12. [20]

    Mor Geva, Roei Schuster, Jonathan Berant, et al. 2020. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913

    2020 arXiv

  13. [21]

    Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. 2021. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 792--801

    2021

  14. [22]

    Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9304--9312

    2020

  15. [23]

    Han Guo, Nazneen Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. 2021. Fastif: Scalable influence functions for efficient model interpretation and debugging. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10333--10350

    2021

  16. [24]

    Xiaochuang Han, Byron C Wallace, and Yulia Tsvetkov. 2020. Explaining black box predictions and unveiling data artifacts through influence functions. arXiv preprint arXiv:2005.06676

    2020 arXiv

  17. [25]

    Moya Hanna, Sandro Pezzelle, and Yonatan Belinkov. 2024. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. arXiv preprint arXiv:2403.17806

    2024 arXiv

  18. [26]

    Zhaozhi He, Xinyuan Ge, Qixun Tang, et al. 2024. Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on othello-gpt. arXiv preprint arXiv:2402.12201

    2024 arXiv

  19. [27]

    Yihuai Hong, Yuelin Zou, Lijie Hu, Ziqian Zeng, Di Wang, and Haiqin Yang. 2024. Dissecting fine-tuning unlearning in large language models. arXiv preprint arXiv:2410.06606

    2024 arXiv

  20. [28]

    Y. Hou, J. Li, Y. Fei, et al. 2023. Towards a mechanistic interpretation of multi-step reasoning capabilities of language models. arXiv preprint arXiv:2310.14491

    2023 arXiv

  21. [29]

    Lijie Hu, Liang Liu, Shu Yang, Xin Chen, Hongru Xiao, Mengdi Li, Pan Zhou, Muhammad Asif Ali, and Di Wang. 2024 a . A hopfieldian view-based interpretation for chain-of-thought reasoning. arXiv preprint arXiv:2406.12255

    2024 arXiv

  22. [30]

    Improving interpretation faithfulness for vision transformers

    Lijie Hu, Yixin Liu, Ninghao Liu, Mengdi Huai, Lichao Sun, and Di Wang. Improving interpretation faithfulness for vision transformers. In Forty-first International Conference on Machine Learning

  23. [31]

    Lijie Hu, Chenyang Ren, Zhengyu Hu, Hongbin Lin, Cheng-Long Wang, Hui Xiong, Jingfeng Zhang, and Di Wang. 2024 b . Editable concept bottleneck models. arXiv preprint arXiv:2405.15476

    2024 arXiv

  24. [32]

    Lijie Hu, Chenyang Ren, Huanyi Xie, Khouloud Saadi, Shu Yang, Jingfeng Zhang, and Di Wang. 2024 c . Dissecting misalignment of multimodal large language models via influence function. arXiv preprint arXiv:2411.11667

    2024 arXiv

  25. [33]

    Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885--1894. PMLR

    2017

  26. [34]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems (NeurIPS)

    2022

  27. [35]

    Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. 2023. Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models. In The Twelfth International Conference on Learning Representations

    2023

  28. [36]

    Jiaqi Liu, Jian Lou, Zhan Qin, and Kui Ren. 2024. Certified minimax unlearning with generalization rates and deletion capacity. Advances in Neural Information Processing Systems, 36

    2024

  29. [37]

    Kevin Meng, David Bau, Alex Andonian, et al. 2022. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359--17372

    2022

  30. [38]

    Edward Michaud, Ziming Liu, Ugur Girit, et al. 2024. The quantization model of neural scaling. Advances in Neural Information Processing Systems, 36

    2024

  31. [39]

    John Miller, Bilal Chughtai, and William Saunders. 2024. Transformer circuit faithfulness metrics are not robust. arXiv preprint arXiv:2407.08734

    2024 arXiv

  32. [40]

    Neel Nanda. 2023. Mechanistic interpretability quickstart guide. Neel Nanda’s Blog. Accessed: 2023-01-26

    2023

  33. [41]

    Chris Olah. 2022. Mechanistic interpretability, variables, and the importance of interpretable bases. https://www.transformer-circuits.pub/2022/mech-interp-essay

    2022

  34. [42]

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. https://doi.org/10.23915/distill.00024.001 Zoom in: An introduction to circuits . Distill

    2020 doi

  35. [43]

    Alec Radford, Jeffrey Wu, Rewon Child, et al. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9

    2019

  36. [44]

    Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In International Conference on Machine Learning, pages 3145--3153. PMLR

    2017

  37. [45]

    Daniel Smilkov, Nikhil Thorat, Been Kim, et al. 2017. Smoothgrad: Removing noise by adding noise. arXiv preprint arXiv:1706.03825

    2017 arXiv

  38. [46]

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319--3328. PMLR

    2017

  39. [47]

    Aaquib Syed, Charles Rager, and Aidan Conmy. 2023. Attribution patching outperforms automated circuit discovery. arXiv preprint arXiv:2310.10348

    2023 arXiv

  40. [48]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems, 30:5998--6008

    2017

  41. [49]

    Hao Wang, Berk Ustun, and Flavio Calmon. 2019. Repairing without retraining: Avoiding disparate impact with counterfactual distributions. In International Conference on Machine Learning, pages 6618--6627. PMLR

    2019

  42. [50]

    Kai Wang, Anna Variengien, Aidan Conmy, et al. 2022. Interpretability in the wild: A circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593

    2022 arXiv

  43. [51]

    Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. 2023. Machine unlearning of features and labels. Network and Distributed System Security (NDSS) Symposium

    2023

  44. [52]

    Alex Warstadt, Alicia Parrish, Haokun Liu, et al. 2020. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377--392

    2020

  45. [53]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS)

    2022

  46. [54]

    Shu Yang, Muhammad Asif Ali, Cheng-Long Wang, Lijie Hu, and Di Wang. 2024 a . Moral: Moe augmented lora for llms' lifelong learning. arXiv preprint arXiv:2402.11260

    2024 arXiv

  47. [55]

    Shu Yang, Shenzhe Zhu, Ruoxuan Bao, Liang Liu, Yu Cheng, Lijie Hu, Mengdi Li, and Di Wang. 2024 b . What makes your model a low-empathy or warmth person: Exploring the origins of personality in llms. arXiv preprint arXiv:2410.10863

    2024 arXiv

  48. [56]

    Yunzhi Yao, Ning Zhang, Zhihao Xi, et al. 2024. Knowledge circuits in pretrained transformers. arXiv preprint arXiv:2405.17969

    2024 arXiv

  49. [57]

    Zhuoran Zhang, Yongxiang Li, Zijian Kan, Keyuan Cheng, Lijie Hu, and Di Wang. 2024. Locate-then-edit for multi-hop factual recall under knowledge editing. arXiv preprint arXiv:2410.06331

    2024 arXiv

  50. [58]

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023. Representation engineering: A top-down approach to ai transparency. CoRR

    2023

  51. [59]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  52. [60]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  53. [61]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  54. [62]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  55. [63]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.