Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that attention in Transformers is a superposition of atomic units that a low-rank sparse decomposition can disentangle into individually comprehensible heads.

desk verdict Lorsa is a genuinely useful sparse attention replacement with solid engineering and a nice rediscovery sanity check, but its 'atomic units' claim needs a ground-truth identifiability test before the strong conclusions are taken at face value. read the letter →

arxiv 2504.20938 v1 pith:3FYM47HO submitted 2025-04-29 cs.LG cs.CL

Zhengfu He , Junxuan Wang , Rui Lin , Xuyang Ge , Wentao Shu , Qiong Tang , Junping Zhang , Xipeng Qiu This is my paper · ORCID
classification cs.LGcs.CL
keywords attentionsuperpositionsparsedictionarylearningmechanisticinterpretabilitylow-rankinductionheadscircuitdiscoverylargelanguagemodelsautoencoders
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Low-Rank Sparse Attention (Lorsa), a replacement module that learns to reproduce a Transformer's multi-head self-attention output using thousands of sparsely activated heads, each reading and writing only a single residual-stream direction. Its central claim is that this decomposition disentangles 'attention superposition': what looks like a polysemantic or uninterpretable attention head is really several atomic attention units packed together, and Lorsa recovers those units as individually comprehensible heads. The paper reports that Lorsa heads rediscover known behaviors such as induction heads, name movers, successor heads, copy suppression, and attention sinks at finer granularity, and uncovers a family of arithmetic-specific heads in Llama-3.1-8B. Automated interpretability scores place Lorsa at parity with sparse autoencoders, and path-patching experiments suggest that Lorsa is better than attention-output SAEs at exposing circuits where several original heads cooperate. If the claim holds, attention-mediated feature interactions in language models become traceable at the level of individual atomic operations.

What carries the argument

The central object is the sparse, low-rank Lorsa head: a one-dimensional OV circuit (one input direction $w_v$, one output direction $w_o$) with a scalar activation $z$, many such heads sharing a single QK circuit, and only the top-K heads active per token. This turns the attention layer into a sparse dictionary over attention directions: the output is a sum of sparsely selected scalar coefficients times output directions. The z pattern, $z_i = A_i v = \sum_j A_{i,j} v_j$, makes each head's activation linearly attributable to previous tokens, which is what lets a single head be interpreted as one atomic 'attend from X to Y and read Z' operation. The QK-sharing groups keep the query-key side high-dimensional because the authors find that lowering QK dimension degrades reconstruction; the sparse top-K selection over thousands of heads is what isolates individual units.

What would settle it

Train a synthetic MHSA layer with, say, five ground-truth attention units superposed over two heads, then train Lorsa on its outputs and check whether the learned heads recover the implanted units one-to-one in both function and attribution. If the recovered units are mixtures, vary across seeds, or depend on hyperparameters, the claim that Lorsa finds the true atomic units would be refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that attention in real Transformers is organized as a sparse superposition of atomic attention units, and that a particular dictionary-learning architecture can expose them. A Lorsa head is an attention head with a one-dimensional OV circuit and a scalar activation; a group of Lorsa heads shares one high-dimensional QK circuit, and at each token only the top-K heads by activation contribute to the output. Trained by squared error against the frozen MHSA output, this model produces heads that are cleaner and finer-grained versions of previously reported MHSA behaviors, and about a quarter of the learned units are spread across multiple original MHSA heads, matching the superposition prediction. The same units also align with SAE features: arithmetic in Llama-3.1-8B decomposes into operand-specific heads such as 'op1 in 27-43' or 'op1 % 10 in [4,5,6]', and automated interpretability gives Lorsa scores comparable to those of SAE features, with better circuit-discovery performance when features are computed collectively.

Load-bearing premise

The argument rests on the premise that the sparse, top-K, single-direction decomposition that best reproduces the original attention output also isolates the model's true atomic attention units; the paper never checks against an example whose true units are known.

Editorial extensions

If this is right

  • If Lorsa heads are the atomic units, interpretability analysis shifts from whole MHSA heads to finer-grained units, so uninterpretable polysemantic heads become collections of comprehensible behaviors.
  • Circuits in which several MHSA heads cooperate to compute one feature become traceable, because each Lorsa head carries its own scalar attribution and can be path-patched independently.
  • The arithmetic-head family implies that arithmetic in LLMs is implemented as a set of per-operand heuristics (range and digit checks) rather than a single algorithm, matching neuron-level findings.
  • Because Lorsa and SAEs trained on the same layers make correlated errors, improvements in one sparse dictionary method may transfer to the other, or the residual error may be a shared 'dark matter' not captured by any sparse linear basis.
  • The architectural constraints identified (QK dimension no smaller than MHSA head dimension, and QK-pair count no fewer than MHSA head count) give concrete design rules for future sparse attention replacement models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the central claim would implant synthetic attention units in a controlled MHSA whose ground truth is known and check whether Lorsa recovers exactly those units; the paper's evidence is limited to rediscovering known heads in real models.
  • If the units are genuine, they should be stable across training seeds and across model sizes; testing whether the same Lorsa heads appear in differently initialized or differently sized models would separate decomposition artifacts from model biology.
  • The 'thematic anchor' heads suggest a mechanism for topical persistence: a small set of heads broadcast a topic representation to all later tokens; patching these heads in long-document tasks would test whether they actually steer vocabulary choice.
  • Since Lorsa error correlates with SAE error, the next step may be a joint model that learns sparse attentional and residual-structure bases together, or a cross-layer version of Lorsa that captures interactions spanning multiple layers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper introduces Low-Rank Sparse Attention (Lorsa), a sparse replacement model for multi-head self-attention. Each Lorsa head has a 1D OV circuit, and the output is a sum over the top-K active heads per token, with QK parameters shared across groups of heads. The model is trained to minimize squared error against the original MHSA output on Pythia-160M and Llama-3.1-8B. The paper reports that Lorsa rediscovers known attention behaviors (induction heads, successor heads, attention sinks, copy suppression), identifies a family of arithmetic-specific heads in Llama-3.1-8B, reaches comparable automated-interpretability scores to SAEs, and presents scaling-law and error analyses. The central claim is that Lorsa decomposes 'attention superposition' into atomic, individually comprehensible attention units, with better circuit-discovery properties than prior methods.

Significance. If the atomicity claim is established, Lorsa would be a significant new tool for mechanistic interpretability, as it is an open-sourced, tractable replacement model that turns attention output into sparse linear components and enables direct QK and OV attribution. The paper's strengths include the clarity of the architecture, the breadth of the empirical study, the open code and weights, and the concrete rediscovery of known head behaviors as an external check. However, the significance is conditional: the identification of atomic units is not currently separated from the choice of sparse factorization, and the 'superior circuit discovery' claim lacks a quantitative baseline.

major comments (4)
  1. [Section 3.1] The scalar least-squares objective L = E||Lorsa(x) - MHSA(x)||^2 does not identify the proposed 'atomic attention units.' For any MHSA head, W_V W_O has rank at most d_h, so replacing (W_V, W_O) by (W_V M^{-1}, M W_O) with any invertible M leaves the head output unchanged while producing a completely different set of 1D OV units. The same non-identifiability applies to sums of heads and to the parameter-shared QK groups used in the paper. Because the paper trains only against MHSA output, infinitely many sparse dictionaries are equivalent under the training loss, and the specific units found by Lorsa need not correspond to ground-truth attention units. The paper cites Jermyn et al. (2024), who construct a case with five ground-truth units in two heads, but it never runs Lorsa on that or any synthetic superposition with known units. Without such a test, or an identifiability condition proved for the Top-K sparse objective, the central claim about recovering atomic units is not supported.
  2. [Appendix F.1/F.2] The evidence that units are spread across MHSA heads is derived from oblique projections of one overcomplete basis onto another (Appendix F.1). For overcomplete dictionaries these coefficients are non-unique, so the 90%-contribution threshold in Appendix F.2 does not establish that a given Lorsa head is a 'true' unit distributed across MHSA heads. This quantity is a property of the chosen factorization, and the claim in Section 2 that about 25% of attention units are spread across multiple heads needs a ground-truth anchor or a uniqueness guarantee.
  3. [Abstract and Section 5] The abstract claims that 'Lorsa exhibits superior circuit discovery properties, especially for features computed collectively by multiple MHSA heads,' but no quantitative circuit-discovery comparison against SAE is provided. Section 5 uses path patching to locate task-relevant Lorsa heads, and Appendix I shows one induction circuit in a 2-layer transformer, yet there is no metric for circuit quality and no SAE-based control. Either provide a controlled comparison or restate the claim as qualitative.
  4. [Section 4.3 and Figure 3] The claim of interpretability parity with SAE rests on t-tests at alpha = 0.05 over 100 samples, yielding 6 wins, 3 losses, and 15 ties across 24 layers. A non-significant difference is not evidence of parity; the report would be stronger with equivalence tests or effect sizes, and the figure should display the score distributions rather than only means and significance indicators.
minor comments (5)
  1. [Section 2] There are typos 'relect' and 'privleged' that should be corrected; additionally, the first piece of evidence (compound-word neurons) refers to MLP neurons, so the text should clarify how it supports polysemy of attention heads.
  2. [Section 3.1] The notation in Algorithm 1 labels w_v^h as a column vector in R^{d x 1} but refers to it as '1-Dim Value weights'; this is correct but should be explained as a map from the residual stream to a scalar.
  3. [Section 3.2] 'with till convergence' should be 'until convergence.'
  4. [Appendix C] The scaling law is fitted to the method's own training data; the text should state explicitly that this is a descriptive fit and not a predictive scaling law, since the relationship between N, K, and loss is only shown at one layer.
  5. [Appendix D] The autointerp prompts contain typos ('indentifying', 'Don´t') that should be fixed; also many code blocks are not typeset as code, making the prompts hard to read.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: Lorsa's reported findings are empirical properties of a fitted replacement model, and the main concerns are identifiability and external validity rather than circular derivation.

full rationale

Lorsa is trained by minimizing the MSE between its output and the original MHSA output, so any behavior present in MHSA that is needed for good reconstruction may be reflected in the replacement model. This makes the reported 'rediscovery' of induction heads, successor heads, copy suppression heads, and attention sinks a consistency check rather than an independent prediction, but it is not a circular step: the training objective does not by itself guarantee that the learned Top-K, 1D-OV heads will be individually interpretable, finer-grained, or aligned with previously known attention mechanisms. Those are empirical properties of the trained decomposition reported in Sections 5.1 and 5.2. The scaling-law experiments in Appendix C fit L(N,K) to the method's own data and present it as descriptive scaling behavior, not as an out-of-sample prediction derived from the same fit. Self-citations such as He et al. 2024a/b and Ge et al. 2024 appear in background and infrastructure roles, alongside independent sources such as Kissane et al. 2024 and Jermyn et al. 2024; no load-bearing premise rests solely on an author-overlapping citation, and no uniqueness theorem is imported from the authors' prior work. The strongest concern, that the MSE objective does not identify a unique sparse dictionary because any invertible linear reparameterization of the 1D OV basis preserves the reconstructed output, is an identifiability and external-validity limitation rather than a circularity. The paper's own Limitations section acknowledges that shared QK circuits raise concerns about independent understandability. Since no equation reduces to its own input and no fitted parameter is relabeled as a prediction, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The method's findings rest on several interpretability-domain assumptions: linear representation, attention superposition, and the belief that MSE-optimal sparse decompositions recover meaningful units. Architecture hyperparameters H, K, QK dimension, and QK binding are hand-chosen and directly shape the discovered units. No new physical entities are introduced; attention units are a postulated conceptual object with only internal behavioral evidence.

free parameters (4)
  • Number of Lorsa heads H = 6144 (Pythia-160M), 32768 (Llama-3.1-8B)
    Chosen as an overcomplete basis for the attention output space; the granularity and interpretability of the decomposition depend on this hand-set number.
  • Number of active heads per token K = 12 (Pythia-160M), 32 (Llama-3.1-8B)
    Sparsity level set equal to the number of original MHSA heads; directly controls the sparse approximation and all downstream head discovery.
  • QK dimension D_QK = 64 (Pythia), 128 (Llama)
    Set equal to the original MHSA head dimension because lower QK dimensions degrade FVU (Appendix B.1); this keeps QK circuits high-dimensional despite the low-rank name.
  • Number of independent QK groups = 96 (Pythia), 256 (Llama)
    Binding ratio choice; performance degrades if the number of independent QK pairs falls below the number of MHSA heads (Appendix B.2), so this is a hand-tuned architectural constraint.
assumptions (5)
  • domain assumption Linear representation hypothesis: residual stream features are 1D linear directions.
    Invoked in Section 3.1 to justify 1D OV circuits; if false, low-rank heads need not correspond to meaningful units.
  • domain assumption Attention superposition hypothesis: MHSA contains atomic attention units superposed across heads.
    Central hypothesis in Section 2; motivates the entire decomposition, but is not directly verified with ground truth.
  • domain assumption MSE-optimal sparse Top-K decomposition recovers interpretable units.
    Training objective in Section 3.1; the paper validates this post hoc via rediscovery and autointerp, but it remains an assumption about the optimization landscape.
  • domain assumption GPT-4o automated interpretability scores measure interpretability.
    Section 4.3 and Appendix D; the parity-with-SAE claim rests on this proxy.
  • domain assumption Known attention mechanisms such as induction heads and successor heads are reliable external benchmarks.
    Section 5.1 uses them as gold labels; they come from prior literature and are themselves partly qualitative.
invented entities (1)
  • Attention units
    purpose: Postulated atomic inter-token feature interactions that Lorsa heads are claimed to realize; the target of decomposition.
    The paper shows behavioral evidence such as induction and arithmetic patterns, but no independent external handle distinguishes these units from other equally valid decompositions. Evidence is internal to the model and the method.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition." pith.science (2026). https://pith.science/paper/3FYM47HO

@misc{pith2026250420938,
  author       = {Pith},
  title        = {Pith review of: Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3FYM47HO}},
  note         = {Machine review of arXiv:2504.20938}
}
read the original abstract

We propose Low-Rank Sparse Attention (Lorsa), a sparse replacement model of Transformer attention layers to disentangle original Multi Head Self Attention (MHSA) into individually comprehensible components. Lorsa is designed to address the challenge of attention superposition to understand attention-mediated interaction between features in different token positions. We show that Lorsa heads find cleaner and finer-grained versions of previously discovered MHSA behaviors like induction heads, successor heads and attention sink behavior (i.e., heavily attending to the first token). Lorsa and Sparse Autoencoder (SAE) are both sparse dictionary learning methods applied to different Transformer components, and lead to consistent findings in many ways. For instance, we discover a comprehensive family of arithmetic-specific Lorsa heads, each corresponding to an atomic operation in Llama-3.1-8B. Automated interpretability analysis indicates that Lorsa achieves parity with SAE in interpretability while Lorsa exhibits superior circuit discovery properties, especially for features computed collectively by multiple MHSA heads. We also conduct extensive experiments on architectural design ablation, Lorsa scaling law and error analysis.

Figures

Figures reproduced from arXiv: 2504.20938 by the authors.

Figure 1
Figure 1. (A) Low-Rank Sparse Attention (Lorsa) comprises thousands of sparsely activated attention heads with 1D outputs, designed to extract interpretable attention units from the original Multi Head Self Attention (MHSA). (B) Lorsa serves as a replacement model for Transformer attention, substituting sparse interpretable components for attention modules. (C) Each Lorsa head explains an atomic feature-feature interaction ac… view at source ↗
Figure 2
Figure 2. Visualization dashboard for a "you"-specific induction Lorsa head. We provide an example [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Automated interpretability scores of Lorsa heads and SAE features. Each distribution is estimated with 100 heads / features. The average score of each group is represented by a horizontal dash line. We highlight distributions with larger mean value suggested by t-tests with α = 0.05. To quantify the interpretability of Lorsa heads in terms of its top activations and z pattern, we perform automated interpretability (… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Examples of Lorsa heads re-discovering previously reported heads. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: For the prompt "36 + 62 =", Lorsa moves two operands to the last position with 3 heads each. The first operand (36) is attended in terms of z pattern by an "op1 ∈ 27 − 43", an "op1 % 10 ∈ [4, 5, 6]" and an "op1 % 10 ∈ [6, 7, 8]" head, which uniquely determines "op1 = 3…
Figure 6
Figure 6. Figure 6: z pattern of a presidency-related topic broadcasting Lorsa head. 6 Discussion and Limitations We report a number of intriguing findings and limitations of Low-Rank Sparse Attention. We believe there remains significant room for improvement for future work in each of th…
Figure 7
Figure 7. Figure 7: Ablation study on the QK dimension using Pythia-160M under different context lengths. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Ablation study on the binding ratio. We vary the number of independent Lorsa QK heads [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Comparison of scaling law of convergence loss with number of parameters and fixed [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Detailed information on Lorsa’s rediscovery of reported functional heads. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Distribution of Lorsa heads based on the number of original attention heads they are [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: FVU of Lorsa and SAE across each layer in Pythia-160M and Llama-3.1-8B. Both models [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Per-Token Error Norms of Lorsa and SAE on Layers 2, 6, and 10 of Pythia-160M for 64 [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: Per-Token error cosine similarity distribution between Lorsa and SAE on Layers 2, 6, and [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: Induction circuit found in our fully sparsified replacement model. [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Targeted Recovery of Weight-Space Mechanisms From Neural Networks

    cs.LG 2026-06 conditional novelty 6.0 of 10

    A targeted decomposition method recovers the weight-space mechanisms behind specific inputs at low FLOPs, enabling focused ablation and rewiring of a 12-block transformer.

Reference graph

Works this paper leans on

51 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Hyland, Daniel C

    Ahmed Abdulaal, Hugo Fry, Nina Monta \ n a Brown, Ayodeji Ijishakin, Jack Gao, Stephanie L. Hyland, Daniel C. Alexander, and Daniel C. Castro. An x-ray is worth 15 features: Sparse autoencoders for interpretable radiology report generation. CoRR, abs/2410.03334, 2024. doi:10.48550/ARXIV.2410.03334. URL https://doi.org/10.48550/arXiv.2410.03334

    arXiv doi:10.48550/arxiv.2410.03334

  2. [2]

    GQA: training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee - Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \' o n, and Sumit Sanghai. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singa...

    doi:10.18653/v1/2023.emnlp-main.298 2023

  3. [3]

    Investigating successor heads

    Emmanuel Ameisen, Joshua Batson, and Jack Lindsey. Investigating successor heads. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/september-update/index.html

    work page 2024

  4. [4]

    Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivo...

    2025

  5. [5]

    Linear algebraic structure of word senses, with applications to polysemy

    Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Trans. Assoc. Comput. Linguistics, 6: 0 483--495, 2018. doi:10.1162/TACL\_A\_00034. URL https://doi.org/10.1162/tacl\_a\_00034

    doi:10.1162/tacl 2018

  6. [6]

    Circuits updates - march 2024

    Joshua Batson, Brian Chen, and Andy Jones. Circuits updates - march 2024. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/march-update/index.html

    work page 2024

  7. [7]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyung...

    2023

  8. [8]

    Language models can explain neurons in language models

    Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html, 2023

    2023

Show all 51 references
  1. [9]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen,...

    2023

  2. [10]

    Learning multi-level features with matryoshka saes

    Bart Bussmann, Patrick Leask, and Neel Nanda. Learning multi-level features with matryoshka saes. LessWrong, 2024. URL https://www.lesswrong.com/posts/rKM9b6B2LqwSB5ToN/learning-multi-level-features-with-matryoshka-saes

    2024

  3. [11]

    Circuits updates - february 2024

    Tom Conerly. Circuits updates - february 2024. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/feb-update/index.html#dict-learning-resampling

    2024

  4. [12]

    Circuits updates - april 2024

    Tom Conerly, Adly Templeton, Trenton Bricken, Jonathan Marcus, and Tom Henighan. Circuits updates - april 2024. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/april-update/index.html#training-saes

    2024

  5. [13]

    Mavor - Parker, Aengus Lynch, Stefan Heimersheim, and Adri \` a Garriga - Alonso

    Arthur Conmy, Augustine N. Mavor - Parker, Aengus Lynch, Stefan Heimersheim, and Adri \` a Garriga - Alonso. Towards automated circuit discovery for mechanistic interpretability. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editor...

    2023

  6. [14]

    Sparse autoencoders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. CoRR, abs/2309.08600, 2023. doi:10.48550/ARXIV.2309.08600. URL https://doi.org/10.48550/arXiv.2309.08600

    2023 doi arXiv

  7. [15]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024 doi arXiv

  8. [16]

    Transcoders find interpretable LLM feature circuits

    Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits. CoRR, abs/2406.11944, 2024. doi:10.48550/ARXIV.2406.11944. URL https://doi.org/10.48550/arXiv.2406.11944

    2024 doi arXiv

  9. [17]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dari...

    2021

  10. [18]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposi...

    2022

  11. [19]

    Privileged bases in the transformer residual stream

    Nelson Elhage, Robert Lasenby, and Christopher Olah. Privileged bases in the transformer residual stream. Transformer Circuits Thread, 2023. URL https://transformer-circuits.pub/2023/privileged-basis/index.html

    2023

  12. [20]

    Decomposing the dark matter of sparse autoencoders

    Joshua Engels, Logan Riggs, and Max Tegmark. Decomposing the dark matter of sparse autoencoders. CoRR, abs/2410.14670, 2024. doi:10.48550/ARXIV.2410.14670. URL https://doi.org/10.48550/arXiv.2410.14670

    2024 doi arXiv

  13. [21]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupr \' e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. CoRR, abs/2406.04093, 2024. doi:10.48550/ARXIV.2406.04093. URL https://doi.org/10.48550/arXiv.2406.04093

    2024 doi arXiv

  14. [22]

    Automatically identifying local and global circuits with linear computation graphs

    Xuyang Ge, Fukang Zhu, Wentao Shu, Junxuan Wang, Zhengfu He, and Xipeng Qiu. Automatically identifying local and global circuits with linear computation graphs. CoRR, abs/2405.13868, 2024. doi:10.48550/ARXIV.2405.13868. URL https://doi.org/10.48550/arXiv.2405.13868

    2024 doi arXiv

  15. [23]

    Successor heads: Recurring, interpretable attention heads in the wild

    Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, interpretable attention heads in the wild. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://open...

    2024

  16. [24]

    Finding neurons in a haystack: Case studies with sparse probing

    Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. Trans. Mach. Learn. Res., 2023, 2023. URL https://openreview.net/forum?id=JYs1R9IMJr

    2023

  17. [25]

    How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model

    Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neura...

    2023

  18. [26]

    Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on othello-gpt

    Zhengfu He, Xuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, and Xipeng Qiu. Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on othello-gpt. CoRR, abs/2402.12201, 2024 a . doi:10.48550/ARXIV.2402.12201. URL https://do...

    2024 doi arXiv

  19. [27]

    Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders

    Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu - Gang Jiang, and Xipeng Qiu. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. CoRR, abs/2410.20526, 2024 ...

    doi arXiv

  20. [28]

    Circuits updates - january 2024

    Adam Jermyn, Chris Olah, and Tom Conerly. Circuits updates - january 2024. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/jan-update/index.html#attn-superposition

    2024

  21. [29]

    Interpreting attention layer outputs with sparse autoencoders

    Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. Interpreting attention layer outputs with sparse autoencoders. CoRR, abs/2406.17759, 2024. doi:10.48550/ARXIV.2406.17759. URL https://doi.org/10.48550/arXiv.2406.17759

    2024 doi arXiv

  22. [30]

    We inspected every head in gpt-2 small using saes so you don’t have to

    Robert Krzyzanowski, Connor Kissane, Arthur Conmy, and Neel Nanda. We inspected every head in gpt-2 small using saes so you don’t have to. Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/xmegeW5mqiBsvoaim/we-inspected-every-head-in-gpt-2-small-using-saes-so-you-don

    2024

  23. [31]

    Dragan, Rohin Shah, and Neel Nanda

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J \' a nos Kram \' a r, Anca D. Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. CoRR, abs/2408.05147, 2024. doi:10.4...

    2024 doi arXiv

  24. [32]

    Sparse crosscoders for cross-layer features and model diffing

    Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/crosscoders/index.html

    2024

  25. [33]

    Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, A...

    2025

  26. [34]

    Copy suppression: Comprehensively understanding an attention head

    Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppression: Comprehensively understanding an attention head. CoRR, abs/2310.04625, 2023. doi:10.48550/ARXIV.2310.04625. URL https://doi.org/10.48550/arXiv.2310.04625

    2023 doi arXiv

  27. [35]

    Locating and editing factual associations in GPT

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT . In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neu...

    2022

  28. [36]

    Arithmetic without algorithms: Language models solve math with a bag of heuristics

    Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic without algorithms: Language models solve math with a bag of heuristics. CoRR, abs/2410.21272, 2024. doi:10.48550/ARXIV.2410.21272. URL https://doi.org/10.48550/arXiv.2410.21272

    2024 doi arXiv

  29. [37]

    interpreting gpt: the logit lens

    nostalgebraist. interpreting gpt: the logit lens. lesswrong, 2020. URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  30. [38]

    Circuits updates - july 2024

    Chris Olah and Adam Jermyn. Circuits updates - july 2024. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/july-update/index.html#hurdles

    2024

  31. [39]

    Zoom in: An introduction to circuits

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi:10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in

    2020 doi

  32. [40]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kam...

    2022

  33. [41]

    Improving dictionary learning with gated sparse autoencoders

    Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, J \' a nos Kram \' a r, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders. CoRR, abs/2404.16014, 2024. doi:10.48550/ARXIV.2404.16014. URL https://doi.org/10...

    2024 doi arXiv

  34. [42]

    SlimPajama: A 627B token cleaned and deduplicated version of RedPajama

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama . https://cerebras.ai/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, 2023. URL ht...

    2023

  35. [43]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. CoRR, abs/2104.09864, 2021. URL https://arxiv.org/abs/2104.09864

    2021 arXiv

  36. [44]

    Circuits updates - january 2024

    Adly Templeton, Joshua Batson, Adam Jermyn, and Chris Olah. Circuits updates - january 2024. Transformer Circuits Thread, 2024 a . URL https://transformer-circuits.pub/2024/jan-update/index.html#predict-future

    2024

  37. [45]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua ...

    2024

  38. [46]

    Towards universality: Studying mechanistic similarity across language model architectures

    Junxuan Wang, Xuyang Ge, Wentao Shu, Qiong Tang, Yunhua Zhou, Zhengfu He, and Xipeng Qiu. Towards universality: Studying mechanistic similarity across language model architectures. CoRR, abs/2410.06672, 2024. doi:10.48550/ARXIV.2410.06672. URL https://doi.org/10.48550/arXiv.2410.06672

    2024 doi arXiv

  39. [47]

    Interpretability in the wild: a circuit for indirect object identification in GPT-2 small

    Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, ...

    2023

  40. [48]

    Addressing feature suppression in saes

    Benjamin Wright and Lee Sharkey. Addressing feature suppression in saes. LessWrong, 2024. URL https://www.lesswrong.com/posts/3JuSjTZyMzaSeTxKk/addressing-feature-suppression-in-saes

    2024

  41. [49]

    Retrieval head mechanistically explains long-context factuality

    Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality. CoRR, abs/2404.15574, 2024. doi:10.48550/ARXIV.2404.15574. URL https://doi.org/10.48550/arXiv.2404.15574

    2024 doi arXiv

  42. [50]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://op...

    2024

  43. [51]

    Towards best practices of activation patching in language models: Metrics and methods

    Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/...

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.