Pith. sign in

REVIEW 3 major objections 6 minor 50 references

A single-token feature's causal necessity is real but depends on which sparse autoencoder family produced it, and layer depth decides whether ablation damage propagates downstream or directly reshapes the output.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:57 UTC pith:2RHNX5NZ

load-bearing objection Genuinely new cross-family causal comparison with an excellent causal arm, but the 'LlamaScope is locally redundant' headline rests on an uncalibrated recovery metric and an overstating abstract. the 3 major comments →

arxiv 2607.20596 v1 pith:2RHNX5NZ submitted 2026-07-22 cs.LG cs.CL

Are Single-Token Sparse Autoencoder Features Causally Necessary? Layer-Depth and SAE-Family Effects

Seonglae Cho , Zekun Wu , Kleyton Da Costa , Rishi Kalra , Ilham Wicaksono , Adriano Koshiyama This is my paper
classification cs.LG cs.CL
keywords sparse autoencoderssingle-token featureszero-ablationcausal necessityinterpretabilitylanguage modelslayer depthSAE families
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Single-token features—SAE directions that activate almost exclusively on one vocabulary item—are the cleanest possible test case for whether sparse-autoencoder interpretations are portable across training setups. The paper finds that zero-ablating these features reduces the target token's logit at Benjamini-Hochberg-significant levels in 178 of 208 full-layer tests across six models and three SAE families: single-token features are causally necessary in a real sense. But that necessity is not a property of the token; it is a property of the SAE. On the same base model, features from one family remain 'causally anchored' and their ablation cascades downstream, while features from another family are 'locally redundant'—the target token's rank recovers to within twice baseline 96–98% of the time. Depth dissociates the two roles: necessity damage grows with depth, while anchoring damage concentrates in early layers. If this is right, cross-family interpretability claims must treat the SAE family as an experimental variable, not a detail.

Core claim

The central claim is that at the single-token endpoint—where ground truth is unambiguous—the causal role of a feature depends on which SAE produced it. Using zero-ablation at full layer depth on 3.9M features across six models and three SAE families, the paper shows single-token features are geometrically distinct (4.7× tighter decoder clustering, 1.72× higher embedding alignment, concentrated in early layers) and causally necessary under ablation in 178 of 208 layer conditions. Yet the same ablation protocol splits the three families: on Gemma models two families anchor downstream layers 92–100% of the time, while on Llama/DeepSeek models the other family anchors only 31–34% and shows 96–98

What carries the argument

The load-bearing instrument is zero-ablation: removing the feature's contribution from the residual stream by subtracting its activation times its decoder vector, then measuring the change in the target token's logit against magnitude-matched random controls. Two derived quantities carry the argument: 'necessity' (statistically significant logit reduction at the source layer) and 'anchoring' (the ablation's downstream effect on logit-lens readouts at later layers), with 'same-layer recovery'—the fraction of features whose target-token rank stays within twice baseline, floor 5—distinguishing anchored from locally redundant regimes. Detection uses decoder-alignment, the cosine between a featur

Load-bearing premise

The split between 'anchored' and 'locally redundant' rests on the same-layer recovery metric—rank within twice baseline after ablation, floor 5—without calibrating it against pre-ablation rank distributions or the size of the logit drop; if that window or baseline differences drive recovery, the family split weakens.

What would settle it

Compute the same-layer recovery metric on magnitude-matched random controls after sorting them into pre-ablation rank bins identical to the single-token features' bins; if controls also show 96–98% recovery in the rank-1 bin, the 'locally redundant' label is a rank-window artifact.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Interpretability claims built on one SAE family should not be assumed to transfer to another, even on the same base model; steering and editing pipelines should re-run ablation checks under the deployed family.
  • Layer depth should be reported as part of causal claims: late-layer features shape the output distribution directly, while early-layer features propagate damage downstream.
  • Activation function is not the decisive factor in cross-family causal differences; training recipe factors such as decoder norms, training scale, and post-hoc conversion are the residual candidates.
  • Single-token features provide a tractable benchmark endpoint where vocabulary-level ground truth allows exact cross-family matching, so they can serve as a diagnostic for SAE evaluation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to train two SAEs with identical recipe, data, and width, differing only in activation function, on the same base model across all layers; if the anchored-versus-redundant split persists, recipe controls are needed, and if it collapses, the split is an artifact of uncontrolled recipe differences.
  • The same-layer recovery metric could be recalibrated as a function of pre-ablation rank and logit-drop magnitude; features starting at rank 1 have a floor-5 window that makes 'recovery' artificially easy.
  • The category-dependent convergence—domain-specific tokens converge across families while function words diverge—suggests that future cross-SAE comparisons should be stratified by token type, not reported only in aggregate.
  • If the family split is training-recipe-driven, then SAE evaluation benchmarks that report only reconstruction fidelity or interpretability are missing a causal dimension; adding per-feature necessity scores would make cross-family comparability measurable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies 'single-token' sparse autoencoder (SAE) features — features whose activation is dominated by a single vocabulary item — across six language models and three SAE families (GemmaScope/res-jb, LlamaScope, community BatchTopK). It reports that such features are geometrically distinct (4.7x tighter decoder clustering, 1.72x higher embedding alignment), concentrated in early layers (91% in GPT2 L0), and causally necessary under zero-ablation at full layer depth, with 178 of 208 full-layer conditions significant under a single global Benjamini-Hochberg correction. The paper's headline claim is a cross-family dissociation: on the same base model, GemmaScope and BatchTopK features are 'causally anchored,' while LlamaScope features are 'locally redundant,' recovering their pre-ablation target-token rank 96-98% of the time. It also reports a depth dissociation: necessity damage increases with depth (rho=0.97 for BatchTopK, 0.70 for GemmaScope on Gemma-2-2B) while downstream anchoring concentrates in early layers (rho=-0.65). The paper concludes that a feature's causal role depends on which SAE produced it, and that SAE family should be treated as an experimental variable.

Significance. If the central dissociation between 'anchored' and 'locally redundant' SAE families held, the paper would make an important contribution: it would show that single-token SAE features, the cleanest possible case for ground-truth comparison, have real but non-portable causal necessity, and that interpretability claims must control for SAE training methodology. The paper has genuine strengths: the causal arm is large and internally consistent (178/208 layer-level tests verified from Table 6), uses a single global BH correction, includes magnitude-matched controls, and reports control-population inertness (B.13, recovery >=99.96%). The inclusion of an alignment-matched null (B.14) is a serious attempt to address selection-geometry circularity. The empirical patterns — early-layer concentration, the L0-to-L1 representational shift, increasing necessity with depth — are plausible and well documented. However, the load-bearing cross-family claim currently rests on an uncalibrated recovery metric and on a same-base comparison that, as written, is not supported by the evaluated model-by-SAE matrix. These issues materially affect the paper's central conclusion, though they appear fixable wit

major comments (3)
  1. [§4.5, Table 6, Table 20, B.13] The anchored-vs-redundant split is read almost entirely off the same-layer recovery metric, defined as the fraction of features whose post-ablation target-token rank stays within twice its pre-ablation rank, with a floor of 5. This metric is never calibrated. Llama-3.1-8B x LlamaScope shows 97.7% recovery while Table 20 reports mean Delta logit = -1.828 at L1, -0.5 to -0.9 across most layers, and BH-significant necessity in 31/32 layers. Under the floor-5 rule, a rank-1 token 'recovers' if it stays anywhere in ranks 1-5; a large logit drop can leave it at rank 2-5 while substantially reducing its probability. The paper does not report pre-ablation rank distributions, nor a null recovery rate under magnitude-matched random ablations. B.13's control population is inert by construction (median Delta logit between -0.00006 and -0.0012), so its >=99.96% recovery does not establish what recove
  2. [Abstract, §1, Table 1, §4.5, Limitations] The abstract and §1 state that 'on the same base model, GemmaScope and BatchTopK features remain causally anchored, while LlamaScope features are locally redundant.' Table 1 shows LlamaScope is evaluated only on Llama-3.1-8B and DeepSeek-R1; no LlamaScope SAE is evaluated on any Gemma model, and no GemmaScope/BatchTopK SAE is evaluated on any Llama model. Thus the LlamaScope half of the family split is conflated with base model, tokenizer, and pretraining data. The Limitations section concedes that cross-family comparisons co-vary training data, but the Conclusion's claim that 'the causal role of a feature depends on which SAE produced it' goes beyond the within-model GemmaScope-vs-BatchTopK evidence. A same-base LlamaScope comparison (or a GemmaScope-style SAE on Llama) is needed before the abstract's 'same base model' phrasing can stand.
  3. [§3.3, B.14, Table 27] Decoder-alignment detection selects features by cosine similarity between the decoder vector and the target token's input embedding, and the zero-ablation removes exactly that decoder direction. The paper's B.14 alignment-matched null is a reasonable response, but the matching gap is large: median |Delta cos| = 0.18 on GemmaScope and 0.51 on LlamaScope. The in-band subset is limited to 315 controls across five Gemma-2-2B layers, with one of five layers inconclusive (L6), and the LlamaScope layer-1 nearest controls themselves carry a -1.318 mean Delta logit. The circularity concern is therefore only partially mitigated. Please report the recovery metric on the in-band alignment-matched null as well as necessity, and stratify results by |Delta cos| or provide a closer-matched set. This directly bears on whether the causal arm measures feature role or selection geometry.
minor comments (6)
  1. [Table 6] The label 'LlamaScope (TopK→JR)' is never defined. §3.1 describes LlamaScope as TopK SAEs; §3.3 mentions a TopK-to-JumpReLU conversion. Explain what conversion was applied and when, since the causal results for LlamaScope depend on the final SAE type actually ablated.
  2. [§4.4, Table 5] Table 5 is labeled 'Causal (N=26,594)' but the main causal analyses use decoder-alignment detection rather than activation-based detection. Clarify which detector produced the semantic-category causal statistics, and whether the category distribution is over the decoder-detected set.
  3. [Figure 4B] The caption says '17 sampled model×layer conditions,' but Table 6 reports 208 full-layer conditions. Explain the sampling and whether Figure 4B is illustrative; the text implies the full-layer coverage is in Table 6.
  4. [§4.5] The text quotes recovery as '96-98%' for LlamaScope, but Table 6 lists 97.7% and 95.5%. State both values consistently and reconcile the 96-98% range with the 95.5% DeepSeek-R1 value.
  5. [§1/Table 6] GPT2-Small is described as a 'single-layer condition' but the abstract and §4.5 say 'across six transformer language models and three SAE families.' Specify that GPT2 contributes one layer (L0) and that full-depth coverage is for the seven configurations.
  6. [§4.5] The anchoring test is described as one-sided Mann-Whitney U at p<0.05, but the necessity test uses global BH correction over 208 layers. Clarify whether anchoring p-values are also BH-corrected, and if not, justify the different multiple-testing treatment.

Circularity Check

1 steps flagged

Decoder-alignment selection makes negative ablation logits partly by construction; the paper's own B.14 null is imperfect, so necessity is partially self-definitional, but the family split rests on independent recovery/anchoring metrics.

specific steps
  1. self definitional [§3.3 Decoder-Alignment Detection; §4.5 Causal Validation; Appendix B.14]
    "For causal experiments requiring cross-family detection, we use decoder-alignment detection: cosine similarity between each decoder vector wdec_i and the model's token embedding matrix E... For a feature i activating with value f_i>0... we replace the residual-stream activation a with a−f_i·wdec_i... and record ablation damage Δlogit_i = log p_ablated(t*_i)−log p_clean(t*_i) for the top activating token t*_i."

    The features used in the causal arm are selected by cosine alignment between wdec and the token embedding of t*; the intervention then subtracts that same wdec from the residual stream while reading t*'s logit. To first order, Δlogit equals −f_i (wdec·U_t*), which is negative for an embedding-aligned decoder (89% logit-lens top-token match), so the BH-significant logit reduction is partly guaranteed by the selection geometry rather than by single-tokenness. The paper's own alignment-matched null is admitted to be only partially constructible: exact matching is impossible at LlamaScope ST alignment levels (median |Δcos| = 0.51), so the necessity effect in those layers is not demonstrably independent of the selection criterion. B.14 concedes 'the necessity effect could in principle be an art

full rationale

The paper is mostly a self-contained empirical study of externally trained SAEs (Neuronpedia checkpoints) with no author self-citations and no fitted parameter later renamed as a prediction. The central family split (GemmaScope/BatchTopK anchored vs LlamaScope locally redundant) is operationalized through same-layer recovery and downstream anchoring statistics, which are not derived from the feature-selection inputs and are partially grounded in same-base Gemma-2-2B/Gemma-3-1B comparisons. However, one load-bearing step is partially circular: the causal-necessity arm selects features by decoder-embedding alignment and then ablates exactly that aligned direction, making a negative target-token logit change a first-order algebraic consequence. The authors attempt an alignment-matched null (B.14), but the matching gap is large where it matters most (LlamaScope layer 1), and they concede the artifact possibility. This keeps the necessity claim from being fully independent, though the anchored/redundant split retains independent content. No other circularity patterns (self-citation chains, imported uniqueness theorems, ansatz smuggled via citation) are present.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

This is a measurement paper, not a derivation, so the ledger consists of measurement and metric choices rather than fitted physics. The two load-bearing hand-set values are the detection operating point (§3.2) and the recovery window (§4.5): the first defines the population for all geometric and prevalence claims, the second defines the anchored-vs-redundant split that is the central finding. Both have only partial robustness checks. The causal intervention (residual-stream linearity) and the logit-lens readout are domain assumptions inherited from the SAE interpretability literature. No invented entities: 'single-token feature', 'anchoring', and 'recovery' are operational definitions and metrics, not new physical or mathematical objects.

free parameters (4)
  • Detection operating point (gap≥0.3, purity≥0.6, complete word, top-k=20) = gap=0.3, purity=0.6, k=20, word-boundary required
    Hand-chosen threshold set that defines the single-token population for all prevalence and geometric claims (§3.2, Table 2). Validated against explanation-match and decoder clustering on the same data, so it is partly fit to the outcome; robustness is shown over a 2× range, but the causal arm uses a different detector, leaving this operating point directly responsible for the geometric and prevalen
  • Recovery window (rank within 2× baseline, floor 5) = 2×, floor 5
    Hand-chosen criterion that defines 'locally redundant' — the split that carries the cross-family headline (§4.5). Never calibrated against pre-ablation rank distributions; a token at baseline rank 1–3 'recovers' if it lands anywhere in ranks 1–5, so the metric can report high recovery while the token's probability mass is destroyed ('al' at L22: Δlogit −0.38 GemmaScope vs −2.72 BatchTopK).
  • Alignment-matching tolerance (B.14) = ±0.02 cosine in-band; nearest-5 otherwise
    Chosen matching windows for the alignment-matched null control. The achieved in-band subset is small (315 controls vs 590 features across 5 layers) and the nearest-null gap is large (median |Δcos| = 0.18 GemmaScope, 0.51 LlamaScope), so the control only partially bounds the selection-geometry component of the necessity effect.
  • Scaling exponent α for prevalence vs model size = −0.51 ± 0.08 (all layers, R²=0.97); −1.33 (Layer 0)
    Power-law fit on n=3 data points (Appendix A.3), labeled 'descriptive only' by the authors. R²=0.97 at n=3 overstates stability; not load-bearing for the main causal claim, included for completeness.
axioms (6)
  • domain assumption Zero-ablation a → a − f_i·w_dec is a valid counterfactual intervention; the feature's residual-stream contribution is linear and independent of other features' contributions at the same position.
    The entire causal arm (§4.5) assumes removing f_i·w_dec isolates the feature's causal role. This is the standard SAE-interpretability intervention model, not proven here; it fails if the feature's contribution is nonlinear in f_i or interacts multiplicatively with other features.
  • domain assumption The same-layer logit lens (w_dec·U_t) faithfully exposes how a layer's features push toward the target token.
    Both anchoring and recovery metrics are defined on same-layer logit-lens readouts (§4.5). Logit-lens reliability is known to be layer-dependent, and the paper cites only the original logit-lens blog post, not the later caveat literature.
  • domain assumption The top-activating token t* is the correct counterfactual target for measuring a feature's necessity.
    Single-token features are evaluated only against their top token; damage to other tokens is not measured (§4.5). If the feature's causal role is to push a family of tokens (its morphological variants), the t*-only readout may under- or over-state necessity.
  • domain assumption Size-matched random controls within a 2× activation-magnitude range, measured at the same positions, adequately separate feature identity from activation magnitude and position frequency.
    B.12 stratifies by magnitude quartile (effect holds in Q1: p<0.0001, r=0.27–0.43), which strengthens this, but controls are not matched on token frequency or position composition beyond the same-text protocol.
  • domain assumption Tokenizer-specific word-boundary rules (BPE space for GPT2/Llama, U+2581 for Gemma) define 'complete word'.
    The 'complete word' criterion (§3.2) depends on tokenizer-specific surface conventions; cross-tokenizer comparisons of single-token status inherit this dependence.
  • standard math Mann-Whitney U with a single global Benjamini-Hochberg correction across all 208 layer tests is a valid control for the multiple-comparison structure.
    Statistical machinery used in §4.5; standard and appropriate, no issue found.

pith-pipeline@v1.3.0-alltime-deepseek · 26178 in / 25576 out tokens · 207936 ms · 2026-08-01T09:57:43.026222+00:00 · methodology

0 comments
read the original abstract

Sparse autoencoder (SAE) features are used to interpret and steer large language models, yet whether a feature's causal role is stable across SAE families remains untested. Single-token features that activate on one vocabulary item provide the diagnostic case where ground truth permits direct comparison. We analyze 3.9M features across six models and three SAE families using zero-ablation at full layer depth. Single-token features cluster 4.7x tighter in decoder space and concentrate in early layers (Layer 0 in GPT2-Small; L0-L4 in Gemma). Ablating them yields Benjamini-Hochberg-significant logit reductions in 178 of 208 full-layer conditions, with depth controlling whether damage cascades downstream or shapes the output directly. Cross-family causal differences exceed within-family scale effects: on the same base model, GemmaScope and BatchTopK features remain causally anchored, while LlamaScope features are locally redundant. The target token's rank recovers to within 2x baseline 96-98% of the time after the same ablation, and a controlled activation-function comparison reverses sign within the same model, leaving training recipe as the residual candidate. Cross-family interpretability claims are therefore sensitive to training methodology, not just activation function or scale.

Figures

Figures reproduced from arXiv: 2607.20596 by Adriano Koshiyama, Ilham Wicaksono, Kleyton da Costa, Rishi Kalra, Seonglae Cho, Zekun Wu.

Figure 1
Figure 1. Figure 1: Single-token prevalence and SAE family effects. (A) Principal component analysis (PCA) scatter of decoder vectors (GPT2-Small L0): green = single-token, gray = polysemantic. (B) Prevalence vs model scale: GemmaScope/res-jb (blue) declines 1.48%→0.14%; LlamaScope (red) near-zero. No fit line (n = 3). (C) GemmaScope shows 46× higher prevalence than LlamaScope at the 8–9B scale (Gemma-2-9B vs Llama-3.1-8B). t… view at source ↗
Figure 2
Figure 2. Figure 2: Three-way validation. (A) Detection schematic: activation metrics and decoder geometry. (B) Gap ratio vs lexical purity separates single-token (green) from polysemantic (gray). (C) Decoder-geometry clustering on GPT2-Small Layer 0: single-token decoder vectors are 7.5× tighter in mean pairwise cosine than a strict polysemantic set (gap < 0.2, purity < 0.4) read at top-10. At the canonical operating point u… view at source ↗
Figure 3
Figure 3. Figure 3: Cross-layer dynamics. (A) Cross-layer similarity matrix (GPT2): L0→L1 shows sharp transition (highlighted), then stabilizes. (B) Feature preservation across layers: both models show low L0→L1 similarity, then recover. (C) Persistence Z-scores: single-token features show 2.7× higher cross-layer persistence (10.7 vs 4.0) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Causal structure across SAE families. (A) Recovery curves for three SAE types on Gemma-2-2B L1: activation function and training recipe each contribute to the recovery gap. (B) Anchoring (solid) and recovery (hatched) across 17 sampled model×layer conditions; per-layer breakdown for the full 208-layer coverage is in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Geometric analysis (mechanism overview). (A) Grassmannian alignment for Gemma-2-2B: low L0→L1 at 0.23, gradual increase through middle layers. (B) Intrinsic dimension: high at L0, compresses through middle layers, and rises again at final layers. (C) Semantic categories: content nouns at 52% and proper nouns at 8.4% dominate. Panel (B) shows the distribution of kurtosis (peakedness) across feature types. S… view at source ↗
Figure 6
Figure 6. Figure 6: Activation shape comparison. (A) Mean activation decay across top-10 tokens: single-token features show sharp drop-off while polysemantic remain flat. (B) Kurtosis distribution: single-token features are sharply peaked (mean 2.65, p < 10−154) vs polysemantic (mean −0.08). (C) Entropy distribution: single-token features show lower entropy (2.23 vs 2.28), confirming concentrated activations. B.2 Token Freque… view at source ↗
Figure 7
Figure 7. Figure 7: Token frequency distribution by feature type. Single-token features show 2.5× enrichment for mid￾frequency tokens (rank 1k–10k), avoiding both high-frequency function words that require compositional encoding and rare tokens that must share capacity. This “Goldilocks zone” pattern suggests single-token features encode tokens that benefit from dedicated, context-independent representations. B.3 Cross-Layer … view at source ↗
Figure 8
Figure 8. Figure 8: Cross-layer persistence by feature type. Single-token features show Z-score 10.7 (vs null distribution of random pairs), 2.7× higher than polysemantic features (Z=4.0, p < 10−73). The dashed line indicates the significance threshold (Z=3). This confirms single-token features serve as stable reference points that persist through the L0→L1 representational shift. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Semantic preservation in persistent features. High-persistence feature pairs (Z > 50) show explanation similarity 0.682, while low-persistence pairs (Z < 5) show only 0.157, a difference of 0.525. This confirms that geometric persistence corresponds to genuine semantic continuity: features that maintain similar directions across layers also maintain similar meanings. Despite different tokenizers, GPT2 and … view at source ↗
Figure 10
Figure 10. Figure 10: Decoder-embedding alignment. Single-token decoder vectors show 1.72× higher cosine similarity with corresponding token embeddings (0.674 vs 0.392, p < 10−42). This mechanistic validation confirms single-token features recover directions close to the original embedding space, functioning as stable reference points for token identity. B.6 Full-Layer Causal Ablation Results Tables 18–19 present layer-wise ne… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 1 canonical work pages

  1. [1]

    Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, and 8 others. 2025. https://transformer-circuits.pub/2025/attribution...

    2025

  2. [2]

    Dana Arad, Aaron Mueller, and Yonatan Belinkov. 2025. https://doi.org/10.18653/v1/2025.emnlp-main.519 SAE s are good for steering -- if you select the right features . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 10241--10259, Suzhou, China. Association for Computational Linguistics

    doi:10.18653/v1/2025.emnlp-main.519 2025

  3. [3]

    Nikita Balagansky, Ian Maksimov, and Daniil Gavrilov. 2025. https://openreview.net/forum?id=MDvecs7EvO Mechanistic permutability: Match features across layers . In The Thirteenth International Conference on Learning Representations

    2025

  4. [4]

    Daniel Balcells, Benjamin Lerner, Michael Oesterle, Ediz Ucar, and Stefan Heimersheim. 2024. https://arxiv.org/abs/2410.08869 Evolution of sae features across layers in llms . Preprint, arXiv:2410.08869

    Pith/arXiv arXiv 2024

  5. [5]

    Joseph Bloom. 2024. Open source sparse autoencoders for all residual stream layers of GPT-2 small. https://www.alignmentforum.org/posts/f9EgfLSurAiqRJySD

    2024

  6. [6]

    Joseph Bloom and Johnny Lin. 2024. Understanding sae features with the logit lens. https://www.lesswrong.com/posts/qykrYY6rXXM7EEs8Q

    2024

  7. [7]

    Dan Braun, Jordan Taylor, Nicholas Goldowsky-Dill, and Lee Sharkey. 2024. https://openreview.net/forum?id=7txPaUpUnc Identifying functionally important features with end-to-end sparse dictionary learning . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  8. [8]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023. Towards monosemanticity: Decomposing language models with d...

    2023

  9. [9]

    Bart Bussmann, Patrick Leask, and Neel Nanda. 2024. https://openreview.net/forum?id=d4dpOCqybL BatchTopK sparse autoencoders . In NeurIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning

    2024

  10. [10]

    Sviatoslav Chalnev, Matthew Siu, and Arthur Conmy. 2024. https://arxiv.org/abs/2411.02193 Improving steering vectors by targeting sparse autoencoder features . Preprint, arXiv:2411.02193

    Pith/arXiv arXiv 2024

  11. [11]

    David Chanin and Joseph Bloom. 2024. https://github.com/jbloomAus/SAELens Saelens: SAE training and analysis library

    2024

  12. [12]

    David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, Satvik Golechha, and Joseph Bloom. 2024. https://arxiv.org/abs/2409.14507 A is for absorption: Studying feature splitting and absorption in sparse autoencoders . Preprint, arXiv:2409.14507

    arXiv 2024

  13. [13]

    Hoagy Cunningham, Aidan Ewart, Logan Riggs Smith, Robert Huben, and Lee Sharkey. 2024. https://openreview.net/forum?id=F76bwRSLeK Sparse autoencoders find highly interpretable features in language models . In The Twelfth International Conference on Learning Representations

    2024

  14. [14]

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. https://transformer-circuits.pub/2022/toy_model/index.html Toy models of superposition . Transfor...

    2022

  15. [15]

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, and 6 others. 2021. https://transformer-circuits.pub/2021/framework/index.html A mat...

    2021

  16. [16]

    Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark

    Joshua Engels, Eric J. Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. 2024. https://arxiv.org/abs/2405.14860 Not all language model features are linear . Preprint, arXiv:2405.14860

    Pith/arXiv arXiv 2024

  17. [17]

    Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2025. https://openreview.net/forum?id=tcsZt9ZNKD Scaling and evaluating TopK sparse autoencoders . In The Thirteenth International Conference on Learning Representations

    2025

  18. [18]

    Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. 2025. https://arxiv.org/abs/2301.04709 Causal abstraction: A theoretical foundation for mechanistic interpretability . Journal of Machine Learning Research, 26

    Pith/arXiv arXiv 2025

  19. [19]

    Gemma Team . 2025. https://arxiv.org/abs/2503.19786 Gemma 3 technical report . Preprint, arXiv:2503.19786

    Pith/arXiv arXiv 2025

  20. [20]

    Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. https://skylion007.github.io/OpenWebTextCorpus/

    2019

  21. [21]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. https://arxiv.org/abs/2407.21783 The llama 3...

    Pith/arXiv arXiv 2024

  22. [22]

    Charles G. Gross. 2002. https://doi.org/10.1177/107385802237175 Genealogy of the ``grandmother cell'' . The Neuroscientist, 8(5):512--518

    work page doi:10.1177/107385802237175 2002

  23. [23]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. https://doi.org/10.1038/s41586-025-09422-z DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement lear...

    doi:10.1038/s41586-025-09422-z 2025

  24. [24]

    Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. 2024. https://arxiv.org/abs/2410.20526 Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders . Preprint, arXiv:2410.20526

    Pith/arXiv arXiv 2024

  25. [25]

    Hindupur, Ekdeep Singh Lubana, Thomas Fel, and Demba Ba

    Sai Sumedh R. Hindupur, Ekdeep Singh Lubana, Thomas Fel, and Demba Ba. 2025. https://arxiv.org/abs/2503.01822 Projecting assumptions: The duality between sparse autoencoders and concept geometry . Preprint, arXiv:2503.01822

    arXiv 2025

  26. [26]

    Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, Matthew Wearden, Arthur Conmy, Samuel Marks, and Neel Nanda. 2025. https://arxiv.org/abs/2503.09532 SAEBench : A comprehensive benchmark for sparse autoencoders in language model interpretability . In Proceedings of ...

    Pith/arXiv arXiv 2025

  27. [27]

    Anton Korznikov, Andrey Galichin, Alexey Dontsov, Oleg Rogov, Ivan Oseledets, and Elena Tutubalina. 2026. https://arxiv.org/abs/2602.14111 Sanity checks for sparse autoencoders: Do SAE s beat random baselines? Preprint, arXiv:2602.14111

    arXiv 2026

  28. [28]

    Michael Lan, Philip Torr, Austin Meek, Ashkan Khakzar, David Krueger, and Fazl Barez. 2024. https://arxiv.org/abs/2410.06981 Sparse autoencoders reveal universal feature spaces across large language models . Preprint, arXiv:2410.06981

    Pith/arXiv arXiv 2024

  29. [29]

    Patrick Leask, Bart Bussmann, Michael T Pearce, Joseph Isaac Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda. 2025. https://openreview.net/forum?id=9ca9eHNrdH Sparse autoencoders do not find canonical units of analysis . In The Thirteenth International Conference on Learning Representations

    2025

  30. [30]

    Elizaveta Levina and Peter Bickel. 2004. https://proceedings.neurips.cc/paper_files/paper/2004/file/74934548253bcab8490ebd74afed7031-Paper.pdf Maximum likelihood estimation of intrinsic dimension . In Advances in Neural Information Processing Systems, volume 17, pages 777--784. MIT Press

    arXiv 2004

  31. [31]

    Michaud, David D

    Yuxiao Li, Eric J. Michaud, David D. Baek, Joshua Engels, Xiaoqing Sun, and Max Tegmark. 2024. https://arxiv.org/abs/2410.19750 The geometry of concepts: Sparse autoencoder feature structure . Preprint, arXiv:2410.19750

    Pith/arXiv arXiv 2024

  32. [32]

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. https://doi.org/10.18653/v1/2024.blackboxnlp-1.19 Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2 . In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreti...

    doi:10.18653/v1/2024.blackboxnlp-1.19 2024

  33. [33]

    Johnny Lin and Joseph Bloom. 2023. Neuronpedia: Interactive reference and tooling for analyzing neural networks. https://neuronpedia.org. Software

    2023

  34. [34]

    Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. 2024. https://transformer-circuits.pub/2024/crosscoders/index.html Sparse crosscoders for cross-layer features and model diffing . Transformer Circuits Thread, Anthropic. Research update, not peer-reviewed

    2024

  35. [35]

    Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Sch \"o lkopf, and Olivier Bachem. 2019. https://proceedings.mlr.press/v97/locatello19a.html Challenging common assumptions in the unsupervised learning of disentangled representations . In Proceedings of the 36th International Conference on Machine Learning, volume 97...

    2019

  36. [36]

    Aleksandar Makelov, George Lange, and Neel Nanda. 2024. https://arxiv.org/abs/2405.08366 Towards principled evaluations of sparse autoencoders for interpretability and control . Preprint, arXiv:2405.08366

    Pith/arXiv arXiv 2024

  37. [37]

    Alireza Makhzani and Brendan Frey. 2014. https://arxiv.org/abs/1312.5663 k-sparse autoencoders . Preprint, arXiv:1312.5663

    Pith/arXiv arXiv 2014

  38. [38]

    Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2025. https://openreview.net/forum?id=I4e82CIDxv Sparse feature circuits: Discovering and editing interpretable causal graphs in language models . In The Thirteenth International Conference on Learning Representations

    2025

  39. [39]

    nostalgebraist. 2020. Interpreting gpt: the logit lens. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru

    2020

  40. [40]

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. https://doi.org/10.23915/distill.00024.001 Zoom in: An introduction to circuits . Distill. Https://distill.pub/2020/circuits/zoom-in

    doi:10.23915/distill.00024.001 2020

  41. [41]

    Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. 2024. https://openreview.net/forum?id=KXuYjuBzKo The geometry of categorical and hierarchical concepts in large language models . In ICML 2024 Workshop on Mechanistic Interpretability

    2024

  42. [42]

    Kiho Park, Yo Joong Choe, and Victor Veitch. 2023. https://openreview.net/forum?id=T0PoOJg8cK The linear representation hypothesis and the geometry of large language models . In Causal Representation Learning Workshop at NeurIPS 2023

    2023

  43. [43]

    Gonçalo Paulo and Nora Belrose. 2025. https://arxiv.org/abs/2501.16615 Sparse autoencoders trained on the same data learn different features . Preprint, arXiv:2501.16615

    Pith/arXiv arXiv 2025

  44. [44]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, and Dario Amodei. 2019. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf Language models are unsupervised multitask learners . Technical report, OpenAI

    2019

  45. [45]

    Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. 2024 a . https://arxiv.org/abs/2404.16014 Improving dictionary learning with gated sparse autoencoders . In Advances in Neural Information Processing Systems

    Pith/arXiv arXiv 2024

  46. [46]

    Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. 2024 b . https://arxiv.org/abs/2407.14435 Jumping ahead: Improving reconstruction fidelity with JumpReLU sparse autoencoders . Preprint, arXiv:2407.14435

    Pith/arXiv arXiv 2024

  47. [47]

    Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. 2025. https://arxiv.org/abs/2503.05613 A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models . Preprint, arXiv:2503.05613

    arXiv 2025

  48. [48]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179 others. 2024. https://arxiv.org/abs/2408.00118 Gemma 2: ...

    Pith/arXiv arXiv 2024

  49. [49]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, and 3 others. 2024. https://transformer-circuits.pub/2024/s...

    2024

  50. [50]

    Constantin Venhoff, Anisoara Calinescu, Philip Torr, and Christian Schroeder de Witt. 2024. https://arxiv.org/abs/2410.07456 Sage: Scalable ground truth evaluations for large sparse autoencoders . Preprint, arXiv:2410.07456

    Pith/arXiv arXiv 2024