Pith. sign in

REVIEW 2 major objections 4 minor 48 references

Where You Measure Decides What You Measure: Position Selection in Ablation-Based SAE Evaluation

T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An unreported choice of which token to measure, not the dictionaries themselves, drives most apparent disagreement in sparse-autoencoder ablation comparisons.

desk verdict A careful, reproducible demonstration that the top-activating-token convention is a hidden, dictionary-chosen measurement facet; the central claim is solid, but the controlled repair would be stronger with a direct check that shared initialisation preserves latent identity post-training. read the letter →

arxiv 2608.13337 v1 pith:64TIWDMN submitted 2026-08-13 cs.LG

Valentin Noël This is my paper · ORCID
classification cs.LG
keywords sparseautoencodersinterpretabilityablation-basedevaluationmeasurementpositionvariancedecompositiongeneralizabilitytheorycausaleffectprotocol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sparse autoencoders are the current tool for naming what a language model computes, and the standard way to test whether a named latent matters is to zero it out and measure how the next-token output changes. The paper shows that this causal number depends on which token the latent is measured at, and that the choice of token is made by the dictionary under evaluation, not by the experimenter. Comparing two dictionaries under the usual top-activation convention therefore compares them at different tokens. In a controlled six-arm experiment holding the latent's identity fixed, the latent-by-dictionary variance that a comparison would attribute to the dictionaries fell from 7.6% and 11.9% to near zero once every dictionary was measured at a shared token. The paper concludes that an ablation-based causal number reported without its position describes the measurement token as much as it describes the latent, and that this problem grows with more evaluation data.

What carries the argument

The central object is the measurement position: the token at which a latent's causal effect is read when it is zero-ablated, conventionally the token where the latent fires hardest. Because that token is computed from the dictionary's own activations, two dictionaries can disagree about a latent even when they encode it nearly identically. The paper's second piece of machinery is a crossed latent-by-arm experimental design with a shared initialisation seed: six autoencoders differ only in fitting choices, so any variance between arms is attributable rather than confounded, and a generalizability-theory decomposition separates variance into latent, arm, latent-by-arm interaction, and position-within-cell components, with the generalizability coefficient $E\rho^2 = v_a/(v_a + v_{ab} + v_e/n)$ tracking how well a latent ranking would survive a different dictionary. Keeping the latent sample fixed and switching from each arm's own positions to a shared position set is the single flag that isolates the convention.

What would settle it

Run the paper's own recipe on any two sparse autoencoders for the same base model: record each dictionary's top-activating token for matched latents. If the two pick the same token for nearly every latent, the claimed position artifact is absent. Then run the six-arm shared-initialisation design at a production-scale token budget; if the latent-by-arm variance does not fall when all arms are measured at a common position, the repair fails. Both can be checked in an afternoon on a single GPU.

Watch

Extended reading notes

Core claim

The paper's central claim is that the effect of zero-ablating a sparse-autoencoder latent is not a property of the latent alone; it is a function of the token at which the intervention is applied, and that token is chosen by the dictionary's own activations. Across released dictionaries for the same base model, matched latents that are nearly identical still pick different top-activating tokens for a large share of pairs, and even the closest near-duplicate pairs agree only about 60% of the time. To separate the convention from the dictionaries, the paper trains six autoencoders from one shared initialisation so that a given latent denotes the same direction in every arm. Measured at each arm's own favourite position, the latent $\times$ arm interaction is 7.6% and 11.9% of variance; measured at one common position, it collapses to 0.0% and 2.4%. The paper also reports that the position a latent is measured at accounts for 67.4% of the total variance, more than latent, arm, and their interaction combined, and that larger evaluation corpora increase disagreement about where to measure rather than reducing it.

Load-bearing premise

The load-bearing premise is that six autoencoders trained from one shared initialisation on 12M tokens make latent i denote the same thing in every arm, so that the controlled comparison can attribute all disagreement to measurement position, and that these 12M-token arms are representative enough of production-scale dictionaries; if shared initialisation does not fix functional identity, the attribution collapses.

Editorial extensions

If this is right

  • A causal number reported without its measurement position cannot be compared across papers; the position is an uncontrolled confound.
  • More evaluation data does not stabilise the convention; across a sixteenfold corpus range, position agreement between arms falls monotonically.
  • Reported differences between dictionaries that disagree about a latent may be dominated by differences in where they chose to measure, not by what the latent does.
  • A one-line evaluation change, measuring every dictionary at a shared token, collapses most of the apparent latent-by-dictionary variance.
  • Within a latent, the choice of firing position explains 67.4% of variance, so scalar causal numbers for a latent are poorly defined; the effect is spread over positions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper only tests two models from one family; if the position artifact persists across architectures, many existing single-latent ablation comparisons in the interpretability literature would need re-reading, not just the five audited papers.
  • A natural extension the paper gestures at is to report a distribution of effects over a latent's firing positions, or an activation-weighted expectation, instead of a scalar; which choice is most stable across dictionaries can be decided empirically on the same crossed design.
  • The audit's finding that none of five papers report position suggests a cheap standardization: requiring the position, and the number of positions per latent, in a protocol would let future causal numbers be pooled across papers.
  • The same controlled design could be applied to rate-based readouts to check whether the position artifact is specific to magnitude readouts or general across evaluation metrics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The manuscript studies a previously unexamined degree of freedom in ablation-based sparse autoencoder evaluation: the token position at which a latent's causal effect is measured. Using released Gemma Scope dictionaries, it shows that matched latents often disagree about the maximally-activating token, and that even near-duplicate decoders agree only about 60% of the time. In a controlled six-arm design trained from a shared initialization, the latent×arm variance component falls from 7.6%/11.9% to 0.0%/2.4% when all arms are measured at a common position rather than each arm's own top position. The paper further shows that more evaluation data does not reduce the disagreement, documents two other reporting conventions that can flip sign, and proposes a protocol for reporting ablation-based causal numbers.

Significance. The strength of the paper is its two-pronged design: the premise that production dictionaries select different positions is tested directly on released dictionaries, and the controlled repair is compared under paired bootstrap CIs with an alternative shared-position rule, a pre-registered decision rule, and an explicit discussion of filter bias and clipping. The authors are also admirably explicit that the Eρ² gain is an arithmetic consequence of the variance-component collapse, not independent evidence. If the controlled attribution to position holds, this identifies a convention that is invisible in current practice yet moves the answer more than the dictionary choice, which would make the paper a useful methodological contribution to SAE evaluation and to interpretability more generally.

major comments (2)
  1. [§3–§4, Table 1] The central attribution of the collapse to position rests on the assertion that shared initialisation makes latent i denote the same thing in every arm, but the paper never measures this after training. The six arms differ in sparsity (k=41 vs 82), learning rate (10× lower), and decoder freezing, and the paper itself cites Leask et al. (2025) and Chanin et al. (2024) showing that such fitting choices split and absorb features. Shared initialisation fixes only the starting decoder direction; it does not prevent trained decoder columns from rotating or drifting. Without a post-training identity check (for example, median decoder cosine between matched columns, activation overlap at shared positions, or matched-feature agreement), the latent×arm component in Table 1 mixes identity drift with position choice, and the 7.6%→0.0% / 11.9%→2.4% collapse cannot be attributed to position alone. Section 7 calls this structural, but the manuscript needs a direct empirical check before the headline attribution is supported.
  2. [§3] The claim that the six shared-initialisation arms are 'the most similar dictionaries a practitioner would ever compare' and hence that the measured instability is a lower bound is not tested. It presupposes that fitting choices (sparsity, learning rate, decoder freezing) induce less feature drift than seed variation, which is exactly what the paper's own cited literature (Paulo and Belrose, 2025) leaves open. This claim should either be verified with a cross-seed comparison or removed.
minor comments (4)
  1. [Figure 2] In panel (b), the y-axis is labelled '% same token', but the panel plots the latent×arm variance component; the label appears to be a copy-paste error from panel (a) and should be corrected.
  2. [Table 2] The caption says that '—' means the sections reviewed do not state the convention, but the symbols in the table body are not rendered consistently; please define the exact mark used in the table itself.
  3. [§2] The reproducibility discrepancy with Cho et al. (0.41–0.44× their prevalence at four times the apparent corpus size) is reported without the detector configuration or the comparison protocol; if it is not moved to an appendix with full details, it reads as an unverifiable aside.
  4. [Abstract] The statement that 'the problem grows with scale' should carry the scope qualifier that the growth is measured on two Gemma models at evaluation corpora of 96–1536 sequences, not established for arbitrary scales.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the per-arm vs shared contrast is an experimental manipulation, and the paper explicitly treats the Eρ2 gain as arithmetic.

full rationale

The paper's central comparison is an experimental manipulation rather than a derivation from its own assumptions. The latent×arm variance component is measured twice on identical latents, identical arms, and identical replication, with the only difference being whether each dictionary selects its own measurement position or all dictionaries are measured at a common position; no fitted parameter is renamed as a prediction. The paper explicitly states that the Eρ2 gain is an arithmetic consequence of the interaction collapse and is not independent evidence, which forecloses the main circularity concern. The shared-initialisation premise ('latent i denotes the same initial direction in every arm') is an assumption about latent identity that the paper flags as structural in Section 7; this is a correctness risk or external-validity limitation, not a reduction by construction. The released Gemma Scope analysis provides independent, externally sourced evidence for the phenomenon, and the shared-position rule is tested against a second arm-symmetric rule with pre-registered outcomes. No self-citation chain, uniqueness theorem, or renamed empirical pattern carries the load-bearing argument. The acknowledged limitations—training scale, retained subpopulation, and unmeasured identity drift—are substantive scientific concerns, but they do not make any quoted equation or protocol step equivalent to its inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters: the six arm variants are treatment conditions, not fitted quantities, and the variance components are estimated, not tuned. The axioms are background assumptions about SAE identity and measurement conventions; none is ad hoc to the paper in a way that would force the result. No invented entities.

assumptions (4)
  • domain assumption Six TopK SAEs sharing initialisation seed 0 make latent i denote the same initial direction in every arm.
    Invoked in Section 4 design; the entire crossed variance decomposition depends on latent identity being fixed across arms.
  • domain assumption Mutual-nearest-neighbour decoder cosine matching identifies the same latent across released dictionaries.
    Appendix B; used to compute position agreement across Gemma Scope dictionaries.
  • standard math The measurement position is a sampled facet whose variance can be decomposed by method-of-moments estimators clipped at zero.
    Equation (2) and Section 3; the estimator is standard generalizability theory.
  • domain assumption The top-activating token is the default convention for selecting measurement positions in the published literature.
    Section 1 and the audit in Table 2; the paper's framing depends on this being the common practice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Where You Measure Decides What You Measure: Position Selection in Ablation-Based SAE Evaluation." pith.science (2026). https://pith.science/paper/64TIWDMN

@misc{pith2026260813337,
  author       = {Pith},
  title        = {Pith review of: Where You Measure Decides What You Measure: Position Selection in Ablation-Based SAE Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/64TIWDMN}},
  note         = {Machine review of arXiv:2608.13337}
}
read the original abstract

Sparse autoencoders are meant to name the things a language model computes, and the usual way to check that a latent matters is to switch it off and see what changes. But a latent fires at many tokens, and the effect has to be measured at one of them. The convention is to measure where the latent fires hardest. That choice is almost never reported, and it is not made by the experimenter: it is made by the dictionary under evaluation. Change the dictionary and the measurement moves to a different token. We show this is not a detail. Take two sparse autoencoders released by Google for the same model and match their latents by decoder similarity: even among the pairs the two dictionaries encode almost identically, they pick different tokens for a large share of them. Two dictionaries compared under the usual protocol are therefore very often compared at different places. To separate the convention from the dictionaries we train six autoencoders from one initialisation, differing only in fitting choices, so that a latent means the same thing in each. Most of the variance such a comparison reads as "these dictionaries disagree about this latent" turns out to be the position instead: it falls from 7.6% and 11.9% of variance to near zero once every dictionary is measured at the same token. More evaluation data does not rescue it. Across a sixteenfold range of corpus sizes the dictionaries agree less about where to measure, not more, so the problem grows with scale. The correction is one line of evaluation code. We give the protocol an ablation-based causal number must report to be comparable across papers, and an audit of five published papers against it. In short: a causal number reported without its position describes the token it was taken at as much as the latent it was taken from.

Figures

Figures reproduced from arXiv: 2608.13337 by the authors.

Figure 1
Figure 1. (a) Distance between the tokens two dictionaries pick for the same latent, over all [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. (a) Position agreement between every pair of [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Agreement between two released Gemma Scope dictionaries against how alike the matched [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 8 canonical work pages

  1. [1]

    Harald Baayen, Douglas J

    R. Harald Baayen, Douglas J. Davidson, and Douglas M. Bates. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59 0 (4): 0 390--412, 2008. doi:10.1016/j.jml.2007.12.005

    doi:10.1016/j.jml.2007.12.005 2008

  2. [2]

    An interpretability illusion for bert

    Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wattenberg. An interpretability illusion for bert. arXiv preprint arXiv:2104.07143, 2021. doi:10.48550/arXiv.2104.07143. URL https://arxiv.org/abs/2104.07143

    arXiv doi:10.48550/arxiv.2104.07143 2021

  3. [3]

    Accounting for variance in machine learning benchmarks

    Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Naz Sepah, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Dmitriy Serdyuk, Tal Arbel, Chris Pal, Ga \"e l Varoquaux, and Pascal Vincent. Accounting for variance in machine learning benchmarks. In Proceedings of Machine ...

    arXiv doi:10.48550/arxiv.2103.03098 2021

  4. [4]

    Identifying functionally important features with end-to-end sparse dictionary learning

    Dan Braun, Jordan Taylor, Nicholas Goldowsky-Dill, and Lee Sharkey. Identifying functionally important features with end-to-end sparse dictionary learning. arXiv preprint arXiv:2405.12241, 2024. doi:10.48550/arXiv.2405.12241. URL https://arxiv.org/abs/2405.12241

    arXiv doi:10.48550/arxiv.2405.12241 2024

  5. [5]

    Robert L. Brennan. Generalizability Theory. Statistics for Social and Behavioral Sciences. Springer, New York, 2001. doi:10.1007/978-1-4757-3456-0

    doi:10.1007/978-1-4757-3456-0 2001

  6. [6]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. Towa...

    work page 2023

  7. [7]

    Batchtopk sparse autoencoders

    Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. arXiv preprint arXiv:2412.06410, 2024. doi:10.48550/arXiv.2412.06410. URL https://arxiv.org/abs/2412.06410

    arXiv doi:10.48550/arxiv.2412.06410 2024

  8. [8]

    With little power comes great responsibility

    Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. With little power comes great responsibility. arXiv preprint arXiv:2010.06595, 2020. doi:10.48550/arXiv.2010.06595. URL https://arxiv.org/abs/2010.06595

    arXiv doi:10.48550/arxiv.2010.06595 2010

Show all 48 references
  1. [9]

    Improving steering vectors by targeting sparse autoencoder features

    Sviatoslav Chalnev, Matthew Siu, and Arthur Conmy. Improving steering vectors by targeting sparse autoencoder features. arXiv preprint arXiv:2411.02193, 2024. doi:10.48550/arXiv.2411.02193. URL https://arxiv.org/abs/2411.02193

    2024 doi arXiv

  2. [10]

    Are sparse autoencoder benchmarks reliable? 2026

    David Chanin. Are sparse autoencoder benchmarks reliable? 2026. doi:10.48550/arXiv.2605.18229. URL https://arxiv.org/abs/2605.18229

    2026 doi arXiv

  3. [11]

    A is for absorption: Studying feature splitting and absorption in sparse autoencoders

    David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, Satvik Golechha, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. arXiv preprint arXiv:2409.14507, 2024. doi:10.48550/arXiv.2409.14507. URL https://arxiv.or...

    2024 doi

  4. [12]

    Evaluating open-source sparse autoencoders on disentangling factual knowledge in gpt-2 small

    Maheep Chaudhary and Atticus Geiger. Evaluating open-source sparse autoencoders on disentangling factual knowledge in gpt-2 small. arXiv preprint arXiv:2409.04478, 2024. doi:10.48550/arXiv.2409.04478. URL https://arxiv.org/abs/2409.04478

    2024 doi arXiv

  5. [13]

    Are single-token sparse autoencoder features causally necessary? L ayer-depth and SAE -family effects

    Seonglae Cho, Zekun Wu, Kleyton Da Costa, Rishi Kalra, Ilham Wicaksono, and Adriano Koshiyama. Are single-token sparse autoencoder features causally necessary? L ayer-depth and SAE -family effects. 2026. doi:10.48550/arXiv.2607.20596. URL https://arxiv.org/abs/2607.20596

    2026 doi arXiv

  6. [14]

    Herbert H. Clark. The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12 0 (4): 0 335--359, 1973. doi:10.1016/s0022-5371(73)80014-3

    1973 doi

  7. [15]

    Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso

    Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. arXiv preprint arXiv:2304.14997, 2023. doi:10.48550/arXiv.2304.14997. URL https://arxiv.org/abs/2304.14997

    2023 doi arXiv

  8. [16]

    Cronbach, Goldine C

    Lee J. Cronbach, Goldine C. Gleser, Harinder Nanda, and Nageswari Rajaratnam. The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. John Wiley and Sons, New York, 1972

    1972

  9. [17]

    Sparse autoencoders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. 2023. doi:10.48550/arXiv.2309.08600. URL https://arxiv.org/abs/2309.08600

    2023 doi arXiv

  10. [18]

    Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat...

    2019 arXiv

  11. [19]

    Transcoders find interpretable llm feature circuits

    Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable llm feature circuits. arXiv preprint arXiv:2406.11944, 2024. doi:10.48550/arXiv.2406.11944. URL https://arxiv.org/abs/2406.11944

    2024 doi arXiv

  12. [20]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposi...

    2022 doi arXiv

  13. [21]

    Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark

    Joshua Engels, Eric J. Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are one-dimensionally linear. arXiv preprint arXiv:2405.14860, 2024. doi:10.48550/arXiv.2405.14860. URL https://arxiv.org/abs/2405.14860

    2024 doi arXiv

  14. [22]

    Applying sparse autoencoders to unlearn knowledge in language models

    Eoin Farrell, Yeu-Tong Lau, and Arthur Conmy. Applying sparse autoencoders to unlearn knowledge in language models. arXiv preprint arXiv:2410.19278, 2024. doi:10.48550/arXiv.2410.19278. URL https://arxiv.org/abs/2410.19278

    2024 doi arXiv

  15. [23]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupr \'e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. 2024. doi:10.48550/arXiv.2406.04093. URL https://arxiv.org/abs/2406.04093

    2024 doi arXiv

  16. [24]

    Automated interpretability metrics do not distinguish trained and random transformers

    Thomas Heap, Tim Lawson, Lucy Farnik, and Laurence Aitchison. Automated interpretability metrics do not distinguish trained and random transformers. 2025. doi:10.48550/arXiv.2501.17727. URL https://arxiv.org/abs/2501.17727

    2025 doi

  17. [25]

    Deep reinforcement learning that matters

    Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. arXiv preprint arXiv:1709.06560, 2017. doi:10.48550/arXiv.1709.06560. URL https://arxiv.org/abs/1709.06560

    2017 doi arXiv

  18. [26]

    RAVEL : Evaluating interpretability methods on disentangling language model representations

    Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. RAVEL : Evaluating interpretability methods on disentangling language model representations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    2024 arXiv

  19. [27]

    Judd, Jacob Westfall, and David A

    Charles M. Judd, Jacob Westfall, and David A. Kenny. Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of Personality and Social Psychology, 103 0 (1): 0 54--69, 2012. doi:10.1037/a0028347

    2012 doi

  20. [28]

    Are sparse autoencoders useful? a case study in sparse probing

    Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, and Neel Nanda. Are sparse autoencoders useful? a case study in sparse probing. arXiv preprint arXiv:2502.16681, 2025. doi:10.48550/arXiv.2502.16681. URL https://arxiv.org/abs/2502.16681

    2025 doi arXiv

  21. [29]

    Measuring progress in dictionary learning for language model interpretability with board game models

    Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Smith, Claudio Mayrink Verdun, David Bau, and Samuel Marks. Measuring progress in dictionary learning for language model interpretability with board game models. arXiv preprint arXiv:2408.00113, 20...

    2024 doi arXiv

  22. [30]

    SAEBench : A comprehensive benchmark for sparse autoencoders in language model interpretability

    Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, Demian Till, Matthew Wearden, Arthur Conmy, Samuel Marks, and Neel Nanda. SAEBench : A comprehensive benchmark for sparse autoencoders i...

    2025 doi arXiv

  23. [31]

    Interpreting attention layer outputs with sparse autoencoders

    Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. Interpreting attention layer outputs with sparse autoencoders. arXiv preprint arXiv:2406.17759, 2024. doi:10.48550/arXiv.2406.17759. URL https://arxiv.org/abs/2406.17759

    2024 doi arXiv

  24. [32]

    Sanity checks for sparse autoencoders: Do SAEs beat random baselines? 2026

    Anton Korznikov, Andrey Galichin, Alexey Dontsov, Oleg Rogov, Ivan Oseledets, and Elena Tutubalina. Sanity checks for sparse autoencoders: Do SAEs beat random baselines? 2026. doi:10.48550/arXiv.2602.14111. URL https://arxiv.org/abs/2602.14111

    2026 doi

  25. [33]

    Atp*: An efficient and scalable method for localizing llm behaviour to components

    János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. Atp*: An efficient and scalable method for localizing llm behaviour to components. arXiv preprint arXiv:2403.00745, 2024. doi:10.48550/arXiv.2403.00745. URL https://arxiv.org/abs/2403.00745

    2024 doi arXiv

  26. [34]

    Sparse autoencoders do not find canonical units of analysis

    Patrick Leask, Bart Bussmann, Michael Pearce, Joseph Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda. Sparse autoencoders do not find canonical units of analysis. arXiv preprint arXiv:2502.04878, 2025. doi:10.48550/arXiv.2502.04878. URL https://arxiv.org/abs...

    2025 doi arXiv

  27. [35]

    Gemma Scope : Open sparse autoencoders everywhere all at once on Gemma 2

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J \'a nos Kram \'a r, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma Scope : Open sparse autoencoders everywhere all at once on Gemma 2. In Proceedings of the 7th BlackboxNLP Wo...

    2024 arXiv

  28. [36]

    Towards principled evaluations of sparse autoencoders for interpretability and control

    Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. 2024. doi:10.48550/arXiv.2405.08366. URL https://arxiv.org/abs/2405.08366

    2024 doi arXiv

  29. [37]

    Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller

    Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In International Conference on Learning Representations (ICLR), 2025. doi:10.48550/arXiv.2...

    2025 doi arXiv

  30. [38]

    Locating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. arXiv preprint arXiv:2202.05262, 2022. doi:10.48550/arXiv.2202.05262. URL https://arxiv.org/abs/2202.05262

    2022 doi arXiv

  31. [39]

    The linear representation hypothesis and the geometry of large language models

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658, 2023. doi:10.48550/arXiv.2311.03658. URL https://arxiv.org/abs/2311.03658

    2023 doi arXiv

  32. [40]

    Sparse autoencoders trained on the same data learn different features

    Gon c alo Paulo and Nora Belrose. Sparse autoencoders trained on the same data learn different features. 2025. doi:10.48550/arXiv.2501.16615. URL https://arxiv.org/abs/2501.16615

    2025 doi arXiv

  33. [41]

    Improving dictionary learning with gated sparse autoencoders

    Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404.16014, 2024 a . doi:10.48550/arXiv.2404.16014. URL https://arxiv....

    2024 doi arXiv

  34. [42]

    Jumping ahead: Improving reconstruction fidelity with JumpReLU sparse autoencoders

    Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, J \'a nos Kram \'a r, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with JumpReLU sparse autoencoders. 2024 b . doi:10.48550/arXiv.2407.14435. URL https://arxiv.org/abs/2407.14435

    2024 doi arXiv

  35. [43]

    Shavelson and Noreen M

    Richard J. Shavelson and Noreen M. Webb. Generalizability Theory: A Primer. Sage Publications, Newbury Park, CA, 1991

    1991

  36. [44]

    Attribution patching outperforms automated circuit discovery

    Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. arXiv preprint arXiv:2310.10348, 2023. doi:10.48550/arXiv.2310.10348. URL https://arxiv.org/abs/2310.10348

    2023 doi arXiv

  37. [45]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, ...

    2024

  38. [46]

    Vazquez, Ulisse Mini, and Monte MacDiarmid

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248, 2023. doi:10.48550/arXiv.2308.10248. URL https://arxiv.org/abs/2308.10248

    2023 doi arXiv

  39. [47]

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022. doi:10.48550/arXiv.2211.00593. URL https://arxiv.org/abs/2211.00593

    2022 doi arXiv

  40. [48]

    Towards best practices of activation patching in language models: Metrics and methods

    Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. arXiv preprint arXiv:2309.16042, 2023. doi:10.48550/arXiv.2309.16042. URL https://arxiv.org/abs/2309.16042

    2023 doi arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.