Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A word-sense benchmark for sparse autoencoders, PS-Eval, shows that models optimized for reconstruction and sparsity separate word meanings worse than plain ReLU — the field's headline metrics diverge from interpretability.

desk verdict PS-Eval is a genuinely useful, honestly-reported benchmark for SAE semantic quality; the headline activation-function ranking rests on an unproven argmax proxy, but the paper's transparency and extra baselines make it worth engaging. read the letter →

arxiv 2501.06254 v2 pith:FWLCQOHO submitted 2025-01-09 cs.CL cs.AIcs.LG

Gouki Minegishi , Hiroki Furuta , Yusuke Iwasawa , Yutaka Matsuo This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords sparseautoencodersmechanisticinterpretabilitypolysemymonosemanticfeaturesevaluationmetricsWord-in-ContextdatasetMSE-L0Paretofrontierlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the standard metrics for sparse autoencoders — reconstruction error (MSE) and sparsity ($L_0$) — do not track whether an SAE is actually doing its advertised job: carving polysemantic LLM activations into monosemantic, single-meaning features. To make that case it introduces PS-Eval, a benchmark built from the Word-in-Context dataset, which checks whether the maximally activated SAE feature for a target word stays the same when the word has the same meaning and changes when its meaning differs. The headline finding is that SAEs tuned to push the MSE-$L_0$ frontier do not improve — and on F1 actually worsen — semantic separation, with the activation-function ranking ReLU > JumpReLU > TopK. A secondary claim locates where polysemy resolution happens in the network: specificity rises in deeper layers and is highest in the attention submodule, even where MSE and $L_0$ look worst. If these results hold, the community's optimization loop has been pointed at a proxy that can diverge from the actual goal of interpretability.

What carries the argument

The load-bearing object is PS-Eval itself: a dataset plus a confusion-matrix protocol. The dataset is a filtered slice of the Word-in-Context (WiC) dataset, keeping only words that tokenize as a single token in GPT-2 small, yielding 1,112 samples split evenly between poly-contexts, where the target word means different things in the two sentences, and mono-contexts, where it means the same thing. The protocol computes the maximally activated SAE feature for the target token in each context and classifies each pair: same top feature under the same meaning (True Positive), different top feature under the same meaning (False Negative), same top feature under different meanings (False Positive), and different top features under different meanings (True Negative). From that 2×2 table the paper computes accuracy, precision, recall, specificity, and F1, and it validates the max-activation convention with a logit-lens check — decoding the top feature through the unembedding matrix yields meaning-consistent vocabulary in the examples shown. The same confusion matrix is then rerun as a network probe, reporting specificity by layer depth and by Transformer component (residual, MLP, attention).

What would settle it

Re-run PS-Eval on the same 1,112 Word-in-Context-derived samples using a distributional similarity over all nonzero SAE features, or a linear probe trained on the full feature vector, instead of the single maximum activation. If TopK or JumpReLU overtakes ReLU on F1 under that scoring, the paper's headline ranking is an artifact of the max-activation convention rather than a property of monosemantic content. A complementary check would apply the logit-lens decoding to every sample and count how often the top feature's strongest decoded tokens relate to the gold meaning; a low fraction would mean the confusion matrix measures contextual pattern differences, not word meanings.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that monosemanticity is a measurable property of SAE features, testable with polysemous words, and that it moves independently of the MSE-$L_0$ trade-off. PS-Eval takes a target word that appears in two contexts — the same meaning in mono-contexts, different meanings in poly-contexts — and compares the single most strongly activated SAE feature for the target token in each context. The same top feature under the same meaning is a True Positive; different top features under different meanings are a True Negative; the two mismatches complete the confusion matrix, and from it the paper derives accuracy, precision, recall, specificity, and F1. Three results follow: expanding the latent dimension improves semantic separation but saturates around an expand ratio of 64; the activation functions built to win the MSE-$L_0$ frontier, TopK and JumpReLU, score below plain ReLU on F1; and both deeper layers and the attention submodule raise specificity even as MSE and $L_0$ worsen. A logit-lens check is offered as evidence that the top features really carry the context-specific senses — 'space' as the universe decodes toward flight and gravity, while 'space' as a gap decodes toward layout and occupancy.

Load-bearing premise

PS-Eval assumes that the single most strongly activated SAE feature for a target word in a given context encodes that word's meaning in that context, so comparing the top features across two contexts reveals whether the two senses have been separated; the paper itself flags in its limitations section that this alignment is not guaranteed.

Editorial extensions

If this is right

  • SAE development should be checked against a semantics-focused metric like PS-Eval alongside MSE and $L_0$, because the paper shows the Pareto-frontier winners (TopK, JumpReLU) can be the semantic losers.
  • Scaling helps meaning separation only up to a point: accuracy saturates near an expand ratio of 64, so indiscriminate dictionary growth has diminishing returns for interpretability.
  • Layer and component choice matters more than reconstruction numbers suggest: deeper layers raise specificity even as MSE and $L_0$ worsen, and the attention submodule separates senses best on specificity despite a lower F1.
  • Because PS-Eval is model-independent, the same yardstick can rank SAEs trained on different base LLMs, as the paper demonstrates on GPT-2 small, Pythia 70M, and Gemma2-2B.
  • The logit-lens results imply that at least some individual SAE features each carry a single sense of a target word, which is direct evidence for the monosemanticity claim at the feature level.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the reported plateau near an expand ratio of 64 implies an efficiency ceiling for dictionary scaling that scaling-law arguments do not model; a testable consequence is that PS-Eval accuracy stops improving with dictionary size even while the MSE-$L_0$ curve keeps improving.
  • The paper's findings point toward a training-side fix it does not propose: a semantic term in the SAE loss (for instance, contrastive separation of word senses) would make PS-Eval a design target rather than a post-hoc check.
  • The attention-specificity result hints at a mechanistic reading the authors leave implicit: polysemy may be disambiguated in the attention stream and propagated through MLP and residual paths, so SAEs trained on attention outputs would be the natural place to find sense-specific features.
  • The logit-lens validation covers only a handful of words ('space', 'save', 'ball'); scaling it to all 1,112 samples with an automated judge would convert a qualitative justification into a quantitative one and would test how often the max feature genuinely carries the gold sense rather than a correlated contextual cue.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PS-Eval, a benchmark built from the WiC dataset, which evaluates sparse autoencoders (SAEs) by comparing the maximally activated SAE feature for a target word in two contexts and forming a confusion matrix over whether the same latent index fires. The authors report that ReLU outperforms JumpReLU and TopK on PS-Eval F1, use this to argue that optimizing the MSE-L0 Pareto frontier does not necessarily improve the extraction of monosemantic features, and further claim that deeper layers and the Attention module contribute to distinguishing polysemy, based on Specificity. The manuscript includes a new dataset and metric, code and data release, and baselines including a random SAE and a dense SAE.

Significance. If the central negative result holds, PS-Eval would be a useful complement to MSE and L0 metrics for SAE development, and the paper's release of code, data, and baselines is a concrete contribution. The random-SAE and dense-SAE controls show awareness of metric pathologies. However, the main result depends on the unvalidated assumption that the argmax feature is the meaning-bearing feature, and the layer/attention conclusions rely on Specificity despite the paper's own warning that Specificity can be trivially high. The significance is therefore conditional on additional validation and re-analysis.

major comments (4)
  1. [Section 4.2, Appendix D, Section 7, Figures 5-6] The paper explicitly warns in Section 4.2 that Specificity 'may reach an unfairly high value by definition' and recommends holistic metrics, and Appendix D shows that a random SAE attains Specificity ≈ 0.99999999668. Yet Section 7 and the abstract use Specificity alone to conclude that deeper layers and the Attention module contribute to distinguishing polysemy. These conclusions are unsupported: high Specificity can simply reflect low chance of identical argmax features rather than semantic discrimination. Please re-analyze the layer and component comparisons with F1, balanced accuracy, or a direct comparison against the random and dense baselines.
  2. [Section 4.2, Section 5, Section 8, Figure 4] PS-Eval classifies each sample by whether the maximally activated SAE feature index is identical across the two contexts, and all headline comparisons (including the ReLU > JumpReLU > TopK ranking) rest on this identity being a faithful proxy for the meaning-bearing monosemantic feature. Section 8 concedes that 'it is not guaranteed that the feature with the maximum activation always aligns with the intended word meanings,' and Section 5 validates this assumption on only three words using qualitative top-7 logits. Because TopK and JumpReLU systematically change the activation distribution, their lower F1 could be an artifact of argmax instability rather than a genuine failure to extract monosemantic features. Please provide quantitative validation on a larger sample (e.g., automated or logit-lens-based interpretability labels for a random subset of PS-Eval instances) or a distribution-based variant of the metric (e.g., overlap or KL divergence of the activation vector) to show the ranking persists.
  3. [Section 3.2, Section 6.3, Appendix B] The default training setting uses in-domain data, namely WiC, which is the same source as the PS-Eval evaluation set, while Appendix B shows that in-domain versus open-domain training changes the precision-recall trade-off substantially. The activation-function comparison in Figure 4 may therefore be affected by differential overfitting to WiC across ReLU, JumpReLU, and TopK. Please report the activation-function ranking for SAEs trained on open-domain data, or at minimum discuss this confound explicitly in the interpretation of the ranking.
  4. [Figure 4, Table 8] The activation-function comparison is not matched for sparsity or reconstruction quality: Table 8 reports L0 = 531 for ReLU, 383 for TopK(k=384), 190 for TopK(k=192), and 829 for JumpReLU(jump=0.0001), while the paper itself shows that recall and F1 vary with k within TopK. The observed ranking may thus reflect different operating points on the sparsity spectrum rather than the activation function per se. Please include comparisons at matched L0 (or a joint analysis controlling for L0), and report confidence intervals in the main figures rather than only in the appendix version with error bars.
minor comments (5)
  1. [Appendix G, Figure 14 caption] The caption of Figure 14 states that 'JumpReLU outperforms both TopK and standard ReLU in terms of F1 score, Precision, and Recall,' which directly contradicts the main-text result in Section 6.3 and Figure 4. This appears to be a copy error and must be corrected.
  2. [Section 4.1, Appendix K] The paper calls PS-Eval model-independent, but the dataset construction filters instances whose target word is tokenized as a single token in GPT-2 small; the open-SAE evaluation on Pythia and Gemma should clarify how multi-token target words in those tokenizers are handled, otherwise the claim of model independence is overstated.
  3. [Appendix D] The text states 'expand ratio × dmodel = 32 × 768 = 23576,' but the correct product is 24576; the following probability calculation uses 24576, so this is a typo in the displayed equation.
  4. [Section 6.2] There is a typo in 'monosemantic fetures' that should be 'features,' and the claim that accuracy saturates around an expand ratio of 64 is only partially supported by the somewhat non-monotonic layer-6 results in Table 10 (e.g., accuracy drops below chance at R=8).
  5. [Figure 5] The caption repeats 'better better' and describes trends starting at 'layer 6 onwards' without specifying which points are statistically distinguishable; please clarify the layer labels and add error bars or a statistical comparison across layers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PS-Eval is an externally grounded evaluation whose metrics do not reduce to the SAE training objective or to a self-citation chain.

full rationale

The paper's central claim, that MSE-L0-optimized SAEs do not necessarily extract monosemantic features, is supported by PS-Eval, a confusion-matrix evaluation built on the external WiC benchmark. The metric compares whether the maximally activated SAE feature for a target token is shared across mono-contexts and differs across poly-contexts; this is not a quantity used in the SAE training loss, nor is it derived from the models' fitted parameters by construction. The paper is transparent that the argmax proxy is an assumption, explicitly conceding in Section 8 that 'it is not guaranteed that the feature with the maximum activation always aligns with the intended word meanings,' which is a validity threat rather than a circular derivation. In-domain training on WiC, the evaluation source, could inflate results, but this is a data-contamination concern, not a circular step, and Appendix B provides open-domain results that partially mitigate it. The logit-lens validation in Section 5 is an independent qualitative check using the decoder and unembedding, not a reintroduction of the evaluation labels. The only self-citations, Furuta et al. (2024) and Minegishi et al. (2024), appear in Appendix P as examples of grokking circuit analysis and are not load-bearing for the paper's claims. The paper also self-corrects on Specificity by noting that a random SAE achieves near-maximal Specificity by definition and therefore recommends balanced metrics such as F1. No equation in the paper reduces PS-Eval to the training objective, and no load-bearing argument depends on an unverified self-citation chain.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

PS-Eval introduces a new evaluation protocol rather than a theoretical derivation; the central claims rest on several domain assumptions about activation-feature correspondence, dataset label quality, and prompt format. The metric itself is external to the SAE training objective, so circularity burden is low, but the validity of the metric depends on these assumptions.

free parameters (5)
  • Sparsity regularization coefficient λ = 0.05
    Default L1 penalty weight used to train SAEs; affects which point on the MSE-L0 frontier is evaluated. Chosen following Templeton et al. 2024.
  • Expand ratio R = 32
    Default latent dimension multiplier (32 x 768) for main experiments; conclusions about activation functions and layer depth use this setting.
  • Layer index l = 4
    Default layer of GPT-2 small for activation extraction; Appendix O tests layer 6 but main claims use layer 4.
  • TopK k values = 384, 192, 96
    Sparsity targets for TopK comparisons, chosen following Gao et al. 2024; the ranking ReLU > TopK may change with k, and k also changes L0.
  • JumpReLU thresholds = 0.001, 0.0001
    Threshold values for JumpReLU and its STE variant, chosen following Rajamanoharan et al. 2024; these affect the sparsity and F1 outcomes.
assumptions (4)
  • domain assumption The maximum-activated SAE feature for a target token encodes its context-specific meaning
    Central to the PS-Eval confusion matrix definition (Section 4.2); the paper itself notes in Section 8 that the max feature may not align with intended meaning.
  • domain assumption WiC labels (same/different meaning) are correct for the selected 1112 single-token instances
    PS-Eval inherits WiC labels as ground truth (Section 4.1).
  • domain assumption The prompt '{context}. The {target word} means' elicits activations localized to the target word's meaning
    Activation extraction relies on this format (Section 4.2), modeled after Wang et al. 2022.
  • domain assumption Logit lens projections of SAE features reflect semantic content
    Used in Section 5 to justify that SAE features encode word meanings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words." pith.science (2026). https://pith.science/paper/FWLCQOHO

@misc{pith2026250106254,
  author       = {Pith},
  title        = {Pith review of: Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FWLCQOHO}},
  note         = {Machine review of arXiv:2501.06254}
}
read the original abstract

Sparse autoencoders (SAEs) have gained a lot of attention as a promising tool to improve the interpretability of large language models (LLMs) by mapping the complex superposition of polysemantic neurons into monosemantic features and composing a sparse dictionary of words. However, traditional performance metrics like Mean Squared Error and L0 sparsity ignore the evaluation of the semantic representational power of SAEs -- whether they can acquire interpretable monosemantic features while preserving the semantic relationship of words. For instance, it is not obvious whether a learned sparse feature could distinguish different meanings in one word. In this paper, we propose a suite of evaluations for SAEs to analyze the quality of monosemantic features by focusing on polysemous words. Our findings reveal that SAEs developed to improve the MSE-L0 Pareto frontier may confuse interpretability, which does not necessarily enhance the extraction of monosemantic features. The analysis of SAEs with polysemous words can also figure out the internal mechanism of LLMs; deeper layers and the Attention module contribute to distinguishing polysemy in a word. Our semantics focused evaluation offers new insights into the polysemy and the existing SAE objective and contributes to the development of more practical SAEs.

Figures

Figures reproduced from arXiv: 2501.06254 by the authors.

Figure 1
Figure 1. Evaluation of SAE’s ability to extract monosemantic features from polysemantic activations in LLM. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. MSE v.s. L0 for expand ratio = 32 across varying L1 coefficients (λ). The color gradient represents the L1 coefficient values, with brighter colors indicating larger λ. As the λ increases, the model becomes sparser (lower L0) while MSE gets worse. As shown in Equation 4, a critical aspect of evaluat￾ing SAE during training lies in balancing two primary metrics: mean squared error for reconstruction accu￾racy and the… view at source ↗
Figure 3
Figure 3. (Left) Cosine distance between LLM activations and SAE features in polysemantic contexts. The histogram compares the distribution of Polysemous Distinction (i.e. 1 - Cosine Similarity) for LLM activations (blue) and SAE features (green), demonstrating that SAE features extract more distinct representations in pol￾ysemous contexts. We adopt R = 128 and L1 coefficient λ = 0.05 for the hyperparameters of SAEs. (Right) … view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Comparison of F1 score, Precision, and Recall across different activation functions. The models [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison of Specificity across different layers of the SAE in relation to L [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Comparison of F1 score and Specificity for SAEs trained on the outputs of the Residual, MLP, and [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: (Left) Changes in the number of dead features during training. The configuration used is expand ratio = 32, L1 coefficient = 0.05 and layer 4. Without Ghost Grad, the number of dead features increases significantly, while using Ghost Grad prevents this growth. (Right) …
Figure 8
Figure 8. Figure 8: Comparison of F1 Score, Recall, and Precision across different Expand Ratios (32, 64, 128) for Open [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Comparison of Accuracy vs MSE and L0 for different Expand Ratios (L1 Coef = 0.05). The left panel shows the relationship between accuracy and mean squared error (MSE), while the right panel presents accuracy versus normalized L0 sparsity. The markers represent various …
Figure 10
Figure 10. Figure 10: Comparison of F1 score, Precision, and Recall across different activation functions. The models [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Comparison of F1 score and Specificity for SAEs trained on the outputs of the Residual, MLP, and [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Comparison of Specificity across different layers of the SAE in relation to L [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Comparison of evaluation metrics between dense autoencoder ( [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Comparison of F1 score, Precision, and Recall across different activation functions. The models [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Comparison of ReLU and JumpReLU activation functions. The ReLU function (green) activates [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Illustration of a single-layer Transformer architecture. The input tokens are embedded and passed [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Comparison of activation functions (ReLU, JumpReLU (STE), and TopK) across different eval [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Evaluation of various performance metrics (Accuracy, Recall, Specificity, Precision, Sensitivity, and [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 19
Figure 19. Figure 19: Evaluation of various performance metrics (Accuracy, Recall, Specificity, Precision, Sensitivity, and [PITH_FULL_IMAGE:figures/full_fig_p026_19.png]
Figure 20
Figure 20. Figure 20: Evaluation of various performance metrics (Accuracy, Recall, Specificity, Precision, Sensitivity, and [PITH_FULL_IMAGE:figures/full_fig_p026_20.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sealing The Backdoor: Unlearning Adversarial Text Triggers In Diffusion Models Using Knowledge Distillation

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    SKD-CAG erases adversarial text triggers from diffusion models by distilling the model's own clean outputs through cross-attention guidance, claiming 100% and 93% removal for pixel and style backdoors.

Reference graph

Works this paper leans on

45 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    Mechanistic interpretability for AI safety - a review

    Leonard Bereska and Stratis Gavves. Mechanistic interpretability for AI safety - a review. Transactions on Machine Learning Research, 2024

    work page 2024

  2. [2]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp.\ 2397--2430. PMLR, 2023

    2023

  3. [3]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Ch...

    2023

  4. [4]

    Evaluating open-source sparse autoencoders on disentangling factual knowledge in gpt-2 small

    Maheep Chaudhary and Atticus Geiger. Evaluating open-source sparse autoencoders on disentangling factual knowledge in gpt-2 small. arXiv preprint arXiv:2409.04478, 2024

    arXiv 2024

  5. [5]

    Sparse autoencoders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023

    arXiv 2023

  6. [6]

    Interpreting and steering features in images, 2024

    Gytis Daujotas. Interpreting and steering features in images, 2024. URL https://www.lesswrong.com/posts/Quqekpvx8BGMMcaem/interpreting-and-steering-features-in-images

    work page 2024

  7. [7]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. URL https://transformer-circuits.pub/20...

    2022

  8. [8]

    Benjamin Erichson, Zhewei Yao, and Michael W

    N. Benjamin Erichson, Zhewei Yao, and Michael W. Mahoney. Jumprelu: A retrofit defense strategy for adversarial attacks. arXiv preprint arXiv:1904.03750, 2019

    arXiv 1904

Show all 45 references
  1. [9]

    Towards empirical interpretation of internal circuits and properties in grokked transformers on modular polynomials

    Hiroki Furuta, Gouki Minegishi, Yusuke Iwasawa, and Yutaka Matsuo. Towards empirical interpretation of internal circuits and properties in grokked transformers on modular polynomials. arXiv preprint arXiv:2402.16726, 2024

    2024 arXiv

  2. [10]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024

    2024 arXiv

  3. [11]

    Gemma Team , Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Ale...

    2024 arXiv

  4. [12]

    Finding neurons in a haystack: Case studies with sparse probing

    Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023

    2023 arXiv

  5. [13]

    Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on othello-gpt

    Zhengfu He, Xuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, and Xipeng Qiu. Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on othello-gpt. arXiv preprint arXiv:2402.12201, 2024

    2024 arXiv

  6. [14]

    Ghost grads: An improvement on resampling

    Adam Jermyn and Adly Templeton. Ghost grads: An improvement on resampling. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/jan-update/index.html#dict-learningresampling

    2024

  7. [15]

    Saebench: A comprehensive benchmark for sparse autoencoders, 2024 a

    Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Arthur Conmy, Callum McDougall, Kola Ayonrinde, Matthew Wearden, Samuel Marks, and Neel Nanda. Saebench: A comprehensive benchmark for sparse autoencoders, 2024 a . URL h...

    2024

  8. [16]

    Evaluating sparse autoencoders on targeted concept erasure tasks

    Adam Karvonen, Can Rager, Samuel Marks, and Neel Nanda. Evaluating sparse autoencoders on targeted concept erasure tasks. arXiv preprint arXiv:2411.18895, 2024 b

    2024 arXiv

  9. [17]

    Interpreting attention layer outputs with sparse autoencoders

    Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. Interpreting attention layer outputs with sparse autoencoders. arXiv preprint arXiv:2406.17759, 2024

    2024 arXiv

  10. [18]

    Gazing in the latent space with sparse autoencoders

    Sangwu Lee. Gazing in the latent space with sparse autoencoders. dev.log, 2024. URL https://re-n-y.github.io/devlog/rambling/sae/

    2024

  11. [19]

    Michaud, David D

    Yuxiao Li, Eric J. Michaud, David D. Baek, Joshua Engels, Xiaoqing Sun, and Max Tegmark. The geometry of concepts: Sparse autoencoder feature structure. arXiv preprint arXiv:2410.19750, 2024

    2024 arXiv

  12. [20]

    Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla

    Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.09458, 2023

    2023 arXiv

  13. [21]

    Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J \'a nos Kram \'a r, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147, 2024

    2024 arXiv

  14. [22]

    Towards principled evaluations of sparse autoencoders for interpretability and control

    Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. arXiv preprint arXiv:2405.08366, 2024

    2024 arXiv

  15. [23]

    Bridging lottery ticket and grokking: Is weight norm sufficient to explain delayed generalization? arXiv preprint arXiv:2310.19470, 2024

    Gouki Minegishi, Yusuke Iwasawa, and Yutaka Matsuo. Bridging lottery ticket and grokking: Is weight norm sufficient to explain delayed generalization? arXiv preprint arXiv:2310.19470, 2024

    2024 arXiv

  16. [24]

    Progress measures for grokking via mechanistic interpretability

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023

    2023 arXiv

  17. [25]

    interpreting gpt: the logit lens

    nostalgebraist. interpreting gpt: the logit lens. Less-Wrong, 2020. URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  18. [26]

    Zoom in: An introduction to circuits

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. URL https://distill.pub/2020/circuits/zoom-in

    2020

  19. [27]

    Wic: the word-in-context dataset for evaluating context-sensitive meaning representations

    Mohammad Taher Pilehvar and Jose Camacho-Collados. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. arXiv preprint arXiv:1808.09121, 2019

    2019 arXiv

  20. [28]

    Grokking: Generalization beyond overfitting on small algorithmic datasets

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022

    2022 arXiv

  21. [29]

    Language models are unsupervised multitask learners, 2019

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019

    2019

  22. [30]

    XL - W i C : A multilingual benchmark for evaluating semantic contextualization

    Alessandro Raganato, Tommaso Pasini, Jose Camacho-Collados, and Mohammad Taher Pilehvar. XL - W i C : A multilingual benchmark for evaluating semantic contextualization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 719...

    2020

  23. [31]

    Improving dictionary learning with gated sparse autoencoders

    Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404.16014, 2024 a

    2024 arXiv

  24. [32]

    Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders

    Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435, 2024 b

    2024 arXiv

  25. [33]

    Interpreting preference models w/ sparse autoencoders, 2024

    Logan Riggs and Jannik Brinkmann. Interpreting preference models w/ sparse autoencoders, 2024. URL https://www.lesswrong.com/posts/5XmxmszdjzBQzqpmz/interpreting-preference-models-w-sparse-autoencoders

    2024

  26. [34]

    Illusionvqa: A challenging optical illusion dataset for vision language models

    Haz Sameen Shahgir, Khondker Salman Sayeed, Abhik Bhattacharjee, Wasi Uddin Ahmad, Yue Dong, and Rifat Shahriyar. Illusionvqa: A challenging optical illusion dataset for vision language models. arXiv preprint arXiv:2403.15952, 2024

    2024 arXiv

  27. [35]

    Taking features out of superposition with sparse autoencoders, 2022

    Lee Sharkey, Dan Braun, and Beren Millidge. Taking features out of superposition with sparse autoencoders, 2022. URL https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition

    2022

  28. [36]

    On the proper treatment of connectionism

    Paul Smolensky. On the proper treatment of connectionism. Behavioral and Brain Sciences, 11 0 (1): 0 1–23, 1988

    1988

  29. [37]

    Unpacking sdxl turbo: Interpreting text-to-image models with sparse autoencoders

    Viacheslav Surkov, Chris Wendler, Mikhail Terekhov, Justin Deschenaux, Robert West, and Caglar Gulcehre. Unpacking sdxl turbo: Interpreting text-to-image models with sparse autoencoders. arXiv preprint arXiv:2410.22366, 2024

    2024

  30. [38]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua ...

    2024

  31. [39]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537, 2020

    1905 arXiv

  32. [40]

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022

    2022 arXiv

  33. [41]

    Redpajama: an open dataset for training large language models

    Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpaja...

    2024 arXiv

  34. [42]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  35. [43]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  36. [44]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  37. [45]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.