Pith. sign in

REVIEW 6 major objections 4 minor 49 references

TopK Language Models

T0 review · 6 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes TopK LMs, transformers whose early layers keep only the $k$ largest activations, making the model's own hidden states serve as sparse, interpretable features and eliminating post-hoc sparse-autoencoder training.

desk verdict A genuinely novel architecture for built-in sparse interpretability, but the performance drops are understated and the interpretability evidence lacks a sparsity-only control. read the letter →

arxiv 2506.21468 v1 pith:UPRZD74K submitted 2025-06-26 cs.CL

Ryosuke Takahashi , Tatsuro Inaba , Kentaro Inui , Benjamin Heinzerling This is my paper · ORCID
classification cs.CL
keywords TopKactivationsparselanguagemodelinherentinterpretabilityautoencoderfeaturesconceptsteeringneuronspecializationtrainingdynamicssparsity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Language models are usually interpreted by training a sparse autoencoder (SAE) on top of their activations after the fact, which leaves open whether a missing feature is absent from the model or just missed by the SAE. This paper proposes TopK LMs, a transformer variant that makes sparsity part of the architecture itself: selected layers keep only the $k$ largest hidden activations and zero the rest. The claim is that the resulting hidden states are exactly the latent features a TopK SAE would produce, so the model is interpretable without any post-hoc training. The paper shows these models stay close to dense baselines on perplexity and downstream accuracy while individual neurons become concept detectors that can be steered, and their formation can be traced across checkpoints and layers.

What carries the argument

The central object is the TopK activation $T_k(x)$, which retains the $k$ largest components of a hidden vector after an elementwise nonlinearity and zeros the rest, plus two design choices: hybrid placement (the last $n_{\mathrm{nontopk}}$ layers stay dense) and sparsity annealing (a factor $\alpha$ linearly decays from 1 to 0, interpolating between dense and TopK activations in the first 20% of training). These make the layer's output a sparse vector whose nonzero coordinates are directly the model's own features, the same representation a TopK SAE would produce, so no separate encoder or decoder is trained. The mechanism carries the paper's argument because it turns a post-hoc interpretability method into an architectural constraint.

What would settle it

Take a dense model and a matched TopK model trained on the same corpus, run a fixed TopK SAE on the dense model, and collect a broad set of its semantically coherent features. If a substantial fraction of those concepts have no corresponding single TopK neuron—no neuron whose selected token set overlaps the concept and whose semantic entropy is comparably low—the claim that TopK hidden states are equivalent to SAE latents is falsified. A cheaper test is to measure, for one behaviorally detectable concept, whether any TopK neuron is both necessary and sufficient for steering that concept in the model.

Watch

Extended reading notes

Core claim

Replacing the final activation function of a transformer layer with the TopK function—keeping the $k$ largest values and masking the rest—makes the model's own hidden states equivalent to the sparse latents of a TopK SAE. The paper demonstrates this across decoder-only models of varying depth and width: with $k=64$, all but the last two layers sparse, widened hidden dimensions, and a sparsity-annealing schedule, TopK LMs reach validation perplexities only moderately above dense baselines (e.g., 14.96 vs. 11.76 for a 24-layer 1024-dimensional model) while staying on par or ahead on several zero-shot benchmarks. The interpretability payoff is measured as low token and semantic entropy per neuron, successful single-neuron steering toward concepts such as 'work', 'numbers', 'history', and 'science', and the ability to follow a particular hidden dimension (e.g., dimension 894 for 'work') as it becomes specialized across training checkpoints and layers.

Load-bearing premise

The interpretability guarantee rests on the empirical bet that plain next-token training, given only a TopK mask, will spontaneously make hidden units semantically monosemantic concept detectors; the paper shows hand-selected low-entropy neurons and steering examples, not a mechanism that enforces monosemanticity.

Editorial extensions

If this is right

  • An LM trained with only a next-token objective and a TopK mask can have SAE-like monosemantic features without any auxiliary reconstruction or interpretability loss.
  • Single-neuron interventions become a viable control mechanism: amplifying one low-semantic-entropy neuron shifts generation toward that neuron's concept.
  • Neuron formation can be observed continuously during training, because a hidden-dimension index is the same object across checkpoints and layers, which post-hoc SAEs cannot offer.
  • Sparsity does not force a large performance hit: at fixed $k=64$, wider hidden dimensions improve both perplexity and specialization, giving a favorable size–sparsity trade-off.
  • Higher layers need not be sparse: keeping the final two layers dense preserves most of the model's expressivity while earlier TopK layers provide interpretability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: if the equivalence holds at scale, SAE interpretability could become a property of architecture rather than an analysis step, eliminating the seed-variability and cross-checkpoint alignment problems that plague post-hoc SAEs; the paper itself does not test multi-seed stability.
  • Extension: the model provides a direct testbed for when and how concepts form during training, since the unit of analysis is exactly one hidden dimension; a natural next experiment is automatic concept labeling of all neurons instead of the paper's manual labels.
  • Extension: comparing steering success of TopK LM neurons against SAE features of a dense twin model on the same data would quantify whether task-aligned features are more causally effective than reconstruction-trained features.
  • Extension: because the sparse mask selects dimensions by magnitude, the network may be learning a routing-like specialization; a testable prediction is that more layers under TopK produce sharper per-neuron selectivity, which the 24-layer entropy plots already suggest.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 4 minor

Summary. The paper proposes TopK LMs, a transformer-based language model variant in which a TopK activation function is inserted in all but the last two layers, retaining only the k=64 largest pre-activations per layer, with a linear annealing schedule from dense to sparse during the first 20% of training. The authors train Llama-style models with hidden dimensions 1024 and 2048 and depths 8, 16, and 24 on about 20B tokens of FineWeb-Edu, and compare them with dense baselines on perplexity and zero-shot benchmarks. They then present three interpretability showcases: token/semantic entropy of individual neurons, single-neuron activation steering, and tracing of a 'work' neuron across training checkpoints and layers. The central claim is that TopK LMs retain competitive language-modeling performance while providing SAE-like, post-hoc-training-free interpretability.

Significance. An architecture that produces sparse, monosemantic, steerable features as a byproduct of standard LM training would be a meaningful contribution: it could remove the need for post-hoc SAE training, stabilize cross-checkpoint feature comparison, and simplify mechanistic interpretability. The paper has several strengths: the TopK mask definition is clean; the study is systematic in depth, width, and sparsity; and the training setup is reported in detail. However, the current evidence does not support the strength of the claims. The performance degradations on ARC-Easy and HellaSwag are large; the entropy and steering analyses lack controls that would rule out trivial sparsity artifacts; and the cross-layer tracing claim is based on index identity rather than functional correspondence. With additional experiments and recalibrated claims, this could become a solid contribution.

major comments (6)
  1. [§2.3, Table 1] The claim (Section 2.3 and Abstract) that TopK LMs 'maintain their original capabilities' and that sparsity 'does not materially degrade language-model performance' is contradicted by the reported numbers. For D=1024, L=24, ARC-Easy drops from 56.27 to 30.51 and HellaSwag from 32.11 to 25.95; for D=2048, L=24, ARC-Easy drops from 64.06 to 32.62 and HellaSwag from 37.49 to 27.90, while LAMBADA perplexity rises from 22.24 to 31.91. These are large degradations. The paper should recalibrate the performance claims and should report parameter counts and FLOPs, since the 'favorable trade-off between model size and computational efficiency' is asserted without those measurements.
  2. [§3.3, Eqs. (4)–(11)] The entropy evidence does not establish SAE-like semantic specialization, because the low token entropy is largely a consequence of hard sparsity: with k=64 out of 1024 or 2048, activations are zero for most neurons on most tokens, so any sparse code—including a random one—will have token entropy far below log|V|. In addition, the semantic entropy in Eq. (11) is computed on a small threshold-selected token subset, so its pairwise-similarity histogram is sparse and low-entropy even without semantic coherence. The paper needs controls: a random-mask TopK LM with the same k, an untrained model, and/or standard SAE interpretability metrics (e.g., SAEBench auto-interpretation) on the same baseline. Without such controls, the claim of 'robust interpretability benefits' is not supported.
  3. [§4, Table 2] The steering demonstration is qualitative and uncontrolled. The four neurons are hand-picked and manually labeled, no quantitative measure of target-concept adherence is reported, and there is no baseline steering of random neurons or random offsets. The 'Number' example produces largely degenerate repeated token output, which further illustrates the need for quantitative evaluation. Please add a controlled steering evaluation, e.g., multiple prompts, automatic concept-classifier scores, and random-neuron controls, before claiming that single-neuron interventions 'steer' the model.
  4. [§5, Figure 5] The claim that the same hidden-dimension index can be tracked 'across layers' as the same neuron is not justified, because each layer has independent parameters and an index match does not imply functional correspondence. Tracking the same index within a fixed layer across checkpoints is well-defined, but the cross-layer identification needs a matching criterion (e.g., activation correlation, ablation, or intervention); otherwise the cross-layer tracing claim should be removed or clearly qualified.
  5. [§2.1, 'Annealed TopK smoothing'] The claim that annealing the TopK sparsity 'greatly improves convergence' is presented as a finding, but no experiment compares the annealed schedule with a fixed (non-annealed) TopK activation. Since this is listed as one of the paper's contributions, please add the corresponding ablation or weaken the claim accordingly.
  6. [Tables 3 and 4] The ablation tables appear to contain duplicated or erroneous rows. In Table 3, the rows for n_nontopk=0 and n_nontopk=1 have identical accuracy values across all benchmarks; in Table 4, the k=8 row matches the k=64 row in all accuracy columns, and the k=128 row matches the baseline row in all accuracy columns. Please verify the ablation data and correct any transcription errors, since these tables are used to support the choice of architecture hyperparameters.
minor comments (4)
  1. [§2.2] There are several typos, including 'he amount of training data' (should be 'the amount') and inconsistent spelling of 'FineWeb-Edu'.
  2. [§2.1, Figure 2] Please clarify exactly where the TopK activation is applied within the transformer block (MLP hidden activation versus residual stream). Figure 2 is ambiguous, and the claim that hidden states are 'equivalent to the latent features of a TopK SAE' depends on this detail.
  3. [§2.3] The sentence beginning 'Following previous research [47], We evaluate...' has an unnecessary capitalization and missing space/period; please clean up the prose.
  4. [§3.3, Figure 3] The separate panels for baseline and TopK models make direct comparison harder than necessary; please consider plotting both curves in the same panel or adding clear labels to each panel.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: TopK LMs' sparse hidden states are definitional by architecture, but the claimed interpretability and performance are supported by external measurements rather than derived from fitted inputs.

full rationale

The paper's central claim has two components: (1) TopK LMs retain competitive language modeling performance, and (2) their hidden states are SAE-like and interpretable. The first is supported by independent perplexity and zero-shot accuracy comparisons against dense baselines in Section 2.3 and Table 1; no parameter is fitted to interpretability data and then reported as a prediction. The second is assessed post-hoc on fully trained models through token entropy (Eq. 6), semantic entropy (Eq. 11), and steering interventions in Section 4. These are measurements taken after standard next-token prediction training with only a TopK mask, so they are not quantities optimized into the model. The abstract's claim that TopK activation makes hidden states 'equivalent to the latent features of a TopK SAE' is definitional, since the TopK activation function by construction produces a top-k sparse vector; however, the paper does not use that equivalence alone as evidence of interpretability. The only self-citation is [17], used to motivate why post-hoc SAEs limit cross-checkpoint analyses; even without that citation, the critique of SAE feature stability is supported by the paper's own discussion and external SAE literature, so it is not load-bearing. The hand-selection of low-semantic-entropy neurons for steering is a selection procedure, not a fitted parameter renamed as a prediction. The cross-layer index tracking in Section 5 and Figure 5 uses hidden-dimension index identity, which may be a validity concern, but the paper reports observed activation patterns rather than deriving a conclusion from an assumed correspondence. Appendix C limitations about scale, evaluation domains, and long context qualify generalization but do not indicate that any metric was constructed from the claim it supports. Overall, the derivation chain is self-contained: no step reduces to its own input, and the minor self-citation and definitional equivalence do not constitute circularity.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central results depend on hyperparameters chosen by hand (k=64, n_nontopk=2), on the empirical assumption that a top-k masked network trains acceptably, and on the assumption that hidden-dimension indices are stable objects for tracking. No new physical entities are introduced.

free parameters (6)
  • topk_k = 64 (default; ablated 8 to 512)
    Number of neurons kept active per layer. Central to the architecture and chosen by hand after ablations.
  • n_nontopk = 2 (default; ablated 0, 1, 2)
    Number of final dense layers left unsparsified. Chosen by hand to preserve representational power.
  • annealing_step_ratio = 0.2
    Fraction of training steps during which sparsity is annealed from dense to sparse. Chosen by hand; no ablation shown.
  • semantic_entropy_threshold = 70% of the 99.9th percentile of mean activations
    Arbitrary threshold used in Equation 7 to define the selected vocabulary subset for semantic entropy.
  • semantic_entropy_bins = 1000
    Number of cosine-similarity bins in Equation 10. Chosen by hand for the semantic entropy histogram.
  • steering_offset = delta in [5, 30]
    Constant offset added to a single neuron's pre-activation in Section 4. Chosen empirically to produce visible steering.
assumptions (3)
  • domain assumption A TopK activation with a hard, piecewise-constant mask provides useful gradients for SGD training of a deep transformer.
    The whole training relies on this being true; the paper demonstrates it empirically for small models but provides no gradient analysis.
  • domain assumption Hidden-dimension indices in the residual stream are stable objects that can be tracked across checkpoints and across layers.
    Section 5's traceability analysis depends on comparing dimension 894 across layers and training steps as if it were the same neuron.
  • domain assumption Results obtained on models with D up to 2048 and L up to 24 will transfer to larger models.
    The paper frames broader interpretability benefits, while the limitations appendix states that larger-scale effectiveness remains open.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TopK Language Models." pith.science (2026). https://pith.science/paper/UPRZD74K

@misc{pith2026250621468,
  author       = {Pith},
  title        = {Pith review of: TopK Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UPRZD74K}},
  note         = {Machine review of arXiv:2506.21468}
}
read the original abstract

Sparse autoencoders (SAEs) have become an important tool for analyzing and interpreting the activation space of transformer-based language models (LMs). However, SAEs suffer several shortcomings that diminish their utility and internal validity. Since SAEs are trained post-hoc, it is unclear if the failure to discover a particular concept is a failure on the SAE's side or due to the underlying LM not representing this concept. This problem is exacerbated by training conditions and architecture choices affecting which features an SAE learns. When tracing how LMs learn concepts during training, the lack of feature stability also makes it difficult to compare SAEs features across different checkpoints. To address these limitations, we introduce a modification to the transformer architecture that incorporates a TopK activation function at chosen layers, making the model's hidden states equivalent to the latent features of a TopK SAE. This approach eliminates the need for post-hoc training while providing interpretability comparable to SAEs. The resulting TopK LMs offer a favorable trade-off between model size, computational efficiency, and interpretability. Despite this simple architectural change, TopK LMs maintain their original capabilities while providing robust interpretability benefits. Our experiments demonstrate that the sparse representations learned by TopK LMs enable successful steering through targeted neuron interventions and facilitate detailed analysis of neuron formation processes across checkpoints and layers. These features make TopK LMs stable and reliable tools for understanding how language models learn and represent concepts, which we believe will significantly advance future research on model interpretability and controllability.

Figures

Figures reproduced from arXiv: 2506.21468 by the authors.

Figure 1
Figure 1. Overview of the TopK LM architecture and its interpretability advantages. Left: The architecture integrates TopK activation functions in selected layers to achieve sparse activation patterns. Right: TopK LMs enjoy SAE-like interpretability. (1) Improved neuron interpretability: individual neurons can be clearly interpreted as representing specific concepts (e.g., "Work"). (2) Steerability: single-neuron intervention… view at source ↗
Figure 2
Figure 2. TopK LM: A sparse neural network archi￾tecture that selectively activates only the k-largest values in early layers while maintaining expressiv￾ity through dense processing in final layers. Annealed TopK smoothing Aiming to im￾prove training stability and convergence, we experimented with varying the degree of spar￾sity throughout training. Based on the fact that current LM training methods and received best practic… view at source ↗
Figure 3
Figure 3. Baseline vs TopK models (24-layer). (Left) Token Entropy. (Right) Semantic Entropy. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Entropy comparison across training checkpoints for the 24-layer TopK model. Left: token [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Visualization of the formation of a concept-specialized neuron across training checkpoints [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Baseline vs TopK models (8-layer). (Left) Token Entropy. (Right) Semantic Entropy. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Baseline vs TopK models (16-layer). (Left) Token Entropy. (Right) Semantic Entropy. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 36 canonical work pages

  1. [1]

    How can we be so dense? the benefits of using highly sparse representations, 2019

    Subutai Ahmad and Luiz Scheinkman. How can we be so dense? the benefits of using highly sparse representations, 2019

    work page 2019

  2. [2]

    Generating long sequences with sparse transformers, 2019

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers, 2019

    2019

  3. [3]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  4. [4]

    [Full Post] Progress Update #1 from the GDM Mech Interp Team

    Arthur Conmy and Neel Nanda. Activation steering with SAEs. https://www.alignmentforum.org/posts/C5KAZQib3bzzpeyrg/ full-post-progress-update-1-from-the-gdm-mech-interp-team , April 2024. Section in “[Full Post] Progress Update #1 from the GDM Mech Interp Team”

    work page 2024

  5. [5]

    Sparse autoencoders find highly interpretable features in language models, 2023

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023

    2023

  6. [6]

    Rigging the lottery: Making all tickets winners

    Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research, pages 2943–2952. PMLR, July 13–18 2020

    work page 2020

  7. [7]

    Sparsity in transformers: A systematic literature review

    Mirko Farina, Usman Ahmad, Ahmad Taha, Hussein Younes, Yusuf Mesbah, Xiao Yu, and Witold Pedrycz. Sparsity in transformers: A systematic literature review. Neurocomputing, 582:127468, 2024

    work page 2024

  8. [8]

    Detecting hallucinations in large language models using semantic entropy

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630, 2024

    2024

Show all 49 references
  1. [9]

    Scaling and evaluating sparse autoencoders, 2024

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024

    2024

  2. [10]

    The language model evaluation harness, 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  3. [11]

    Causal Abstractions of Neural Networks

    Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal Abstractions of Neural Networks. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 9574–9586, 2021

    2021

  4. [12]

    The State of Sparse Training in Deep Reinforcement Learning

    Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. The State of Sparse Training in Deep Reinforcement Learning. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 7766–7792, 2022

    2022

  5. [13]

    Memory-efficient transformers via top-k attention, 2021

    Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, and Jonathan Berant. Memory-efficient transformers via top-k attention, 2021

    2021

  6. [14]

    Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders, 2024

    Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders, 2024. 10

    2024

  7. [15]

    Hindupur, Ekdeep Singh Lubana, Thomas Fel, and Demba Ba

    Sai Sumedh R. Hindupur, Ekdeep Singh Lubana, Thomas Fel, and Demba Ba. Projecting assumptions: The duality between sparse autoencoders and concept geometry, 2025

    2025

  8. [16]

    Two sparsities are better than one: Unlocking the performance benefits of sparse-sparse networks

    Kevin Lee Hunter, Lawrence Spracklen, and Subutai Ahmad. Two sparsities are better than one: Unlocking the performance benefits of sparse-sparse networks. CoRR, abs/2112.13896, 2021

    2021 arXiv

  9. [17]

    How llms learn: Tracing internal representations with sparse autoencoders, 2025

    Tatsuro Inaba, Kentaro Inui, Yusuke Miyao, Yohei Oseki, Benjamin Heinzerling, and Yu Takagi. How llms learn: Tracing internal representations with sparse autoencoders, 2025

    2025

  10. [18]

    Sparse is enough in scaling transformers, 2021

    Sebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin, Łukasz Kaiser, Wojciech Gajew- ski, Henryk Michalewski, and Jonni Kanerva. Sparse is enough in scaling transformers, 2021

    2021

  11. [19]

    Jayakumar, Razvan Pascanu, Jack W

    Siddhant M. Jayakumar, Razvan Pascanu, Jack W. Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. arXiv preprint arXiv:2106.03517, 2021

    2021 arXiv

  12. [20]

    Saebench: A comprehensive benchmark for sparse autoen- coders in language model interpretability, 2025

    Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, Matthew Wearden, Arthur Conmy, Samuel Marks, and Neel Nanda. Saebench: A comprehensive benchmark for sparse autoen- coders in language m...

    2025

  13. [21]

    Concept steerers: Leveraging k-sparse autoencoders for controllable generations, 2025

    Dahye Kim and Deepti Ghadiyaram. Concept steerers: Leveraging k-sparse autoencoders for controllable generations, 2025

    2025

  14. [22]

    Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. In The Eleventh International Conference on Learning Representations, 2023

    2023

  15. [23]

    Soft Threshold Weight Reparameterization for Learnable Sparsity

    Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft Threshold Weight Reparameterization for Learnable Sparsity. In Proceedings of the 37th International Conference on Machine Learning , volume 119, pages 5544–5555, 2020

    2020

  16. [24]

    Sparse autoencoders do not find canonical units of analysis, 2025

    Patrick Leask, Bart Bussmann, Michael Pearce, Joseph Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda. Sparse autoencoders do not find canonical units of analysis, 2025

    2025

  17. [25]

    Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024

    2024

  18. [26]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations, 2019

    2019

  19. [27]

    Learning sparse neural networks through l_0 regularization

    Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through l_0 regularization. In International Conference on Learning Representations , 2018

    2018

  20. [28]

    Fineweb-edu: the finest collection of educational content, 2024

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024

    2024

  21. [29]

    Winner-take-all autoencoders

    Alireza Makhzani and Brendan J Frey. Winner-take-all autoencoders. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015

    2015

  22. [30]

    Pointer Sentinel Mixture Models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer Sentinel Mixture Models. In International Conference on Learning Representations , 2017

    2017

  23. [31]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2381–2391, 2018

    2018

  24. [32]

    Nguyen, Madeleine Gibescu, Antonio Liotta, and et al

    Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, Antonio Liotta, and et al. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(2383), 2018. 11

    2018

  25. [33]

    The LAMBADA dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of t...

    2016

  26. [34]

    Sparse autoencoders trained on the same data learn different features, 2025

    Gonçalo Paulo and Nora Belrose. Sparse autoencoders trained on the same data learn different features, 2025

    2025

  27. [35]

    Automatically interpreting millions of features in large language models, 2024

    Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. Automatically interpreting millions of features in large language models, 2024

    2024

  28. [36]

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

    Guilherme Penedo, Hynek Kydlíˇcek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro V on Werra, and Thomas Wolf. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In The Thirty-eight Conference on Neural Information Processing ...

    2024

  29. [37]

    Improving dictionary learning with gated sparse autoencoders, 2024

    Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders, 2024

    2024

  30. [38]

    Winogrande: an adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106, 2021

    2021

  31. [39]

    Taking features out of superposition with sparse autoencoders

    Lee Sharkey, Dan Braun, and Beren Millidge. Taking features out of superposition with sparse autoencoders. https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/ interim-research-report-taking-features-outof-superposition , 2023. Ac- cessed: 2025-05-15

    2023

  32. [40]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017

    2017

  33. [41]

    A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models, 2025

    Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models, 2025

    2025

  34. [42]

    Codebook features: Sparse and discrete interpretability for neural networks

    Alex Tamkin, Mohammad Taufeeque, and Noah D Goodman. Codebook features: Sparse and discrete interpretability for neural networks. arXiv preprint arXiv:2310.17230, 2023

    2023 arXiv

  35. [43]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua ...

    2024

  36. [44]

    LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Langua...

    2023 arXiv

  37. [45]

    Meta Lingua: A minimal PyTorch LLM training library, 2024

    Mathurin Videau, Badr Youbi Idrissi, Daniel Haziza, Luca Wehrstedt, Jade Copet, Olivier Teytaud, and David Lopez-Paz. Meta Lingua: A minimal PyTorch LLM training library, 2024

    2024

  38. [46]

    Tracking the feature dynamics in llm training: A mechanistic study, 2025

    Yang Xu, Yi Wang, and Hao Wang. Tracking the feature dynamics in llm training: A mechanistic study, 2025

    2025

  39. [47]

    STEP: Staged parameter-efficient pre-training for large language models

    Kazuki Yano, Takumi Ito, and Jun Suzuki. STEP: Staged parameter-efficient pre-training for large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computa- tion...

    2025

  40. [48]

    HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4791–4800, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4791–4800, 2019

    2019

  41. [49]

    PyTorch FSDP: Experiences on scaling fully sharded data parallel

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. PyTorch FSDP: Experiences on sc...

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.