Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Self-Ablating Transformers: More Interpretability, Less Sparsity

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A k-winner-takes-all training constraint makes transformers more interpretable without making them sparser.

desk verdict The self-ablation training mechanism is genuinely novel and the interpretability results are promising, but the headline 'less sparsity' claim rests on a metric that doesn't measure sparsity, so the central surprise is unsupported as written. read the letter →

arxiv 2505.00509 v1 pith:6M5OGOZO submitted 2025-05-01 cs.LG

Jeremias Ferrao , Luhan Mikaelson , Keenan Pepper , Natalia Perez-Campanero Antolin This is my paper · ORCID
classification cs.LG
keywords self-ablationk-winner-takes-alltransformerinterpretabilitymechanisticsparsityneuronspecializationcircuitlocalizationTinyStories
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that interpretability can be built into a transformer during training rather than inferred after the fact. It introduces a self-ablation mechanism that enforces a $k$-winner-takes-all constraint, keeping only the top-$k$ neurons and attention heads active for each input while a straight-through estimator lets gradients flow through the hard selection. Trained on a small synthetic-story corpus, the resulting models show substantially smaller task circuits, sparser autoencoder features, and more explainable, specialized neurons at only a modest perplexity cost. The paper's headline result is that these gains come with lower global sparsity, so interpretability here is driven by specialization and localization rather than by widespread inactivity.

What carries the argument

The load-bearing object is the self-ablation gate built on a $k$-winner-takes-all constraint. For each set of gating activations, the model sorts the values, sets a threshold $\gamma=(x_k+x_{k+1})/2$ and a temperature $T=x_k-x_{k+1}$, and derives softmax weights $w_i$ for gradient computation while a hard top-$k$ mask is used in the forward pass. These gates sit on MLP neurons and attention-head outputs, are trained jointly with the base transformer weights, and are switched off at inference, leaving a standard transformer architecture. A dual residual stream computes clean and ablated outputs in a single pass, and the total loss is the sum of the clean cross-entropy and the ablated cross-entropy. The gates are what force the model to specialize components from the start of training, and their removal at inference is what makes the interpretability gain free at deployment.

What would settle it

Count the actual fraction of dead or near-zero-weight components in both baseline and self-ablated models; if the self-ablated models have fewer such components rather than more, the paper's claim of decreased overall sparsity is contradicted. The same test could be run with an exact zero-weight count or a per-layer activity census to settle whether the L1 result reflects true density or just larger magnitudes.

Watch

Extended reading notes

Core claim

The central claim is that self-ablation, applied during training, produces more interpretable transformers without sacrificing language-modelling quality, and that the improvement arises through local specialization rather than reduced overall activity. Concretely, the paper reports that a local variant with $k=4$ cuts the Indirect Object Identification circuit from 79 to 30 edges and reduces the average number of active sparse-autoencoder features from 7.22 to 4.01, while raising validation perplexity from 5.73 to 6.58. The same model family yields higher automated neuron explanation scores and sparser, more selective token-activation graphs than the baseline. The broader conclusion is that the common intuition tying sparsity to interpretability is incomplete: decreased global sparsity can coexist with increased local specialization, and the latter is what makes circuits and neurons more legible.

Load-bearing premise

The paper's surprising 'less sparsity' claim rests on treating the L1 norm of the model's weights as a measure of overall sparsity, and if weight magnitude is not a valid proxy for how many components are actually active, that headline result collapses.

Editorial extensions

If this is right

  • Interpretability can be engineered into a model during training without changing its inference-time architecture, so existing deployment and analysis tooling continues to work.
  • Task circuits and sparse-autoencoder feature sets can shrink sharply under the constraint; the paper's runs show up to a 62% reduction in circuit edges and a 44% reduction in active features.
  • If higher weight L1 norms really indicate lower global sparsity, then global weight sparsity is the wrong target for interpretability; selective per-input activation with dense weights retained is what matters.
  • Local ablation generally beats global ablation on both circuit size and perplexity in the paper's comparisons, pointing to layer-local decisions as the preferred design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit extension is that the same training-time constraint could be applied during fine-tuning of a larger pretrained model, testing whether localization benefits survive when the base weights already encode distributed circuits.
  • The 'less sparsity' result suggests that interpretability metrics and sparsity metrics are tracking different quantities; a natural next experiment is to vary $k$ continuously and watch whether circuit size and weight L1 norm move independently.
  • If the reported neuron specialization persists at scale, interventions like targeted unlearning or concept removal could become easier on self-ablated models, because the relevant computation is confined to fewer, more legible units.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a self-ablation mechanism for transformers: learned gating weights select top-k MLP neurons and attention heads during training via a straight-through estimator, with an auxiliary loss on the ablated output, while the gates are removed at inference. Models are trained on TinyStories at roughly 3M scale. The authors report improved interpretability relative to a standard GPT-Neo baseline, including smaller ACDC IOI circuits, lower SAE L0 norms, higher neuron explanation scores, and sparser neuron graphs, at a modest perplexity cost. They also report higher values in the 'Sparsity (L1)' column of Table 1 under self-ablation, which they interpret as a surprising decrease in overall sparsity.

Significance. If the findings hold, the paper makes a useful contribution: an ante-hoc training intervention that improves several interpretability indicators without changing the inference-time architecture, together with a challenge to the common assumption that sparsity and interpretability always go hand in hand. The code release and the use of multiple standard interpretability tools are strengths. However, the central 'Less Sparsity' claim rests on a metric that is not a sparsity measure, and the quantitative comparisons lack repeated-run variance, so the significance of the paper is currently conditional on a corrected analysis.

major comments (4)
  1. [§4.5, Table 1] The 'Sparsity (L1)' column is defined in Section 4.5 as 'the normalized sum of the model's weights.' The L1 norm of weights is a magnitude measure, not a sparsity measure: a dense weight matrix with small entries can have a lower L1 norm than a sparse matrix with a few large entries. Thus the observed increase in L1 from 0.44 to roughly 0.7 does not establish a decrease in sparsity, and the title and abstract's 'Less Sparsity' claim is unsupported. Please report a true sparsity metric (for example, the fraction of weights or activations that are exactly zero or below a threshold) for both weights and activations, and state exactly which parameters are included (in particular, whether the auxiliary gating weights are counted).
  2. [§5.5, §6] The conclusion that interpretability improves through increased specialization rather than through sparsity relies on the L1 result as the only global-sparsity evidence. The other reported metrics — ACDC edge count, SAE L0, and N2G density — are local circuit-level or representation-level measures, and lower values of these metrics do not imply that the model's overall weight or activation vector is more sparse. Consequently, even if all the interpretability metrics improve, the 'decreased global sparsity' conclusion does not follow from the data presented.
  3. [§5, Tables 1 and 2] All comparisons are based on single training runs with no error bars or significance tests. For example, the ACDC edge counts vary from 30 to 54 across ablation configurations with no indication of run-to-run variability, so claims such as 'the local approach generally outperformed global ablation' are not supported. Please provide multiple seeds, at least for the baseline and one or two ablation configurations, and report means and standard deviations, or otherwise justify why single runs are sufficient for the comparative claims.
  4. [§4.1, Appendix D] The ACDC results use a single fixed threshold τ=0.03 for all models, with no sensitivity analysis. Circuit edge counts can depend substantially on the threshold and on the patching metric; a sweep over τ, or a demonstration that the edge-count ranking is stable across thresholds, is needed to support the claim that self-ablation produces smaller circuits rather than that the chosen threshold happens to favor the ablated models. Additionally, Section 4.1 and Appendix D make contradictory statements about whether lower τ yields sparser circuits or more edges; this must be corrected.
minor comments (5)
  1. [Abstract, §4.5, Table 1] The term 'L1 norm' is used inconsistently: the abstract and introduction refer to 'L1 activation norms,' while Section 4.5 defines it as the normalized sum of weights and Table 1 labels it 'Sparsity (L1).' This inconsistency should be resolved and the exact definition should be stated in the table caption.
  2. [References] The reference entry 'Curt Tigges Joseph Bloom and David Chanin. Saelens' is malformed; the author list should be corrected (Bloom, Joseph, Curt Tigges, and David Chanin).
  3. [Figure 2 (right), §5.3] The explanation-score histograms are shown for the baseline and one ablated model, but the text reports only the two means; please report the number of neurons, the standard deviation, and a significance test for the difference.
  4. [§5.3] Section 5.3 analyzes only the global k=2 model; the conclusion that self-ablation improves neuron explainability should be qualified as limited to that configuration, or additional configurations should be analyzed.
  5. [Appendix C] The appendix claims 'constant time complexity of self-ablation,' but the described sorting step is O(N log N); the intended claim appears to be that the overhead is independent of the input length, and this should be stated precisely.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the empirical comparisons are self-contained and externally benchmarked.

full rationale

The paper's claims are empirical measurements on trained models compared against a baseline: ACDC edge counts, SAE L0 norms, explanation scores, N2G graph metrics, and perplexity are all external evaluation tools applied after training. The self-ablation mechanism (kWTA with straight-through estimator) is a training intervention, not a fitted parameter that is later renamed as an outcome; there is no equation in the paper that defines an interpretability gain in terms of the kWTA mask itself. No load-bearing self-citation chain appears: the mechanism cites standard kWTA literature (Majani et al., 1988), and the evaluation tools are independent external works (Conmy et al., 2023; Foote et al., 2023a; Bills et al., 2023). The 'decreased sparsity' claim rests on the L1 norm defined as a normalized sum of weights, which is a questionable proxy for sparsity and is a correctness or validity concern rather than a circular derivation: the result is an observed number, not constructed from the definition of the method. Accordingly, no circular step meets the bar of being equivalent to its inputs by construction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central empirical claims depend on a small set of hand-chosen hyperparameters (k, tau, SAE L1 coefficient) and on three domain assumptions about the validity of interpretability proxies. The most fragile is the assertion that weight L1 norm captures sparsity, which is used to support the 'less sparsity' headline finding.

free parameters (3)
  • k (kWTA winner count) = 1, 2, 4, 8 (scanned)
    The number of components kept active by self-ablation is a hand-chosen hyperparameter scanned across values; all interpretability claims are conditioned on k.
  • ACDC threshold tau = 0.03
    The threshold for retaining circuit edges in ACDC is fixed at 0.03 across all models; edge counts depend heavily on this choice.
  • SAE L1 coefficient = 5
    SAE L0 norms and CE scores depend on the L1 penalty coefficient used to train the autoencoders, yet no sensitivity analysis is reported.
assumptions (3)
  • domain assumption The straight-through estimator provides sufficiently accurate gradients for the top-k selection to train the model end-to-end.
    The paper relies on this to train through the non-differentiable hard mask in Section 3.1, but it is not verified.
  • domain assumption ACDC circuit edge count is a valid proxy for interpretability.
    The paper states that fewer edges indicates better localization; this is a domain assumption that is not independently justified.
  • ad hoc to paper The L1 norm of model weights measures sparsity.
    Used in Section 4.5 and Table 1 to claim decreased overall sparsity; weight L1 does not count zero components and is not a sparsity metric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Ablating Transformers: More Interpretability, Less Sparsity." pith.science (2026). https://pith.science/paper/6M5OGOZO

@misc{pith2026250500509,
  author       = {Pith},
  title        = {Pith review of: Self-Ablating Transformers: More Interpretability, Less Sparsity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6M5OGOZO}},
  note         = {Machine review of arXiv:2505.00509}
}
read the original abstract

A growing intuition in machine learning suggests a link between sparsity and interpretability. We introduce a novel self-ablation mechanism to investigate this connection ante-hoc in the context of language transformers. Our approach dynamically enforces a k-winner-takes-all constraint, forcing the model to demonstrate selective activation across neuron and attention units. Unlike post-hoc methods that analyze already-trained models, our approach integrates interpretability directly into model training, promoting feature localization from inception. Training small models on the TinyStories dataset and employing interpretability tests, we find that self-ablation leads to more localized circuits, concentrated feature representations, and increased neuron specialization without compromising language modelling performance. Surprisingly, our method also decreased overall sparsity, indicating that self-ablation promotes specialization rather than widespread inactivity. This reveals a complex interplay between sparsity and interpretability, where decreased global sparsity can coexist with increased local specialization, leading to enhanced interpretability. To facilitate reproducibility, we make our code available at https://github.com/keenanpepper/self-ablating-transformers.

Figures

Figures reproduced from arXiv: 2505.00509 by the authors.

Figure 1
Figure 1. Comparison of global and local ablation mechanisms. Both models use a transformer with [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Self-ablation improves interpretability as shown through (left) IOI circuit simplification [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 10 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Language Models Can Explain Neurons in Language Models , May 2023

    Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language Models Can Explain Neurons in Language Models , May 2023. URL https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html. [Online; accessed 18. Dec. 2024]

    work page 2023

  3. [3]

    GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow , March 2021

    Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow , March 2021. URL https://doi.org/10.5281/zenodo.5297715

    doi:10.5281/zenodo.5297715 2021

  4. [4]

    Gradient routing: Masking gradients to localize computation in neural networks, 2024

    Alex Cloud, Jacob Goldman-Wetzler, Evžen Wybitul, Joseph Miller, and Alexander Matt Turner. Gradient routing: Masking gradients to localize computation in neural networks, 2024. URL https://arxiv.org/abs/2410.04332

    arXiv 2024

  5. [5]

    Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri \`a Garriga-Alonso

    Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri \`a Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=89ia77nZ8u

    work page 2023

  6. [6]

    Tinystories: How small can language models be and still speak coherent english?, 2023

    Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023. URL https://arxiv.org/abs/2305.07759

    arXiv 2023

  7. [7]

    Neuron to graph: Interpreting language model neurons at scale

    Alex Foote, Neel Nanda, Esben Kran, Ioannis Konstas, Shay Cohen, and Fazl Barez. Neuron to graph: Interpreting language model neurons at scale. arXiv preprint arXiv:2305.19911, 2023 a

    arXiv 2023

  8. [8]

    N2g: A scalable approach for quantifying interpretable neuron representations in large language models, 2023 b

    Alex Foote, Neel Nanda, Esben Kran, Ionnis Konstas, and Fazl Barez. N2g: A scalable approach for quantifying interpretable neuron representations in large language models, 2023 b . URL https://arxiv.org/abs/2304.12918

    arXiv 2023

Show all 41 references
  1. [9]

    Stabilizing the lottery ticket hypothesis

    Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.01611, 2019

    1903 arXiv

  2. [10]

    The pile: An 800gb dataset of diverse text for language modeling, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https://arxiv.org/abs/2101.00027

    2020 arXiv

  3. [11]

    Scaling and evaluating sparse autoencoders, 2024

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024. URL https://arxiv.org/abs/2406.04093

    2024 arXiv

  4. [12]

    Gemini: A Family of Highly Capable Multimodal Models , 2024

    Gemini Team, Google . Gemini: A Family of Highly Capable Multimodal Models , 2024. URL https://arxiv.org/abs/2312.11805

    2024 arXiv

  5. [13]

    Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai, 2024

    Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, L...

    2024 arXiv

  6. [14]

    Openwebtext corpus

    Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  7. [15]

    Sparse autoencoders find highly interpretable features in language models

    Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=F76bwRSLeK

    2024

  8. [16]

    Two sparsities are better than one: unlocking the performance benefits of sparse–sparse networks

    Kevin Hunter, Lawrence Spracklen, and Subutai Ahmad. Two sparsities are better than one: unlocking the performance benefits of sparse–sparse networks. Neuromorphic Computing and Engineering, 2 0 (3): 0 034004, jul 2022. doi:10.1088/2634-4386/ac7c8a. URL https://dx.doi.org/10.1...

    2022 doi

  9. [17]

    Curt Tigges Joseph Bloom and David Chanin. Saelens. https://github.com/jbloomAus/SAELens, 2024

    2024

  10. [18]

    Lazzaro, S

    J. Lazzaro, S. Ryckebusch, M.A. Mahowald, and C. A. Mead. Winner-take-all networks of o(n) complexity. In D. Touretzky (ed.), Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmann, 1988. URL https://proceedings.neurips.cc/paper_files/paper/1988/file/a8f1...

    1988

  11. [19]

    Lecun, L

    Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 0 (11): 0 2278--2324, 1998. doi:10.1109/5.726791

    1998 doi

  12. [20]

    Deep learning

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 0 (7553): 0 436--444, May 2015. ISSN 1476-4687. doi:10.1038/nature14539. URL https://doi.org/10.1038/nature14539

    2015 doi

  13. [21]

    Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang,...

    2024 arXiv

  14. [22]

    Hou, and Max Tegmark

    Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y. Hou, and Max Tegmark. Kan: Kolmogorov-arnold networks, 2024. URL https://arxiv.org/abs/2404.19756

    2024 arXiv

  15. [23]

    Majani, Ruth Erlanson, and Yaser Abu-Mostafa

    E. Majani, Ruth Erlanson, and Yaser Abu-Mostafa. On the k-winners-take-all network. In D. Touretzky (ed.), Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmann, 1988. URL https://proceedings.neurips.cc/paper_files/paper/1988/file/6c4b761a28b734fe93831e3...

    1988

  16. [24]

    Transformer circuit evaluation metrics are not robust

    Joseph Miller, Bilal Chughtai, and William Saunders. Transformer circuit evaluation metrics are not robust. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=zSf8PJyQb2

    2024

  17. [25]

    GPT-4o mini: advancing cost-efficient intelligence , 2024

    OpenAI . GPT-4o mini: advancing cost-efficient intelligence , 2024. URL https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence

    2024

  18. [26]

    GPT-4 Technical Report , 2024

    OpenAI. GPT-4 Technical Report , 2024. URL https://arxiv.org/abs/2303.08774

    2024 arXiv

  19. [27]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pp.\ 1135–1144, New York, NY, USA, 2...

    2016

  20. [28]

    Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, 2019

    Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, 2019. URL https://arxiv.org/abs/1811.10154

    2019 arXiv

  21. [29]

    SAEBench: A Comprehensive Benchmark for Sparse Autoencoders - Dec 2024 , January 2025

    SAE Bench . SAEBench: A Comprehensive Benchmark for Sparse Autoencoders - Dec 2024 , January 2025. URL https://www.neuronpedia.org/sae-bench/info

    2024

  22. [30]

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter . In 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing @ NeurIPS 2019, 2019. URL http://arxiv.org/abs/1910.01108

    2019 arXiv

  23. [31]

    The neural lasso: Local linear sparsity for interpretable explanations

    Andrew Slavin, Isaac Lage, and Finale Doshi-Velez. The neural lasso: Local linear sparsity for interpretable explanations. 2018. URL https://api.semanticscholar.org/CorpusID:52953225

    2018

  24. [32]

    Dropout: A simple way to prevent neural networks from overfitting

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15 0 (56): 0 1929--1958, 2014. URL http://jmlr.org/papers/v15/srivastava14a.html

    1929

  25. [33]

    Tamkin, M

    A. Tamkin, M. Taufeeque, and N. D. Goodman. Codebook features: Sparse and discrete interpretability for neural networks. arXiv preprint arXiv:2310.17230, 2023

    2023 arXiv

  26. [34]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Info...

    2017

  27. [35]

    Interpretability in the wild: a circuit for indirect object identification in GPT -2 small

    Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT -2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevi...

    2023

  28. [36]

    MMLU -pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU -pro: A more robust and challenging multi-task language under...

    2024

  29. [37]

    Understanding deep learning requires rethinking generalization, 2017

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization, 2017. URL https://arxiv.org/abs/1611.03530

    2017 arXiv

  30. [38]

    Towards best practices of activation patching in language models: Metrics and methods

    Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Hf17y6u9BC

    2024

  31. [39]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  32. [40]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  33. [41]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.