Pith. sign in

REVIEW 2 major objections 3 minor 35 references

Understanding sparse autoencoder scaling in the presence of feature manifolds

T0 review · 2 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Sparse autoencoder scaling under feature manifolds reduces to a latent-allocation problem, and one exponent comparison decides whether SAEs discover new features or spend latents tiling common manifolds.

desk verdict A clean, honest adaptation of Brill's allocation model to SAE scaling that identifies a plausible pathological regime but stops short of showing it occurs in real SAEs; worth refereeing, not worth over-trusting. read the letter →

arxiv 2509.02565 v2 pith:VE7YB4QM submitted 2025-09-02 cs.LG

Eric J. Michaud , Liv Gorton , Tom McGrath This is my paper · ORCID
classification cs.LG
keywords sparseautoencodersscalinglawsfeaturemanifoldslatentallocationdiscoveryinterpretabilityneuraldictionarylearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to explain SAE scaling laws as the outcome of allocating a fixed number of latents among features, and to show that feature manifolds (multi-dimensional features) can make that allocation pathological. Under the model, feature frequencies follow a power law $p_i \propto i^{-(1+\alpha)}$, each feature's loss with $n_i$ latents follows $L_i(n_i) \propto n_i^{-\beta}$, and the SAE's total loss is $\sum_i p_i L_i(n_i)$. Solving the allocation problem yields a sharp regime switch: when $\beta < \alpha$, the optimal SAE keeps shaving loss by tiling common feature manifolds, and the number of features it discovers grows sublinearly with the number of latents. The authors show that such shallow per-feature loss curves are plausible for hollow hyperspheres but plateau when features have radial intensity variation, and they leave open whether real SAEs are in the pathological regime. This matters because SAEs are a main tool for interpreting neural networks: if many latents are silently spent re-tiling common manifolds, rare features remain undiscovered.

What carries the argument

The carrying object is the latent-allocation reduction: under assumptions of feature-specific latents and orthogonal feature subspaces, SAE training is equivalent to choosing $n_i$ latents per feature to minimize $\sum_i p_i L_i(n_i)$ subject to $\sum_i n_i = N$. Solving the continuous relaxation gives $n_i \propto i^{-(1+\alpha)/(1+\beta)}$ for power-law frequencies and loss curves. The decisive identity is the comparison of $\alpha$ and $\beta$: when $\beta < \alpha$, the cutoff feature index grows like $N^{(1+\beta)/(1+\alpha)}$, so the discovery count $D(N)$ grows sublinearly; when $\alpha < \beta$, $D(N)$ grows linearly with $N$. This machinery converts a question about neural interpretability into a measurable relationship between two power-law exponents.

What would settle it

Train an SAE on a synthetic mixture containing one common hypersphere manifold plus many rare discrete features, with parameters chosen so $\beta < \alpha$, and measure the discovered-feature count $D(N)$ as width grows by orders of magnitude: linear growth in $N$ would falsify the sublinear-discovery prediction. On real data, directly measuring $L_i(n_i)$ for identified features at several widths would show whether any manifold sustains the slow decay needed for the pathological regime.

Watch

Extended reading notes

Core claim

The paper's central claim is that, under explicit idealizing assumptions, SAE training is exactly a latent-allocation problem, and feature manifolds introduce a pathology the discrete-feature picture misses. When feature frequencies obey $p_i \propto i^{-(1+\alpha)}$ and per-feature loss obeys $L_i(n_i) \propto n_i^{-\beta}$, the optimal allocation solves a Lagrange-multiplier problem with $n_i \propto i^{-(1+\alpha)/(1+\beta)}$. Total loss then scales as $L(N) \propto N^{-\min(\alpha,\beta)}$, but discovered features behave differently in the two regimes: $D(N) \propto N$ when $\alpha < \beta$, while $D(N) \propto N^{(1+\beta)/(1+\alpha)}$ when $\beta < \alpha$. In the latter case $D(N)/N \to 0$, so an SAE keeps improving reconstruction loss while its latent budget is overwhelmingly spent on common feature manifolds rather than on rarer features. Toy-manifold experiments show that the shape of $L_i(n_i)$ depends on geometry, and the real-network question is left open.

Load-bearing premise

The model treats each feature as a separate, non-overlapping piece of activation space served by its own dedicated latents, so total loss decomposes into per-feature terms; if real SAEs absorb features or let latents mix across overlapping feature subspaces, that decomposition no longer holds, and the authors explicitly admit this happens in practice.

Editorial extensions

If this is right

  • A clean power-law SAE loss curve does not reveal which regime the SAE is in; because total loss falls as $N^{-\min(\alpha,\beta)}$ in both cases, scaling behavior alone cannot detect pathological tiling.
  • Even very large SAEs can have most latents allocated to a few common feature manifolds while rare features go undiscovered, decoupling reconstruction quality from interpretability coverage.
  • Per-feature loss curves are geometry-dependent: hollow hyperspheres can absorb thousands of latents, while radial intensity variation makes loss plateau near $2d_i$ latents and would suppress the pathology.
  • Latent-firing-frequency slopes measured on real SAEs give tentative $\alpha$ estimates around 0.5–0.7, but feature absorption and tiling distort the mapping from latents to true features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical diagnostic follows directly: fit per-feature loss curves at small widths for frequent features and measure the feature-frequency exponent; if the per-feature curve is shallower, scaling up will mostly re-tile manifolds.
  • The same allocation logic should apply to crosscoders and transcoders, whose features span shared layer spaces; manifold tiling could be more or less severe there.
  • If real-world features carry radial intensity variation, the benign regime holds; the open risk is ripples, where intrinsically low-dimensional manifolds wander through many dimensions, a case the toy experiments do not cover.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. This paper adapts Brill's (2024) capacity-allocation model to sparse autoencoders. It assumes activations are sums of sparse features living in orthogonal subspaces, and that an SAE assigns latents to features so that total loss is Σ_i p_i L_i(n_i) with Σ_i n_i = N. Assuming feature frequencies p_i ∝ i^{-(1+α)} and identical per-feature loss curves L_i(n) = n^{-β}, the authors derive two scaling regimes: when β < α, L(N) ∝ N^{-β} and the number of discovered features grows only as D(N) ∝ N^{(1+β)/(1+α)}, so D(N)/N → 0; when α < β, D(N) ∝ N and L(N) ∝ N^{-α}. They support the per-feature loss curves with SAE experiments on hyperspheres and spherical shells, run a numerical simulation of the allocation model, and present preliminary measurements of latent firing frequencies and decoder-neighbor cosine similarities on Gemma Scope and Inception-v1 SAEs. The paper concludes that whether real SAEs are in the pathological regime is unresolved.

Significance. If the model's assumptions hold, the derivation in Appendix B is clean and yields a crisp, falsifiable scaling prediction that connects SAE scaling to the geometry of feature manifolds. The paper is explicit about the idealized nature of its assumptions and does not overclaim empirical confirmation. The toy experiments directly show that hollow hyperspheres permit gradual loss reduction while radial-variation manifolds saturate, which is a useful empirical fact. The framework provides a concrete vocabulary (α, β, D(N)) for future work on SAE scaling. Its main value is as a formal model and a source of hypotheses; its main weakness is the unvalidated transfer from the allocation model to real SAE training.

major comments (2)
  1. [§2.2, Eq. (1)] The allocation objective in Eq. (1) is derived from three premises: feature-specific latents, decoder columns lying in each feature's subspace, and mutually orthogonal feature subspaces. The paper explicitly concedes that feature absorption [27] and non-orthogonality violate these assumptions in practice. This matters because the per-feature loss curves L_i(n_i) used in the allocation model are measured on isolated manifolds (Figure 3); if in a joint SAE latents mix features or absorb them, these curves are not the actual marginal cost curves inside the SAE. I recommend adding a direct empirical check on synthetic data: train a single SAE on data containing one manifold feature plus several discrete features, then measure (i) the number of latents whose decoder direction lies in each feature subspace, (ii) whether latents fire only when their feature is active, and (iii) whether the per-feature loss as a function of allocated latents matches the isolated curves. Without such a check, the paper's prediction of a pathological regime, while correct for the idealized model, is not yet connected to actual SAE training.
  2. [§2.5 and Appendix B.3/B.4] The asymptotic results D(N) ∝ N^{(1+β)/(1+α)} (β < α) and D(N) ∝ N (α < β) assume that every feature has the same per-feature loss L_i(n_i) = n_i^{-β} for all n_i, with no saturation. The paper's own experiments in Figure 3 and Appendix A.1 show that L(n) curves can plateau sharply (spherical shells at n ≈ 2d_i), and even the hypersphere curves are only measured over 10^2 to 10^4 latents, with β ≈ 0.05 for dimensions 6–8. If a real manifold's L(n) eventually saturates, the infinite-N limit in Section 2.5 no longer follows: once the manifold is fully tiled, additional latents would go to discrete features and D(N)/N would recover. I ask for an analysis of the allocation problem with saturating per-feature loss curves (e.g., L(n) = n^{-β} + c, or a sharp cutoff at some n_max), to characterize when the pathological regime is realized and how large N must be before it sets in. The current numerical simulation in Figure 2 (right) uses a non-saturating power-law manifold plus step-function features, so it does not address this.
minor comments (3)
  1. [Figure 2 (right) caption] The caption says a 'simulated SAE' discovers only 3 million features, but the simulation optimizes the allocation objective Σ_i p_i L_i(n_i) rather than training an SAE; please rename it a 'simulation of the allocation model' to avoid implying end-to-end SAE training.
  2. [Appendix A.1, Figure 4] The paragraph estimating α from Gemma Scope latent firing frequencies notes the distortion caused by feature absorption and compositional features, but the figure may still be read as evidence about α; add an explicit statement that this estimate cannot be used to test the model because the mapping from features to latents is exactly what the model is trying to characterize.
  3. [§2.3] The definition 'L(n) = 1 if n_i < 1 else λ' uses n_i inconsistently with the function argument n; use a single symbol (e.g., L_i(n_i)) throughout the section.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the scaling derivation is a self-contained allocation model with alpha and beta as stated inputs; self-citations are peripheral.

full rationale

The paper's central result (Sections 2.3–2.5 and Appendix B) is a mathematical derivation from explicitly stated assumptions: feature frequencies pi ~ i^{-(1+alpha)}, per-feature loss curves Li(n) ~ n^{-beta}, and an allocation objective sum_i p_i Li(n_i) subject to sum_i n_i = N. The exponents alpha and beta are inputs, not outputs fitted to the derived scaling laws, so the derived L(N) and D(N) expressions are not predictions equivalent to their inputs by construction. The numerical simulation in Figure 2 merely instantiates the derived allocation regimes and does not present itself as an independent empirical test. The paper also explicitly declines to claim that real SAEs are in the pathological regime: 'Unfortunately, we do not resolve the question of whether SAEs are in this pathological scaling regime in practice.' The admitted idealizations in Section 2.2 (feature-specific latents, orthogonal subspaces, no feature absorption) weaken the transfer from model to real SAEs, but this is a limitation, not a circularity; the derivation is internally coherent under those stated assumptions. The self-citations are not load-bearing: [28] (Michaud et al., the authors' prior 'quanta' model) is invoked only as a conceptual analogue ('this model of SAE scaling mirrors the quanta model of neural scaling'), and [22] (Engels, Michaud, et al.) supports the empirical premise that multidimensional features exist, which is used as motivation and tested on synthetic manifolds rather than derived from it. The capacity-allocation machinery is transparently attributed to Brill (2024), an external source, and the paper notes its derivations 'closely follow those in Brill (2024)'. No step reduces to a self-citation chain, no fitted parameter is renamed as a prediction, and no known result is repackaged as new. Overall, the derivation is self-contained and honest about its scope.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. Its load-bearing assumptions are the sparse orthogonal-subspace data model, feature-specific latents, and power-law forms for p_i and L_i(n). The free parameters alpha and beta are inputs to the model and are not derived from first principles.

free parameters (2)
  • alpha (feature frequency power-law exponent) = Simulation uses 0.5; Gemma Scope estimate 0.5 to 0.7
    Controls the tail of feature frequencies p_i proportional to i^(-(1+alpha)). The regime split beta < alpha versus alpha < beta is the paper's central dichotomy, but alpha is not derived from first principles.
  • beta (per-feature loss power-law exponent) = Measured near 0.05 for 6 to 8 dimensional hollow hyperspheres; not fixed in the model
    Controls the per-feature loss curve L_i(n) proportional to n^(-beta). The paper measures beta on synthetic manifolds but does not derive it, and the real-world value is unknown.
assumptions (4)
  • domain assumption Activations are generated as x = sum S_i f_i with sparse features on orthogonal subspaces (multi-dimensional linear representation hypothesis).
    Section 2.1 and Section 2.2, inherited from refs [24,26]. Orthogonality is load-bearing for the Pythagorean loss decomposition in Appendix B.1.
  • domain assumption Each SAE latent is feature-specific: it fires only when its assigned feature fires, and its decoder direction lies in that feature's subspace.
    Section 2.2. Acknowledged as violated in practice by feature absorption [27]; without it, per-feature loss curves L_i(n_i) need not exist.
  • domain assumption Feature frequencies decay as a power law p_i proportional to i^(-(1+alpha)) and per-feature loss curves are identical power laws L_i(n) = n^(-beta).
    Sections 2.5 and B.2. The analytic regime split is derived from these forms, and Appendix A.1 notes that real data may not even be a power law.
  • standard math Continuous Lagrange relaxation approximates the integer latent allocation problem.
    Appendix B.2 replaces discrete latents with continuous n_i. The cutoff argument is heuristic and integer rounding effects are not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding sparse autoencoder scaling in the presence of feature manifolds." pith.science (2026). https://pith.science/paper/VE7YB4QM

@misc{pith2026250902565,
  author       = {Pith},
  title        = {Pith review of: Understanding sparse autoencoder scaling in the presence of feature manifolds},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VE7YB4QM}},
  note         = {Machine review of arXiv:2509.02565}
}
read the original abstract

Sparse autoencoders (SAEs) model the activations of a neural network as linear combinations of sparsely occurring directions of variation (latents). The ability of SAEs to reconstruct activations follows scaling laws w.r.t. the number of latents. In this work, we adapt a capacity-allocation model from the neural scaling literature (Brill, 2024) to understand SAE scaling, and in particular, to understand how "feature manifolds" (multi-dimensional features) influence scaling behavior. Consistent with prior work, the model recovers distinct scaling regimes. Notably, in one regime, feature manifolds have the pathological effect of causing SAEs to learn far fewer features in data than there are latents in the SAE. We provide some preliminary discussion on whether or not SAEs are in this pathological regime in the wild.

Figures

Figures reproduced from arXiv: 2509.02565 by the authors.

Figure 1
Figure 1. SAE scaling on a toy feature manifold S 1 . We train ReLU SAEs with an L1 sparsity penalty to reconstruct points on the circle S 1 ⊆ R 2 . We find that SAEs can slightly reduce their total loss by “tiling” the manifold with more sparsely activating latents. For SAEs with 4, 8, and 24 latents, we show the data (S 1 ) in grey, the SAE’s reconstruction in black, the decoder latent directions as arrows, and indicate the… view at source ↗
Figure 2
Figure 2. Left: Application of Brill’s (2024) [25] capacity-allocation model to SAE scaling. Right: numerical simulation of SAE scaling when the most frequently occurring feature is a manifold with loss scaling as L(n) ∝ n −β and all other features are discrete. We see that if β ≪ α, then a simulated SAE with 100 million latents discovers only 3 million features. With total loss P i piL(ni) and N latents in our SAE we see tha… view at source ↗
Figure 3
Figure 3. ReLU SAE scaling on individual toy feature manifolds, showing how [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Frequencies at which latents fire in gemma scope SAEs, sorted by frequency. We measure [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Measured slopes of Li curves for ReLU L1 SAEs trained on hyperspheres. Lastly, we note that we are unsure how “ripples” [29] in feature manifolds could affect SAE scaling on them. If a feature manifold is intrinsically low dimensional, but ripples through a large numbe…
Figure 6
Figure 6. Figure 6: JumpReLU SAE scaling on individual toy feature manifolds, showing how [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Distribution over cosine similarities between decoder vectors and their nearest neighbor. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Distribution over pairwise cosine similarities for Inception V1. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 13 canonical work pages

  1. [27]

    A is for absorption: Studying feature splitting and absorption in sparse autoen- coders

    David Chanin, James Wilken-Smith, Tom´aˇs Dulka, Hardik Bhatnagar, Satvik Golechha, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoen- coders. arXiv preprint arXiv:2409.14507, 2024

    arXiv 2024

  2. [1]

    Sparse autoencoder

    Andrew Ng et al. Sparse autoencoder. CS294A Lecture notes, 72(2011):1–19, 2011

    2011

  3. [2]

    Sparse autoen- coders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023

    arXiv 2023

  4. [3]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and ...

    2023

  5. [4]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupr ´e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024

    arXiv 2024

  6. [5]

    Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders

    Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, J´anos Kram´ar, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435, 2024

    arXiv 2024

  7. [6]

    Batchtopk sparse autoencoders

    Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. arXiv preprint arXiv:2412.06410, 2024

    arXiv 2024

  8. [7]

    Archetypal sae: Adaptive and stable dictionary learning for concept extraction in large vision models

    Thomas Fel, Ekdeep Singh Lubana, Jacob S Prince, Matthew Kowal, Victor Boutin, Isabel Papadimitriou, Binxu Wang, Martin Wattenberg, Demba Ba, and Talia Konkle. Archetypal sae: Adaptive and stable dictionary learning for concept extraction in large vision models. arXiv preprint arXiv:2502.12892, 2025

    arXiv 2025

Show all 35 references
  1. [8]

    Incorporating hierarchical semantics in sparse autoencoder architectures

    Mark Muchane, Sean Richardson, Kiho Park, and Victor Veitch. Incorporating hierarchical semantics in sparse autoencoder architectures. arXiv preprint arXiv:2506.01197, 2025

    2025 arXiv

  2. [9]

    From flat to hierarchical: Extracting sparse representations with matching pursuit

    Val´erie Costa, Thomas Fel, Ekdeep Singh Lubana, Bahareh Tolooshams, and Demba Ba. From flat to hierarchical: Extracting sparse representations with matching pursuit. arXiv preprint arXiv:2506.03093, 2025

    2025

  3. [10]

    Transcoders find interpretable llm feature circuits

    Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable llm feature circuits. Advances in Neural Information Processing Systems, 37:24375–24410, 2024. 5

    2024

  4. [11]

    Sparse crosscoders for cross-layer features and model diffing, October 2024

    Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christo- pher Olah. Sparse crosscoders for cross-layer features and model diffing, October 2024. URL https://transformer-circuits.pub/2024/crosscoders/index.html. Research up- date

    2024

  5. [12]

    Group crosscoders for mechanistic analysis of symmetry

    Liv Gorton. Group crosscoders for mechanistic analysis of symmetry. arXiv preprint arXiv:2410.24184, 2024

    2024 arXiv

  6. [13]

    Robustly identifying concepts introduced during chat fine-tuning using crosscoders

    Julian Minder, Cl ´ement Dumas, Caden Juang, Bilal Chugtai, and Neel Nanda. Robustly identifying concepts introduced during chat fine-tuning using crosscoders. arXiv preprint arXiv:2504.02922, 2025

    2025

  7. [14]

    Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, A...

    2025

  8. [15]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua ...

    2024

  9. [16]

    Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J´anos Kram´ar, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147, 2024

    2024 arXiv

  10. [17]

    Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, A...

    2025

  11. [18]

    Scaling laws for dictionary learning, April 2024

    Jack Lindsey, Tom Conerly, Adly Templeton, Jonathan Marcus, and Tom Henighan. Scaling laws for dictionary learning, April 2024. URL https://transformer-circuits.pub/ 2024/april-update/index.html#scaling-laws. Circuits Updates — April 2024

    2024

  12. [19]

    The dark matter of neural networks? Transformer Circuits Thread, July 2024

    Chris Olah and Adam Jermyn. The dark matter of neural networks? Transformer Circuits Thread, July 2024. URL https://transformer-circuits.pub/2024/july-update/ index.html#dark-matter. Part of Circuits Updates - July 2024, Anthropic Interpretability Team

    2024

  13. [20]

    Recurrent neural networks learn to store and generate sequences using non-linear representations

    R´obert Csord´as, Christopher Potts, Christopher D Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-linear representations. arXiv preprint arXiv:2408.10920, 2024

    2024 arXiv

  14. [21]

    Projecting assumptions: The duality between sparse autoencoders and concept geometry

    Sai Sumedh R Hindupur, Ekdeep Singh Lubana, Thomas Fel, and Demba Ba. Projecting assumptions: The duality between sparse autoencoders and concept geometry. arXiv preprint arXiv:2503.01822, 2025

    2025

  15. [22]

    Not all language model features are one-dimensionally linear

    Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are one-dimensionally linear. arXiv preprint arXiv:2405.14860, 2024

    2024 arXiv

  16. [23]

    The origins of representation manifolds in large language models

    Alexander Modell, Patrick Rubin-Delanchy, and Nick Whiteley. The origins of representation manifolds in large language models. arXiv preprint arXiv:2505.18235, 2025. 6

    2025 arXiv

  17. [24]

    Decomposing the dark matter of sparse autoencoders

    Joshua Engels, Logan Riggs, and Max Tegmark. Decomposing the dark matter of sparse autoencoders. arXiv preprint arXiv:2410.14670, 2024

    2024 arXiv

  18. [25]

    Neural scaling laws rooted in the data distribution

    Ari Brill. Neural scaling laws rooted in the data distribution. arXiv preprint arXiv:2412.07942, 2024

    2024 arXiv

  19. [26]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposi...

    2022

  20. [28]

    The quantization model of neural scaling

    Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. Advances in Neural Information Processing Systems, 36:28699–28722, 2023

    2023

  21. [29]

    What is a linear representation? what is a multidimensional feature? Trans- former Circuits Thread, July 2024

    Chris Olah. What is a linear representation? what is a multidimensional feature? Trans- former Circuits Thread, July 2024. URL https://transformer-circuits.pub/2024/ july-update/index.html#linear-representations. Part of Circuits Updates - July 2024, Anthropic Interpretability Team

    2024

  22. [30]

    true features

    Demian Till. Do sparse autoencoders find “true features”? https://www.lesswrong. com/posts/QoR8noAB3Mp2KBA4B/do-sparse-autoencoders-find-true-features , 02 2024. Accessed: 2025-08-11

    2024

  23. [31]

    Evan Anders, Clement Neo, Jason Hoelscher-Obermaier, and Jessica N. Howard. Sparse autoencoders find composed features in small toy models. https://www.lesswrong.com/posts/a5wwqza2cY3W7L9cj/ sparse-autoencoders-find-composed-features-in-small-toy , 03 2024. Ac- cessed: 2025-08-11

    2024

  24. [32]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  25. [33]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  26. [34]

    Up- date on how we train saes

    Tom Conerly, Adly Templeton, Trenton Bricken, Jonathan Marcus, and Tom Henighan. Up- date on how we train saes. Transformer Circuits Thread, April 2024. URL https:// transformer-circuits.pub/2024/april-update/index.html#training-saes . Part of Circuits Updates - April 2024, An...

    2024

  27. [35]

    features

    Tom Conerly, Hoagy Cunningham, Adly Templeton, Jack Lindsey, Basil Hosmer, and Adam Jermyn. Dictionary learning optimization techniques. Transformer Circuits Thread, Jan- uary 2025. URL https://transformer-circuits.pub/2025/january-update/index. html#DL. Part of Circuits Updat...

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.