Pith. sign in

REVIEW 1 major objections 1 minor 31 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

T0 review · grok-4.3

Stable SAE features carry most reconstruction and prediction signal while unstable features concentrate in reproducible lower-rank subspaces across seeds.

2026-06-27 10:37 UTC pith:UH6ZYSVR

load-bearing objection Stable SAE features do most of the reconstruction and prediction work while unstable ones sit in reproducible low-rank subspaces, but the stable/unstable split rests on an unvalidated similarity threshold for cross-seed matching. the 1 major comments →

arxiv 2606.12138 v1 pith:UH6ZYSVR submitted 2026-06-10 cs.LG cs.AIcs.CL

Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders

classification cs.LG cs.AIcs.CL
keywords sparse autoencodersfeature stabilityseed dependencereproducible subspacesneural network interpretabilitydictionary learningactivation subspaces
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper defines per-feature stability as the probability that a similar feature reappears across independently trained sparse autoencoders. It finds that stable features account for nearly all reconstruction quality and downstream predictive utility, whereas unstable features contribute little on their own and are driven by low-frequency surface-form patterns. Geometrically, the unstable features vary individually from seed to seed but reliably occupy the same lower-dimensional subspaces, implying that training-seed differences largely reflect different choices of basis within a shared activation region rather than unrelated noise. Experiments across models, layers, and dictionary sizes, plus a controlled synthetic low-rank model, support this separation. Pooling distinct features from multiple seeds then produces SAEs with greater stability at comparable explained variance.

Core claim

Stable features carry most of the reconstruction- and prediction-relevant signal, while unstable features have weak marginal impact and are dominated by low-frequency surface-form triggers in both activation statistics and automatic explanations. Geometrically, unstable features are individually non-reproducible but concentrate in reproducible lower-rank subspaces, suggesting that seed dependence often reflects basis ambiguity within a shared region of activation space rather than pure noise. A controlled synthetic model makes this mechanism explicit, showing that low-rank ground-truth features can be recovered at the subspace level while remaining non-identifiable as individual SAE latents

What carries the argument

Per-feature reappearance probability estimated by matching similar features across independently trained SAEs, which isolates stable from unstable latents and reveals their grouping into shared lower-rank subspaces.

Load-bearing premise

The chosen similarity threshold and matching procedure for deciding whether two features from different SAEs count as similar correctly identifies meaningful correspondence.

What would settle it

Measure the principal angles or reconstruction overlap between the subspace spanned by unstable features from one SAE and the subspace spanned by unstable features from an independent SAE; close alignment across seeds would support the claim while large angles would contradict it.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Stable features account for the bulk of explained variance and downstream task performance.
  • Unstable features exhibit weak individual functional impact yet align with reproducible low-dimensional structure.
  • Pooling unique cross-seed features produces SAEs with higher stability while preserving explained variance.
  • Low-rank ground-truth signals remain recoverable at the subspace level even when individual latents are not identifiable across seeds.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Interpretation pipelines could restrict attention to stable features to reduce run-to-run variability without large loss of coverage.
  • Methods that learn subspaces directly rather than individual sparse directions might reduce seed dependence in practice.
  • The observed low-rank concentration may appear in other overcomplete dictionary-learning settings beyond SAEs.
  • The stability signal could serve as a diagnostic for choosing dictionary size or regularization strength.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The paper defines per-feature stability in SAEs as the estimated probability that a similar feature reappears across independently trained models with different seeds. A large-scale empirical study across seeds, models, layers, dictionary sizes and SAE variants reports a functional asymmetry: stable features account for the bulk of reconstruction and prediction performance while unstable features show weak marginal contributions and are driven by low-frequency surface-form patterns. Geometrically, unstable features are shown to concentrate in reproducible lower-rank subspaces. A controlled synthetic low-rank model demonstrates that individual latents can be non-identifiable across seeds while the underlying subspace remains recoverable. The authors also construct more stable SAEs by pooling unique cross-seed features.

Significance. If the reported asymmetry and subspace reproducibility hold, the work supplies a mechanistic account of seed dependence in SAEs that distinguishes basis ambiguity from noise and supplies a practical construction for improved stability. The scale of the empirical sweep, the explicit synthetic model that isolates the low-rank mechanism, and the constructive pooling result are concrete strengths that would be useful to the interpretability community.

major comments (1)
  1. [Methods (feature-stability definition and estimation)] The central partition into stable versus unstable features rests on an unvalidated similarity threshold and matching procedure used to compute reappearance probability. Because every downstream claim (functional asymmetry, subspace concentration, and the synthetic-model explanation) is conditioned on this partition, the manuscript must demonstrate that the reported effects are robust to reasonable variations in the threshold (e.g., cosine-similarity cutoffs) and to alternative matching rules; without such analysis the empirical conclusions remain sensitive to an arbitrary modeling choice.
minor comments (1)
  1. [Abstract] The abstract refers to 'automatic explanations' without indicating the method used to generate or compare them; the main text should supply the precise procedure and any controls applied.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for highlighting the importance of validating the feature-stability partition. We address the single major comment below.

read point-by-point responses
  1. Referee: The central partition into stable versus unstable features rests on an unvalidated similarity threshold and matching procedure used to compute reappearance probability. Because every downstream claim (functional asymmetry, subspace concentration, and the synthetic-model explanation) is conditioned on this partition, the manuscript must demonstrate that the reported effects are robust to reasonable variations in the threshold (e.g., cosine-similarity cutoffs) and to alternative matching rules; without such analysis the empirical conclusions remain sensitive to an arbitrary modeling choice.

    Authors: We agree that the stability definition depends on a similarity threshold and matching rule whose sensitivity has not been fully documented. The original manuscript selects a cosine-similarity cutoff of 0.8 via preliminary checks but does not report systematic variation or alternative matchers. In the revision we will add an appendix containing (i) results for thresholds in {0.65, 0.7, 0.75, 0.8, 0.85, 0.9} and (ii) a comparison against bipartite (Hungarian) matching on the full similarity matrix. We will show that the functional asymmetry, subspace concentration, and synthetic-model conclusions remain qualitatively unchanged across these choices. This directly addresses the concern that downstream claims rest on an arbitrary modeling decision. revision: yes

Circularity Check

0 steps flagged

Observational study with independent cross-seed measurements; no derivation reduces to its inputs by construction

full rationale

The paper performs an empirical comparison of SAEs trained on independent random seeds. Feature stability is defined via a cross-seed reappearance probability computed from a similarity threshold and bipartite matching procedure; the subsequent claims about reconstruction error, downstream task impact, activation statistics, and subspace geometry are then measured on those partitioned sets using separate statistics that are not algebraically or statistically forced by the partitioning rule itself. No equation equates a reported functional or geometric quantity to a fitted parameter on the same data, no self-citation supplies a uniqueness theorem that forbids alternatives, and the synthetic low-rank model is presented as an illustrative construction rather than a fitted predictor of the real-data results. The analysis is therefore self-contained against external benchmarks of cross-run reproducibility.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claims rest on empirical cross-seed comparisons and a synthetic model; no free parameters, ad-hoc axioms, or invented entities are introduced beyond standard SAE training assumptions.

axioms (1)
  • domain assumption Neural network activations admit a useful sparse linear decomposition into features.
    Implicit background assumption of all SAE work invoked throughout the abstract.

pith-pipeline@v0.9.1-grok · 5815 in / 1287 out tokens · 36804 ms · 2026-06-27T10:37:52.323337+00:00 · methodology

0 comments
read the original abstract

Sparse autoencoders (SAEs) are widely used to interpret neural network representations, but their utility depends on whether the learned features are reproducible across training runs. We study this question through \emph{feature stability}: for each SAE feature, we estimate the probability that a similar feature reappears in an independently trained SAE. This yields a scalable per-feature signal that separates stable from unstable features. In a large-scale study across seeds, models, layers, dictionary sizes, and SAE variants, we find a pronounced functional asymmetry: stable features carry most of the reconstruction- and prediction-relevant signal, while unstable features have weak marginal impact and are dominated by low-frequency surface-form triggers in both activation statistics and automatic explanations. Geometrically, unstable features are individually non-reproducible but concentrate in reproducible lower-rank subspaces, suggesting that seed dependence often reflects basis ambiguity within a shared region of activation space rather than pure noise. A controlled synthetic model makes this mechanism explicit, showing that low-rank ground-truth features can be recovered at the subspace level while remaining non-identifiable as individual SAE latents across seeds. Finally, by pooling unique cross-seed features, we construct more stable SAEs while preserving explained variance in this setting. Together, these results show that unstable features are not merely failed or noisy latents: they have weak individual functional impact, but reflect reproducible low-dimensional structure that standard SAEs resolve differently across seeds.

Figures

Figures reproduced from arXiv: 2606.12138 by Daniil Gavrilov, Daniil Laptev, Gleb Gerasimov, Nikita Balagansky, Timofei Rusalev, Vadim Kurochkin.

Figure 1
Figure 1. Figure 1: Feature reappearance across seeds in the main [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Token diversity for stable vs. unstable fea￾tures. Token entropy Hi with representative feature interpretations. matter. Concretely, we show that: (i) unstable fea￾tures activate less frequently and (on average) with smaller magnitude tails (as quantified in Section 5.1 and shown in Appendix B.3, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Impact of masking stable vs. unstable features on reconstruction and next-token loss. Left: explained variance (EV) under feature masking. Right: change in next-token loss under activation patching with masked￾feature reconstructions. We mask N stable features and 4N unstable features to approximately match expected active mass. Solid vs. dashed curves correspond to using reweighting vs. not using reweight… view at source ↗
Figure 4
Figure 4. Figure 4: shows that only a modest number of source SAEs is needed: the most-probable con￾struction quickly becomes dominated by high￾probability features as the pool grows. After tuning, these dictionaries recover near-baseline explained variance, while least-probable dictionaries remain 10 1 10 2 Number of SAEs 0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 Fraction of Unstable Features | | = 20463 EV = 0.928 | | = … view at source ↗
Figure 5
Figure 5. Figure 5: Explained variance of singular values of decoder submatrices versus SVD rank: within-seed (solid) and cross-seed transfer (dashed). This indi￾cates that the subspaces learned in one seed accurately approximate those in other seeds. ciently large N they saturate at approximately 0.73 for ε = 0 and 0.67 for ε = 0.1. Next, to compare decoder vectors across seeds in more detail, we consider the explained varia… view at source ↗
Figure 6
Figure 6. Figure 6: Synthetic low-rank model (d = 32, r = 2, k = 8). The first 80 ground-truth features are full-rank features and the last 20 lie in a shared rank-2 subspace. Full-rank features (blue) have near-perfect cross-seed reappearance probability and cosine similarity to their matched ground-truth features, whereas low-rank fea￾tures (red) do not. k dictionary rows, summing them with unit coef￾ficients, and training … view at source ↗
Figure 8
Figure 8. Figure 8: Dead-salmon control: stability on trained vs. random transformers. Fractions of stable and un￾stable features as a function of cosine matching thresh￾old θ, comparing SAEs trained on a trained GPT-2 ver￾sus a randomly initialized GPT-2 (same architecture and SAE setup). the reconstruction- and prediction-relevant signal and more often correspond to structural or com￾positional patterns, while unstable feat… view at source ↗
Figure 7
Figure 7. Figure 7: Unstable fraction vs. SAE training tokens. Fraction of unstable features in the main TopK setting as a function of total SAE training tokens. initialized GPT-2 (same architecture and SAE hy￾perparameters). Our stability metric sharply distin￾guishes these cases [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Endpoint fractions vs. cosine threshold θ. Fractions of stable and unstable features as a function of cosine matching threshold θ for several SAE types in the same base-model setting. GPT-2 setting [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Activation frequency and conditional mean magnitude for stable vs. unstable features. Un￾stable features are concentrated at lower frequencies, while stable features exhibit a heavier high-magnitude tail. We use the natural logarithm. Features that never activate on the evaluation set are excluded from entropy plots. B.5 Masking protocol, uncertainty, and reweighting For a masked index set M ⊆ {1, . . . ,… view at source ↗
Figure 11
Figure 11. Figure 11: Explained variance during brief tuning of SAEs constructed from different feature subsets. Initialization from the most probable features recov￾ers nearly the same explained variance as the original SAE, the equiprobable construction performs somewhat worse, and the least-probable construction lags substan￾tially behind. Metric Standard Most-probable Sparse Probing (Top-1) 0.656 0.673 AutoInterp (mean) 0.… view at source ↗
Figure 14
Figure 14. Figure 14: Routes to SAE stability in the EV– instability plane. Orange points compare SAE archi￾tecture/objective variants, including TopK, JumpReLU, Vanilla, and HierarchicalTopK. Green points vary the Mahalanobis/whitening interpolation for TopK SAEs. Blue points show most-probable feature-pool SAEs af￾ter post-training with different numbers of source SAEs; rightmost blue point corresponds to the post-trained To… view at source ↗
Figure 15
Figure 15. Figure 15: Feature reappearance probabilities before and after post-training. The most-probable construc￾tion initializes from relatively high-probability pooled features, but post-training shifts some features toward lower reappearance probabilities. Lower-stability direc￾tions can therefore be reconstruction-useful rather than mere random artifacts. tures using automatic feature interpretation and explanation text… view at source ↗
Figure 17
Figure 17. Figure 17: F1 score for classification separating un￾stable features from all others. Solid curves report within-seed evaluation, while dashed curves report trans￾fer of the classifier to a different seed. F Additional Details for Geometric Analysis [PITH_FULL_IMAGE:figures/full_fig_p018_17.png] view at source ↗
Figure 19
Figure 19. Figure 19: reports the corresponding effective￾rank and cross-seed subspace-similarity diagnos￾tics. 12 13 14 15 16 17 18 19 20 Effective Rank 0 1 2 3 4 5 6 7 8 Count Full-rank Low-rank 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Projection Similarity 0 5 10 15 20 25 30 Count Full-rank Low-rank Random baseline = 0.625 [PITH_FULL_IMAGE:figures/full_fig_p019_19.png] view at source ↗
Figure 18
Figure 18. Figure 18: Additional synthetic settings. The same full-rank versus low-rank split appears across multiple values of the subspace rank r and sparsity level k: full￾rank features retain high reappearance probabilities and high cosine similarity to ground truth, whereas low-rank features do not. Intuitively, EV(a→s) SVD (r) measures how well the top￾r singular subspace learned in seed a explains the feature subspace i… view at source ↗
Figure 21
Figure 21. Figure 21: shows that stable SAE features become more prevalent with layer depth and are much more likely than unstable features to admit a next-layer 0 1 2 3 4 5 6 7 8 9 10 11 Layer 0.0 0.2 0.4 0.6 0.8 Fraction of Features Feature type Stable Unstable 0→1 1→2 2→3 3→4 4→5 5→6 6→7 7→8 8→9 9→10 10→11 Layer Transition 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of Features with Next-layer Match 0.001 0.002 0.005 0.005 0.007 0.017… view at source ↗
Figure 22
Figure 22. Figure 22: Stable fraction vs. SAE training tokens. Fraction of stable features in the main TopK setting as a function of total SAE training tokens. 0.0 0.2 0.4 0.6 0.8 1.0 Autointerp Score 0 1000 2000 3000 4000 5000 Count Random Model Baseline SAE Mean Score: 0.737 Mean Score: 0.818 [PITH_FULL_IMAGE:figures/full_fig_p021_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Automatic interpretation scores for SAEs trained on trained vs. random base models. High de￾tection scores are achievable even on random-model ac￾tivations, despite the absence of cross-seed reproducibil￾ity. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_23.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 11 canonical work pages · 2 internal anchors

  1. [1]

    2026 , eprint=

    Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learning , author=. 2026 , eprint=

  2. [2]

    2024 , eprint=

    Anisotropy Is Inherent to Self-Attention in Transformers , author=. 2024 , eprint=

  3. [3]

    Train One Sparse Autoencoder Across Multiple Sparsity Budgets to Preserve Interpretability and Accuracy

    Balagansky, Nikita and Aksenov, Yaroslav and Laptev, Daniil and Kurochkin, Vadim and Gerasimov, Gleb and Koriagin, Nikita and Gavrilov, Daniil. Train One Sparse Autoencoder Across Multiple Sparsity Budgets to Preserve Interpretability and Accuracy. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v...

  4. [4]

    Sparse autoencoders trained on the same data learn different features

    Sparse Autoencoders Trained on the Same Data Learn Different Features , author =. 2025 , eprint =. doi:10.48550/arXiv.2501.16615 , url =

  5. [5]

    The Thirteenth International Conference on Learning Representations , year =

    Sparse Autoencoders Do Not Find Canonical Units of Analysis , author =. The Thirteenth International Conference on Learning Representations , year =

  6. [6]

    Prince and Matthew Kowal and Victor Boutin and Isabel Papadimitriou and Binxu Wang and Martin Wattenberg and Demba E

    Thomas Fel and Ekdeep Singh Lubana and Jacob S. Prince and Matthew Kowal and Victor Boutin and Isabel Papadimitriou and Binxu Wang and Martin Wattenberg and Demba E. Ba and Talia Konkle , booktitle =. Archetypal. 2025 , url =

  7. [7]

    arXiv preprint arXiv:2505.16077 , year=

    Ensembling Sparse Autoencoders , author =. 2025 , eprint =. doi:10.48550/arXiv.2505.16077 , url =

  8. [8]

    Taming Polysemanticity in

    Siyu Chen and Heejune Sheen and Xuyuan Xiong and Tianhao Wang and Zhuoran Yang , year =. Taming Polysemanticity in. doi:10.48550/arXiv.2506.14002 , url =. 2506.14002 , archivePrefix =

  9. [9]

    2025 , eprint =

    Seonglae Cho and Harryn Oh and Donghyun Lee and Luis Eduardo Rodrigues Vieira and Andrew Bermingham and Ziad. 2025 , eprint =. doi:10.48550/arXiv.2506.17673 , url =

  10. [10]

    2025 , eprint =

    Enforcing Orderedness to Improve Feature Consistency , author =. 2025 , eprint =. doi:10.48550/arXiv.2512.02194 , url =

  11. [11]

    arXiv preprint arXiv:2501.17727 , year=

    Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers , author =. 2025 , eprint =. doi:10.48550/arXiv.2501.17727 , url =

  12. [12]

    Do Sparse Autoencoders Capture Concept Manifolds?

    Do Sparse Autoencoders Capture Concept Manifolds? , author =. 2026 , eprint =. doi:10.48550/arXiv.2604.28119 , url =

  13. [13]

    Evolution of

    Daniel Balcells and Benjamin Lerner and Michael Oesterle and Ediz Ucar and Stefan Heimersheim , year =. Evolution of. doi:10.48550/arXiv.2410.08869 , url =. 2410.08869 , archivePrefix =

  14. [14]

    The Thirteenth International Conference on Learning Representations , year =

    Mechanistic Permutability: Match Features Across Layers , author =. The Thirteenth International Conference on Learning Representations , year =

  15. [15]

    The Thirteenth International Conference on Learning Representations , year =

    Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures , author =. The Thirteenth International Conference on Learning Representations , year =

  16. [16]

    2026 , howpublished =

    Benchmarking Cross-Seed Feature Correspondence in Sparse Autoencoders , author =. 2026 , howpublished =

  17. [17]

    2025 , eprint =

    Analyze Feature Flow to Enhance Interpretation and Steering in Language Models , author =. 2025 , eprint =. doi:10.48550/arXiv.2502.03032 , url =

  18. [18]

    2025 , journal =

    SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability , author =. 2025 , journal =

  19. [19]

    , title =

    Kuhn, Harold W. , title =. Naval Research Logistics Quarterly , volume =. 1955 , doi =

  20. [20]

    Giornale dell'Istituto Italiano degli Attuari , volume =

    Glivenko, Valery , title =. Giornale dell'Istituto Italiano degli Attuari , volume =

  21. [21]

    Giornale dell'Istituto Italiano degli Attuari , volume =

    Cantelli, Francesco Paolo , title =. Giornale dell'Istituto Italiano degli Attuari , volume =

  22. [22]

    The Thirteenth International Conference on Learning Representations , year=

    Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures , author=. The Thirteenth International Conference on Learning Representations , year=

  23. [23]

    2025 , journal =

    The Dead Salmons of AI Interpretability , author =. 2025 , journal =

  24. [24]

    2025 , journal =

    Sparse Autoencoders Can Interpret Randomly Initialized Transformers , author =. 2025 , journal =

  25. [25]

    2025 , eprint=

    Data Whitening Improves Sparse Autoencoder Learning , author=. 2025 , eprint=

  26. [26]

    Sparse coding with an overcomplete basis set: A strategy employed by

    Olshausen, Bruno A and Field, David J , journal=. Sparse coding with an overcomplete basis set: A strategy employed by. 1997 , publisher=

  27. [27]

    Transformer Circuits Thread , volume=

    Towards monosemanticity: Decomposing language models with dictionary learning , author=. Transformer Circuits Thread , volume=

  28. [28]

    Sparse Autoencoders Find Highly Interpretable Features in Language Models

    Sparse autoencoders find highly interpretable features in language models , author=. arXiv preprint arXiv:2309.08600 , year=

  29. [29]

    The Thirteenth International Conference on Learning Representations , year=

    Scaling and evaluating sparse autoencoders , author=. The Thirteenth International Conference on Learning Representations , year=

  30. [30]

    2024 , eprint=

    Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders , author=. 2024 , eprint=

  31. [31]

    2024 , eprint=

    BatchTopK Sparse Autoencoders , author=. 2024 , eprint=