REVIEW 1 major objections 1 minor 31 references
Reviewed by Pith at T0; open to challenge.
T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →
T0 review · grok-4.3
Stable SAE features carry most reconstruction and prediction signal while unstable features concentrate in reproducible lower-rank subspaces across seeds.
2026-06-27 10:37 UTC pith:UH6ZYSVR
load-bearing objection Stable SAE features do most of the reconstruction and prediction work while unstable ones sit in reproducible low-rank subspaces, but the stable/unstable split rests on an unvalidated similarity threshold for cross-seed matching. the 1 major comments →
Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Stable features carry most of the reconstruction- and prediction-relevant signal, while unstable features have weak marginal impact and are dominated by low-frequency surface-form triggers in both activation statistics and automatic explanations. Geometrically, unstable features are individually non-reproducible but concentrate in reproducible lower-rank subspaces, suggesting that seed dependence often reflects basis ambiguity within a shared region of activation space rather than pure noise. A controlled synthetic model makes this mechanism explicit, showing that low-rank ground-truth features can be recovered at the subspace level while remaining non-identifiable as individual SAE latents
What carries the argument
Per-feature reappearance probability estimated by matching similar features across independently trained SAEs, which isolates stable from unstable latents and reveals their grouping into shared lower-rank subspaces.
Load-bearing premise
The chosen similarity threshold and matching procedure for deciding whether two features from different SAEs count as similar correctly identifies meaningful correspondence.
What would settle it
Measure the principal angles or reconstruction overlap between the subspace spanned by unstable features from one SAE and the subspace spanned by unstable features from an independent SAE; close alignment across seeds would support the claim while large angles would contradict it.
If this is right
- Stable features account for the bulk of explained variance and downstream task performance.
- Unstable features exhibit weak individual functional impact yet align with reproducible low-dimensional structure.
- Pooling unique cross-seed features produces SAEs with higher stability while preserving explained variance.
- Low-rank ground-truth signals remain recoverable at the subspace level even when individual latents are not identifiable across seeds.
Where Pith is reading between the lines
- Interpretation pipelines could restrict attention to stable features to reduce run-to-run variability without large loss of coverage.
- Methods that learn subspaces directly rather than individual sparse directions might reduce seed dependence in practice.
- The observed low-rank concentration may appear in other overcomplete dictionary-learning settings beyond SAEs.
- The stability signal could serve as a diagnostic for choosing dictionary size or regularization strength.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper defines per-feature stability in SAEs as the estimated probability that a similar feature reappears across independently trained models with different seeds. A large-scale empirical study across seeds, models, layers, dictionary sizes and SAE variants reports a functional asymmetry: stable features account for the bulk of reconstruction and prediction performance while unstable features show weak marginal contributions and are driven by low-frequency surface-form patterns. Geometrically, unstable features are shown to concentrate in reproducible lower-rank subspaces. A controlled synthetic low-rank model demonstrates that individual latents can be non-identifiable across seeds while the underlying subspace remains recoverable. The authors also construct more stable SAEs by pooling unique cross-seed features.
Significance. If the reported asymmetry and subspace reproducibility hold, the work supplies a mechanistic account of seed dependence in SAEs that distinguishes basis ambiguity from noise and supplies a practical construction for improved stability. The scale of the empirical sweep, the explicit synthetic model that isolates the low-rank mechanism, and the constructive pooling result are concrete strengths that would be useful to the interpretability community.
major comments (1)
- [Methods (feature-stability definition and estimation)] The central partition into stable versus unstable features rests on an unvalidated similarity threshold and matching procedure used to compute reappearance probability. Because every downstream claim (functional asymmetry, subspace concentration, and the synthetic-model explanation) is conditioned on this partition, the manuscript must demonstrate that the reported effects are robust to reasonable variations in the threshold (e.g., cosine-similarity cutoffs) and to alternative matching rules; without such analysis the empirical conclusions remain sensitive to an arbitrary modeling choice.
minor comments (1)
- [Abstract] The abstract refers to 'automatic explanations' without indicating the method used to generate or compare them; the main text should supply the precise procedure and any controls applied.
Simulated Author's Rebuttal
We thank the referee for highlighting the importance of validating the feature-stability partition. We address the single major comment below.
read point-by-point responses
-
Referee: The central partition into stable versus unstable features rests on an unvalidated similarity threshold and matching procedure used to compute reappearance probability. Because every downstream claim (functional asymmetry, subspace concentration, and the synthetic-model explanation) is conditioned on this partition, the manuscript must demonstrate that the reported effects are robust to reasonable variations in the threshold (e.g., cosine-similarity cutoffs) and to alternative matching rules; without such analysis the empirical conclusions remain sensitive to an arbitrary modeling choice.
Authors: We agree that the stability definition depends on a similarity threshold and matching rule whose sensitivity has not been fully documented. The original manuscript selects a cosine-similarity cutoff of 0.8 via preliminary checks but does not report systematic variation or alternative matchers. In the revision we will add an appendix containing (i) results for thresholds in {0.65, 0.7, 0.75, 0.8, 0.85, 0.9} and (ii) a comparison against bipartite (Hungarian) matching on the full similarity matrix. We will show that the functional asymmetry, subspace concentration, and synthetic-model conclusions remain qualitatively unchanged across these choices. This directly addresses the concern that downstream claims rest on an arbitrary modeling decision. revision: yes
Circularity Check
Observational study with independent cross-seed measurements; no derivation reduces to its inputs by construction
full rationale
The paper performs an empirical comparison of SAEs trained on independent random seeds. Feature stability is defined via a cross-seed reappearance probability computed from a similarity threshold and bipartite matching procedure; the subsequent claims about reconstruction error, downstream task impact, activation statistics, and subspace geometry are then measured on those partitioned sets using separate statistics that are not algebraically or statistically forced by the partitioning rule itself. No equation equates a reported functional or geometric quantity to a fitted parameter on the same data, no self-citation supplies a uniqueness theorem that forbids alternatives, and the synthetic low-rank model is presented as an illustrative construction rather than a fitted predictor of the real-data results. The analysis is therefore self-contained against external benchmarks of cross-run reproducibility.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Neural network activations admit a useful sparse linear decomposition into features.
read the original abstract
Sparse autoencoders (SAEs) are widely used to interpret neural network representations, but their utility depends on whether the learned features are reproducible across training runs. We study this question through \emph{feature stability}: for each SAE feature, we estimate the probability that a similar feature reappears in an independently trained SAE. This yields a scalable per-feature signal that separates stable from unstable features. In a large-scale study across seeds, models, layers, dictionary sizes, and SAE variants, we find a pronounced functional asymmetry: stable features carry most of the reconstruction- and prediction-relevant signal, while unstable features have weak marginal impact and are dominated by low-frequency surface-form triggers in both activation statistics and automatic explanations. Geometrically, unstable features are individually non-reproducible but concentrate in reproducible lower-rank subspaces, suggesting that seed dependence often reflects basis ambiguity within a shared region of activation space rather than pure noise. A controlled synthetic model makes this mechanism explicit, showing that low-rank ground-truth features can be recovered at the subspace level while remaining non-identifiable as individual SAE latents across seeds. Finally, by pooling unique cross-seed features, we construct more stable SAEs while preserving explained variance in this setting. Together, these results show that unstable features are not merely failed or noisy latents: they have weak individual functional impact, but reflect reproducible low-dimensional structure that standard SAEs resolve differently across seeds.
Figures
Reference graph
Works this paper leans on
-
[1]
2026 , eprint=
Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learning , author=. 2026 , eprint=
2026
-
[2]
2024 , eprint=
Anisotropy Is Inherent to Self-Attention in Transformers , author=. 2024 , eprint=
2024
-
[3]
Balagansky, Nikita and Aksenov, Yaroslav and Laptev, Daniil and Kurochkin, Vadim and Gerasimov, Gleb and Koriagin, Nikita and Gavrilov, Daniil. Train One Sparse Autoencoder Across Multiple Sparsity Budgets to Preserve Interpretability and Accuracy. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v...
-
[4]
Sparse autoencoders trained on the same data learn different features
Sparse Autoencoders Trained on the Same Data Learn Different Features , author =. 2025 , eprint =. doi:10.48550/arXiv.2501.16615 , url =
-
[5]
The Thirteenth International Conference on Learning Representations , year =
Sparse Autoencoders Do Not Find Canonical Units of Analysis , author =. The Thirteenth International Conference on Learning Representations , year =
-
[6]
Prince and Matthew Kowal and Victor Boutin and Isabel Papadimitriou and Binxu Wang and Martin Wattenberg and Demba E
Thomas Fel and Ekdeep Singh Lubana and Jacob S. Prince and Matthew Kowal and Victor Boutin and Isabel Papadimitriou and Binxu Wang and Martin Wattenberg and Demba E. Ba and Talia Konkle , booktitle =. Archetypal. 2025 , url =
2025
-
[7]
arXiv preprint arXiv:2505.16077 , year=
Ensembling Sparse Autoencoders , author =. 2025 , eprint =. doi:10.48550/arXiv.2505.16077 , url =
-
[8]
Siyu Chen and Heejune Sheen and Xuyuan Xiong and Tianhao Wang and Zhuoran Yang , year =. Taming Polysemanticity in. doi:10.48550/arXiv.2506.14002 , url =. 2506.14002 , archivePrefix =
-
[9]
Seonglae Cho and Harryn Oh and Donghyun Lee and Luis Eduardo Rodrigues Vieira and Andrew Bermingham and Ziad. 2025 , eprint =. doi:10.48550/arXiv.2506.17673 , url =
-
[10]
Enforcing Orderedness to Improve Feature Consistency , author =. 2025 , eprint =. doi:10.48550/arXiv.2512.02194 , url =
-
[11]
arXiv preprint arXiv:2501.17727 , year=
Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers , author =. 2025 , eprint =. doi:10.48550/arXiv.2501.17727 , url =
-
[12]
Do Sparse Autoencoders Capture Concept Manifolds?
Do Sparse Autoencoders Capture Concept Manifolds? , author =. 2026 , eprint =. doi:10.48550/arXiv.2604.28119 , url =
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.28119 2026
-
[13]
Daniel Balcells and Benjamin Lerner and Michael Oesterle and Ediz Ucar and Stefan Heimersheim , year =. Evolution of. doi:10.48550/arXiv.2410.08869 , url =. 2410.08869 , archivePrefix =
-
[14]
The Thirteenth International Conference on Learning Representations , year =
Mechanistic Permutability: Match Features Across Layers , author =. The Thirteenth International Conference on Learning Representations , year =
-
[15]
The Thirteenth International Conference on Learning Representations , year =
Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures , author =. The Thirteenth International Conference on Learning Representations , year =
-
[16]
2026 , howpublished =
Benchmarking Cross-Seed Feature Correspondence in Sparse Autoencoders , author =. 2026 , howpublished =
2026
-
[17]
Analyze Feature Flow to Enhance Interpretation and Steering in Language Models , author =. 2025 , eprint =. doi:10.48550/arXiv.2502.03032 , url =
-
[18]
2025 , journal =
SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability , author =. 2025 , journal =
2025
-
[19]
, title =
Kuhn, Harold W. , title =. Naval Research Logistics Quarterly , volume =. 1955 , doi =
1955
-
[20]
Giornale dell'Istituto Italiano degli Attuari , volume =
Glivenko, Valery , title =. Giornale dell'Istituto Italiano degli Attuari , volume =
-
[21]
Giornale dell'Istituto Italiano degli Attuari , volume =
Cantelli, Francesco Paolo , title =. Giornale dell'Istituto Italiano degli Attuari , volume =
-
[22]
The Thirteenth International Conference on Learning Representations , year=
Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures , author=. The Thirteenth International Conference on Learning Representations , year=
-
[23]
2025 , journal =
The Dead Salmons of AI Interpretability , author =. 2025 , journal =
2025
-
[24]
2025 , journal =
Sparse Autoencoders Can Interpret Randomly Initialized Transformers , author =. 2025 , journal =
2025
-
[25]
2025 , eprint=
Data Whitening Improves Sparse Autoencoder Learning , author=. 2025 , eprint=
2025
-
[26]
Sparse coding with an overcomplete basis set: A strategy employed by
Olshausen, Bruno A and Field, David J , journal=. Sparse coding with an overcomplete basis set: A strategy employed by. 1997 , publisher=
1997
-
[27]
Transformer Circuits Thread , volume=
Towards monosemanticity: Decomposing language models with dictionary learning , author=. Transformer Circuits Thread , volume=
-
[28]
Sparse Autoencoders Find Highly Interpretable Features in Language Models
Sparse autoencoders find highly interpretable features in language models , author=. arXiv preprint arXiv:2309.08600 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[29]
The Thirteenth International Conference on Learning Representations , year=
Scaling and evaluating sparse autoencoders , author=. The Thirteenth International Conference on Learning Representations , year=
-
[30]
2024 , eprint=
Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders , author=. 2024 , eprint=
2024
-
[31]
2024 , eprint=
BatchTopK Sparse Autoencoders , author=. 2024 , eprint=
2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.