Pith. sign in

REVIEW 11 cited by

Sparse Autoencoders Trained on the Same Data Learn Different Features

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2501.16615 v2 pith:BQF3XB6N submitted 2025-01-28 cs.LG

Gonçalo Paulo , Nora Belrose This is my paper
classification cs.LG
keywords featuressaesdifferenttrainedacrossmodelusedactivation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Sparse autoencoders (SAEs) are a useful tool for uncovering human-interpretable features in the activations of large language models (LLMs). While some expect SAEs to find the true underlying features used by a model, our research shows that SAEs trained on the same model and data, differing only in the random seed used to initialize their weights, identify different sets of features. For example, in an SAE with 131K latents trained on a feedforward network in Llama 3 8B, only 30% of the features were shared across different seeds. We observed this phenomenon across multiple layers of three different LLMs, two datasets, and several SAE architectures. While ReLU SAEs trained with the L1 sparsity loss showed greater stability across seeds, SAEs using the state-of-the-art TopK activation function were more seed-dependent, even when controlling for the level of sparsity. Our results suggest that the set of features uncovered by an SAE should be viewed as a pragmatically useful decomposition of activation space, rather than an exhaustive and universal list of features "truly used" by the model.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Brand-as-Memory: Vision-Language Models Encode Causal, Mechanistically Localizable Credibility Priors for News Sources

    cs.CV 2026-07 conditional novelty 7.0 of 10

    VLMs encode a dual-coded, scale-strengthening brand-credibility prior that overrides article evidence ~1.8×, is causally formed at layers 19–21, and is selectively steerable.

  2. Sign-Aware Gated Sparse Autoencoders: Modeling Anticorrelated Features with Bi-Jump-ReLU Activations

    cs.LG 2026-05 conditional novelty 7.0 of 10

    SA-GSAE with Bi-Jump-ReLU enables one latent to encode both polarities of anticorrelated features, Pareto-dominating or matching full-width gated SAEs while reducing dead latents by up to 500x on some LLM hookpoints.

  3. From Found to Designed: Concepts as a Design Axis for Large Language Models

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Concepts should be an explicit design axis for LLMs, organized by pipeline stage and internal-vs-external origin, rather than recovered post-hoc.

  4. Are Single-Token Sparse Autoencoder Features Causally Necessary? Layer-Depth and SAE-Family Effects

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A single-token feature's causal necessity under zero-ablation depends on which SAE family found it: GemmaScope and BatchTopK features stay causally anchored while LlamaScope features are locally redundant.

  5. FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Dataset Dependencies

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Training sparse autoencoders on a language model's own generated text can improve seed stability and downstream probing relative to training on web text.

  6. Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEs

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A position paper proposing feature consistency, measured by PW-MCC, as a core SAE evaluation criterion, with evidence that TopK SAEs achieve high consistency on LLM activations.

  7. Stable and Steerable Sparse Autoencoders with Weight Regularization

    stat.ML 2026-03 conditional novelty 5.0 of 10

    L2 weight regularization in TopK SAEs increases cross-seed feature overlap and roughly doubles measured steering success on Pythia-70M, at the cost of collapsing most latents to zero.

  8. Distribution-Aware Feature Selection for SAEs

    cs.LG 2025-08 conditional novelty 5.0 of 10

    Sampled-SAE pre-selects a candidate pool of features using batch-level norms or entropy before batch top-K, creating a tunable family that trades reconstruction fidelity for improved probing and reduced absorption on ...

  9. On the transferability of Sparse Autoencoders for interpreting compressed models

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Pruning a pretrained sparse autoencoder can produce an interpretability tool for a WANDA-pruned LLM that is roughly comparable to retraining an SAE on the pruned model, though with notable caveats in the reported metrics.

  10. Cross-Layer Discrete Concept Discovery for Interpreting Language Models

    cs.LG 2025-06 reject novelty 5.0 of 10

    CLVQ-VAE maps lower-layer transformer activations to higher-layer ones through a discrete codebook, yielding concept vectors evaluated with probe ablation and human annotation.

  11. Prompting as Scientific Inquiry

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Position paper arguing that prompting LLMs is a form of behavioral science and should be recognized as a core scientific method alongside mechanistic interpretability.

Pith tools