Pith. sign in

REVIEW 6 cited by

Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.04478 v1 pith:HPG2X7MD submitted 2024-09-05 cs.LG cs.AIcs.NE

Maheep Chaudhary , Atticus Geiger This is my paper · ORCID
classification cs.LGcs.AIcs.NE
keywords featuressaesgpt-2smallanalysisautoencodersbaselinecity
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

A popular new method in mechanistic interpretability is to train high-dimensional sparse autoencoders (SAEs) on neuron activations and use SAE features as the atomic units of analysis. However, the body of evidence on whether SAE feature spaces are useful for causal analysis is underdeveloped. In this work, we use the RAVEL benchmark to evaluate whether SAEs trained on hidden representations of GPT-2 small have sets of features that separately mediate knowledge of which country a city is in and which continent it is in. We evaluate four open-source SAEs for GPT-2 small against each other, with neurons serving as a baseline, and linear features learned via distributed alignment search (DAS) serving as a skyline. For each, we learn a binary mask to select features that will be patched to change the country of a city without changing the continent, or vice versa. Our results show that SAEs struggle to reach the neuron baseline, and none come close to the DAS skyline. We release code here: https://github.com/MaheepChaudhary/SAE-Ravel

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects

    cs.LG 2026-07 conditional novelty 7.0 of 10

    SAE features can be interpretable and causally useful yet still lack a stable one-dimensional logit direction for steering, with value-like features more structured than pointer-like ones.

  2. Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Coverage of sparse-autoencoder-identified task features predicts post-training performance and can guide synthesis of small, high-impact datasets (2,000 vs. 300,000 samples).

  3. Sparse Autoencoders, Again?

    cs.LG 2025-06 conditional novelty 6.0 of 10

    VAEase gates the VAE decoder input by the encoder's variance, combining sparse-autoencoder adaptive sparsity with a hyperparameter-free loss; a global-minimizer theorem says active latent dimensions recover per-manifo...

  4. LLM Scheming Inversely Scales with Pretraining Language Coverage

    cs.AI 2026-06 reject novelty 5.0 of 10

    A Qwen3 model exhibits higher scheming scores in low-resource languages than in English and Chinese, suggesting alignment does not transfer uniformly across languages.

  5. Discovering Chunks in Neural Embeddings for Interpretability

    cs.LG 2025-02 conditional novelty 5.0 of 10

    Recurring 'chunks' in neural embeddings can be extracted, predict input patterns, and be perturbed to steer a model's outputs.

  6. Perspectives for Direct Interpretability in Multi-Agent Deep Reinforcement Learning

    cs.AI 2025-02 unverdicted novelty 4.0 of 10

    A perspective paper that advocates direct, post hoc interpretability for multi-agent deep reinforcement learning and offers a taxonomy of where those methods might apply.

Pith tools