Pith. sign in

REVIEW 5 cited by

Enhancing Neural Network Interpretability with Feature-Aligned Sparse Autoencoders

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.01220 v2 pith:25WXSAZF submitted 2024-11-02 cs.LG

Luke Marks , Alasdair Paren , David Krueger , Fazl Barez This is my paper · ORCID
classification cs.LG
keywords featuressaestextscdatainputinterpretabilitylearnlearned
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Sparse Autoencoders (SAEs) have shown promise in improving the interpretability of neural network activations, but can learn features that are not features of the input, limiting their effectiveness. We propose \textsc{Mutual Feature Regularization} \textbf{(MFR)}, a regularization technique for improving feature learning by encouraging SAEs trained in parallel to learn similar features. We motivate \textsc{MFR} by showing that features learned by multiple SAEs are more likely to correlate with features of the input. By training on synthetic data with known features of the input, we show that \textsc{MFR} can help SAEs learn those features, as we can directly compare the features learned by the SAE with the input features for the synthetic data. We then scale \textsc{MFR} to SAEs that are trained to denoise electroencephalography (EEG) data and SAEs that are trained to reconstruct GPT-2 Small activations. We show that \textsc{MFR} can improve the reconstruction loss of SAEs by up to 21.21\% on GPT-2 Small, and 6.67\% on EEG data. Our results suggest that the similarity between features learned by different SAEs can be leveraged to improve SAE training, thereby enhancing performance and the usefulness of SAEs for model interpretability.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Query Circuits: Explaining How Language Models Answer User Prompts

    cs.AI 2025-09 conditional novelty 6.0 of 10

    Language models contain per-query circuits using a few percent of their edges that reproduce much of their answer behavior, and paraphrase-based Best-of-N search finds them.

  2. Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEs

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A position paper proposing feature consistency, measured by PW-MCC, as a core SAE evaluation criterion, with evidence that TopK SAEs achieve high consistency on LLM activations.

  3. Cross-seed explainability using Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoders

    cs.CL 2026-07 conditional novelty 5.5 of 10

    Procrustes-conditioned joint End-to-end Top-K SAEs recover more cross-seed universal features (r≥0.70) from independent BERT seeds than post-hoc alignment baselines on three NLP datasets.

  4. Stable and Steerable Sparse Autoencoders with Weight Regularization

    stat.ML 2026-03 conditional novelty 5.0 of 10

    L2 weight regularization in TopK SAEs increases cross-seed feature overlap and roughly doubles measured steering success on Pythia-70M, at the cost of collapsing most latents to zero.

  5. Interpreting CFD Surrogates through Sparse Autoencoders

    cs.CE 2025-07 conditional novelty 4.0 of 10

    Sparse autoencoders trained on frozen MeshGraphNets embeddings yield feature dictionaries whose mesh-space activations align with high-vorticity regions better than embedding-norm, PCA, or random baselines.

Pith tools