REVIEW 2 major objections 5 minor 1 cited by
Can sparse autoencoders be used to decompose and interpret steering vectors?
T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Sparse autoencoders cannot faithfully decompose steering vectors because such vectors are out-of-distribution and carry negative feature projections SAEs cannot represent.
desk verdict A clean diagnostic paper: the OOD/encoder-bias story is solid, but the negative-coefficient prevalence is backed by an unvalidated proxy rather than direct measurement. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument turns on two mechanisms. Contrastive activation addition builds a steering vector as the averaged difference between activations on positive and negative prompt pairs; this subtraction removes the default components that are present in every activation, so the resulting vector sits far from the distribution the SAE encoder was calibrated for, and the encoder's bias term dominates the decomposition. The second mechanism is the SAE encoder's non-negative activation function, such as ReLU or JumpReLU, which forces every reconstruction coefficient to be non-negative; a contrastive difference vector, however, naturally has negative projections, and because SAE features can have negative cosine similarity with one another, a true negative projection can appear as a spurious positive activation in a different feature. Together these make direct SAE decomposition unreliable.
What would settle it
Take the corrigibility steering vector and compute its signed coefficients in the SAE feature basis with unconstrained least squares, allowing negative values. If the largest-magnitude coefficients are not predominantly negative, or if they disagree with the positive-versus-negative prompt activation differences, the paper's evidence for widespread negative projections is undermined.
Extended reading notes
Core claim
The paper's central claim is that direct SAE-decomposition of steering vectors is misleading for two structural reasons. First, steering vectors fall outside the distribution of model activations SAEs are trained on: they have much smaller L2 norms and, because contrastive activation addition subtracts paired activations, they lack the default context components that SAE encoder biases are calibrated to offset; as a result, the decomposition is dominated by encoder bias, as shown by the near-identical top features for a steering vector and a zero vector. Second, SAEs enforce non-negative reconstruction coefficients, while a steering vector formed by subtracting activations naturally has meaningful negative projections in feature directions; the paper finds that for seven behaviours roughly half of the large-magnitude feature differences are negative, and that negative projections can masquerade as positive activations in features with negative cosine similarity. The conclusion is that the apparent SAE interpretation of a steering vector is not a reliable guide to what the vector does.
Load-bearing premise
The load-bearing assumption is that differences in SAE feature activations between positive and negative prompts reveal how strongly the steering vector points along each feature; the authors state this is only a proxy and not a ground-truth decomposition.
Editorial extensions
If this is right
- Any SAE feature ranking obtained by feeding a steering vector through the encoder is dominated by encoder bias; the top features are essentially the same as those produced by a zero vector.
- Negative feature projections are not a corner case: across the seven behaviours studied, roughly half of the largest-magnitude feature differences are negative, so SAE interpretations silently omit a large fraction of the steering mechanism.
- A true negative projection can appear as a false positive activation of a different feature when the corresponding feature directions have negative cosine similarity, so even the positive features reported by an SAE can point to the wrong concept.
- Alternative sparse approximation methods that replace the SAE encoder address the out-of-distribution problem but still fail unless they allow negative coefficients, making negativity the more fundamental obstacle.
- A more promising route is to decompose positive and negative prompt activations separately and then subtract the decompositions, keeping inputs in-distribution and permitting signed coefficients; the paper proposes this as future work.
Reading between the lines
- If this diagnosis holds, the meaning of a steering vector may be context-dependent: in different activation contexts the same vector can act as a positive write to one feature or a negative write to a feature with opposite direction, so any context-free interpretation may be impossible, a possibility the paper sketches in its appendix.
- The out-of-distribution argument generalises beyond SAEs: any linear readout trained with a bias on activations will misread contrastive difference vectors, so other interpretability tools that operate in residual-stream space may carry the same blind spot.
- A testable extension would be to train an SAE directly on contrastive difference vectors, or to compare the paper's prompt-activation proxy against an unconstrained least-squares decomposition of the steering vector in SAE feature space, which would provide the ground-truth check the paper's method lacks.
- If negative projections are as common as reported, interpretability work should consider signed dictionary learning or separate positive and negative feature decompositions, not just non-negative sparse coding, for any object built from activation differences.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper asks whether sparse autoencoders (SAEs) can be used to directly decompose steering vectors, and answers that they cannot be trusted for this purpose. Using a corrigibility steering vector extracted from Gemma 2 2B via contrastive activation addition and a Gemma Scope layer-14 SAE, the authors identify two reasons: (1) steering vectors are out-of-distribution for SAEs, so the encoder bias dominates the decomposition; and (2) steering vectors can have meaningful negative projections in feature directions, which non-negative SAE encoders cannot represent and which can also cause spurious positive activations through negative cosine similarity between features. The evidence includes a zero-vector control showing nearly identical top features, per-behavior top-feature tables, and comparisons of SAE activations on positive versus negative prompts. The paper concludes that direct SAE decomposition should not be used and sketches a future approach that decomposes contrastive activations before subtraction.
Significance. If the paper's diagnostic claims are correct, they provide a useful caveat for the interpretability community and help explain why prior attempts to interpret steering vectors with SAEs produced mixed results. The zero-vector control and the near-identical top-feature tables in Tables 1 and 2 are a clean, convincing demonstration that the direct encoding of a small-norm steering vector is dominated by the SAE encoder bias. The use of external, openly available SAEs, datasets, and code supports reproducibility. The logical argument that SAEs cannot output negative coefficients is sound, but the empirical claim that negative projections are widespread (51.2% in Section 3.2 and Table 3) currently rests on an acknowledged proxy that has not been validated against direct measurements. If that proxy is biased by JumpReLU thresholding, the prevalence estimate could be off, although the qualitative message that direct SAE decomposition is unreliable is independently supported by the out-of-distribution result.
major comments (2)
- [Section 3.2 / Appendix C.2, Table 3] The headline statistic that 51.2% of active features are 'negative' is computed by comparing mean SAE activations on positive versus negative prompts, not by decomposing the steering vector itself. Because the JumpReLU encoder is nonlinear with per-feature thresholds, sign(f_i(a_+) - f_i(a_-)) need not equal sign(<d_i, v>) or the sign of a sparse reconstruction coefficient of v; thresholding can suppress one of the paired activations and flip the apparent sign. The paper concedes that this method 'does not provide a ground truth decomposition,' yet it is the only quantitative support for the claim that meaningful negative projections are common. I would like to see the proxy validated against direct projections <d_i, v> or against coefficients obtained by a linear sparse approximation of v in the SAE dictionary, together with a robustness check that the sign pattern is stable under threshold variations. Without this, the paper's second headline reason is under-supported.
- [Section 3.1, Figure 2] The zero-vector control convincingly shows that the encoder bias dominates for the original, small-norm steering vector. However, the additional claim that scaling the steering vector does not fix the out-of-distribution problem is supported by the default-component argument but not quantified. A direct comparison of f(alpha*v) and f(0) over a range of alpha, including values chosen to match the median activation norm, would make the claim quantitative and would rule out the alternative that a sufficiently large alpha makes W_enc v dominate b_enc. As written, Figure 2 is suggestive but the scaling claim is not demonstrated as rigorously as the zero-vector control.
minor comments (5)
- [Section 3.2, Figure 3] The right panel would be clearer if it stated that the 6.12 activation comes from direct decomposition of the steering vector rather than from the prompt-conditioned activations; as written, 'rarely activates' and 'SAE activation 6.12' appear contradictory.
- [Section 1 / Section 3] The text says 'steeringcorrigibility' in §3; this should be 'steering corrigibility'.
- [Equation (1)] The notation |X| is standard but should be defined as the number of contrastive prompt pairs; also, the subscript L is typeset inconsistently as a_L and aL across the paper.
- [Appendix C.2] The top-100 cutoff is admittedly arbitrary; reporting the number of negative features at several cutoffs (e.g., top 50 and top 200) would make the 'around 50' claim more robust.
- [Appendix A.2] The sentence 'our arguments and findings in this paper are independent of whether ReLU or JumpReLU is used' is too strong in light of the threshold-sensitivity concern for the positive/negative prompt proxy; please qualify it to refer to the logical non-negativity argument rather than all empirical findings.
Circularity Check
No circularity: both headline reasons are supported by independent empirical controls or explicit definitional derivations, not by fitting, renaming, or load-bearing self-citation.
full rationale
The paper's two headline reasons are not derived from its own outputs. Reason (1) is supported by an independent empirical control: decomposing the zero vector through the same SAE produces nearly identical top features (Table 1), showing that the encoder bias dominates; this control does not presuppose the out-of-distribution conclusion and is externally checkable with the released code and Gemma Scope checkpoints. The 'default component' argument is an empirical observation about activation statistics, not an artifact of fitting. Reason (2) is a mathematical implication of the stated definitions (Equation 1: v = mean(a+ - a-); Equation 2: the encoder activation function is non-negative): a difference of two non-negative combinations can have negative coefficients, and the SAE clamps them to zero. This is a valid derivation with stated premises, not a circular reduction. The prevalence estimate (51.2%, Table 3) is explicitly an admitted proxy rather than a ground-truth decomposition, and the authors flag that it 'does not provide a ground truth decomposition'; a proxy validity concern is a correctness risk, not circularity, because the number is not a fitted parameter renamed as a prediction and no self-citation carries the argument. No load-bearing self-citations were found; all cited SAEs, datasets, and baseline methods are external or code-reproducible, and the paper's own conclusions do not depend on accepting any author's prior uniqueness claim.
Assumptions & free parameters
free parameters (1)
- top-100 feature cutoff =
100 features by magnitude
assumptions (4)
- domain assumption SAE reconstruction coefficients are non-negative due to the activation function in the encoder (Equation 2).
- domain assumption Model activations contain prompt-independent default components that SAE encoder biases offset, and steering vectors lack these due to subtraction.
- domain assumption Differences between SAE activations on positive and negative contrastive prompts are indicative of the steering vector's projections in feature directions.
- domain assumption Negative cosine similarity between SAE features causes negative projections in one direction to appear as positive projections in another.
Cite this review
Pith. "Pith review of Can sparse autoencoders be used to decompose and interpret steering vectors?." pith.science (2026). https://pith.science/paper/4QQ672QD
@misc{pith2026241108790,
author = {Pith},
title = {Pith review of: Can sparse autoencoders be used to decompose and interpret steering vectors?},
year = {2026},
howpublished = {\url{https://pith.science/paper/4QQ672QD}},
note = {Machine review of arXiv:2411.08790}
}
read the original abstract
Steering vectors are a promising approach to control the behaviour of large language models. However, their underlying mechanisms remain poorly understood. While sparse autoencoders (SAEs) may offer a potential method to interpret steering vectors, recent findings show that SAE-reconstructed vectors often lack the steering properties of the original vectors. This paper investigates why directly applying SAEs to steering vectors yields misleading decompositions, identifying two reasons: (1) steering vectors fall outside the input distribution for which SAEs are designed, and (2) steering vectors can have meaningful negative projections in feature directions, which SAEs are not designed to accommodate. These limitations hinder the direct use of SAEs for interpreting steering vectors.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMs
BiasLens uses concept activation vectors and sparse autoencoders to estimate LLM bias from internal representations, reporting moderate to strong agreement with behavioral bias metrics in a small evaluation.
Reference graph
Works this paper leans on
-
[1]
Foundational challenges in assuring alignment and safety of large language models
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932, 2024
arXiv 2024
-
[2]
Refusal in language models is mediated by a single direction
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717, 2024
arXiv 2024
-
[3]
Towards monosemanticity: Decomposing language models with dictionary learning
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2, 2023
2023
-
[4]
Progress update #1 from the GDM mech interp team
Arthur Conmy, Neel Nanda, Lewis Smith, Senthooran Rajamanoharan, Tom Lieberum, János Kramár, and Vikrant Varma. Progress update #1 from the GDM mech interp team. Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/C5KAZQib3bzzpeyrg/ full-post-progress-update-1-from-the-gdm-mech-interp-team . Activation Steer- ing with SAEs
work page 2024
-
[5]
Sparse autoen- coders find highly interpretable features in language models
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023
arXiv 2023
-
[6]
The Pile: An 800GB dataset of diverse text for language modeling
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020
arXiv 2020
-
[7]
Scaling and evaluating sparse autoencoders
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024
arXiv 2024
-
[8]
Extract- ing sae task features for in-context learning
Dmitrii Kharlapenko, neverix, Neel Nanda, and Arthur Conmy. Extract- ing sae task features for in-context learning. AI Alignment Forum , August
Show all 25 references
-
[9]
Gemma Scope: Open sparse autoencoders everywhere all at once on Gemma 2
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma Scope: Open sparse autoencoders everywhere all at once on Gemma 2. arXiv preprint arXiv:2408.05147, 2024
2024 arXiv
-
[10]
In-context vectors: Making in context learning more effective and controllable through latent space steering
Sheng Liu, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint arXiv:2311.06668, 2023
2023 arXiv
-
[11]
Sparse feature circuits: Discovering and editing interpretable causal graphs in language models
Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647, 2024
2024 arXiv
-
[12]
Sparse autoencoders enable scalable and reliable circuit identification in language models
Charles O’Neill and Thang Bui. Sparse autoencoders enable scalable and reliable circuit identification in language models. arXiv preprint arXiv:2405.12522, 2024
2024 arXiv
-
[13]
Steering llama 2 via contrastive activation addition
Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023
2023 arXiv
-
[14]
Discovering language model behaviors with model-written evaluations
Ethan Perez, Sam Ringer, Kamil˙e Lukoši¯ut˙e, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251, 2022
2022 arXiv
-
[15]
Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435, 2024. 6
2024 arXiv
-
[16]
Progress update #1 from the gdm mech interp team
Lewis Smith, Arthur Conmy, Neel Nanda, Senthooran Rajamanoharan, Tom Lieberum, János Kramár, and Vikrant Varma. Progress update #1 from the gdm mech interp team. Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/C5KAZQib3bzzpeyrg/ full-post-progress-update-1-from...
2024
-
[17]
Steering vectors github, 2024
Daniel Tan and David Chanin. Steering vectors github, 2024. URL https://github.com/ steering-vectors/steering-vectors . Accessed: 2024-08-28
2024
-
[18]
Analyzing the generalization and reliability of steering vectors
Daniel Tan, David Chanin, Aengus Lynch, Dimitrios Kanoulas, Brooks Paige, Adria Garriga- Alonso, and Robert Kirk. Analyzing the generalization and reliability of steering vectors. arXiv preprint arXiv:2407.12404, 2024
2024 arXiv
-
[19]
Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, and Tristan Hume. Scaling monosem...
2024
-
[20]
Vazquez, Ulisse Mini, and Monte MacDiarmid
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization,
-
[21]
Model editing as a robust and denoised variant of dpo: A case study on toxicity, 2024
Rheeya Uppaal, Apratim Dey, Yiting He, Yiqiao Zhong, and Junjie Hu. Model editing as a robust and denoised variant of dpo: A case study on toxicity, 2024. URL https://arxiv. org/abs/2405.13967
2024 arXiv
-
[22]
URL https://arxiv.org/abs/2308.10248
-
[23]
(A)” and “(B)
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to AI transparency. arXiv preprint arXiv:2310.01405, 2023. 7 A Methodological de...
-
[24]
Extending activation steering to broad skills and multiple behaviours
Teun van der Weij, Massimo Poesio, and Nandi Schoots. Extending activation steering to broad skills and multiple behaviours. arXiv preprint arXiv:2403.05767, 2024
2024 arXiv
-
[2024]
URL https://www.alignmentforum.org/posts/5FGXmJ3wqgGRcbyH7/ extracting-sae-task-features-for-in-context-learning
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.