Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Can sparse autoencoders be used to decompose and interpret steering vectors?

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read Sparse autoencoders cannot faithfully decompose steering vectors because such vectors are out-of-distribution and carry negative feature projections SAEs cannot represent.

desk verdict A clean diagnostic paper: the OOD/encoder-bias story is solid, but the negative-coefficient prevalence is backed by an unvalidated proxy rather than direct measurement. read the letter →

arxiv 2411.08790 v1 pith:4QQ672QD submitted 2024-11-13 cs.LG cs.AIcs.CL

classification cs.LGcs.AIcs.CL
keywords sparseautoencoderssteeringvectorsactivationcontrastiveadditionmechanisticinterpretabilitylargelanguagemodelsout-of-distributionnegativefeatureprojections
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Steering vectors let researchers push a language model's behaviour in a desired direction, but what those vectors actually represent remains unclear. This paper argues that the natural tool for answering that question, sparse autoencoders (SAEs), cannot be applied to steering vectors directly: the vectors fall outside the distribution SAEs were trained on, and the non-negative coefficients SAEs produce cannot express the negative projections such vectors contain. If the paper is right, existing SAE-based interpretations of steering vectors are systematically misleading, and interpretability work should decompose positive and negative prompt activations separately before subtracting, or find another way to represent signed coefficients. The stakes are practical: a correct interpretation would help explain why steering works for some behaviours and not others, and why combining steering vectors often fails.

What carries the argument

The argument turns on two mechanisms. Contrastive activation addition builds a steering vector as the averaged difference between activations on positive and negative prompt pairs; this subtraction removes the default components that are present in every activation, so the resulting vector sits far from the distribution the SAE encoder was calibrated for, and the encoder's bias term dominates the decomposition. The second mechanism is the SAE encoder's non-negative activation function, such as ReLU or JumpReLU, which forces every reconstruction coefficient to be non-negative; a contrastive difference vector, however, naturally has negative projections, and because SAE features can have negative cosine similarity with one another, a true negative projection can appear as a spurious positive activation in a different feature. Together these make direct SAE decomposition unreliable.

What would settle it

Take the corrigibility steering vector and compute its signed coefficients in the SAE feature basis with unconstrained least squares, allowing negative values. If the largest-magnitude coefficients are not predominantly negative, or if they disagree with the positive-versus-negative prompt activation differences, the paper's evidence for widespread negative projections is undermined.

Watch

Extended reading notes

Core claim

The paper's central claim is that direct SAE-decomposition of steering vectors is misleading for two structural reasons. First, steering vectors fall outside the distribution of model activations SAEs are trained on: they have much smaller L2 norms and, because contrastive activation addition subtracts paired activations, they lack the default context components that SAE encoder biases are calibrated to offset; as a result, the decomposition is dominated by encoder bias, as shown by the near-identical top features for a steering vector and a zero vector. Second, SAEs enforce non-negative reconstruction coefficients, while a steering vector formed by subtracting activations naturally has meaningful negative projections in feature directions; the paper finds that for seven behaviours roughly half of the large-magnitude feature differences are negative, and that negative projections can masquerade as positive activations in features with negative cosine similarity. The conclusion is that the apparent SAE interpretation of a steering vector is not a reliable guide to what the vector does.

Load-bearing premise

The load-bearing assumption is that differences in SAE feature activations between positive and negative prompts reveal how strongly the steering vector points along each feature; the authors state this is only a proxy and not a ground-truth decomposition.

Editorial extensions

If this is right

  • Any SAE feature ranking obtained by feeding a steering vector through the encoder is dominated by encoder bias; the top features are essentially the same as those produced by a zero vector.
  • Negative feature projections are not a corner case: across the seven behaviours studied, roughly half of the largest-magnitude feature differences are negative, so SAE interpretations silently omit a large fraction of the steering mechanism.
  • A true negative projection can appear as a false positive activation of a different feature when the corresponding feature directions have negative cosine similarity, so even the positive features reported by an SAE can point to the wrong concept.
  • Alternative sparse approximation methods that replace the SAE encoder address the out-of-distribution problem but still fail unless they allow negative coefficients, making negativity the more fundamental obstacle.
  • A more promising route is to decompose positive and negative prompt activations separately and then subtract the decompositions, keeping inputs in-distribution and permitting signed coefficients; the paper proposes this as future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this diagnosis holds, the meaning of a steering vector may be context-dependent: in different activation contexts the same vector can act as a positive write to one feature or a negative write to a feature with opposite direction, so any context-free interpretation may be impossible, a possibility the paper sketches in its appendix.
  • The out-of-distribution argument generalises beyond SAEs: any linear readout trained with a bias on activations will misread contrastive difference vectors, so other interpretability tools that operate in residual-stream space may carry the same blind spot.
  • A testable extension would be to train an SAE directly on contrastive difference vectors, or to compare the paper's prompt-activation proxy against an unconstrained least-squares decomposition of the steering vector in SAE feature space, which would provide the ground-truth check the paper's method lacks.
  • If negative projections are as common as reported, interpretability work should consider signed dictionary learning or separate positive and negative feature decompositions, not just non-negative sparse coding, for any object built from activation differences.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper asks whether sparse autoencoders (SAEs) can be used to directly decompose steering vectors, and answers that they cannot be trusted for this purpose. Using a corrigibility steering vector extracted from Gemma 2 2B via contrastive activation addition and a Gemma Scope layer-14 SAE, the authors identify two reasons: (1) steering vectors are out-of-distribution for SAEs, so the encoder bias dominates the decomposition; and (2) steering vectors can have meaningful negative projections in feature directions, which non-negative SAE encoders cannot represent and which can also cause spurious positive activations through negative cosine similarity between features. The evidence includes a zero-vector control showing nearly identical top features, per-behavior top-feature tables, and comparisons of SAE activations on positive versus negative prompts. The paper concludes that direct SAE decomposition should not be used and sketches a future approach that decomposes contrastive activations before subtraction.

Significance. If the paper's diagnostic claims are correct, they provide a useful caveat for the interpretability community and help explain why prior attempts to interpret steering vectors with SAEs produced mixed results. The zero-vector control and the near-identical top-feature tables in Tables 1 and 2 are a clean, convincing demonstration that the direct encoding of a small-norm steering vector is dominated by the SAE encoder bias. The use of external, openly available SAEs, datasets, and code supports reproducibility. The logical argument that SAEs cannot output negative coefficients is sound, but the empirical claim that negative projections are widespread (51.2% in Section 3.2 and Table 3) currently rests on an acknowledged proxy that has not been validated against direct measurements. If that proxy is biased by JumpReLU thresholding, the prevalence estimate could be off, although the qualitative message that direct SAE decomposition is unreliable is independently supported by the out-of-distribution result.

major comments (2)
  1. [Section 3.2 / Appendix C.2, Table 3] The headline statistic that 51.2% of active features are 'negative' is computed by comparing mean SAE activations on positive versus negative prompts, not by decomposing the steering vector itself. Because the JumpReLU encoder is nonlinear with per-feature thresholds, sign(f_i(a_+) - f_i(a_-)) need not equal sign(<d_i, v>) or the sign of a sparse reconstruction coefficient of v; thresholding can suppress one of the paired activations and flip the apparent sign. The paper concedes that this method 'does not provide a ground truth decomposition,' yet it is the only quantitative support for the claim that meaningful negative projections are common. I would like to see the proxy validated against direct projections <d_i, v> or against coefficients obtained by a linear sparse approximation of v in the SAE dictionary, together with a robustness check that the sign pattern is stable under threshold variations. Without this, the paper's second headline reason is under-supported.
  2. [Section 3.1, Figure 2] The zero-vector control convincingly shows that the encoder bias dominates for the original, small-norm steering vector. However, the additional claim that scaling the steering vector does not fix the out-of-distribution problem is supported by the default-component argument but not quantified. A direct comparison of f(alpha*v) and f(0) over a range of alpha, including values chosen to match the median activation norm, would make the claim quantitative and would rule out the alternative that a sufficiently large alpha makes W_enc v dominate b_enc. As written, Figure 2 is suggestive but the scaling claim is not demonstrated as rigorously as the zero-vector control.
minor comments (5)
  1. [Section 3.2, Figure 3] The right panel would be clearer if it stated that the 6.12 activation comes from direct decomposition of the steering vector rather than from the prompt-conditioned activations; as written, 'rarely activates' and 'SAE activation 6.12' appear contradictory.
  2. [Section 1 / Section 3] The text says 'steeringcorrigibility' in §3; this should be 'steering corrigibility'.
  3. [Equation (1)] The notation |X| is standard but should be defined as the number of contrastive prompt pairs; also, the subscript L is typeset inconsistently as a_L and aL across the paper.
  4. [Appendix C.2] The top-100 cutoff is admittedly arbitrary; reporting the number of negative features at several cutoffs (e.g., top 50 and top 200) would make the 'around 50' claim more robust.
  5. [Appendix A.2] The sentence 'our arguments and findings in this paper are independent of whether ReLU or JumpReLU is used' is too strong in light of the threshold-sensitivity concern for the positive/negative prompt proxy; please qualify it to refer to the logical non-negativity argument rather than all empirical findings.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: both headline reasons are supported by independent empirical controls or explicit definitional derivations, not by fitting, renaming, or load-bearing self-citation.

full rationale

The paper's two headline reasons are not derived from its own outputs. Reason (1) is supported by an independent empirical control: decomposing the zero vector through the same SAE produces nearly identical top features (Table 1), showing that the encoder bias dominates; this control does not presuppose the out-of-distribution conclusion and is externally checkable with the released code and Gemma Scope checkpoints. The 'default component' argument is an empirical observation about activation statistics, not an artifact of fitting. Reason (2) is a mathematical implication of the stated definitions (Equation 1: v = mean(a+ - a-); Equation 2: the encoder activation function is non-negative): a difference of two non-negative combinations can have negative coefficients, and the SAE clamps them to zero. This is a valid derivation with stated premises, not a circular reduction. The prevalence estimate (51.2%, Table 3) is explicitly an admitted proxy rather than a ground-truth decomposition, and the authors flag that it 'does not provide a ground truth decomposition'; a proxy validity concern is a correctness risk, not circularity, because the number is not a fitted parameter renamed as a prediction and no self-citation carries the argument. No load-bearing self-citations were found; all cited SAEs, datasets, and baseline methods are external or code-reproducible, and the paper's own conclusions do not depend on accepting any author's prior uniqueness claim.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The ledger is small because the paper is an empirical diagnostic rather than a derivation. The main unverified input is the default-component model of activations and the proxy used to infer negative projections. No new entities are introduced.

free parameters (1)
  • top-100 feature cutoff = 100 features by magnitude
    The number of features used to count negative projections in Table 3 was chosen arbitrarily. The authors state results were similar when varying this, so it does not drive the qualitative conclusion, but it is a hand-chosen threshold.
assumptions (4)
  • domain assumption SAE reconstruction coefficients are non-negative due to the activation function in the encoder (Equation 2).
    Used to argue that negative projections cannot be represented. This is a property of the SAE architecture, not an assumption introduced by the paper.
  • domain assumption Model activations contain prompt-independent default components that SAE encoder biases offset, and steering vectors lack these due to subtraction.
    Invoked in Section 3.1 to argue that scaling does not fix the out-of-distribution problem. Supported by citation [21] and one feature example, not independently proven.
  • domain assumption Differences between SAE activations on positive and negative contrastive prompts are indicative of the steering vector's projections in feature directions.
    Used in Section 3.2 and Appendix C.2 to quantify negative coefficients. The authors note this is not a ground truth decomposition.
  • domain assumption Negative cosine similarity between SAE features causes negative projections in one direction to appear as positive projections in another.
    Assumed in Section 3.2 and Appendix D. Relies on linearity of the SAE encoder and the geometry of learned features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can sparse autoencoders be used to decompose and interpret steering vectors?." pith.science (2026). https://pith.science/paper/4QQ672QD

@misc{pith2026241108790,
  author       = {Pith},
  title        = {Pith review of: Can sparse autoencoders be used to decompose and interpret steering vectors?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4QQ672QD}},
  note         = {Machine review of arXiv:2411.08790}
}
read the original abstract

Steering vectors are a promising approach to control the behaviour of large language models. However, their underlying mechanisms remain poorly understood. While sparse autoencoders (SAEs) may offer a potential method to interpret steering vectors, recent findings show that SAE-reconstructed vectors often lack the steering properties of the original vectors. This paper investigates why directly applying SAEs to steering vectors yields misleading decompositions, identifying two reasons: (1) steering vectors fall outside the input distribution for which SAEs are designed, and (2) steering vectors can have meaningful negative projections in feature directions, which SAEs are not designed to accommodate. These limitations hinder the direct use of SAEs for interpreting steering vectors.

Figures

Figures reproduced from arXiv: 2411.08790 by the authors.

Figure 1
Figure 1. Steering vectors are out-of￾distribution for SAEs. The L2-norm of the corri￾gibility steering vector is outside the distribution of L2-norms of layer 14 model activations, caus￾ing the encoder bias to skew the SAE decomposi￾tion. Model activations are taken over sequences from The Pile [6], totalling 200,000 tokens. Corrigibility Zero vector steering vector Feature Activation Feature Activation 4888 95.04 4888 89.06… view at source ↗
Figure 2
Figure 2. Scaled steering vectors remain out-of-distribution in certain directions. Model acti￾vations contain some default components that exist regardless of the prompt. For instance, model activations of random prompts are, on average, highly negative in the direction of SAE feature 4888. The SAE offsets this default component with a positive encoder bias term (86.20), resulting in SAE activations around zero (right-hand a… view at source ↗
Figure 3
Figure 3. Negative projections can cause misleading positive activations in SAE decompositions. Left: Feature 14004 activates more strongly on negative corrigibility prompts than positive ones, indicating its relevance to the steering vector. However, while the steering vector has a strong negative projection in this direction, SAEs are not designed to accommodate negative coefficients, resulting in an activation of 0.00. Rig… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The corrigibility steering vector extracted at layer 14 has the highest steerability. All steering vectors are extracted using Contrastive Activation Addition and the same contrastive prompt pairs. Steerability is defined as in [18]. increasing since propensity to exhi…
Figure 5
Figure 5. Figure 5: Steering vector L2-norms. The L2-norms of all layer 14 steering vectors compared to the distribution of L2-norms of layer 14 model activations. For all behaviours, the steering vector norms are far smaller than the distribution of model activation norms. Model activati…
Figure 6
Figure 6. Figure 6: Illustration of why steering vector interpretability may depend on the model activations the vector is added to. Depending on the model activations the steering vector is added to, the same vector could be interpreted as (A) writing positively to a feature or (B) writi…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    BiasLens uses concept activation vectors and sparse autoencoders to estimate LLM bias from internal representations, reporting moderate to strong agreement with behavioral bias metrics in a small evaluation.

Reference graph

Works this paper leans on

25 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    Foundational challenges in assuring alignment and safety of large language models

    Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932, 2024

  2. [2]

    Refusal in language models is mediated by a single direction

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717, 2024

  3. [3]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2, 2023

  4. [4]

    Progress update #1 from the GDM mech interp team

    Arthur Conmy, Neel Nanda, Lewis Smith, Senthooran Rajamanoharan, Tom Lieberum, János Kramár, and Vikrant Varma. Progress update #1 from the GDM mech interp team. Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/C5KAZQib3bzzpeyrg/ full-post-progress-update-1-from-the-gdm-mech-interp-team . Activation Steer- ing with SAEs

  5. [5]

    Sparse autoen- coders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023

  6. [6]

    The Pile: An 800GB dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

  7. [7]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024

  8. [8]

    Extract- ing sae task features for in-context learning

    Dmitrii Kharlapenko, neverix, Neel Nanda, and Arthur Conmy. Extract- ing sae task features for in-context learning. AI Alignment Forum , August

Show all 25 references
  1. [9]

    Gemma Scope: Open sparse autoencoders everywhere all at once on Gemma 2

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma Scope: Open sparse autoencoders everywhere all at once on Gemma 2. arXiv preprint arXiv:2408.05147, 2024

  2. [10]

    In-context vectors: Making in context learning more effective and controllable through latent space steering

    Sheng Liu, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint arXiv:2311.06668, 2023

  3. [11]

    Sparse feature circuits: Discovering and editing interpretable causal graphs in language models

    Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647, 2024

  4. [12]

    Sparse autoencoders enable scalable and reliable circuit identification in language models

    Charles O’Neill and Thang Bui. Sparse autoencoders enable scalable and reliable circuit identification in language models. arXiv preprint arXiv:2405.12522, 2024

  5. [13]

    Steering llama 2 via contrastive activation addition

    Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023

  6. [14]

    Discovering language model behaviors with model-written evaluations

    Ethan Perez, Sam Ringer, Kamil˙e Lukoši¯ut˙e, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251, 2022

  7. [15]

    Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders

    Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435, 2024. 6

  8. [16]

    Progress update #1 from the gdm mech interp team

    Lewis Smith, Arthur Conmy, Neel Nanda, Senthooran Rajamanoharan, Tom Lieberum, János Kramár, and Vikrant Varma. Progress update #1 from the gdm mech interp team. Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/C5KAZQib3bzzpeyrg/ full-post-progress-update-1-from...

  9. [17]

    Steering vectors github, 2024

    Daniel Tan and David Chanin. Steering vectors github, 2024. URL https://github.com/ steering-vectors/steering-vectors . Accessed: 2024-08-28

  10. [18]

    Analyzing the generalization and reliability of steering vectors

    Daniel Tan, David Chanin, Aengus Lynch, Dimitrios Kanoulas, Brooks Paige, Adria Garriga- Alonso, and Robert Kirk. Analyzing the generalization and reliability of steering vectors. arXiv preprint arXiv:2407.12404, 2024

  11. [19]

    Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, and Tristan Hume. Scaling monosem...

  12. [20]

    Vazquez, Ulisse Mini, and Monte MacDiarmid

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization,

  13. [21]

    Model editing as a robust and denoised variant of dpo: A case study on toxicity, 2024

    Rheeya Uppaal, Apratim Dey, Yiting He, Yiqiao Zhong, and Junjie Hu. Model editing as a robust and denoised variant of dpo: A case study on toxicity, 2024. URL https://arxiv. org/abs/2405.13967

  14. [22]

    URL https://arxiv.org/abs/2308.10248

  15. [23]

    (A)” and “(B)

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to AI transparency. arXiv preprint arXiv:2310.01405, 2023. 7 A Methodological de...

  16. [24]

    Extending activation steering to broad skills and multiple behaviours

    Teun van der Weij, Massimo Poesio, and Nandi Schoots. Extending activation steering to broad skills and multiple behaviours. arXiv preprint arXiv:2403.05767, 2024

  17. [2024]

    URL https://www.alignmentforum.org/posts/5FGXmJ3wqgGRcbyH7/ extracting-sae-task-features-for-in-context-learning

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.