Pith. sign in

REVIEW 3 major objections 5 minor 61 references

Where Steering Signals Come From: Activation Source Selection in Activation Steering

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Activation steering works best when the signal is read from the instant before the model acts, not from text that already shows the behavior.

desk verdict Source choice is the real story here, and tail subtraction is worth stealing; the 'execution-boundary' interpretation is plausible but the position/source-text confound means the theory is not settled. read the letter →

arxiv 2607.25270 v1 pith:Q3VOAIE4 submitted 2026-07-28 cs.CL cs.AIcs.LG

Jiaran Ye , Lingxu Ran , Zijun Yao , Chenpeng Wang , Yong Jiang , Lei Hou , Juanzi Li , Liangming Pan This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords activationsteeringsourceselectionexecution-boundarystatespost-realizationtracestailsubtractionhiddenstatereadoutinstruction-tunedlanguagemodelsinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks where the hidden states used to steer a language model should come from. Holding the downstream intervention fixed, it shows that changing only the source activations—what text elicits them and which positions are read—substantially changes steering success. The central claim is that strong signals come from execution-boundary states, the hidden state at the final token just before the model produces or continues the target behavior, rather than from states that already contain the target text. Answer-based sources are weak on their own; when they work, their useful component aligns with an execution-boundary direction. The paper introduces tail subtraction, subtracting a matched continuation state to isolate the boundary signal, and reports consistent gains across three instruction-tuned models and four task families.

What carries the argument

Execution-boundary states: the hidden state h_{ℓ,|p|}(p) read at the last token of an open context p, immediately before the target-bearing continuation is generated. The paper contrasts these with post-realization trace states read after the target has appeared, and with mixed states produced by averaging across answer positions. The last-token readout policy is the instrument that selects the boundary, and tail subtraction—subtracting a matched tail-only state (same final question and assistant prefix, no target conditioning) from the boundary state—is the construction that isolates a cleaner boundary signal. These devices carry the argument by showing that position relative to imminent pr

What would settle it

Take one target instruction and one final user question, and vary only the readout position within the same prompt: if steering success is equal at the last token before the answer and at an earlier or later token, the execution-boundary theory is false. The paper's own Table 17 shows sequence-mean readout sometimes competing with last-token readout for ICL sources, so a position-vs-content ablation is the decisive test.

Watch

Extended reading notes

Core claim

The paper's central claim is a pre-/post-realization distinction. For a source instance (p,a), define the execution-boundary state b_l(z) = h_{l,|p|}(p), the hidden state read immediately before the target-bearing continuation, and a post-realization trace r_{l,k}(z) = h_{l,|p|+k}(p ⊕ a_{1:k}) after the target has appeared. Across entity, persona, refusal, and nonsense tasks, boundary states read with a last-token policy steer most reliably; answer-only post-realization sources are among the weakest. In relation-completion tasks, where target appearance and continuation semantics are unentangled, all three boundary sources far outperform the answer-bearing mean source, and projecting out the

Load-bearing premise

The execution-boundary interpretation rests on hand-built prompts whose assistant prefix stops just before the target behavior; if those prompts simply induce stronger instruction following at any readout position, the special status of the boundary position—rather than better-specified source text—carries the result.

Editorial extensions

If this is right

  • Steering pipelines should read source activations at a pre-production boundary position rather than averaging over target-bearing completions.
  • Answer-based sources that appear to work do so because they contain an execution-continuation component; removing that component measurably weakens them.
  • Subtracting a matched tail-only state from boundary states improves over positive-only and ordinary contrastive baselines across models and construction methods.
  • Source context and readout policy must be treated as first-class variables in activation steering; the effect persists when hyperparameters are selected on held-out questions.
  • Relation-completion tasks provide a clean diagnostic separating pre-realization from post-realization sources, with boundary sources outperforming answer-bearing mean sources on all three models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension: automatically locate execution boundaries (e.g., by next-token prediction confidence or a probe) instead of hand-built prompts; the theory predicts automatically found boundaries match hand-built ones.
  • If the boundary account is right, it connects to the broader principle that models commit to a behavior before the behavior is visible; the same readout position may improve other interventions such as refusal directions or feature amplification.
  • Tail subtraction suggests a diagnostic for steering failure: collapses and flooding may be caused by post-realization continuation noise, and subtracting matched tails should reduce them in deployed settings.
  • The mixed-state account predicts that as tasks get longer or more compositional, the useful readout position shifts later—right before each continuation step—so per-step boundary readout could outperform a single fixed boundary.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes "activation source selection" as the upstream design choice in activation steering: the combination of source context (text used to elicit activations) and readout policy (which hidden states are aggregated). Holding the downstream intervention fixed, the authors compare source conditions across three instruction-tuned models and four task families. They report that source selection substantially changes steering success, that prompt-only final-token sources are strongest, and that answer-only sources are weak even though they contain the target behavior. They then define execution-boundary states b_ℓ(z) read immediately before a target-bearing continuation and post-realization trace states r_{ℓ,k}(z) read after target tokens have appeared. Refined "boundary" sources (instruction, in-context learning, hybrid) with last-token readout outperform post-realization baselines, and a relation-completion task shows a similar advantage. The paper also introduces "tail subtraction," which subtracts a matched tail-only state from each boundary state to isolate target-preparation signal, reporting improved steering across models and methods.

Significance. If the central interpretation is established, the paper makes a practical and conceptual contribution: it would show that activation steering should read source activations at the pre-production boundary rather than from already-realized target text, and it offers a simple construction (tail subtraction) that improves steering reliability. The empirical work is broad: three models, four task families, five vector-construction methods, held-out hyperparameter selection (Appendix D), and automatic-judge validation. The paper is generally careful to separate source selection from downstream intervention, and the held-out analysis strengthens the claim that the source-condition effect is not merely a test-set artifact.

major comments (3)
  1. [§5.2, Fig. 2, Table 17] The central comparison between boundary sources and post-realization baselines is confounded: the boundary conditions (instruction/icl/hybrid) differ from answer-only and prompt-and-answer conditions in both readout position and source content. The boundary prompts contain an instruction and/or demonstrations and an open prefix designed to make the target continuation likely; the baselines contain neither. The paper's position control (Table 17) compares last-token vs sequence-mean readout within the same boundary prompts; it does not hold source text constant across the pre-/post-realization divide, and for icl the last-token advantage reverses on two models. A position-matched control—e.g., reading the same natural assistant response at the token immediately before the target is generated versus one or more tokens after—is needed to support the claim that pre-production state, rather t
  2. [§6.3, Table 5] The projection ablation assumes that the boundary Diff-PCA direction is specifically 'execution-relevant.' Without a control direction—for example, a boundary-derived direction from a matched tail-only or an unrelated task—the drop in Table 5 could equally be explained by removing a generic instruction-following component. The relation-task dissociation in §6.2 inherits the same source-text confound, so it does not independently resolve the boundary-vs-instruction ambiguity.
  3. [§7.2, Table 7] Tail subtraction relies on hand-constructed tail-only prompts that remove target conditioning while keeping the local query and assistant prefix. The diagnostic in Table 6 shows high cosine similarity between full and tail states, but this similarity may largely reflect the shared question/prefix rather than 'local continuation semantics.' The practical gains are plausible, but the interpretation that this isolates a phase-specific execution signal needs a more controlled comparison, e.g., subtracting a matched unrelated prompt with the same prefix but different local semantics.
minor comments (5)
  1. [Table 2] The check-marks table is hard to parse; labeling each condition explicitly (e.g., 'Prompt-only, Last' vs 'Prompt-only, Mean') would improve readability.
  2. [Fig. 2] The numbers on bars are small and sometimes overlap; consider annotating with values in a table or enlarging the font.
  3. [Appendix D] 'Paired hierarchical uncertainty intervals' is not defined; please specify the model and how the intervals were computed.
  4. [§3.3] The SAE construction differs from 'feature steering' in prior work; the paper should state explicitly that the SAE is used only as a source of candidate vectors, not as a reconstruction-based intervention, which is done later in Appendix C but could be moved earlier.
  5. [References] Several references are dated 2026 (e.g., You et al., Spherical Steering), which is fine for an arXiv preprint but should be verified as published or clearly marked as preprints.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the empirical derivation is self-contained, with held-out evaluation and hypothesis-driven ablations.

full rationale

The paper does not derive its central claim from its definitions by construction. The execution-boundary state is defined positionally in Section 5.1 as b_l(z)=h_{l,|p|}(p), i.e., the hidden state immediately before a target-bearing continuation, and then tested against post-realization trace states; steerability is measured, not assumed, from this definition. The refined boundary prompts are constructed to make the target continuation imminent, but the outcome variable (steering success on held-out questions) is independent of the source construction, and Appendix D shows the main source-condition gains persist when source layer, intervention layer, and strength are selected only on validation questions. Tail subtraction (Section 7.2) is a motivated construction whose benefit is measured out-of-sample, and the projection ablation in Table 5 removes a boundary-derived direction from answer-based vectors and measures a drop, rather than fitting a parameter to reproduce the target. There are no load-bearing self-citations, no imported uniqueness theorems, and no renamed known results that function as the argument. The main residual concern raised by the skeptic is construct validity: boundary prompts differ from post-realization baselines in both source text and readout position, so the advantage could partly reflect better-specified source text. That is a confounding/interpretation issue, not a circularity-by-construction issue, and the paper partially addresses it with the last-token vs. sequence-mean readout comparison (Table 17) and the matched tail subtraction diagnostic. The paper's own Limitations section honestly discloses the best-over-grid protocol and reliance on an automatic judge, but these are evaluation caveats rather than circular reasoning. Accordingly, no specific circular step can be quoted, and the appropriate score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The paper contributes a measurement and an interpretation, not a derivation. Its claims rest on standard activation-steering practice (additive residual-stream vectors transfer behavior), the domain assumption that the last pre-answer token state is an 'execution boundary', an automatic judge assumed to track human labels, and hand-built boundary prompts that encode the theory. Free parameters (steering strength, layer choices, demo/source counts) are fitted to evaluation data or chosen by hand; none of the headline conclusions are parameter-free derivations.

free parameters (5)
  • Steering strength α = grid maximum, per method/model/condition; values not reported
    Swept in 'method- and model-appropriate ranges' and the best score is retained (Sections 3.4, 4.2). The headline scores are the result of this fit.
  • Source layer ℓs and intervention layer ℓi = best of roughly four evenly spaced depths per model
    Selected per condition by grid search on the evaluation data (Section 3.4); Appendix D mitigates with validation-only selection for a subset.
  • In-context demonstration count = 3
    Hand-chosen for icl and hybrid sources (Section 5.2); no ablation on demo count.
  • Source prompt count = 16
    Hand-chosen for vector construction (Appendix A); no sensitivity analysis.
  • SAE consensus-feature selection threshold = unspecified
    Features 'consistently activated' across a batch are selected (Appendix C); the exact threshold and feature count are not given.
assumptions (5)
  • domain assumption Additive residual-stream steering: adding direction v to hidden states transfers the behavior v encodes (Eq. 1 makes this the fixed intervention).
    The entire success metric presupposes that additive activation steering works as a behavioral control; the paper holds this fixed rather than deriving it (Section 2.1).
  • domain assumption The last-token hidden state at |p| represents imminent production of the continuation (execution-boundary state b_ℓ(z) = h_{ℓ,|p|}(p)).
    Defined in Section 5.1 and instantiated in all boundary-source experiments; if this position does not carry 'about-to-do' semantics, the interpretation collapses.
  • domain assumption Qwen2.5-14B-Instruct judge labels (target expression + generation validity) match human labels on unvalidated outputs.
    Validated at 87.0% agreement on n=400 (Table 13); a Gemma-3-27B rescore preserves the main pattern (Table 14), but the full behavioral evaluation still uses only automatic labels.
  • ad hoc to paper The hand-built open prompt–prefix pairs stop 'just before the target behavior would be realized or continued'.
    This construction (Section 5.2, Appendix A) embeds the paper's own boundary theory in the stimulus; a different theory of what those prefixes do would change the interpretation.
  • domain assumption In relation tasks, the model at the boundary is genuinely about to execute the intended mapping (e.g., big→small).
    Required for Figure 3 to be a clean dissociation between target-appearance and execution-continuation (Section 6.2).
invented entities (3)
  • Execution-boundary state b_ℓ(z) independent evidence
    purpose: Theoretical construct claimed to carry target-preparation signal in steering; defined as h_{ℓ,|p|}(p) (Section 5.1); motivates tail subtraction and the projection ablation.
    Has falsifiable handles inside the paper (position sensitivity Table 17, ablation Table 5, relation-task split Figure 3), but all handles are produced by the same authors; no external verification.
  • Post-realization trace state r_{ℓ,k}(z)
    purpose: Contrast construct for states after the target has appeared; used to define the weak-source condition.
    Defined contrastively in Section 5.1; its predictive power is only exercised internally.
  • Tail-only state h_tail
    purpose: Matched local-continuation reference subtracted in tail subtraction (Section 7.2).
    A procedural baseline construct; no independent evidence beyond the reported gains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Where Steering Signals Come From: Activation Source Selection in Activation Steering." pith.science (2026). https://pith.science/paper/Q3VOAIE4

@misc{pith2026260725270,
  author       = {Pith},
  title        = {Pith review of: Where Steering Signals Come From: Activation Source Selection in Activation Steering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q3VOAIE4}},
  note         = {Machine review of arXiv:2607.25270}
}
read the original abstract

Activation steering controls language models by adding vectors or features to hidden states at inference time, but the upstream source of these steering signals is often treated as a secondary detail. We study this source choice as activation source selection: the combination of source context and activation readout policy used to collect the hidden states from which a steering signal is built. Holding the downstream intervention fixed, we show across three instruction-tuned models and four steering task families that changing only the source activations substantially changes steering success. We further find that effective steering is not explained simply by whether the desired behavior appears in the source text. Instead, strong signals come from execution-boundary states, where the model is about to produce or continue the target behavior. This pre-/post-realization distinction explains why answer-based sources sometimes work: their useful component aligns with execution-boundary directions rather than target appearance alone. Building on this view, we introduce tail subtraction, which removes shared prompt and continuation semantics from boundary states and yields cleaner, more stable steering signals. Overall, our results suggest that steering depends on representations of what the model is about to do, not merely on what has already appeared.

Figures

Figures reproduced from arXiv: 2607.25270 by the authors.

Figure 1
Figure 1. Overview of our activation source view. With the downstream steering pipeline fixed, different source [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Refined semantic source comparison. The first three conditions are boundary sources with last-token [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Relation-task source comparison. Boundary [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 15 linked inside Pith

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  2. [2]

    Publications Manual , year = "1983", publisher =

    1983

  3. [3]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    arXiv 1981

  4. [4]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  5. [5]

    Dan Gusfield , title =. 1997

    1997

  6. [6]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  7. [7]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  8. [9]

    Kenneth Li and Oam Patel and Fernanda B. Vi. Inference-Time Intervention: Eliciting Truthful Answers from a Language Model , booktitle =. 2023 , url =

    2023

Show all 61 references
  1. [11]

    The Thirteenth International Conference on Learning Representations,

    Alessandro Stolfo and Vidhisha Balachandran and Safoora Yousefi and Eric Horvitz and Besmira Nushi , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =

    2025

  2. [12]

    The Thirteenth International Conference on Learning Representations,

    Weixuan Wang and Jingyuan Yang and Wei Peng , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =

    2025

  3. [16]

    Advances in Neural Information Processing Systems , volume =

    Andy Arditi and Oscar Obeso and Aaquib Syed and Daniel Paleka and Nina Panickssery and Wes Gurnee and Neel Nanda , title =. Advances in Neural Information Processing Systems , volume =. 2024 , url =

    2024

  4. [18]

    Turner and Callum McDougall and Monte MacDiarmid and C

    Adly Templeton and Tom Conerly and Jonathan Marcus and Jack Lindsey and Trenton Bricken and Brian Chen and Adam Pearce and Craig Citro and Emmanuel Ameisen and Andy Jones and Hoagy Cunningham and Nicholas L. Turner and Callum McDougall and Monte MacDiarmid and C. Daniel Freema...

  5. [24]

    Can Activation Steering Generalize Across Languages?

    Gabriele Maraia and Leonardo Ranaldi and Marco Valentino and Fabio Massimo Zanzotto , editor =. Can Activation Steering Generalize Across Languages?. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics,. 2026 , url =

    2026

  6. [25]

    Mitigating Content Effects on Reasoning in Language Models Through Fine-Grained Activation Steering , booktitle =

    Marco Valentino and Geonhee Kim and Dhairya Dalal and Zhixue Zhao and Andr. Mitigating Content Effects on Reasoning in Language Models Through Fine-Grained Activation Steering , booktitle =. 2026 , url =

    2026

  7. [26]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Nina Rimsky and Nick Gabrieli and Julian Schulz and Meg Tong and Evan Hubinger and Alexander Matt Turner , title =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2024 , url =

    2024

  8. [27]

    Jentzsch and Diaoul

    Kai Konen and Sophie F. Jentzsch and Diaoul. Style Vectors for Steering Generative Large Language Models , booktitle =. 2024 , url =

    2024

  9. [30]

    Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 , booktitle =

    Tom Lieberum and Senthooran Rajamanoharan and Arthur Conmy and Lewis Smith and Nicolas Sonnerat and Vikrant Varma and J. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 , booktitle =. 2024 , url =

    2024

  10. [36]

    Lee and Inkit Padhi and Karthikeyan Natesan Ramamurthy and Erik Miehling and Pierre L

    Bruce W. Lee and Inkit Padhi and Karthikeyan Natesan Ramamurthy and Erik Miehling and Pierre L. Dognin and Manish Nagireddy and Amit Dhurandhar , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =

    2025

  11. [39]

    The Twelfth International Conference on Learning Representations,

    Robert Huben and Hoagy Cunningham and Logan Riggs Smith and Aidan Ewart and Lee Sharkey , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    2024

  12. [40]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages =

    Sewon Min and Xinxi Lyu and Ari Holtzman and Mikel Artetxe and Mike Lewis and Hannaneh Hajishirzi and Luke Zettlemoyer , title =. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages =. 2022 , doi =

    2022

  13. [41]

    Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022 , year =

    Shivam Garg and Dimitris Tsipras and Percy Liang and Gregory Valiant , title =. Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022 , year =

    2022

  14. [42]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages =

    Roee Hendel and Mor Geva and Amir Globerson , title =. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages =. 2023 , doi =

    2023

  15. [43]

    Li and Arnab Sen Sharma and Aaron Mueller and Byron C

    Eric Todd and Millicent L. Li and Arnab Sen Sharma and Aaron Mueller and Byron C. Wallace and David Bau , title =. International Conference on Learning Representations , year =

  16. [46]

    Manning and Christopher Potts , editor =

    Zhengxuan Wu and Aryaman Arora and Atticus Geiger and Zheng Wang and Jing Huang and Dan Jurafsky and Christopher D. Manning and Christopher Potts , editor =. AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders , booktitle =. 2025 , url =

    2025

  17. [47]

    Zhang, and James Y

    Abubakar Abid, Vivek Kumar Bagaria, Martin J. Zhang, and James Y. Zou. 2017. https://arxiv.org/abs/1709.06716 Contrastive principal component analysis . CoRR, abs/1709.06716

    2017 arXiv

  18. [48]

    Guillaume Alain and Yoshua Bengio. 2016. https://arxiv.org/abs/1610.01644 Understanding intermediate layers using linear classifier probes . CoRR, abs/1610.01644

    2016 arXiv

  19. [49]

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. https://doi.org/10.52202/079017-4322 Refusal in language models is mediated by a single direction . Advances in Neural Information Processing Systems, 37

    2024 doi

  20. [50]

    Yonatan Belinkov and James R. Glass. 2019. https://doi.org/10.1162/TACL\_A\_00254 Analysis methods in neural language processing: A survey . Trans. Assoc. Comput. Linguistics, 7:49--72

    2019 doi

  21. [51]

    Sviatoslav Chalnev, Matthew Siu, and Arthur Conmy. 2024. https://doi.org/10.48550/ARXIV.2411.02193 Improving steering vectors by targeting sparse autoencoder features . CoRR, abs/2411.02193

    2024 doi arXiv

  22. [52]

    Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. 2025. https://doi.org/10.48550/ARXIV.2507.21509 Persona vectors: Monitoring and controlling character traits in language models . CoRR, abs/2507.21509

    2025 doi arXiv

  23. [53]

    Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/hash/c529dba08a146ea8d6cf715ae8930cbe-Abstract-Conference.html What can transformers learn in-context? A case study of simple function classes . In Adva...

    2022

  24. [54]

    Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu - Gang Jiang, and Xipeng Qiu. 2024. https://doi.org/10.48550/ARXIV.2410.20526 Llama scope: Extracting millions of features from llama-3.1-8b with ...

    2024 doi arXiv

  25. [55]

    Zirui He, Haiyan Zhao, Yiran Qiao, Fan Yang, Ali Payani, Jing Ma, and Mengnan Du. 2025. https://doi.org/10.48550/ARXIV.2502.11356 SAIF: A sparse autoencoder framework for interpreting and steering instruction following of language models . CoRR, abs/2502.11356

    2025 doi arXiv

  26. [56]

    Roee Hendel, Mor Geva, and Amir Globerson. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.624 In-context learning creates task vectors . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318--9333

    2023 doi

  27. [57]

    Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2024. https://openreview.net/forum?id=F76bwRSLeK Sparse autoencoders find highly interpretable features in language models . In The Twelfth International Conference on Learning Representations, IC...

    2024

  28. [58]

    Ole Jorgensen, Dylan Cope, Nandi Schoots, and Murray Shanahan. 2023. https://doi.org/10.48550/ARXIV.2312.03813 Improving activation steering in language models with mean-centring . CoRR, abs/2312.03813

    2023 doi arXiv

  29. [59]

    Jentzsch, Diaoul \' e Diallo, Peer Sch \"u tt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias Hecking

    Kai Konen, Sophie F. Jentzsch, Diaoul \' e Diallo, Peer Sch \"u tt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias Hecking. 2024. https://doi.org/10.18653/v1/2024.findings-eacl.52 Style vectors for steering generative large language models . In Findings of the Assoc...

    2024 doi

  30. [60]

    Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre L

    Bruce W. Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre L. Dognin, Manish Nagireddy, and Amit Dhurandhar. 2025. https://openreview.net/forum?id=Oi47wc10sm Programming refusal with conditional activation steering . In The Thirteenth International Confer...

    2025

  31. [61]

    Vi \' e gas, Hanspeter Pfister, and Martin Wattenberg

    Kenneth Li, Oam Patel, Fernanda B. Vi \' e gas, Hanspeter Pfister, and Martin Wattenberg. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/81b8390039b7302c909cb769f8b6cd93-Abstract-Conference.html Inference-time intervention: Eliciting truthful answers from a l...

    2023

  32. [62]

    Yichen Li, Zhiting Fan, Ruizhe Chen, Xiaotang Gai, Luqi Gong, Yan Zhang, and Zuozhu Liu. 2025. https://doi.org/10.18653/v1/2025.findings-acl.589 FairSteer: inference time debiasing for LLMs with dynamic activation steering . In Findings of the Association for Computational Lin...

    2025 doi

  33. [63]

    Dragan, Rohin Shah, and Neel Nanda

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J \' a nos Kram \' a r, Anca D. Dragan, Rohin Shah, and Neel Nanda. 2024. https://doi.org/10.18653/v1/2024.blackboxnlp-1.19 Gemma scope: Open sparse autoencoders everywhere all ...

    2024 doi

  34. [64]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.759 Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on ...

    2022 doi

  35. [65]

    Joris Postmus and Steven Abreu. 2024. https://doi.org/10.48550/ARXIV.2410.16314 Steering large language models using conceptors: Improving addition-based activation engineering . CoRR, abs/2410.16314

    2024 doi arXiv

  36. [66]

    Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.828 Steering llama 2 via contrastive activation addition . In Proceedings of the 62nd Annual Meeting of the Association for Computatio...

    2024 doi

  37. [67]

    Marco Scialanga, Thibault Laugel, Vincent Grari, and Marcin Detyniecki. 2025. https://doi.org/10.18653/v1/2025.acl-long.777 SAKE: steering activations for knowledge editing . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: ...

    2025 doi

  38. [68]

    Niklas Stoehr, Kevin Du, V \' e steinn Sn bjarnarson, Robert West, Ryan Cotterell, and Aaron Schein. 2024. https://doi.org/10.18653/V1/2024.FINDINGS-EMNLP.479 Activation scaling for steering and interpreting language models . In Findings of the Association for Computational Li...

    2024 doi

  39. [69]

    Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. 2025. https://openreview.net/forum?id=wozhdnRCtw Improving instruction-following in language models through activation steering . In The Thirteenth International Conference on Learning ...

    2025

  40. [70]

    Jingran Su, Jingfan Chen, Hongxin Li, Yuntao Chen, Li Qing, and Zhaoxiang Zhang. 2025. https://doi.org/10.18653/v1/2025.acl-long.634 Activation steering decoding: Mitigating hallucination in large vision-language models through bidirectional hidden state intervention . In Proc...

    2025 doi

  41. [71]

    Nishant Subramani, Nivedita Suresh, and Matthew E. Peters. 2022. https://doi.org/10.18653/v1/2022.findings-acl.48 Extracting latent steering vectors from pretrained language models . In Findings of the Association for Computational Linguistics: ACL 2022 , pages 566--581. Assoc...

    2022 doi

  42. [72]

    Gemma Team. 2024 a . https://doi.org/10.48550/ARXIV.2408.00118 Gemma 2: Improving open language models at a practical size . CoRR, abs/2408.00118

    2024 doi arXiv

  43. [73]

    Gemma Team. 2025. https://doi.org/10.48550/ARXIV.2503.19786 Gemma 3 technical report . CoRR, abs/2503.19786

    2025 doi arXiv

  44. [74]

    Llama Team. 2024 b . https://doi.org/10.48550/ARXIV.2407.21783 The llama 3 herd of models . CoRR, abs/2407.21783

    2024 doi arXiv

  45. [75]

    Turner, Callum McDougall, Monte MacDiarmid, C

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua...

    2024

  46. [76]

    Li, Arnab Sen Sharma, Aaron Mueller, Byron C

    Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. 2024. https://openreview.net/forum?id=AwyxtyMwaG Function vectors in large language models . In International Conference on Learning Representations

    2024

  47. [77]

    Vazquez, Ulisse Mini, and Monte MacDiarmid

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. https://doi.org/10.48550/ARXIV.2308.10248 Steering language models with activation engineering . CoRR, abs/2308.10248

    2023 doi arXiv

  48. [78]

    Weixuan Wang, Jingyuan Yang, and Wei Peng. 2025 a . https://openreview.net/forum?id=8WQ7VTfPTl Semantics-adaptive activation intervention for LLMs via dynamic steering vectors . In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April...

    2025

  49. [79]

    Xinpeng Wang, Mingyang Wang, Yihong Liu, Hinrich Sch \" u tze, and Barbara Plank. 2025 b . https://doi.org/10.48550/ARXIV.2505.17306 Refusal direction is universal across safety-aligned languages . CoRR, abs/2505.17306

    2025 doi

  50. [80]

    Manning, and Christopher Potts

    Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. 2025. https://proceedings.mlr.press/v267/wu25a.html Axbench: Steering llms? even simple baselines outperform sparse autoencoders . In Forty-second ...

    2025

  51. [81]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2024. https://doi.org/10.485...

    2024 doi arXiv

  52. [82]

    Zejia You, Chunyuan Deng, and Hanjie Chen. 2026. https://doi.org/10.48550/ARXIV.2602.08169 Spherical steering: Geometry-aware activation rotation for language models . CoRR, abs/2602.08169

    2026 doi arXiv

  53. [83]

    Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, and 2 others

    Andy Zou, Long Phan, Sarah Li Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann - Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, an...

    2023 doi arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.