Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Understanding (Un)Reliability of Steering Vectors in Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Steering vectors work only when a behavior has one coherent direction in activation space.

desk verdict Solid empirical study with a useful cheap predictor for CAA steering success, but the central interpretation is undercut by a possible answer-token identity confound. read the letter →

arxiv 2505.22637 v1 pith:STOKPVAD submitted 2025-05-28 cs.LG

Joschka Braun , Carsten Eickhoff , David Krueger , Seyed Ali Bahrainian , Dmitrii Krasheninnikov This is my paper
classification cs.LG
keywords steeringvectorscontrastiveactivationadditiongeometrydirectionalagreementdiscriminabilityindexlinearrepresentationlanguagemodelreliabilitylogitdifference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks why activation steering sometimes fails, and argues that the answer lies in the geometry of the model's activations, not in the choice of prompt wording. Using Contrastive Activation Addition on Llama2-7B-Chat across 36 behavior datasets, the authors find that all seven prompt types they tried give a net positive steering effect on average, yet nearly a third of individual samples move in the opposite direction. They then show that two cheaply computable geometric measurements—the cosine similarity between individual activation differences and the steering vector, and the separability of positive versus negative activations along the difference-of-means line—predict both the size of the steering effect and how often steering backfires. The conclusion is that a behavior is reliably steerable by CAA exactly when it is represented by a coherent linear direction in activation space.

What carries the argument

The carrying object is the CAA steering vector, $\mathbf{s}_l = \frac{1}{|\mathcal{D}|}\sum_{(x,y^+,y^-)\in\mathcal{D}}(\mathbf{a}_l(x,y^+) - \mathbf{a}_l(x,y^-))$, a single vector added to residual-stream activations at layer 13 during inference. Around this, the paper builds two geometric diagnostics. The first is directional agreement: the cosine similarity between each training activation difference and $\mathbf{s}_l$, averaged over the dataset. The second is the difference-of-means line, the line through the mean positive activation $\boldsymbol{\mu}_{l,+}$ and mean negative activation $\boldsymbol{\mu}_{l,-}$, parameterized as $\mathrm{doml}_l(\boldsymbol{\mu}_+,\boldsymbol{\mu}_-) = \frac{1+\kappa}{2}\boldsymbol{\mu}_{l,+} + \frac{1-\kappa}{2}\boldsymbol{\mu}_{l,-}$, onto which activations are projected and summarized by the discriminability index $d' = |\mu_+ - \mu_-| / \sqrt{\tfrac{1}{2}(\sigma_+^2 + \sigma_-^2)}$. Steering success is measured by $\Delta m_{\mathrm{LD}}$, the change in logit-difference between the desired and undesired answer token when steering is applied. The machinery shows that the same geometry that defines the steering vector also predicts whether the vector will work.

What would settle it

Compute the same predictors at other layers and multipliers: if there exists a layer where all 36 datasets show high $d'$ and high cosine similarity yet steering effect is weak or anti-steerable, or a setting where steering works well despite low coherence, the proposed explanation fails. A direct test would be to measure Spearman correlation between mean cosine similarity and $\Delta m_{\mathrm{LD}}$ at layers other than 13 (e.g., layers 5, 20, 30) and at multipliers $\lambda=0.5$ and $\lambda=3$; the claim predicts the correlations survive.

Watch

Extended reading notes

Core claim

The paper's central claim is that steering-vector reliability is a property of the dataset's activation geometry. For each of the 36 datasets, they compute the CAA steering vector as the mean difference between positive and negative residual-stream activations at layer 13, then measure, per training sample, the cosine similarity between the individual activation difference and the steering vector, and the discriminability index $d'$ of the two activation classes projected onto the difference-of-means line. Datasets ranked as most steerable have mean cosine similarities near 0.48, while the least steerable hover near 0.19, and $d'$ drops correspondingly; both quantities correlate strongly (Spearman 0.76 for effect size, -0.78 for anti-steerable fraction) with steering performance. The paper interprets this as evidence that unreliable steering occurs when the target behavior is not consistently encoded as a single linear direction in the residual stream. Prompt type has only a limited influence: vectors trained with different prompt formats point in quite different directions (cosine similarities as low as 0.07), yet all produce similar average steering effects, reinforcing that dataset-specific geometry dominates.

Load-bearing premise

The whole explanation rests on assuming that steering success is faithfully captured by the change in logit-difference measured at layer 13 with multiplier $\lambda=1$, and that the activation geometry at that single layer is the geometry that governs steering; if the link between coherence and effectiveness shifts with layer or steering strength, the proposed predictor may not generalize.

Editorial extensions

If this is right

  • Steering effectiveness can be predicted before applying the intervention by averaging per-sample cosine similarity between activation differences and the steering vector, or by computing $d'$ along the difference-of-means line.
  • Prompt-type choice has limited impact on average steering effect, so tuning prompt phrasing cannot fix unreliability that stems from a scattered activation geometry.
  • Reliability varies by dataset: roughly one-third of samples are anti-steerable on average, ranging from 3% to 50% per dataset, so per-sample variance is inherent to datasets with weak linear structure.
  • The results delimit the applicability of CAA-style vector steering to behaviors that are linearly represented; for other behaviors, the same method will be unreliable regardless of prompt format.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If coherence of activation differences is the true driver, then failure cases are not a tuning problem: no multiplier or prompt format can make a scattered direction behave like a coherent one, and efforts should shift to steering methods that do not assume a single linear shift.
  • A practical extension suggested by the paper is a pre-hoc 'steerability score' for new datasets: compute mean cosine similarity and $d'$ on a small training set before deciding whether vector steering is worth applying.
  • The geometry measured here is layer-13-specific; a natural test is whether the coherence-to-steerability relationship holds at all layers or only at the layer where the steering vector is applied, which would sharpen the mechanistic story.
  • Because the paper only tests CAA, the same two predictors could be measured for other linear steering methods (e.g., function vectors) to see whether directional agreement is a universal requirement or an artifact of the mean-difference construction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies when and why Contrastive Activation Addition (CAA) steering vectors are reliable, using Llama2-7B-Chat on 36 multiple-choice behavior datasets. It compares seven prompt types for training steering vectors, finding that all yield a net positive average steering effect but with high per-sample variance, and that no prompt type clearly outperforms the others. The main contribution is a proposed explanation based on activation geometry: datasets whose training activation differences are well aligned with the steering vector (high cosine similarity) and where positive and negative activations are well separated along the difference-of-means line (high d') exhibit stronger and more reliable steering. These geometric predictors correlate with held-out steering success (Spearman 0.76 for effect size and -0.78 for anti-steerable fraction), leading the authors to conclude that steering is unreliable when the target behavior is not represented as a coherent linear direction.

Significance. If the geometric predictors genuinely capture whether a behavior is linearly represented, the paper offers a cheap, pre-hoc diagnostic for when vector steering will work, which would be practically useful and conceptually clarifying. The reported correlations are strong and are computed against held-out evaluation prompts, and the paper is transparent about its limitations, including the restriction to a single model, a single layer, and CAA only. The central interpretation, however, is threatened by a confound in the way the prefilled training activations are constructed, which I detail in the major comments. The paper's strengths are its clear experimental design, direct comparison to prior work, and honest acknowledgment of scope; its main claim needs additional work to rule out the token-identity alternative.

major comments (3)
  1. [§2 (Steering Method) and §3 (Directional Agreement Predicts Steerability)] The geometric predictors are computed on activations from the prefilled prompt type, where the positive and negative examples differ only by the appended answer token (e.g., 'A' versus 'B'). Because the residual stream at the answer-token position encodes the identity of that token, the activation differences and the steering vector contain a substantial component tied to answer-token identity rather than to the target behavior. The held-out evaluation uses plain prompts without an appended answer token, so the reported correlations (Appendix D: Spearman 0.76 for steerability and -0.78 for anti-steerable fraction) may reflect a token-prior effect rather than a linear representation of the behavior. This directly affects the central claim in Section 4 that 'steering vectors are not universally applicable, and that their effectiveness depends on whether the targeted behavior is well-represented as a linear direction.' I request that the authors recompute the predictors on non-prefilled prompt types (instruction and/or 5-shot), where the positive and negative prompts differ in content rather than in the final token, or otherwise control for the answer-token direction (e.g., by subtracting the mean answer-token embedding direction) and show that the predictive relationship persists.
  2. [§2 (Experimental Setup) and §4 (Limitations)] All geometric predictors and steering evaluations are performed with a single layer, l=13, and a single multiplier, λ=1. The paper's conceptual conclusion is that steering reliability depends on whether the behavior is linearly represented in activation space, which is a general claim about the model's geometry. As it stands, the evidence is limited to one depth and one intervention strength. The limitations discussion lists breadth of models, datasets, and steering methods, but does not address sensitivity to layer or multiplier. I ask the authors to report whether the geometry-effectiveness relationship holds at other layers (e.g., layers 10 and 16) and for at least one other multiplier (e.g., λ=0.5 and λ=2). If the relationship reverses or weakens, the statement in Section 4 that 'both directional consistency of activation differences and separability of activations along the difference-of-means line are conceptually intuitive explanations and empirical predictors of steering vector performance' would need to be qualified.
  3. [§3 (Effect of Prompt Types on Steering Vectors) and §4 (Methodology for Prompt Type Comparison)] The comparison of prompt types is presented graphically without error bars or significance tests. The paper states that 'all seven prompt types produce a net positive steering effect' and that 'no prompt type clearly outperforms the others,' but Figure 1 shows only per-sample distributions and means. The authors acknowledge in Section 4 that statistical comparison is highly sensitive to hyperparameters and that they did not run such tests. This is acceptable as an exploratory finding, but the claims are stronger than the evidence supports. Please either add confidence intervals or paired significance tests, or soften the claims to state that no prompt type is consistently ranked best in this dataset set.
minor comments (6)
  1. [Figure 1 and Figure 5 captions] The text 'faction of such anti-steerable samples' should be 'fraction of such anti-steerable samples.'
  2. [Appendix C.2] The sentence 'now single prompt type is systematically better than the others' should read 'no single prompt type is systematically better than the others.'
  3. [Appendix A example] In the example prompt, the prefilled answer token is written as 'Answer: (A' without a closing parenthesis; it should be 'Answer: (A)' for consistency with the other choices.
  4. [Figure 2 caption and Section 3] The caption says the datasets are ordered by 'steerability rank from Tan et al. (2024),' but Section 3 also discusses the authors' own steerability measurements. This is potentially confusing; please clarify whether the ordering in Figure 2 is based on Tan et al.'s ranking, the authors' own evaluation, or both, and how that relates to the Spearman correlations in Appendix D.
  5. [Appendix D, Figure 6] The x-axis label 'Mean per-sample steerability' is not defined in the caption; please define it explicitly (e.g., mean Δm_LD over the held-out test set).
  6. [§2 (Evaluation of Steering Success)] The paper uses λ=1 for 'most' experiments but does not specify which analyses use other multipliers. Please state explicitly whether the geometric-predictor correlations and the prompt-type comparisons all use λ=1, and whether any results use different values.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the geometric predictors are computed from training activations before held-out steering evaluation, so the central correlation is not forced by construction.

full rationale

The paper's central empirical claim is that directional agreement (mean cosine similarity between per-sample activation differences and the steering vector) and separability (d' along the difference-of-means line) predict steering effectiveness. The steering vector s_l is defined as the mean of the training activation differences (Section 2), and both predictors are computed from the same training activations, so there is an element of self-reference. However, the predicted quantity, ΔmLD and the fraction of anti-steerable samples, is measured on held-out plain prompts after adding λs_l at layer 13 (Section 2), and no parameter is fitted to that held-out outcome. The Spearman correlations in Appendix D are pre-hoc geometric statistics versus independently measured steering outcomes, not a fitted parameter renamed as a prediction. The mathematical relation between the mean vector and per-sample cosine does not by itself determine the held-out logit change; the empirical correlation is a substantive result, even though it may partly reflect the norm of the steering vector. The paper's own limitations (single model, CAA only, layer 13, λ=1) and the potential confound that prefilled answer-token activations encode answer-token identity are external-validity concerns, not circular reductions. Self-citations in the introduction and Appendix B (Braun et al. 2024; Brumley et al. 2024; Pres et al. 2024; Krasheninnikov and Krueger 2024) are contextual background and are not load-bearing for the paper's empirical conclusion. No equation in the paper makes the predictor equal to the outcome by construction, so no specific circular step can be identified.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities. It relies on standard domain assumptions about the validity of logit-difference evaluation, representativeness of the benchmark suite, and the choice of prefilled prompts for geometric analysis. The only hand-chosen experimental constants are λ=1, layer 13, and the 250-sample training set size, each of which could affect the reported correlations.

free parameters (3)
  • Steering multiplier λ = λ = 1
    Chosen by hand and used for nearly all experiments; the central results are not shown to be robust to this choice.
  • Layer l = l = 13
    Chosen following (Tan et al., 2024); the layer at which residual activations are recorded and steering is applied. The geometry-behavior relations may depend on this choice.
  • Training set size = 250 training samples, 500 evaluation samples
    The authors note in Section 4 that prompt-type comparisons are highly sensitive to this hyperparameter, so it is an influential hand-chosen setting.
assumptions (3)
  • domain assumption The logit-difference propensity metric Δm_LD accurately reflects steering success for the behaviors studied.
    The paper defines reliability via Δm_LD and anti-steerable fraction (Section 2), but does not validate against free-form generation or downstream task metrics.
  • domain assumption The 36 multiple-choice behavior datasets by Rogers et al. and Llama2-7B-Chat are representative enough to support general claims about steering reliability.
    The paper's conclusions are drawn from one model and one dataset family; the authors acknowledge limited breadth in Section 4.
  • domain assumption Activation differences recorded at the answer token position (prefilled) capture the same behavior-relevant information as other prompt formats.
    After the prompt-type comparison, all geometry analysis uses only the prefilled prompt type, assuming its activations are the appropriate object for studying steerability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding (Un)Reliability of Steering Vectors in Language Models." pith.science (2026). https://pith.science/paper/STOKPVAD

@misc{pith2026250522637,
  author       = {Pith},
  title        = {Pith review of: Understanding (Un)Reliability of Steering Vectors in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/STOKPVAD}},
  note         = {Machine review of arXiv:2505.22637}
}
read the original abstract

Steering vectors are a lightweight method to control language model behavior by adding a learned bias to the activations at inference time. Although steering demonstrates promising performance, recent work shows that it can be unreliable or even counterproductive in some cases. This paper studies the influence of prompt types and the geometry of activation differences on steering reliability. First, we find that all seven prompt types used in our experiments produce a net positive steering effect, but exhibit high variance across samples, and often give an effect opposite of the desired one. No prompt type clearly outperforms the others, and yet the steering vectors resulting from the different prompt types often differ directionally (as measured by cosine similarity). Second, we show that higher cosine similarity between training set activation differences predicts more effective steering. Finally, we observe that datasets where positive and negative activations are better separated are more steerable. Our results suggest that vector steering is unreliable when the target behavior is not represented by a coherent direction.

Figures

Figures reproduced from arXiv: 2505.22637 by the authors.

Figure 1
Figure 1. Steering vectors trained with different prompt types all increase the mean logit-difference [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. We group the 36 datasets by how effective the resulting steering vector is (“steerability [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. For datasets where the behavior is steerable, activations are clearly separated along the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Steering vectors (SVs) trained on the same datasets but with different prompt types have [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Steering vectors trained with different prompt types all increase the mean logit-difference [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: The mean cosine similarity of activation differences on the training dataset, are a predictor [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: The nine most steerable datasets have high discriminability along the difference-of-means [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: The nine next most steerable datasets are slightly less discriminable. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: As steerability decreases, discriminability decreases as well and distributions of positive [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: The nine least steerable datasets overlap along the difference-of-means line and also have [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Conditional Optimal Bridge for Riemannian Activation Steering

    cs.LG 2026-07 accept novelty 7.0 of 10

    Casting activation steering as a Schrödinger Bridge on the residual hypersphere derives the log-density-ratio objective and yields query-adaptive directions that beat fixed baselines without OOD collapse.

  2. Temporal Preference Concepts and their Functions in a Large Language Model

    cs.LG 2026-05 unverdicted novelty 6.5 of 10

    Temporal preference in Qwen3-4B-Instruct-2507 localizes to layers 17–35 (especially L24 attention), has curved residual-stream geometry, is behaviorally unstable, and can be bidirectionally steered.

Reference graph

Works this paper leans on

37 extracted references · 9 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Refusal in language models is mediated by a single direction, 2024

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction, 2024. URL https://arxiv.org/abs/2406.11717

    arXiv 2024

  3. [3]

    Cats: Customizable abstractive topic-based summarization

    Seyed Ali Bahrainian, George Zerveas, Fabio Crestani, and Carsten Eickhoff. Cats: Customizable abstractive topic-based summarization. ACM Trans. Inf. Syst., 40 0 (1), oct 2021. ISSN 1046-8188. doi:10.1145/3464299. URL https://doi.org/10.1145/3464299

    doi:10.1145/3464299 2021

  4. [4]

    NEWTS : A corpus for news topic-focused summarization

    Seyed Ali Bahrainian, Sheridan Feucht, and Carsten Eickhoff. NEWTS : A corpus for news topic-focused summarization. In Findings of the Association for Computational Linguistics: ACL 2022, pp.\ 493--503, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi:10.18653/v1/2022.findings-acl.42. URL https://aclanthology.org/2022.findings-acl.42

    doi:10.18653/v1/2022.findings-acl.42 2022

  5. [5]

    Controllable topic-focused abstractive summarization, 2023

    Seyed Ali Bahrainian, Martin Jaggi, and Carsten Eickhoff. Controllable topic-focused abstractive summarization, 2023. URL https://doi.org/10.48550/arXiv.2311.06724

    arXiv doi:10.48550/arxiv.2311.06724 2023

  6. [6]

    Text simplification via adaptive teaching

    Seyed Ali Bahrainian, Jonathan Dou, and Carsten Eickhoff. Text simplification via adaptive teaching. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 6574--6584, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi:10.18653/v1/2024.findings-acl.392....

    work page doi:10.18653/v1/2024.findings-acl.392 2024

  7. [7]

    SIMSUM : Document-level text simplification via simultaneous summarization

    Sofia Blinova, Xinyu Zhou, Martin Jaggi, Carsten Eickhoff, and Seyed Ali Bahrainian. SIMSUM : Document-level text simplification via simultaneous summarization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 9927--9944, Tor...

    work page doi:10.18653/v1/2023.acl-long.552 2023

  8. [8]

    A sober look at steering vectors for llms

    Joschka Braun, Dmitrii Krasheninnikov, Usman Anwar, Robert Kirk, Daniel Chee Hian Tan, and David Scott Krueger. A sober look at steering vectors for llms. AI Alignment Forum, nov 2024. URL https://www.alignmentforum.org/posts/QQP4nq7TXg89CJGBh/a-sober-look-at-steering-vectors-for-llms. Publication Date: 2024-11-23

    work page 2024

Show all 37 references
  1. [9]

    Comparing bottom-up and top-down steering approaches on in-context learning tasks, 2024

    Madeline Brumley, Joe Kwon, David Krueger, Dmitrii Krasheninnikov, and Usman Anwar. Comparing bottom-up and top-down steering approaches on in-context learning tasks, 2024. URL https://arxiv.org/abs/2411.07213

    2024 arXiv

  2. [10]

    Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization

    Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen. Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization. arXiv preprint arXiv:2406.00045, 2024

    2024 arXiv

  3. [11]

    In-context learning creates task vectors

    Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 9318--9333, Singapore, December 2023. Association for Computati...

    2023 doi

  4. [12]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  5. [13]

    Style Vectors for Steering Generative Large Language Models

    Kai Konen, Sophie Freya Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias Hecking. Style Vectors for Steering Generative Large Language Models . In European Chapter of the ACL: (EACL) 2024, St Julians, Malta, 2024. URL https://eli...

    2024

  6. [14]

    Steering clear: A systematic study of activation steering in a toy setup

    Dmitrii Krasheninnikov and David Krueger. Steering clear: A systematic study of activation steering in a toy setup. In MINT workshop: Foundation Model Interventions, 2024

    2024

  7. [15]

    Inference-time intervention: Eliciting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id...

    2023

  8. [16]

    The geometry of truth: Emergent linear structure in large language model representations of true/false datasets, 2024 a

    Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets, 2024 a . URL https://arxiv.org/abs/2310.06824

    2024 arXiv

  9. [17]

    The geometry of truth: Emergent linear structure in large language model representations of true/false datasets

    Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In First Conference on Language Modeling, 2024 b . URL https://openreview.net/forum?id=aajyHYjjsk

    2024

  10. [18]

    Refusal in llms is an affine function

    Thomas Marshall, Adam Scherlis, and Nora Belrose. Refusal in llms is an affine function. arXiv preprint arXiv:2411.09003, 2024

    2024 arXiv

  11. [19]

    Towards reliable evaluation of behavior steering interventions in llms

    Itamar Pres, Laura Ruis, Ekdeep Singh Lubana, and David Krueger. Towards reliable evaluation of behavior steering interventions in llms. arXiv preprint arXiv:2410.17245, 2024

    2024 arXiv

  12. [20]

    Steering llama 2 via contrastive activation addition

    Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama 2 via contrastive activation addition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational ...

    2024 doi

  13. [21]

    Discovering Language Model Behaviors with Model-Written Evaluations, Toronto, Canada, July 2023

    Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Discovering Language Model Behaviors with Model-Written Evaluations, Toronto, Canada, July 2023. Association for Computational Linguistics. doi:10.18653/v1/2023.findings-acl.847. URL https://aclanthology.org/2023.find...

    2023 doi

  14. [22]

    Representation surgery: Theory and practice of affine steering

    Shashwat Singh, Shauli Ravfogel, Jonathan Herzig, Roee Aharoni, Ryan Cotterell, and Ponnurangam Kumaraguru. Representation surgery: Theory and practice of affine steering. In Forty-first International Conference on Machine Learning, 2024

    2024

  15. [23]

    Asa Cooper Stickland, Alexander Lyzhov, Jacob Pfau, Salsabila Mahdi, and Samuel R. Bowman. Steering without side effects: Improving post-deployment control of language models. In Neurips Safe Generative AI Workshop 2024, 2024. URL https://openreview.net/forum?id=tfXIZ8P4ZU

    2024

  16. [24]

    Extracting Latent Steering Vectors from Pretrained Language Models

    Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting Latent Steering Vectors from Pretrained Language Models . In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics : ACL 2022 , pp.\ 566--581...

    2022 doi

  17. [25]

    Analysing the generalisation and reliability of steering vectors

    Daniel Chee Hian Tan, David Chanin, Aengus Lynch, Brooks Paige, Dimitrios Kanoulas, Adri \`a Garriga-Alonso, and Robert Kirk. Analysing the generalisation and reliability of steering vectors. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024...

    2024

  18. [26]

    Linear representations of sentiment in large language models, 2023

    Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models, 2023. URL https://arxiv.org/abs/2310.15154

    2023 arXiv

  19. [27]

    Hollinsworth, Atticus Geiger, and Neel Nanda

    Curt Tigges, Oskar J. Hollinsworth, Atticus Geiger, and Neel Nanda. Language models linearly represent sentiment. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackboxNLP Workshop: Analyzing and I...

    2024 doi

  20. [28]

    Function vectors in large language models

    Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint arXiv:2310.15213, 2023

    2023 arXiv

  21. [29]

    Function vectors in large language models

    Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=AwyxtyMwaG

    2024

  22. [30]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  23. [31]

    Activation addition: Steering language models without optimization

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv e-prints, pp.\ arXiv--2308, 2023

    2023

  24. [32]

    Controllable text summarization: Unraveling challenges, approaches, and prospects - a survey

    Ashok Urlana, Pruthwik Mishra, Tathagato Roy, and Rahul Mishra. Controllable text summarization: Unraveling challenges, approaches, and prospects - a survey. In ACL (Findings), pp.\ 1603--1623, 2024. URL https://doi.org/10.18653/v1/2024.findings-acl.93

    2024 doi

  25. [33]

    A comprehensive survey on process-oriented automatic text summarization with exploration of llm-based methods, 2025

    Yang Zhang, Hanlei Jin, Dan Meng, Jun Wang, and Jinghua Tan. A comprehensive survey on process-oriented automatic text summarization with exploration of llm-based methods, 2025. URL https://arxiv.org/abs/2403.02901

    2025

  26. [35]

    Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico...

    2025 arXiv

  27. [36]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  28. [37]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  29. [38]

    3!( 4˜ "3!( 4˒

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    1976

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.