Pith. sign in

REVIEW 3 major objections 6 minor 68 references

Linearly Decoding Refused Knowledge in Aligned Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Instruction-tuning suppresses refused answers in outputs but leaves the underlying values linearly readable in hidden states, where those values continue to line up with the models' downstream judgments.

desk verdict Worth a serious look: the decodability and transfer results are real, but the abstract's 'actively used' claim outruns the correlation evidence and should be tempered before publication. read the letter →

arxiv 2507.00239 v1 pith:2Y2RPHVW submitted 2025-06-30 cs.CL cs.AI

Aryan Shrivastava , Ari Holtzman This is my paper
classification cs.CLcs.AI
keywords linearprobingjailbreakpromptsrefusalbehaviorinstructiontuninghiddenstaterepresentationsalignmentsafetyBradley-Terrymodelrepresentationhypothesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether instruction-tuning, the stage at which language models learn to refuse harmful requests, actually removes the underlying knowledge or merely hides its expression. The authors show that the numbers a jailbroken model will produce—the average IQ of a country, an occupation's substance-abuse rate, a political figure's net worth—can be recovered by a linear probe trained on hidden states from the innocuous prompt 'This document describes [entity],' with Pearson correlations above 0.8 in the strongest cases. Probes trained on base models that never refuse sometimes transfer to the instruction-tuned versions, indicating the refused attributes persist in much the same linear form through alignment. The sharpest claim is that these traces are not inert: probe-predicted values correlate with the model's own pairwise-comparison rankings, which suggests the suppressed information still influences how the model makes judgments. If true, alignment training changes what models are allowed to say more than what they represent and believe.

What carries the argument

The argument is carried by three components: linear ridge-regression probes on residual-stream hidden states, $\hat{w} = (A^\top A + \lambda I)^{-1}A^\top y$, which test whether a scalar attribute lies along a single direction in representation space; the cross-model transfer design, in which a probe trained on base-model hidden states and base-model labels is applied unchanged to instruction-tuned hidden states, with the tuned model's jailbroken responses as the held-out test set; and a Bradley-Terry paired-comparison model that converts the tuned model's pairwise generations into a latent ordinal score per entity. The transfer step is the operational definition of persistence: if a readout learned on the base model keeps predicting after alignment, the linear geometry of the attribute survived. The innocuous probe prompt 'This document describes [entity]' is equally load-bearing, because it extracts representations without ever mentioning the attribute, so high decodability shows the attribute is spontaneously present in the entity's representation.

What would settle it

Run the Section 4 transfer pipeline with base-model labels scrambled, or with random probe directions matched in norm: if shuffled-label or random probes reach transfer correlations comparable to the real ones, the persistence result is an artifact of marginal statistics rather than a preserved linear direction. A complementary check compares the best base-model probe weight vector with the best instruction-tuned probe weight vector at the same layer via cosine similarity: high similarity would confirm one shared geometry, while near-orthogonal weights would show the transfer rides on something other than a common direction.

Watch

Extended reading notes

Core claim

The paper's central claim is that instruction-tuning does not wholly eliminate, or even relocate, harmful information in representation space: it merely suppresses its direct expression. Concretely, the authors fit ridge-regression probes to last-token hidden states from 'This document describes [entity]' and show that the resulting readouts predict, often with Pearson correlation above 0.8, the numerical answers a jailbroken model gives for attributes the tuned model refuses to state, across the three instruction-tuned models studied (gemma-2-9b-it, gemma-2-2b-it, and Yi-6B-Chat). Because that prompt never mentions the attribute, high decodability means the attribute is spontaneously present in the entity's representation rather than induced by the question. Transfer experiments then show that probes trained on the base, non-refusing model predict the instruction-tuned model's jailbroken responses on many attribute pairs, which the authors read as evidence that these linear representations survive instruction-tuning largely unchanged. Finally, using a Bradley-Terry model fit to pairwise comparisons, they find Spearman correlations up to 0.824 between probe predictions and the model's latent rankings, concluding that the decoded representations align with, and may steer, suppressed generative behavior expressed more subtly elsewhere.

Load-bearing premise

The transfer experiment applies a probe trained on base-model hidden states directly to instruction-tuned hidden states, which assumes the two models' residual streams at corresponding layers share a coordinate frame in which a single linear readout stays meaningful; if instruction-tuning rotates or rescales that subspace, the reported transfer correlations could be an artifact rather than evidence that the refused knowledge persisted.

Editorial extensions

If this is right

  • Refusal becomes a behavioral gate over unchanged representations: a linear probe trained on innocuous hidden states can read the refused answer without any jailbreak prompt.
  • Alignment evaluations that inspect only generated text will miss this channel; hidden-state probing offers a complementary audit that catches what suppression leaves behind.
  • Because base-model probes transfer to tuned models, alignment data need not mention an attribute for its representation to persist, so safety tuning cannot be assumed to scrub any particular fact.
  • The correlation between probe values and pairwise-comparison rankings implies that a refused attribute can leak into decisions that never name it, such as recommending a course of study based on the model's unstated association between an occupation and IQ.
  • Probes trained on the jailbreak prompts themselves improve only modestly over innocuous-prompt probes, suggesting the information is present before jailbreaking and is not confabulated on demand.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A causal test the paper leaves implicit: editing or ablating the probe direction in the tuned model's activations should move the Bradley-Terry rankings if the representation actually drives the comparisons, turning a correlational result into an interventional one.
  • The base-to-tuned transfer suggests a cheap audit procedure: train probes once on an open base model and apply them to any fine-tuned variant of the same weights, scanning for refused attributes without jailbreaking the target.
  • The entity set itself may carry confounds: if probes are reading stereotype statistics of the name or category rather than the attribute, then re-running the comparison task with randomized entity labels would tell whether the probe-model alignment survives the loss of real-world correlates.
  • For safety engineering, the results point toward interventions in representation space, such as steering or erasing specific directions, rather than output filtering, since the expressive channel can be suppressed without touching the underlying information.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies whether information that instruction-tuned language models refuse to generate is nevertheless linearly decodable from their hidden states. Using linear probes trained on innocuous prompts such as "This document describes [entity]", the authors report that jailbroken responses about attributes like country IQ or occupation substance-abuse rates can be predicted with Pearson correlations often above 0.6 and sometimes above 0.8. They further report that probes trained on base (non-instruction-tuned) models sometimes transfer to instruction-tuned models, suggesting that instruction-tuning preserves linear representations of refused content. Finally, they show that probe predictions correlate with Bradley-Terry scores estimated from jailbroken pairwise comparisons, and interpret this as evidence that the decoded information is actively used in downstream behavior. The experiments use three open-weight models (gemma-2-9b-it, gemma-2-2b-it, Yi-6B-Chat), four entity types, and two jailbreak prompts.

Significance. If the central claim holds, the paper would be a useful bridge between two usually separate literatures: jailbreak attacks on aligned LMs and linear-probing analyses of internal representations. It would also strengthen the practical concern that open-weight models retain linearly accessible traces of refused information after instruction-tuning. The paper has several concrete strengths: the measurement pipeline is straightforward, code is released, the probing targets are specified in detail, and the pairwise-comparison experiment is an attempt to move beyond the probe-fitting loop toward behavioral evidence. The headline claims, however, are currently stronger than the evidence: the "actively used" and "indirectly influential" language in the abstract goes beyond what a correlational study can establish, and the statistical reporting (best-layer selection without confidence intervals or multiple-comparison correction) makes the quantitative headline numbers less solid than they appear. With appropriate controls and tempered claims, the paper would be a useful empirical contribution to the alignment and interpretability communities.

major comments (3)
  1. [Abstract and Section 5] The claim that refused information is 'actively used' and 'indirectly influential in downstream behavior' outruns the evidence. Section 5 correlates probe predictions (from probes trained on ICL-jailbroken responses, Section 3) with Bradley-Terry scores estimated from pairwise comparisons that are also elicited under the same ICL jailbreak from the same model. This correlation could reflect a shared entity-attribute association or a common prompt-induced response style rather than the model's decision process reading the probed linear direction. The paper provides no control probe (e.g., permuted labels or an unrelated attribute), no partial correlation controlling for a broad status/development dimension, and no intervention such as ablating the probed direction. The full results in Figures 10-12 also contain many near-zero and negative Spearman correlations, and the cross-task correlations in Figure 13 are modest (0.463 for gemma-2-9b-it between main and Bradley-Terry results). The Limitations section concedes that only pairwise comparisons were tested and that non-jailbroken downstream tasks are left to future work, yet the abstract asserts active use and downstream influence. I request either tempering these claims to correlational language or adding controls/interventions that support the causal reading.
  2. [Section 3.1, Figures 2 and 7] The headline correlations (e.g., >0.8 for country IQ) are reported after selecting the best layer across all layers, with no confidence intervals and no correction for multiple comparisons. Because the layer is chosen on the same held-out set used to evaluate the probe, the reported performance is an optimistic estimate; leave-one-out cross-validation tunes lambda only, not the layer. I ask for confidence intervals or nested cross-validation over layer selection, and a shuffled-label or random-direction baseline to calibrate the expected correlation under the null. This is load-bearing for the paper's core claim that refused content is linearly decodable from innocuous hidden states.
  3. [Sections 4.1 and 4.2] The base-to-instruct transfer experiment applies probes trained on base model hidden states directly to instruction-tuned hidden states, but the paper does not justify that the two models' residual streams share a common linear coordinate system or that a single linear readout remains valid after fine-tuning. Because base and instruct checkpoints share pretrained weights, transfer is plausible, but controls are needed: for example, probes trained on shuffled labels, random directions, or an attribute that should not transfer. Without such controls, the observed transfer correlations could reflect generic shared structure rather than persistence of the specific refused attribute. Please add these controls or explicitly discuss the coordinate-alignment assumption and its failure modes.
minor comments (6)
  1. [Section 5 opening] The sentence 'above shown initially refused information indicates can be linearly decodable' contains a grammatical error; 'indicates' should be removed or the sentence restructured.
  2. [Figure 4 caption] The caption says 'These two entities were the same...' but IQ and Percent Women are attributes, not entities; please use 'attributes' throughout the discussion.
  3. [Appendix A] The word 'langauge' should be 'language' in the synthetic-names construction paragraph.
  4. [Section 6] The phrase 'remaining largely in-tact' should be 'remaining largely intact'.
  5. [Section 4.1] The phrase 'The goal is the assess whether' should be 'The goal is to assess whether'.
  6. [Appendix B] The description of ICL prompt construction says answers were 'randomly generated ... within a reasonable range' but does not specify the ranges or random seed; please provide the exact ranges or a reference to the code for reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: probing is a held-out supervised measurement and the Section 5 correlation is a genuine behavioral anchor; only a minor non-load-bearing self-citation appears.

full rationale

The central results are measurements rather than derivations. In Section 3, probes are fit to jailbroken responses and evaluated on held-out entities, so the reported correlations are not forced by construction. Section 4's base-to-instruct transfer applies a probe fit to base-model responses to a different model's hidden states and compares against that model's jailbroken responses; this is a falsifiable generalization, not a renamed fit. Section 5 correlates probe predictions (from innocuous hidden states) with Bradley-Terry scores estimated from a separate pairwise-comparison generation; the two quantities are not equal by construction, so the correlation is a real (though correlational) behavioral anchor rather than a tautology. The abstract's 'actively used' and 'indirectly influential' phrasing exceeds what a correlation can support, and the Limitations section concedes that only pairwise comparisons were tested and that non-jailbroken downstream tasks remain future work; this is an over-claim about external validity, not circularity. The only self-citation is [50] (Shrivastava et al. 2024), cited alongside [60] and [51] to support the caveat that LMs are inconsistent under different prompts; it is not load-bearing for any central claim and does not smuggle in an ansatz or uniqueness assumption. No fitted parameter is renamed as a prediction, and no derivation step reduces to its own input.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The ledger is small because this is an empirical probing study, not a derivation. The main fitted objects are the probe weights themselves, plus the tuning choices (ridge lambda, layer selection, parsing rule) that shape reported correlations. No new theoretical entities are introduced.

free parameters (3)
  • Ridge regularization lambda = tuned per probe via leave-one-out cross-validation
    Controls probe generalization; not reported as a fixed value, and selection is per entity-attribute-layer.
  • Layer selection for best correlation = best layer per entity-attribute pair
    Reported Pearson/Spearman correlations are maxima over layers, which can inflate apparent decodability without multiple-comparison correction.
  • Response parsing rule = first number (or first number after 'AIM:')
    Parsed jailbroken responses become regression labels; the rule was only qualitatively verified.
assumptions (5)
  • domain assumption Linear representation hypothesis: high-level concepts are linearly encoded in residual streams
    Invoked in Section 2 and cited prior work; the study's choice of linear probes depends on it, though the authors state they are not testing linearity itself.
  • domain assumption Base and instruction-tuned residual streams are in a compatible coordinate frame
    Section 4.1 applies base-trained probe weights directly to instruction-tuned hidden states without alignment or calibration.
  • domain assumption Parsed first-number labels faithfully represent the model's intended jailbroken answer
    Section 3.1; parsing was qualitatively verified but not quantitatively.
  • domain assumption Dropping failed jailbreak samples does not bias the measured correlations
    Section 3.1 and Appendix B.1; refusal/ASR rates differ by model and attribute, so exclusions may be non-random.
  • standard math Closed-form ridge regression (Eq. 3) follows standard linear algebra
    Used to fit probes; uncontroversial.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Linearly Decoding Refused Knowledge in Aligned Language Models." pith.science (2026). https://pith.science/paper/2Y2RPHVW

@misc{pith2026250700239,
  author       = {Pith},
  title        = {Pith review of: Linearly Decoding Refused Knowledge in Aligned Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2Y2RPHVW}},
  note         = {Machine review of arXiv:2507.00239}
}
abstract

Most commonly used language models (LMs) are instruction-tuned and aligned using a combination of fine-tuning and reinforcement learning, causing them to refuse users requests deemed harmful by the model. However, jailbreak prompts can often bypass these refusal mechanisms and elicit harmful responses. In this work, we study the extent to which information accessed via jailbreak prompts is decodable using linear probes trained on LM hidden states. We show that a great deal of initially refused information is linearly decodable. For example, across models, the response of a jailbroken LM for the average IQ of a country can be predicted by a linear probe with Pearson correlations exceeding $0.8$. Surprisingly, we find that probes trained on base models (which do not refuse) sometimes transfer to their instruction-tuned versions and are capable of revealing information that jailbreaks decode generatively, suggesting that the internal representations of many refused properties persist from base LMs through instruction-tuning. Importantly, we show that this information is not merely "leftover" in instruction-tuned models, but is actively used by them: we find that probe-predicted values correlate with LM generated pairwise comparisons, indicating that the information decoded by our probes align with suppressed generative behavior that may be expressed more subtly in other downstream tasks. Overall, our results suggest that instruction-tuning does not wholly eliminate or even relocate harmful information in representation space-they merely suppress its direct expression, leaving it both linearly accessible and indirectly influential in downstream behavior.

Figures

Figures reproduced from arXiv: 2507.00239 by the authors.

Figure 1
Figure 1. (a) In Section 3, we obtain the hidden states of a model when processing an innocuous prompt. Then, we jailbreak a model to obtain responses to harmful questions. We then train a linear probe to predict the responses from the obtained hidden states. (b) In Section 4, we train a linear probe on the hidden states from a base LM (which do not need to be jailbroken) and test whether this probe can be applied to the orig… view at source ↗
Figure 2
Figure 2. Linear decodability of Occupations attributes using probes trained on an innocuous prompt [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Linear decodability of Occupations attributes using probes trained on base model to predict [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Correlation between predicted probe value and Bradley-Terry score on the Percent Women [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Hypothetical implication of persistent harmful representations influencing downstream [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Correlations between results from all sections for [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Main experiment results for all entity types, across both jailbreak prompts (AIM, ICL). [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Difference in probe performance between probes trained on hidden states from innocuous [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Transferability of linear probes trained on base model representations to instruction-tuned [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Full results for the Occupations entity type on the generative comparisons experiments. [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: Full results for the Countries entity type on the generative comparisons experiments. [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]
Figure 12
Figure 12. Figure 12: Full results for the Political Figures entity type on the generative comparisons experiments. [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]
Figure 13
Figure 13. Figure 13: Correlations between results from all sections for all models. Main results, specific results, [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 29 canonical work pages

  1. [1]

    Yi-6b-chat

    01.AI. Yi-6b-chat. https://huggingface.co/01-ai/Yi-6B-Chat , 2024. Accessed: 2025- 05-15

    work page 2024

  2. [2]

    Fine-grained analysis of sentence embeddings using auxiliary prediction tasks

    Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. InInternational Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=BJh6Ztuxl

    work page 2017

  3. [3]

    Understanding intermediate layers using linear classifier probes, 2017

    Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2017. URL https://openreview.net/forum?id=ryF7rTqgl

    2017

  4. [4]

    Bowman, Ethan Perez, Roger Baker Grosse, and David Duve- naud

    Cem Anil, Esin DURMUS, Nina Rimsky, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel J Ford, Francesco Mosconi, Rajashree Agrawal, Rylan Schaeffer, Naomi Bashkansky, Samuel Svenningsen, Mike Lambert, Ansh Radhakrishnan, Carson Denison, Evan J Hubinger, Yuntao Bai, Trenton Bricken, Timothy Maxwell, Nicholas Schiefer, Ja...

    work page 2024

  5. [5]

    Refusal in language models is mediated by a single direction

    Andy Arditi, Oscar Balcells Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. URL https://openreview.net/forum?id=pH3XAQME6c

    work page 2024

  6. [6]

    Language models can predict their own behavior

    Dhananjay Ashok and Jonathan May. Language models can predict their own behavior. arXiv preprint arXiv:2502.13329, 2025

    arXiv 2025

  7. [7]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  8. [8]

    Probing classifiers: Promises, shortcomings, and advances

    Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Com- putational Linguistics , 48(1):207–219, March 2022. doi: 10.1162/coli_a_00422. URL https://aclanthology.org/2022.cl-1.7/

    doi:10.1162/coli_a_00422 2022

Show all 68 references
  1. [9]

    Emergent misalignment: Narrow finetuning can produce broadly misaligned llms

    Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans. Emergent misalignment: Narrow finetuning can produce broadly misaligned llms. arXiv preprint arXiv:2502.17424, 2025

    2025

  2. [10]

    Wedded to prosperity? informal influence and regional favoritism

    Pietro Bomprezzi, Axel Dreher, Andreas Fuchs, Teresa Hailer, Andreas Kammerlander, Lennart Kaplan, Silvia Marchesi, Tania Masi, Charlotte Robert, and Kerstin Unfried. Wedded to prosperity? informal influence and regional favoritism. Discussion Paper 18878, Centre for Economic ...

    2025

  3. [11]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. ISSN 00063444, 14643510. URL http://www.jstor.org/stable/2334029

    1952

  4. [12]

    List of countries | Britannica

    Encyclopedia Britannica. List of countries | Britannica. https://www.britannica.com/ topic/list-of-countries-1993160 , 2025. [Accessed 10-05-2025]

    2025

  5. [13]

    From imitation to introspection: Probing self-consciousness in language models

    Sirui Chen, Shu Yu, Shengjie Zhao, and Chaochao Lu. From imitation to introspection: Probing self-consciousness in language models. arXiv preprint arXiv:2410.18819, 2024

    2024 arXiv

  6. [14]

    Probing linguistic information for logical inference in pre-trained language models

    Zeming Chen and Qiyue Gao. Probing linguistic information for logical inference in pre-trained language models. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):10509– 10517, Jun. 2022. doi: 10.1609/aaai.v36i10.21294. URL https://ojs.aaai.org/index. php/AA...

    2022 doi

  7. [15]

    Leverag- ing the context through multi-round interactions for jailbreaking attacks

    Yixin Cheng, Markos Georgopoulos, V olkan Cevher, and Grigorios G Chrysos. Leverag- ing the context through multi-round interactions for jailbreaking attacks. arXiv preprint arXiv:2402.09177, 2024

    2024 arXiv

  8. [16]

    Breaking down the defenses: A comparative survey of attacks on large language models

    Arijit Ghosh Chowdhury, Md Mofijul Islam, Vaibhav Kumar, Faysal Hossain Shezan, Vinija Jain, and Aman Chadha. Breaking down the defenses: A comparative survey of attacks on large language models. arXiv preprint arXiv:2403.04786, 2024

    2024

  9. [17]

    Com- prehensive assessment of jailbreak attacks against llms

    Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, and Yang Zhang. Com- prehensive assessment of jailbreak attacks against llms. CoRR, abs/2402.05668, 2024. URL https://doi.org/10.48550/arXiv.2402.05668

    2024 doi arXiv

  10. [18]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024

    2024

  11. [19]

    Pawan Kumar, and Adel Bibi

    Francisco Eiras, Aleksandar Petrov, Philip Torr, M. Pawan Kumar, and Adel Bibi. Do as i do (safely): Mitigating task-specific fine-tuning risks in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview. net/f...

    2025

  12. [20]

    Dissecting recall of factual associations in auto-regressive language models

    Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proce...

    2023 doi

  13. [21]

    Estimating knowledge in large language models without generating a single token

    Daniela Gottesman and Mor Geva. Estimating knowledge in large language models without generating a single token. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 3994–4019, ...

    2024 doi

  14. [22]

    Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligen...

    2023

  15. [23]

    Language models represent space and time

    Wes Gurnee and Max Tegmark. Language models represent space and time. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview. net/forum?id=jE8xbmvFin

    2024

  16. [24]

    The Elements of Statistical Learning: Data Mining, Inference, and Prediction, volume 2

    Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, volume 2. Springer, New York, 2nd edition, 2009. ISBN 978-0-387-84857-0

    2009

  17. [25]

    Do LLMs “know” internally when they follow instructions? In The Thirteenth International Conference on Learning Representations, 2025

    Juyeon Heo, Christina Heinze-Deml, Oussama Elachqar, Kwan Ho Ryan Chan, Shirley You Ren, Andrew Miller, Udhyakumar Nallasamy, and Jaya Narain. Do LLMs “know” internally when they follow instructions? In The Thirteenth International Conference on Learning Representations, 2025....

    2025

  18. [26]

    Linearity of relation decoding in transformer language models

    Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://op...

    2024

  19. [27]

    John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational 12 Linguisti...

    2019 doi

  20. [28]

    Refusal tokens: A simple way to calibrate refusals in large language models

    Neel Jain, Aditya Shrivastava, Chenyang Zhu, Daben Liu, Alfy Samuel, Ashwinee Panda, Anoop Kumar, Micah Goldblum, and Tom Goldstein. Refusal tokens: A simple way to calibrate refusals in large language models. arXiv preprint arXiv:2412.06748, 2024

    2024 arXiv

  21. [29]

    Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip Torr, Amartya Sanyal, and Puneet K. Dokania. What makes safety fine-tuning methods safe? a mechanistic study. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openr...

    2024

  22. [30]

    Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large language and vision-language models

    Haibo Jin, Leyang Hu, Xinuo Li, Peiyan Zhang, Chonghan Chen, Jun Zhuang, and Haohan Wang. Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large language and vision-language models. arXiv preprint arXiv:2407.01599, 2024

    2024

  23. [31]

    Alignment of language agents

    Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. Alignment of language agents. arXiv preprint arXiv:2103.14659, 2021

    2021 arXiv

  24. [32]

    Linear representations of political perspective emerge in large language models

    Junsol Kim, James Evans, and Aaron Schein. Linear representations of political perspective emerge in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=rwqShzb9li

    2025

  25. [33]

    Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b

    Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. arXiv preprint arXiv:2310.20624, 2023

    2023 arXiv

  26. [34]

    Re- vealing the intrinsic ethical vulnerability of aligned large language models

    Jiawei Lian, Jianhong Pan, Lefan Wang, Yi Wang, Shaohui Mei, and Lap-Pui Chau. Re- vealing the intrinsic ethical vulnerability of aligned large language models. arXiv preprint arXiv:2504.05050, 2025

    2025 arXiv

  27. [35]

    The unlocking spell on base LLMs: Rethinking alignment via in-context learning

    Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base LLMs: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Representatio...

    2024

  28. [36]

    Towards understanding jailbreak attacks in llms: A representation space analysis

    Yuping Lin, Pengfei He, Han Xu, Yue Xing, Makoto Yamada, Hui Liu, and Jiliang Tang. Towards understanding jailbreak attacks in llms: A representation space analysis. arXiv preprint arXiv:2406.10794, 2024

    2024 arXiv

  29. [37]

    Formalizing and benchmarking prompt injection attacks and defenses

    Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24), pp. 1831–1847, 2024

    2024

  30. [38]

    Keep- ing llms aligned after fine-tuning: The crucial role of prompt templates

    Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Keep- ing llms aligned after fine-tuning: The crucial role of prompt templates. arXiv preprint arXiv:2402.18540, 2024

    2024 arXiv

  31. [39]

    The geometry of truth: Emergent linear structure in large lan- guage model representations of true/false datasets

    Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large lan- guage model representations of true/false datasets. In First Conference on Language Modeling,

  32. [40]

    Occupation Data - O*NET 29.2 Data Dictionary at O*NET Re- source Center

    O*NET Resource Center. Occupation Data - O*NET 29.2 Data Dictionary at O*NET Re- source Center. https://www.onetcenter.org/dictionary/29.2/excel/occupation_ data.html, 2025. [Accessed 10-05-2025]

    2025

  33. [41]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  34. [42]

    The linear representation hypothesis and the geometry of large language models

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In ICML, 2024. URL https://openreview.net/ forum?id=UGpGkLzwpP

    2024

  35. [43]

    Ignore previous prompt: Attack techniques for language models

    Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527, 2022

    2022 arXiv

  36. [44]

    Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations , 2024

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations , 2024. URL https://open...

    2024

  37. [45]

    Safety alignment should be made more than just a few tokens deep

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. In The Thirteenth International Conference on Learning Representations, 2025. URL https://open...

    2025

  38. [46]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openr...

    2023

  39. [47]

    Multi- task prompted training enables zero-shot task generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyo...

    2022

  40. [48]

    Scalable and transferable black-box jailbreaks for language models via persona modulation

    Rusheb Shah, Quentin Feuillade-Montixi, Soroush Pour, Arush Tagade, Stephen Casper, and Javier Rando. Scalable and transferable black-box jailbreaks for language models via persona modulation. CoRR, abs/2311.03348, 2023. URL https://doi.org/10.48550/arXiv.2311. 03348

    2023 doi arXiv

  41. [49]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp. 1671–1...

    2024

  42. [50]

    Measuring free-form decision- making inconsistency of language models in military crisis simulations

    Aryan Shrivastava, Jessica Hullman, and Max Lamparth. Measuring free-form decision- making inconsistency of language models in military crisis simulations. arXiv preprint arXiv:2410.13204, 2024

    2024 arXiv

  43. [51]

    Large language models are incon- sistent and biased evaluators

    Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. Large language models are incon- sistent and biased evaluators. arXiv preprint arXiv:2405.01724, 2024

    2024 arXiv

  44. [52]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  45. [53]

    What do you learn from context? probing for sentence structure in contextualized word representations

    Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint arXiv:1905...

    1905 arXiv

  46. [54]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural I...

    2017

  47. [55]

    White-box multimodal jailbreaks against large vision-language models

    Ruofan Wang, Xingjun Ma, Hanxu Zhou, Chuanjun Ji, Guangnan Ye, and Yu-Gang Jiang. White-box multimodal jailbreaks against large vision-language models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 6920–6928, 2024

    2024

  48. [56]

    Jailbroken: How does LLM safety training fail? In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=jA235JGM09

    2023

  49. [57]

    Assessing the brittleness of safety alignment via pruning and low-rank modifications

    Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint arXiv:2402.05162, 2024

    2024 arXiv

  50. [58]

    Jailbreak and guard aligned language models with only few in-context demonstrations

    Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387, 2023

    2023 arXiv

  51. [59]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=NG7sS51zVF

    2024

  52. [60]

    Assessing hidden risks of llms: an empirical study on robustness, consistency, and credibility

    Wentao Ye, Mingfeng Ou, Tianyi Li, Xuetao Ma, Yifan Yanggong, Sai Wu, Jie Fu, Gang Chen, Haobo Wang, Junbo Zhao, et al. Assessing hidden risks of llms: an empirical study on robustness, consistency, and credibility. arXiv preprint arXiv:2305.10235, 2023

    2023 arXiv

  53. [61]

    On the vulnerability of safety alignment in open-access LLMs

    Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. On the vulnerability of safety alignment in open-access LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Lingu...

    2024 doi

  54. [62]

    Jailbreak attacks and defenses against large language models: A survey

    Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. Jailbreak attacks and defenses against large language models: A survey. arXiv preprint arXiv:2407.04295, 2024

    2024 arXiv

  55. [63]

    Yi: Open foundation models by 01

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024

    2024 arXiv

  56. [64]

    Don’t listen to me: understanding and exploring jailbreak prompts of large language models

    Zhiyuan Yu, Xiaogeng Liu, Shunning Liang, Zach Cameron, Chaowei Xiao, and Ning Zhang. Don’t listen to me: understanding and exploring jailbreak prompts of large language models. In 33rd USENIX Security Symposium (USENIX Security 24), pp. 4675–4692, 2024

    2024

  57. [65]

    Removing RLHF protections in GPT-4 via fine-tuning

    Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing RLHF protections in GPT-4 via fine-tuning. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Associati...

    2024 doi

  58. [66]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006–55021, 2023

    2023

  59. [67]

    diverse set of fictional names

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. 15 A Further Details on Data Here, we provide further details on the data u...

    2023 arXiv

  60. [2024]

    URL https://openreview.net/forum?id=aajyHYjjsk

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.