Pith. sign in

REVIEW 3 major objections 7 minor 3 cited by

The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper establishes that safety-aligned refusal behavior in large language models is jointly controlled by a dominant direction and multiple smaller, interpretable feature directions in a safety residual space, and that removing…

desk verdict Multi-dimensional safety directions are real, and the trigger-removal attack is worth taking seriously, but PLRP faithfulness and missing controls need fixing before the mechanism claims fully land. read the letter →

arxiv 2502.09674 v4 pith:YKH7BFTM submitted 2025-02-13 cs.CL cs.AI

Wenbo Pan , Zhichao Liu , Qiguang Chen , Xiangyang Zhou , Haining Yu , Xiaohua Jia This is my paper · ORCID
classification cs.CLcs.AI
keywords safetyalignmentmechanisticinterpretabilityjailbreakattackslayer-wiserelevancepropagationrefusaldirectionresidualspacetriggertokenslinearrepresentationhypothesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a large language model's learned refusal behavior lives not in one 'refusal direction' but in a low-rank subspace of orthogonal directions, learned as the activation shift of safety fine-tuning. A dominant direction predicts whether the model refuses harmful queries, while smaller directions encode distinct side features such as hypothetical-narrative framing, role-playing, and the 'Sure, I'm happy to help' affirmative pattern common in PAIR attacks. The paper introduces Partial Layer-wise Relevance Propagation to attribute each direction to the input tokens that activate it, and then shows that these non-dominant directions causally shape the dominant direction and refusal behavior. It concludes by demonstrating that rephrasing harmful prompts to avoid the attributed trigger tokens weakens learned safety alignment even after fine-tuning on examples of the attack, framing safety alignment as vulnerable through spurious surface correlations.

What carries the argument

Safety Residual Space: the affine map $S(x)=Wx+b$ fit to the representation shift between the pre-finetuning and post-finetuning model at each layer, so that the learned alignment behavior lives in the span of $W-I$; singular value decomposition yields orthogonal components $L_N$-$C_K$, with $L_N$-$C_1$ the dominant refusal direction. Partial Layer-wise Relevance Propagation (PLRP) projects activations onto the span of selected directions, decomposes the projected norm into relevance scores, and back-propagates those scores through the transformer with the epsilon rule to assign token-level relevance. The trigger-removal attack uses these token attributions to build a blacklist, and an LLM iteratively rephrases harmful prompts to avoid the blacklisted tokens.

What would settle it

Take a fixed PAIR jailbreak prompt, compute PLRP relevance for the non-dominant direction L14-C6 on the safety fine-tuned model, and remove only the single highest-relevance trigger token while keeping all other tokens fixed. If the model's refusal rate on PAIR prompts does not fall in the direction predicted by the component's projection shift, the PLRP attribution is unfaithful and the causal link between trigger tokens, non-dominant directions, and refusal is broken.

Watch

Extended reading notes

Core claim

The central discovery is that the activation shift induced by safety fine-tuning on Llama 3.1 8B is approximately low-rank and linear, and its first singular component (the dominant direction) predicts refusal behavior, while the remaining orthogonal components are individually interpretable features such as hypothetical narrative, creative-writing framing, and the 'Sure, I'm happy to help' affirmative pattern. The paper provides causal evidence that suppressing the component associated with PAIR-like affirmative patterns selectively removes the model's ability to refuse PAIR prompts, and that suppressing most non-dominant directions while leaving dominant components intact lowers refusal rates on jailbreak prompts. It also introduces a trigger-removal attack: PLRP relevance scores identify the tokens that activate the safety directions, an LLM rephrases harmful prompts to avoid those tokens, and the rewritten prompts retain roughly 40% success after safety fine-tuning on samples of the attack while baseline attacks drop to near zero.

Load-bearing premise

The load-bearing premise is that Partial Layer-wise Relevance Propagation faithfully attributes the projected activation components back to specific input tokens; if that back-propagation is unfaithful, the semantic labels on the directions and the trigger-removal attack lose their grounding.

Editorial extensions

If this is right

  • If the multi-direction picture is right, refusal cannot be fully understood or manipulated through a single probe vector; safety mechanisms live on a subspace and depend on the balance among its components.
  • Suppressing one non-dominant direction (L14-C6) selectively disables refusal for PAIR prompts while preserving refusal for other attack families, so interventions on learned safety must be evaluated per jailbreak family, not just by overall refusal rate.
  • Trigger tokens are causally relevant: rephrasing a harmful prompt to avoid PLRP-identified triggers keeps an attack near 40% success after 80-shot safety fine-tuning, while PAIR and other baselines fall below 5%, so a safety policy that only adds examples of known attacks will not fix the vulnerability.
  • The dominant component's projection on harmful prompts is shifted by both non-dominant suppression and trigger removal, meaning secondary features act through the dominant direction rather than around it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if PLRP attribution is faithful, the trigger-removal procedure is not specific to these jailbreak families; any prompt pattern that activates a learned non-dominant feature direction should act as a bypass, which would predict that safety training over-relies on surface n-grams.
  • Editorial inference: the subspace view suggests a detector-side defense: monitor projections on non-dominant directions during inference, since an anomalous surge in the hypothetical-narrative or affirmative-response direction could flag a jailbreak before the refusal decision is made.
  • Editorial inference: the paper's empirical scope (one 8B model, 2600 training samples) likely understates the effective rank of the residual space; on larger models or more diverse alignment data one would expect more feature directions and correspondingly more attack families that follow the same trigger-removal mechanism.
  • Editorial inference: a direct testable extension is to retrain with counterfactual augmentation that varies trigger phrasing while keeping harmful intent, forcing the model to encode refusal in the dominant direction alone; if safety then becomes robust to trigger removal, the spurious-correlation story is confirmed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper studies the activation-space changes induced by safety fine-tuning in Llama 3.1 8B Instruct. The authors define a 'safety residual space' as the affine least-squares fit of the representation shift between the base model and the SSFT/DPO-tuned model, decompose it with SVD, and investigate the resulting orthogonal directions. Their claims are: (i) a dominant singular direction predicts refusal behavior with accuracy comparable to probe vectors; (ii) non-dominant directions capture distinct jailbreak-style features such as hypothetical narrative, role-playing, and the 'Sure, I\'m happy to help' pattern; (iii) non-dominant directions causally modulate the dominant direction and refusal behavior, as supported by direction-suppression interventions; and (iv) a 'trigger removal' attack that uses Partial Layer-wise Relevance Propagation (PLRP) to blacklist tokens from harmful prompts remains partially effective after safety fine-tuning, which the authors attribute to the multi-dimensional structure of learned safety. The paper validates the linearity of the residual fit (Table 4), evaluates predictions and interventions on a held-out test set, documents the dataset and training protocol in detail, and releases code.

Significance. If established, this is a meaningful step beyond the single-direction refusal picture of Arditi et al. (2024): refusal would be jointly controlled by a low-dimensional subspace of directions, some tracking jailbreak format rather than harmfulness, with an attribution-guided token-removal attack that survives fine-tuning exposure. The strongest concrete results are the residual-space linearity check (Table 4), the causal selectivity of the layer-14 component-6 ablation for PAIR prompts (Figure 4), and the documented resilience of the trigger-removal attack (Table 3). The paper is also commendable for releasing code, specifying hyperparameters and dataset composition, and reporting perplexity checks on intervention side effects. The significance is conditional, however: the semantic labels and the attack mechanism rest on unvalidated PLRP attributions to a subspace-projection target, and the trigger-removal claim lacks a resampling-only control, so the multi-dimensional causal story requires the additional experiments proposed below.

major comments (3)
  1. [Section 5 (Our Approach); Tables 1-2; Appendix B (Algorithm 1)] The PLRP target is changed from the network output to the squared projection norm PV(X^l) = sum over v in V of ||v^T X^l||^2, and faithfulness is imported from Achtibat et al. (2024), which validates attention-aware LRP for output logits rather than for a squared projection onto a residual-direction subspace. The LRP conservation property must be re-established for the composed function X^l -> PV(X^l), since the projection discards the off-span component and squares the retained one; the paper's proportionality sign and its partial re-normalization note do not substitute for a conservation check. No perturbation test (e.g., ablating the top-ranked tokens and measuring the change in PV or in refusal probability) or comparison against a gradient-based attribution baseline is reported. Because the semantic labels in Tables 1-2, the trigger blacklist in Algorithm 1, and the mechanism of the trigger-removal attack all rest on these attributions, this validation gap is load-bearing and must be closed with a quantitative faithfulness evaluation.
  2. [Section 6.1 (Table 3); Appendix B (Algorithm 1)] The trigger-removal attack is an iterative pipeline that generates ten resampled variants per round, scores them with the Strong Reject metric, applies PLRP to a selected subset, and resamples with a growing token blacklist for up to three rounds, allowing at most thirty attempts per sample. The causal claim that removing the identified trigger tokens mitigates the safety directions requires a control that isolates the PLRP blacklist from the resampling procedure: the identical iterative scoring-and-resampling loop should be run without the blacklist, or with an equally sized blacklist of random tokens drawn from the original prompts. Without such a control, the high pass rates in Table 3 and their resilience to fine-tuning could be explained entirely by the distributional shift of paraphrasing, independent of the identified directions. Since the abstract presents the trigger-removal result as direct evidence for the multi-dimensional vulnerability thesis, this control is essential rather than cosmetic.
  3. [Figure 4; Section 6 (Table 8); Appendix D] The specificity claims in Figure 4 (L14-C6 selectively ablates PAIR refusal) and in Table 8 (non-dominance suppression preserves 80% refusal on direct harmful prompts while jailbreak refusal drops to 14.4%) would be considerably stronger with a magnitude-matched random-direction ablation at the same layers and with a sensitivity sweep over alpha in Equation 2; the L14-C4/L14-C5 conditions control for other learned components but not for a generic perturbation of identical norm. In addition, the suppression experiment excludes directions whose harmfulness correlation exceeds 0.7 (Appendix D), and the paper should state explicitly which data split is used to compute these correlations; if the correlations are computed on the test set, the selectivity of Table 8 becomes partly a selection effect, and the result should be re-derived with the correlations computed on the training split only.
minor comments (7)
  1. [Table 3 vs. Appendix C.2 (Tables 5-7)] Please reconcile the definition of the 0-shot condition and the relationship between the Attack Pass Rate and the Strong Reject Scores. The FLIP row of Table 3 reports a 0-shot pass rate of 0.78, which appears to contradict the observation in Appendix C.2 that 'Flip and CodeChaeleom remains ineffective' and the Flip Strong Reject score of 0.000 in Table 5. The pass-rate threshold and the exact model condition (base model vs. N=0) for the 0-shot column should be stated explicitly.
  2. [Figure 3 (Section 4)] The Best-of-N baseline selects the singular vector with the highest refusal prediction accuracy on the test set; to avoid any ambiguity, the selection for this baseline should be performed on a validation split, with the test set used only for the final comparison.
  3. [Definition 3.1 and Section 5 (PLRP formula)] The notation in Definition 3.1 is confusing, with S appearing as both the transformation and the objective value being minimized, and the proportionality sign in PV(X^l) = sum over v in V of ||v^T X^l||^2 proportional to R^l should be replaced by the actual normalization, since the conservation argument for LRP requires exact equality after renormalization.
  4. [Corollary 3.3] Corollary 3.3 states that the safety residual space is the span of feature directions developed during safety training; this is a modeling assumption about which features the residual captures rather than a consequence of Definition 3.1, and it should be presented as an explicit assumption.
  5. [Appendix B (Algorithm 1)] Algorithm 1 and its surrounding prose disagree on which variants PLRP is applied to: the pseudocode uses 'Pvariants with top k score', while the prose says the variants with lower scores, being more likely to be rejected, are analyzed. Since the main text states that PLRP identifies tokens that explain the refusal output, the pseudocode should be aligned with that description and the scoring direction should be unambiguous.
  6. [Appendix C.2 (Tables 5-7)] The OR-Bench rows in Tables 5-7 need an explicit metric definition: for harmless samples, a high value of the reported score is ambiguous without knowing whether the quantity is the Strong Reject score of the model's response, an over-refusal rate, or a helpfulness score.
  7. [Appendix C.2; Section 7; Table 1] Minor typos and infelicities: 'we we label them as unsafe' in Appendix C.2, 'Neverthless' in Section 7, and 'a experimental ChatGPT' in Table 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical, evaluated on held-out data, and no load-bearing claim reduces by construction to its inputs.

full rationale

The paper makes empirical claims rather than deriving predictions from definitions. The safety residual space S is fitted on a training split and evaluated with held-out MSE (Table 4) and refusal prediction accuracy on a disjoint test set (Figure 3). The multi-directional conclusion rests on SVD components of W−I and on intervention experiments (suppressing L14-C6, non-dominant suppression) whose behavioral outcomes, refusal rates and attack pass rates, are independent of how the directions are defined. The trigger-removal attack uses PLRP to identify tokens, but its reported pass rates and post-fine-tuning resilience are measured on held-out STRONG REJECT samples, so the attack results are not forced by the PLRP definition. The cited faithfulness authority, Achtibat et al. (2024), is external and not by the present authors; whether PLRP remains faithful for projected residual directions is a validation concern, not a circular reduction. Self-citations (Chen et al. 2025; Zhou et al. 2024a,b) appear only in related work and are not load-bearing. No equation in the paper equates a prediction to its fitted input by construction. The Best-of-N baselines select components using test accuracy, which makes them upper-bound controls rather than genuine predictions, but this is a statistical caveat and does not affect the main empirical derivation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The paper's central contribution is an analytical decomposition of activation shifts, so the ledger contains few invented entities. The main hand-chosen quantities are the intervention thresholds and analysis layers, which affect the magnitude of reported effects but not the existence of the dominant direction. The domain assumptions are the linearity of residual shifts and the faithfulness of PLRP, both standard for this line of work but not fully proven.

free parameters (5)
  • Harmfulness correlation exclusion threshold = 0.7
    Hand-chosen cutoff in Section 6 for excluding non-dominant components when suppressing them. The resulting refusal drop (96.7% to 14.4%) depends on this threshold.
  • Energy threshold tau for effective rank = 0.4 to 0.9
    Defines effective rank in Figure 2; the observation that effective rank is 1 in early SSFT layers depends on the chosen tau values.
  • Analysis layer selection = Layer 14 and Layer 25
    Token attribution focuses on Layer 14 and dominant components on Layer 25. The paper says other layers show similar patterns but does not quantify this.
  • Activation suppression strength alpha = not reported
    Interventions using x := x minus alpha*v in Equation 2 depend on an unreported alpha magnitude, so the effect sizes in Figures 4 and 6 are not fully specified.
  • Trigger removal iteration budget = n=3 iterations, 10 variants each
    Appendix B fixes these values; the attack success rates in Table 3 depend on the budget and on access to Llama 3.1 405B as resampler.
assumptions (5)
  • domain assumption Linear Representation Hypothesis: features are linear directions in activation space
    Adopted in Section 2 from Park et al. (2023); the entire residual-space framework treats safety features as linear directions.
  • domain assumption The fine-tuning activation shift is approximately affine, S(x)=Wx+b
    Definition 3.1 minimizes MSE against the true shift T; Table 4 reports low MSE, but the linearity is an assumption about the training dynamics.
  • domain assumption The safety residual space spans only features developed during safety training
    Corollary 3.3 states this, assuming that pre-existing features (e.g., toxicity recognition) and features irrelevant to the objective are absent from the residual shift.
  • standard math PLRP relevance propagation is faithful for projected SVD directions
    Section 5 adopts the LRP implementation of Achtibat et al. (2024) and the epsilon rule from Bach et al. (2015); faithfulness for the new projection setting is not validated.
  • domain assumption The first generated token position represents the model's safety decision
    Section 3 states activations are taken at the first generated token per layer; this is a methodological choice that could miss refusal dynamics at other positions.
invented entities (1)
  • Safety Residual Space independent evidence
    purpose: A linear span of representation shifts during safety fine-tuning, decomposed by SVD into dominant and non-dominant feature directions.
    The construct has falsifiable handles: its components predict refusal on a held-out test set, and interventions on components change refusal behavior in specific ways (e.g., L14-C6 removal specifically hurts PAIR refusal). Other researchers can test these predictions on other models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions." pith.science (2026). https://pith.science/paper/YKH7BFTM

@misc{pith2026250209674,
  author       = {Pith},
  title        = {Pith review of: The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YKH7BFTM}},
  note         = {Machine review of arXiv:2502.09674}
}
read the original abstract

Large Language Models' safety-aligned behaviors, such as refusing harmful queries, can be represented by linear directions in activation space. Previous research modeled safety behavior with a single direction, limiting mechanistic understanding to an isolated safety feature. In this work, we discover that safety-aligned behavior is jointly controlled by multi-dimensional directions. Namely, we study the vector space of representation shifts during safety fine-tuning on Llama 3 8B for refusing jailbreaks. By studying orthogonal directions in the space, we first find that a dominant direction governs the model's refusal behavior, while multiple smaller directions represent distinct and interpretable features like hypothetical narrative and role-playing. We then measure how different directions promote or suppress the dominant direction, showing the important role of secondary directions in shaping the model's refusal representation. Finally, we demonstrate that removing certain trigger tokens in harmful queries can mitigate these directions to bypass the learned safety capability, providing new insights on understanding safety alignment vulnerability from a multi-dimensional perspective. Code and artifacts are available at https://github.com/BMPixel/safety-residual-space.

Figures

Figures reproduced from arXiv: 2502.09674 by the authors.

Figure 1
Figure 1. Illustration of the Safety Residual Space. The safety residual space is the linear span of representation shifts during safety fine-tuning. In our experiments, the dominant direction predicts safety behavior, while non-dominant directions capture different indirect safety features. A concrete example is the refusal direction identified by Arditi et al. (2024), where they construct contrast pairs by comparing inputs … view at source ↗
Figure 2
Figure 2. Effective rank of the residual space by layer. optimize the model to generate refusal for harmful queries with instruction fine-tuning. For DPO, we additionally cre￾ate a preference dataset with prefered helpful responses for harmless queries and refusals with disclaimers for harmful queries. We use Llama 3.1 405B Instruct (Dubey et al., 2024) to generate reference responses for the preference dataset. The effective… view at source ↗
Figure 4
Figure 4. Intervention results after removing the direction of the 6th component of layer 14 (L14-C6) from the hidden states during generation. L14-C6 is identified as representing the specific ability to recognize the PAIR Attack. Additionally, we remove the dominant direction (L25-C1), which completely eliminates the fine-tuned model’s ability to refuse. In comparison, L14-C4 and L14-C3 also affect model behavior but do not… view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Top 3: Adjacent layer relevance scores among top direc￾tions. Rel Comp 1: relevance scores to first component in next layer. Bottom: Log-likelihood of predicting aligned behavior with different directions. in subsection C.4, ensuring that removing refusal does not degr…
Figure 6
Figure 6. Figure 6: Mean projection of activations on top components under different settings in SSFT. The projection on component 1 is strongly correlated with the model’s safety behavior. Harmful: activations are from harmful samples. Benign: activations are from benign samples. Non-Dom…
Figure 7
Figure 7. Figure 7: Sample division illustration for N-SHOT Security Training. Left: The case when the number of alignment blindspot samples N is 80 in the dynamic division of the training set. Right: Fixed test set division, where or-bench consists of harmless samples, and other parts ar…
Figure 8
Figure 8. Figure 8: The visualization of the changes in Strong Reject Scores for all jailbreak methods as the number of exposure examples in￾creases during SSFT training [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: The visualization of the changes in Strong Reject Scores for all jailbreak methods as the number of exposure examples in￾creases during DPO training [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: The visualization of the changes in Strong Reject Scores on Ministral-8B-Instruct during DPO training [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Safety residual space analysis of safety finetuning the Llama 3.2 3B Instruct using DPO. The dataset and experimental setup for generating plot (a), (b) and (c) are the same as in [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 13
Figure 13. Figure 13: Harmfulness correlation for each component. As discussed in section 6, the non-dominant suppression technique involves excluding components with high harm￾fulness correlations (above 0.7) to preserve the model’s ability to refuse plainly harmful prompts while investi￾…
Figure 14
Figure 14. Figure 14: Projection value distributions for the non-dominance suppression experiment (Section 6). 20 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GloSS over Toxicity: Understanding and Mitigating Toxicity in LLMs via Global Toxic Subspace

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Detoxifying LLMs by deleting a global, cross-layer 'toxic subspace' from feed-forward weights reduces toxic outputs more than layer-local subspace methods.

  2. The Geometry of Harmfulness in LLMs through Subconcept Probing

    cs.AI 2025-07 conditional novelty 5.0 of 10

    Fifty-five harmfulness subconcept directions in Llama-3.1-8B-Instruct form a nearly rank-1 subspace, and steering along the dominant direction cuts jailbreak success but costs accuracy and fails on Qwen.

  3. Re-Emergent Misalignment: How Narrow Fine-Tuning Erodes Safety Alignment in LLMs

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Narrow fine-tuning on insecure code appears to erode prior safety alignment in Qwen2.5-Coder, with the misaligned model's internal activations moving back toward the base model.

Reference graph

Works this paper leans on

53 extracted references · 12 canonical work pages · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Achtibat, R., Hatefi, S. M. V., Dreyer, M., Jain, A., Wiegand, T., Lapuschkin, S., and Samek, W. Attnlrp: attention-aware layer-wise relevance propagation for transformers. arXiv preprint arXiv:2402.05602, 2024

    arXiv 2024

  3. [3]

    Refusal in language models is mediated by a single direction

    Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., and Nanda, N. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717, 2024

    arXiv 2024

  4. [4]

    On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation

    Bach, S., Binder, A., Montavon, G., Klauschen, F., M \"u ller, K.-R., and Samek, W. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10 0 (7): 0 e0130140, 2015

    2015

  5. [5]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  6. [6]

    Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and Generation

    Balestriero, R., Cosentino, R., and Shekkizhar, S. Characterizing large language model geometry solves toxicity detection and generation. ArXiv, abs/2312.01648, 2023. URL https://api.semanticscholar.org/CorpusID:265609911

    work page Pith review arXiv 2023

  7. [7]

    Understanding jailbreak success: A study of latent space dynamics in large language models

    Ball, S., Kreuter, F., and Rimsky, N. Understanding jailbreak success: A study of latent space dynamics in large language models. ArXiv, abs/2406.09289, 2024. URL https://api.semanticscholar.org/CorpusID:270440981

    arXiv 2024

  8. [8]

    E., Hume, T., Carter, S., Henighan, T., and Olah, C

    Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield-Dodds, Z., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. Towards monosemanticity: Decomposing languag...

    2023

Show all 53 references
  1. [9]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  2. [10]

    A., Jagielski, M., Gao, I., Koh, P

    Carlini, N., Nasr, M., Choquette-Choo, C. A., Jagielski, M., Gao, I., Koh, P. W. W., Ippolito, D., Tramer, F., and Schmidt, L. Are aligned neural networks adversarially aligned? Advances in Neural Information Processing Systems, 36, 2024

    2024

  3. [11]

    J., and Wong, E

    Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., and Wong, E. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023

    2023 arXiv

  4. [12]

    Towards reasoning era: A survey of long chain-of-thought for reasoning large language models

    Chen, Q., Qin, L., Liu, J., Peng, D., Guan, J., Wang, P., Hu, M., Zhou, Y., Gao, T., and Che, W. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567, 2025

    2025 arXiv

  5. [13]

    Or-bench: An over-refusal benchmark for large language models

    Cui, J., Chiang, W.-L., Stoica, I., and Hsieh, C.-J. Or-bench: An over-refusal benchmark for large language models. arXiv preprint arXiv:2405.20947, 2024

    2024 arXiv

  6. [14]

    A wolf in sheep's clothing: Generalized nested jailbreak prompts can fool large language models easily

    Ding, P., Kuang, J., Ma, D., Cao, X., Xian, Y., Chen, J., and Huang, S. A wolf in sheep's clothing: Generalized nested jailbreak prompts can fool large language models easily. arXiv preprint arXiv:2311.08268, 2023

    2023 arXiv

  7. [15]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  8. [16]

    J., Liao, I., Gurnee, W., and Tegmark, M

    Engels, J., Michaud, E. J., Liao, I., Gurnee, W., and Tegmark, M. Not all language model features are linear. arXiv preprint arXiv:2405.14860, 2024

    2024 arXiv

  9. [17]

    Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462, 2020

    2009 arXiv

  10. [18]

    Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2 0 (11): 0 665--673, 2020

    2020

  11. [19]

    Llama guard: Llm-based input-output safeguard for human-ai conversations

    Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023

    2023 arXiv

  12. [20]

    S., Oksuz, K., Joy, T., Torr, P

    Jain, S., Lubana, E. S., Oksuz, K., Joy, T., Torr, P. H., Sanyal, A., and Dokania, P. K. What makes and breaks safety fine-tuning? a mechanistic study. arXiv preprint arXiv:2407.10264, 2024

    2024 arXiv

  13. [21]

    Artprompt: Ascii art-based jailbreak attacks against aligned llms

    Jiang, F., Xu, Z., Niu, L., Xiang, Z., Ramasubramanian, B., Li, B., and Poovendran, R. Artprompt: Ascii art-based jailbreak attacks against aligned llms. In Annual Meeting of the Association for Computational Linguistics, 2024. URL https://api.semanticscholar.org/CorpusID:267750708

    2024

  14. [22]

    Hiddendetect: Detecting jailbreak attacks against large vision-language models via monitoring hidden states

    Jiang, Y., Gao, X., Peng, T., Tan, Y., Zhu, X., Zheng, B., and Yue, X. Hiddendetect: Detecting jailbreak attacks against large vision-language models via monitoring hidden states. arXiv preprint arXiv:2502.14744, 2025

    2025 arXiv

  15. [23]

    and Moeller, M

    L \"a hner, Z. and Moeller, M. On the direct alignment of latent spaces. In Proceedings of UniReps: the First Workshop on Unifying Representations in Neural Models, pp.\ 158--169. PMLR, 2024

    2024

  16. [24]

    K., and Mihalcea, R

    Lee, A., Bai, X., Pres, I., Wattenberg, M., Kummerfeld, J. K., and Mihalcea, R. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. arXiv preprint arXiv:2401.01967, 2024

    2024 arXiv

  17. [25]

    Inference-time intervention: Eliciting truthful answers from a language model

    Li, K., Patel, O., Vi \'e gas, F., Pfister, H., and Wattenberg, M. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024 a

    2024

  18. [26]

    Safety layers in aligned large language models: The key to llm security

    Li, S., Yao, L., Zhang, L., and Li, Y. Safety layers in aligned large language models: The key to llm security. arXiv preprint arXiv:2408.17003, 2024 b

    2024 arXiv

  19. [27]

    Flipattack: Jailbreak llms via flipping

    Liu, Y., He, X., Xiong, M., Fu, J., Deng, S., and Hooi, B. Flipattack: Jailbreak llms via flipping. arXiv preprint arXiv:2410.02832, 2024

    2024 arXiv

  20. [28]

    Codechameleon: Personalized encryption framework for jailbreaking large language models

    Lv, H., Wang, X., Zhang, Y., Huang, C., Dou, S., Ye, J., Gui, T., Zhang, Q., and Huang, X. Codechameleon: Personalized encryption framework for jailbreaking large language models. arXiv preprint arXiv:2402.16717, 2024

    2024 arXiv

  21. [29]

    Latent space translation via semantic alignment

    Maiorca, V., Moschella, L., Norelli, A., Fumero, M., Locatello, F., and Rodol \`a , E. Latent space translation via semantic alignment. Advances in Neural Information Processing Systems, 36: 0 55394--55414, 2023

    2023

  22. [30]

    Tree of attacks: Jailbreaking black-box llms automatically

    Mehrotra, A., Zampetakis, M., Kassianik, P., Nelson, B., Anderson, H., Singer, Y., and Karbasi, A. Tree of attacks: Jailbreaking black-box llms automatically. arXiv preprint arXiv:2312.02119, 2023

    2023 arXiv

  23. [31]

    Tree of attacks: Jailbreaking black-box llms automatically

    Mehrotra, A., Zampetakis, M., Kassianik, P., Nelson, B., Anderson, H., Singer, Y., and Karbasi, A. Tree of attacks: Jailbreaking black-box llms automatically. Advances in Neural Information Processing Systems, 37: 0 61065--61105, 2024

    2024

  24. [32]

    Introducing the world's best edge models

    Mistral. Introducing the world's best edge models. URL https://mistral.ai/news/ministraux/. Mistral AI blog post announcing Ministral 3B and 8B models for edge computing

  25. [33]

    Interpreting gpt: The logit lens

    Nostalgebraist. Interpreting gpt: The logit lens. LessWrong, 2020. URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  26. [34]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  27. [35]

    J., and Veitch, V

    Park, K., Choe, Y. J., and Veitch, V. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658, 2023

    2023 arXiv

  28. [36]

    Qin, L., Chen, Q., Zhou, Y., Chen, Z., Li, Y., Liao, L., Li, M., Che, W., and Yu, P. S. Multilingual large language model: A survey of resources, taxonomy and frontiers. arXiv preprint arXiv:2404.04925, 2024

    2024 arXiv

  29. [37]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  30. [38]

    A strongreject for empty jailbreaks, 2024

    Souly, A., Lu, Q., Bowen, D., Trinh, T., Hsieh, E., Pandey, S., Abbeel, P., Svegliato, J., Emmons, S., Watkins, O., and Toyer, S. A strongreject for empty jailbreaks, 2024

    2024

  31. [39]

    Mission impossible: A statistical perspective on jailbreaking llms

    Su, J., Kempe, J., and Ullrich, K. Mission impossible: A statistical perspective on jailbreaking llms. arXiv preprint arXiv:2408.01420, 2024

    2024 arXiv

  32. [40]

    Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  33. [41]

    Hermes 3 technical report, 2024

    Teknium, R., Quesnelle, J., and Guang, C. Hermes 3 technical report, 2024. URL https://arxiv.org/abs/2408.11857

    2024 arXiv

  34. [42]

    Detox: Toxic subspace projection for model editing

    Uppaal, R., Dey, A., He, Y., Zhong, Y., and Hu, J. Detox: Toxic subspace projection for model editing. arXiv e-prints, pp.\ arXiv--2405, 2024

    2024

  35. [43]

    Assessing the brittleness of safety alignment via pruning and low-rank modifications

    Wei, B., Huang, K., Huang, Y., Xie, T., Qi, X., and Xia. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint arXiv:2402.05162, 2024

    2024 arXiv

  36. [44]

    a ger, T., Elstner, J., Geisler, S., Cohen-Addad, V., G \

    Wollschl \"a ger, T., Elstner, J., Geisler, S., Cohen-Addad, V., G \"u nnemann, S., and Gasteiger, J. The geometry of refusal in large language models: Concept cones and representational independence. arXiv preprint arXiv:2502.17420, 2025

    2025

  37. [45]

    Y., and Poovendran, R

    Xu, Z., Jiang, F., Niu, L., Jia, J., Lin, B. Y., and Poovendran, R. Safedecoding: Defending against jailbreak attacks via safety-aware decoding. ArXiv, abs/2402.08983, 2024. URL https://api.semanticscholar.org/CorpusID:267658033

    2024 arXiv

  38. [46]

    Beyond toxic neurons: A mechanistic analysis of dpo for toxicity reduction

    Yang, Y., Sondej, F., Mayne, H., and Mahdi, A. Beyond toxic neurons: A mechanistic analysis of dpo for toxicity reduction. 2024. URL https://api.semanticscholar.org/CorpusID:273963284

    2024

  39. [47]

    A safety realignment framework via subspace-oriented model fusion for large language models

    Yi, X., Zheng, S., Wang, L., Wang, X., and He, L. A safety realignment framework via subspace-oriented model fusion for large language models. arXiv preprint arXiv:2405.09055, 2024

    2024 arXiv

  40. [48]

    Yong, Z.-X., Menghini, C., and Bach, S. H. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446, 2023

    2023 arXiv

  41. [49]

    Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts

    Yu, J., Lin, X., Yu, Z., and Xing, X. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253, 2023

    2023 arXiv

  42. [50]

    X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023

    2023 arXiv

  43. [51]

    How alignment and jailbreak work: Explain llm safety through intermediate hidden states

    Zhou, Z., Yu, H., Zhang, X., Xu, R., Huang, F., and Li, Y. How alignment and jailbreak work: Explain llm safety through intermediate hidden states. In Conference on Empirical Methods in Natural Language Processing, 2024 a . URL https://api.semanticscholar.org/CorpusID:270371990

    2024

  44. [52]

    On the role of attention heads in large language model safety

    Zhou, Z., Yu, H., Zhang, X., Xu, R., Huang, F., Wang, K., Liu, Y., Fang, J., and Li, Y. On the role of attention heads in large language model safety. ArXiv, abs/2410.13708, 2024 b . URL https://api.semanticscholar.org/CorpusID:273403424

    2024 arXiv

  45. [53]

    Z., and Fredrikson, M

    Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.