Pith. sign in

REVIEW 5 major objections 6 minor 3 cited by

Advancing LLM Safe Alignment with Safety Representation Ranking

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A frozen LLM's hidden states can rank its own replies by safety.

desk verdict SRR is a clear, lightweight ranking idea, but the keyword-derived labels make the reported accuracies uninterpretable as safety metrics. read the letter →

arxiv 2505.15710 v1 pith:6G4TBLAT submitted 2025-05-21 cs.CL cs.LG

Tianqi Du , Zeming Wei , Quan Chen , Chenheng Zhang , Yisen Wang This is my paper
classification cs.CLcs.LG
keywords safetyalignmentlearningtorankhiddenstatesrepresentationengineeringjailbreakrobustnesslistwiserankingLLMpost-generationdefense
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Safety Representation Ranking (SRR) is a post-generation safety filter that never touches the base model's decoding logic. The paper claims that a frozen LLM's internal hidden states carry enough safety signal that a tiny single-layer transformer can rank candidate responses safe-first, outperforming a much larger GPT-2 reward-model baseline on HarmBench, SorryBench, and JailbreakBench. If the claim holds, a lightweight ranker could act as a universal safeguard, reducing harmful outputs under jailbreak attacks while leaving benign task performance essentially unchanged. The paper also argues that the safety signal generalizes across datasets and can be redirected to privacy and fairness concerns, though fairness accuracy hovers near chance.

What carries the argument

The load-bearing object is the Safety Representation Ranking (SRR) scorer: a shared linear projection plus one transformer block that takes the sequence [instruction hidden state, response hidden states] and outputs cosine-similarity logits between instruction and each response. It is trained end-to-end with the base LLM frozen, using a listwise KL objective against a uniform distribution over the safe responses in each candidate list. This machinery turns safety into a compatibility score: safe responses are ones whose projected hidden states align with the contextualized instruction representation.

What would settle it

Build a test set where harmful responses include polite refusal words like 'Sorry' and safe responses avoid them, or where harmful answers say 'I understand, but here is...' while safe ones use other phrasing, then run the published SRR training and evaluation pipeline; if accuracy collapses toward chance, the ranker was detecting keywords rather than safety.

Watch

Extended reading notes

Core claim

The central discovery is that safety can be treated as a ranking problem over the LLM's own internal representations rather than as a property of surface text. For each instruction, SRR samples several candidate completions, extracts hidden-state vectors from a lower intermediate layer of the frozen model, projects them down, concatenates them with the instruction vector, and passes the sequence through a single-layer transformer encoder. It then scores each response by cosine similarity to the contextualized instruction vector and trains with a listwise KL divergence that pushes probability mass onto the safe candidates. In the paper's experiments, this procedure selects safe responses with roughly 88-90% accuracy across datasets, transfers to unseen safety benchmarks, improves accuracy over the base model's default first-choice response under real jailbreak attacks, and leaves MATH accuracy unchanged.

Load-bearing premise

The labels that define 'safe' and 'harmful' are keyword patterns (responses containing 'Sorry', 'unable', 'illegal', or 'understand' count as safe, those with 'sure' or 'certainly' count as harmful), and the whole accuracy story depends on these surface patterns being faithful proxies for true safety.

Editorial extensions

If this is right

  • SRR can be bolted onto any frozen LLM as a post-generation ranker, imposing no change to token distributions or decoding strategy.
  • A ranker trained on one safety benchmark transfers to other benchmarks, suggesting the hidden-state safety signal is not dataset-specific.
  • Under real jailbreak attacks, selecting SRR's top-ranked response beats the model's natural first response, reducing harmful outputs.
  • Because the ranker is trained only on safety labels, its ranking does not measurably shift accuracy on benign math problems.
  • The same ranking machinery can be retargeted to privacy infringement detection with high accuracy, and to fairness with near-chance accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the hidden-state safety signal is real, SRR could serve as a cheap probe for auditing other alignment properties without retraining the base model, since only the small ranker changes.
  • The near-chance fairness results suggest that safety and fairness are not carried by the same representation directions; a single similarity space may not unify all alignment goals.
  • The method's dependence on keyword-derived labels is testable: replacing the keyword labels with human or judge labels would reveal whether SRR is learning surface refusal markers or deeper harmfulness.
  • Because ranking depends on sampling diverse candidates, SRR's utility in deployment is bounded by the base model's candidate diversity; a model that always refuses will give the ranker no harmful candidate to downrank.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript proposes Safety Representation Ranking (SRR), a listwise ranking framework that scores candidate responses by safety using hidden states of a frozen LLM. A lightweight single-layer transformer ranker is trained with a KL listwise loss on instruction-response lists with binary safety labels; at inference it ranks candidates by cosine similarity between instruction and response representations. Experiments report high accuracy on three safety benchmarks, cross-dataset transfer, extensions to privacy and fairness, improved robustness over a GPT-2 baseline in real-world attack settings, and no degradation on MATH. The method is clearly described, but the empirical evaluation is compromised by the keyword-based definition of safety and by weak or underspecified baselines and deployment experiments.

Significance. If the central claim held, SRR would be an attractive safeguard: it is lightweight, does not change the base model's decoding logic, and uses internal representations that output-only judges may miss. The paper deserves credit for a clear method description, a standard listwise ranking formulation, and for including cross-dataset and natural-performance experiments. However, the evidence is currently not sufficient to support the abstract's claim that SRR 'significantly improves robustness to adversarial prompts.' The evaluation relies on surface-cue labels in both training and testing, compares only against an untuned GPT-2, and omits essential protocol details in the real-world experiments. The central idea is plausible, but the operational definition of safety is the weakest link and must be re-established with independent ground truth before the empirical claims can be accepted.

major comments (5)
  1. [Section 4.1, Tables 1-2] The safety labels are defined by keyword matching: a response is safe if it contains 'Sorry', 'unable', 'illegal', or 'understand', and harmful if it contains 'sure' or 'certainly'. Since the same keyword scheme creates both the training labels and the test correctness labels, the reported accuracies in Tables 1 and 2 are consistent with the ranker detecting refusal phrase patterns in hidden states rather than evaluating safety. The central robustness claim is therefore not established. Please redo the evaluation with human-annotated or independently judged labels (with inter-annotator agreement reported), evaluate on responses generated by attack families not used for candidate generation, and include an error analysis showing that accuracy is not explained by keyword presence or absence.
  2. [Section 4.1, Baseline] A pretrained GPT-2 is not a reward model; without fine-tuning on preference or safety data it cannot serve as a representative baseline for reward-model-based ranking. Consequently, the claim in Section 4.2 that SRR 'significantly outperforms the reward model' is unsupported. Compare against an established safety reward model or safety classifier (e.g., Llama Guard, Beaver, or a trained reward model), and report statistical significance over multiple seeds and evaluation runs.
  3. [Section 5.1, Tables 5-7] The 'real-world' experiments omit crucial protocol details: the attack method used to generate jailbroken candidates, the number of candidates per prompt, the ground-truth labeling procedure for 'safe vs harmful' in this setting, and the definition of 'first accuracy'. Vicuna is absent from Tables 5 and 6, and no variance or significance measures are reported. Without this information and an independent label source, Tables 5-7 cannot support the claim that SRR reduces harmful outputs under attack.
  4. [Section 4.4, Table 4] The fairness experiment reports 52.52% average accuracy, which is near chance, yet the text claims SRR 'shows a foundational ability to distinguish between biased and unbiased responses.' Since contribution 2 explicitly includes fairness adaptation, this near-chance result should be reported as negative or inconclusive, or the fairness claim should be removed.
  5. [Section 5.2, Table 8] The natural-performance claim rests on a single model (Qwen2.5-7B), a single benchmark (MATH), and no repeated runs; the reported ±0.2% fluctuations are asserted to be within the margin of noise without confidence intervals. Please provide multiple runs with standard errors, and ideally an additional benign benchmark, or temper the claim accordingly.
minor comments (6)
  1. [Section 3.2] Typo: 'intermeidate' should be 'intermediate'.
  2. [Section 4.2] The sentence 'As depicted in the 1' should read 'As depicted in Table 1'.
  3. [Section 5.1] The dataset name is written inconsistently as 'JailbreakingBench' here and 'JailbreakBench' elsewhere.
  4. [Section 4.1] The exact hidden layer used for representation extraction, the softmax temperature τ, and the number of training epochs are not given; 'approximately the bottom 25% of layers' is too vague for replication.
  5. [Tables 1, 5-7] No standard deviations, confidence intervals, or numbers of runs are reported for the ranking accuracy values.
  6. [General] The paper states that code will be available upon publication; for reproducibility, the code and data should be released with the revision.

Circularity Check

1 steps flagged · score 6.0 of 10

Safety-ranking accuracy reduces to keyword-label prediction: the same keyword heuristic defines both the training labels and the evaluation labels.

  1. fitted input called prediction [Section 4.1 (Experiment Set-up, data filtering and Metrics)]
    "After sampling, we check whether the answers are valid by examining the keywords in them. For safe answers, we examine "Sorry", "unable", "illegal", and "understand". For harmful answers, we examine "sure", "certainly". We then filter the answers using the above criterion to get high-quality data. ... In the experiments in this section, we request the model to choose between the safe answer and the harmful answer, and verify its correctness, referring to its label."

    The training signal and the evaluation ground truth are produced by the same keyword-based rule: a response is 'safe' iff it contains Sorry/unable/illegal/understand and 'harmful' iff it contains sure/certainly. The ranker is trained to assign high similarity to keyword-safe responses and low similarity to keyword-harmful responses, and accuracy is then measured by checking whether the selected response carries the same keyword label. The reported 'safety ranking accuracy' is therefore, by construction, accuracy at predicting this surface keyword criterion, not at judging safety. The abstract's robustness claim rests on Tables 5-7, which report accuracy without specifying an independent label protocol; if those labels use the same keyword rule, the same reduction applies.

full rationale

The SRR method itself is not mathematically circular: the listwise ranking loss, cosine-similarity scorer, and frozen-LLM feature extraction are internally consistent, and the self-citations (e.g., [35] for In-context Attack/Defense, [44] for representation engineering) are method citations rather than load-bearing uniqueness theorems. The circularity is in the evaluation construct. Section 4.1 defines the safety ground truth by keyword occurrence and reuses that same definition for both filtering training data and verifying test correctness. A ranker reading hidden states can detect refusal/compliance phrasing with high accuracy, so the 80-96% accuracies in Tables 1-2 and the cross-dataset numbers in Table 2 demonstrate generalization across prompts under one fixed keyword label rule, not generalization to safety as an external construct. The 'real-world' experiments (Tables 5-7) are presented as evidence of robustness to jailbreak attacks but report only accuracy with no stated independent labeling protocol, so they do not break the self-fulfilling loop. This is a partial circularity: the central empirical claim reduces to keyword-label prediction by construction, while the underlying representation-ranking idea retains independent content.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper's central claims depend on a keyword-based safety label, a hidden-state similarity assumption, and several unreported hyperparameters. No new physical or architectural entities are introduced.

free parameters (5)
  • Safety label keywords = 'Sorry', 'unable', 'illegal', 'understand' for safe; 'sure', 'certainly' for harmful
    Used to filter and label training/eval responses (Section 4.1). The ranker may learn these surface cues rather than robust safety semantics.
  • Layer fraction for hidden-state extraction = bottom ~25%
    Ranker uses features from approximately the bottom 25% of layers (Section 4.1, Ranker Settings). No ablation or selection rationale.
  • Softmax temperature tau = not reported
    Temperature scaling parameter in Eq. (3) affects the listwise loss sharpness; value never given (Section 3.3).
  • Number of candidate responses m = 2
    All training/eval groups contain exactly two candidates (Section 4.1), so the 'listwise' ranking reduces to pairwise classification.
  • Sampling count per prompt = 20 per attack type (in-context attack and defense)
    Candidate generation protocol in Section 4.1; diversity of candidates is acknowledged as a limitation (Section 6).
assumptions (4)
  • domain assumption Intermediate hidden states of an LLM contain linear or similarity-based safety signals (refs [34], [49]).
    The SRR design assumes cosine similarity in a learned projection captures safety-relevant information (Section 3.2).
  • ad hoc to paper Keyword matching over 'Sorry', 'unable', etc. is a valid binary safety label.
    Used to construct and label the training/test pairs in Section 4.1. No human verification or independent classifier.
  • domain assumption In-context Attack and In-context Defense prompts produce representative safe and unsafe candidate responses.
    Training candidates are generated exclusively by these two prompting strategies (Section 4.1), limiting the attack distribution.
  • domain assumption Sampling from the base model with moderate temperature yields diverse candidates without changing the generation distribution.
    Assumed in Section 3.1 and in the MATH natural-performance study (Section 5.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Advancing LLM Safe Alignment with Safety Representation Ranking." pith.science (2026). https://pith.science/paper/6G4TBLAT

@misc{pith2026250515710,
  author       = {Pith},
  title        = {Pith review of: Advancing LLM Safe Alignment with Safety Representation Ranking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6G4TBLAT}},
  note         = {Machine review of arXiv:2505.15710}
}
read the original abstract

The rapid advancement of large language models (LLMs) has demonstrated milestone success in a variety of tasks, yet their potential for generating harmful content has raised significant safety concerns. Existing safety evaluation approaches typically operate directly on textual responses, overlooking the rich information embedded in the model's internal representations. In this paper, we propose Safety Representation Ranking (SRR), a listwise ranking framework that selects safe responses using hidden states from the LLM itself. SRR encodes both instructions and candidate completions using intermediate transformer representations and ranks candidates via a lightweight similarity-based scorer. Our approach directly leverages internal model states and supervision at the list level to capture subtle safety signals. Experiments across multiple benchmarks show that SRR significantly improves robustness to adversarial prompts. Our code will be available upon publication.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. One Anchor for All: Unified Multilingual and Multimodal Safety Alignment for LVLMs

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A small set of model neurons shared across languages and modalities can transfer English-only safety training to multilingual and multimodal refusal behavior.

  2. Targeted Interpretable Safety Neuron Enhancement for Multilingual Vision-Language Large Models

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    Precise Shield identifies safety neurons in VLLMs via activation contrasts and aligns only them with gradient masking, boosting safety, preserving generalization, and enabling zero-shot cross-lingual and cross-modal transfer.

  3. SEALGuard: Safeguarding the Multilingual Conversations in Southeast Asian Languages for LLM Software Systems

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A LoRA-adapted SeaLLM model detects unsafe and jailbreak prompts in nine Southeast Asian languages with 97% recall and 98% F1 on the authors' new SEALSBench benchmark, far above zero-shot LlamaGuard.

Reference graph

Works this paper leans on

50 extracted references · 32 canonical work pages · cited by 3 Pith papers

  1. [1]

    Foundational challenges in assuring alignment and safety of large language models

    Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. Transactions on Machine Learning Research, 2024. 1, 2

    work page 2024

  2. [2]

    Constitutional ai: Harmlessness from ai feedback, 2022

    Yuntao Bai et al. Constitutional ai: Harmlessness from ai feedback, 2022. 1

    work page 2022

  3. [3]

    Safeinfer: Context adaptive decoding time safety alignment for large language models

    Somnath Banerjee, Sayan Layek, Soham Tripathy, Shanu Kumar, Animesh Mukherjee, and Rima Hazra. Safeinfer: Context adaptive decoding time safety alignment for large language models. In AAAI, volume 39, pages 27188–27196, 2025. 1

    work page 2025

  4. [4]

    Le, Christopher Ré, and Azalia Mirhoseini

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V . Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024. 2

    arXiv 2024

  5. [5]

    Pappas, Florian Tramer, Hamed Hassani, and Eric Wong

    Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jailbreaking large language models, 2024. 5

    work page 2024

  6. [6]

    Finding safety neurons in large language models

    Jianhui Chen, Xiaozhi Wang, Zijun Yao, Yushi Bai, Lei Hou, and Juanzi Li. Finding safety neurons in large language models. arXiv preprint arXiv:2406.14144, 2024. 3

    arXiv 2024

  7. [7]

    Safe rlhf: Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, et al. Safe rlhf: Safe reinforcement learning from human feedback. In ICLR, 2024. 1

    work page 2024

  8. [8]

    Hierarchical neural story generation

    Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), 2018. 3

    work page 2018

Show all 50 references
  1. [9]

    Controlling linguistic style aspects in neural language generation

    Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic V ariation, 2017. 3

    2017

  2. [10]

    Evolving neural turing machines for reward-based learning

    Rasmus Boll Greve, Emil Juul Jacobsen, and Sebastian Risi. Evolving neural turing machines for reward-based learning. In Proceedings of the Genetic and Evolutionary Computation Conference 2016, pages 117–124, Denver Colorado USA, 2016. ACM. 2

    2016

  3. [11]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Advances in Neural Information Processing Systems , 2021. 9

    2021

  4. [12]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations , 2020. 3

    2020

  5. [13]

    Learning to write with cooperative discriminators

    Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), 2018. 3

    2018

  6. [14]

    Llama guard: Llm-based input-output safeguard for human-ai conversations

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023. 1

    2023 arXiv

  7. [15]

    Ai alignment: A comprehensive survey

    Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852, 2023. 1, 2

    2023 arXiv

  8. [16]

    Mistral 7b

    Albert Q Jiang, A Sablayrolles, A Mensch, C Bamford, D Singh Chaplot, Ddl Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b. arxiv. arXiv preprint arXiv:2310.06825, 10,

    arXiv

  9. [17]

    Buckley, Jason Phang, Samuel R

    Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. Pretraining language models with human preferences. In ICML, 2023. 1

    2023

  10. [18]

    Contrastive decoding: Open-ended text generation as optimization

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis...

    2023

  11. [19]

    Lipo: Listwise preference optimization through learning-to-rank

    Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, et al. Lipo: Listwise preference optimization through learning-to-rank. arXiv preprint arXiv:2402.01878, 2024. 4

    2024 arXiv

  12. [20]

    Jailbreaking chatgpt via prompt engineering: An empirical study, 2023

    Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study, 2023. 1

    2023

  13. [21]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In ICML,

  14. [22]

    Llm improvement for jailbreak defense: Analysis through the lens of over-refusal

    Swetasudha Panda, Naveen Jafer Nizar, and Michael L Wick. Llm improvement for jailbreak defense: Analysis through the lens of over-refusal. In Neurips Safe Generative AI Workshop 2024, 2024. 1

    2024

  15. [23]

    Learning to rank from relevance judgments distributions

    Alberto Purpura, Gianmaria Silvello, and Gian Antonio Susto. Learning to rank from relevance judgments distributions. Journal of the Association for Information Science and Technology , 73(9):1236–1252, 2022. 4

    2022

  16. [24]

    Safety alignment should be made more than just a few tokens deep

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. arXiv preprint arXiv:2406.05946, 2024. 3

    2024 arXiv

  17. [25]

    Language models are unsupervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. 5

    2019

  18. [26]

    Rewarding progress: Scaling automated process verifiers for llm reasoning

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for llm reasoning. arXiv preprint arXiv:2410.08146, 2024. 3

    arXiv

  19. [27]

    Does representation matter? exploring intermediate layers in large language models

    Oscar Skean, Md Rifat Arefin, Yann LeCun, and Ravid Shwartz-Ziv. Does representation matter? exploring intermediate layers in large language models. arXiv preprint arXiv:2412.09563, 2024. 4

    2024 arXiv

  20. [28]

    Reft: Reason- ing with reinforced fine-tuning

    Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reason- ing with reinforced fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), 2024. 3

    2024

  21. [29]

    Interpretable prefer- ences via multi-objective reward modeling and mixture-of-experts, 2024

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable prefer- ences via multi-objective reward modeling and mixture-of-experts, 2024. 3

    2024

  22. [30]

    Self-consistency improves chain of thought reasoning in language models, 2023

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023. 3

    2023

  23. [31]

    Chain-of-thought reasoning without prompting

    Xuezhi Wang and Denny Zhou. Chain-of-thought reasoning without prompting. arXiv preprint arXiv:2402.10200, 2024. 3

    2024 arXiv

  24. [32]

    A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more

    Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Mao, Sitaram Asur, et al. A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407.16216, 2024. 1

    2024 arXiv

  25. [33]

    Jailbroken: How does llm safety training fail? In NeurIPS, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? In NeurIPS, 2023. 1

    2023

  26. [34]

    Assessing the brittleness of safety alignment via pruning and low-rank modifications

    Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. In ICML, 2024. 2, 3

    2024

  27. [35]

    Jailbreak and guard aligned language models with only few in-context demonstrations

    Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387, 2023. 1, 3

    2023 arXiv

  28. [36]

    Sorry-bench: Systematically evaluating large language model safety refusal

    Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, et al. Sorry-bench: Systematically evaluating large language model safety refusal. In ICLR, 2025. 1, 5

    2025

  29. [37]

    Defending chatgpt against jailbreak attack via self-reminders

    Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 2023. 1 11

    2023

  30. [38]

    Safedecoding: Defending against jailbreak attacks via safety-aware decoding

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Pooven- dran. Safedecoding: Defending against jailbreak attacks via safety-aware decoding. In ACL, pages 5587–5605. Association for Computational Linguistics (ACL), 2024. 1

    2024

  31. [39]

    SafeDecoding: Defending against jailbreak attacks via safety-aware decoding

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Pooven- dran. SafeDecoding: Defending against jailbreak attacks via safety-aware decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Lo...

    2024

  32. [40]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. 5

    2024 arXiv

  33. [41]

    The ai alignment problem: why it is hard, and where to start

    Eliezer Yudkowsky. The ai alignment problem: why it is hard, and where to start. Symbolic Systems Distinguished Speaker, 4(1), 2016. 2

    2016

  34. [42]

    Rest-mcts*: Llm self-training via process reward guided tree search, 2024

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search, 2024. 2

    2024

  35. [43]

    General preference modeling with preference representations for aligning language models

    Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, and Quanquan Gu. General preference modeling with preference representations for aligning language models. arXiv preprint arXiv:2410.02197, 2024. 3

    2024 arXiv

  36. [44]

    Adversarial representation engineering: A general model editing framework for large language models

    Yihao Zhang, Zeming Wei, Jun Sun, and Meng Sun. Adversarial representation engineering: A general model editing framework for large language models. arXiv preprint arXiv:2404.13752,

    arXiv

  37. [45]

    Alleviating hallucinations of large language models through induced hallucinations

    Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi. Alleviating hallucinations of large language models through induced hallucinations. arXiv preprint arXiv:2312.15710, 2024. 3

    2024 arXiv

  38. [46]

    Identifying and tuning safety neurons in large language models

    Yiran Zhao, Wenxuan Zhang, Yuxi Xie, Anirudh Goyal, Kenji Kawaguchi, and Michael Shieh. Identifying and tuning safety neurons in large language models. In ICLR, 2025. 3

    2025

  39. [47]

    On prompt-driven safeguarding for large language models

    Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. On prompt-driven safeguarding for large language models. In International Conference on Machine Learning, pages 61593–61613. PMLR, 2024. 1, 3

    2024

  40. [48]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36:46595–46623, 2023. 2, 5

    2023

  41. [49]

    Representation engineering: A top-down approach to ai transparency

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023. 2, 3

    arXiv

  42. [50]

    Universal and transferable adversarial attacks on aligned language models

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. 1, 3 12

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.