Pith. sign in

REVIEW 3 major objections 6 minor 59 references

Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Safety alignment can plant an input-independent bias toward rejective tokens, and the paper's TSDI method estimates and subtracts that bias at decoding time to improve helpfulness without sacrificing safety.

desk verdict TSDI is a useful, simple debiasing trick for DPO-style safety alignment, with solid but partly circular evaluation; the transfer assumption needs a direct test, but the core Pareto-front improvement is credible. read the letter →

arxiv 2502.02153 v1 pith:CQCQLA5S submitted 2025-02-04 cs.AI cs.CLcs.LG

Thien Q. Tran , Akifumi Wachi , Rei Sato , Takumi Tanabe , Youhei Akimoto This is my paper · ORCID
classification cs.AIcs.CLcs.LG
keywords safetyalignmenttoken-leveldebiasinglogitbiasover-refusalhelpfulness-safetytrade-offdirectpreferenceoptimizationinference-timeSALAD-Bench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that DPO-style safety alignment does more than suppress harmful outputs: it plants an input-independent bias that makes the model prefer negative, rejective tokens even on harmless prompts. Because that bias sits in the output logits, the authors argue it can be estimated cheaply and removed at inference time without retraining. Their Token-level Safety-Debiased Inference (TSDI) estimates a per-position bias vector from 500 random word-salad prompts, then subtracts it from logits during generation. Across the PKU-SafeRLHF/SACPO setup, they report that TSDI raises compliance and helpfulness win rates while keeping MD-Judge and Llama Guard 3 safety scores at the same level, improving the safety-helpfulness trade-off. If true, this gives a plug-in remedy for over-refusal in safety-aligned models that only requires logit access.

What carries the argument

The load-bearing object is the per-position logit bias vector $b_i$, defined as the average difference between the safety-aligned policy's logits and the reference policy's logits over randomly constructed prompt-response pairs $\tilde{D}$: $b_i = \frac{1}{|\tilde{D}|}\sum_{(x,y)\in\tilde{D}} [f_{\pi_\theta}(x\oplus y_{1:i-1}) - f_{\pi_r^*}(x\oplus y_{1:i-1})]$. TSDI subtracts $b_i$ from the aligned model's logits at the $i$-th generated token, i.e. it samples from $\mathrm{softmax}(f_{\pi_\theta}(x\oplus y_{1:i-1}) - b_i)$. Proposition 1 shows this is equivalent to replacing the implicitly learned safety function $g_\theta$ by a version centered with respect to the random-prompt distribution, removing the part of the safety signal that is independent of the input. The random prompts are built by sampling tokens from an MMLU-derived pool, and the paper uses $L=20$ and $|\tilde{D}|=500$; the bias is recomputed for each trained model.

What would settle it

Compute the per-position bias $b_i$ from random prompts and separately from 500 ordinary harmless prompts; if the two vectors diverge enough that subtracting the random-prompt $b_i$ increases the refusal rate on the harmless set or produces unsafe completions on a red-team suite, TSDI's transfer premise is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that safety alignment introduces a measurable, context-free shift in next-token logits: after DPO-based safety alignment, the logits of negative tokens such as 'sorry', 'unfortunately', 'none', and 'cannot' rise substantially even when the input is a randomly assembled token salad. The size of the shift grows as the KL penalty shrinks and varies by generation position. TSDI treats this shift as a bias vector and subtracts it during decoding, which the authors formalize in Proposition 1 as re-centering the implicit safety function used by the aligned policy. In their experiments on an Alpaca-7B replica aligned via SACPO, TSDI improves the compliance rate on harmless prompts without reducing safety scores, and the helpfulness win rate against the SFT model rises from 0.59 to 0.67 for the $\beta/\lambda = 0.025$, 200-iteration model while safety is preserved. They also report that the same pattern holds under Llama Guard 3 evaluation and that the method is stable to the choice of debiasing length and token pool.

Load-bearing premise

The load-bearing premise is that the bias measured on 500 random word-salad prompts is the same input-independent bias that acts on real user prompts, so subtracting it helps real responses rather than distorting them.

Editorial extensions

If this is right

  • Safety-aligned models trained by DPO-style safety alignment carry an input-independent bias toward rejection tokens, so even harmless prompts can draw openings such as 'I\'m sorry' or 'Unfortunately'.
  • Subtracting the estimated per-position bias at decoding raises the compliance rate on harmless prompts without lowering MD-Judge or Llama Guard 3 safety scores.
  • The helpfulness win rate versus the SFT model improves while safety is maintained, for example from 0.59 to 0.67 for the debiased model trained with $\beta/\lambda = 0.025$ for 200 iterations.
  • Because TSDI needs only logit access, it can be applied to any logit-accessible safety-aligned model without retraining or fitting a separate reward model.
  • Similar results hold for debiasing lengths $L=5,10,20$ and for token pools built from either MMLU or MS MARCO, indicating the bias estimate is not tied to one prompt-construction recipe.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the bias is estimated once per model and then applied indiscriminately, the same technique should transfer to other tasks and prompt distributions, and its failure modes would show up as a mismatch between the random-prompt bias and the actual bias on real prompt distributions.
  • Editorial inference: the paper records cases where removing the initial negative tokens still leaves an unhelpful continuation, so token-level debiasing is a partial fix; a natural next step is to apply the same re-centering idea to hidden states or later decoding positions.
  • Editorial inference: a cheap test of the transfer assumption would be to compute the bias vector on ordinary harmless prompts and compare it with the random-prompt vector; if they diverge, TSDI could be restricted to prompts that are already classified as safe.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies safety-aligned language models and argues that safety alignment induces a context-free token-level bias toward negative/refusal tokens, which hurts helpfulness on harmless prompts. It proposes TSDI, a learning-free decoding method that estimates per-position logit biases on randomly constructed prompts and subtracts them at generation time. Experiments with SACPO-style DPO safety alignment on an Alpaca-7B replica show improvements in compliance rate and GPT-4 helpfulness win rate while aggregate safety scores on SALAD-Bench remain high, yielding a claimed improvement of the safety-helpfulness Pareto front.

Significance. If validated, TSDI would be a simple, training-free intervention that improves the safety-helpfulness trade-off for DPO-style safety-aligned models using only logit access. The paper has notable strengths: the Proposition 1 algebra is clean; the qualitative Pareto-front improvement is reported across two safety evaluators, two token pools, several β/λ settings and iteration counts, and three seeds; and the ablations on L and token-pool choice are useful. The main limitations are that the central transfer assumption is not independently verified, the compliance-rate metric is partly aligned with the mechanism TSDI suppresses, and the safety evaluation does not isolate whether debiasing erodes refusals on harmful prompts.

major comments (3)
  1. [Sections 4.1-4.3] The load-bearing premise of TSDI is that the per-position bias vector b_i estimated from 500 random word-salad prompts (Section 4.1, Appendix C.4) is a context-free side effect of safety alignment that transfers to all real prompts, including harmful red-team prompts. Proposition 1 establishes only the algebraic identity that debiasing replaces g_theta by g_theta - E_{\tilde rho}[g_theta]; it does not show that E_{\tilde rho}[g_theta] is an unintended bias rather than an out-of-distribution artifact, nor that the debiased policy still assigns high safety value to refusal tokens on harmful inputs. The argument in Section 4.3 that random prompts have no overlap with training distribution D and hence should have zero g_theta is not implied by DPO training, because parameter sharing lets the safety update alter logits on arbitrary inputs. I ask for a direct test of the transfer premise, for example comparing b_i estimated on random prompts with b_i estimated on harmful prompts, and an analysis of whether debiasing flips any refusals on harmful queries into unsafe completions. Without such a test, the 'maintaining safety' claim is not fully established.
  2. [Section 5.1] The compliance-rate metric is partly circular with TSDI. Appendix C.6 lists 53 keywords including 'I'm sorry', 'Sorry', 'I apologize', 'I cannot', 'I do not', 'Unfortunately', and 'No,', which are exactly the refusal-token families whose logits TSDI is designed to suppress. Consequently, the large compliance-rate improvement in Figure 5a is expected by construction and should not be presented as independent evidence of helpfulness. The GPT-4 helpfulness win rate is the non-circular outcome and should be the primary helpfulness measure; please report its confidence intervals and explicitly state that the compliance metric is a mechanism sanity check rather than a helpfulness evaluation.
  3. [Section 5.2 and Appendix C.8] The safety-maintenance claim relies on aggregate per-category safety scores from MD-Judge and Llama Guard 3, but no confidence intervals are reported for these scores or for the GPT-4 win rates. The hypervolume significance test in Tables 3-4 reports zero standard deviation for the without-TSDI condition on two of three categories, which is not explained. More importantly, an aggregate safety score can remain high while debiasing erodes refusals on a subset of harmful prompts; the red-team examples in Appendix D.1 are anecdotal and show identical w/o and w/ TSDI outputs for several categories. Please provide per-category or per-prompt safety changes before and after debiasing, and a test of whether any previously safe refusal becomes unsafe after debiasing.
minor comments (6)
  1. [Section 3.1] The text says 66 safety categories with 68 prompts per category, giving 4488 prompts, but Figure 1 and Table 5 report only 16 category numbers; please clarify the relationship between subcategories and the 16 macro categories used in the figures.
  2. [Section 2.1] There is a typo in the second paragraph: 'Speficially' should be 'Specifically'.
  3. [Table 5] The category label 'Influence 0perations' contains a typo ('0' for 'O').
  4. [Equation (6)] The softmax function is denoted σ in Eq. (6) but SOFTMAX earlier in Section 4.1; please unify the notation.
  5. [Appendix C.4] The example random prompt is word-salad; the statement that this construction is 'reasonably close to the normal distribution of text' is overstated. Please soften this or provide a quantitative justification.
  6. [Section 5.1] The AlpacaEval dataset is used but not cited; please add the appropriate reference.

Circularity Check

1 steps flagged · score 3.0 of 10

One helpfulness metric is definitionally aligned with the debiasing operation, but the central Pareto claim rests on independent GPT-4 and external safety evaluations.

  1. self definitional [Section 5.1 (Evaluation) and Section 5.2 ("Can TSDI effectively remove negative tokens?"), with Appendix C.6 and Eq. (6)]
    "The compliance rate assesses if the models refuse to respond with expressions like “I’m sorry” or “Unfortunately.” We use 53 keywords, of which 47 keywords are from Zou et al. [2023]... This metric assesses the helpfulness at the token level, aligning with the intention of the proposed method. ... Importantly, the improvement is consistent across all training settings, which matches our expectation since TSDI is token-based, aligning with how the compliance rate is measured."

    The compliance metric counts absence of refusal phrases like 'I'm sorry', 'Unfortunately', 'I cannot', 'No' (Appendix C.6). TSDI is exactly a logit-level debiasing of the same negative-token family: Section 4.1 estimates b_i on random prompts, Figure 4 tracks 'sorry', 'unfortunately', 'no', 'cannot', and Eq. (6) subtracts b_i from logits. So 'TSDI significantly enhances the compliance rate' is largely a restatement of the debiasing operation rather than an independent helpfulness measurement: lowering logits of the exact strings the metric flags makes those strings less likely by construction. Independent evidence is the GPT-4 win rate (0.59→0.67) and external MD-Judge/Llama Guard 3 safety scores, which are not defined via the debiased tokens.

full rationale

The paper is largely self-contained. The bias vector b_i is a plug-in estimate of logit differences on random prompts; it is not fitted to maximize the reported safety or win-rate metrics. Proposition 1 is an algebraic identity showing what subtracting b_i does to the implicit safety function g_theta, and it does not by itself establish the transfer claim that the random-prompt bias equals a context-free alignment side effect on real prompts; that is a correctness risk, not circularity. The paper also does not lean on a load-bearing self-citation chain: SACPO is used as a concrete alignment method and is evaluated alongside the proposed method. The one genuinely definitional element is the compliance-rate metric, which counts the absence of refusal phrases (Appendix C.6) while TSDI suppresses exactly those token families (Figure 4, Eq. (6)); the paper itself notes the metric is 'aligned with the intention of the proposed method.' Because the GPT-4 helpfulness win rate and the SALAD-Bench safety scores are external to the debiasing construction, the central Pareto-front claim retains independent content. The circularity is therefore partial and confined to one auxiliary helpfulness metric, giving a score of 3 rather than a higher value.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

TSDI's central premise is that a bias vector estimated from random word-salad prompts transfers to all real inputs, and that this bias can be subtracted without eroding true safety behavior. The ledger lists the hand-chosen hyperparameters L, sample count, and token pool, plus the bias vectors themselves and the imported DPO/SACPO policy form. No new physical or architectural entities are introduced.

free parameters (5)
  • Bias vectors b_i for i=1..20 = Estimated from 500 random prompt-response pairs per model
    Core ingredient of TSDI: the correction subtracted from output logits is computed from synthetic random prompts, not from an independent theory. Its value depends on L, the token pool, and the aligned/reference model pair.
  • L, number of debiased output positions = 20 (robustness checked at 5 and 10)
    Hand-chosen design choice; TSDI only debiases the first L tokens, and the claim of improved Pareto front depends on L being adequate for refusal behavior.
  • Number of random prompt-response pairs = 500
    Hand-chosen sample size for bias estimation; no convergence or variance analysis of the bias estimate itself is provided.
  • Random prompt token pool = MMLU test-slice words; MS MARCO used for robustness
    The bias estimate depends on the distribution of random prompts; robustness to MMLU vs MS MARCO is shown, but the transfer to real prompts is assumed.
  • Safety alignment strengths beta/lambda = {0.1, 0.05, 0.025, 0.01} with 100, 200, 300 iterations
    Experimental grid; the headline TSDI point in Figure 1 uses beta/lambda=0.025 with 200 iterations, selected after seeing the full set of results.
assumptions (5)
  • domain assumption DPO/SACPO optimal policy form: pi_theta(y|x) proportional to pi*_r(y|x) exp((lambda/beta) g_theta(x,y))
    Imported from Rafailov et al. 2024 and Wachi et al. 2024; used in Section 4.3 and Appendix A to interpret TSDI as subtracting expected safety function values.
  • domain assumption Random prompts from the MMLU word pool are almost always irrelevant to the safety preference dataset topics
    Section 4.3 relies on the support of the random prompt distribution being outside the preference dataset so that nonzero expected safety values are unintended side effects. This is plausible but not formally established.
  • ad hoc to paper The safety bias estimated from random prompts is additive and context-free across all real prompts
    Section 4.1 assumes the logit difference on random word-salad inputs transfers to real harmless and harmful prompts. This is the load-bearing transfer premise of the method.
  • domain assumption MD-Judge and Llama Guard 3 provide valid per-category safety labels at the 0.5 threshold
    All safety scores in Figures 1, 5, and Table 5 rest on these classifiers, and the paper does not audit their errors on the SALAD-Bench subset.
  • domain assumption GPT-4 pairwise judgments are a valid helpfulness measure
    Helpfulness win rates rely on the GPT-4 prompt in Appendix C.5, following prior SACPO and Safe RLHF evaluation practice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing." pith.science (2026). https://pith.science/paper/CQCQLA5S

@misc{pith2026250202153,
  author       = {Pith},
  title        = {Pith review of: Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CQCQLA5S}},
  note         = {Machine review of arXiv:2502.02153}
}
read the original abstract

Safety alignment is an essential research topic for real-world AI applications. Despite the multifaceted nature of safety and trustworthiness in AI, current safety alignment methods often focus on a comprehensive notion of safety. By carefully assessing models from the existing safety-alignment methods, we found that, while they generally improved overall safety performance, they failed to ensure safety in specific categories. Our study first identified the difficulty of eliminating such vulnerabilities without sacrificing the model's helpfulness. We observed that, while smaller KL penalty parameters, increased training iterations, and dataset cleansing can enhance safety, they do not necessarily improve the trade-off between safety and helpfulness. We discovered that safety alignment could even induce undesired effects and result in a model that prefers generating negative tokens leading to rejective responses, regardless of the input context. To address this, we introduced a learning-free method, Token-level Safety-Debiased Inference (TSDI), to estimate and correct this bias during the generation process using randomly constructed prompts. Our experiments demonstrated that our method could enhance the model's helpfulness while maintaining safety, thus improving the trade-off Pareto-front.

Figures

Figures reproduced from arXiv: 2502.02153 by the authors.

Figure 1
Figure 1. (Left) Safety score for different safety categories evaluated by MD-Judge across different models. (Right) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Helpfulness win rate and safety score of Adult Content category for various [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. (a) Safety probabilities evaluated by MD-Judge for [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Token-wise differences in logits before and after safety alignment. (Left) logit differences for the first output [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Trade-offs between MD-Judge’s safety score of three different categories and (a) compliance rate to harmless [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Trade-offs between adult-related safety score and the compliance rate to harmless prompts for models trained [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Trade-offs between adult-related safety score and the helpful win rate versus SFT model for models trained [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Token-wise differences in logits before and after safety alignment. These models are trained with a safety [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Safety Level of Adult Content Category (Category 03) by MD-Judge for models trained with entire dataset [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Safety Level of Adult Content Category (Category 03) by Llama Guard 3 for models trained with entire [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: (Left panel) Safety score for different safety categories and helpfulness scores across different models. (Right panel) Trade-off between Llama Guard 3’s safety score and helpfulness win rate against the SFT model. 0.80 0.85 0.90 0.95 1.00 Compliance Rate 0.70 0.75 0.…
Figure 12
Figure 12. Figure 12: Trade-offs between Llama Guard’s safety score of three different categories and the compliance rate to harmless prompts. The number in bracket indicates the category number. Different points correspond to the combinations of different β/λ and number of iterations. 0.5…
Figure 13
Figure 13. Figure 13: Trade-offs between Llama Guard’s safety score of three different categories and the helpful win rate versus SFT model. The number in bracket indicates the category number. Different points correspond to the combinations of different β/λ and number of iterations. 18 […
Figure 14
Figure 14. Figure 14: Trade-offs between MD-Judge’s safety score for the Adult-content category and the helpfulness win rate compared to the SFT model when constructing random prompts with length L=5 and L=10 19 [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Trade-offs between MD-Judge’s safety score across three different categories and the compliance rate to harmless prompts when using a token pool constructed from the MS MARCO dataset. 0.5 0.6 0.7 Helpful Win Rate 0.5 0.6 0.7 0.8 0.9 Adult Content (Md-Judge) 0.5 0.6 0.…
Figure 16
Figure 16. Figure 16: Trade-offs between MD-Judge’s safety score across three different categories and the helpfulness win rate compared to the SFT model when using a token pool constructed from the MS MARCO dataset. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: Trade-offs between MD-Judge’s safety score for the Adult-content category and the helpfulness win rate compared to the SFT model when using a token pool constructed from the MS MARCO dataset. Here, we conduct comparison experiments with two safety baselines: SafeDecod…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 18 canonical work pages

  1. [1]

    GPT -4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT -4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Concrete problems in AI safety

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man \'e . Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016

    arXiv 2016

  3. [3]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2024

    work page 2024

  4. [4]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  5. [5]

    The ethics of artificial intelligence

    Nick Bostrom and Eliezer Yudkowsky. The ethics of artificial intelligence. In Artificial intelligence safety and security, pages 57--69. Chapman and Hall/CRC, 2018

    work page 2018

  6. [6]

    Driving with llms: Fusing object-level vector modality for explainable autonomous driving

    Long Chen, Oleg Sinavski, Jan H \"u nermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with llms: Fusing object-level vector modality for explainable autonomous driving. arXiv preprint arXiv:2310.01957, 2023

    arXiv

  7. [7]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  8. [8]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS), 2017

    work page 2017

Show all 59 references
  1. [9]

    Chatlaw: Open-source legal large language model with integrated external knowledge bases

    Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306.16092, 2023

    2023 arXiv

  2. [10]

    Safe RLHF : Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF : Safe reinforcement learning from human feedback. In International Conference on Learning Representations (ICLR), 2024

    2024

  3. [11]

    Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model

    Haikang Deng and Colin Raffel. Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model. arXiv preprint arXiv:2310.09520, 2023

    2023 arXiv

  4. [12]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  5. [13]

    KTO : Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO : Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  6. [14]

    Pal: Program-aided language models

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning (ICML), 2023

    2023

  7. [15]

    Realtoxicityprompts: Evaluating neural toxic degeneration in language models

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462, 2020

    2009 arXiv

  8. [16]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  9. [17]

    Reference-free monolithic preference optimization with odds ratio

    Jiwoo Hong, Noah Lee, and James Thorne. Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691, 2024

    2024 arXiv

  10. [18]

    One-shot safety alignment for large language models via optimal dualization

    Xinmeng Huang, Shuo Li, Edgar Dobriban, Osbert Bastani, Hamed Hassani, and Dongsheng Ding. One-shot safety alignment for large language models via optimal dualization. arXiv preprint arXiv:2405.19544, 2024

    2024 arXiv

  11. [19]

    Llama guard: Llm-based input-output safeguard for human-ai conversations

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023

    2023 arXiv

  12. [20]

    Reference point specification in hypervolume calculation for fair comparison and efficient search

    Hisao Ishibuchi, Ryo Imada, Yu Setoguchi, and Yusuke Nojima. Reference point specification in hypervolume calculation for fair comparison and efficient search. In Proceedings of the genetic and evolutionary computation conference, pages 585--592, 2017

    2017

  13. [21]

    AI alignment: A comprehensive survey

    Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. AI alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852, 2023

    2023 arXiv

  14. [22]

    Pku-saferlhf: A safety alignment preference dataset for llama family models

    Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. Pku-saferlhf: A safety alignment preference dataset for llama family models. arXiv preprint arXiv:2406.15513, 2024 a

    2024 arXiv

  15. [23]

    Beavertails: Towards improved safety alignment of LLM via a human-preference dataset

    Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. Advances in Neural Information Processing Systems (NeurIPS), 2024 b

    2024

  16. [24]

    SALAD-Bench : A hierarchical and comprehensive safety benchmark for large language models

    Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. SALAD-Bench : A hierarchical and comprehensive safety benchmark for large language models. arXiv preprint arXiv:2402.05044, 2024 a

    2024 arXiv

  17. [25]

    Rethinking jailbreaking through the lens of representation engineering, 2024 b

    Tianlong Li, Shihan Dou, Wenhao Liu, Muling Wu, Changze Lv, Rui Zheng, Xiaoqing Zheng, and Xuanjing Huang. Rethinking jailbreaking through the lens of representation engineering, 2024 b . URL https://arxiv.org/abs/2401.06824

    2024 arXiv

  18. [26]

    Controllable text generation for large language models: A survey

    Xun Liang, Hanyu Wang, Yezhaohui Wang, Shichao Song, Jiawei Yang, Simin Niu, Jie Hu, Dan Liu, Shunyu Yao, Feiyu Xiong, et al. Controllable text generation for large language models: A survey. arXiv preprint arXiv:2408.12599, 2024

    2024 arXiv

  19. [27]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021

    2021 arXiv

  20. [28]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451, 2023 a

    2023 arXiv

  21. [29]

    Trustworthy LLMs : a survey and guideline for evaluating large language models' alignment

    Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy LLMs : a survey and guideline for evaluating large language models' alignment. arXiv preprint arXiv:2308.05374, 2023 b

    2023 arXiv

  22. [30]

    Enhancing LLM safety via constrained direct preference optimization

    Zixuan Liu, Xiaolin Sun, and Zizhan Zheng. Enhancing LLM safety via constrained direct preference optimization. arXiv preprint arXiv:2403.02475, 2024

    2024 arXiv

  23. [31]

    Meta llama guard 2

    Llama-Team. Meta llama guard 2. https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL_CARD.md, 2024

    2024

  24. [32]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024

    2024 arXiv

  25. [33]

    Controlled decoding from language models

    Sidharth Mudgal, Jong Lee, Harish Ganapathy, Yaguang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled decoding from language models. In Proceedings of the 41st International Conf...

    2024

  26. [34]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  27. [35]

    Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023

    2023 arXiv

  28. [36]

    Safety alignment should be made more than just a few tokens deep

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. arXiv preprint arXiv:2406.05946, 2024

    2024 arXiv

  29. [37]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2024

    2024

  30. [38]

    Benchmarking safe exploration in deep reinforcement learning

    Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 2019

    1910 arXiv

  31. [39]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  32. [40]

    LM-Nav : Robotic navigation with large pre-trained models of language, vision, and action

    Dhruv Shah, B a \.z ej Osi \'n ski, Sergey Levine, et al. LM-Nav : Robotic navigation with large pre-trained models of language, vision, and action. In Conference on Robot Learning (CoRL), pages 492--504. PMLR, 2023

    2023

  33. [41]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca : An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  34. [42]

    Large language models in medicine

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29 0 (8): 0 1930--1940, 2023

    1930

  35. [43]

    TRL : Transformer reinforcement learning

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. TRL : Transformer reinforcement learning. https://github.com/huggingface/trl, 2020

    2020

  36. [44]

    Tran, Rei Sato, Takumi Tanabe, and Youhei Akimoto

    Akifumi Wachi, Thien Q. Tran, Rei Sato, Takumi Tanabe, and Youhei Akimoto. Stepwise alignment for constrained language model policy optimization. arXiv preprint arXiv:2404.11049, 2024

    2024 arXiv

  37. [45]

    Decodingtrust: A comprehensive assessment of trustworthiness in GPT models

    Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in GPT models. arXiv preprint arXiv:2306.11698, 2023 a

    2023 arXiv

  38. [46]

    Do-not-answer: A dataset for evaluating safeguards in llms

    Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. Do-not-answer: A dataset for evaluating safeguards in llms. arXiv preprint arXiv:2308.13387, 2023 b

    2023 arXiv

  39. [47]

    Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36, 2024

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36, 2024

    2024

  40. [48]

    Defending chatgpt against jailbreak attack via self-reminders

    Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 5 0 (12): 0 1486--1496, 2023

    2023

  41. [49]

    Safedecoding: Defending against jailbreak attacks via safety-aware decoding

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. Safedecoding: Defending against jailbreak attacks via safety-aware decoding. arXiv preprint arXiv:2402.08983, 2024 a

    2024 arXiv

  42. [50]

    Uncovering safety risks of large language models through concept activation vector, 2024 b

    Zhihao Xu, Ruixuan Huang, Changyu Chen, and Xiting Wang. Uncovering safety risks of large language models through concept activation vector, 2024 b . URL https://arxiv.org/abs/2404.12038

    2024 arXiv

  43. [51]

    Shadow alignment: The ease of subverting safely-aligned language models

    Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949, 2023

    2023 arXiv

  44. [52]

    On the vulnerability of safety alignment in open-access llms

    Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. On the vulnerability of safety alignment in open-access llms. In Findings of the Association for Computational Linguistics ACL 2024, pages 9236--9260, 2024

    2024

  45. [53]

    Wordcraft: story writing with large language models

    Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. Wordcraft: story writing with large language models. In International Conference on Intelligent User Interfaces, pages 841--852, 2022

    2022

  46. [54]

    Prompting large language model for machine translation: A case study

    Biao Zhang, Barry Haddow, and Alexandra Birch. Prompting large language model for machine translation: A case study. In International Conference on Machine Learning (ICML), 2023

    2023

  47. [55]

    Panacea: Pareto alignment via preference adaptation for LLM s

    Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. Panacea: Pareto alignment via preference adaptation for LLM s. arXiv preprint arXiv:2402.02030, 2024

    2024 arXiv

  48. [56]

    Beyond one-preference-for-all: Multi-objective direct preference optimization

    Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preference-for-all: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708, 2023

    2023 arXiv

  49. [57]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

  50. [58]

    Universal and transferable adversarial attacks on aligned language models

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023

    2023 arXiv

  51. [59]

    Improving alignment and robustness with short circuiting

    Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with short circuiting. arXiv preprint arXiv:2406.04313, 2024

    2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.