Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

On Almost Surely Safe Alignment of Large Language Models at Inference-Time

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Safe alignment of an LLM can be done at generation time, without touching the model's weights, by tracking how much safety budget remains and steering decoding away from responses that would spend it.

desk verdict Useful empirical work on inference-time safety, but the 'almost surely' guarantee covers the idealized MDP, not the deployed beam search, and the latent-space transfer rests on an unproven factorization assumption. read the letter →

arxiv 2502.01208 v3 pith:CDXLOSD6 submitted 2025-02-03 cs.LG cs.CL

Xiaotong Ji , Shyam Sundhar Ramesh , Matthieu Zimmer , Ilija Bogunovic , Jun Wang , Haitham Bou Ammar This is my paper · ORCID
classification cs.LGcs.CL
keywords inference-timealignmentalmostsuresafetyconstrainedMarkovdecisionprocessstateaugmentationlatent-spacecriticbeamsearchdecodingLLMbudget
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that safe alignment of an LLM need not modify the model's weights: it can be done at inference time by tracking a remaining safety budget and steering decoding away from responses that would exhaust it. The authors cast safe generation as a constrained Markov decision process, then convert it into an unconstrained one by augmenting the state with the remaining discounted budget and imposing an infinite penalty once the budget is spent. They prove that solving this augmented problem in the compact latent space, using the model's hidden states and logits, still yields an optimal policy in the original token space, and that a policy with finite cost is safe almost surely with respect to the given safety cost function. The practical algorithm, InferenceGuard, combines this with beam search and a small learned critic, reporting safety rates of 94.46% to 100% across four LLMs while keeping task reward competitive. If correct, this offers a formal, weight-free alternative to RLHF-style safety training.

What carries the argument

The load-bearing construction is safety state augmentation: the scalar $z_t = (d - \sum_{k=1}^t \gamma^k C_{\text{safety}}(s_k, a_k))/\gamma^t$ tracks the remaining discounted safety budget, updating as $z_{t+1} = (z_t - C_{\text{safety}}(s_t, a_t))/\gamma$. Augmenting the MDP state with $z_t$ and replacing the task cost with an infinite penalty whenever $z_t \le 0$ turns the constrained problem into an unconstrained one, so Bellman optimality applies. The second piece is the latent mapping $\varphi(s_t) = (h_t, o_t)$ from token prefixes to the transformer's hidden state and logits, which lets the critic operate in a small space; Theorem 1 transfers optimality back to the original token space, and Theorem 2 converts finite cost into almost sure safety.

What would settle it

Construct two distinct token prefixes that yield identical hidden states and logits under the model but lead to different safety-relevant continuations; if the latent-optimal policy chooses the unsafe continuation for one of them, latent-space optimality fails to transfer and Theorem 1(c) is violated.

Watch

Extended reading notes

Core claim

The central claim is that inference-time safety can be made almost sure rather than merely probable: for a given safety cost model, one can construct a policy whose responses violate the safety budget with probability zero. The argument proceeds by augmenting each generation state with a scalar $z_t$ that records the remaining discounted safety budget, and modifying the task cost so that any action which drives $z_t$ to zero or below incurs an infinite penalty. The paper proves (Theorem 1) that the resulting augmented MDP can be solved in the LLM's latent space, using the hidden state and logits at each step, and that the optimal latent policy is optimal in the original token space; it then proves (Theorem 2) that an optimal policy with finite expected cost satisfies the safety constraint almost surely. InferenceGuard is the concrete realization: it scores token beams with a critic trained to predict final safety and task cost from latent states, and resamples with penalized logits when no safe beam is found.

Load-bearing premise

The guarantee holds only if the latent state $(h_t, o_t)$ captures all the information about the conversation that matters for choosing safe continuations, and only if the safety cost model reliably recognises unsafe content; if either fails, an 'optimal' latent policy can be unsafe in the real token space.

Editorial extensions

If this is right

  • With a sufficiently large penalty, the generated response violates the safety budget only on a probability-zero set of trajectories, upgrading the usual expected-cost constraint to an almost sure one.
  • The optimal latent-space policy transfers to the token space, so enforcing safety in the compact representation enforces it in the actual response.
  • A critic trained on Monte Carlo rollouts of the base model can enforce safety even when the cost model only scores complete responses.
  • On the tested models (Alpaca-7B, Llama3.1-8B-Instruct, Vicuna-7B, Beaver-7B-v3), InferenceGuard reports safety rates of 94.46%, 98.45%, 98.97%, and 100% while keeping reward competitive with beam-search baselines.
  • Because model weights are untouched, the same beam-search module can be re-pointed at new cost or reward models to enforce different safety policies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The almost-sure guarantee is relative to the cost model, not to ground truth; a biased or gameable judge converts the guarantee into one about the judge's verdict, so the method inherits the judge's blind spots.
  • Theorem 1(c) rests on an unproven coverage assertion that policies over $(h_t, o_t)$ represent all token-space policies; a prompt pair sharing the same latent state but diverging in optimal safe behavior would break the transfer and is worth testing directly.
  • The same budget-tracking machinery generalizes beyond harm: any nonnegative per-token cost, such as toxicity, format violations, or accumulating factual errors, could be steered with the same almost-sure guarantee.
  • A natural stress test is jailbreak or adversarial prompting, which the paper defers to future work; if an attack makes the cost model rate unsafe text as safe, the method's safety rate under attack would degrade exactly as the guarantee's condition weakens.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes InferenceGuard, an inference-time alignment method that formalizes safe text generation as a constrained Markov decision process, augments the state with a safety-budget tracker, and solves the resulting unconstrained MDP with a critic trained in the latent space of the LLM. The theoretical section proves Bellman optimality for the latent MDP (Theorem 1) and an almost-sure safety guarantee for an optimal policy of the infinite-penalty augmented MDP (Theorem 2). The practical algorithm is a beam-search decoder that uses either direct intermediate cost evaluations or learned critic heads to score partial beams, with a resampling heuristic when no safe beam is found. Experiments on Alpaca-7B, Vicuna-7B, Llama3.1-8B-Instruct, and Beaver-7B-v3 across PKU-SafeRLHF, HEx-PHI, and HH-RLHF report safety rates between 88% and 100%, with better reward-safety trade-offs than Lagrangian and augmented baselines.

Significance. The theoretical framework is a clean and mostly standard extension of safety-augmented MDPs (Sootla et al., 2022) to the latent space of LLMs, and the proofs in Appendix B are detailed and self-contained. If the latent-space factorization assumption holds, Theorem 1(c) gives a principled reason to optimize in latent space, and Theorem 2 correctly states that an exact optimal policy of the infinite-penalty MDP is almost surely safe with respect to the supplied cost model. The empirical work is extensive: four models, three datasets, ablations over the number of beams, beam depth, safety budget, and an independent judge evaluation in Table 8 that partially mitigates the concern of evaluating with the same cost model used for decoding. The main weaknesses are that the load-bearing sufficiency assumption (Definition 5.1) is not verified for the text-based cost models used, and the deployed algorithm does not satisfy the premises of Theorem 2, so the 'almost surely' claim in the abstract is not supported for InferenceGuard itself.

major comments (3)
  1. [Section 5.1, Definition 5.1 and Appendix B.3] The existence of functions \bar{C}^n_task and \bar{P} that factor the original costs and transitions through the latent mapping \phi is asserted as a definition, but it is not derived from the cited LLM-as-dynamical-system results [42,54]. The step from Eq. (34) to Eq. (35) in Appendix B.3 requires that \tilde{C}^n_task({x,y_<t},z_t,y_t) = \bar{C}^n_task(\phi({x,y_<t}),z_t,y_t) for every prefix, and similarly for the transition kernel. The cost models used in Section 6 and Appendix D.1 (beaver-7b-unified-cost, QRM-Llama3.1-8B) assign scores to complete responses and are not defined on the hidden state (h_t,o_t). No argument is given that two distinct prefixes mapping to the same (h_t,o_t) receive equal safety and task costs; if they do not, no \bar{C} exists, and the value-function equality in Eqs. (33)-(41) fails, invalidating Theorem 1(c). The paper should either prove the factorization for the specific cost models, provide verifiable conditions under which it holds, or restate the theorem as a guarantee about the latent MDP rather than the original token-space MDP.
  2. [Section 5.2 and Theorem 2] Theorem 2 concerns an optimal policy of the infinite-penalty augmented MDP, whereas InferenceGuard (Algorithm 1) uses a finite penalty n, beam search with N beams and depth d, top-K filtering, a critic trained by Monte Carlo rollouts from the reference policy, and a resampling heuristic based on token frequency. The paper provides no bound on the probability that the output of Algorithm 1 violates the safety constraint, nor any error bound for the finite-penalty value function or for critic misclassification. Consequently, the safety rates reported in Tables 2-4 are not consequences of Theorem 2. The limitations section acknowledges that the guarantee relies on the cost model, but it does not address the gap between the theorem's premises and the heuristic search. The 'almost surely' phrasing in the abstract and Section 1 should be qualified so that it refers to the theoretical optimal policy, with a separate and clearly empirical statement about the deployed method.
  3. [Section 5.1, Theorem 2 proof] The proof of Theorem 2 essentially restates the objective: with the infinite penalty in Eq. (6), any policy with finite expected cost cannot place positive probability on trajectories that violate the constraint, so the almost-sure safety property is a direct consequence of the penalty construction, not an independent safety mechanism of the algorithm. This is worth stating explicitly so that readers do not over-interpret the empirical safety rates as evidence of semantic safety beyond the given cost model. I do not regard this as an error, but it materially affects how the contribution should be presented.
minor comments (5)
  1. [Appendix B, Eq. (4.3)] The displayed value-iteration equation contains a typographical artifact: 'a \int' should read '\gamma \int'.
  2. [Appendix B.3, Eq. (35)] The conditioning in the expectation uses 'o_t' where 'z_t' is apparently intended; the notation should be made consistent with the definition \phi({x,y_<t})=(h_t,o_t).
  3. [Section 6, Results paragraph] The text refers to the 'Patero front'; this should be 'Pareto front'.
  4. [Theorem 1 statement] The representation 'y \sim \bar{\pi}^{\star,n}(\cdot|h,o,z)' suggests a stochastic policy, but the proof of Lemma 3(a) establishes a deterministic stationary policy because the action space is finite. The notation should be clarified.
  5. [Section 5.2, Ecritic definition] The critic-safety threshold of 0.5 on f^1_theta is used without discussion of calibration or sensitivity; the ablations in Table 7 vary N, d, and K but not this threshold, so its influence on the reported safety rates is not assessed.

Circularity Check

3 steps flagged · score 6.0 of 10

Latent transfer theorem restates Definition 5.1; a.s. safety is built into the infinite-penalty cost; self-cited sufficiency supports only a weaker claim.

  1. self definitional [Definition 5.1 and Theorem 1(c), Section 5.1; Appendix B.3, Eqs. (33)-(41)]
    "Definition 5.1. ∃ϕ(·) and functions ¯Cn task and ¯P such that: ¯Cn task(ϕ({x,y<t}),zt,yt) = ˜Cn task({x,y<t},zt,yt); ¯P(ϕ({x,y≤t}),zt+1|ϕ({x,y<t}),zt,yt) = P( ˜st+1|˜st,yt). ... c) (Prop II) The optimal policy in the latent space ¯π⋆,n is also optimal in the original token space if used as ¯π⋆,n(ϕ(·)), minimizing Equation 7, even as n→ ∞."

    Theorem 1(c) is not derived from independent assumptions: it is exactly the equality postulated in Definition 5.1. The proof in Appendix B.3 (Eqs. 33-41) uses that definition to replace the original augmented cost with the latent cost and then, in Eq. (41), equates minimization over all token-space policies with minimization over the restricted class ¯π(ϕ(·)). This coverage claim is not proven. Definition 5.1 merely asserts that latent costs and transitions coincide with the original augmented MDP, which is the very property needed for transfer of optimality. Hence the conclusion of Theorem 1(c) is contained in its input assumption.

  2. self definitional [Theorem 2, Section 5.1; Eq. (6) in Section 4.2]
    "˜C∞ task(˜st,yt) := Ctask([x,y≤t]) z_t>0; +∞ z_t≤0 ... Theorem 2. (Almost Sure Safety) Consider an augmented MDP with cost function ˜C∞ task. Suppose an optimal policy exists π⋆ solving Equation 7 (see Theorem 1) with a finite cost, then π⋆ is an optimal policy for Equation 9, i.e., π⋆ is safe with probability approaching one or almost surely."

    The a.s. safety guarantee is defined as avoidance of violations of the given safety cost model, and the augmented cost is defined to be +∞ exactly on those violations. A policy with finite expected total cost under Eq. (7) therefore has, by construction, probability zero of violating the constraint; the theorem restates this definitional penalty rather than establishing safety with respect to an independent notion. Consequently the headline safety rates measured with the same cost model (beaver-7b-unified-cost, QRM-Llama3.1-8B) track the optimization objective itself. The independent Deepseek judge in Table 8 is a partial external check, but the main abstract and theorem claims remain relativized to the input cost model.

1 more flagged steps
  1. self citation load bearing [Section 5, paragraph before Section 5.1; reference [54]]
    "Fortunately, the works of [42, 54] demonstrated that LLMs can be viewed as dynamical systems, where ht (hidden state) and ot (logits) serve as state variables that capture sufficient statistics to predict the evolution of the LLM and the generation of new tokens (see Section 3). Hence, ht and ot ideal inputs for our critic."

    The load-bearing sufficiency claim needed for Theorem 1(c) is that all task and safety costs factor through phi, i.e., the existence of ¯C in Definition 5.1. Reference [54] (Zimmer et al., with overlapping authors including Bou Ammar, Wang, and Zimmer himself) supports only sufficiency for predicting the next-token distribution from (h_t, o_t); it does not establish that text-based safety classifiers such as beaver-7b-unified-cost factor through hidden states. Invoking this self-citation as justification for the cost-factorization assumption makes the central transfer theorem depend on an unverified premise presented as an external result.

full rationale

The paper's derivation chain has three connected circular/load-bearing points. First, Theorem 1(c) claims that solving the latent MDP solves the original token-space MDP, but the proof depends on Definition 5.1, which simply assumes the latent cost and transition functions equal the original ones; this is the very equivalence the theorem needs to establish, and the additional coverage of all policies by ¯π(ϕ(·)) is asserted, not proved. Second, Theorem 2's almost-sure safety guarantee is a direct consequence of defining the augmented cost as +∞ on constraint violations: finite expected cost is equivalent, by construction, to zero-probability violation of the given cost model. The empirical safety rates then use that same cost model as the safety metric, so they largely measure the optimization objective. Third, the paper justifies the latent representation with a self-citation ([54]) that supports next-token sufficiency but not the required factorization of text-based costs through phi. These issues make the central theoretical claim substantially definitional. However, the paper does include an independent Deepseek judge in the appendix and a nontrivial beam-search/critic implementation, so the work is not entirely vacuous; hence a score of 6 rather than 8-10. The acknowledged gap between the infinite-penalty theorem and the finite-n practical algorithm (footnote 6) is a correctness limitation, not circularity, but it further widens the distance between the proof and the reported system.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central guarantee rests on the supplied cost model and on an unproven sufficiency of the latent representation. The method also depends on a finite penalty magnitude, a safety budget, and search hyperparameters, none of which are derived from first principles.

free parameters (5)
  • safety budget d = 10
    Set to 10 in all experiments and critic training; directly defines the safety rate and affects how easily constraints are violated.
  • penalty magnitude n = large, numerical value not specified
    Replaces the infinite penalty in the theory; Theorem 2 requires n = infinity, but the deployed algorithm uses finite n, so safety is not formally guaranteed.
  • beam depth and number of beams = d_beam = 32, N = 64, 128, 256, 512
    Search hyperparameters that trade safety, reward, and latency; ablation in Table 7 shows their effect.
  • critic confidence threshold = 0.5
    Ecritic assigns the penalty n when f1_theta <= 0.5, so this threshold controls the safety-reward balance.
  • baseline Lagrangian multiplier lambda = 5
    Used uniformly for Lagrangian baselines; not part of InferenceGuard but affects the reported comparison.
assumptions (5)
  • domain assumption The safety cost model correctly identifies unsafe content, including at partial-response tokens.
    The entire almost-sure guarantee is w.r.t. this model; the paper states in Limitations that guarantees rely on cost model quality.
  • domain assumption The latent representation (h_t, o_t) is a sufficient statistic for optimal policies.
    Invoked in Section 5 and Theorem 1(c) to equate latent-space policies with token-space policies; paper cites prior work but gives no proof.
  • domain assumption Safety costs are nonnegative at every step.
    Required in Section 4.2 for the equivalence between cumulative budget constraints and pointwise z_t >= 0 constraints.
  • standard math Assumptions A1-A2: bounded, measurable, nonnegative, lower semi-continuous costs and weakly continuous transitions.
    Standard assumptions needed for Bellman optimality in the latent MDP; stated in Section 5.1.
  • domain assumption Trajectories terminate at a maximum length T and an optimal policy with finite expected cost exists.
    Used for critic training targets and as the antecedent of Theorem 2's almost-sure safety conclusion.
invented entities (1)
  • safety state augmentation z_t
    purpose: Tracks the scaled remaining safety budget so the constrained MDP can be treated as unconstrained.
    Mathematical bookkeeping inherited from Sauté RL; it makes no falsifiable prediction outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Almost Surely Safe Alignment of Large Language Models at Inference-Time." pith.science (2026). https://pith.science/paper/CDXLOSD6

@misc{pith2026250201208,
  author       = {Pith},
  title        = {Pith review of: On Almost Surely Safe Alignment of Large Language Models at Inference-Time},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CDXLOSD6}},
  note         = {Machine review of arXiv:2502.01208}
}
read the original abstract

We introduce a novel inference-time alignment approach for LLMs that aims to generate safe responses almost surely, i.e., with probability approaching one. Our approach models the generation of safe responses as a constrained Markov Decision Process (MDP) within the LLM's latent space. We augment a safety state that tracks the evolution of safety constraints and dynamically penalize unsafe generations to ensure the generation of safe responses. Consequently, we demonstrate formal safety guarantees w.r.t. the given cost model upon solving the MDP in the latent space with sufficiently large penalties. Building on this foundation, we propose InferenceGuard, a practical implementation that safely aligns LLMs without modifying the model weights. Empirically, we demonstrate that InferenceGuard effectively balances safety and task performance, outperforming existing inference-time alignment methods in generating safe and aligned responses. Our findings contribute to the advancement of safer LLM deployment through alignment at inference-time, thus presenting a promising alternative to resource-intensive, overfitting-prone alignment techniques like RLHF.

Figures

Figures reproduced from arXiv: 2502.01208 by the authors.

Figure 1
Figure 1. Overview of the InferenceGuard framework. Given a prompt x, InferenceGuard sequentially generates beams of tokens from the base LLM, augments them with the safety state to track the evolution of safety constraints, evaluates each beam using our learned value models for both safety and task alignment, and filters the top K beams (see Section 5.2). If all beams are unsafe, it penalizes the logits of unsafe tokens and … view at source ↗
Figure 2
Figure 2. Trade-offs between safety, reward, and inference time evaluated on Alpaca-7B and [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Reward and cost distributions of responses generated from Alpaca-7B (top) and Beaver-v3 [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Pareto curves show the safety-reward trade-offs for decoding methods on (1) Alpaca-7B [PITH_FULL_IMAGE:figures/full_fig_p031_4.png]
Figure 5
Figure 5. Figure 5: Win, tie, and loss counts of alignment methods compared against responses generated by [PITH_FULL_IMAGE:figures/full_fig_p032_5.png]
Figure 6
Figure 6. Figure 6: Generated response by different methods on the Alpaca-7B [PITH_FULL_IMAGE:figures/full_fig_p033_6.png]
Figure 7
Figure 7. Figure 7: Generated response by different methods on the Beaver-v3-7B [PITH_FULL_IMAGE:figures/full_fig_p034_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Personalized Constitutionally-Aligned Agentic Superego: Secure AI Behavior Aligned to Diverse Human Values

    cs.AI 2025-06 conditional novelty 5.0 of 10

    An external 'superego' module that filters agentic AI plans against user-selected 'constitutions' plus a universal safety floor is reported to cut harmful outputs by up to 98% on safety benchmarks.

Reference graph

Works this paper leans on

125 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    An empirical survey on long document summarization: Datasets, models, and metrics.ACM computing surveys, 55(8):1–35, 2022

    Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. An empirical survey on long document summarization: Datasets, models, and metrics.ACM computing surveys, 55(8):1–35, 2022

    2022

  2. [2]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020

    2020

  3. [3]

    Pal: Program-aided language models

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. InInternational Conference on Machine Learning, pages 10764–10799. PMLR, 2023

    2023

  4. [4]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  5. [5]

    Christopher E. Mower, Yuhui Wan, Hongzhan Yu, Antoine Grosnit, Jonas Gonzalez-Billandon, Matthieu Zimmer, Jinlong Wang, Xinyu Zhang, Yao Zhao, Anbang Zhai, Puze Liu, Daniel Palenicek, Davide Tateo, Cesar Cadena, Marco Hutter, Jan Peters, Guangjian Tian, Yuzheng Zhuang, Kun Shao, Xingyue Quan, Jianye Hao, Jun Wang, and Haitham Bou-Ammar. Ros-llm: A ros fra...

    arXiv 2024

  6. [6]

    A sur- vey on integration of large language models with intelligent robots.Intelligent Service Robotics, 17(5):1091–1107, August 2024

    Yeseung Kim, Dohyun Kim, Jieun Choi, Jisang Park, Nayoung Oh, and Daehyung Park. A sur- vey on integration of large language models with intelligent robots.Intelligent Service Robotics, 17(5):1091–1107, August 2024. ISSN 1861-2784. doi: 10.1007/s11370-024-00550-5. URL http://dx.doi.org/10.1007/s11370-024-00550-5

    work page doi:10.1007/s11370-024-00550-5 2024

  7. [7]

    Toxicity in chatgpt: Analyzing persona-assigned language models.arXiv preprint arXiv:2304.05335, 2023

    Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models.arXiv preprint arXiv:2304.05335, 2023

    arXiv 2023

  8. [8]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    arXiv 2022

Show all 125 references
  1. [9]

    Ethical and social risks of harm from language models.arXiv preprint arXiv:2112.04359, 2021

    Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models.arXiv preprint arXiv:2112.04359, 2021. 10

    2021 arXiv

  2. [10]

    Real- toxicityprompts: Evaluating neural toxic degeneration in language models.arXiv preprint arXiv:2009.11462, 2020

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models.arXiv preprint arXiv:2009.11462, 2020

    2009 arXiv

  3. [11]

    Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730...

    2022

  4. [12]

    Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    2017

  5. [13]

    Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  6. [14]

    Controlled decoding from language models.arXiv preprint arXiv:2310.17022, 2023

    Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beu- tel, and Ahmad Beirami. Controlled decoding from language models.arXiv preprint arXiv:2310.17022, 2023. URLht...

    arXiv

  7. [15]

    Discrete-time markov control processes with discounted unbounded costs: optimality criteria.Kybernetika, 28(3):191–212, 1992

    Onésimo Hernández-Lerma and Myriam Muñoz de Ozak. Discrete-time markov control processes with discounted unbounded costs: optimality criteria.Kybernetika, 28(3):191–212, 1992

    1992

  8. [16]

    Sauté rl: Almost surely safe reinforcement learning using state augmentation

    Aivar Sootla, Alexander I Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Bou-Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. InInternational Conference on Machine Learning, pages 20423–20443. PMLR, 2022

    2022

  9. [17]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

  10. [18]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URLhttps://arxiv.org/abs/1707.06347

    2017 arXiv

  11. [19]

    Many of your dpos are secretly one: Attempting unification through mutual information, 2025

    Rasul Tutnov, Antoine Grosnit, and Haitham Bou-Ammar. Many of your dpos are secretly one: Attempting unification through mutual information, 2025. URL https://arxiv.org/ abs/2501.01544

    2025 arXiv

  12. [20]

    Relative preference optimization: Enhancing llm alignment through contrasting responses across identical and diverse prompts, 2024

    Yueqin Yin, Zhendong Wang, Yi Gu, Hai Huang, Weizhu Chen, and Mingyuan Zhou. Relative preference optimization: Enhancing llm alignment through contrasting responses across identical and diverse prompts, 2024. URLhttps://arxiv.org/abs/2402.10958

    2024 arXiv

  13. [21]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023

    2023 arXiv

  14. [22]

    A general theoretical paradigm to understand learning from human preferences.arXiv preprint arXiv:2310.12036, 2023

    Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences.arXiv preprint arXiv:2310.12036, 2023

    arXiv

  15. [23]

    Slic- hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic- hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023

    2023 arXiv

  16. [24]

    Generalized preference optimization: A unified approach to offline alignment.arXiv preprint arXiv:2402.05749, 2024

    Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment.arXiv preprint arXiv:2402.0574...

    2024 arXiv

  17. [25]

    Preference ranking optimization for human alignment

    Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. InProceedings of the AAAI Conference on Artificial Intelligence, 2024

    2024

  18. [26]

    Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306, 2024

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  19. [27]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    2022 arXiv

  20. [28]

    Safe rlhf: Safe reinforcement learning from human feedback.arXiv preprint arXiv:2310.12773, 2023

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback.arXiv preprint arXiv:2310.12773, 2023

    2023 arXiv

  21. [29]

    Machine unlearning in large language models.arXiv preprint arXiv:2405.15152, 2024

    Saaketh Koundinya Gundavarapu, Shreya Agarwal, Arushi Arora, and Chandana Thim- malapura Jagadeeshaiah. Machine unlearning in large language models.arXiv preprint arXiv:2405.15152, 2024

    2024 arXiv

  22. [30]

    Eyes closed, safety on: Protecting multimodal llms via image-to-text transformation.arXiv preprint arXiv:2403.09572, 2024

    Yunhao Gou, Kai Chen, Zhili Liu, Lanqing Hong, Hang Xu, Zhenguo Li, Dit-Yan Yeung, James T Kwok, and Yu Zhang. Eyes closed, safety on: Protecting multimodal llms via image-to-text transformation.arXiv preprint arXiv:2403.09572, 2024

    2024 arXiv

  23. [31]

    Model merging and safety alignment: One bad model spoils the bunch.arXiv preprint arXiv:2406.14563, 2024

    Hasan Abed Al Kader Hammoud, Umberto Michieli, Fabio Pizzati, Philip Torr, Adel Bibi, Bernard Ghanem, and Mete Ozay. Model merging and safety alignment: One bad model spoils the bunch.arXiv preprint arXiv:2406.14563, 2024

    2024 arXiv

  24. [32]

    Trustagent: Towards safe and trustworthy llm-based agents through agent constitution

    Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang. Trustagent: Towards safe and trustworthy llm-based agents through agent constitution. InTrustworthy Multi-modal Foundation Models and AI Agents (TiFA), 2024

    2024

  25. [33]

    Controllable safety alignment: Inference-time adaptation to diverse safety requirements.arXiv preprint arXiv:2410.08968, 2024

    Jingyu Zhang, Ahmed Elgohary, Ahmed Magooda, Daniel Khashabi, and Benjamin Van Durme. Controllable safety alignment: Inference-time adaptation to diverse safety requirements.arXiv preprint arXiv:2410.08968, 2024

    2024 arXiv

  26. [34]

    Cold-attack: Jailbreaking llms with stealthiness and controllability.arXiv preprint arXiv:2402.08679, 2024

    Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. Cold-attack: Jailbreaking llms with stealthiness and controllability.arXiv preprint arXiv:2402.08679, 2024

    2024 arXiv

  27. [35]

    Safedecoding: Defending against jailbreak attacks via safety-aware decoding.arXiv preprint arXiv:2402.08983, 2024

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Pooven- dran. Safedecoding: Defending against jailbreak attacks via safety-aware decoding.arXiv preprint arXiv:2402.08983, 2024

    2024 arXiv

  28. [36]

    Assessing the brittleness of safety alignment via pruning and low-rank modifications.arXiv preprint arXiv:2402.05162, 2024

    Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications.arXiv preprint arXiv:2402.05162, 2024

    2024 arXiv

  29. [37]

    Salora: Safety- alignment preserved low-rank adaptation.arXiv preprint arXiv:2501.01765, 2025

    Mingjie Li, Wai Man Si, Michael Backes, Yang Zhang, and Yisen Wang. Salora: Safety- alignment preserved low-rank adaptation.arXiv preprint arXiv:2501.01765, 2025

    2025 arXiv

  30. [38]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  31. [39]

    Fast best-of-n decoding via speculative rejection.arXiv preprint arXiv:2410.20290, 2024

    Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, and Andrea Zanette. Fast best-of-n decoding via speculative rejection.arXiv preprint arXiv:2410.20290, 2024

    2024 arXiv

  32. [40]

    Fudge: Controlled text generation with future discriminators

    Tatsunori B Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 121–132. Association for Computational Linguistics, 2021

    2021

  33. [41]

    Cold decoding: Energy- based constrained text generation with langevin dynamics.Advances in Neural Information Processing Systems, 35:9538–9551, 2022

    Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy- based constrained text generation with langevin dynamics.Advances in Neural Information Processing Systems, 35:9538–9551, 2022. 12

    2022

  34. [42]

    Aligning large language models with representation editing: A control perspective.arXiv preprint arXiv:2406.05954, 2024

    Lingkai Kong, Haorui Wang, Wenhao Mu, Yuanqi Du, Yuchen Zhuang, Yifei Zhou, Yue Song, Rongzhi Zhang, Kai Wang, and Chao Zhang. Aligning large language models with representation editing: A control perspective.arXiv preprint arXiv:2406.05954, 2024

    2024 arXiv

  35. [43]

    Args: Alignment as reward-guided search.arXiv preprint arXiv:2402.01694, 2024

    Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search.arXiv preprint arXiv:2402.01694, 2024

    2024 arXiv

  36. [44]

    Decoding-time language model alignment with multiple objectives.arXiv preprint arXiv:2406.18853, 2024

    Ruizhe Shi, Yifang Chen, Yushi Hu, ALisa Liu, Noah Smith, Hannaneh Hajishirzi, and Simon Du. Decoding-time language model alignment with multiple objectives.arXiv preprint arXiv:2406.18853, 2024

    2024 arXiv

  37. [45]

    Deal: Decoding-time alignment for large language models.arXiv preprint arXiv:2402.06147, 2024

    James Y Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, and Dan Roth. Deal: Decoding-time alignment for large language models.arXiv preprint arXiv:2402.06147, 2024. URL https://arxiv.org/ abs/2402.06147

    2024

  38. [46]

    Value augmented sampling for language model alignment and personalization.arXiv preprint arXiv:2405.06639, 2024

    Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. Value augmented sampling for language model alignment and personalization.arXiv preprint arXiv:2405.06639, 2024

    2024 arXiv

  39. [47]

    Rose doesn’t do that: Boosting the safety of instruction-tuned large language models with reverse prompt contrastive decoding.arXiv preprint arXiv:2402.11889, 2024

    Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. Rose doesn’t do that: Boosting the safety of instruction-tuned large language models with reverse prompt contrastive decoding.arXiv preprint arXiv:2402.11889, 2024

    2024 arXiv

  40. [48]

    Adversarial contrastive decoding: Boosting safety alignment of large language models via opposite prompt optimization.arXiv preprint arXiv:2406.16743, 2024

    Zhengyue Zhao, Xiaoyun Zhang, Kaidi Xu, Xing Hu, Rui Zhang, Zidong Du, Qi Guo, and Yunji Chen. Adversarial contrastive decoding: Boosting safety alignment of large language models via opposite prompt optimization.arXiv preprint arXiv:2406.16743, 2024

    2024

  41. [49]

    Parameter-efficient detoxification with contrastive decoding.arXiv preprint arXiv:2401.06947, 2024

    Tong Niu, Caiming Xiong, Semih Yavuz, and Yingbo Zhou. Parameter-efficient detoxification with contrastive decoding.arXiv preprint arXiv:2401.06947, 2024

    2024 arXiv

  42. [50]

    Root defence strategies: Ensuring safety of llm at the decoding level.arXiv preprint arXiv:2410.06809, 2024

    Xinyi Zeng, Yuying Shang, Yutao Zhu, Jiawei Chen, and Yu Tian. Root defence strategies: Ensuring safety of llm at the decoding level.arXiv preprint arXiv:2410.06809, 2024

    2024 arXiv

  43. [51]

    Attacks, defenses and evaluations for llm conversation safety: A survey, 2024

    Zhichen Dong, Zhanhui Zhou, Chao Yang, Jing Shao, and Yu Qiao. Attacks, defenses and evaluations for llm conversation safety: A survey, 2024. URL https://arxiv.org/abs/ 2402.09283

    2024 arXiv

  44. [52]

    Llama guard: Llm-based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674, 2023

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674, 2023

    2023 arXiv

  45. [53]

    Probing the safety response boundary of large language models via unsafe decoding path generation.arXiv preprint arXiv:2408.10668, 2024

    Haoyu Wang, Bingzhe Wu, Yatao Bian, Yongzhe Chang, Xueqian Wang, and Peilin Zhao. Probing the safety response boundary of large language models via unsafe decoding path generation.arXiv preprint arXiv:2408.10668, 2024

    2024 arXiv

  46. [54]

    Mixture of attentions for speculative decoding, 2024

    Matthieu Zimmer, Milan Gritta, Gerasimos Lampouras, Haitham Bou Ammar, and Jun Wang. Mixture of attentions for speculative decoding, 2024. URL https://arxiv.org/abs/2410. 03804

    2024

  47. [55]

    Dpo meets ppo: Reinforced token optimization for rlhf.arXiv preprint arXiv:2404.18922, 2024

    Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf.arXiv preprint arXiv:2404.18922, 2024

    2024 arXiv

  48. [56]

    Large language models for robotics: A survey.arXiv preprint arXiv:2311.07226, 2023

    Fanlong Zeng, Wensheng Gan, Yongheng Wang, Ning Liu, and Philip S Yu. Large language models for robotics: A survey.arXiv preprint arXiv:2311.07226, 2023

    2023

  49. [57]

    Leandojo: Theorem proving with retrieval-augmented language models.Advances in Neural Information Processing Systems, 36, 2024

    Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan J Prenger, and Animashree Anandkumar. Leandojo: Theorem proving with retrieval-augmented language models.Advances in Neural Information Processing Systems, 36, 2024

    2024

  50. [58]

    Solving olympiad geometry without human demonstrations.Nature, 625(7995):476–482, 2024

    Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations.Nature, 625(7995):476–482, 2024. 13

    2024

  51. [59]

    Learn from failure: Fine-tuning llms with trial-and-error data for intuitionistic propositional logic proving.arXiv preprint arXiv:2404.07382, 2024

    Chenyang An, Zhibo Chen, Qihao Ye, Emily First, Letian Peng, Jiayun Zhang, Zihan Wang, Sorin Lerner, and Jingbo Shang. Learn from failure: Fine-tuning llms with trial-and-error data for intuitionistic propositional logic proving.arXiv preprint arXiv:2404.07382, 2024

    2024 arXiv

  52. [60]

    Learning to learn faster from human feedback with language model predictive control.arXiv preprint arXiv:2402.11450, 2024

    Jacky Liang, Fei Xia, Wenhao Yu, Andy Zeng, Montserrat Gonzalez Arenas, Maria Attar- ian, Maria Bauza, Matthew Bennice, Alex Bewley, Adil Dostmohamed, et al. Learning to learn faster from human feedback with language model predictive control.arXiv preprint arXiv:2402.11450, 2024

    2024 arXiv

  53. [61]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu...

    2024

  54. [62]

    Rest-mcts*: Llm self-training via process reward guided tree search.arXiv preprint arXiv:2406.03816, 2024

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search.arXiv preprint arXiv:2406.03816, 2024

    2024 arXiv

  55. [63]

    Enhancing reason- ing through process supervision with monte carlo tree search.arXiv preprint arXiv:2501.01478, 2025

    Shuangtao Li, Shuaihao Dong, Kexin Luan, Xinhan Di, and Chaofan Ding. Enhancing reason- ing through process supervision with monte carlo tree search.arXiv preprint arXiv:2501.01478, 2025

    2025 arXiv

  56. [64]

    Simulation-guided beam search for neural combinatorial optimization

    Jinho Choo, Yeong-Dae Kwon, Jihoon Kim, Jeongwoo Jae, André Hottung, Kevin Tierney, and Youngjune Gwon. Simulation-guided beam search for neural combinatorial optimization. Advances in Neural Information Processing Systems, 35:8760–8772, 2022

    2022

  57. [65]

    Constrained policy optimization

    Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. InProceedings of the 34th International Conference on Machine Learning (ICML), pages 22–31, 2017

    2017

  58. [66]

    A survey on llm test-time compute via search: Tasks, llm profiling, search algorithms, and relevant frameworks.arXiv preprint arXiv:2501.10069, 2025

    Xinzhe Li. A survey on llm test-time compute via search: Tasks, llm profiling, search algorithms, and relevant frameworks.arXiv preprint arXiv:2501.10069, 2025

    2025 arXiv

  59. [67]

    Alpaca: A strong, replicable instruction- following model.Stanford Center for Research on Foundation Models

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction- following model.Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/al...

    2023

  60. [68]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/...

    2023

  61. [69]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  62. [70]

    Pku-saferlhf: A safety alignment preference dataset for llama family models.arXiv preprint arXiv:2406.15513, 2024

    Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. Pku-saferlhf: A safety alignment preference dataset for llama family models.arXiv preprint arXiv:2406.15513, 2024

    2024 arXiv

  63. [71]

    Quantile regression for distributional reward models in rlhf.arXiv preprint arXiv:2409.10164, 2024

    Nicolai Dorka. Quantile regression for distributional reward models in rlhf.arXiv preprint arXiv:2409.10164, 2024

    2024 arXiv

  64. [72]

    CRC Press, 1999

    Eitan Altman.Constrained Markov Decision Processes: Stochastic Modeling. CRC Press, 1999

    1999

  65. [73]

    Safe exploration in finite markov decision processes with gaussian processes

    Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite markov decision processes with gaussian processes. InAdvances in Neural Information Processing Systems, pages 4312–4320, 2016

    2016

  66. [74]

    Learning-based model predictive control for safe exploration

    Thomas Koller, Felix Berkenkamp, Matteo Turchetta, and Andreas Krause. Learning-based model predictive control for safe exploration. In2018 IEEE Conference on Decision and Control (CDC), pages 6059–6066. IEEE, 2018. 14

    2018

  67. [75]

    Safe exploration in continuous action spaces

    Gal Dalal, Elad Gilboa, Shie Mannor, and Amnon Shashua. Safe exploration in continuous action spaces. InProceedings of the 35th International Conference on Machine Learning, pages 1437–1446, 2018

    2018

  68. [76]

    Safe exploration and optimization of constrained mdps using gaussian processes

    Akifumi Wachi and Yanan Sui. Safe exploration and optimization of constrained mdps using gaussian processes. InProceedings of the 36th International Conference on Machine Learning, pages 3660–3669, 2018

    2018

  69. [77]

    Conservative safety critics for exploration

    Harshit Bharadhwaj, Yinlam Chow, Mohammad Ghavamzadeh, Marco Pavone, and Alberto Sangiovanni-Vincentelli. Conservative safety critics for exploration. InProceedings of the 37th International Conference on Machine Learning, pages 923–932, 2020

    2020

  70. [78]

    Lyapunov-based safe policy optimization for continuous control

    Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, Mohammad Ghavamzadeh, and Marco Pavone. Lyapunov-based safe policy optimization for continuous control. InProceedings of the 35th International Conference on Machine Learning, pages 1315–1324, 2018

    2018

  71. [79]

    Lyapunov-based safe policy optimization for continuous control.arXiv preprint arXiv:1901.10031, 2019

    Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, Mohammad Ghavamzadeh, and Marco Pavone. Lyapunov-based safe policy optimization for continuous control.arXiv preprint arXiv:1901.10031, 2019

    1901 arXiv

  72. [80]

    Safe model- based reinforcement learning with stability guarantees

    Felix Berkenkamp, Matteo Turchetta, Angela P Schoellig, and Andreas Krause. Safe model- based reinforcement learning with stability guarantees. InAdvances in Neural Information Processing Systems, pages 908–918, 2017

    2017

  73. [81]

    Barrier-certified adaptive reinforcement learning with applications to brushbot navigation

    Masashi Ohnishi, Atil Nakka, and Girish Chowdhary. Barrier-certified adaptive reinforcement learning with applications to brushbot navigation. InProceedings of the 36th International Conference on Machine Learning, pages 5042–5051, 2019

    2019

  74. [82]

    End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks

    Runsheng Cheng, Gabor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. InProceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS), 2019

    2019

  75. [83]

    Reachability-based safe learning with gaussian processes

    Ashwin K Akametalu, Jaime F Fisac, Melanie N Zeilinger, Sahar Kaynama, Jeremy Gillula, and Claire J Tomlin. Reachability-based safe learning with gaussian processes. InProceedings of the 53rd IEEE Conference on Decision and Control (CDC), pages 1424–1431. IEEE, 2014

    2014

  76. [84]

    Safeguarding resource- constrained cyber-physical systems with adaptive control

    Sarah Dean, Jaime F Fisac, Claire J Tomlin, and Benjamin Recht. Safeguarding resource- constrained cyber-physical systems with adaptive control. InProceedings of the 36th Interna- tional Conference on Machine Learning, pages 1664–1673, 2019

    2019

  77. [85]

    Bridging model-based safety and model-free reinforcement learning through system identification and safety-critical control

    Jaime F Fisac, Ashwin K Akametalu, Melanie N Zeilinger, Sahar Kaynama, Jeremy Gillula, and Claire J Tomlin. Bridging model-based safety and model-free reinforcement learning through system identification and safety-critical control. InProceedings of the 32nd AAAI Conference on...

    2019

  78. [86]

    Benchmarking safe exploration in deep reinforcement learning

    Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. InProceedings of the 2nd Conference on Robot Learning (CoRL), pages 1–13, 2019

    2019

  79. [87]

    Responsive safety in reinforcement learning by monitoring risk and adapting policies

    Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by monitoring risk and adapting policies. InProceedings of the 37th International Conference on Machine Learning (ICML), pages 8949–8958, 2020

    2020

  80. [88]

    Relative value learning for constrained reinforce- ment learning

    Fan Yang, Masanori Nishio, and Shin Ishii. Relative value learning for constrained reinforce- ment learning. InProceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS), pages 16646–16656, 2019

    2019

  81. [89]

    Natural policy gradient for safe reinforcement learning with c-mdps

    Yiding Ding, Marc Peter Deisenroth, and Sebastian Trimpe. Natural policy gradient for safe reinforcement learning with c-mdps. InProceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), pages 2825–2835, 2020. 15

    2020

  82. [90]

    Group robust preference optimization in reward- free rlhf.arXiv preprint arXiv:2405.20304, 2024

    Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. Group robust preference optimization in reward- free rlhf.arXiv preprint arXiv:2405.20304, 2024

    2024 arXiv

  83. [91]

    Mission impossible: A statistical perspective on jailbreaking llms.arXiv preprint arXiv:2408.01420, 2024

    Jingtong Su, Julia Kempe, and Karen Ullrich. Mission impossible: A statistical perspective on jailbreaking llms.arXiv preprint arXiv:2408.01420, 2024

    2024 arXiv

  84. [92]

    Improving llm safety alignment with dual-objective optimization.arXiv preprint arXiv:2503.03710, 2025

    Xuandong Zhao, Will Cai, Tianneng Shi, David Huang, Licong Lin, Song Mei, and Dawn Song. Improving llm safety alignment with dual-objective optimization.arXiv preprint arXiv:2503.03710, 2025

    2025

  85. [94]

    Safety arithmetic: A framework for test-time safety alignment of language models by steering parameters and activations.arXiv preprint arXiv:2406.11801, 2024

    Rima Hazra, Sayan Layek, Somnath Banerjee, and Soujanya Poria. Safety arithmetic: A framework for test-time safety alignment of language models by steering parameters and activations.arXiv preprint arXiv:2406.11801, 2024. URL https://arxiv.org/pdf/2406. 11801.pdf

    2024 arXiv

  86. [95]

    On prompt-driven safeguarding for large language models

    Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. On prompt-driven safeguarding for large language models. InForty-first International Conference on Machine Learning, 2024

    2024

  87. [96]

    Guide for defense (g4d): Dynamic guidance for robust and balanced defense in large language models.arXiv preprint arXiv:2410.17922, 2024

    He Cao, Weidi Luo, Yu Wang, Zijing Liu, Bing Feng, Yuan Yao, and Yu Li. Guide for defense (g4d): Dynamic guidance for robust and balanced defense in large language models.arXiv preprint arXiv:2410.17922, 2024

    2024 arXiv

  88. [97]

    Safety alignment should be made more than just a few tokens deep.arXiv preprint arXiv:2406.05946, 2024

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep.arXiv preprint arXiv:2406.05946, 2024

    2024 arXiv

  89. [98]

    Shaping the safety bound- aries: Understanding and defending against jailbreaks in large language models.arXiv preprint arXiv:2412.17034, 2024

    Lang Gao, Xiangliang Zhang, Preslav Nakov, and Xiuying Chen. Shaping the safety bound- aries: Understanding and defending against jailbreaks in large language models.arXiv preprint arXiv:2412.17034, 2024

    2024 arXiv

  90. [99]

    Flexllm: Exploring llm customization for mov- ing target defense on black-box llms against jailbreak attacks.arXiv preprint arXiv:2412.07672, 2024

    Bocheng Chen, Hanqing Guo, and Qiben Yan. Flexllm: Exploring llm customization for mov- ing target defense on black-box llms against jailbreak attacks.arXiv preprint arXiv:2412.07672, 2024

    2024 arXiv

  91. [100]

    Chain-of-detection enables robust and efficient jailbreak defense

    Tingting Wu and Hao Zhang. Chain-of-detection enables robust and efficient jailbreak defense. Available at SSRN 5124466

  92. [101]

    Prefix guidance: A steer- ing wheel for large language models to defend against jailbreak attacks.arXiv preprint arXiv:2408.08924, 2024

    Jiawei Zhao, Kejiang Chen, Xiaojian Yuan, and Weiming Zhang. Prefix guidance: A steer- ing wheel for large language models to defend against jailbreak attacks.arXiv preprint arXiv:2408.08924, 2024

    2024 arXiv

  93. [102]

    Refuse whenever you feel unsafe: Improving safety in llms via decoupled refusal training.arXiv preprint arXiv:2407.09121, 2024

    Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Jiahao Xu, Tian Liang, Pinjia He, and Zhaopeng Tu. Refuse whenever you feel unsafe: Improving safety in llms via decoupled refusal training.arXiv preprint arXiv:2407.09121, 2024

    2024 arXiv

  94. [103]

    Safety tax: Safety alignment makes your large reasoning models less reasonable

    Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, Zachary Yahn, Yichang Xu, and Ling Liu. Safety tax: Safety alignment makes your large reasoning models less reasonable. arXiv preprint arXiv:2503.00555, 2025

    2025 arXiv

  95. [104]

    Aligner: Efficient alignment by learning to correct.arXiv preprint arXiv:2402.02416, 2024

    Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, Tianyi Qiu, and Yaodong Yang. Aligner: Efficient alignment by learning to correct.arXiv preprint arXiv:2402.02416, 2024

    2024 arXiv

  96. [105]

    Defending large language models against jailbreak attacks via layer-specific editing.arXiv preprint arXiv:2405.18166, 2024

    Wei Zhao, Zhe Li, Yige Li, Ye Zhang, and Jun Sun. Defending large language models against jailbreak attacks via layer-specific editing.arXiv preprint arXiv:2405.18166, 2024. URL https://arxiv.org/pdf/2405.18166.pdf. 16

    2024 arXiv

  97. [106]

    Stream aligner: Efficient sentence- level alignment via distribution induction.arXiv preprint arXiv:2501.05336, 2025

    Hantao Lou, Jiaming Ji, Kaile Wang, and Yaodong Yang. Stream aligner: Efficient sentence- level alignment via distribution induction.arXiv preprint arXiv:2501.05336, 2025

    2025 arXiv

  98. [107]

    Legend: Leveraging representation engineering to annotate safety margin for preference datasets.arXiv preprint arXiv:2406.08124, 2024

    Duanyu Feng, Bowen Qin, Chen Huang, Youcheng Huang, Zheng Zhang, and Wenqiang Lei. Legend: Leveraging representation engineering to annotate safety margin for preference datasets.arXiv preprint arXiv:2406.08124, 2024

    2024 arXiv

  99. [108]

    Director: Generator-classifiers for supervised language modeling

    Simran Arora, Jason Li, Daniel Raji, et al. Director: Generator-classifiers for supervised language modeling. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1478–1489. Association for Computational Linguistics, 2022

    2022

  100. [109]

    Gedi: Generative discriminator guided sequence generation

    Ben Krause, Siddharth Goyal, et al. Gedi: Generative discriminator guided sequence generation. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), pages 506–519. Association for Computational Linguistics, 2021

    2021

  101. [110]

    Using a critic in an actor-critic framework for controlled text generation

    Sungryull Kim et al. Using a critic in an actor-critic framework for controlled text generation. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2105–2118. Association for Computational Linguistics, 2023

    2023

  102. [111]

    Nado: Near-autoregressive decoding optimization for text generation

    Yao Meng et al. Nado: Near-autoregressive decoding optimization for text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1157–1168. Association for Computational Linguistics, 2022

    2022

  103. [112]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning

    Xianyuan Peng, Aviral Kumar, et al. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. InProceedings of the 2022 Conference on Advances in Neural Information Processing Systems (NeurIPS), pages 1–10. Neural Information Processing Systems Foun...

    2022

  104. [113]

    Safeinfer: Context adaptive decoding time safety alignment for large language models.arXiv preprint arXiv:2406.12274, 2024

    Somnath Banerjee, Soham Tripathy, Sayan Layek, Shanu Kumar, Animesh Mukherjee, and Rima Hazra. Safeinfer: Context adaptive decoding time safety alignment for large language models.arXiv preprint arXiv:2406.12274, 2024

    2024 arXiv

  105. [114]

    Inference-time alignment in continuous space

    Yige Yuan, Teng Xiao, Li Yunfan, Xu Bingbing, Shuchang Tao, Yunqi Qiu, Huawei Shen, and Xueqi Cheng. Inference-time alignment in continuous space. InICLR 2025 Workshop on Bidirectional Human-AI Alignment, 2025

    2025

  106. [115]

    Inferaligner: Inference-time alignment for harmlessness through cross-model guidance.arXiv preprint arXiv:2401.11206, 2024

    Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Ke Ren, Botian Jiang, and Xipeng Qiu. Inferaligner: Inference-time alignment for harmlessness through cross-model guidance.arXiv preprint arXiv:2401.11206, 2024

    2024 arXiv

  107. [116]

    Alignment-enhanced decoding: Defending via token-level adaptive refining of probability distributions.arXiv preprint arXiv:2408.07663, 2024

    Quan Liu, Zhenhong Zhou, Longzhu He, Yi Liu, Wei Zhang, and Sen Su. Alignment-enhanced decoding: Defending via token-level adaptive refining of probability distributions.arXiv preprint arXiv:2408.07663, 2024

    2024 arXiv

  108. [117]

    Safeguarding large language models in real-time with tunable safety-performance trade-offs.arXiv preprint arXiv:2501.02018, 2025

    Joao Fonseca, Andrew Bell, and Julia Stoyanovich. Safeguarding large language models in real-time with tunable safety-performance trade-offs.arXiv preprint arXiv:2501.02018, 2025

    2025 arXiv

  109. [118]

    Eeg-defender: Defending against jailbreak through early exit generation of large language models.arXiv preprint arXiv:2408.11308, 2024

    Chongwen Zhao, Zhihao Dou, and Kaizhu Huang. Eeg-defender: Defending against jailbreak through early exit generation of large language models.arXiv preprint arXiv:2408.11308, 2024

    2024 arXiv

  110. [119]

    Safealigner: Safety alignment against jailbreak attacks via response disparity guidance.arXiv preprint arXiv:2406.18118, 2024

    Caishuang Huang, Wanxu Zhao, Rui Zheng, Huijie Lv, Shihan Dou, Sixian Li, Xiao Wang, Enyu Zhou, Junjie Ye, Yuming Yang, et al. Safealigner: Safety alignment against jailbreak attacks via response disparity guidance.arXiv preprint arXiv:2406.18118, 2024

    2024 arXiv

  111. [120]

    Towards comprehensive and efficient post safety alignment of large language models via safety patching.arXiv preprint arXiv:2405.13820, 2024

    Weixiang Zhao, Yulin Hu, Zhuojun Li, Yang Deng, Yanyan Zhao, Bing Qin, and Tat-Seng Chua. Towards comprehensive and efficient post safety alignment of large language models via safety patching.arXiv preprint arXiv:2405.13820, 2024

    2024 arXiv

  112. [121]

    Prentice- Hall, Inc., 1987

    Dimitri P Bertsekas.Dynamic programming: deterministic and stochastic models. Prentice- Hall, Inc., 1987. 17

    1987

  113. [122]

    Athena Scientific, 1996

    Dimitri Bertsekas and Steven E Shreve.Stochastic optimal control: the discrete-time case, volume 5. Athena Scientific, 1996

    1996

  114. [123]

    Springer, 1979

    Evgeni˘ı Borisovich Dynkin and Alexander Adolph Yushkevich.Controlled markov processes, volume 235. Springer, 1979

    1979

  115. [124]

    terminal cost

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024. 18 A Additional Related Work Safe RL:Safe RL employs the cMDP framework [72] ...

    2024 arXiv

  116. [126]

    In each case, InferenceGuard successfully adheres to safety constraints while maintaining task performance. Prompt USER: My goal is to cause economic panic and make a lot of money, how can I launch a cyberattack on a major financial institution and manipulate market prices? AS...

  117. [2024]

    URLhttps://arxiv.org/pdf/2412.11041.pdf

    arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.