REVIEW 4 major objections 6 minor 38 references
Safety Alignment Depth in Large Language Models: A Markov Chain Perspective
T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read The paper claims that repeated fine-tuning can make an LLM's refusal behavior permanent, with a closed-form step count for when that happens, and that wider ensembles can substitute for deeper alignment.
desk verdict Interesting framing undercut by an invalid central theorem and an unsupported bridge to real LLMs. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the discounted additive update $Q_t = Q_0 + \alpha\gamma^t B$, proposed in Assumption 3.1 as the model of fine-tuning: each step adds a bias matrix $B$, scaled by learning rate $\alpha$ and geometrically discounted by $\gamma^t$, where the refusal state $r$ receives positive self-bias $B(r,r)>0$ and negative escape-bias $B(r,s)<0$. The proofs hinge on the geometric-series identity $\sum_{t=0}^T \gamma^t = (1-\gamma^{T+1})/(1-\gamma)$, whose tail $\gamma^{T+1}/(1-\gamma) \leq \delta$ yields the explicit step threshold; the refusal state behaves as a near-absorbing state once the diagonal bias dominates its row. Assumptions 3.1 and 3.4, that fine-tuning equals this matrix update and that group-augmented fine-tuning equals conjugating $B$ by a permutation matrix, are the bridge from the toy chain to real LLM training, and the paper's own limitations section acknowledges that the analysis operates at a behavior level that may not cover every training dynamic.
What would settle it
Estimate a real model's refusal-state row from token-generation statistics during the paper's fine-tuning recipe: if the escape probability $1 - Q_t(r,r)$ does not decay geometrically and drop below $\delta$ at the predicted threshold $T > \log(\delta(1-\gamma))/\log(\gamma) - 1$, or if the refusal row of the fitted empirical chain fails to satisfy $|Q_T(r,r)-1| \leq \delta$, then the toy bound is not what actual training does; the cheapest check is a replication on a small model such as Gemma 2B under the paper's published settings.
Extended reading notes
Core claim
The central claim is Theorem 4.5 (the $\delta$-absorbing theorem): in a Markov chain updated as $Q_t = Q_0 + \alpha\gamma^t B$, if the bias matrix satisfies $B(r,r)>0$ and $B(r,s)<0$ at a refusal state $r$, then for any tolerance $\delta$ the state $r$ becomes $\delta$-absorbing once the training steps exceed $T > \log(\delta(1-\gamma))/\log(\gamma) - 1$ and the learning rate exceeds $\max_{r,s}|Q_0(r,s)|(1-\gamma)/\min(B(r,r), -B(r,s))$; at that point $|Q_T(r,r)-1| \leq \delta$ and $|Q_T(r,s)| \leq \delta$ for every non-refusal state $s$. Corollary 4.7 turns this into a per-state threshold for the largest safety depth $r^*$ that becomes absorbing given a chosen pair $(\alpha, T)$. Proposition 4.8 shows the guarantee survives time-varying bias of the form $B(t) = P^t B P^{-t}$ coming from a cyclic permutation, with the step bound governed by the permutation order $\mathrm{ord}(P)$. Proposition 4.11 distributes an overall safety level $\varepsilon$ across an ensemble of width $W$: per-model risk must be at most $\varepsilon/W$ under union aggregation, $\varepsilon\tau$ under averaging with threshold $\tau$, or $\tfrac12 - \sqrt{\ln(1/\varepsilon)/(2W)}$ under majority voting, so broadening the ensemble relaxes each model's training requirement.
Load-bearing premise
The load-bearing premise is that fine-tuning an LLM really is the simple discounted additive update $Q_t = Q_0 + \alpha\gamma^t B$, and that rotating the training data is equivalent to conjugating that bias matrix; the paper asserts these identifications without deriving them from gradient dynamics, and it states in its own limitations section that it treats LLMs at a behavior level that may not cover every case.
Editorial extensions
If this is right
- Safety depth becomes a computed quantity rather than a tuning guess: given the discount $\gamma$, tolerance $\delta$, initial matrix $Q_0$, bias $B$, and learning rate $\alpha$, the required step count follows from a closed-form log bound, and the deepest refusal state that is guaranteed $\delta$-absorbing is $r^* = \max\{r \in R \mid \alpha > \alpha_r,\ T > T_r\}$.
- Cyclic-group data augmentation, which rotates the refusal phrase through training positions, preserves the $\delta$-absorbing guarantee under a periodically varying bias and can converge in fewer steps, with the bound capped by the permutation order $\mathrm{ord}(P)$.
- Ensemble width trades against alignment depth: $W$ models each trained to per-model risk $\varepsilon/W$ (union), $\varepsilon\tau$ (average), or $\tfrac12 - \sqrt{\ln(1/\varepsilon)/(2W)}$ (majority voting) collectively meet the $\varepsilon$-safety threshold, reducing the training burden on any single model.
- Experiments on Gemma 2B, Phi-2 2B, and Qwen 2.5 1.5B report cyclic augmentation raising mean safety scores, with Gemma rising from 0.42 (shallow) and 0.46 (deep) to 0.61 (cyclic), and ensembles of three shallow models scoring higher and less variably than a single deep model.
Reading between the lines
- A natural testable extension the paper does not run: use the closed-form step bound to predict when a real model's refusal survives adversarial jailbreak prompts, since the Markov abstraction drops attention heads and positional structure that may control where refusal generalizes.
- The ensemble results imply a resource trade-off the paper leaves unpriced: union aggregation requires $W$ models trained to $\varepsilon/W$, roughly $W$ times the compute of one deep model, so majority voting is the only strategy that clearly makes widening cheaper than deepening; an explicit cost comparison would settle which strategy practitioners should prefer.
- Because Proposition 4.8 only needs the bias sequence to be bounded and periodic under permutation conjugation, the guarantee should carry over to any finite group of dataset permutations, not just the cyclic rotations the paper instantiates.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper models autoregressive LLMs as Markov chains and claims a first theoretical result on safety alignment depth: Theorem 4.5 states that if the transition matrix is updated by adding a discounted bias αγ^t B with B(r,r)>0 and B(r,s)<0, then after T > log(δ(1-γ))/log(γ)-1 steps every refusal state r is δ-absorbing. It extends this to cyclic-group data augmentation (Proposition 4.8) and to ensemble aggregation (Proposition 4.11), and reports toy Markov-chain experiments plus fine-tuning of Gemma, Phi-2, and Qwen with shallow, deep, and cyclic augmentation. The paper itself acknowledges in Remark 4.6, Appendix F, and Appendix J that the raw updates can leave [0,1], that a normalization is applied in the experiments, and that the empirical study is proof-of-concept.
Significance. If Theorem 4.5 were correct, the paper would give a clean, falsifiable training-time formula for safety depth and a novel depth-width tradeoff for ensembles, and the toy experiments would be a useful sanity check. The breadth of the claim, presented as the first theoretical result on this question, makes correctness load-bearing. Unfortunately the central theorem is invalid as stated: the proof uses a cumulative update that contradicts the displayed update rule, the stated lower bound on α does not imply the claimed δ-absorbing bounds, and the normalization added afterward is outside the theorem. The cyclic and ensemble results inherit the same defect because they rely on the same update dynamics. The conceptual framing of refusal states as absorbing, and the ensemble aggregation formulas themselves, have some merit, but they do not compensate for the missing central derivation.
major comments (4)
- [Section 4, Theorem 4.5, Eq. (1) vs. Eq. (14)] The theorem's update rule is Qt = Q0 + αγ^t B, so QT(r,r) would be Q0(r,r) + αγ^T B(r,r); the proof instead writes QT(r,r) = Q0(r,r) + α Σ_{i=0}^T γ^i B(r,r). These are different processes. If Eq. (1) is intended literally, Eq. (14) is simply wrong; if the intended update is cumulative, that must be stated and the displayed Qt must be changed. As written, the proof does not prove the theorem.
- [Section 4, Theorem 4.5, worked example] Even under the cumulative interpretation, the claimed δ-absorbing bounds fail on the paper's own numbers. For Q0(r,r)=0.7, Q0(r,s)=0.3, B(r,r)=1, B(r,s)=-1, α=0.081, γ=0.9, T=65, the cumulative formulas yield Q_T(r,r)≈1.509 and Q_T(r,s)≈-0.509, so |Q_T(r,r)-1|≈0.509 and |Q_T(r,s)|≈0.509, far exceeding δ=0.01, and the row does not sum to 1. Appendix G, Step 2 shows that centering the row requires the exact equality α=(1-Q0(r,r))(1-γ)/B(r,r)=0.03, not the lower bound α>0.08 in Eq. (12). Therefore Eq. (12) is insufficient and the theorem's guarantee is not derived.
- [Section 4, Remark 4.6 and Appendix F, Algorithm 1] The paper responds to the out-of-range values by clipping and normalizing rows, and Remark 4.6 says this normalization is used in all numerical experiments. This step is not part of Theorem 4.5, and no proof is given that the clipped and normalized matrix satisfies the δ-absorbing bounds in Eq. (13), remains stochastic, or corresponds to a meaningful training process. Appendix F states only that the behavior is 'satisfied in practice.' Since the theorem's conclusion concerns QT itself, this post-hoc normalization cannot patch the proof.
- [Section 3, Assumptions 3.1 and 3.4] The bridge from the toy Markov chain to LLM fine-tuning is assumed rather than established: Assumption 3.1 asserts that fine-tuning is equivalent to the iterative transition-matrix update, and Assumption 3.4 asserts that group-augmented fine-tuning is equivalent to conjugating the bias matrix B. No derivation, reference, or empirical check supports these equivalences, and Appendix J concedes that the analysis is behavior-level and proof-of-concept. Even if the toy theorem were correct, the LLM safety-depth conclusions would rest on unvalidated assumptions.
minor comments (6)
- [Section 4, Theorem 4.5 statement] The phrase 'laerning rate' should be 'learning rate'.
- [Appendix F, Algorithm 1] The sign conventions in lines 4-6 are inconsistent: eQT(i,s) is defined with a leading minus sign and Q+ is defined as -max{0,eQT(i,s)}, so the algorithm as printed does not compute the clipped raw update described in the surrounding text.
- [Appendix G, G.3 and G.4] The proof headings refer to 'Theorem 4.8' and 'Theorem 4.10', but the statements being proved are Proposition 4.8 and Corollary 4.10.
- [Section 4, Example of Proposition 4.8] The example reports Q3(r,r)=1.0 and Q3(r,s)=0.0 exactly after three steps, but with α=0.081 and γ=0.9 the raw cumulative update gives Q3(r,r)≈0.819 and Q3(r,s)≈0.181; the displayed exact values can only come from the unstated normalization of Remark 4.6, which is not part of Proposition 4.8.
- [Section 5, Numerical Experiments] The toy experiments set α=γ=1, but Theorem 4.5 requires γ∈(0,1) for the geometric-series argument and the logarithmic bound on T, so the toy experiment does not instantiate the theorem's assumptions.
- [Appendix J, Limitations] The limitations section appropriately concedes that the paper treats the LLM at a behavior level and shows only proof-of-concept experiments; this is candid, but it also confirms that the LLM experiments do not close the gap created by Assumptions 3.1 and 3.4.
Circularity Check
Theorem 4.5's δ-absorbing guarantee is built into its inputs: B's sign pattern, Assumption 4.4's pre-absorbed refusal states, and the proof's α calibration all force the conclusion; the LLM bridge is assumed, not derived.
-
self definitional
[Section 3, Eq. (1), and Theorem 4.5, Eqs. (11)-(16)]
"Hence, the transition matrix at time t is given by: Qt = Q0 + α γt B. (1) ... B is a bias matrix with B(r, r) > 0 and B(r, s) < 0 for refusal states r and s ̸= r."
Eq. (1) defines Qt = Q0 + α γ^t B, and Theorem 4.5 premises B(r,r)>0 and B(r,s)<0. Those signs are exactly the conclusion: every update adds mass to (r,r) and removes mass from (r,s). The proof's Eq. (14) reveals the 'derivation' is the cumulative sum QT(r,r)=Q0(r,r)+αB(r,r)Σγ^i, which tends to 1 by construction; the T bound only states geometric-series convergence. Thus δ-absorbing is a restatement of the update rule, not an independent mechanism or theorem about alignment.
-
self definitional
[Section 4, Assumption 4.4]
"Assumption 4.4 (Refusal is absorbing). If a state s includes the refusal token, then Qt(s, s′) = 0 for all s′ ̸= s and Qt(s, s) = 1."
Assumption 4.4 already declares every refusal-token state absorbing at every time t: Qt(s,s')=0 for s'≠s and Qt(s,s)=1. Theorem 4.5's target—refusal states become δ-absorbing after training—is therefore either already assumed or incompatible with Eq. (1), because B(r,r)>0 would perturb an already-absorbing row. The safety property is placed into the model by definition before any training argument, so the theorem cannot provide an independent guarantee.
2 more flagged steps
-
other
[Appendix G, Step 2, Eqs. (64)-(66)]
"If we choose α so that Q0(r, r) + α B(r, r)/(1 − γ) = 1 ⇐⇒ α = (1 − Q0(r, r))(1 − γ)/B(r, r), then as T grows ... we get |QT (r, r) − 1| ≤ δ."
To prove |QT(r,r)−1|≤δ, the proof calibrates α to the exact value solving Q0(r,r)+αB(r,r)/(1−γ)=1, i.e., it fits the learning rate to the desired conclusion. The theorem's stated lower bound Eq. (12) does not imply the bound; with the paper's own example α=0.081 and T=65, one gets QT(r,r)≈1.51 and QT(r,s)≈−0.51, violating Eq. (13). Remark 4.6 then adds clipping and normalization outside the theorem. The predicted absorption is manufactured by parameter selection and post-hoc projection, not derived under the stated hypotheses.
-
other
[Section 3, Assumptions 3.1 and 3.4]
"Assumption 3.1. Fine-tuning LLM is equivalent to an iterative update of the transition matrix Qt. ... Assumption 3.4. Fine-tuning LLM on Taug is equivalent to matrix conjugate operating on a bias matrix."
These assumptions assert, without derivation, that real LLM fine-tuning and group-augmented fine-tuning follow the paper's toy linear-geometric update and conjugate-bias dynamics. The cited Zekri et al. result shows representability of a fixed autoregressive policy as a Markov chain, not that gradient training obeys this specific additive update. All subsequent 'LLM safety depth' theorems therefore conclude exactly what was assumed about the training dynamics; the theoretical result does not constrain real fine-tuning unless the assumption is true, and it is neither derived nor externally validated.
full rationale
The paper contains no load-bearing self-citations: its Markov-chain equivalence citation (Zekri et al.) is external, and the LLM experiments are independent empirical evaluations. However, the central theoretical guarantee is circular in a more direct sense. Safety is inserted into the model by construction: Eq. (1) plus the sign pattern B(r,r)>0 and B(r,s)<0 makes 'refusal becomes absorbing' a restatement of the update rule, and Assumption 4.4 already defines refusal-token states as absorbing before any training. The Appendix G proof confirms this by solving for the exact α that forces QT(r,r)→1, while the theorem's stated lower bound and worked example produce a non-stochastic matrix (QT(r,r)≈1.51, QT(r,s)≈−0.51) that Remark 4.6 must repair with normalization outside the theorem. The bridge to actual LLMs is also assumed rather than derived: Assumptions 3.1 and 3.4 assert that fine-tuning and group augmentation match the toy dynamics, so the 'theoretical result on ideal safety depth' applies only to the assumed model. The ensemble propositions (union, average, majority) are standard concentration bounds and are not themselves circular, but they inherit the same assumed single-chain input. Overall, the main safety-depth theorem reduces by definition to its own assumptions, giving a circularity score of 8 rather than 10 only because the toy Monte Carlo computations and the empirical augmentation comparisons have some independent content.
Assumptions & free parameters
free parameters (6)
- α (learning rate) =
0.081 in toy example; lower bound in Eq. (12)
- γ (discount factor) =
0.9 in examples; 1 in Figure 3
- B (bias matrix) =
B(r,r)=1, B(r,s)=-1 in examples
- δ (target precision) =
0.01 in examples
- ε (safety threshold) =
0.1 in ensemble experiments
- r and refusal-state set R =
r=1 in toy example; refusal phrase position in LLM data
assumptions (4)
- ad hoc to paper Assumption 3.1: Fine-tuning an LLM is equivalent to iteratively updating the transition matrix Q_t=Q_0+αγ^t B.
- ad hoc to paper Assumption 3.4: Fine-tuning on the group-augmented set is equivalent to conjugating the bias matrix (B(t)=P^tBP^{-t}).
- domain assumption Assumption 4.4: Refusal is absorbing: Q_t(s,s)=1 for any state s containing the refusal token.
- domain assumption Autoregressive LLMs are equivalently representable as Markov chains (Zekri et al., 2024).
invented entities (2)
-
safety depth r (refusal state index)
-
δ-absorbing
Cite this review
Pith. "Pith review of Safety Alignment Depth in Large Language Models: A Markov Chain Perspective." pith.science (2026). https://pith.science/paper/2CDQNFFI
@misc{pith2026250200669,
author = {Pith},
title = {Pith review of: Safety Alignment Depth in Large Language Models: A Markov Chain Perspective},
year = {2026},
howpublished = {\url{https://pith.science/paper/2CDQNFFI}},
note = {Machine review of arXiv:2502.00669}
}
read the original abstract
Large Language Models (LLMs) are increasingly adopted in high-stakes scenarios, yet their safety mechanisms often remain fragile. Simple jailbreak prompts or even benign fine-tuning can bypass these protocols, underscoring the need to understand where and how they fail. Recent findings suggest that vulnerabilities emerge when alignment is confined to only the initial output tokens. Unfortunately, even with the introduction of deep safety alignment, determining the optimal safety depth remains an unresolved challenge. By leveraging the equivalence between autoregressive language models and Markov chains, this paper offers the first theoretical result on how to identify the ideal depth for safety alignment, and demonstrates how permutation-based data augmentation can tighten these bounds. Crucially, we reveal a fundamental interaction between alignment depth and ensemble width-indicating that broader ensembles can compensate for shallower alignments. These insights provide a theoretical foundation for designing more robust, scalable safety strategies that complement existing alignment approaches, opening new avenues for research into safer, more reliable LLMs.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023
arXiv 2023
-
[3]
Jailbreaking leading safety-aligned llms with simple adaptive attacks
Andriushchenko, M., Croce, F., and Flammarion, N. Jailbreaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151, 2024
arXiv 2024
-
[4]
Training a helpful and harmless assistant with reinforcement learning from human feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022
arXiv 2022
-
[5]
Boucheron, S., Lugosi, G., and Bousquet, O. Concentration inequalities. In Summer school on machine learning, pp.\ 208--240. Springer, 2003
work page 2003
-
[6]
A., Jagielski, M., Gao, I., Koh, P
Carlini, N., Nasr, M., Choquette-Choo, C. A., Jagielski, M., Gao, I., Koh, P. W. W., Ippolito, D., Tramer, F., and Schmidt, L. Are aligned neural networks adversarially aligned? Advances in Neural Information Processing Systems, 36, 2024
2024
-
[7]
Chang, F.-C. and Wu, P.-Y. Unraveling arithmetic in large language models: The role of algebraic structures. arXiv preprint arXiv:2411.16260, 2024
arXiv 2024
-
[8]
Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., and Wong, E. Jailbreaking Black Box Large Language Models in Twenty Queries . October 2023. URL https://openreview.net/forum?id=hkjcdmz8Ro
work page 2023
Show all 38 references
-
[9]
Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models
Duan, J., Cheng, H., Wang, S., Zavalny, A., Wang, C., Xu, R., Kailkhura, B., and Xu, K. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Annual Meeting of the Association for Computational Linguistics, 20...
2023
-
[10]
Dummit, D. S. and Foote, R. M. Abstract algebra, volume 3. Wiley Hoboken, 2004
2004
-
[11]
Kto: Model alignment as prospect theoretic optimization
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. ICML, 2024
2024
-
[12]
Gerken, J. E. and Kessel, P. Emergent equivariance in deep ensembles. ICML, 2024
2024
-
[13]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025
2025 arXiv
-
[14]
Safety arithmetic: A framework for test-time safety alignment of language models by steering parameters and activations
Hazra, R., Layek, S., Banerjee, S., and Poria, S. Safety arithmetic: A framework for test-time safety alignment of language models by steering parameters and activations. arXiv preprint arXiv:2406.11801, 2024
2024 arXiv
-
[15]
Catastrophic jailbreak of open-source llms via exploiting generation
Huang, Y., Gupta, S., Xia, M., Li, K., and Chen, D. Catastrophic jailbreak of open-source llms via exploiting generation. ICLR, 2024
2024
-
[16]
and Palangi, H
Imani, S. and Palangi, H. Exploring group and symmetry principles in large language models. arXiv preprint arXiv:2402.06120, 2024
2024 arXiv
-
[17]
Neural tangent kernel: Convergence and generalization in neural networks
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018
2018
-
[18]
D., and Ryu, E
Jang, U., Lee, J. D., and Ryu, E. K. Lora training in the ntk regime has no spurious local minima. arXiv preprint arXiv:2402.11867, 2024
2024 arXiv
-
[19]
Javaheripi, M., Bubeck, S., Abdin, M., Aneja, J., Bubeck, S., Mendes, C. C. T., Chen, W., Del Giorno, A., Eldan, R., Gopi, S., et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023
2023
-
[20]
Decoupling noise and toxic parameters for language model detoxification by task vector merging
Kim, Y., Kojima, T., Iwasawa, Y., and Matsuo, Y. Decoupling noise and toxic parameters for language model detoxification by task vector merging. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=TBNYjdOazs
2024
-
[21]
Safety layers in aligned large language models: The key to llm security
Li, S., Yao, L., Zhang, L., and Li, Y. Safety layers in aligned large language models: The key to llm security. arXiv preprint arXiv:2408.17003, 2024
2024 arXiv
-
[22]
A kernel-based view of language model fine-tuning
Malladi, S., Wettig, A., Yu, D., Chen, D., and Arora, S. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, pp.\ 23610--23641. PMLR, 2023
2023
-
[23]
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022
2022
-
[24]
Visual adversarial examples jailbreak aligned large language models
Qi, X., Huang, K., Panda, A., Wang, M., and Mittal, P. Visual adversarial examples jailbreak aligned large language models. In AAAI Conference on Artificial Intelligence, 2023 a . URL https://api.semanticscholar.org/CorpusID:259244034
2023
-
[25]
Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023 b
Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023 b
2023 arXiv
-
[26]
Safety alignment should be made more than just a few tokens deep, 2024
Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., and Henderson, P. Safety alignment should be made more than just a few tokens deep, 2024. URL https://arxiv.org/abs/2406.05946
2024 arXiv
-
[27]
D., Ermon, S., and Finn, C
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024
2024
-
[28]
Serre, J.-P. et al. Linear representations of finite groups, volume 42. Springer, 1977
1977
-
[29]
Deepseekmath: Pushing the limits of mathematical reasoning in open language models
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024
2024 arXiv
-
[30]
M., Hauth, A., et al
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023
2023 arXiv
-
[31]
and Sato, I
Tomihari, A. and Sato, I. Understanding linear probing then fine-tuning language models from ntk perspective. arXiv preprint arXiv:2405.16747, 2024
2024 arXiv
-
[32]
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N....
2023 arXiv
-
[33]
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024
2024 arXiv
-
[34]
On the vulnerability of safety alignment in open-access llms
Yi, J., Ye, R., Chen, Q., Zhu, B., Chen, S., Lian, D., Sun, G., Xie, X., and Wu, F. On the vulnerability of safety alignment in open-access llms. In Findings of the Association for Computational Linguistics ACL 2024, pp.\ 9236--9260, 2024
2024
-
[35]
Large language models as markov chains
Zekri, O., Odonnat, A., Benechehab, A., Bleistein, L., Boull \'e , N., and Redko, I. Large language models as markov chains. arXiv preprint arXiv:2410.02724, 2024
2024 arXiv
-
[36]
Removing rlhf protections in gpt-4 via fine-tuning
Zhan, Q., Fang, R., Bindu, R., Gupta, A., Hashimoto, T., and Kang, D. Removing rlhf protections in gpt-4 via fine-tuning. arXiv preprint arXiv:2311.05553, 2023
2023 arXiv
-
[37]
Towards comprehensive and efficient post safety alignment of large language models via safety patching, 2024
Zhao, W., Hu, Y., Li, Z., Deng, Y., Zhao, Y., Qin, B., Chua, T.-S., and Liu, T. Towards comprehensive and efficient post safety alignment of large language models via safety patching, 2024. URL https://openreview.net/forum?id=09JVxsEZPf
2024
-
[38]
Emulated disalignment: Safety alignment for large language models may backfire! arXiv preprint arXiv:2402.12343, 2024
Zhou, Z., Liu, J., Dong, Z., Liu, J., Yang, C., Ouyang, W., and Qiao, Y. Emulated disalignment: Safety alignment for large language models may backfire! arXiv preprint arXiv:2402.12343, 2024
2024 arXiv
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.