REVIEW 4 major objections 5 minor 4 cited by
Safety Reasoning with Guidelines
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Refusal-trained LLMs already know how to be safe; training them to reason through explicit guidelines unlocks that knowledge and roughly halves out-of-distribution jailbreak success.
desk verdict A useful and cheap safety-training recipe with large reported gains, but the paper's central diagnostic claim that refusal-trained models have adequate latent safety knowledge is an artifact of its any-safe Best-of-N rule. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two mechanisms carry the argument. The first is Best-of-N (BoN) evaluation: per query the model samples N responses, a guard classifier judges each, and the attack is scored unsuccessful if at least one response is safe, converting "occasionally safe" into a measurable scaling curve that supplies the evidence of latent knowledge. The second is the Safety Reasoning with Guidelines (SRG) pipeline: an expert model is prompted with a guideline block $C$, reasoning patterns, self-reflection via <reflection> tags, per-step safety scores via <safety degree> tags, and a closing self-refinement step that ties the final refusal to the lowest score, to write a reasoning trace and final answer; rejection sampling keeps only guideline-adherent, refusal-behaving traces; and context distillation trains the student on $(x, y^c)$ with $C$ removed from the input so the reasoning must be internalized rather than read. The diagnostic lens is a classic domain-adaptation bound, $\epsilon_{\mathrm{test}}(M_\theta) \leq \epsilon_{\mathrm{train}}(M_\theta) + d_{M_\theta}(p_{\mathrm{train}}, p_{\mathrm{test}}) + \lambda$, which attributes RT's OOD failure to the representation-divergence term $d$ and SRG's success to shrinking it.
What would settle it
Probe the same refusal-trained models without sampling: take a single response per OOD query at temperature zero, and separately read the hidden states at the layers where harmfulness is represented, using the same RepE-style linear method the paper uses for visualization, on ID attacks, OOD attacks, and helpful queries. If OOD attack activations cannot be separated from helpful ones above chance while the BoN improvement persists, the "adequate latent knowledge" conclusion is a sampling artifact; if they separate cleanly, the diagnosis holds. Independently, the SRG claim would be settled by holding out a genuinely novel OOD attack type, one not used in training, evaluation, or the guideline design, and checking whether SRG's greedy-decoding advantage over RT persists on it.
Extended reading notes
Core claim
The paper's central claim is that the failure of Refusal Training (RT) against out-of-distribution jailbreaking attacks is an elicitation failure rather than a capability failure: the model carries adequate safety knowledge from pretraining, yet the deliberately simple "refuse directly" supervision trains it to lean on surface shortcuts that break when an attack is paraphrased, written in the past tense, hidden in cipher, or wrapped in persuasion. The supporting evidence is Best-of-N evaluation, where sampling N responses per query and scoring the attack as unsuccessful whenever any one response is safe drives average attack success down by roughly 60% at N=128, with Jailbreak Chat and Self Cipher falling below 10%, and where enlarging the training set improves the trend further. From a domain-adaptation perspective the paper attributes this to an under-specified supervision signal: a large representation-space divergence between in-distribution and OOD attacks under RT, visible in PCA of internal activations, whereas representation-controlling defenses (LAT and RR) align the two. On that diagnosis it constructs Safety Reasoning with Guidelines (SRG): a teacher model generates step-by-step reasoning traces under explicit guidelines, including self-reflection tags and per-step safety scores; rejection sampling filters the traces; and context distillation trains the student on the original query alone, forcing it to internalize the reasoning. The result is that SRG-trained models lower greedy-decoding OOD attack success by 46.3% (8B) and 54.8% (70B) relative to RT, reach near-zero ASR on several attacks, and can accept a fresh guideline at inference time, cutting PAIR attack success from 58% to 4% on the 70B model.
Load-bearing premise
The load-bearing premise is that the Best-of-N rule, an attack fails if any one of the N sampled answers is judged safe, is a faithful probe of latent safety knowledge; under that rule the attack-success rate is forced to fall as N grows for any model that can ever answer safely, so the claim that "the knowledge is there" inherits the definition rather than resting on an independent test.
Editorial extensions
If this is right
- Guidelines become a runtime control surface: adding one new guideline at inference time, without retraining, drops PAIR attack success from 68% to 14% on the 8B model and from 58% to 4% on the 70B model.
- SRG's OOD gains are not bought with helpfulness: average scores on coding (HumanEval, MBPP), math (GSM8K, MATH), and tool-use (BFCL) benchmarks match or exceed RT, and over-refusal on XSTest stays near the instruct-model level at 92%.
- The method stacks with representation-level defenses: combining SRG with RR beats each alone on every evaluated attack.
- BoN still helps SRG, reaching about 0% ASR at N=64 on all attacks, which the paper reads as headroom for reinforcement learning: with reliable reward signals, sampling gains could be converted into single-sample performance, with the guidelines serving as verification criteria.
- The recipe transfers across base models (Gemma-2-9B, Mistral-7B, Qwen2-7B) and across teachers (GPT-4o, Qwen2.5-72B-Instruct, DeepSeek-R1-Qwen-32B), with richer guidelines and stronger teachers giving larger safety gains.
Reading between the lines
- The BoN evidence for "latent knowledge" is partly definitional: because an attack counts as failed whenever any of N responses is safe, ASR must decay with N for any model that is ever safe; the diagnosis would be stronger if the knowledge claim were probed directly, for instance by linearly classifying hidden states on OOD inputs or by measuring single-response safety under attacks designed to pr
- The guideline mechanism hints at test-time alignment as a general pattern: if a deployment-time guideline can patch behavior for PAIR attacks without retraining, guidelines are effectively a human-readable interface for updating safety policy, which could matter more for long-lived deployed models than for training pipelines.
- A useful stress test would vary the teacher: the paper's results show that the DeepSeek-R1 teacher yields worse safety data than GPT-4o, so the approach's ceiling is set by the teacher's own safety reasoning; testing whether a guideline-reasoning loop can bootstrap a weak teacher's safety data would settle how far the method scales without a strong proprietary teacher.
- The domain-adaptation diagnosis is illustrated qualitatively with PCA; quantifying the divergence term $d_{M_\theta}$ with a measurable distance between ID-attack, OOD-attack, and helpful representations would make the shortcut-learning story directly testable and could predict which new attacks a given model will resist before evaluating them.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper challenges the assumption that OOD jailbreak attacks inherently exceed the capability of vanilla Refusal Training (RT). Through Best-of-N (BoN) sampling, the authors observe that RT models' ASR drops substantially as the number of samples increases, and they interpret this as evidence that RT models possess adequate latent safety knowledge that RT fails to elicit consistently. A domain-adaptation analysis with RepE visualizations argues that RT relies on superficial shortcuts causing non-generalizable representations. Building on this, the authors propose Safety Reasoning with Guidelines (SRG), a training framework that synthesizes step-by-step reasoning supervision aligned with explicit safety guidelines, filters responses via rejection sampling, and distills the reasoning through Context Distillation. Experiments on LLaMA-3.1-8B and 70B across multiple attacks and dataset scales report large greedy-decoding ASR reductions over RT (average 46.3% and 54.8% on the two model sizes), with maintained helpfulness on code, math, and tool benchmarks.
Significance. If the empirical claims hold, SRG provides a simple and effective recipe for improving OOD safety generalization of LLMs, and the paper's BoN-based diagnostic is a thought-provoking way to analyze refusal training. The experiments are unusually broad: they cover two model sizes, four dataset scales, multiple base models, multiple attack types, helpfulness benchmarks, and an over-refusal test, and they include a detailed appendix with guidelines and further ablations. The method is clearly specified and the main greedy-decoding improvements over RT are large and consistent. However, the central diagnostic claim about latent safety knowledge is weakened by the definitional nature of the BoN metric, and a few missing controls and reporting gaps currently limit the strength of the causal story.
major comments (4)
- [Sec. 3.1, Fig. 1] The inference that RT models possess "adequate latent safety knowledge" is substantially definitional. Under the paper's any-safe criterion, an attack is unsuccessful if at least one of N samples is judged safe, so for any model with a fixed nonzero safe-response probability p, the ASR at sample size N equals (1-p)^N and must decay monotonically to zero as N grows. The observed monotone decrease in Figure 1 is therefore guaranteed by the evaluation rule and cannot by itself discriminate between a model that genuinely 'knows' safety and one that occasionally stumbles into a safe response. Please add an independent probe of latent knowledge (e.g., representation-based analysis with quantitative separation, or a direct-prompting control that elicits safe responses without search) to support the claim, or explicitly restate the conclusion to acknowledge that BoN measures the existence of safe responses rather than their accessibility under ordinary decoding.
- [Sec. 5.1, Appendix A.1] The manuscript states "We also carry out necessary human evaluation" but no human evaluation results are reported in the main text or the appendix. Since all reported ASR numbers rely on Llama-Guard-3-8B as the judge, the validity of the headline improvements depends on showing that the classifier's judgments agree with human judgments. Please either include the human evaluation protocol and quantitative agreement with Llama-Guard, or remove the claim that human evaluation was carried out.
- [Table 5, Sec. 5.2] The comparison with state-of-the-art methods shows that SRG is substantially worse than both LAT and RR on PastTense and Persuasive Attack (e.g., 26% vs. 4% and 2% on Persuasive Attack for 8B models). The text says the reasons "have been discussed in previous evaluation," but no such discussion appears. This discrepancy weakens the general claim that SRG "significantly improves model generalization against OOD attacks" relative to prior methods. Please either provide a quantitative explanation for the gap, or reframe the claims to make clear that SRG is primarily an improvement over RT and is complementary to LAT/RR, and support the complementarity claim with more than the single SRG+RR result in Appendix C.3.
- [Sec. 4, Sec. 5.2] The SRG training data contains long reasoning traces, while the RT baseline is trained on short refusals. The reported improvements of SRG over RT could therefore be partly due to the additional training tokens (or the different token-level compute) rather than to the guideline-based reasoning itself. Please add a control experiment that trains RT on extended refusal explanations of comparable length (without the structured reasoning tags) or trains SRG with guidelines but without the reasoning structure, to isolate the effect of the proposed reasoning supervision.
minor comments (5)
- [Table 3, Table 7] The word "Table" is misspelled as "Tabel" in the captions of Tables 3 and 7.
- [Sec. 3.2] The domain-adaptation bound uses notation d_{M_θ}(p_train, p_test) but does not specify the divergence measure; the bound is presented as an informal analogy, but the notation should be defined or the paragraph should clarify that it is illustrative.
- [Fig. 2, Appendix D] The RepE/PCA analysis is qualitative and the claim that SRG better aligns OOD and ID representations is asserted from visual inspection of plots; please report a quantitative separation metric (e.g., distance between cluster centroids or a classifier-based alignment score) to support this claim.
- [Sec. 4, Fig. 4] The safety-degree thresholds (0.8, 0.5) are presented without justification; a brief sensitivity analysis or a citation to related work would clarify how these thresholds were chosen.
- [Table 4] In the PAIR evaluation, adding the 'role awareness' guideline at inference also improves the RT baseline (e.g., 70B-RT drops from 94% to 84% ASR), so the benefit of the new guideline is not unique to SRG; this should be acknowledged when highlighting the flexibility of the method.
Circularity Check
BoN-based latent-knowledge claim is definitional; SRG improvements are independently validated.
-
self definitional
[Section 3 (evaluation definition) and Section 3.1 (Figure 1 discussion)]
"Specifically, we sample N outputs per query from the RT model and consider an attack unsuccessful if at least one response is classified as safe by an external classifier (Llama-Guard-3-8B). Notably, as N increases, the model’s ASR drops significantly across all OOD attacks. Compared to greedy decoding, BoN with N = 128 reduces ASR by an average of 60%... This underscores that the model possesses sufficient safety-related latent knowledge."
Under the paper’s own success criterion, an attack is unsuccessful if any of the N sampled responses is safe. If each sample has probability p of being judged safe, the reported ASR satisfies ASR(N) = (1-p)^N; more generally, the any-safe rule makes ASR nonincreasing in N for any stochastic model with a nonzero safe-response probability. The observed monotone drop as N grows is therefore guaranteed by the evaluation definition, not by any measured property of 'latent safety knowledge.' The conclusion that RT models 'possess sufficient safety-related latent knowledge' is a restatement of the metric’s construction rather than an independent probe: it cannot distinguish a model that knows safety from one that occasionally produces a safe refusal.
full rationale
One definitional reduction is present. The paper defines BoN success as 'at least one safe response' and then reads the guaranteed monotone ASR decrease as evidence of latent safety knowledge (Sec. 3/3.1). This is a self-definitional inference: any model with a nonzero probability of emitting a safe response will exhibit a decreasing ASR under the any-safe criterion, so the direction of the effect is fixed by the metric, not by the model’s internal knowledge state. I nevertheless keep the score at 4 rather than higher because the paper’s proposed SRG training, and its headline greedy-decoding ASR reductions (46.3%/54.8% in Table 1), are evaluated against an external classifier on held-out ID/OOD attack benchmarks and do not depend on the BoN argument. The RepE/PCA visualization is qualitative and would be stronger with a quantitative separation measure, but that is an evidentiary weakness, not circular reasoning. No other load-bearing step reduces to a fit or to a self-citation chain: the supervision synthesis, rejection sampling, and context distillation are external inputs, and the method is compared against open baselines.
Assumptions & free parameters
free parameters (4)
- Guideline templates C
- Safety degree thresholds =
0.8 and 0.5
- Reasoning step budget =
6 (default), 15 (extra)
- Rejection sampling judge models =
LLaMA-3.1-8B-instruct and Llama-Guard-3
assumptions (6)
- domain assumption LLaMA-Guard-3-8B accurately labels harmful content for ASR evaluation.
- domain assumption GPT-4o-generated reasoning traces are correct and safe.
- standard math The Ben-David et al. (2006) bound informally applies to safety OOD generalization.
- ad hoc to paper BoN with at least-one-safe-response criterion reveals latent safety knowledge.
- domain assumption RepE PCA visualizations correspond to the model's safety-relevant representation space.
- domain assumption Context distillation transfers the reasoning behavior to the target model.
invented entities (1)
-
safety degree
Cite this review
Pith. "Pith review of Safety Reasoning with Guidelines." pith.science (2026). https://pith.science/paper/IYT3OGXG
@misc{pith2026250204040,
author = {Pith},
title = {Pith review of: Safety Reasoning with Guidelines},
year = {2026},
howpublished = {\url{https://pith.science/paper/IYT3OGXG}},
note = {Machine review of arXiv:2502.04040}
}
read the original abstract
Training safe LLMs remains a critical challenge. The most widely used method, Refusal Training (RT), struggles to generalize against various Out-of-Distribution (OOD) jailbreaking attacks. Although various advanced methods have been proposed to address this issue, we instead question whether OOD attacks inherently surpass the capability of vanilla RT. Evaluations using Best-of-N (BoN) reveal significant safety improvements as N increases, indicating models possess adequate latent safety knowledge but RT fails to consistently elicit it under OOD scenarios. Further domain adaptation analysis reveals that direct RT causes reliance on superficial shortcuts, resulting in non-generalizable representation mappings. Inspired by our findings, we propose training model to perform safety reasoning for each query. Specifically, we synthesize reasoning supervision aligned with specified guidelines that reflect diverse perspectives on safety knowledge. This encourages model to engage in deeper reasoning, explicitly eliciting and utilizing latent safety knowledge for each query. Extensive experiments show that our method significantly improves model generalization against OOD attacks.
Figures
Figures from the paper (10 more)
Forward citations
Cited by 4 Pith papers
-
Reasoning Up the Instruction Ladder for Controllable Language Models
RLVR on ~7K verifiable system/user conflict examples teaches LLMs to prioritize higher-priority instructions, improving instruction-hierarchy and safety benchmarks.
-
Lifelong Safety Alignment for Language Models
A co-evolutionary attacker-defender loop, warmed up by strategies extracted from jailbreak papers, reduces jailbreak success rate on a robust model from 73% to 7% in two iterations.
-
Reason Before You Retrieve: Agentic Planning for Multi-modal RAG
MM-R2 claims SOTA multimodal RAG accuracy on InfoSeek and Encyclopedic VQA via intent grounding plus a 10-topic KnowledgeMap, but its teacher trajectories leak the gold Wikipedia page and omit the image.
-
AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning
AlphaAlign uses pure reinforcement learning with a verifiable safety reward to make LLMs refuse harmful requests with explicit reasoning while preserving helpfulness on benign queries.
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023
arXiv 2023
-
[3]
Concrete problems in ai safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Man \'e , D. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016
arXiv 2016
-
[4]
Andriushchenko, M. and Flammarion, N. Does refusal training in llms generalize to the past tense? arXiv preprint arXiv:2407.11969, 2024
arXiv 2024
-
[5]
A general language assistant as a laboratory for alignment
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021
arXiv 2021
-
[6]
Program synthesis with large language models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021
arXiv 2021
-
[7]
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023
arXiv 2023
-
[8]
Constitutional ai: Harmlessness from ai feedback
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022
arXiv 2022
Show all 80 references
-
[9]
Analysis of representations for domain adaptation
Ben-David, S., Blitzer, J., Crammer, K., and Pereira, F. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006
2006
-
[10]
Safety-tuned LL a MA s: Lessons from improving the safety of large language models that follow instructions
Bianchi, F., Suzgun, M., Attanasio, G., Rottger, P., Jurafsky, D., Hashimoto, T., and Zou, J. Safety-tuned LL a MA s: Lessons from improving the safety of large language models that follow instructions. In The Twelfth International Conference on Learning Representations, 2024
2024
-
[11]
J., and Wong, E
Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., and Wong, E. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023
2023 arXiv
-
[12]
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., B...
2021
-
[13]
Understanding and improving feature learning for out-of-distribution generalization
Chen, Y., Huang, W., Zhou, K., Bian, Y., Han, B., and Cheng, J. Understanding and improving feature learning for out-of-distribution generalization. Advances in Neural Information Processing Systems, 36: 0 68221--68275, 2023
2023
-
[14]
Training verifiers to solve math word problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021
-
[15]
Ultrafeedback: Boosting language models with high-quality feedback, 2023
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback, 2023
2023
-
[16]
D., et al
D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 23 0 (226): 0 1--61, 2022
2022
-
[17]
Enhancing chat language models by scaling high-quality instructional conversations
Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023
2023 arXiv
-
[18]
Scaling laws for reward model overoptimization
Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp.\ 10835--10866. PMLR, 2023
2023
-
[19]
a ger, T., Abdalla, M., Gasteiger, J., and G \
Geisler, S., Wollschl \"a ger, T., Abdalla, M., Gasteiger, J., and G \"u nnemann, S. Attacking large language models with projected gradient descent. arXiv preprint arXiv:2402.09154, 2024
2024 arXiv
-
[20]
Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Heylar, A., Dias, R., Vallone, A., Ren, H., Wei, J., et al
Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Heylar, A., Dias, R., Vallone, A., Ren, H., Wei, J., et al. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339, 2024
2024 arXiv
-
[21]
L., Liu, Y., Shang, N., Sun, Y., Zhu, Y., Yang, F., and Yang, M
Guan, X., Zhang, L. L., Liu, Y., Shang, N., Sun, Y., Zhu, Y., Yang, F., and Yang, M. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519, 2025
2025 arXiv
-
[22]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025
2025 arXiv
-
[23]
Overthinking the truth: Understanding how language models process false demonstrations
Halawi, D., Denain, J.-S., and Steinhardt, J. Overthinking the truth: Understanding how language models process false demonstrations. In The Twelfth International Conference on Learning Representations, 2024
2024
-
[24]
Measuring mathematical problem solving with the math dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021
2021
-
[25]
J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021
2021 arXiv
-
[26]
P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024
2024 arXiv
-
[27]
Izmailov, P., Kirichenko, P., Gruver, N., and Wilson, A. G. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems, 35: 0 38516--38532, 2022
2022
-
[28]
Pku-saferlhf: Towards multi-level safety alignment for llms with human preference
Ji, J., Hong, D., Zhang, B., Chen, B., Dai, J., Zheng, B., Qiu, T., Li, B., and Yang, Y. Pku-saferlhf: Towards multi-level safety alignment for llms with human preference. arXiv preprint arXiv:2406.15513, 2024
2024 arXiv
-
[29]
Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023
2023 arXiv
-
[30]
website template1, 2024
Klieger, B. website template1, 2024. URL https://github.com/bklieger-groq/g1
2024
-
[31]
Fine-tuning can distort pretrained features and underperform out-of-distribution
Kumar, A., Raghunathan, A., Jones, R., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054, 2022
2022 arXiv
-
[32]
H., Gonzalez, J
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023
2023
-
[33]
Diversify and disambiguate: Learning from underspecified data
Lee, Y., Yao, H., and Finn, C. Diversify and disambiguate: Learning from underspecified data. arXiv preprint arXiv:2202.03418, 2022
2022 arXiv
-
[34]
Common 7b language models already possess strong math capabilities
Li, C., Wang, W., Hu, J., Wei, Y., Zheng, N., Hu, H., Zhang, Z., and Peng, H. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.04706, 2024
2024 arXiv
-
[35]
Let's verify step by step
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023
2023 arXiv
-
[36]
S., Wang, Y., and Zhang, L
Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023 a
2023
-
[37]
Autodan: Generating stealthy jailbreak prompts on aligned large language models
Liu, X., Xu, N., Chen, M., and Xiao, C. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451, 2023 b
2023 arXiv
-
[38]
Llama Team, A. . M. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783
2024 arXiv
-
[39]
Y., Roongta, M., Cai, C., Luo, J., Li, L
Luo, M., Tan, S., Wong, J., Shi, X., Tang, W. Y., Roongta, M., Cai, C., Luo, J., Li, L. E., Popa, R. A., and Stoica, I. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025. Notion Blog
2025
-
[40]
Self-refine: Iterative refinement with self-feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2023
2023
-
[41]
Harmbench: A standardized evaluation framework for automated red teaming and robust refusal
Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249, 2024
2024 arXiv
-
[42]
Orca-math: Unlocking the potential of slms in grade school math
Mitra, A., Khanpour, H., Rosset, C., and Awadallah, A. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402.14830, 2024
2024 arXiv
-
[43]
Rule based rewards for language model safety
Mu, T., Helyar, A., Heidecke, J., Achiam, J., Vallone, A., Kivlichan, I., Lin, M., Beutel, A., Schulman, J., and Weng, L. Rule based rewards for language model safety. arXiv preprint arXiv:2411.01111, 2024
2024 arXiv
-
[44]
Introducing ChatGPT , 2022
OpenAI. Introducing ChatGPT , 2022. URL https://openai.com/blog/chatgpt
2022
-
[45]
Learning to reason with llms, 2024
OpenAI. Learning to reason with llms, 2024. URL https://openai.com/index/learning-to-reason-with-llms/
2024
-
[46]
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022
2022
-
[47]
Safety alignment should be made more than just a few tokens deep
Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., and Henderson, P. Safety alignment should be made more than just a few tokens deep. arXiv preprint arXiv:2406.05946, 2024
2024 arXiv
-
[48]
H., Fung, Y
Qin, Z., Dong, Q., Zhang, X., Dong, L., Huang, X., Yang, Z., Khademi, M., Zhang, D., Awadalla, H. H., Fung, Y. R., et al. Scaling laws of synthetic data for language models. arXiv preprint arXiv:2503.19551, 2025
2025
-
[49]
Recursive introspection: Teaching language model agents how to self-improve
Qu, Y., Zhang, T., Garg, N., and Kumar, A. Recursive introspection: Teaching language model agents how to self-improve. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024
2024
-
[50]
R., Vidgen, B., Attanasio, G., Bianchi, F., and Hovy, D
R \"o ttger, P., Kirk, H. R., Vidgen, B., Attanasio, G., Bianchi, F., and Hovy, D. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263, 2023
2023 arXiv
-
[51]
website template2, 2024
Schmid, P. website template2, 2024. URL https://x.com/_philschmid/status/1842846050320544016
2024
-
[52]
do anything now
Shen, X., Chen, Z., Backes, M., Shen, Y., and Zhang, Y. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp.\ 1671--1685, 2024
2024
-
[53]
C., Perez, E., Hadfield-Menell, D., et al
Sheshadri, A., Ewart, A., Guo, P., Lynch, A., Wu, C., Hebbar, V., Sleight, H., Stickland, A. C., Perez, E., Hadfield-Menell, D., et al. Latent adversarial training improves robustness to persistent harmful behaviors in llms. arXiv preprint arXiv:2407.15549, 2024
2024 arXiv
-
[54]
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in neural information processing systems, 33: 0 3008--3021, 2020
2020
-
[55]
G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024
2024 arXiv
-
[56]
Team, K., Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C., et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025
2025 arXiv
-
[57]
Qwen2.5: A party of foundation models, September 2024
Team, Q. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/
2024
-
[59]
Llama 2: Open foundation and fine-tuned chat models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b
2023 arXiv
-
[60]
Solving math word problems with process-and outcome-based feedback
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, I. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022
2022 arXiv
-
[61]
Do-anything-now, 2022
Walkerspider. Do-anything-now, 2022. URL https://www.reddit.com/r/ChatGPT/comments/zlcyr9/dan_is_my_new_friend/?rdt=63210
2022
-
[62]
Step-on-feet tuning: Scaling self-alignment of llms via bootstrapping
Wang, H., Ma, G., Meng, Z., Qin, Z., Shen, L., Zhang, Z., Wu, B., Liu, L., Bian, Y., Xu, T., et al. Step-on-feet tuning: Scaling self-alignment of llms via bootstrapping. arXiv preprint arXiv:2402.07610, 2024
2024 arXiv
-
[63]
Lifelong safety alignment for language models, 2025
Wang, H., Qin, Z., Zhao, Y., Du, C., Lin, M., Wang, X., and Pang, T. Lifelong safety alignment for language models, 2025. URL https://arxiv.org/abs/2505.20259
2025 arXiv
-
[64]
Do-not-answer: A dataset for evaluating safeguards in llms
Wang, Y., Li, H., Han, X., Nakov, P., and Baldwin, T. Do-not-answer: A dataset for evaluating safeguards in llms. arXiv preprint arXiv:2308.13387, 2023
2023 arXiv
-
[65]
Jailbroken: How does LLM safety training fail? In Thirty-seventh Conference on Neural Information Processing Systems, 2023
Wei, A., Haghtalab, N., and Steinhardt, J. Jailbroken: How does LLM safety training fail? In Thirty-seventh Conference on Neural Information Processing Systems, 2023
2023
-
[66]
Reward hacking in reinforcement learning
Weng, L. Reward hacking in reinforcement learning. lilianweng.github.io, Nov 2024. URL https://lilianweng.github.io/posts/2024-11-28-reward-hacking/
2024
-
[67]
Wizardlm: Empowering large language models to follow complex instructions
Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023
2023 arXiv
-
[68]
C.-J., Zhang, T., Patil, S
Yan, F., Mao, H., Ji, C. C.-J., Zhang, T., Patil, S. G., Stoica, I., and Gonzalez, J. E. Berkeley function calling leaderboard. 2024
2024
-
[69]
T., Li, Z., Weller, A., and Liu, W
Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023
2023 arXiv
-
[70]
Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher
Yuan, Y., Jiao, W., Wang, W., Huang, J.-t., He, P., Shi, S., and Tu, Z. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463, 2023 a
2023 arXiv
-
[72]
Refuse whenever you feel unsafe: Improving safety in llms via decoupled refusal training
Yuan, Y., Jiao, W., Wang, W., Huang, J.-t., Xu, J., Liang, T., He, P., and Tu, Z. Refuse whenever you feel unsafe: Improving safety in llms via decoupled refusal training. arXiv preprint arXiv:2407.09121, 2024 b
2024 arXiv
-
[73]
Scaling relationship on learning mathematical reasoning with large language models
Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023 b
2023 arXiv
-
[74]
Star: Bootstrapping reasoning with reasoning
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35: 0 15476--15488, 2022
2022
-
[75]
How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms
Zeng, Y., Lin, H., Zhang, J., Yang, D., Jia, R., and Shi, W. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373, 2024
2024 arXiv
-
[76]
Rich feature construction for the optimization-generalization dilemma
Zhang, J., Lopez-Paz, D., and Bottou, L. Rich feature construction for the optimization-generalization dilemma. In International Conference on Machine Learning, pp.\ 26397--26411. PMLR, 2022
2022
-
[77]
Llamafactory: Unified efficient fine-tuning of 100+ language models
Zheng, Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z., Feng, Z., and Ma, Y. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Th...
2024 arXiv
-
[78]
Lima: Less is more for alignment
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2023
2023
-
[79]
Representation engineering: A top-down approach to ai transparency
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023 a
2023 arXiv
-
[80]
Z., and Fredrikson, M
Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023 b
2023 arXiv
-
[81]
Z., Fredrikson, M., and Hendrycks, D
Zou, A., Phan, L., Wang, J., Duenas, D., Lin, M., Andriushchenko, M., Kolter, J. Z., Fredrikson, M., and Hendrycks, D. Improving alignment and robustness with circuit breakers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 a
2024
-
[82]
Improving alignment and robustness with short circuiting
Zou, A., Phan, L., Wang, J., Duenas, D., Lin, M., Andriushchenko, M., Wang, R., Kolter, Z., Fredrikson, M., and Hendrycks, D. Improving alignment and robustness with short circuiting. arXiv preprint arXiv:2406.04313, 2024 b
2024 arXiv
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.