REVIEW 4 major objections 5 minor 29 references
R1-ACT: Efficient Reasoning Model Safety Alignment by Activating Safety Knowledge
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Large reasoning models store safety knowledge but fail to activate it during chain-of-thought reasoning; this paper shows that a training structure with an explicit harmfulness-assessment step activates that knowledge, sharply reducing…
desk verdict Useful, cheap safety-alignment recipe with a plausible activation-failure diagnosis, but the headline numbers need an overlap check and error bars before I'd trust them. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a three-step reasoning structure—problem understanding → harmfulness assessment → solution reasoning—inserted into the model's chain of thought. The harmfulness assessment step acts as a trigger that raises stored safety knowledge from a latent state to an active state before the model commits to a solution, analogous to the psychological notion of knowledge activation invoked from Higgins (1996). The paper's training data is built by taking the problem-understanding first sentence from R1-70B traces, eliciting a one-sentence harmfulness judgment from GPT-4o, and appending a fixed safe-termination sentence for harmful cases; benign cases add a full R1-70B solution and answer to preserve helpfulness. Because each training chain is only about 171 tokens, the method is 2–6× cheaper in tokens than the baselines and needs just 1,000 examples.
What would settle it
Measure the probe accuracy on a larger and more diverse set of harmful and benign instructions; if a model whose probe accuracy is at chance nonetheless shows large safety gains after R1-Act training, the stored-knowledge diagnosis would be wrong. Alternatively, fine-tune an LRM on the same 1,000 examples but with the harmfulness-assessment step removed from the reasoning structure; if safety improves just as much without that step, activation of stored knowledge is not what drives the results.
Extended reading notes
Core claim
The central discovery is Finding 1: LRMs already possess sufficient safety knowledge yet fail to activate it during reasoning. The paper establishes this by showing that DeepSeek-R1-Distill models at 7B and 8B, and SafeChain variants, perform competitively with instruction-tuned LLMs on binary classification and token-level probing over benign and harmful WildJailbreak queries—so the knowledge is stored—while their compliance rates on harmful prompts remain high, often above 70%. A simple prompt that asks the model to determine whether an instruction is harmful or benign before responding cuts compliance substantially, confirming that activation, not knowledge, is the missing piece. R1-Act operationalizes this by supervised fine-tuning on a dataset whose reasoning chains insert an explicit harmfulness assessment between problem understanding and solution reasoning, using 900 harmful and 100 benign examples, and the resulting models refuse harmful instructions far more often while keeping math and coding performance roughly intact.
Load-bearing premise
The entire diagnosis rests on the assumption that the binary classification and token-level probes really reveal the model's stored safety knowledge—if those probes only capture surface-level classification ability, or the 100-query sample is unrepresentative, the claim that safety knowledge is "stored but inactivated" loses its foundation.
Editorial extensions
If this is right
- R1-Act reduces harmful compliance on StrongReject, WildJailbreak, and JBB-Behaviors across 1.5B to 14B models while preserving reasoning scores on GSM8K, MATH-500, AIME24, and HumanEval.
- The method outperforms SafeChain, including when SafeChain is given 40k training examples, and beats STAR-1's safety while showing less over-refusal on benign-but-tricky prompts.
- The three-step reasoning structure itself, not just alignment training, is what protects reasoning performance: ablations show that alignment training without the structure degrades reasoning substantially.
- Only 100 benign examples are enough to bring over-refusal from around 22–25% down to 69–88% compliance on benign queries, showing the data efficiency.
- Training an 8B model takes 90 minutes on a single RTX A6000 GPU, so the method is accessible without large compute.
Reading between the lines
- If the activation diagnosis generalizes, safety evaluation of LRMs should measure not just final answers but whether safety knowledge is engaged during the reasoning trace; models that fail probes should be treated as knowledge-deficient rather than simply misaligned.
- The 1,000-example recipe suggests that many safety failures in reasoning models may be correctable by small, structured interventions rather than large preference datasets; a testable extension is whether the same three-step trigger works when inserted at inference time without any fine-tuning, as a decoding constraint.
- The sensitivity to trigger words (e.g., refusing benign queries containing "kill", missing "unanimously") hints that the harmfulness assessment step is still shallow—an extension would be to train or prompt the model to reason about context and intent rather than lexically flagging topics.
- The success of a single-sentence harmfulness assessment by GPT-4o as training data raises the question of whether the trigger could be learned from the model's own self-generated assessments, removing the reliance on a proprietary teacher.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that large reasoning models (LRMs) already store sufficient safety knowledge but fail to activate it during reasoning, and proposes R1-Act, a post-training method that prepends a mandatory harmfulness assessment step to the model's reasoning structure. The method is trained on only 1,000 examples (900 harmful, 100 benign) and is claimed to substantially reduce harmful compliance on StrongReject, WildJailbreak, and JBB-Behaviors while preserving math and coding performance, with training taking about 90 minutes on a single RTX A6000 GPU. Experiments cover DeepSeek-R1-Distill models at 1.5B, 7B, 8B, and 14B scales, and compare against SafeChain and STAR-1.
Significance. If the claims hold, R1-Act is a practically valuable and unusually cheap alignment recipe: it requires only 1,000 examples, 171 tokens per example, and a single-GPU fine-tune, and the authors release code, data, and model weights. The paper also makes a conceptually interesting diagnostic claim (activation failure rather than missing knowledge) and backs it with a concrete prompt-based intervention. The main limitations are that the headline numbers rest on an unchecked possible train/eval overlap, on single-run point estimates without uncertainty, and on a probing analysis with only 50+50 examples. The compliance-rate results in Table 2 are at least partly independent of the GPT-4o-based circularity that affects Table 4, because the former use Llama-3.1-8B-Instruct as the judge, but the diagnostic foundation of Finding 1 is thin.
major comments (4)
- [Section 4.2; Section 5.1 (Table 2)] The 900 harmful training instructions are randomly sampled from the dataset released by Jiang et al. (2025) (SafeChain), while safety is evaluated on WildJailbreak, StrongReject, and JBB-Behaviors. The paper does not report any deduplication or overlap analysis between the training sample and these evaluation sets. If SafeChain's released data is assembled from the same public benchmarks, the large reductions in Table 2 — for example, WildJailbreak compliance falling from 89.6 to 21.2 for R1-8B — could be inflated by memorization of benchmark phrasing rather than by activated safety knowledge. This is directly testable with the released training data and evaluation sets, and the authors should report exact and near-duplicate overlap statistics and, if overlaps exist, retrain on deduplicated data.
- [Section 3 (Figure 1)] Finding 1 — that LRMs already possess sufficient safety knowledge but fail to activate it — is supported only by two probes on 50 benign and 50 harmful WildJailbreak queries, with no confidence intervals or statistical tests. Both probes are direct classification tasks: a yes/no question and a forced-choice logit comparison between the words 'benign' and 'harmful'. Such tasks can be solved by surface-level cues and do not establish that safety knowledge is available during free-form generation. Since the design rationale of R1-Act (inserting a harmfulness assessment step) rests on this diagnosis, the paper should either substantially strengthen the probing evidence (more examples, error bars, out-of-distribution queries, or an intervention analysis) or soften the causal claim that LRMs 'fail to activate' stored knowledge.
- [Section 5.1; Tables 2 and 4] All safety and reasoning numbers are single-run point estimates without error bars, multiple seeds, or significance tests. For a 250-item WildJailbreak sample or a 310-item StrongReject sample, a difference of a few percentage points is well within sampling noise, and several comparisons that underlie claims of 'competitive' or 'superior' performance (e.g., R1-Act vs. STAR-1 in Table 4) are of this size. The authors should report bootstrap confidence intervals or repeated-seed results, especially for the headline compliance reductions and for the over-refusal numbers.
- [Section 4.2 vs. Section 5.1 (Table 4)] The safe@1 metric in Table 4 uses GPT-4o as the safety classifier, while the harmfulness-assessment labels in the training data were also generated by GPT-4o. Table 4 therefore partly measures agreement with GPT-4o's safety judgments, and the reported safe@1 gains may overstate generalizable safety. The compliance-rate tables (Tables 1–3) are less affected because they use a different judge, but any claims based on safe@1 — including the statement that R1-Act is 'competitive with STAR-1' under this metric — should either use an independent judge or explicitly acknowledge and control for this shared-source bias.
minor comments (5)
- [Section 5.1, Tables 2 and 4] The column heading 'Over Refusal (↑)' is confusing: the reported value is compliance on XsTest, where higher is better (i.e., less refusal), not a refusal rate. The label should be something like 'XsTest non-refusal (↑)' or 'Over-refusal avoidance (↑)'.
- [Table 3] The abbreviations TR, RS, and BE are used in the table but defined only in the prose; adding a definition to the caption would make the table self-contained.
- [Section 4.2] The fixed closing sentence 'Therefore, there is no further consideration is necessary for this instruction. </think>' contains a grammatical error, and the paper should clarify whether this full string is included when computing the reported 171-token average.
- [Abstract] The abstract contains a grammar error: 'Our code are available' should be 'Our code is available'.
- [Figure 1] The figure would be much more informative if the numerical scores, sample sizes, and confidence intervals were reported in the caption or adjacent text, rather than only as bar heights.
Circularity Check
No load-bearing circularity: the activation-failure diagnosis and the R1-Act training intervention are supported by independent probes and external safety benchmarks; the only noteworthy issue is a non-load-bearing self-citation and a shared GPT-4o annotator between one training component and one secondary metric.
full rationale
The paper's central derivation chain is not circular. Finding 1 is grounded in Section 3's untrained probe experiments (binary yes/no classification and token-level logit AUC) on 50 benign and 50 harmful WildJailbreak queries; these probes do not train the evaluated models and do not presuppose the conclusion that safety knowledge is stored but under-activated. Finding 2 is supported by a zero-training prompting intervention reported in Table 1, which independently reduces compliance rates. R1-Act's training construction (Section 4.2) uses GPT-4o-generated harmfulness assessments and harmful instructions seeded from SafeChain, while the main safety results in Table 2 use compliance rate judged by Llama-3.1-8B-Instruct on StrongReject, WildJailbreak, and JBB-Behaviors; these evaluation judgments are external to the training labels. The safe@1 metric in Table 4 does share GPT-4o as both the training labeler (Section 4.2) and the evaluation judge (Section 5.1), which is a shared-annotator bias worth disclosing, but it is not a construction identity: the model must still generalize refusal behavior to unseen prompts, and the central safety claim does not rest on safe@1 alone. The only self-citation is In et al. (2025) in the evaluation-protocol list, and it is not load-bearing. The skeptic's train/eval provenance concern (900 harmful training prompts sampled from the SafeChain release versus evaluation on WildJailbreak, StrongReject, and JBB-Behaviors) is a testable data-leakage risk rather than a demonstrated circularity, because the paper does not state that evaluation prompts were used to construct the training set.
Assumptions & free parameters
free parameters (2)
- Training set composition (number of harmful/benign examples) =
900 harmful / 100 benign
- Training epochs =
15
assumptions (4)
- domain assumption The binary classification and token-level probing in Section 3 measure whether safety knowledge is stored in model parameters.
- domain assumption Supervised finetuning on 1,000 examples with the three-step structure generalizes to unseen harmful and benign instructions rather than memorizing the training patterns.
- domain assumption Compliance rate judged by Llama-3.1-8B-Instruct and safe@1 judged by GPT-4o are valid and reliable safety measures.
- ad hoc to paper The fixed closing sentence for harmful examples ('Therefore, there is no further consideration is necessary for this instruction. </think>') acts as a safe refusal and omitting the final answer ai does not hurt safety.
Cite this review
Pith. "Pith review of R1-ACT: Efficient Reasoning Model Safety Alignment by Activating Safety Knowledge." pith.science (2026). https://pith.science/paper/TDKRLCS3
@misc{pith2026250800324,
author = {Pith},
title = {Pith review of: R1-ACT: Efficient Reasoning Model Safety Alignment by Activating Safety Knowledge},
year = {2026},
howpublished = {\url{https://pith.science/paper/TDKRLCS3}},
note = {Machine review of arXiv:2508.00324}
}
read the original abstract
Although large reasoning models (LRMs) have demonstrated impressive capabilities on complex tasks, recent studies reveal that these models frequently fulfill harmful user instructions, raising significant safety concerns. In this paper, we investigate the underlying cause of LRM safety risks and find that models already possess sufficient safety knowledge but fail to activate it during reasoning. Based on this insight, we propose R1-Act, a simple and efficient post-training method that explicitly triggers safety knowledge through a structured reasoning process. R1-Act achieves strong safety improvements while preserving reasoning performance, outperforming prior alignment methods. Notably, it requires only 1,000 training examples and 90 minutes of training on a single RTX A6000 GPU. Extensive experiments across multiple LRM backbones and sizes demonstrate the robustness, scalability, and practical efficiency of our approach.
Reference graph
Works this paper leans on
-
[1]
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. Advances in Neural Information Processing Systems, 37:55005--55029
work page 2024
-
[2]
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374
arXiv 2021
-
[3]
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168
arXiv 2021
-
[4]
Michael Han Daniel Han and Unsloth team. 2023. http://github.com/unslothai/unsloth Unsloth
2023
-
[5]
Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. 2024. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339
arXiv 2024
-
[6]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948
arXiv 2025
-
[7]
Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, et al. 2025. Can large language models detect errors in long chain-of-thought reasoning? arXiv preprint arXiv:2502.19361
arXiv 2025
-
[8]
E. Higgins. 1996. Knowledge activation: Accessibility, applicability, and salience. Social Psychology: Handbook of basic Principles
work page 1996
Show all 29 references
-
[9]
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, Zachary Yahn, Yichang Xu, and Ling Liu. 2025. Safety tax: Safety alignment makes your large reasoning models less reasonable. arXiv preprint arXiv:2503.00555
2025 arXiv
-
[10]
Yeonjun In, Wonjoong Kim, Kanghoon Yoon, Sungchul Kim, Mehrab Tanjim, Kibum Kim, and Chanyoung Park. 2025. Is safety standard same for everyone? user-specific safety evaluation of large language models. arXiv preprint arXiv:2502.15086
2025
-
[11]
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720
2024 arXiv
-
[12]
Fengqing Jiang, Zhangchen Xu, Yuetai Li, Luyao Niu, Zhen Xiang, Bo Li, Bill Yuchen Lin, and Radha Poovendran. 2025. Safechain: Safety of language models with long chain-of-thought reasoning capabilities. arXiv preprint arXiv:2502.12025
2025 arXiv
-
[13]
Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, et al. 2024. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. Advances in Neural Informati...
2024
-
[14]
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. In The Twelfth International Conference on Learning Representations
2023
-
[15]
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393
2025 arXiv
-
[16]
Paul R \"o ttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2023. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263
2023 arXiv
-
[17]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300
2024 arXiv
-
[18]
Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, et al. 2024. A strongreject for empty jailbreaks. arXiv preprint arXiv:2402.10260
2024 arXiv
-
[19]
Changsheng Wang, Chongyu Fan, Yihua Zhang, Jinghan Jia, Dennis Wei, Parikshit Ram, Nathalie Baracaldo, and Sijia Liu. 2025 a . Reasoning model unlearning: Forgetting traces, not just answers, while preserving reasoning skills. arXiv preprint arXiv:2506.12963
2025
-
[20]
Zijun Wang, Haoqin Tu, Yuhan Wang, Juncheng Wu, Jieru Mei, Brian R Bartoldson, Bhavya Kailkhura, and Cihang Xie. 2025 b . Star-1: Safer alignment of reasoning llms with 1k data. arXiv preprint arXiv:2504.01903
2025
-
[21]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837
2022
-
[22]
Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, et al. 2024. Sorry-bench: Systematically evaluating large language model safety refusal behaviors. arXiv preprint arXiv:2406.14598
2024 arXiv
-
[23]
Sangyeon Yoon, Wonje Jeung, and Albert No. 2025. R-tofu: Unlearning in large reasoning models. arXiv preprint arXiv:2505.15214
2025 arXiv
-
[24]
Yichi Zhang, Zihao Zeng, Dongbai Li, Yao Huang, Zhijie Deng, and Yinpeng Dong. 2025 a . Realsafe-r1: Safety-aligned deepseek-r1 without compromising reasoning capability. arXiv preprint arXiv:2504.10081
2025 arXiv
-
[25]
Zhexin Zhang, Xian Qi Loye, Victor Shea-Jay Huang, Junxiao Yang, Qi Zhu, Shiyao Cui, Fei Mi, Lifeng Shang, Yingkang Wang, Hongning Wang, et al. 2025 b . How should we enhance the safety of large reasoning models: An empirical study. arXiv preprint arXiv:2505.15404
2025 arXiv
-
[26]
Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Shreedhar Jangam, Jayanth Srinivasa, Gaowen Liu, Dawn Song, and Xin Eric Wang. 2025 a . The hidden risks of large reasoning models: A safety assessment of r1. arXiv preprint arXiv:2502.12659
2025
-
[27]
Kaiwen Zhou, Xuandong Zhao, Gaowen Liu, Jayanth Srinivasa, Aosong Feng, Dawn Song, and Xin Eric Wang. 2025 b . Safekey: Amplifying aha-moment insights for safety reasoning. arXiv preprint arXiv:2505.16186
2025
-
[28]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
-
[29]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.