{"id":"26ec590f-00e2-41d7-afb3-e9b70605de2b","arxiv_id":"2411.17075","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Encouraging LLMs to analyze user requests step-by-step (System-2 Alignment) modestly improves safety on open-source models, but with trade-offs and limited evidence.","lead":"This study tests whether making language models reason step-by-step about safety makes them safer. It finds that some prompting and fine-tuning methods improve safety on open-source models, but results are inconsistent and the evaluation is limited.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Safety gains may rest on a closed GPT-4 loop: the same model family labels training data, generates CoT distillations, and scores the evaluation; human or independent validation is needed.","rationale":"The reader's weakest-assumption analysis matches mine: the validity of GPT-4-based evaluation is the single most load-bearing condition for the central claim. Because GPT-4o generates the CoT training signal and GPT-4 scores the outputs, a systematic preference for GPT-4-like reasoning style would inflate all post-training results. The tag-stripping and human-label check would distinguish genuine safety from format-matching; without it, 'some simple methods improve safety' is not established. Mixed results across models and methods are secondary because the abstract explicitly limits the claim to 'some methods,' and the primary threat is measurement bias rather than statistical inconsistency. The paper is otherwise honest and exploratory, with released code and a manual o1 evaluation as a positive sign. Therefore the reader's conditional verdict is appropriate, and I would not move it further without additional validation.","tokens_in":18840,"tokens_out":7726,"duration_ms":71721,"concrete_test":"Re-score Table 5 with an independent judge: for a random subset of 100 adversarial harmful and 100 adversarial benign responses from Llama3-8B SFT-Baseline and SFT-CoT, strip all <Analysis> tags (leaving only the final answer), and have three human annotators follow the definitions in Tables 9 and 12 to label safety and overrefusal. If the SFT-CoT improvement over SFT-Baseline is not reproduced on human labels, the reported gains reflect GPT-4 style matching rather than genuine safety.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that methods encouraging careful analysis improve safety—rests entirely on a closed evaluation loop. In Section 3.1, GPT-4 scores all open-source results using the evaluation prompt in Table 9. In Section 3.2, GPT-4o is used both to assign safety categories to SFT training data and to distill the reasoning-annotated CoT responses from which models learn. DPO and RL in Sections 3.3–3.4 then construct preference/reward labels from the same safety taxonomy, so the optimization target and the measurement are the same model family. If GPT-4-family judges rate responses that mimic its own structured, 'thoughtful' refusal style as safe, then SFT-CoT and RL models may improve on the metric without improving real-world safety. Manual evaluation is reported only for the o1 study in Section 2; no human validation is described for the main open-source comparisons, even though the test set is only 420 samples. The lack of any independent judge is therefore the most load-bearing weakness: every quantitative gain in Tables 4–6 could be an artifact of GPT-4 self-preference.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper explores 'System-2 alignment' for large language model safety, i.e., encouraging models to reason carefully about user requests before responding. It first evaluates OpenAI's o1 model on adversarial natural-language jailbreaks and math-encoded jailbreaks using 20 samples per condition with manual evaluation (Section 2). It then applies prompt engineering, supervised fine-tuning (SFT), direct preference optimization (DPO), and outcome-supervised reinforcement learning (RL) to Mistral-7B, Qwen-7B, and Llama3-8B, evaluating all open-source results with a GPT-4 judge on 420 WildJailbreak samples (Section 3). The paper also proposes a process-supervision RL framework (Section 3.5) but presents no experiments for it. The central claim is that methods encouraging deliberate analysis improve model safety, with the abstract noting that 'some simple methods' are beneficial.","tokens_in":21,"tokens_out":8025,"duration_ms":127924,"significance":"If the central claim held, the results would be practically useful: simple prompting and fine-tuning interventions could improve the safety of open-source models, complementing OpenAI's deliberative alignment approach. The paper's strengths are its open release of code and data, the manual evaluation and detailed case studies for the o1 model, and the honest reporting of trade-offs such as increased overrefusal and model-dependent failures. However, the significance is limited by the closed evaluation loop (GPT-4o labels/distills training data, GPT-4 scores outputs), the absence of uncertainty quantification, and the fact that several reported results contradict the broad central claim. The contribution is best read as an exploratory study that identifies directions for further work rather than a definitive demonstration.","major_comments":[{"comment":"The central claim that System-2 methods improve safety rests on a closed evaluation loop. Section 3.1 uses GPT-4 as the judge for all open-source outputs, with the prompt in Table 9, while Section 3.2 uses GPT-4o to assign safety labels to the SFT training data and to distill the reasoning-annotated CoT responses. DPO and RL in Sections 3.3 and 3.4 construct preference/reward labels from the same safety taxonomy. No human validation or independent judge is reported for any of the 420 open-source test responses. Since the judge and the teacher belong to the same model family, the improvements in Tables 4-6 could reflect the models learning to imitate GPT-4o's structured refusal style rather than genuine safety. This is load-bearing for the abstract's claim. Please report inter-annotator agreement between GPT-4 judgments and human labels on a random subset (e.g., 50 responses per condition), or compare GPT-4 judgments with a second independent judge.","section":"Section 3.1/3.2; Tables 4-6"},{"comment":"The central claim, stated in the Introduction as 'methods encouraging models to carefully analyze user inputs improve safety,' is not uniformly supported by the reported results. In Table 4, Llama3-8B with zero-shot CoT (safe) has average 0.691 versus 0.779 for the baseline, driven by a drop in not_overrefuse from 0.790 to 0.533. In Table 5, Qwen-7B SFT-CoT has average 0.771 versus 0.788 for SFT-Baseline, with not_unsafe falling from 0.690 to 0.609. The abstract's weaker 'some simple methods... are beneficial' is compatible with the data, but the Introduction's general statement is contradicted. Please either revise the central claim to be method- and model-specific, or provide an analysis of the conditions under which System-2 alignment helps.","section":"Section 1; Tables 4-5"},{"comment":"No uncertainty quantification is provided for any of the headline numbers. The o1 evaluation in Section 2 uses only 20 samples per condition (Tables 2-3); for example, the difference between o1 (17/20 not_unsafe) and GPT-4o (13/20) is not shown to be significant. The open-source evaluation has 420 test samples but reports point estimates only, with no confidence intervals or significance tests across random seeds or sampling. Without error bars, the improvements in Tables 4-6 (e.g., Llama3-8B SFT-CoT average 0.905 vs SFT-Baseline 0.760) cannot be distinguished from noise. Please add bootstrap or multiple-seed evaluation results, or at minimum report per-condition counts and confidence intervals.","section":"Tables 2-6"},{"comment":"The process supervision section (Section 3.5) is a proposal with no experiments; the abstract states that 'implementation details and experimental results will be provided in future versions,' and the first page labels the paper 'A study in preparation.' Yet the Conclusion claims that 'we showed that incorporating slow-thinking processes improves safety performance' and that the study 'employs diverse methodologies—... reinforcement learning—to validate the efficacy.' Since the process-supervised RL is not implemented or evaluated, the validated scope is limited to prompting, SFT, DPO, and outcome-supervised RL. Please either implement and evaluate the process supervision, or explicitly frame it as future work and revise the Conclusion and abstract accordingly.","section":"Section 3.5; Abstract"}],"minor_comments":[{"comment":"There is a typo in the caption: 'bottm' should be 'bottom.'","section":"Section 2.1, Figure 1 caption"},{"comment":"The word 'implementaions' should be 'implementations.'","section":"Section 3, opening paragraph"},{"comment":"The notation 'Few-shot CoT (+ -)' is unexplained; please define the symbols, presumably indicating one adversarial benign and one adversarial harmful in-context example.","section":"Table 4"},{"comment":"The phrase 'as shown in Table 1314' should read 'Tables 13 and 14.'","section":"Section 3.2, page 8"},{"comment":"Reference [12] is incomplete: 'Yuxiang Zhang and et. al. o1-coder: an o1 replication for coding. arXiv preprint, 2024' lacks a full title and arXiv identifier; also 'and et. al.' is redundant.","section":"References"},{"comment":"The relationship between the three label columns and the row categories should be explained; the identical counts in the Allowed, Disallowed, and Safe Completion columns for each category are confusing.","section":"Table 10"},{"comment":"The line 'Fault information and storyline written by I'm' appears to be a broken phrase; please correct.","section":"Figure 2, Thoughts section"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is explicitly marked as a preliminary study ('A study in preparation', page 2) and reports that process-supervision experiments will appear in future versions. Given the journal context, the editors may wish to consider whether the paper meets the completeness bar for a full archival contribution. The closed GPT-4 evaluation loop is the most serious technical issue; the authors should be encouraged to add human evaluation or an independent judge. The code and data release, and the manual o1 case studies, are positive aspects that partially offset these concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. This is an honest exploratory paper, and the empirical core is a useful addition to the small but growing literature on reasoning-style safety alignment. What's actually new: a clean comparison of prompting, SFT, DPO, and RL for 'System-2 alignment' on three open-weight models, with code and data released. The authors also frame their work as a complement to Deliberative Alignment rather than a competitor, which is fair. The mixed results help: SFT-CoT helps Mistral and Llama but not Qwen, and zero-shot safe prompting hurts Llama. If the GPT-4 judge were simply rewarding longer CoT-style answers, you'd expect more uniform gains.\n\nThe soft spots are real and load-bearing. The whole pipeline runs on one model family: GPT-4o assigns safety categories to the SFT training data, distills the reasoning traces, and GPT-4 scores the evaluation. The test set is 420 samples with no human validation on the main open-source comparisons—the manual eval is only on the o1 case studies. That means the measured safety improvements could be style matching rather than genuine safety. The math-encoded jailbreak evaluation is 20 samples with no error bars; it's suggestive, not conclusive. The process-supervision section (3.5) is a proposal, no experiments, and the paper itself says the implementation details will come later. For an arXiv exploratory study that's acceptable, but it should be labeled as such in the claims.\n\nThe o1 case studies are actually the most interesting part. Seeing the model's reasoning chain go from safe to unsafe mid-thought (Case 2) and the mathematical-encoding attacks slipping through are concrete, falsifiable observations. Those are worth building on.\n\nMy bottom line: the central claim—that simple methods encouraging careful analysis improve safety—is plausible and partially supported, but the closed evaluation loop is the key weakness. It needs independent judgment (human or a second model family) before I'd take the numbers at face value. The paper deserves serious review; it's a legitimate exploratory study with a clear research direction. I'd send it to peer review, and ask for human evaluation, error bars, and either results for the process-supervision plan or a clear statement that it's future work.","headline":"Useful exploratory study, but the safety gains rest on a GPT-4-family evaluation loop that needs independent validation before the headline claim is trusted.","tokens_in":19570,"tokens_out":2140,"would_cite":true,"duration_ms":20517,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Encouraging language models to scrutinize user requests step by step reduces harmful outputs, while math-encoded jailbreaks remain a gap.","keywords":["System-2 alignment","LLM safety","jailbreak","chain-of-thought","prompt engineering","supervised fine-tuning","direct preference optimization","reinforcement learning"],"falsifier":"Collect human-annotated safety labels for the same WildJailbreak and math-encoded test prompts, then score the SFT-CoT and RL models against those labels; if human agreement with the model judge is low or the human-annotated safety gain vanishes, the central claim fails. A cheaper check: attack the fine-tuned models with a math-encoded prompt built from a harm category excluded from the SFT taxonomy and see whether the reasoning chain still flags it.","tokens_in":18684,"feed_emoji":"🧠","tokens_out":4098,"duration_ms":37778,"temperature":0.7,"pith_summary":"This paper asks whether deliberately slow, step-by-step reasoning can make language models safer, not just smarter. It first stress-tests the o1 model against adversarial natural-language jailbreaks and prompts encoded in mathematical symbols, finding better overall safety than GPT-4o but persistent failures on math-encoded attacks. It then tries to transfer System-2 alignment to open-source 7B-8B models by prompting, supervised fine-tuning, direct preference optimization, and reinforcement learning. Across those experiments the consistent result is that teaching or instructing a model to scrutinize user requests raises its refusal of harmful content, sometimes at the cost of over-refusing benign requests. The paper argues this is a viable complement to alignment methods that assume strong reasoning is already present.","feed_headline":"Step-by-step safety reasoning makes LLMs harder to jailbreak","feed_subtitle":"Prompt and fine-tune models to scrutinize requests and harmful outputs drop, but math-encoded attacks still slip through.","key_machinery":"System-2 alignment is operationalized as chain-of-thought safety analysis. The model is asked, through prompts or fine-tuning, to produce steps such as identifying potential hazards, analyzing the user's intended use, and categorizing the request as Allowed, Disallowed, or Safe Completion before giving a final answer. In supervised fine-tuning this reasoning is distilled from a teacher model using hand-written safety specifications based on a harm taxonomy; in reinforcement learning, a reward model scores final answers, and the paper proposes a process reward model that scores each reasoning step. The mechanism is the analysis step itself: it slows the model down and makes the safety decision depend on explicit evidence rather than the model's first associative response.","core_discovery":"The paper's central claim is that System-2 alignment, steering a model to evaluate user intent, weigh risks, and reason before answering, improves safety even in models that do not natively reason at length. Evidence comes from WildJailbreak adversarial prompts: adding a safe-alert or few-shot analysis raises the not_unsafe rate, for example Mistral-7B from 0.186 to 0.695 with few-shot chain-of-thought prompting. Supervised fine-tuning with distilled reasoning steps improves both safety and over-refusal for Mistral-7B and Llama3-8B, and adding reinforcement learning gives the most balanced result. The same scrutiny, however, does not yet defeat math-encoded jailbreaks: o1 refuses only 15% of these without a safety alert, and the fine-tuned open models still answer such prompts as benign academic requests. The paper locates the vulnerability in the model's reasoning chain itself, where a step can misread the request as safe and thereby enlarge the attack surface.","pith_inferences":["My inference: if the safety gains come from genuine deliberation rather than style matching, the methods should transfer to attacks never seen during training, such as new encodings, new languages, or novel harm categories; measuring that out-of-distribution transfer would separate mechanism from memorization.","My inference: since a GPT-4-family model is both the teacher that distills reasoning and the judge that scores safety, a human-annotated evaluation could reveal whether the apparent improvements are absolute or an artifact of judge preference for reasoning-shaped answers.","My inference: math-encoded jailbreaks succeed because the model treats the request as a formal proof task, so a concrete testable extension is to force the safety taxonomy to run before any mathematical solution, even when the prompt appears purely academic."],"forward_implications":["Chain-of-thought prompting with safety alerts improves refusal of harmful prompts across Mistral-7B, Qwen-7B, and Llama3-8B, though often with more over-refusals on benign requests.","Supervised fine-tuning on reasoning-annotated safety data improves both safety and helpfulness more than fine-tuning on raw responses, with average scores rising from 0.752 to 0.883 for Mistral-7B and from 0.760 to 0.905 for Llama3-8B.","Reinforcement learning with outcome supervision balances safety and helpfulness better than direct preference optimization, which tends to make the model reject too many benign requests.","Mathematical encoding remains a blind spot: even with added safety alerts, the o1 model only reaches 0.8 not_unsafe on math-encoded attacks, and fine-tuned open models classify such prompts as safe academic requests.","Because the attack can hide inside a reasoning chain, the authors propose process supervision, rewarding each intermediate step, as the next step toward safer System-2 alignment."],"supporting_citations":[{"why":"Supplies the WildJailbreak adversarial benign and harmful prompts used for evaluation and fine-tuning.","marker":"[3]"},{"why":"Supplies the mathematical-encoding jailbreak construction used in the o1 and open-model tests.","marker":"[1]"},{"why":"Provides the o1 characterization connecting slow reasoning to robustness, which motivates the study.","marker":"[6]"},{"why":"Presents the deliberation-based alignment approach that this paper positions itself as complementing.","marker":"[2]"},{"why":"Provides the safety category definitions used to label the SFT training data.","marker":"[8]"},{"why":"Supplies the step-level reward collection method that the process-supervision proposal builds on.","marker":"[10]"},{"why":"Supplies the automated process-supervision approach referenced for training process reward models.","marker":"[5]"},{"why":"Supplies the self-play iteration structure for the proposed reinforcement-learning framework.","marker":"[12]"}],"fun_headline_variants":["Teaching LLMs to think before answering cuts jailbreaks","Reason-before-answer prompts harden LLMs, but math attacks persist","System-2 alignment: safer LLMs via scrutiny, yet math jailbreaks remain","Scrutinize requests, not just obey: LLM safety boost","Cultivate reasoning, not commands: safer models, math gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4-family judge and teacher labels reflect true safety and harmfulness; if those judgments are biased, the measured safety improvements could be style matching rather than real safety.","fun_headline_variants_meta":{"raw":{"variants":["Teaching LLMs to think before answering cuts jailbreaks","Reason-before-answer prompts harden LLMs, but math attacks persist","System-2 alignment: safer LLMs via scrutiny, yet math jailbreaks remain","Scrutinize requests, not just obey: LLM safety boost","Cultivate reasoning, not commands: safer models, math gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000535,"raw_usage":{"total_tokens":2566,"prompt_tokens":933,"completion_tokens":1633,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":1539}},"tokens_in":549,"tokens_out":1633,"duration_ms":11456,"temperature":1.0,"reasoning_tokens":1539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:32:11.730041+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect human-annotated safety labels for the same WildJailbreak and math-encoded test prompts, then score the SFT-CoT and RL models against those labels; if human agreement with the model judge is low or the human-annotated safety gain vanishes, the central claim fails. A cheaper check: attack the fine-tuned models with a math-encoded prompt built from a harm category excluded from the SFT taxonomy and see whether the reasoning chain still flags it.","supporting_citations":[{"cited_title":"Openai o1 system card","cited_arxiv_id":null,"evidence_quote":"Provides the o1 characterization connecting slow reasoning to robustness, which motivates the study."},{"cited_title":"Math-shepherd: Verify and reinforce llms step-by-step without human annotations","cited_arxiv_id":null,"evidence_quote":"Supplies the step-level reward collection method that the process-supervision proposal builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the self-play iteration structure for the proposed reinforcement-learning framework."}],"review_version":1}