{"id":"fe516238-8899-4656-b901-cb16b424fba1","arxiv_id":"2505.21784","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-agent deliberation and refinement pipeline generates policy-embedded chain-of-thought data that improves safety fine-tuning and jailbreak resistance in open LLMs.","lead":"This paper introduces AIDSAFE, a multi-agent process that writes step-by-step safety reasoning chains for training language models to follow safety policies. Fine-tuning open models on these chains improved safety generalization and jailbreak robustness in the reported evaluations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All headline safety and quality numbers rest on unvalidated LLM judges; grader bias toward policy-reasoning outputs could explain part of the improvements, so the central claim is not yet fully supported.","rationale":"The most load-bearing premise of the paper's empirical argument is that the automated judges used throughout are valid proxies for safety and reasoning quality. The full experimental pipeline—from CoT quality (Table 1) to downstream safety (Table 2) and DPO outcomes (Table 5)—is scored exclusively by ShieldGemma-9B and Claude-3 Sonnet, with no human validation or inter-annotator agreement reported. The paper does include a Limitations section, but it omits this limitation, so the omission is not an artifact of the review pipeline. I agree with the reader's identification of this as the weakest assumption. A concrete validation study would settle whether the concern lands: if human labels reproduce the SFT_DB improvements, the central claim holds; if not, the large reported effects (e.g., WildChat safety 31→86) are suspect. Given the paper's otherwise careful data-generation description and external benchmarks, the appropriate verdict remains CONDITIONAL, pending such validation and code release. I therefore recommend UNCHANGED.","tokens_in":22395,"tokens_out":5362,"duration_ms":54103,"concrete_test":"Select a stratified random sample of 250 responses per condition (Base, SFT_OG, SFT_DB) from each of BeaverTails, WildChat, and StrongREJECT evaluations, plus 250 AIDSAFE and LLM_ZS CoT pairs. Have 3 trained annotators (with a safety-policy rubric) label response safety and CoT quality/faithfulness, and compute human-judge agreement (Cohen's kappa) and re-estimate the safety and quality gaps using human labels. If the SFT_DB-vs-baseline improvements do not survive human scoring, or if judge-human agreement is low, the central claim is not supported and the paper must add human-validated evaluation before acceptance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that SFT on AIDSAFE-generated CoTs significantly improves safety generalization and jailbreak robustness—depends on the validity of the automated judges used to measure every outcome. Safety rates on BeaverTails, WildChat, and StrongREJECT are computed using ShieldGemma-9B (Appendix C.1); CoT quality, faithfulness, over-refusal, and MMLU correctness are graded by Claude-3 Sonnet (with Command used only for pairwise CoT preference). No human-annotated validation set, agreement statistic, or alternative-judge robustness check is reported. This matters because SFTDB models are explicitly trained to produce policy-reasoning language, which may inflate judge scores: ShieldGemma may classify responses containing explicit policy citations or refusal boilerplate as 'safe' even when the actual helpfulness or safety is unchanged, and Claude-3 Sonnet may favor deliberation-style CoTs on stylistic grounds. The Limitations section (Section 8) notes issues such as policy coverage and deliberation interruptions, but it does not address the absence of judge validation. Since the effect sizes are large (e.g., Mixtral WildChat safety 31.00% to 85.95%), grader bias would have to be substantial to erase the qualitative conclusion, but the paper provides no evidence that it is not. Thus the empirical support for the abstract's claim is conditional on independent validation that the judges track human judgments in these settings.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AIDSAFE, a multi-agent deliberation framework for generating policy-embedded chain-of-thought (CoT) data for LLM safety training. The framework has three stages: intent decomposition and seed CoT generation, iterative deliberation between two agents over a fixed set of safety policies, and a refiner agent that filters redundant or deceptive reasoning. The authors also introduce an 'ear-whisperer' recipe for creating preference data for Direct Preference Optimization (DPO) by prepending adversarially learned 'bad beliefs' to the target LLM when sampling rejected CoTs. The paper evaluates the generated data with LLM auto-graders and then fine-tunes Mixtral-7B and Qwen2.5-7B on AIDSAFE-generated CoTs, reporting safety, over-refusal, utility, and jailbreak robustness results. The central claim is that SFT on AIDSAFE CoTs significantly improves safety generalization and jailbreak robustness while maintaining acceptable utility and over-refusal accuracy.","tokens_in":22793,"tokens_out":4571,"duration_ms":43378,"significance":"If the reported results hold, the paper offers a scalable and relatively low-cost data-generation recipe for safety reasoning, with the dataset released publicly. The evaluation across two base models and several benchmarks (BeaverTails, WildChat, XSTest, MMLU, StrongREJECT) is a strength, as is the use of two different LLM graders for pairwise CoT comparisons. However, the central empirical claims rest on automated judges (ShieldGemma-9B for safety, Claude-3 Sonnet for CoT quality, over-refusal, and MMLU correctness) that are not validated against human judgments, and the effect sizes are large (e.g., WildChat safety from 31.00% to 85.95%). The absence of human validation, single-run results, and missing tabulated numbers for the key SFTZS baseline mean that the results are promising but conditional on further evidence.","major_comments":[{"comment":"All safety rates in Table 2 are computed with ShieldGemma-9B at a fixed harm threshold of 0.5, and no human-annotated validation or agreement statistic is reported. Because the SFTDB models are trained to emit policy-reasoning language, ShieldGemma may systematically classify such outputs as safe regardless of objective safety, so the headline improvement on WildChat (31.00% to 85.95%) could be in part a grader artifact. Please validate on a human-annotated subset of responses (e.g., 100-200 per condition) or report agreement with a second independent moderation model; this is necessary to support the abstract's safety-generalization claim.","section":"Appendix C.1 (safety evaluation)"},{"comment":"CoT quality (relevance, coherence, completeness), faithfulness, over-refusal, and MMLU correctness are all graded by Claude-3 Sonnet, with Command used only for pairwise CoT preference. No inter-annotator agreement with human raters or correlation with an alternative grader is reported. Since the same LLM family is used both to generate the training data and to grade its quality, stylistic preference could inflate the apparent advantage of AIDSAFE CoTs over LLMZS CoTs; please add a second judge or a human-annotated sample to rule out this bias.","section":"Section 3.2 and Appendix B"},{"comment":"The comparison against single-LLM CoT data (SFTZS) is presented only as a figure, so the exact safety, over-refusal, and utility numbers for SFTZS are not available in the paper. This is the key ablation that isolates the contribution of the multi-agent deliberation and the refiner, and it should be reported as a table with the same metrics as Table 2, without which the claim that AIDSAFE improves over single-LLM generation is not fully supported.","section":"Section 4.3 and Figure 3"},{"comment":"All SFT results appear to be from a single run with a fixed data subsample and fixed hyperparameters (e.g., LoRA r=256, alpha=128, 3 epochs), with no variance reporting. Given the size of the reported effects and the 9:1 data split, reporting means and standard deviations over at least three seeds, or a bootstrap over the evaluation set, would materially strengthen the claim that the improvements are stable rather than artifacts of a particular initialization or data draw.","section":"Appendix C.2 and Table 2"}],"minor_comments":[{"comment":"The dataset name is inconsistent: 'Alpagsus' appears in several places, while the correct name 'Alpagasus' is used in the reference; please unify the spelling throughout.","section":"Section 4.1 and Appendix A.4"},{"comment":"The Helpfulness and Respectfulness Policy description is identical to the Hate-Harass-Violence Policy description; this appears to be a copy-paste error and should be corrected.","section":"Appendix A.1"},{"comment":"The Table 5 header contains a typo ('Eval Dimention'), and the DPO experiments are reported only for Mixtral; please note this limitation explicitly in the main text.","section":"Appendix D.2 and Table 5"},{"comment":"The code is described as to be 'released upon acceptance'; please include the code or a detailed configuration file with the final version to support reproducibility of the data-generation pipeline and training runs.","section":"Appendix A.3 and A.5"},{"comment":"Figure 3 is difficult to read at the current size; consider adding numerical labels to the points or moving the underlying values into a table.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is from an industrial lab and the dataset is publicly released, which is a positive feature. The main issue is the lack of human validation for the automated judges, which is load-bearing for the abstract's claims. I recommend requesting a human-annotated sample and a second judge as conditions for acceptance, rather than rejecting outright, since the core methodology is sound and the effects are large. There is no apparent novelty-disclosure concern beyond the usual need to cite concurrent work on safety reasoning and multi-agent deliberation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper worth your attention is AIDSAFE, a recipe for generating policy-embedded chain-of-thought data by having LLM agents deliberate, then a refiner clean up the output. The combination—intent decomposition, iterative multi-agent deliberation, refiner, plus a belief-augmentation 'ear-whisperer' for preference data—is new, and the paper shows it works in the sense that matters: SFT on this data improves in-domain and out-of-domain safety, jailbreak robustness, and over-refusal behavior compared to both base models and standard safety SFT. The Mixtral numbers are striking: WildChat safety jumps from 31% to 86%, StrongREJECT from 51% to 94%. Qwen, already safety-trained, also improves and, notably, does not show the degradation that plain SFT on safe responses causes. That is a useful result for open-source teams who cannot afford deliberative alignment at inference time.\n\nCredit: the recipe is described in enough detail to reproduce, the dataset is released, and the ablations (deliberation vs. refinement, SFT on zero-shot CoTs) support the claim that the iterative deliberation is doing the work. The paper is honest about limitations, including policy coverage and the ear-whisperer's weakness against already-safe models.\n\nSoft spots: the evidence is thinner than the headline numbers look. Every quality and safety measurement comes from an LLM judge—ShieldGemma for safety, Claude-3 Sonnet for over-refusal, MMLU grading, CoT quality, and faithfulness. There is no human-annotated validation set or agreement statistic. If ShieldGemma or Sonnet systematically prefer policy-reasoning style over substance, some of the gap could be judge bias. The effect sizes are large enough that it would take a lot of bias to erase the qualitative conclusion, but I would not call the claim fully established without a judge-validation study. The numbers are single-run; no error bars or seeds. Qwen's MMLU drops from 75.8 to 60.5 (SFTDB), which is not 'minimal regression' in absolute terms, even if it is far better than the traditional SFT drop to 55.7. The DPO section is thin: one model, one run, and the over-refusal accuracy falls from 91.8 to 80.7 after DPO, which the paper mentions but could emphasize more. Code is not yet available; the dataset is.\n\nWho it's for: anyone working on safety data generation, alignment, or jailbreak robustness. It is a solid incremental contribution, not a paradigm shift.\n\nRecommendation: send it to peer review. The central claim is plausible and the recipe is useful, but the review should require human validation of the judges, multiple seeds, and ideally the full SFTZS numbers in the main table. My own verdict would be conditional acceptance with those additions.","headline":"A practical data-generation recipe for safety-reasoning CoTs that mostly delivers on its claims, but the evidence would be stronger with human-validated judges and reported variance.","tokens_in":23214,"tokens_out":2356,"would_cite":true,"duration_ms":21732,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-agent deliberation pipeline can generate safety CoT data that, used for supervised fine-tuning, sharply lifts LLM safety generalization and jailbreak robustness while keeping utility and over-refusal costs modest.","keywords":["safety reasoning","chain-of-thought","multi-agent deliberation","jailbreak robustness","supervised fine-tuning","direct preference optimization","policy adherence","data generation"],"falsifier":"A human evaluation of a random sample of model outputs that finds ShieldGemma-9B's safety labels diverge from expert judgments on a substantial fraction of cases (or that Claude-3 Sonnet's CoT-quality judgments disagree with human rankings more than a minimal threshold) would falsify the claim that the SFT gains reflect true safety reasoning rather than grader alignment. Concretely, if expert annotators rate AIDSAFE-SFT and LLMZS-SFT responses equally safe on the same BeaverTails and StrongREJECT samples, the central claim collapses.","tokens_in":22189,"feed_emoji":"🛡️","tokens_out":13008,"duration_ms":95292,"temperature":0.7,"pith_summary":"The paper proposes AIDSAFE, a recipe for generating policy-embedded chain-of-thought (CoT) data in which multiple LLM agents debate how to respond to a user query under a set of written safety policies, and then a refiner agent compresses the deliberation into a concise CoT and response. The authors claim that fine-tuning open-source LLMs on this generated data significantly improves safety generalization and jailbreak robustness while keeping utility and over-refusal accuracy acceptable. They also introduce an 'ear-whisperer' agent that prepends deceptive beliefs to create clearly inferior 'rejected' examples for DPO preference learning. If the recipe works as reported, it offers a low-cost way for open-source models to acquire reasoning-style safety without expensive reasoning-capable teacher models.","feed_headline":"Agent-debated CoT data lifts jailbreak safety to 94%+","feed_subtitle":"Open models reach 94% safety on jailbreak prompts with limited utility and over-refusal trade-offs.","key_machinery":"The central mechanism is AIDSAFE's three-stage pipeline: (1) initialization with intent decomposition and a seed CoT-response pair; (2) a deliberation stage where two LLM agents alternately propose corrections and additions to the reasoning and response until one says 'I agree with previous agent' or a budget of three rounds is exhausted; and (3) a refinement stage where a third agent acts as an impartial judge, discarding repetitive, redundant, or deceptive thoughts and reordering the survivors into a concise first-person CoT, together with a revised response. The supplemental 'ear-whisperer' mechanism generates preference pairs by prepending iteratively refined deceptive belief prefixes to the target LLM, yielding rejected CoTs that genuinely violate policy while selected CoTs are sampled normally.","core_discovery":"The central discovery is that iterative multi-agent deliberation followed by a filtering refiner turns a single capable but not safety-specialized LLM (Mixtral 8x22B) into a generator of high-quality policy-embedded CoTs: the generated CoTs are more faithful to safety policies than single-shot LLM CoTs (policy faithfulness 4.27 vs 3.85 on a 1–5 scale) and win pairwise comparisons with two different auto-graders. Fine-tuning Mixtral-7B and Qwen2.5-7B on these CoTs lifts in-domain safety (e.g., Mixtral from 76% to 96% on BeaverTails) and, more strikingly, out-of-domain safety (from 31% to 85.95% on WildChat) and jailbreak robustness (from 51.09% to 94.04% on StrongREJECT), with only modest falls in MMLU accuracy and over-refusal performance. The paper also finds that standard sampling-based DPO preference data has nearly identical 'selected' and 'rejected' CoTs, and that an adversarial ear-whisperer agent using belief augmentation and iterative in-context learning produces rejected CoTs with genuinely worse policy adherence, creating a usable preference signal.","pith_inferences":["If AIDSAFE's gains hold up under human evaluation, the same deliberation-plus-refinement recipe could be exported to other policy domains — privacy, copyright, instruction hierarchies — since the framework is not tied to the five safety policies used here.","The dependence on auto-graders (Claude-3 Sonnet for reasoning quality, ShieldGemma for safety) as both data filters and evaluation metrics raises the possibility that part of the reported improvement is an alignment-of-graders effect; a human-annotation study on a random sample would be the natural check.","The ear-whisperer approach may be more broadly useful as adversarial data augmentation for alignment: by explicitly generating 'bad reasoning' examples, it targets the failure mode of models that give safe final answers via fabricated or policy-conflicting rationales.","Because the generator and refiner are the same model (Mixtral 8x22B), a testable extension is whether a weaker or stronger generator changes the quality frontier, which would identify which stage of the pipeline — deliberation or refinement — is doing the heavy lifting."],"forward_implications":["Supervised fine-tuning on AIDSAFE-generated CoTs can replace expensive reasoning-model distillation for safety reasoning; the generator is a single readily available Mixtral 8x22B model.","Safety gains generalize out of distribution: Mixtral's WildChat safe-response rate rises from 31% to 85.95% and StrongREJECT jailbreak safety from 51.09% to 94.04% after SFT on only 5,000 AIDSAFE CoTs.","Deliberation-driven data mitigates the 'safety fine-tuning overrides pretrained safety' failure: Qwen SFT on original responses drops from 94.14% to 87.95% BeaverTails safety, whereas SFT on AIDSAFE CoTs raises it to 97%.","Standard sampling produces indistinguishable selected and rejected CoTs for preference learning; the ear-whisperer's belief augmentation creates a measurable distribution shift, improving DPO jailbreak robustness (94.04% to 94.91%) and WildChat safety (85.95% to 94.22%) for Mixtral.","The data-generation recipe is practical at moderate compute: roughly 35 seconds per prompt on 4×A100 GPUs, with refinement cutting average CoT length from about 900 to about 600 tokens."],"supporting_citations":[{"why":"Supplies the 5,000 harmful prompts used for CoT generation and the original safe responses for the SFT-OG baseline.","marker":"(Ji et al., 2024)"},{"why":"Is the Mixtral 8x22B model running all AIDSAFE agents and the source of generated CoTs.","marker":"(Jiang et al., 2024)"},{"why":"Is ShieldGemma-9B, the safety judge producing every safe-response rate and scoring ear-whisperer beliefs.","marker":"(Zeng et al., 2024)"},{"why":"Provides the relevance/coherence/completeness rubric and the LLM-auto-grader evaluation methodology.","marker":"(Jacovi et al., 2024)"},{"why":"Defines the safety-reasoning paradigm AIDSAFE targets and the WildChat top-200 evaluation protocol.","marker":"(Guan et al., 2024)"},{"why":"Supplies the five safety policies and documents that extra safety SFT can degrade an already-aligned model.","marker":"(Qi et al., 2023)"},{"why":"Is the DPO objective that the ear-whisperer preference recipe is designed to feed.","marker":"(Rafailov et al., 2024)"},{"why":"Provides StrongREJECT, the jailbreak benchmark behind the robustness numbers.","marker":"(Souly et al., 2024)"},{"why":"Is the JAB belief-augmentation method adapted for the ear-whisperer's iterative in-context learning.","marker":"(Mehrabi et al., 2023)"}],"fun_headline_variants":["Agent-debated CoT lifts jailbreak safety to 94%","Multi-agent debate powers safer LLM reasoning","AI debate generates safer CoT data for LLMs","Deliberative agents trim jailbreak failures to 6%","Safety reasoning via agentic deliberation hits 94%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All headline safety, reasoning-quality, over-refusal, and utility numbers are produced by automated judges — Claude-3 Sonnet and Command for CoT quality and faithfulness, Claude-3 Sonnet for over-refusal and MMLU grading, and ShieldGemma-9B for safety labels — and no human agreement study is reported, so grader bias could account for part of the improvements.","fun_headline_variants_meta":{"raw":{"variants":["Agent-debated CoT lifts jailbreak safety to 94%","Multi-agent debate powers safer LLM reasoning","AI debate generates safer CoT data for LLMs","Deliberative agents trim jailbreak failures to 6%","Safety reasoning via agentic deliberation hits 94%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000821,"raw_usage":{"total_tokens":3658,"prompt_tokens":1073,"completion_tokens":2585,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":689,"completion_tokens_details":{"reasoning_tokens":2505}},"tokens_in":689,"tokens_out":2585,"duration_ms":19737,"temperature":1.0,"reasoning_tokens":2505,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:22:19.124265+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A human evaluation of a random sample of model outputs that finds ShieldGemma-9B's safety labels diverge from expert judgments on a substantial fraction of cases (or that Claude-3 Sonnet's CoT-quality judgments disagree with human rankings more than a minimal threshold) would falsify the claim that the SFT gains reflect true safety reasoning rather than grader alignment. Concretely, if expert annotators rate AIDSAFE-SFT and LLMZS-SFT responses equally safe on the same BeaverTails and StrongREJECT samples, the central claim collapses.","supporting_citations":[{"cited_title":"trainer","cited_arxiv_id":null,"evidence_quote":"Is the JAB belief-augmentation method adapted for the ear-whisperer's iterative in-context learning."}],"review_version":1}