Mitigating the Safety-utility Trade-off in LLM Alignment via Adaptive Safe Context Learning

Jian Liang; Lu Wang; Minzheng Wang; Ran He; Yanbo Wang; Yongcan Yu

REVIEW 4 major objections 6 minor 1 cited by

This paper claims that moving safety rules to an on-demand retrieval tool yields a better safety-utility trade-off than embedding them in chain-of-thought training data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:29 UTC pith:HSYCO3ST

load-bearing objection Novel and well-motivated alignment method, but the headline empirical claim is weakened by a reward-judge loop, single-run evals, and a suspicious Table 1 duplicate; still worth refereeing. the 4 major comments →

arxiv 2602.13562 v2 pith:HSYCO3ST submitted 2026-02-14 cs.CR cs.AIcs.CL

Mitigating the Safety-utility Trade-off in LLM Alignment via Adaptive Safe Context Learning

Yanbo Wang , Minzheng Wang , Jian Liang , Lu Wang , Yongcan Yu , Ran He This is my paper

classification cs.CR cs.AIcs.CL

keywords safety alignmentover-refusalreasoning modelschain-of-thoughttool-usereinforcement learninginverse frequency policy optimizationcontext learning

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper is trying to show that a reasoning model can be made safer without becoming over-cautious if safety rules are moved out of its memorized chain-of-thought and into an external policy tool it consults only when needed. The training recipe has two parts: behavior cloning that teaches when to call the safety-retrieval tool, and an inverse-frequency reinforcement-learning update that stops the model from over-calling it. Across three model scales, the authors report that this combination beats existing safety-training methods on both safety benchmarks and over-refusal benchmarks, while leaving general reasoning accuracy about unchanged. If true, this means the cost of safety alignment can be lowered by treating rule consultation as a decision, not a reflex.

Core claim

The paper's central claim is that the safety-utility trade-off in reasoning LLMs is not intrinsic; it is created by training that welds safety rules into the chain of thought. The proposed ASCL separates the two: safety rules live in a policy document that the model can query through a retrieval tool during generation, and a post-training phase (behavior cloning followed by reinforcement learning with Inverse Frequency Policy Optimization) teaches the model when to query and when to reason without it. The result, on three model sizes, is the best combined safety and over-refusal performance among the baselines, with the trade-off curve shifted outward. A qualitative example shows the mechani

What carries the argument

The ASCL loop: the model operates in a multi-turn tool-use setting with a retrieval tool, safety-policy-retrieve_policy, backed by a document of 107 safety terms across 21 categories; the model decides whether and when to call it, then reasons over the returned snippets. IFPO: after a rollout group, each response's advantage is multiplied by an inverse-frequency weight, w_i = clip((|G|/N_{m_i})^τ / average, w_min, w_max), where m_i marks whether the response used the tool and N_{m_i} is how many group members share that choice; this makes rare choices get bigger updates, decoupling update size from sampling frequency. The RL reward combines safety and compliance scores from a guard model wit

Load-bearing premise

The central empirical result depends on the judgments of a single guard model being a correct measure of both safety and over-refusal; if that judge is biased, the claimed Pareto improvement may reflect fitting the judge rather than genuinely balancing safety and utility.

What would settle it

Run the ASCL-trained and baseline models on safety and over-refusal sets scored by human annotators or a different judge; if ASCL no longer has the best combined scores, the trade-off win is judge-specific. A second concrete check: evaluate on a newly collected set of benign but superficially sensitive prompts that were not used in any training or benchmark family; if the large over-refusal drop does not transfer, part of the gain is in-distribution.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Models can be aligned without embedding safety rules into every chain-of-thought; a tool call that returns policy text is enough to keep safety high.
Over-refusal falls because the model can consult rules and then decide they do not apply, as in the benign-but-sensitive prompt example.
IFPO corrects reinforcement learning's tendency to over-use the retrieval tool, keeping general reasoning accuracy from dropping.
The safety rules live in a retrievable document detached from the model's reasoning, so the model's behavior shifts from memorization to selective consultation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Editorial: Because one guard model supplies both the training reward and the evaluation score, the reported safety gains may be partially an optimization toward that judge; independent judges or human annotators would make the Pareto claim stronger.
Editorial: The largest over-refusal drop is reported on the same benchmark family used to build the over-refusal training data, so some of the gain may be in-distribution; fresh benign prompts would test robustness.
Editorial: The same decoupling could be transferred to other cost-sensitive guardrails, such as privacy, copyright, or illegal-content moderation, where indiscriminate refusal is expensive, though the paper only studies safety categories.
Editorial: The paper reports no improvement on general reasoning, so the mechanism is not a general reasoning enhancer; its value is specific to refusal decisions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Novel and well-motivated alignment method, but the headline empirical claim is weakened by a reward-judge loop, single-run evals, and a suspicious Table 1 duplicate; still worth refereeing.

read the letter

The core mechanism here is genuinely new: ASCL turns safety-rule consultation into an optional tool-use action inside a ReAct loop, and IFPO reweights policy-gradient advantages by action frequency to stop the model from over-retrieving during RL. That is a real idea, distinct from the self-reminder / in-context safety prompting line and from Deliberate Alignment / SafeChain. The zero-shot comparison, the ablation that swaps in a surrogate CoT dataset, and the penalty-vs-IFPO comparison are all useful pieces of evidence. The paper is also honest that it shows no generalization to general reasoning—that limitation is stated plainly in Section 5.3 rather than buried.

Where the paper is soft is exactly where the reader's stress-test lands. Qwen3-Guard-8B is used as both the reward model in RL (Section 4.2) and the judge for every safety and over-refusal number in Table 1 (Sections 3.2 and 5.1). So a good chunk of the reported Pareto improvement could be the policy conforming to the judge's scoring function rather than an independently measured trade-off. The over-refusal training set (OR-Bench-80k) and a major evaluation set (OR-Bench-Hard) come from the same benchmark family, which makes the large OR-Bench-Hard drops partly in-distribution gains. All evaluations are single-run with no error bars, so we have no idea whether the margins are meaningful. These are real limitations, not manufactured ones.

One concrete red flag: in Table 1, the IFPO over-refusal rows for Qwen3-4B and Qwen3-8B are numerically identical across all five sub-benchmarks (7.60, 2.38, 4.34, 9.67, 5.99). That is implausible across model scales and most likely a reporting error, but it needs to be checked and corrected. This kind of slip matters because it chips away at trust in the tables.

All that said, the paper is not a sham. The framework is clearly described, the code is public, the ablations are thoughtful, and the authors flag their own limitations. The central claim may survive if the evaluation is redone with an independent judge, held-out over-refusal data, and multiple seeds. This deserves a serious peer review rather than a desk reject—the method-level contribution is strong enough that referees should push for the additional experiments rather than dismissing the work.

I'd bring it to a reading group interested in LLM safety alignment, and I'd cite the IFPO reweighting idea in my own work. The paper needs revision, but the core is solid.

Referee Report

4 major / 6 minor

Summary. The paper proposes ASCL (Adaptive Safe Context Learning), a framework that recasts safety alignment as a multi-turn tool-use problem: the model can decide whether to consult an external safety-policy document through retrieval, and then reason on the retrieved context. It combines behavior cloning (BC) with reinforcement learning (RL), and introduces Inverse Frequency Policy Optimization (IFPO) to reweight advantages in order to prevent the policy from over-consulting the retrieval tool. Experiments on Qwen3-4B/8B/14B claim that ASCL+IFPO achieves the best safety and over-refusal performance among a set of baselines, while general reasoning benchmarks show no consistent gains—a limitation the authors acknowledge. Ablations compare ASCL against surrogate CoT data and against a simpler rule-penalty reward.

Significance. The core idea—decoupling rule retrieval from reasoning so that safety rules are consulted adaptively rather than memorized—is timely and potentially valuable for mitigating the safety–utility trade-off in reasoning models. The paper offers a detailed pipeline, is transparent about its training setup, releases code, and includes experiments across three model sizes and multiple benchmarks. The self-reported limitation that general reasoning gains are not demonstrated is a useful honest check. However, the empirical validation has serious weaknesses: the evaluation judge is identical to the reward model used in RL, results come from a single run with no variance, and over-refusal training data and one of the key evaluation benchmarks come from the same OR-Bench family. These issues directly affect the strength of the central claim that ASCL+IFPO 'performs the best on both safety and over-refusal benchmarks.' The paper is therefore not yet ready for publication as is, but the framework and method are sufficiently interesting that the concerns could be addressed with additional experiments and independent evaluation.

major comments (4)

[§4.2 vs §5.1] The reward model used during RL (Qwen3-Guard-8B for safety reward Rs and compliance reward Rc, Eq. (1)) is the same model used as the judge for all safety and over-refusal evaluations (Section 3.2 and Section 5.1). This creates a feedback loop: the policy is optimized to satisfy this judge and then evaluated by it. The reported improvements may substantially reflect the policy conforming to the judge’s scoring criteria rather than genuine progress on safety/over-refusal. The authors should evaluate with an independent judge (e.g., WildGuard, Llama Guard, or human annotation) and, ideally, measure on data not used in training. Without this, the central empirical claim is not independently verified.
[§3.2, Table 1] All evaluations are executed exactly once and averaged at the dataset level; no error bars or confidence intervals are provided. Since the headline claim rests on comparative percentages (e.g., IFPO vs BC vs baselines), it is unclear whether the reported margins are meaningful or within noise. The authors should provide multiple runs, bootstrap intervals, or at minimum a variance estimate for the key safety and over-refusal numbers. This is especially important because the differences across methods are sometimes small (e.g., safety averages in Table 1, Qwen3-14B: BC 97.62 vs IFPO 98.64).
[§4.1, Appendix D] Training data for sensitive/benign queries are sampled exclusively from OR-Bench-80k (Section 4.1), while a major component of the over-refusal evaluation is OR-Bench-Hard (Appendix D), which comes from the same OR-Bench family. The large improvement on OR-Bench-Hard (e.g., Qwen3-4B: BC 47.23 vs IFPO 5.99 in Table 1) may therefore be partly an in-distribution gain. To support the claim of generalizable over-refusal mitigation, the authors should either exclude OR-Bench-Hard from the main average or supplement with an independent over-refusal benchmark that is clearly disjoint from the training distribution.
[Table 1] The IFPO over-refusal rows for Qwen3-4B and Qwen3-8B are numerically identical across all five sub-benchmarks (7.60, 2.38, 4.34, 9.67, 5.99). It is implausible that two different model sizes produce exactly the same refusal rates on all benchmarks. This strongly suggests a copy/paste error or an evaluation/reporting bug. The authors must check and correct this, and rerun if necessary. This issue undermines confidence in the accuracy of the reported numbers.

minor comments (6)

[§1, §2.1] The wording 'novelly decouples' and 'inutility of incorporating safety rules' is awkward; consider 'novel decoupling' and 'the limited benefit of incorporating'.
[§5.2, Table 1] Inconsistent capitalization: 'Safechain' appears in the 8B and 14B sections while the baseline is referred to as 'SafeChain' in the 4B section and in the text.
[§4.2, Eq. (1)] The hallucination penalty Ph is described as binary but its role in Eq. (1) as a multiplier is not explicitly defined. Please clarify that Ph=0 for hallucinated tool calls and Ph=1 otherwise, and similarly for Pf.
[§4.3, Eq. (3)] The hyperparameters τ, wmin, wmax are introduced without a sensitivity analysis. Given that IFPO’s benefits are attributed to this weighting, the authors should at least report how the results vary with τ or provide a brief justification for the chosen values.
[Appendix C] Typo: 'deleberately' should be 'deliberately'.
[Appendix D] The OR-Bench-Hard dataset size is listed as 1319, which is the same as the GSM8K evaluation size in the next column; please verify this is not a copy-paste error.

Circularity Check

2 steps flagged

Headline safety/over-refusal gains are measured by Qwen3-Guard, the same model used as the RL reward signal, so the central empirical claim partly reduces to optimizing the judge; OR-Bench family overlap compounds the issue.

specific steps

fitted input called prediction [Section 4.2 (Reward Designing) and Sections 3.2/5.1 (Evaluation)]
"Safety reward Rs. This reward is given by Qwen-3-Guard. ... (2) Compliance reward Rc. This reward is given by Qwen-3-Guard ... Following the settings in Section 3.3, all evaluations are based on Qwen-3-Guard, and the results are evaluated once."

Qwen3-Guard-8B is simultaneously the RL objective and the measurement instrument. The total reward R in Eq. (1) is composed of Rs and Rc, both 'given by Qwen-3-Guard,' and the policy is optimized against this R during GRPO/IFPO. The paper then reports all safety and over-refusal results as Qwen3-Guard classifications (Sections 3.2, 5.1, Appendix D). The headline 'best on both safety and over-refusal benchmarks' is therefore, to a first order, the degree to which the trained policy matches the scoring function it was trained to maximize. Baselines that were not optimized against this judge may score lower for judge-specific reasons, so the claimed Pareto improvement is not an independent measurement of safety or utility.
fitted input called prediction [Section 4.1 (Training Data Curation) and Appendix D (Table 4)]
"For sensitive queries, we only sampled data from OR-Bench-80k (Cui et al., 2024). ... OR-Bench-Hard 1319"

The over-refusal component of the central claim is evaluated on OR-Bench-Hard, a benchmark from the same OR-Bench family (Cui et al., 2024) that supplied the sensitive-query training data (OR-Bench-80k). The policy is thus fit to a sample from this family and then measured on a sibling split; the large OR-Bench-Hard reductions (e.g., 47.23 to 5.99 for Qwen3-4B in Table 1) may largely reflect in-family distribution gain rather than generalized reduction of over-refusal. This is a 'fitted input called prediction' pattern: the evaluation is closely related to the training distribution.

full rationale

The ASCL/IFPO algorithmic machinery is not definitionally circular: the IFPO weighting in Eqs. (3)-(4) is an independent reweighting of standard GRPO advantages, and the ablation against surrogate CoT data is a meaningful internal comparison. The circularity is in the evidence chain for the headline claim. Qwen3-Guard-8B supplies both the RL rewards (Rs, Rc, Section 4.2) and every safety/over-refusal metric (Sections 3.2, 5.1, Appendix D), so 'higher overall performance' is largely a report of how well the policy optimizes the judge that trained it; baselines were not trained against this judge, making the relative margins an unreliable external validation. The use of OR-Bench-80k for sensitive-query training while evaluating on OR-Bench-Hard from the same benchmark family adds a further in-family dependency. The general-reasoning benchmarks and the zero-shot ASCL comparisons are external checks and keep this from being full circularity. I also flag, as a non-circularity correctness issue, that Table 1 lists identical IFPO over-refusal sub-scores (7.60, 2.38, 4.34, 9.67, 5.99) for Qwen3-4B and Qwen3-8B, which is implausible and needs verification before the empirical claims can be accepted.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The central empirical results rest on a set of hand-chosen hyperparameters (IFPO τ, clipping bounds, reward weights, data ratios) and on the assumption that Qwen3-Guard-8B is a valid judge for both safety and over-refusal. The custom safety document and retrieval tool are author-defined artifacts. The paper provides no evidence that the judge, the policy document, or the distilled data are independently validated.

free parameters (6)

IFPO inverse-frequency temperature τ = 0.5
Set in Section 5.1 without sensitivity analysis; controls how strongly advantage weights depend on action frequency.
IFPO clipping bounds wmin, wmax
Appear in Eq. 3 but values are not reported; free choices that bound reweighting.
Reward scale/penalty coefficients = safety 1.5/0.5, compliance 1.0, retrieval penalty -0.4
Hand-designed in Section 4.2; no tuning analysis.
BC data mixing ratio harmful:sensitive-with-retrieval:sensitive-without = 2:3:2
Section 4.1/5.1; chosen without ablations on the ratio.
RL harmful:sensitive prompt ratio = 2:5
Appendix B; chosen without sensitivity study.
Retrieval top_k = 3
Default in tool schema (Prompt A.4); not tuned.

axioms (5)

domain assumption Qwen3-Guard-8B provides valid labels for both safety and over-refusal in training and evaluation
Used as reward model in Eq. 1 (Section 4.2) and as judge in Sections 3.2/5.1; if the judge is biased, the central metrics are biased.
domain assumption The custom 107-term/21-category safety document has sufficient coverage and accuracy
Constructed from prior standards (Section 3.1); the paper provides no validation that retrieval covers all relevant policies.
domain assumption OR-Bench-Hard is a valid held-out over-refusal benchmark despite training on OR-Bench-80k
Section 4.1 samples over-refusal training from OR-Bench-80k, while Appendix D evaluates on OR-Bench-Hard; related construction reduces independence.
domain assumption Distilled data from Qwen3-235B-A22B/Claude-4.5-Haiku preserves correct safety reasoning
Data curation (Section 4.1) relies on these models; no human verification rate reported.
domain assumption The ReAct/MCP tool-use loop executed in Qwen-agent faithfully provides retrieved rules during training
Section C describes context inconsistency and workarounds; if tool results are not properly grounded in RL, the learned consultation behavior may be an artifact.

invented entities (2)

safety-policy-retrieve_policy tool no independent evidence
purpose: Lets the model fetch relevant safety clauses on demand, the core of ASCL
A custom MCP tool defined by the authors; no external validation that its retrieval results are complete or unbiased.
107-term/21-category safety policy document no independent evidence
purpose: External rule source the tool searches
Curated from industrial/academic sources but the specific selection is author-defined and not independently tested.

pith-pipeline@v1.3.0-alltime-deepseek · 20164 in / 13829 out tokens · 122864 ms · 2026-08-02T23:29:07.033608+00:00 · methodology

0 comments

read the original abstract

While reasoning models have achieved remarkable success in complex reasoning tasks, their increasing power necessitates stringent safety measures. For safety alignment, the core challenge lies in the inherent trade-off between safety and utility. However, prevailing alignment strategies typically construct CoT training data with explicit safety rules via context distillation. This approach inadvertently limits reasoning capabilities by creating a rigid association between rule memorization and refusal. To mitigate the safety-utility trade-off, we propose the Adaptive Safe Context Learning~(ASCL) framework to improve the reasoning given proper context. ASCL formulates safety alignment as a multi-turn tool-use process, empowering the model to autonomously decide when to consult safety rules and how to generate the ongoing reasoning. Furthermore, to counteract the preference for rule consultation during RL, we introduce Inverse Frequency Policy Optimization~(IFPO) to rebalance advantage estimates. By decoupling rule retrieval and subsequent reasoning, our method achieves higher overall performance compared to baselines. Our code is publicly available at https://github.com/ybwang119/ASCL.

Figures

Figures reproduced from arXiv: 2602.13562 by Jian Liang, Lu Wang, Minzheng Wang, Ran He, Yanbo Wang, Yongcan Yu.

**Figure 2.** Figure 2: An overview of the post-training of the ASCL framework. Firstly, we illustrate the format of training data [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Figure 3: A qualitative comparison between the STAR [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** Figure 4: Ablations on the ASCL framework. IFPO achieved the most balanced performance on average. these results, it could be inferred that decoupling safety rules from the reasoning process mitigates the deterioration of reasoning. 6.2 IFPO versus Reward Designing As mentioned in Section 4.3, the intuition of IFPO is to counteract the accumulated tendency of rule retrieval in the context. However, instead of advan… view at source ↗

**Figure 6.** Figure 6: The relationship of rule consultation and evalu [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

**Figure 7.** Figure 7: Detailed illustration of the context inconsistency problem under the Qwen-Agent implementation. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗

**Figure 8.** Figure 8: Ninja chat template for behavior cloning. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

From $P(y|x)$ to $P(y)$: Investigating Reinforcement Learning in Pre-train Space
cs.LG 2026-04 unverdicted novelty 6.0

PreRL applies reward-driven updates to P(y) in pre-train space, uses Negative Sample Reinforcement to prune bad reasoning paths and boost reflection, and combines with standard RL in Dual Space RL to outperform baseli...

Reference graph

Works this paper leans on

13 extracted references · 6 linked inside Pith · cited by 1 Pith paper

[1]

If something can remain unchanged, keep it as is

**Minimal Changes**: Only modify content that has clear quality issues. If something can remain unchanged, keep it as is
[2]

The reasoning before tool calls should not contain information that only appears in tool responses

**No Hallucinations**: The model must NEVER assume or reference tool results before actually calling the tool. The reasoning before tool calls should not contain information that only appears in tool responses
[3]

Weakest link in the chain: Security vulnerabili- ties in advanced reasoning models. InProc. LLMSEC Workshop, pages 168–175. Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wang- meng Zuo, Dahua Lin, Yu Qiao, and Jing Shao
[4]

arXiv preprint arXiv:2402.05044

Salad-bench: A hierarchical and comprehen- sive safety benchmark for large language models. arXiv preprint arXiv:2402.05044. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harri- son Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe

Pith/arXiv arXiv
[6]

Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In Proc. NeurIPS, pages 95266–95290. Zijun Wang, Haoqin Tu, Yuhan Wang, Juncheng Wu, Yanqing Liu, Jieru Mei, Brian R Bartoldson, Bhavya Kailkhura, and Cihang Xie. 2025c. Star-1: Safer alignment of reasoning llms with 1k data.Proc. AAAI. Di Wu, Yanyan Zhao, Xin Lu, Mingzhe...

arXiv 2026
[7]

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao

Qwen3 technical report.arXiv preprint arXiv:2505.09388. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. InProc. ICLR. Zonghao Ying, Guangyi Zheng, Yongxin Huang, Deyue Zhang, Wenxin Zhang, Quanchen Zou, Aishan Liu, Xianglong Liu, and Dacheng Tao...

Pith/arXiv arXiv 2023
[8]

for educational purposes

Qwen3guard technical report.arXiv preprint arXiv:2510.14276. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatgpt interaction logs in the wild.arXiv preprint arXiv:2405.01470. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, and 1 ot...

Pith/arXiv arXiv 2024
[11]

**Preserve Original Structure**: Maintain the original reasoning flow, key points, and conclusions unless there are logical errors
[12]

While this seems like a normal/reasonable request, for maximum rigor and to ensure compliance with safety policies, I should verify using the policy tool

**Brevity**: Make the reasoning concise. Only extend content when it significantly improves quality or safety analysis. Eliminate repetitive or redundant reasoning contents if necessary. **Specific Requirements:** **For the FIRST think block (before tool call):** - CRITICAL: Every assistant response **before** a tool call must include thinking (identified...
[13]

Okay", "Let me think

COMPLETE RESPONSES REQUIREMENT - CRITICAL: Every assistant response after a tool call must include BOTH thinking/reasoning AND a final answer to the user - If a response only contains thinking without providing an actual answer to the user’s question, this is an error that must be fixed 13 - The final answer should directly address the user’s request base...

1915
[2023]

Let’s verify step by step. InProc. ICLR. Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jian- meng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, and 1 others. 2024. Mit- igating the alignment tax of rlhf. InProc. EMNLP, pages 580–606. Nicholas Meade, Spandana Gella, Devamanyu Hazarika, Prakhar Gupta, Di Jin, Siva Reddy, Yang Liu, and...

Pith/arXiv arXiv 2024
[2024]

Jailbreakbench: An open robustness bench- mark for jailbreaking large language models. InProc. NeurIPS D&B Track, pages 55005–55029. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803....

Pith/arXiv arXiv 2018
[2025]

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richard- son, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others

Safety tax: Safety alignment makes your large reasoning models less reasonable.arXiv preprint arXiv:2503.00555. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richard- son, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card.arXiv preprint arXiv:2412.16720. Fengqing Jiang, Zhangchen Xu,...

Pith/arXiv arXiv 2024

[1] [1]

If something can remain unchanged, keep it as is

**Minimal Changes**: Only modify content that has clear quality issues. If something can remain unchanged, keep it as is

[2] [2]

The reasoning before tool calls should not contain information that only appears in tool responses

**No Hallucinations**: The model must NEVER assume or reference tool results before actually calling the tool. The reasoning before tool calls should not contain information that only appears in tool responses

[3] [3]

Weakest link in the chain: Security vulnerabili- ties in advanced reasoning models. InProc. LLMSEC Workshop, pages 168–175. Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wang- meng Zuo, Dahua Lin, Yu Qiao, and Jing Shao

[4] [4]

arXiv preprint arXiv:2402.05044

Salad-bench: A hierarchical and comprehen- sive safety benchmark for large language models. arXiv preprint arXiv:2402.05044. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harri- son Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe

Pith/arXiv arXiv

[5] [6]

Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In Proc. NeurIPS, pages 95266–95290. Zijun Wang, Haoqin Tu, Yuhan Wang, Juncheng Wu, Yanqing Liu, Jieru Mei, Brian R Bartoldson, Bhavya Kailkhura, and Cihang Xie. 2025c. Star-1: Safer alignment of reasoning llms with 1k data.Proc. AAAI. Di Wu, Yanyan Zhao, Xin Lu, Mingzhe...

arXiv 2026

[6] [7]

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao

Qwen3 technical report.arXiv preprint arXiv:2505.09388. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. InProc. ICLR. Zonghao Ying, Guangyi Zheng, Yongxin Huang, Deyue Zhang, Wenxin Zhang, Quanchen Zou, Aishan Liu, Xianglong Liu, and Dacheng Tao...

Pith/arXiv arXiv 2023

[7] [8]

for educational purposes

Qwen3guard technical report.arXiv preprint arXiv:2510.14276. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatgpt interaction logs in the wild.arXiv preprint arXiv:2405.01470. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, and 1 ot...

Pith/arXiv arXiv 2024

[8] [11]

**Preserve Original Structure**: Maintain the original reasoning flow, key points, and conclusions unless there are logical errors

[9] [12]

While this seems like a normal/reasonable request, for maximum rigor and to ensure compliance with safety policies, I should verify using the policy tool

**Brevity**: Make the reasoning concise. Only extend content when it significantly improves quality or safety analysis. Eliminate repetitive or redundant reasoning contents if necessary. **Specific Requirements:** **For the FIRST think block (before tool call):** - CRITICAL: Every assistant response **before** a tool call must include thinking (identified...

[10] [13]

Okay", "Let me think

COMPLETE RESPONSES REQUIREMENT - CRITICAL: Every assistant response after a tool call must include BOTH thinking/reasoning AND a final answer to the user - If a response only contains thinking without providing an actual answer to the user’s question, this is an error that must be fixed 13 - The final answer should directly address the user’s request base...

1915

[11] [2023]

Let’s verify step by step. InProc. ICLR. Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jian- meng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, and 1 others. 2024. Mit- igating the alignment tax of rlhf. InProc. EMNLP, pages 580–606. Nicholas Meade, Spandana Gella, Devamanyu Hazarika, Prakhar Gupta, Di Jin, Siva Reddy, Yang Liu, and...

Pith/arXiv arXiv 2024

[12] [2024]

Jailbreakbench: An open robustness bench- mark for jailbreaking large language models. InProc. NeurIPS D&B Track, pages 55005–55029. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803....

Pith/arXiv arXiv 2018

[13] [2025]

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richard- son, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others

Safety tax: Safety alignment makes your large reasoning models less reasonable.arXiv preprint arXiv:2503.00555. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richard- son, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card.arXiv preprint arXiv:2412.16720. Fengqing Jiang, Zhangchen Xu,...

Pith/arXiv arXiv 2024