Pith. sign in

REVIEW 4 major objections 6 minor 68 references

Zero-RL constructive safety training can match far larger models on safety while nearly preserving a 14B base model’s abilities and style.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 06:08 UTC pith:6WO3KMQD

load-bearing objection Practical Zero-RL recipe that lifts a 14B model to near-frontier constructive safety with clean ablations and a useful long-to-short generalization result; softest link is unvalidated LLM judges plus internal data. the 4 major comments →

arxiv 2607.02914 v1 pith:6WO3KMQD submitted 2026-07-03 cs.AI

Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models

Jiyang Guan , Yong Xie , Jun Chen , Jiexi Liu , Zipeng Ye , Defeng Li , Jiayu Shen , Jialing Tao
show 1 more author
Hui Xue
This is my paper
classification cs.AI
keywords constructive safety alignmentZero-RLlong-context safetyinstruction hierarchySERLover-refusalmulti-stage reinforcement learningLLM safety
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that refusal-based safety training leaves legitimate needs unmet, and that Oyster-I’s supervised constructive approach still generalizes poorly and over-applies safety reasoning to ordinary queries. Oyster-II instead applies reinforcement learning directly to the base model (Zero-RL) in a multi-stage curriculum, with length rewards to stop collapse into terse refusals, active sample filtering to cut reward noise, long-query safety data that transfers to short queries, and SERL to enforce developer-over-user instruction priority. The authors report safety scores that beat the 14B base and Oyster-I and reach levels comparable to models hundreds of times larger, while general capability and response-style metrics stay close to the base model. A central empirical claim is that training only on long-query safety data already yields strong short-query safety and higher response rates by moving beyond keyword matching. A reader who wants assistants that stay helpful on borderline requests, rather than blanket refusals, would care because the method aims to make safety and usefulness joint outcomes of the same training loop.

Core claim

Oyster-II shows that a Zero-RL multi-stage reinforcement pipeline for constructive safety—composite rewards that jointly gate safety, response rate, length, and format; active-learning difficulty control; long-context safety data with cross-length generalization; and SERL for instruction hierarchy—can surpass both Qwen3-14B and Oyster-I on safety benchmarks, reach performance comparable to much larger models, and do so non-invasively so that general capabilities and response style remain essentially intact.

What carries the argument

Zero-RL multi-stage training with SERL (Semi-Exploratory Reinforcement Learning with Prior-Guided Anchoring). Zero-RL means RL is run directly on the base model without intermediate supervised fine-tuning; SERL injects one offline gold-standard anchor into each rollout group so advantage signals never vanish under sparse safety rewards; length and benign-sample controls keep the policy from collapsing into uninformative refusals.

Load-bearing premise

The whole result rests on the assumption that the LLM-as-judge safety and hierarchy rewards, plus the chosen high-recall thresholds, are clean enough that the policy is learning real constructive safety rather than judge-specific artifacts.

What would settle it

Have independent human raters score the same long-query safety and instruction-conflict sets for Oyster-II, the base model, and Oyster-I; if human safety or constructive-helpfulness gains reverse while the automated judges still report large gains, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Safety alignment need not impose a large alignment tax on math, code, or style if Zero-RL replaces synthetic SFT traces.
  • Training only on long-query safety data can be sufficient for strong short-query safety and lower over-refusal.
  • Developer-over-user instruction hierarchy can be improved inside the same RL loop without forcing pure refusal.
  • Curriculum staging by language, length, and task type outperforms joint multi-task safety RL on the reported metrics.
  • Multiplicative rewards that zero out unsafe answers while still paying for response rate and length block safety-driven collapse into blanket refusal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If long-context training truly forces semantic rather than keyword safety, the same recipe may cut over-refusal in medical, legal, or other high-stakes advice domains.
  • Heavy dependence on LLM-as-judge rewards implies that further gains may be limited by judge quality more than by policy capacity.
  • Cross-length generalization from long to short (but not the reverse) suggests future safety datasets should prioritize long naturalistic contexts over ever-larger short-query corpora.
  • SERL’s anchor idea may transfer to other sparse-reward alignment problems—honesty, multi-agent priority, or tool-use constraints—where pure group-relative methods stall early.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript presents Oyster-II, a Zero-RL multi-stage reinforcement learning framework for constructive (response-oriented rather than refusal-oriented) safety alignment of LLMs. Building on Oyster-I, it targets two SFT limitations: weak OOD safety generalization and “safety CoT over-generalization” on benign queries. Core technical pieces are (1) a multiplicative composite reward (Eq. 1) with length-based entropy control and a benign-sample additive reward (Eq. 2); (2) SERL, a GSPO/GRPO-style mix-policy algorithm that injects a prior-guided gold anchor into each rollout group to avoid sparse-reward collapse; (3) a five-stage curriculum (long Chinese business → Chinese LSB/MSJ → English long → English short → instruction hierarchy) with active-learning sample filtering; and (4) long-query-only safety training claimed to transfer to short-query safety while reducing keyword-level over-refusal. Instruction hierarchy is trained and evaluated on a new InstructConflict-Bench (8 system-prompt scenarios × 3 attack levels). On public and internal suites (Table 4, Figs. 1, 13), a 14B model is reported to match or exceed Qwen3-14B and Oyster-I on safety, approach Qwen3-Max / Qwen3.5-397B on several safety axes, and largely preserve general capability and response-style consistency via Zero-RL.

Significance. If the empirical claims hold under independent scrutiny, the work is a solid contribution to constructive safety alignment. Strengths that deserve explicit credit include: extensive staged ablations isolating length reward, multi-stage curriculum vs joint training, active difficulty filtering, long- vs short-query data, and SERL vs GRPO (Figs. 3–9, 11–13); a clear non-invasive Zero-RL design that keeps general OpenCompass scores and style metrics near the base model (Table 4); and a useful practical finding that long-context safety training can improve short-query safety and response rates. SERL’s anchor mechanism is a concrete, transferable fix for sparse-reward collapse in non-verifiable safety RL. The instruction-hierarchy framing (Root > Developer > User) with a graded helpfulness rubric is timely for controllable deployment. These elements make the paper more than an incremental SFT-to-RL swap; they supply a reproducible training recipe and several falsifiable empirical claims about cross-length generalization and style preservation.

major comments (4)
  1. [§§2.2, 3.5, 4.2.1; Table 4] §§2.2, 3.5, 4.2.1 and the central non-invasive constructive-safety claim: safety, response-rate, and hierarchy rewards are produced by LLM-as-Judge models (plus Skywork). The only human calibration described is an unquantified iterative prompt-refinement for the hierarchy judge (§4.2.1); safety thresholds are chosen by F1/recall on the judge itself (§3.5) with no large-scale human correlation or inter-annotator agreement reported. Because multi-stage Zero-RL optimizes exclusively against these signals, and the largest gains (e.g., Chinese long-query Bullet-point Enumeration 97.05% and Summarization 99.20% vs much lower scores for Qwen3.5-397B in Table 4) occur on internal task-augmented business data that may share distributional features with the judges, it remains possible that the policy has learned judge-specific artifacts rather than human-preferred constructive behavior. A load-bea
  2. [Table 4; §5.1] Table 4 and §5.1: several of the strongest reported results (Model Specification safety/response rate, Chinese short- and long-query safety averages, and the six task-augmented long-query categories) rest on internal Alibaba AAIG datasets that are not released and are only partially described. Without public data, prompts, or a documented construction protocol comparable to InstructConflict-Bench, independent verification of the cross-scale claim (“comparable to Qwen3-Max and Qwen3.5-397B”) is not possible. At minimum, release evaluation prompts and scoring scripts for the internal suites, or re-run the same comparisons on fully public long-context safety benchmarks (e.g., LongSafety / LongSafetyBench / MSJ) with identical judge settings and report those numbers as the primary evidence.
  3. [§2.1, Eq. (1); Fig. 3] Eq. (1) and §2.1: the length score is written S_length = length / (target − length). As length approaches or exceeds the target this term diverges or becomes negative, which can invert the intended incentive and interact unpredictably with the multiplicative safety gate. The manuscript does not specify clipping, a soft target band, or the numerical target used in experiments. Clarify the exact functional form used in training (including any clipping), report sensitivity to the target hyperparameter, and confirm that the reported stage-1 response-rate gains in Fig. 3 are robust under a well-behaved alternative (e.g., a bounded length bonus).
  4. [§3.2–3.3; Fig. 8] §3.2–3.3 and Fig. 8: the claim that “training exclusively on long-query safety data is sufficient to achieve SOTA short-query safety” is central and interesting, but the short-query baselines shown are the authors’ own stage-1 short-query run, not a fully optimized short-query RL or SFT safety baseline under the same reward stack and compute. To support the cross-length generalization thesis as stated, add a matched-budget short-query-only RL control (same total tokens / same reward models) and report short- and long-query metrics for both, so the transfer claim is not confounded by under-training of the short-only arm.
minor comments (6)
  1. [Fig. 1] Fig. 1 caption and §5: WildChat is used as the representative Short-query Safety (EN) bar, but Table 4 also reports XSTest / StrongReject / DoNotAnswer. State explicitly in the figure caption which metric and judge are plotted so the radar is not misread as a single unified safety score.
  2. [§2.4; Fig. 7] §2.4 and Fig. 7: the SFT+RL comparison is informative, but the SFT data volume, epochs, and whether Lingo-BP/ORPO hyperparameters match Oyster-I are not stated. A short paragraph or appendix table would make the “Zero-RL preserves style better than SFT” claim easier to reproduce.
  3. [§4.2.3; Fig. 10] §4.2.3 / Fig. 10: SERL is described as built “upon GSPO”; a one-sentence formal difference from GRPO (advantage formula, KL term, whether the anchor is treated as on-policy for the importance ratio) would help readers implement the method without reverse-engineering the figure.
  4. [§5.3] §5.3 response-consistency metrics: the ad-hoc rescaling “Lrr divided by 3 and −KLp divided by 30” before averaging is under-motivated. Either justify the constants from the empirical ranges or report the four metrics separately without a composite average.
  5. [§3 title; §5.3; Fig. 5] Typos / polish: “Shot-context” in the §3 title; “Oy2” / “OysterII” / “Oyster-II” naming is inconsistent; “th››e” in §5.3; “active-leaning” in Fig. 5 caption. Standardize naming and fix OCR-like artifacts.
  6. [§6.4] Related work §6.2–6.4 is generally appropriate; a brief explicit comparison of SERL to other sparse-reward / offline-anchor RL methods (beyond GRPO) would situate the algorithmic contribution more clearly.

Circularity Check

0 steps flagged

Empirical multi-stage Zero-RL safety paper; no derivation reduces claimed benchmark gains to fitted inputs or self-citation by construction.

full rationale

Oyster-II is an empirical RL alignment paper. Its central claims (superior constructive safety on S-eval, WildChat, long-query Chinese/English suites, InstructConflict-Bench, and cross-scale parity with Qwen3-Max/Qwen3.5-397B while preserving general capabilities via Zero-RL) are measured outcomes of training against external composite rewards (Eq. 1 multiplicative safety/response/length/format; Eq. 2 additive Skywork+format-overrefusal+length; LLM-as-Judge hierarchy rubric) followed by evaluation on held-out or public benchmarks. No equation or theorem asserts that a safety score or response rate equals a fitted constant by definition. Ablations (length reward on/off, active difficulty control, multi-stage vs joint, long-query-only vs short, SERL vs GRPO, Zero-RL vs SFT) are comparative experiments, not circular reductions. Self-citation of Oyster-I supplies the constructive-safety paradigm and the Lingo-BP refinement tool used only for offline anchor construction; it is not invoked as a uniqueness theorem that forces the reported numbers. Reward-model threshold choices and judge calibration are methodological assumptions (correctness risk), not circular steps that make the evaluation scores tautological. The derivation chain is therefore self-contained against the reported benchmarks.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 3 invented entities

As an empirical alignment paper the load-bearing commitments are design choices (reward factors, filtering thresholds, curriculum order, judge models) rather than mathematical axioms. Free parameters are the numerical knobs that shape the reward and sample selection; invented entities are the named algorithmic constructs introduced here.

free parameters (5)
  • S_Response refusal soft-penalty (0.5)
    Hand-chosen multiplicative factor that trades off refusal vs. engagement; no sensitivity sweep reported.
  • Length-score target length
    Implicit target in S_length = length / (target - length); value not stated, yet controls entropy.
  • Active-learning N and keep-threshold (all-N-safe discard; later 6/8 full-score discard)
    Determines which samples enter the RL pool; chosen without ablation on N.
  • Safety-reward-model decision threshold (high-recall preference)
    Selected after F1 analysis; directly gates the binary S_safety multiplier.
  • SERL group size G=8 (7 online + 1 anchor)
    Fixed hyper-parameter of the hybrid rollout; affects advantage variance.
axioms (4)
  • ad hoc to paper Multiplicative composite reward enforces joint satisfaction of safety, response rate, length and format
    Eq. (1); design choice that any zero factor nullifies the whole reward.
  • domain assumption LLM-as-Judge scores are sufficiently accurate and unbiased proxies for true constructive safety and hierarchy compliance
    Used throughout §§2–4; only iterative prompt refinement against human agreement is mentioned.
  • domain assumption Zero-RL (RL directly on base model) preserves general capabilities better than SFT+RL
    Core methodological premise of §2.4; supported by one ablation (Fig. 7) but treated as given.
  • ad hoc to paper Curriculum ordering (long Chinese business → Chinese LSB/MSJ → English long → English short → hierarchy) is superior to joint multi-task
    §2.3; justified by one comparison (Fig. 6) and data-heterogeneity argument.
invented entities (3)
  • SERL (Semi-Exploratory RL with Prior-Guided Anchoring) no independent evidence
    purpose: Guarantee non-zero advantage in sparse-reward instruction-hierarchy groups by injecting one offline gold anchor per group
    Defined in §4.2.3; algorithmic novelty relative to GRPO.
  • Safety CoT over-generalization no independent evidence
    purpose: Name the failure mode of SFT constructive safety that Oyster-II claims to fix
    Introduced in abstract and §1 as diagnostic of Oyster-I.
  • InstructConflict-Bench (8 scenarios × 3 attack levels) no independent evidence
    purpose: Evaluate hierarchical instruction following under realistic developer–user conflicts
    Constructed in §4.1; new evaluation resource.

pith-pipeline@v1.1.0-grok45 · 30842 in / 3151 out tokens · 31372 ms · 2026-07-12T06:08:38.965104+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) have demonstrated remarkable capabilities across diverse applications, yet ensuring their simultaneous safety, helpfulness, and trustworthiness remains a persistent challenge. Conventional refusal-oriented alignment strategies mitigate harmful content generation but systematically fail to serve legitimate user needs, often withholding information that could safely and constructively address the underlying intent of sensitive queries. Building upon the constructive safety paradigm pioneered by Oyster-I, which moves beyond blanket refusal toward thoughtful, response-oriented safety alignment, we identify two critical limitations of its Supervised Fine-Tuning (SFT)-based scheme: insufficient safety generalization to out-of-distribution scenarios and a phenomenon we term safety chain-of-thought (CoT) over-generalization, wherein safety-oriented reasoning patterns are excessively applied to benign queries, degrading helpfulness and user experience. To address these limitations, we propose Oyster-II, a reinforcement learning (RL)-based constructive safety alignment framework that adopts a Zero-RL paradigm combined with a multi-stage reinforcement learning strategy.Evaluated across extensive benchmarks, Oyster-II comprehensively surpasses both Qwen3-14B and its predecessor Oyster-I on safety dimensions, achieving cross-scale performance comparable to Qwen3-Max and Qwen3.5-397B.

Figures

Figures reproduced from arXiv: 2607.02914 by Defeng Li, Hui Xue, Jialing Tao, Jiayu Shen, Jiexi Liu, Jiyang Guan, Jun Chen, Yong Xie, Zipeng Ye.

Figure 1
Figure 1. Figure 1: Performance of Oyster-II, Oyster-I, and Qwen3-14B, where WildChat is used as the repre [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training Pipeline of Oyster-II. Equipped with active-learning-based safety control, the zero-RL paradigm, and SERL, Oyster-II achieves state-of-the-art safety without compromising its helpfulness on both benign and malicious samples. To overcome these limitations, Oyster-II introduces a reinforcement learning (RL)-based framework for constructive safety alignment. By replacing static supervised signals wit… view at source ↗
Figure 3
Figure 3. Figure 3: Performance of Oyster-II with and without length reward. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Framework of active-learning-based sample selection. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of safety alignment with and without active-leaning based sample difficulty [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of safety alignment with and without multi-stage training. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of safety alignment with Zero-RL or SFT+RL. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Comparison of performance of Oyster-II safety alignment with short queries and long [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Performance of Oyster-II with and without involving benign samples during its safety [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Overall framework of SERL. Top: Standard GRPO pipeline. Middle: SERL replaces one online sample within the rollout group with an offline anchor o ∗ , and constructs a differential reward ∆ = R(o ∗ ) − R(oi) via Anchor-Aware Group Computation. Bottom: Offline construction pipeline of the gold-standard anchor—prior-injected rejection sampling yields the highest-scoring candidate o ⋆ ; if S = 1 it is adopted… view at source ↗
Figure 11
Figure 11. Figure 11: Training reward comparison between SERL and GRPO on the instruction hierarchy [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Evaluation performance on InstructConflict-Bench (8 scenarios [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Performance of Oyster-II on different stages. Q3 represents Qwen3-14B. S1-S5 represent [PITH_FULL_IMAGE:figures/full_fig_p023_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 29 linked inside Pith

  1. [1]

    Claude’s character

    Anthropic. Claude’s character. https://www.anthropic.com/research/ claude-character, 2024

    2024

  2. [2]

    Many-shot jailbreaking, 2024

    Anthropic. Many-shot jailbreaking, 2024. Research blog post

    2024

  3. [3]

    A general language assistant as a laboratory for alignment.arXiv preprint arXiv:2112.00861, 2021

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment.arXiv preprint arXiv:2112.00861, 2021

    Pith/arXiv arXiv 2021

  4. [4]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  5. [5]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Pith/arXiv arXiv 2022

  6. [6]

    Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Pith/arXiv arXiv 2022

  7. [7]

    LongAlign: A recipe for long context alignment of large language models

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. LongAlign: A recipe for long context alignment of large language models. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 1376–1395, Miami, Florida, USA, November 2024. Association for Computational Linguistics

    2024

  8. [8]

    Curriculum learning

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. InProceedings of the 26th Annual International Conference on Machine Learning (ICML), pages 41–48. ACM, 2009

    2009

  9. [9]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  10. [10]

    Think you have solved question answering? try ARC, the AI2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  11. [11]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  12. [12]

    Opencompass: A universal evaluation platform for foundation models.https://github.com/open-compass/opencompass, 2023

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models.https://github.com/open-compass/opencompass, 2023

    2023

  13. [13]

    Safe rlhf: Safe reinforcement learning from human feedback

    Juntao Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. InInternational Conference on Learning Representations, volume 2024, pages 50750–50777, 2024

    2024

  14. [14]

    Oyster-i: Beyond refusal–constructive safety alignment for responsible language models.arXiv preprint arXiv:2509.01909, 2025

    Ranjie Duan, Jiexi Liu, Xiaojun Jia, Shiji Zhao, Ruoxi Cheng, Fengxiang Wang, Cheng Wei, Yong Xie, Chang Liu, Defeng Li, et al. Oyster-i: Beyond refusal–constructive safety alignment for responsible language models.arXiv preprint arXiv:2509.01909, 2025

    arXiv 2025

  15. [15]

    Control illusion: The failure of instruction hierarchies in large language models

    Yilin Geng, Haonan Li, Honglin Mu, Xudong Han, Timothy Baldwin, Omri Abend, Eduard Hovy, and Lea Frermann. Control illusion: The failure of instruction hierarchies in large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 30816–30824, 2026

    2026

  16. [16]

    Evaluating the sensitivity of llms to harmful contents in long input.arXiv preprint arXiv:2510.05864, 2025

    Faeze Ghorbanpour and Alexander Fraser. Evaluating the sensitivity of llms to harmful contents in long input.arXiv preprint arXiv:2510.05864, 2025. 26

    Pith/arXiv arXiv 2025

  17. [17]

    Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. InProceedings of the 16th ACM workshop on artificial intelligence and security, pages 79–90, 2023

    2023

  18. [18]

    Deliberative alignment: Reasoning enables safer language models.arXiv preprint arXiv:2412.16339, 2024

    Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. Deliberative alignment: Reasoning enables safer language models.arXiv preprint arXiv:2412.16339, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Chinese simpleqa: A chinese factuality evaluation for large language models.arXiv preprint arXiv:2411.07140, 2024

    Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, Zhuoran Lin, Xuepeng Liu, Dekai Sun, Shirong Lin, Zhicheng Zheng, Xiaoyong Zhu, Wenbo Su, and Bo Zheng. Chinese simpleqa: A chinese factuality evaluation for large language models.arXiv preprint arXiv:2411.07140, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  22. [22]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. InAdvances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2021

    2021

  23. [23]

    Orpo: Monolithic preference optimization without reference model

    Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170–11189, 2024

    2024

  24. [24]

    Longsafety: Enhance safety for long-context llms.arXiv preprint arXiv:2411.06899, 2024

    Mianqiu Huang, Xiaoran Liu, Shaojun Zhou, Mozhi Zhang, Qipeng Guo, Linyang Li, Chenkun Tan, Yang Gao, Pengyu Wang, Linlin Li, Qun Liu, Yaqian Zhou, Xipeng Qiu, and Xuanjing Huang. Longsafety: Enhance safety for long-context llms.arXiv preprint arXiv:2411.06899, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  26. [26]

    Safe rlhf-v: Safe reinforcement learning from multi-modal human feedback.Advances in Neural Information Processing Systems, 38:46146–46182, 2026

    Jiaming Ji, Xinyu Chen, Rui Pan, Han Zhu, Jiahao Li, Donghai Hong, Boyuan Chen, Jiayi Zhou, Kaile Wang, Juntao Dai, et al. Safe rlhf-v: Safe reinforcement learning from multi-modal human feedback.Advances in Neural Information Processing Systems, 38:46146–46182, 2026

    2026

  27. [27]

    Safedpo: A simple approach to direct preference optimization with enhanced safety.arXiv preprint arXiv:2505.20065, 2025

    Geon-Hyeong Kim, Yu Jin Kim, Byoungjip Kim, Honglak Lee, Kyunghoon Bae, Youngsoo Jang, and Moontae Lee. Safedpo: A simple approach to direct preference optimization with enhanced safety.arXiv preprint arXiv:2505.20065, 2025

    arXiv 2025

  28. [28]

    On information and sufficiency.The Annals of Mathematical Statistics, 22(1):79–86, 1951

    Solomon Kullback and Richard A Leibler. On information and sufficiency.The Annals of Mathematical Statistics, 22(1):79–86, 1951

    1951

  29. [29]

    RACE: Large-scale ReAding comprehension dataset from examinations

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 785–794, 2017

    2017

  30. [30]

    Tülu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, et al. Tülu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Binary codes capable of correcting deletions, insertions, and reversals

    Vladimir I Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8):707–710, 1966

    1966

  32. [32]

    Model spec midtraining: Improving how alignment training generalizes.arXiv preprint arXiv:2605.02087, 2026

    Chloe Li, Sara Price, Samuel Marks, and Jon Kutasov. Model spec midtraining: Improving how alignment training generalizes.arXiv preprint arXiv:2605.02087, 2026. 27

    Pith/arXiv arXiv 2026

  33. [33]

    Optimizing safe and aligned language generation: A multi-objective grpo approach.arXiv preprint arXiv:2503.21819, 2025

    Xuying Li, Zhuo Li, Yuji Kosuga, and Victor Bian. Optimizing safe and aligned language generation: A multi-objective grpo approach.arXiv preprint arXiv:2503.21819, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    TruthfulQA: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. InProceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (ACL), pages 3214–3252, 2022

    2022

  36. [36]

    Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Longsafety: Evaluating long-context safety of large language models

    Yida Lu, Jiale Cheng, Zhexin Zhang, Shiyao Cui, Cunxiang Wang, Xiaotao Gu, Yuxiao Dong, Jie Tang, Hongning Wang, and Minlie Huang. Longsafety: Evaluating long-context safety of large language models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 31705–31725, Vienna, Austria, July

  38. [38]

    Association for Computational Linguistics

  39. [39]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2381– 2391, 2018

    2018

  40. [40]

    Enhancing llm safety through reasoningbased alignment.URL https://arxiv

    Yutao Mou, Yuxiao Luo, Shikun Zhang, and Wei Ye Saro. Enhancing llm safety through reasoningbased alignment.URL https://arxiv. org/abs/2504.09420, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    The model spec, 2024.https://model-spec.openai.com/

    OpenAI. The model spec, 2024.https://model-spec.openai.com/

    2024

  42. [42]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  43. [43]

    Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023

    2023

  45. [45]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    Pith/arXiv arXiv 2023

  46. [46]

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models

    Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers),...

    2024

  47. [47]

    Ignore this title and hackaprompt: Exposing systemic vulnerabilities of llms through a global prompt hacking competition

    Sander Schulhoff, Jeremy Pinto, Anaum Khan, Louis-François Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Kost, Christopher Carnahan, and Jordan Lee Boyd-Graber. Ignore this title and hackaprompt: Exposing systemic vulnerabilities of llms through a global prompt hacking competition. InProceedings of the 2023 Conference on Empirical Methods ...

    2023

  48. [48]

    thefuzz: Fuzzy string matching in Python, 2024

    SeatGeek. thefuzz: Fuzzy string matching in Python, 2024. 28

    2024

  49. [49]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    Openai gpt-5 system card.arXiv preprint arXiv:2601.03267, 2025

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card.arXiv preprint arXiv:2601.03267, 2025

    Pith/arXiv arXiv 2025

  51. [51]

    A strongreject for empty jailbreaks.Advances in Neural Information Processing Systems, 37:125416–125440, 2024

    Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, et al. A strongreject for empty jailbreaks.Advances in Neural Information Processing Systems, 37:125416–125440, 2024

    2024

  52. [52]

    DetectLLM: Leveraging log rank information for zero-shot detection of machine-generated text

    Jinyan Su, Terry Zhuo, Di Wang, and Preslav Nakov. DetectLLM: Leveraging log rank information for zero-shot detection of machine-generated text. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 12395–12412, Singapore, 2023. Association for Computational Linguistics

    2023

  53. [53]

    Investigating prior knowledge for challenging chinese machine reading comprehension.Transactions of the Association for Computational Linguistics (TACL), 8:141–155, 2020

    Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. Investigating prior knowledge for challenging chinese machine reading comprehension.Transactions of the Association for Computational Linguistics (TACL), 8:141–155, 2020

    2020

  54. [54]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V . Le, Ed H. Chi, Denny Zhou, and Jason Wei. Chal- lenging BIG-Bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261, 2022

    Pith/arXiv arXiv 2022

  55. [55]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, 2019

    2019

  56. [56]

    The instruction hierarchy: Training llms to prioritize privileged instructions.arXiv preprint arXiv:2404.13208, 2024

    Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training llms to prioritize privileged instructions.arXiv preprint arXiv:2404.13208, 2024

    Pith/arXiv arXiv 2024

  57. [57]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. InAdvances in Neural Information Processing Systems (NeurIPS)...

    2024

  58. [58]

    Do-not-answer: Evaluating safeguards in llms

    Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. Do-not-answer: Evaluating safeguards in llms. InFindings of the Association for Computational Linguistics: EACL 2024, 2024

    2024

  59. [59]

    Measuring short-form factuality in large language models

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024

    Pith/arXiv arXiv 2024

  60. [60]

    Benchmarking and defending against indirect prompt injection attacks on large language models

    Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Benchmarking and defending against indirect prompt injection attacks on large language models. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V . 1, pages 1809–1820, 2025

    2025

  61. [61]

    S-eval: Towards automated and comprehensive safety evaluation for large language models

    Xiaohan Yuan, Jinfeng Li, Dongxia Wang, Yuefeng Chen, Xiaofeng Mao, Longtao Huang, Jialuo Chen, Hui Xue, Xiaoxia Liu, Wenhai Wang, Kui Ren, and Jingyi Wang. S-eval: Towards automated and comprehensive safety evaluation for large language models. InInternational Symposium on Software Testing and Analysis (ISSTA), 2025

    2025

  62. [62]

    From hard refusals to safe-completions: Toward output-centric safety training.arXiv preprint arXiv:2508.09224, 2025

    Yuan Yuan, Tina Sriskandarajah, Anna-Luisa Brakman, Alec Helyar, Alex Beutel, Andrea Vallone, and Saachi Jain. From hard refusals to safe-completions: Toward output-centric safety training.arXiv preprint arXiv:2508.09224, 2025. 29

    Pith/arXiv arXiv 2025

  63. [63]

    Many-tier instruction hierarchy in llm agents.arXiv preprint arXiv:2604.09443, 2026

    Jingyu Zhang, Tianjian Li, William Jurayj, Hongyuan Zhan, Benjamin Van Durme, and Daniel Khashabi. Many-tier instruction hierarchy in llm agents.arXiv preprint arXiv:2604.09443, 2026

    Pith/arXiv arXiv 2026

  64. [64]

    Iheval: Evaluating language models on following the instruction hierarchy

    Zhihan Zhang, Shiyang Li, Zixuan Zhang, Xin Liu, Haoming Jiang, Xianfeng Tang, Yifan Gao, Zheng Li, Haodong Wang, Zhaoxuan Tan, et al. Iheval: Evaluating language models on following the instruction hierarchy. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Techno...

    2025

  65. [65]

    Wildchat: 1m chatgpt interaction logs in the wild

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chatgpt interaction logs in the wild. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  66. [66]

    Improving llm safety alignment with dual-objective optimization.arXiv preprint arXiv:2503.03710, 2025

    Xuandong Zhao, Will Cai, Tianneng Shi, David Huang, Licong Lin, Song Mei, and Dawn Song. Improving llm safety alignment with dual-objective optimization.arXiv preprint arXiv:2503.03710, 2025

    arXiv 2025

  67. [67]

    AGIEval: A human-centric benchmark for evaluating foundation models.arXiv preprint arXiv:2304.06364, 2023

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A human-centric benchmark for evaluating foundation models.arXiv preprint arXiv:2304.06364, 2023

    Pith/arXiv arXiv 2023

  68. [68]

    Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023. 30

    Pith/arXiv arXiv 2023