The reviewed record of science sign in
Pith

arxiv: 2607.06326 · v1 · pith:5E7YKNEJ · submitted 2026-07-07 · cs.AI

DT-Guard: Intent-Driven Reasoning-Active Training for Reasoning-Free LLM Safety Guardrail

He Liu , Changtao Miao , Xinjie Yang , Tianle Song , Yin Wu , Junchi Chen , Bintao He , Xinyuan Zhang
show 7 more authors
Bo Zhang Shi Yan Wei Lu Wei Wang Danyang Xu Jiansheng Cai Zhe Li
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 09:39 UTCglm-5.2pith:5E7YKNEJrecord.jsonopen to challenge →

classification cs.AI
keywords safety guardrailreasoning internalizationchain-of-thoughtdirect preference optimizationcontent moderationrollout consistencyhard-case optimization
0
0 comments X

The pith

4B guardrail beats 8B rivals by reasoning at training, not inference

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DT-Guard proposes a safety guardrail model that uses reasoning supervision during training while emitting only structured labels at inference time. The core claim is that explicit chain-of-thought reasoning, when used as a training signal rather than a deployment-time output, can be internalized into a smaller model to improve safety discrimination without adding latency. The paper formulates safety judgment as a progressive decision process—Intent → Category → Safety—and introduces a three-stage training pipeline called RG-PHO that uses multi-rollout consistency to identify stably mastered, persistently failed, and preference-unstable samples, then applies targeted supervised fine-tuning or contrastive preference optimization to each group. With a 4B-parameter backbone, DT-Guard achieves average F1 scores of 0.886 on prompt-side and 0.870 on response-side safety benchmarks, outperforming 8B guardrail baselines while requiring no reasoning-chain generation at deployment.

Core claim

The central mechanism is the separation of reasoning supervision from reasoning inference. During training, DT-Guard mixes chain-of-thought outputs with structured-label outputs, selectively assigning CoT to borderline and ambiguous cases while using compact labels for clear cases. After initial training, the model is rolled out three times per sample: samples that are correct 3/3 times are set aside, samples correct 0/3 times receive supervised correction with CoT, and samples correct 1/3 or 2/3 times become preference pairs for DPO. This rollout-consistency-based stratification routes each residual error to the optimization method best matched to its failure type. At inference, the model直接

What carries the argument

Reasoning-Active Training, Reasoning-Free Inference

If this is right

  • Smaller guardrail models trained with internalized reasoning could replace larger or slower reasoning-based guards in latency-sensitive production systems, reducing both compute cost and response time for real-time content moderation.
  • The rollout-consistency stratification method is domain-agnostic: it could be applied to any classification task where a model produces variable outputs across repeated generations, such as medical triage, legal risk assessment, or fraud detection.
  • The finding that selective CoT allocation to borderline cases outperforms both full-CoT and no-CoT training suggests that reasoning supervision is most valuable exactly where decision boundaries are ambiguous, not uniformly across all samples.

Load-bearing premise

The paper attributes its performance gains to the reasoning-active training paradigm and the RG-PHO pipeline, but the 811k-sample proprietary training dataset—built via distillation from a large model and expert verification—may itself be the primary driver. The ablation shows the baseline variant without intent or CoT already achieves a dual-side average F1 of 0.851, close to the 0.858 of an 8B baseline, making it difficult to isolate how much of the final 0.878 comes from

What would settle it

If a model trained on the same 811k dataset with standard SFT and no reasoning supervision, intent labels, or RG-PHO stages achieved comparable F1 scores, the core claim that reasoning internalization drives the improvement would be undermined.

Figures

Figures reproduced from arXiv: 2607.06326 by Bintao He, Bo Zhang, Changtao Miao, Danyang Xu, He Liu, Jiansheng Cai, Junchi Chen, Shi Yan, Tianle Song, Wei Lu, Wei Wang, Xinjie Yang, Xinyuan Zhang, Yin Wu, Zhe Li.

Figure 1
Figure 1. Figure 1: DT-Guard achieves the top F1 on representative safety benchmarks under reasoning [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Intent-driven data construction pipeline: heterogeneous safety sources are distilled, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of unsafe risk categories [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: RG-PHO training pipeline: mixed-mode SFT is followed by rollout-based hard-case [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
read the original abstract

Large language models deployed in open-world applications require safety guardrails that are both robust to complex risks and efficient enough for low-latency runtime moderation. Existing guardrails face a practical trade-off between lightweight classification-based models, which are efficient but often struggle with concealed intent, ambiguous semantics, and borderline safety decisions, and reasoning-based guards, which improve judgment quality but introduce additional token generation and inference latency. We present DT-Guard, a content safety guardrail model based on a Reasoning-Active Training, Reasoning-Free Inference paradigm. The key idea is to use reasoning supervision during training while emitting only structured safety labels at inference time. DT-Guard formulates safety judgment as a progressive decision process, Intent - Category - Safety, and constructs an intent-driven dataset with intent labels, risk categories, safety labels, and structured reasoning trajectories. To further improve hard-case robustness, we propose Rollout-Guided Progressive Hard-Case Optimization (RG-PHO), which uses multi-rollout consistency to identify stably mastered, persistently failed, and preference-unstable samples, and applies targeted supervised and preference optimization accordingly. At inference time, DT-Guard directly generates structured labels without explicit reasoning traces, preserving deployment efficiency. Experiments on prompt-side and response-side safety benchmarks show that DT-Guard achieves average F1 scores of 0.886 and 0.870, respectively. With only a 4B backbone, it reaches a dual-side average F1 of 0.878, outperforming strong 8B guardrail baselines. These results demonstrate that reasoning supervision can be effectively internalized into low-latency safety discrimination.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 7 minor

Summary. The paper presents DT-Guard, a 4B-parameter safety guardrail model that uses reasoning supervision (intent labels, chain-of-thought trajectories, hard-case SFT, and rollout-contrastive DPO) during training while emitting only structured labels at inference time. The core idea is to internalize reasoning without incurring inference-time latency. The authors construct an 811k-sample intent-driven corpus and propose Rollout-Guided Progressive Hard-Case Optimization (RG-PHO), a three-stage pipeline that stratifies training samples by multi-rollout consistency and applies targeted SFT or DPO accordingly. Experiments on 10 prompt-side and 7 response-side benchmarks show DT-Guard achieving a dual-side average F1 of 0.878, outperforming 8B baselines. The progressive ablation (Table 9) demonstrates that each RG-PHO stage contributes incremental gains over the Stage1-SFT-v0 baseline.

Significance. The reasoning-active training, reasoning-free inference paradigm addresses a genuine practical tension in deployed guardrails. The progressive ablation in Table 9 is a commendable design choice that isolates each component's contribution. The rollout-consistency-based sample stratification (§4.2) is a principled mechanism for routing hard cases to the appropriate optimization objective. The intent-driven decision structure (Intent → Category → Safety) provides a clean intermediate supervision signal. However, the significance of the methodological contribution is partially obscured by the confound between the proprietary 811k dataset and the RG-PHO pipeline itself, which the ablation structure does not fully resolve. Neither the dataset nor the code is publicly available, which limits independent verification.

major comments (2)
  1. §5.2, final paragraph: The claim that 'the gain therefore comes primarily from intent-driven supervision and RG-PHO rather than model scale' is supported by the comparison against larger baselines, but the ablation in Table 9 reveals a more nuanced picture that the paper does not adequately address. Stage1-SFT-v0 (no intent, no CoT, no RG-PHO) already achieves a dual-side average F1 of 0.851, which exceeds YuFeng-XGuard-Reason-8B (0.845) and nearly matches Qwen3Guard-8B (0.858). The entire RG-PHO pipeline adds +2.7 points (0.851 → 0.878). This means a substantial portion of the competitive advantage over 8B baselines is attributable to the 811k proprietary dataset alone, not to the reasoning-active training paradigm that the paper centers as its primary contribution. The paper should explicitly acknowledge this decomposition and reframe the headline claim accordingly. As currently stated
  2. §5.1–5.2, Tables 5–7: The baseline comparison conflates training data, model scale, and training method. The 8B baselines (Qwen3Guard-8B, YuFeng-XGuard-Reason-8B) are trained on their own proprietary data of unknown scale and quality. Without knowing whether these baselines had access to comparable data resources, the '4B beats 8B' framing cannot be cleanly attributed to either the dataset or the method. The paper should either (a) provide a baseline trained on the same 811k corpus with a comparable 8B backbone to isolate the method's contribution from data effects, or (b) explicitly state that the comparison conflates these factors and temper the attribution claims. This is load-bearing for the central claim that reasoning-active training is the primary driver of the competitive result.
minor comments (7)
  1. §4.2: The choice of K=3 rollouts is stated without justification. A brief discussion of why three rollouts suffice for reliable stratification, or a sensitivity analysis over K, would strengthen the rollout-consistency design.
  2. §4.4, Eq. (3): The value of β (preference strength) is not specified. This and other hyperparameters (learning rate, batch size, number of epochs per stage) should be reported for reproducibility.
  3. §3.2: The GLM-5.1 distillation prompts and the expert verification criteria are not described in detail. Providing the annotation schema or example prompts would aid reproducibility.
  4. §3.3: The target Safe:Unsafe:Borderline ratio of 5.5:4:0.5 is stated without justification. Was this ratio selected empirically? A brief rationale or ablation over alternative ratios would help.
  5. Figure 4: The diagram is dense and some labels are difficult to read. The flow of data through the three stages could be clarified by simplifying or enlarging the figure.
  6. Table 5: The prompt-side F1 values for some benchmarks (e.g., SimpST, HarmB) show near-perfect scores (0.995–1.000) across multiple models. A brief discussion of ceiling effects on these benchmarks would help contextualize the average F1 comparisons.
  7. The dataset and code are not publicly available. While proprietary constraints may prevent full release, providing evaluation scripts or a data sample would support independent verification of the reported results.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises two related points about attribution: (1) the Stage1-SFT-v0 baseline already achieves strong performance, suggesting the 811k dataset contributes substantially to the competitive advantage over 8B baselines, and (2) the baseline comparison conflates training data, model scale, and training method, making it impossible to cleanly attribute the '4B beats 8B' result to the reasoning-active training paradigm. Both points are well-taken. We agree that the current framing overstates the attribution to RG-PHO and does not adequately acknowledge the role of the dataset. We will revise the manuscript to explicitly decompose the contribution and temper the headline claim. We cannot fully resolve the data-scale confound with 8B baselines because we lack access to their proprietary training corpora, but we will state this limitation transparently.

read point-by-point responses

  1. Referee: §5.2, final paragraph: The claim that 'the gain therefore comes primarily from intent-driven supervision and RG-PHO rather than model scale' is supported by the comparison against larger baselines, but the ablation in Table 9 reveals a more nuanced picture that the paper does not adequately address. Stage1-SFT-v0 (no intent, no CoT, no RG-PHO) already achieves a dual-side average F1 of 0.851, which exceeds YuFeng-XGuard-Reason-8B (0.845) and nearly matches Qwen3Guard-8B (0.858). The entire RG-PHO pipeline adds +2.7 points (0.851 → 0.878). This means a substantial portion of the competitive advantage over 8B baselines is attributable to the 811k proprietary dataset alone, not to the reasoning-active training paradigm that the paper centers as its primary contribution. The paper should explicitly acknowledge this decomposition and reframe the headline claim accordingly.

    Authors: The referee is correct. The decomposition is as follows: Stage1-SFT-v0, which uses the 811k corpus with standard SFT and no intent labels, CoT, or RG-PHO, already achieves 0.851 dual-side average F1, exceeding YuFeng-XGuard-Reason-8B (0.845) and nearly matching Qwen3Guard-8B (0.858). The full RG-PHO pipeline then adds +2.7 points on top of this baseline (0.851 → 0.878). This means the competitive advantage over 8B baselines has two distinct sources: (a) the 811k intent-driven dataset, which accounts for the bulk of the gap relative to 8B baselines, and (b) the reasoning-active training paradigm (intent labels, selective CoT, hard-case SFT, rollout-contrastive DPO), which provides an additional +2.7 points. The current claim that the gain comes 'primarily from intent-driven supervision and RG-PHO rather than model scale' is imprecise because it does not distinguish between the dataset contribution and the method contribution. We will revise §5.2 to explicitly present this decomposition, acknowledge that a substantial portion of the competitive advantage over 8B baselines is attributable to the dataset, and reframe the headline claim to state that RG-PHO provides a meaningful incremental improvement on top of an already strong data-driven baseline. We will also add a sentence in the abstract and introduction clarifying that the 4B-vs-8B advantage reflects both data quality and training methodology. revision: yes

  2. Referee: §5.1–5.2, Tables 5–7: The baseline comparison conflates training data, model scale, and training method. The 8B baselines (Qwen3Guard-8B, YuFeng-XGuard-Reason-8B) are trained on their own proprietary data of unknown scale and quality. Without knowing whether these baselines had access to comparable data resources, the '4B beats 8B' framing cannot be cleanly attributed to either the dataset or the method. The paper should either (a) provide a baseline trained on the same 811k corpus with a comparable 8B backbone to isolate the method's contribution from data effects, or (b) explicitly state that the comparison conflates these factors and temper the attribution claims. This is load-bearing for the central claim that reasoning-active training is the primary driver of the competitive result.

    Authors: The referee is correct that the baseline comparison conflates training data, model scale, and training method. The 8B baselines (Qwen3Guard-8B, YuFeng-XGuard-Reason-8B) are trained on their own proprietary corpora, whose scale and quality we cannot inspect. Without controlling for training data, the '4B beats 8B' result cannot be cleanly attributed to either the dataset or the method. Regarding option (a): we considered training an 8B backbone on the same 811k corpus to isolate the method's contribution from data effects. However, this experiment requires computational resources beyond what is available for this revision cycle, and we cannot honestly report results we have not yet obtained. We therefore adopt option (b): we will explicitly state in §5.1–5.2 that the baseline comparison conflates training data, model scale, and training method, and that the '4B beats 8B' framing reflects the combined effect of our data pipeline and training methodology rather than the training method alone. We will also temper the attribution claims throughout the paper, including in the abstract, introduction, and conclusion, to avoid implying that reasoning-active training is the sole or primary driver of the competitive result. The within-method ablation in Table 9 (which controls for data by holding the corpus fixed and varying only the training procedure) does provide a clean estimate of RG-PHO's contribution (+2.7 points), and we will foreground this as the primary evidence for the method's effectiveness rather than the cross-baseline comparison. revision: yes

Circularity Check

0 steps flagged

No significant circularity: the paper's training pipeline and ablation are self-contained, with no self-citation chain or fitted-input-as-prediction reduction.

full rationale

The paper presents DT-Guard, a safety guardrail model trained via a three-stage pipeline (mixed-mode SFT, hard-case SFT, rollout-contrastive DPO). Walking the derivation chain: (1) The dataset is constructed from heterogeneous public sources, distilled via GLM-5.1, filtered by multi-round voting and expert verification — no step here is defined in terms of the final evaluation metrics. (2) The RG-PHO pipeline uses rollout consistency (Eq. 2) to stratify samples, then applies SFT (Eq. 1) and DPO (Eq. 3) — these are standard training objectives, not circularly defined. (3) The ablation study (Tables 5–9) progressively adds components and measures F1 on external benchmarks. The baseline Stage1-SFT-v0 is trained on the same 811k corpus without intent/CoT, providing a fair within-dataset comparison. The gains from each component (+0.5 for intent, +0.9 for selective CoT, +1.3 for hard-case SFT, +1.4 for DPO) are measured against this baseline, not fitted to evaluation data. (4) No self-citation chain is load-bearing: the paper cites external work (Qwen3Guard, YuFeng-XGuard, GuardReasoner, DPO, RLHF) but does not invoke any prior theorem or result by the same authors as a premise for its central claim. The 'reasoning supervision can be internalized' claim is supported by the ablation showing selective CoT training + RG-PHO improves over no-CoT baseline under reasoning-free inference. The reader's concern about the proprietary dataset being a confound is a correctness/external-validity concern, not a circularity issue — the ablation does include a same-data baseline (Stage1-SFT-v0), and the incremental gains are measured against it. No step in the paper reduces to its own inputs by construction. Score 1 reflects one minor concern: the dataset is proprietary and not independently reproducible, but this is an openness issue, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 1 invented entities

The axiom ledger captures the key parameters and assumptions. The free parameters are standard for LLM training but their values are fitted or chosen by hand. The domain assumptions rely on the quality of GLM-5.1 and expert verification, which are not independently verified in the paper.

free parameters (3)
  • Safe:Unsafe:Borderline ratio = 5.5:4:0.5
    Chosen by hand to balance the training distribution (Section 3.3).
  • K (rollout count) = 3
    Number of independent rollouts for consistency scoring (Section 4.2).
  • beta (DPO preference strength) = Not specified
    Controls preference strength in DPO objective (Eq. 3), value not given.
axioms (3)
  • domain assumption GLM-5.1 distillation provides accurate CoT, intent, and safety labels
    Section 3.2 relies on GLM-5.1 for automatic annotation and quality filtering.
  • domain assumption Expert verification corrects remaining annotation noise
    Section 3.2 assumes expert verification of unanimous relabeling results is ground truth.
  • standard math F1 score is the appropriate metric for safety guardrail evaluation
    Section 5.1 uses F1 for all experiments.
invented entities (1)
  • RG-PHO (Rollout-Guided Progressive Hard-Case Optimization) independent evidence
    purpose: Training pipeline to internalize reasoning supervision
    Ablation study (Table 9) provides falsifiable evidence of its contribution.

pith-pipeline@v1.1.0-glm · 15071 in / 1955 out tokens · 405426 ms · 2026-07-08T09:39:26.707387+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages · 10 internal anchors

  1. [1]

    Qwen3.5: Accelerating productivity with native multimodal agents, February

    Qwen Team. Qwen3.5: Accelerating productivity with native multimodal agents, February

    work page

  2. [2]

    URLhttps://qwen.ai/blog?id=qwen3.5

    work page

  3. [3]

    OpenAI GPT-5 System Card

    OpenAI. Openai gpt-5 system card, 2026. URL https://arxiv.org/abs/2601.03267

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  4. [4]

    Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale

    Ling Team. Ling and ring 2.6 technical report: Efficient and instant agentic intelligence at trillion-parameter scale, 2026. URLhttps://arxiv.org/abs/2606.15079

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  5. [5]

    Universal and Transferable Adversarial Attacks on Aligned Language Models

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. URL https://arxiv.org/abs/2307.15043

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  6. [6]

    AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models

    Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: Interpretable gradient-based adversarial attacks on large language models, 2023. URLhttps://arxiv.org/abs/2310.15140

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  7. [7]

    Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023. URL https: //arxiv.org/abs/2312.06674

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    Shieldgemma: Generative ai content moderation based on gemma,

    Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, Olivia Sturman, and Oscar Wahltinez. Shieldgemma: Generative ai content moderation based on gemma,

    work page

  9. [9]

    URLhttps://arxiv.org/abs/2407.21772

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Qwen3Guard Technical Report

    Haiquan Zhao, Chenhan Yuan, Fei Huang, Xiaomeng Hu, Yichang Zhang, An Yang, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin, et al. Qwen3guard technical report, 2025. URLhttps://arxiv.org/abs/2510.14276

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  11. [11]

    Yufeng-xguard: A reasoning-centric, interpretable, and flexible guardrail model for large language models, 2026

    Junyu Lin, Meizhen Liu, Xiufeng Huang, Jinfeng Li, Haiwen Hong, Xiaohan Yuan, Yuefeng Chen, Longtao Huang, Hui Xue, Ranjie Duan, Zhikai Chen, Yuchuan Fu, Defeng Li, Lingyao Gao, and Yitong Yang. Yufeng-xguard: A reasoning-centric, interpretable, and flexible guardrail model for large language models, 2026. URL https://arxiv.org/abs/2601 .15588

    work page 2026

  12. [12]

    AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts

    Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. Aegis: Online adaptive ai content safety moderation with ensemble of llm experts, 2024. URL https: //arxiv.org/abs/2404.05993

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [13]

    Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms

    Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. Advances in neural information processing systems, 37: 8093–8131, 2024

    work page 2024

  14. [14]

    PolyGuard: A Multilingual Safety Moderation Tool for 17 Languages

    Priyanshu Kumar, Devansh Jain, Akhila Yerukola, Liwei Jiang, Himanshu Beniwal, Thomas Hartvigsen, and Maarten Sap. Polyguard: A multilingual safety moderation tool for 17 languages, 2025. URLhttps://arxiv.org/abs/2504.04377

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  15. [15]

    Mitigating Jailbreaks with Intent-Aware LLMs

    Wei Jie Yeo, Ranjan Satapathy, and Erik Cambria. Mitigating jailbreaks with intent-aware llms, 2025. URLhttps://arxiv.org/abs/2508.12072. 14

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  16. [16]

    Dagher, and Haohan Wang

    Jun Zhuang, Haibo Jin, Ye Zhang, Zhengjian Kang, Wenbin Zhang, Gaby G. Dagher, and Haohan Wang. Exploring the vulnerability of the content moderation guardrail in large language models via intent manipulation, 2025. URL https://arxiv.org/abs/2505.1 8556

    work page 2025

  17. [17]

    Li, Hui Xiong, and Bryan Hooi

    Yue Liu, Hongcheng Gao, Shengfang Zhai, Yufei He, Jun Xia, Zhengyu Hu, Yulin Chen, Xihong Yang, Jiaheng Zhang, Stan Z. Li, Hui Xiong, and Bryan Hooi. Guardreasoner: Towards reasoning-based llm safeguards, 2025. URL https://arxiv.org/abs/2501.1 8492

    work page 2025

  18. [18]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

    work page 2022

  19. [19]

    Constitu- tional ai: Harmlessness from ai feedback, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from ai feedback, 2022. URL https://arxiv.org/abs/2212.0 8073

    work page 2022

  20. [20]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023. 15

    work page 2023