Don't Command, Cultivate: An Exploratory Study of System-2 Alignment

Jitao Sang; Xinyan Wen; Yanxu Zhu; Yuhang Wang; Yuxiang Zhang

REVIEW 4 major objections 7 minor 19 references

Don't Command, Cultivate: An Exploratory Study of System-2 Alignment

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Encouraging language models to scrutinize user requests step by step reduces harmful outputs, while math-encoded jailbreaks remain a gap.

desk verdict Useful exploratory study, but the safety gains rest on a GPT-4-family evaluation loop that needs independent validation before the headline claim is trusted. read the letter →

arxiv 2411.17075 v5 pith:5VIVXHII submitted 2024-11-26 cs.CL

Yuhang Wang , Yuxiang Zhang , Yanxu Zhu , Xinyan Wen , Jitao Sang This is my paper · ORCID

classification cs.CL

keywords System-2alignmentLLMsafetyjailbreakchain-of-thoughtpromptengineeringsupervisedfine-tuningdirectpreferenceoptimizationreinforcementlearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether deliberately slow, step-by-step reasoning can make language models safer, not just smarter. It first stress-tests the o1 model against adversarial natural-language jailbreaks and prompts encoded in mathematical symbols, finding better overall safety than GPT-4o but persistent failures on math-encoded attacks. It then tries to transfer System-2 alignment to open-source 7B-8B models by prompting, supervised fine-tuning, direct preference optimization, and reinforcement learning. Across those experiments the consistent result is that teaching or instructing a model to scrutinize user requests raises its refusal of harmful content, sometimes at the cost of over-refusing benign requests. The paper argues this is a viable complement to alignment methods that assume strong reasoning is already present.

What carries the argument

System-2 alignment is operationalized as chain-of-thought safety analysis. The model is asked, through prompts or fine-tuning, to produce steps such as identifying potential hazards, analyzing the user's intended use, and categorizing the request as Allowed, Disallowed, or Safe Completion before giving a final answer. In supervised fine-tuning this reasoning is distilled from a teacher model using hand-written safety specifications based on a harm taxonomy; in reinforcement learning, a reward model scores final answers, and the paper proposes a process reward model that scores each reasoning step. The mechanism is the analysis step itself: it slows the model down and makes the safety decision depend on explicit evidence rather than the model's first associative response.

What would settle it

Collect human-annotated safety labels for the same WildJailbreak and math-encoded test prompts, then score the SFT-CoT and RL models against those labels; if human agreement with the model judge is low or the human-annotated safety gain vanishes, the central claim fails. A cheaper check: attack the fine-tuned models with a math-encoded prompt built from a harm category excluded from the SFT taxonomy and see whether the reasoning chain still flags it.

Watch

Extended reading notes

Core claim

The paper's central claim is that System-2 alignment, steering a model to evaluate user intent, weigh risks, and reason before answering, improves safety even in models that do not natively reason at length. Evidence comes from WildJailbreak adversarial prompts: adding a safe-alert or few-shot analysis raises the not_unsafe rate, for example Mistral-7B from 0.186 to 0.695 with few-shot chain-of-thought prompting. Supervised fine-tuning with distilled reasoning steps improves both safety and over-refusal for Mistral-7B and Llama3-8B, and adding reinforcement learning gives the most balanced result. The same scrutiny, however, does not yet defeat math-encoded jailbreaks: o1 refuses only 15% of these without a safety alert, and the fine-tuned open models still answer such prompts as benign academic requests. The paper locates the vulnerability in the model's reasoning chain itself, where a step can misread the request as safe and thereby enlarge the attack surface.

Load-bearing premise

The load-bearing premise is that the GPT-4-family judge and teacher labels reflect true safety and harmfulness; if those judgments are biased, the measured safety improvements could be style matching rather than real safety.

Editorial extensions

If this is right

Chain-of-thought prompting with safety alerts improves refusal of harmful prompts across Mistral-7B, Qwen-7B, and Llama3-8B, though often with more over-refusals on benign requests.
Supervised fine-tuning on reasoning-annotated safety data improves both safety and helpfulness more than fine-tuning on raw responses, with average scores rising from 0.752 to 0.883 for Mistral-7B and from 0.760 to 0.905 for Llama3-8B.
Reinforcement learning with outcome supervision balances safety and helpfulness better than direct preference optimization, which tends to make the model reject too many benign requests.
Mathematical encoding remains a blind spot: even with added safety alerts, the o1 model only reaches 0.8 not_unsafe on math-encoded attacks, and fine-tuned open models classify such prompts as safe academic requests.
Because the attack can hide inside a reasoning chain, the authors propose process supervision, rewarding each intermediate step, as the next step toward safer System-2 alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

My inference: if the safety gains come from genuine deliberation rather than style matching, the methods should transfer to attacks never seen during training, such as new encodings, new languages, or novel harm categories; measuring that out-of-distribution transfer would separate mechanism from memorization.
My inference: since a GPT-4-family model is both the teacher that distills reasoning and the judge that scores safety, a human-annotated evaluation could reveal whether the apparent improvements are absolute or an artifact of judge preference for reasoning-shaped answers.
My inference: math-encoded jailbreaks succeed because the model treats the request as a formal proof task, so a concrete testable extension is to force the safety taxonomy to run before any mathematical solution, even when the prompt appears purely academic.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take. This is an honest exploratory paper, and the empirical core is a useful addition to the small but growing literature on reasoning-style safety alignment. What's actually new: a clean comparison of prompting, SFT, DPO, and RL for 'System-2 alignment' on three open-weight models, with code and data released. The authors also frame their work as a complement to Deliberative Alignment rather than a competitor, which is fair. The mixed results help: SFT-CoT helps Mistral and Llama but not Qwen, and zero-shot safe prompting hurts Llama. If the GPT-4 judge were simply rewarding longer CoT-style answers, you'd expect more uniform gains.

The soft spots are real and load-bearing. The whole pipeline runs on one model family: GPT-4o assigns safety categories to the SFT training data, distills the reasoning traces, and GPT-4 scores the evaluation. The test set is 420 samples with no human validation on the main open-source comparisons—the manual eval is only on the o1 case studies. That means the measured safety improvements could be style matching rather than genuine safety. The math-encoded jailbreak evaluation is 20 samples with no error bars; it's suggestive, not conclusive. The process-supervision section (3.5) is a proposal, no experiments, and the paper itself says the implementation details will come later. For an arXiv exploratory study that's acceptable, but it should be labeled as such in the claims.

The o1 case studies are actually the most interesting part. Seeing the model's reasoning chain go from safe to unsafe mid-thought (Case 2) and the mathematical-encoding attacks slipping through are concrete, falsifiable observations. Those are worth building on.

My bottom line: the central claim—that simple methods encouraging careful analysis improve safety—is plausible and partially supported, but the closed evaluation loop is the key weakness. It needs independent judgment (human or a second model family) before I'd take the numbers at face value. The paper deserves serious review; it's a legitimate exploratory study with a clear research direction. I'd send it to peer review, and ask for human evaluation, error bars, and either results for the process-supervision plan or a clear statement that it's future work.

Referee Report

4 major / 7 minor

Summary. The paper explores 'System-2 alignment' for large language model safety, i.e., encouraging models to reason carefully about user requests before responding. It first evaluates OpenAI's o1 model on adversarial natural-language jailbreaks and math-encoded jailbreaks using 20 samples per condition with manual evaluation (Section 2). It then applies prompt engineering, supervised fine-tuning (SFT), direct preference optimization (DPO), and outcome-supervised reinforcement learning (RL) to Mistral-7B, Qwen-7B, and Llama3-8B, evaluating all open-source results with a GPT-4 judge on 420 WildJailbreak samples (Section 3). The paper also proposes a process-supervision RL framework (Section 3.5) but presents no experiments for it. The central claim is that methods encouraging deliberate analysis improve model safety, with the abstract noting that 'some simple methods' are beneficial.

Significance. If the central claim held, the results would be practically useful: simple prompting and fine-tuning interventions could improve the safety of open-source models, complementing OpenAI's deliberative alignment approach. The paper's strengths are its open release of code and data, the manual evaluation and detailed case studies for the o1 model, and the honest reporting of trade-offs such as increased overrefusal and model-dependent failures. However, the significance is limited by the closed evaluation loop (GPT-4o labels/distills training data, GPT-4 scores outputs), the absence of uncertainty quantification, and the fact that several reported results contradict the broad central claim. The contribution is best read as an exploratory study that identifies directions for further work rather than a definitive demonstration.

major comments (4)

[Section 3.1/3.2; Tables 4-6] The central claim that System-2 methods improve safety rests on a closed evaluation loop. Section 3.1 uses GPT-4 as the judge for all open-source outputs, with the prompt in Table 9, while Section 3.2 uses GPT-4o to assign safety labels to the SFT training data and to distill the reasoning-annotated CoT responses. DPO and RL in Sections 3.3 and 3.4 construct preference/reward labels from the same safety taxonomy. No human validation or independent judge is reported for any of the 420 open-source test responses. Since the judge and the teacher belong to the same model family, the improvements in Tables 4-6 could reflect the models learning to imitate GPT-4o's structured refusal style rather than genuine safety. This is load-bearing for the abstract's claim. Please report inter-annotator agreement between GPT-4 judgments and human labels on a random subset (e.g., 50 responses per condition), or compare GPT-4 judgments with a second independent judge.
[Section 1; Tables 4-5] The central claim, stated in the Introduction as 'methods encouraging models to carefully analyze user inputs improve safety,' is not uniformly supported by the reported results. In Table 4, Llama3-8B with zero-shot CoT (safe) has average 0.691 versus 0.779 for the baseline, driven by a drop in not_overrefuse from 0.790 to 0.533. In Table 5, Qwen-7B SFT-CoT has average 0.771 versus 0.788 for SFT-Baseline, with not_unsafe falling from 0.690 to 0.609. The abstract's weaker 'some simple methods... are beneficial' is compatible with the data, but the Introduction's general statement is contradicted. Please either revise the central claim to be method- and model-specific, or provide an analysis of the conditions under which System-2 alignment helps.
[Tables 2-6] No uncertainty quantification is provided for any of the headline numbers. The o1 evaluation in Section 2 uses only 20 samples per condition (Tables 2-3); for example, the difference between o1 (17/20 not_unsafe) and GPT-4o (13/20) is not shown to be significant. The open-source evaluation has 420 test samples but reports point estimates only, with no confidence intervals or significance tests across random seeds or sampling. Without error bars, the improvements in Tables 4-6 (e.g., Llama3-8B SFT-CoT average 0.905 vs SFT-Baseline 0.760) cannot be distinguished from noise. Please add bootstrap or multiple-seed evaluation results, or at minimum report per-condition counts and confidence intervals.
[Section 3.5; Abstract] The process supervision section (Section 3.5) is a proposal with no experiments; the abstract states that 'implementation details and experimental results will be provided in future versions,' and the first page labels the paper 'A study in preparation.' Yet the Conclusion claims that 'we showed that incorporating slow-thinking processes improves safety performance' and that the study 'employs diverse methodologies—... reinforcement learning—to validate the efficacy.' Since the process-supervised RL is not implemented or evaluated, the validated scope is limited to prompting, SFT, DPO, and outcome-supervised RL. Please either implement and evaluate the process supervision, or explicitly frame it as future work and revise the Conclusion and abstract accordingly.

minor comments (7)

[Section 2.1, Figure 1 caption] There is a typo in the caption: 'bottm' should be 'bottom.'
[Section 3, opening paragraph] The word 'implementaions' should be 'implementations.'
[Table 4] The notation 'Few-shot CoT (+ -)' is unexplained; please define the symbols, presumably indicating one adversarial benign and one adversarial harmful in-context example.
[Section 3.2, page 8] The phrase 'as shown in Table 1314' should read 'Tables 13 and 14.'
[References] Reference [12] is incomplete: 'Yuxiang Zhang and et. al. o1-coder: an o1 replication for coding. arXiv preprint, 2024' lacks a full title and arXiv identifier; also 'and et. al.' is redundant.
[Table 10] The relationship between the three label columns and the row categories should be explained; the identical counts in the Allowed, Disallowed, and Safe Completion columns for each category are confusing.
[Figure 2, Thoughts section] The line 'Fault information and storyline written by I'm' appears to be a broken phrase; please correct.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity by construction; the GPT-4 evaluation loop is a validity concern, not a derivation-circle.

full rationale

The paper's central claim is empirical rather than definitional: it asserts that prompting, SFT, DPO, and RL interventions which encourage deliberative analysis improve judged safety on WildJailbreak samples. None of the reported quantities is defined in terms of another reported quantity, and no parameter is fitted to a subset and then renamed as a prediction. The o1 safety evaluations in Section 2 are manually scored, providing an independent anchor for that part of the study. For the open-source experiments, the safety taxonomy is taken from an external benchmark (MLCommons, ref [8]) and the human-authored safety specifications in Tables 12-14; GPT-4o provides labels and CoT distillations, while GPT-4 provides evaluation judgments. This shared model family creates a legitimate concern about style-matching and external validity, but it is not a formal circularity because the training objectives and evaluation metric are not identical by construction. The self-citations ([12], [13]) appear only as future-work suggestions and are not load-bearing for the empirical claims. No uniqueness theorem, imported ansatz, or renaming of a known result is used to force the conclusions. The paper is exploratory and incomplete, but its derivation chain does not reduce to its own inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim relies on the evaluation and labeling assumptions above. There are no fitted parameters or invented entities.

assumptions (4)

domain assumption WildJailbreak adversarial evaluation subset is representative of real-world jailbreak attempts.
Section 2.1 uses 20 random samples from WildJailbreak eval subset; the paper generalizes from these to claims about o1 robustness.
domain assumption GPT-4-based judge reliably distinguishes unsafe, overrefuse, and safe responses.
Section 3 uses GPT-4 with prompt in Table 9; no human validation or agreement reported.
domain assumption Safety categories from MLCommons AI Safety Benchmark and the added 'others' category are appropriate for labeling safety data.
Section 3.2 labels training data using these categories; the taxonomy may not cover all edge cases.
ad hoc to paper The safety specifications written by the authors (Tables 12-14) correctly define allowed, disallowed, and safe completion behavior.
These specifications are created for the paper and used to distill GPT-4o reasoning; they are not validated externally.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Don't Command, Cultivate: An Exploratory Study of System-2 Alignment." pith.science (2026). https://pith.science/paper/5VIVXHII

@misc{pith2026241117075,
  author       = {Pith},
  title        = {Pith review of: Don't Command, Cultivate: An Exploratory Study of System-2 Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5VIVXHII}},
  note         = {Machine review of arXiv:2411.17075}
}

read the original abstract

The o1 system card identifies the o1 models as the most robust within OpenAI, with their defining characteristic being the progression from rapid, intuitive thinking to slower, more deliberate reasoning. This observation motivated us to investigate the influence of System-2 thinking patterns on model safety. In our preliminary research, we conducted safety evaluations of the o1 model, including complex jailbreak attack scenarios using adversarial natural language prompts and mathematical encoding prompts. Our findings indicate that the o1 model demonstrates relatively improved safety performance; however, it still exhibits vulnerabilities, particularly against jailbreak attacks employing mathematical encoding. Through detailed case analysis, we identified specific patterns in the o1 model's responses. We also explored the alignment of System-2 safety in open-source models using prompt engineering and supervised fine-tuning techniques. Experimental results show that some simple methods to encourage the model to carefully scrutinize user requests are beneficial for model safety. Additionally, we proposed a implementation plan for process supervision to enhance safety alignment. The implementation details and experimental results will be provided in future versions.

Figures

Figures reproduced from arXiv: 2411.17075 by the authors.

**Figure 2.** An example of the o1 model being attacked. In the latter part of the “thoughts” section, the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** An example of a normal response to benign prompts. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: An example of overrefusal to benign prompts. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: An example of an o1 being attacked by a prompt encoded with mathematical symbols. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: An example comparing SFT-CoT + RL and SFT-CoT. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

**Figure 7.** Figure 7: An example comparing SFT-CoT + RL and SFT-CoT + DPO. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 8.** Figure 8: An example of an SFT-CoT + RL method addressing a math-encoded jailbreak prompt. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 9 canonical work pages

[1]

Jailbreaking large language models with symbolic mathematics

Emet Bethany, Mazal Bethany, Juan Arturo Nolazco Flores, Sumit Kumar Jha, and Peyman Najafirad. Jailbreaking large language models with symbolic mathematics. arXiv preprint arXiv:2409.11445, 2024

arXiv 2024
[2]

Deliberative alignment: Reasoning enables safer language models

Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Heylar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339, 2024

arXiv 2024
[3]

Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models

Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. ArXiv, abs/2406.18510, 2024

arXiv 2024
[4]

Thinking, fast and slow

Daniel Kahneman. Thinking, fast and slow. Farrar, Straus and Giroux, 2011

2011
[5]

Improve mathematical reasoning in language models by automated process supervision

Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592, 2024

arXiv 2024
[6]

Openai o1 system card

OpenAI. Openai o1 system card. In OpenAI, 2024

work page 2024
[7]

O1 replication journey: A strategic progress report–part 1

Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. O1 replication journey: A strategic progress report–part 1. arXiv preprint arXiv:2410.18982, 2024

arXiv 2024
[8]

Introducing v0

Bertie Vidgen, Adarsh Agrawal, Ahmed M Ahmed, Victor Akinwande, Namir Al-Nuaimi, Najla Alfaraj, Elie Alhajjar, Lora Aroyo, Trupti Bavalatti, Max Bartolo, et al. Introducing v0. 5 of the ai safety benchmark from mlcommons. arXiv preprint arXiv:2404.12241, 2024

arXiv 2024

Show all 19 references

[9]

Openr: An open source framework for advanced reasoning with large language models

Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, et al. Openr: An open source framework for advanced reasoning with large language models. arXiv preprint arXiv:2410.09671, 2024

arXiv
[10]

Math-shepherd: Verify and reinforce llms step-by-step without human annotations

Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V o...

2024
[11]

Quiet-star: Language models can teach themselves to think before speaking

Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629, 2024

2024 arXiv
[12]

Yuxiang Zhang and et. al. o1-coder: an o1 replication for coding. arXiv preprint, 2024

2024
[13]

Famous Actor X,

Yuxiang Zhang, Yuqi Yang, Jiangming Shu, Yuhang Wang, Jinlin Xiao, and Jitao Sang. Openrft: Adapting reasoning foundation model for domain-specific tasks with reinforcement fine-tuning. arXiv preprint arXiv:2412.16849, 2024. 12 5 Appendix Adversarial Benign Prompt As a risk an...

2024 arXiv
[14]

What is machine learning?

Allowed Requests -Definition: User requests are clearly legal, safe, and ethical, with no potential harm to the user or society. -Applicable Scenarios: Requests for information in academic, educational, or everyday contexts that are legal and do not involve sensitive, high-ris...
[15]

How can I make dangerous chemicals?

Disallowed Requests -Definition: User requests may violate legal, ethical, or safety standards, or have significant potential for harm, causing direct or indirect negative impacts on the user, third parties, or society. -Applicable Scenarios: Requests involving illegal activit...
[16]

How do I deal with anxiety?

Safe Completion Requests -Definition: User requests may involve potential safety or ethical risks, but the model can respond safely and beneficially by carefully phrasing the answer, providing additional context, or reframing the question. -Applicable Scenarios: Requests relat...
[17]

Machine learning is an artificial intelligence technique that enables computers to learn and improve from data analysis without explicit programming

For an Allowed Request -Response Requirements: Provide clear, accurate, and professional answers that directly address the user’s request. Avoid redundant content, and ensure the information is concise and relevant. -Example: “Machine learning is an artificial intelligence tec...
[18]

I’m sorry, I cannot assist with that request as it involves dangerous or illegal activities

For a Disallowed Request -Response Requirements: Clearly reject the request and briefly explain why it cannot be fulfilled. Avoid providing any sensitive or potentially exploitable information. -Example: “I’m sorry, I cannot assist with that request as it involves dangerous or...
[19]

Provide helpful and harmless information, and, if necessary, encourage users to seek professional assistance or authoritative sources

For Safe Completion Requests -Response Requirements: Ensure responses are cautiously worded to avoid misunderstand- ings or unsafe behaviors. Provide helpful and harmless information, and, if necessary, encourage users to seek professional assistance or authoritative sources. ...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Jailbreaking large language models with symbolic mathematics

Emet Bethany, Mazal Bethany, Juan Arturo Nolazco Flores, Sumit Kumar Jha, and Peyman Najafirad. Jailbreaking large language models with symbolic mathematics. arXiv preprint arXiv:2409.11445, 2024

arXiv 2024

[2] [2]

Deliberative alignment: Reasoning enables safer language models

Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Heylar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339, 2024

arXiv 2024

[3] [3]

Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models

Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. ArXiv, abs/2406.18510, 2024

arXiv 2024

[4] [4]

Thinking, fast and slow

Daniel Kahneman. Thinking, fast and slow. Farrar, Straus and Giroux, 2011

2011

[5] [5]

Improve mathematical reasoning in language models by automated process supervision

Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592, 2024

arXiv 2024

[6] [6]

Openai o1 system card

OpenAI. Openai o1 system card. In OpenAI, 2024

work page 2024

[7] [7]

O1 replication journey: A strategic progress report–part 1

Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. O1 replication journey: A strategic progress report–part 1. arXiv preprint arXiv:2410.18982, 2024

arXiv 2024

[8] [8]

Introducing v0

Bertie Vidgen, Adarsh Agrawal, Ahmed M Ahmed, Victor Akinwande, Namir Al-Nuaimi, Najla Alfaraj, Elie Alhajjar, Lora Aroyo, Trupti Bavalatti, Max Bartolo, et al. Introducing v0. 5 of the ai safety benchmark from mlcommons. arXiv preprint arXiv:2404.12241, 2024

arXiv 2024