Does More Inference-Time Compute Really Help Robustness?

Chawin Sitawarin; Chong Xiang; Jiachen T. Wang; Prateek Mittal; Tong Wu; Vikash Sehwag; Weichen Yu

REVIEW 5 major objections 5 minor 34 references

Does More Inference-Time Compute Really Help Robustness?

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read When reasoning chains are exposed, more inference-time compute makes open-source reasoning models less robust.

desk verdict Useful warning about exposed reasoning chains, but the 'inverse scaling law' is definitional, not empirical; the paper overclaims in the abstract. read the letter →

arxiv 2507.15974 v1 pith:SUQBDZ2Y submitted 2025-07-21 cs.AI

Tong Wu , Chong Xiang , Jiachen T. Wang , Weichen Yu , Chawin Sitawarin , Vikash Sehwag , Prateek Mittal This is my paper

classification cs.AI

keywords inference-timescalingrobustnessreasoningchainexposurepromptinjectionextractionharmfulrequestsbudgetforcinginverselaw

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that the robustness gains from allocating more inference-time compute are conditional on a hidden assumption: the adversary cannot see the model's intermediate reasoning. It first shows that open-source reasoning models such as DeepSeek R1, Qwen3, and Phi-reasoning can become more robust on final outputs against prompt injection and prompt extraction when a budget-forcing strategy forces longer thinking. The central claim is an inverse scaling law: once the reasoning chain itself is exposed, longer chains consistently make the model less robust, because every added token is another chance for a malicious string to appear and be observed. The paper derives this from probability monotonicity, verifies it on SEP, Tensor Trust, SorryBench, and a tool-integrated LLM-PIEVal proof of concept, and argues that hiding chains does not fully remove the risk. The practical consequence is that a deployment's threat model, not just the compute budget, decides whether spending more thinking time helps or hurts.

What carries the argument

The central object is an exposed-chain failure event. For a predefined malicious-token set $M \subset \Sigma$ and a reasoning chain $T_1, T_2, \ldots$, the paper defines failure of a length-$L$ exposed chain as $E_L = \{\exists\, i \leq L : T_i \in M\}$, so robustness is $1 - \Pr[E_L]$. Because the events are nested, $\Pr[E_L]$ is non-decreasing in $L$; with any positive per-step risk $p_*$, leakage approaches certainty as $L$ grows. The scaling method is budget forcing: when the chain is under budget, the model is prompted with "Wait," to keep reasoning, and at budget an end-of-thinking delimiter followed by "Final Answer:" forces the response, letting the experiments set $L$ directly across 100 to 16,000 tokens.

What would settle it

A decisive check scores the same exposed-chain runs by end-to-end harm, not token presence: did the injected instruction get followed, did the secret leave the system, did the model comply with the harmful request? If final-output harm does not rise with the thinking budget even though the visible chains contain more malicious tokens, the security-relevant version of the inverse scaling law is falsified; the monotonicity identity itself would survive, but without the claimed practical teeth.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is an inverse scaling law for robustness under exposed reasoning chains: across twelve open-source reasoning models, raising the reasoning-token budget from 100 to 16,000 makes it more likely that the visible chain contains a token from a predefined malicious set, so chain-level robustness consistently falls. The formal reason is monotonicity of nested events: with $M$ a fixed set of malicious tokens, the failure event $E_L = \{\exists\, i \leq L : T_i \in M\}$ satisfies $\Pr[E_L] \le \Pr[E_{L+1}]$, and if each decoding step has positive risk $p_*$ then $\Pr[E_L] \ge 1-(1-p_*)^L$. The paper verifies this empirically on prompt injection, prompt extraction, and harmful requests, and it qualifies the practical severity: leaked secrets and unsafe step-by-step instructions in chains are directly dangerous, while chain-level prompt-injection leakage is less practically concerning because attackers usually target the final output. On top of this, the same budget-forcing schedule that improves final-output prompt-injection and prompt-extraction robustness leaves harmful-request robustness roughly flat, so the direction of the effect depends on what is being scored.

Load-bearing premise

The inverse scaling law is measured on the paper's chosen definition of failure—whether any token from a fixed list of malicious strings appears among the first $L$ reasoning tokens—and the security conclusion assumes that this token-presence measure is the right proxy for adversarial harm.

Editorial extensions

If this is right

Any deployment that exposes reasoning traces should expect the leakage probability of secrets or unsafe strings to rise with the thinking budget, regardless of how safe the final answer looks.
In prompt-extraction settings, the inverse law is directly exploitable: once the secret appears in the visible chain, the attacker has succeeded before the model produces its final response.
For harmful requests, longer exposed chains can leak detailed unsafe instructions even when the model refuses at the final answer level, so hiding the final refusal is not enough.
Hiding the chain is not a complete mitigation: tool-integrated reasoning adds API-call surfaces whose prompt-injection robustness degrades with compute, and hidden chains have been successfully extracted from proprietary models in red-teaming.
The favorable direction reported for proprietary models still holds for open-source models when only final outputs are scored, so the practical answer to whether more compute helps robustness depends on what the adversary can see and what counts as failure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: the formal inverse law is a monotonicity identity for a fixed token set, so the empirical content lives in the per-step leakage rate $p_*$; measuring $p_*$ per model and budget would show whether the phenomenon is an intrinsic property of reasoning or an artifact of the chosen malicious-token sets.
Editorial extension: parallel scaling strategies such as Best-of-N sampling, which spread extra compute over multiple short chains rather than one long chain, may avoid the inverse law altogether; this is testable on the same three benchmarks because the budget is spent without lengthening any single exposed trace.
Editorial extension: the paper's own qualification suggests a mitigation the paper does not develop: filter or redact the reasoning trace before release, which would decouple the length of the internal chain from the length of the exposed surface.
Editorial extension: the token-presence metric treats any member of $M$ as equally dangerous even when it appears in a harmless context; context-sensitive detection of actual harmful content could change the measured scaling direction on some datasets.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper gives the field a genuinely useful warning: if you expose a reasoning model's chain-of-thought, spending more inference compute can leak more malicious tokens. The authors show this across 12 open-source models and three benchmarks, and they also extend Zaremba et al. by showing budget forcing improves final-output robustness against prompt extraction in open models. That final-output result is new and worth having.

But the headline 'inverse scaling law' is not an empirical discovery; it is a consequence of the way they define failure. In Section 4.1 they define E_L as 'any token from a fixed malicious set M appears in the first L reasoning tokens,' note that this event is monotone, and then Section 4.2 measures exactly this quantity. For any autoregressive model whatsoever, Pr[E_L] is non-decreasing in L. So the monotone degradation in their figures is forced. The empirical content is only that malicious tokens occur with non-negligible frequency—and they don't report error bars, per-token rates, or ablations on M. The authors themselves include a Remark saying prompt-injection degradation in chains is less practically concerning because attackers usually target final outputs. That runs against the abstract's blanket claim that more compute 'consistently reduces model robustness.'

The evidential base is thinner than the headline: no code or data release, single-run measurements, and an unvalidated GPT-4o-mini judge for the harmful-request curves. The tool-use section is a proof-of-concept with a simulated API, and the hidden-chain extraction discussion leans on a red-team competition. These are minor-to-moderate issues, but together they mean the paper should be read as a 'consider your threat model' reminder rather than a discovered scaling law.

That said, the work is clear and honest. It flags its own caveats, cites related work fairly, and the formal monotonicity argument is correct. I would send it to peer review, but I'd expect the reviewers to push for reframing: the contribution is the demonstration that exposed chains invert the final-output trend, not the monotonicity itself. With error bars, code, a validated judge, and a softer abstract, this would be a solid paper. As is, it is a useful, if overclaimed, empirical note.

Recommendation: engage with it, but require the revisions above before accepting.

Referee Report

5 major / 5 minor

Summary. The paper studies how increasing inference-time compute (via budget forcing) affects robustness of open-source reasoning LLMs. Section 3 shows that for final outputs, larger thinking budgets improve robustness against prompt injection and prompt extraction, with limited change on harmful requests. Section 4 introduces an 'inverse scaling law': if intermediate reasoning tokens are exposed, robustness, defined as the absence of any malicious token in the first L reasoning tokens, degrades as L grows. Section 5 argues that hiding reasoning chains does not fully mitigate the issue, providing a tool-use simulation and citing chain-extraction attacks. The paper concludes that robustness benefits of inference-time scaling depend heavily on the adversarial setting.

Significance. If the inverse scaling law were an empirical discovery, it would be an important caution for deployments that expose chain-of-thought. However, as the paper itself formalizes in Section 4.1, the monotone degradation in Section 4.2 follows from the nested event definition: for any model, the probability that a fixed malicious token set is hit within the first L tokens is non-decreasing in L. The paper's contribution therefore rests on the rate at which this probability grows and on the practical relevance of the metric, both of which are underdeveloped. The Section 3 results extending inference-time scaling benefits to open-source models are useful, and Section 5 raises legitimate concerns about tool-augmented reasoning and chain extraction. With reframing and additional analysis, the paper could be a valuable contribution; in its current form, the abstract and conclusion overstate the finding.

major comments (5)

[Section 4.1, definition of E_L; Section 4.2, Figure 4] The 'inverse scaling law' is a mathematical consequence of the nested-event metric. Because E_L = {∃ i ≤ L : T_i ∈ M} is monotone increasing in L, Pr[E_L] is non-decreasing for any autoregressive model regardless of its safety behavior. The decreasing robustness curves in Figure 4 therefore do not reveal a property of the models; they reveal that longer prefixes provide more chances to contain a token from M. The only empirically non-trivial quantity is the per-step hazard Pr[T_i ∈ M | T_{<i}, P], but the paper never isolates it. I recommend either reporting the per-token hazard as a function of budget or explicitly framing the finding as 'cumulative leakage grows with exposed chain length by construction, and the empirical question is the rate.' Without such a reframing, the abstract's statement that increased compute 'consistently reduces model robustness' is unsupported.
[Abstract and Conclusion; Remark in Section 4.2] The headline claim overstates the finding. The Remark correctly states that for prompt injection, robustness decreases measured solely in intermediate reasoning are less practically concerning because attackers target final outputs; for harmful requests, leaked unsafe instructions in chains are a serious risk; for prompt extraction, chain leakage is a genuine secret-disclosure threat. The abstract and conclusion do not carry this qualification and give the impression of a uniform inverse scaling law. Please qualify the claims to the metric and to the specific threat models.
[Section 4.2 and Appendix A] The malicious-token set M is never operationally defined. For SEP, Tensor Trust, and SorryBench, the paper must specify how M is constructed (e.g., exact secret phrases, tokens from the injected instruction, a curated unsafe-word list). Without this definition, the empirical evaluation in Section 4.2 is not reproducible, and the interpretation of the degradation magnitude is ambiguous because the result depends heavily on the size and content of M.
[Figures 3 and 4] All results are single-run measurements with no confidence intervals, and the harmful-request evaluation uses GPT-4o-mini as an automated judge without reporting validation against human labels. The paper should provide variance estimates (e.g., across seeds or via bootstrap over dataset items) and a judge validation, since the claimed 20–40% declines in SorryBench (Figure 4c) and the 'no obvious gains' for harmful requests (Figure 3c) may be within measurement noise.
[Section 5.1] The tool-integrated reasoning experiment is a proof-of-concept that simulates API calls via prompts on models without native tool-calling. The observed degradation is modest (e.g., Phi-4-reason from 100% to 87%), and the Discussion acknowledges this limitation. I have no objection to a proof-of-concept, but the claim that this 'highlights serious security concerns' should be tempered; without experiments on models with real tool-use integration, the practical severity is not established.

minor comments (5)

[Section 2.1 heading] 'Prelimiary' is a typo for 'Preliminary'.
[Section 2.2] The text inconsistently renders 'GPT-4 O-MINI' and 'GPT-4o-mini'; please standardize the model name.
[Figures 3 and 4 captions] The y-axis label 'Robustness' obscures the fact that Figure 4 uses a different metric (absence of malicious tokens in the reasoning prefix); the caption should define the metric explicitly.
[Figure 1 caption] 'Averaged over 12 open-source reasoning models' is ambiguous; clarify whether averaging is across models and datasets, and report the spread.
[General / reproducibility] The paper does not state a data/code availability commitment; given the sensitivity of red-teaming, consider at least releasing the prompts and judge outputs to enable reproducibility.

Circularity Check

2 steps flagged · score 6.0 of 10

The §4.2 inverse scaling law is the monotonicity of the paper's own prefix-event metric, so the headline degradation is forced by definition; independent empirical content remains in §3 and in the tool-use proof-of-concept.

self definitional [Section 4.1 (formal hypothesis) and Section 4.2 (empirical verification), event E_L and Figure 4]
"Define the event EL = {∃ i ≤ L : Ti ∈ M}, i.e., at least one malicious token appears in the first L positions. Because the set of trajectories satisfying Ek is contained in the set satisfying Ek+1, probability measure monotonicity gives Pr [Ek] ≤ Pr [Ek+1]. Hence, the success probability is non-decreasing with the length of the exposed chain. ... Specifically, we assess robustness based on whether the reasoning chains themselves contain malicious tokens (e.g., unsafe or adversarial instructions), regardless of the final model response."

The Section 4.2 'inverse scaling law' measures exactly Pr[E_L], the event defined in Section 4.1. Since E_L is monotonically nested, Pr[E_L] must be non-decreasing in L for every autoregressive model; equivalently, robustness 1 − Pr[E_L] must be non-increasing. The empirical curves in Figure 4 therefore cannot test any behavioral hypothesis about reasoning models—they plot a mathematical consequence of the chosen metric. The only empirical content is that malicious tokens occur with non-negligible per-step probability, yet the paper presents the forced monotonicity as a discovered 'law' and as evidence that more compute 'consistently reduces model robustness.' The Remark softens practical implications, but the headline claim is definitional.
self definitional [Section 5.1, tool-integrated reasoning evaluation (Figure 5)]
"We evaluate their robustness using the LLM-PIEVAL dataset ... specifically measuring the models' susceptibility to making unsafe API calls during intermediate reasoning. ... longer reasoning chains inherently provide adversaries with more opportunities to trigger unsafe API interactions—an issue not adequately addressed simply by hiding intermediate reasoning steps."

The tool-use robustness metric is again an any-unsafe-call-in-prefix event. With a fixed per-token probability of producing a malicious API call, the chance of at least one such call in the reasoning prefix is non-decreasing in L for any model; the paper's own wording ('inherently provide... more opportunities') concedes the mechanism is definitional rather than a discovered model property. The empirical result is therefore of the same monotone form, so the decline in Figure 5 is forced by the measurement choice, not by an independent behavioral law.

full rationale

The paper's central novel claim—the inverse scaling law of Section 4—is built into its own definition of robustness. Section 4.1 defines the failure event E_L as 'some malicious token appears among the first L reasoning tokens' and proves Pr[E_L] is non-decreasing in L; Section 4.2 then reports the empirical curves of exactly this monotone quantity. The trend is therefore a logical consequence of the metric for any autoregressive model, and the empirical evaluation cannot independently establish a behavioral law about reasoning models. Section 5.1's tool-use result inherits the same prefix-event structure. That said, the paper is not wholly circular: Section 3's finding that increased inference budgets improve final-output robustness is a genuine empirical measurement not forced by the metric, and Section 5.2 relies on an independent red-teaming competition result. The authors' self-citations appear only as related work, not as load-bearing support for the inverse scaling claim. The in-paper Remark also appropriately warns that practical risk depends on the task, but the abstract and Section 4.2 headline the inverse scaling law without that caveat. Overall, the headline claim reduces by construction to the chosen prefix-event metric, while meaningful independent content remains elsewhere—hence a partial circularity score of 6.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No numerical parameters are fitted to the data; the thinking-budget levels, temperature 0.6, repetition penalty 1.15, and per-dataset sample sizes are fixed experimental choices, not fitted values, so the free-parameter ledger is empty. No new entities are introduced; the malicious-token set M in Section 4.1 is a notational device for analysis, not a new object with independent empirical support requirements.

assumptions (5)

standard math For an autoregressive language model, token probabilities satisfy Pr[E_k] <= Pr[E_{k+1}] for nested events E_k.
Section 4.1 relies on probability measure monotonicity for the inverse scaling argument.
domain assumption The set M of malicious tokens (secret strings, unsafe words, injected instructions) can be identified in reasoning chains by the evaluation datasets or the evaluator.
Section 4.2 measures robustness as absence of such tokens; the validity of the whole exposed-chain metric rests on this.
domain assumption Budget forcing via appending 'Wait,' implements a valid increase of inference-time compute that is representative of reasoning scaling.
Sections 2.1 and 3; the paper's conclusions about inference-time scaling depend on this being an adequate operationalization from Muennighoff et al., 2025.
domain assumption GPT-4o-mini as automated judge correctly classifies refusals for harmful requests.
Section 2.2; no validation or agreement analysis is reported for the harmful-request robustness curves.
domain assumption Simulated tool-use prompting in open-source models is a representative proxy for real tool-integrated reasoning models such as o3 and Gemini.
Section 5.1 and Discussion; the paper itself flags this as a proxy, so the Section 5.1 robustness decline is conditional on this representativeness.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Does More Inference-Time Compute Really Help Robustness?." pith.science (2026). https://pith.science/paper/SUQBDZ2Y

@misc{pith2026250715974,
  author       = {Pith},
  title        = {Pith review of: Does More Inference-Time Compute Really Help Robustness?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SUQBDZ2Y}},
  note         = {Machine review of arXiv:2507.15974}
}

read the original abstract

Recently, Zaremba et al. demonstrated that increasing inference-time computation improves robustness in large proprietary reasoning LLMs. In this paper, we first show that smaller-scale, open-source models (e.g., DeepSeek R1, Qwen3, Phi-reasoning) can also benefit from inference-time scaling using a simple budget forcing strategy. More importantly, we reveal and critically examine an implicit assumption in prior work: intermediate reasoning steps are hidden from adversaries. By relaxing this assumption, we identify an important security risk, intuitively motivated and empirically verified as an inverse scaling law: if intermediate reasoning steps become explicitly accessible, increased inference-time computation consistently reduces model robustness. Finally, we discuss practical scenarios where models with hidden reasoning chains are still vulnerable to attacks, such as models with tool-integrated reasoning and advanced reasoning extraction attacks. Our findings collectively demonstrate that the robustness benefits of inference-time scaling depend heavily on the adversarial setting and deployment context. We urge practitioners to carefully weigh these subtle trade-offs before applying inference-time scaling in security-sensitive, real-world applications.

Figures

Figures reproduced from arXiv: 2507.15974 by the authors.

**Figure 1.** Inference-time scaling and robustness. (Left) We show that increasing inferencetime computation, by extending reasoning chains, can either improve robustness or at least maintain model robustness when only the final output is considered. (Right) However, we also identify an inverse scaling law: when intermediate reasoning steps are exposed to adversaries, increased inference-time computation consistently reduces ro… view at source ↗

**Figure 2.** Illustrative examples of three common attack types on LLM-based systems: [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Robustness evaluation across inference-time computation for multiple open-source [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Robustness evaluation of multiple reasoning models with varying inference-time [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Robustness of reasoning models against prompt injection attacks targeting tool-augmented reasoning. Robustness declines as inference-time computation increases. Recently, reasoning chains have been augmented with tool-use capabilities, as exemplified by OpenAI’s o3 and…

**Figure 6.** Figure 6: An example of a prompt injection attack. The main instruction, associated data, [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

**Figure 7.** Figure 7: A demonstration of a prompt extraction attack. The system instruction and [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

**Figure 8.** Figure 8: An illustrative example demonstrating a prompt injection attack targeting tool [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 3 canonical work pages

[2]

Training a helpful and harmless assistant with reinforcement learning from human feedback

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862,

arXiv
[5]

Brown, Nicholas Joseph, Sam McCandlish, Christopher Olah, Jared Kaplan, and Jack Clark

Deep Ganguli, Liane Lovitt, John Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Benjamin Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zachary Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnst...

arXiv
[6]

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen

Available at https://storage.googleapis.com/ deepmind-media/gemini/gemini_v2_5_report.pdf. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452,

arXiv
[7]

org/abs/2506.15674

URL https://arxiv. org/abs/2506.15674. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security , AISec ’23, pp. 79–90, N...

arXiv
[9]

Under review

10 Preprint. Under review. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

arXiv
[10]

Openai o1 system card

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720,

arXiv
[11]

Safechain: Safety of language models with long chain-of-thought reasoning capabilities

Fengqing Jiang, Zhangchen Xu, Yuetai Li, Luyao Niu, Zhen Xiang, Bo Li, Bill Yuchen Lin, and Radha Poovendran. Safechain: Safety of language models with long chain-of-thought reasoning capabilities. arXiv preprint arXiv:2502.12025,

arXiv
[12]

Search- r1: Training llms to reason and leverage search engines with reinforcement learning.ArXiv, abs/2503.09516,

Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search- r1: Training llms to reason and leverage search engines with reinforcement learning.ArXiv, abs/2503.09516,

arXiv

Show all 34 references

[13]

Martin Kuo, Jianyi Zhang, Aolin Ding, Qinsi Wang, Louis DiValentin, Yujia Bao, Wei Wei, Hai Li, and Yiran Chen

URL https://api.semanticscholar.org/CorpusID:276937772. Martin Kuo, Jianyi Zhang, Aolin Ding, Qinsi Wang, Louis DiValentin, Yujia Bao, Wei Wei, Hai Li, and Yiran Chen. H-cot: Hijacking the chain-of-thought safety reasoning mechanism to jailbreak large reasoning models, includi...

arXiv
[14]

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe

URL https://arxiv.org/abs/2503.04625. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050,

arXiv
[15]

Deepseek-r1 thoughtology: Let’s think about llm reasoning

Sara Vera Marjanovi´ c, Arkil Patel, Vaibhav Adlakha, Milad Aghajohari, Parishad BehnamGhader, Mehar Bhatia, Aditi Khandelwal, Austin Kraft, Benno Krojer, Xing Han Lù, et al. Deepseek-r1 thoughtology: Let’s think about llm reasoning. arXiv preprint arXiv:2504.07128,
[16]

Saro: Enhancing llm safety through reasoning-based alignment

Yutao Mou, Yuxiao Luo, Shikun Zhang, and Wei Ye. Saro: Enhancing llm safety through reasoning-based alignment. arXiv preprint arXiv:2504.09420,

arXiv
[17]

s1: Simple test-time scaling

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393,

arXiv
[18]

net/forum?id=qzEzXnw4ng

URL https://openreview. net/forum?id=qzEzXnw4ng. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

arXiv
[19]

R1-searcher: Incentivizing the search capability in llms via re- inforcement learning

Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via re- inforcement learning. ArXiv, abs/2503.05592,

arXiv
[20]

net/forum?id=fsW7wJGLBd

URLhttps://openreview. net/forum?id=fsW7wJGLBd. Eric Wallace, Kai Yuanqing Xiao, Reimar Heinrich Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training llms to prioritize privileged instruc- tions. arXiv preprint arXiv:2404.13208,

arXiv
[21]

Safety in large reasoning models: A survey

Cheng Wang, Yue Liu, Baolong Bi, Duzhen Zhang, Zhong-Zhi Li, Yingwei Ma, Yufei He, Shengju Yu, Xinfeng Li, Junfeng Fang, et al. Safety in large reasoning models: A survey. arXiv preprint arXiv:2504.17704, 2025a. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sha...

arXiv
[22]

Star-1: Safer alignment of reasoning llms with 1k data

Zijun Wang, Haoqin Tu, Yuhan Wang, Juncheng Wu, Jieru Mei, Brian R Bartoldson, Bhavya Kailkhura, and Cihang Xie. Star-1: Safer alignment of reasoning llms with 1k data. arXiv preprint arXiv:2504.01903, 2025b. Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, ...
[23]

Autogen: Enabling next-gen llm applica- tions via multi-agent conversations

Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applica- tions via multi-agent conversations. In First Conference on Language Modeling, 2024a. Tong Wu, Chong Xiang,...

arXiv
[24]

Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models

Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724, 2024b. xAI. Grok 3 beta — the age of reasoning agents, February 19

arXiv
[25]

Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, et al

URL https://x.ai/ news/grok-3. Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, et al. Sorry-bench: Systematically evaluating large language model safety refusal behaviors. arXiv preprint arXiv...

arXiv
[26]

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan

URLhttps://arxiv.org/abs/2505.09388. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822,

arXiv
[27]

A mousetrap: Fooling large reasoning models for jailbreak with chain of iterative chaos

Yang Yao, Xuan Tong, Ruofan Wang, Yixu Wang, Lujundong Li, Liang Liu, Yan Teng, and Yingchun Wang. A mousetrap: Fooling large reasoning models for jailbreak with chain of iterative chaos. arXiv preprint arXiv:2502.15806,

arXiv
[28]

Under review

12 Preprint. Under review. Wojciech Zaremba, Evgenia Nitishinskaya, Boaz Barak, Stephanie Lin, Sam Toyer, Yaodong Yu, Rachel Dias, Eric Wallace, Kai Xiao, Johannes Heidecke, et al. Trading inference-time compute for adversarial robustness. arXiv preprint arXiv:2501.18841,

arXiv
[29]

Realsafe- r1: Safety-aligned deepseek-r1 without compromising reasoning capability

Yichi Zhang, Zihao Zeng, Dongbai Li, Yao Huang, Zhijie Deng, and Yinpeng Dong. Realsafe- r1: Safety-aligned deepseek-r1 without compromising reasoning capability. arXiv preprint arXiv:2504.10081, 2025a. Yichi Zhang, Siyuan Zhang, Yao Huang, Zeyu Xia, Zhengwei Fang, Xiao Yang, ...

arXiv
[30]

Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang

URL https: //openreview.net/forum?id=0o95CVdNuz. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406,

arXiv
[31]

The hidden risks of large reasoning models: A safety assessment of r1

Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Shreedhar Jangam, Jayanth Srinivasa, Gaowen Liu, Dawn Song, and Xin Eric Wang. The hidden risks of large reasoning models: A safety assessment of r1. arXiv preprint arXiv:2502.12659,
[32]

13 Preprint

URL https:// openreview.net/forum?id=8EtSBX41mt. 13 Preprint. Under review. A Details of Experiments In this appendix, we provide complete details about the experiments conducted in the main text, including prompt injection, prompt extraction, and harmful requests. We then pre...

2025
[33]

The main instruction, associated data, low-priority query, and witness are shown

Figure 6: An example of a prompt injection attack. The main instruction, associated data, low-priority query, and witness are shown. We highlight the added guidance intended to help the model mitigate the attack. The model is expected to follow the main instruction and ignore ...

2024
[34]

The system instruction and malicious user prompt are shown

Figure 7: A demonstration of a prompt extraction attack. The system instruction and malicious user prompt are shown. A detailed specification is highlighted. The model is expected to avoid revealing the secret passcode in its response. 14 Preprint. Under review. Harmful Reques...

2024
[2022]

Theoretical guarantees on the best-of-n alignment policy

Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D’Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. arXiv preprint arXiv:2401.01879,

arXiv
[2023]

ISBN 9798400702600

Association for Computing Machinery. ISBN 9798400702600. doi: 10.1145/3605764. 3623985. URL https://doi.org/10.1145/3605764.3623985. Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. De...
[2024]

Large language monkeys: Scaling inference compute with repeated sampling

Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787,

arXiv
[2025]

Pranjal Aggarwal and Sean Welleck

URLhttps://arxiv.org/abs/2504.21318. Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697,

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [2]

Training a helpful and harmless assistant with reinforcement learning from human feedback

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862,

arXiv

[2] [5]

Brown, Nicholas Joseph, Sam McCandlish, Christopher Olah, Jared Kaplan, and Jack Clark

Deep Ganguli, Liane Lovitt, John Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Benjamin Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zachary Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnst...

arXiv

[3] [6]

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen

Available at https://storage.googleapis.com/ deepmind-media/gemini/gemini_v2_5_report.pdf. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452,

arXiv

[4] [7]

org/abs/2506.15674

URL https://arxiv. org/abs/2506.15674. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security , AISec ’23, pp. 79–90, N...

arXiv

[5] [9]

Under review

10 Preprint. Under review. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

arXiv

[6] [10]

Openai o1 system card

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720,

arXiv

[7] [11]

Safechain: Safety of language models with long chain-of-thought reasoning capabilities

Fengqing Jiang, Zhangchen Xu, Yuetai Li, Luyao Niu, Zhen Xiang, Bo Li, Bill Yuchen Lin, and Radha Poovendran. Safechain: Safety of language models with long chain-of-thought reasoning capabilities. arXiv preprint arXiv:2502.12025,

arXiv

[8] [12]

Search- r1: Training llms to reason and leverage search engines with reinforcement learning.ArXiv, abs/2503.09516,

Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search- r1: Training llms to reason and leverage search engines with reinforcement learning.ArXiv, abs/2503.09516,

arXiv