Pith. sign in

REVIEW 4 major objections 6 minor 5 cited by

Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuning

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper establishes a general negative correlation between LLM reasoning gains and safety, and shows fine-tuning on long chain-of-thought data can cut safety scores by two-thirds.

desk verdict Broad, useful map of safety-reasoning trade-offs, but the single unvalidated judge and 1024-token truncation keep the headline numbers from being established. read the letter →

arxiv 2502.09673 v2 pith:SHX6WXL7 submitted 2025-02-13 cs.CL cs.AI

Ang Li , Yichuan Mo , Mingjie Li , Yifei Wang , Yisen Wang This is my paper · ORCID
classification cs.CLcs.AI
keywords LLMsafetychain-of-thoughtjailbreakattacksreasoningfine-tuningsafety-reasoningtrade-offtoxicthoughtleakageHarmBenchGSM8K
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that methods for improving LLM reasoning, whether through prompting or fine-tuning, tend to degrade safety, and that the effect can be catastrophic for long chain-of-thought fine-tuning. Across 12 prompts on 7 open-source LLMs, reasoning accuracy and safety are negatively correlated (Pearson C less than -0.75, p less than 0.01) under three jailbreak attacks. Fine-tuning on o1-style long CoT data drops Llama-2-7B-Chat's direct safety score from 0.844 to 0.331 while raising GSM8K accuracy from 16% to 41.4%. The paper also documents failure modes such as toxic thought leakage and shows that mixing reflective safety data into fine-tuning recovers much of the safety loss without sacrificing accuracy.

What carries the argument

The central object is the empirical safety-reasoning trade-off, measured by pairing GSM8K accuracy with safety scores under three jailbreak attacks (GCG, AIR, and the decomposition-based attack). The machinery that carries the argument is a controlled comparison across prompt variants, including chain-of-thought prompts, reflection prompts, and safety-oriented prompts, and across fine-tuning datasets such as Alpaca, math-code, and the long CoT datasets OpenO1-SFT and Sky-T1. Safety is scored by an automated judge that reads only the last 1024 tokens of each response, and the long CoT fine-tuning transformation is what produces the catastrophic drops.

What would settle it

Re-evaluate a random sample of the long-CoT fine-tuned models' responses (e.g., Llama-2-7B-Chat trained on full OpenO1-SFT) using human annotators or a second judge that reads entire responses, and check whether the direct safety score still drops from 0.844 to around 0.331; additionally, re-optimize the GCG suffix against the fine-tuned weights and see whether the negative correlation across prompts survives.

Watch

Extended reading notes

Core claim

The central claim is that there is a general trade-off between reasoning and safety: methods that improve reasoning on GSM8K systematically lower measured safety, and the cost is particularly severe when models are fine-tuned on long chain-of-thought data in the style of OpenAI-o1. For prompting, the paper reports a strong negative Pearson correlation between accuracy and jailbreak safety across 12 prompts and 7 models for all three attacks (GCG, AIR, and the decomposition-based attack), with C less than -0.75 and p less than 0.01. For fine-tuning, it finds that CoT-style responses induce an additional safety tax over default and helpful response styles, and that training on the full OpenO1-SFT dataset reduces Llama-2-7B-Chat's direct safety score from 0.844 to 0.331, a 66% drop. The paper identifies three failure modes in long-CoT models: more practical unsafe responses, toxic thought leakage (unsafe content in reasoning traces despite a safe final answer), and user-intention alternation. Finally, it proposes mitigation: mixing in a small amount of safety reflection data substantially recovers safety without sacrificing accuracy.

Load-bearing premise

All safety measurements rest on one automated judge, a Llama-2-13B model fine-tuned by the paper's safety evaluation framework, scoring only the last 1024 tokens of each response; if that judge is biased or insensitive, especially on long chain-of-thought outputs, the reported safety drops would not reflect true model safety.

Editorial extensions

If this is right

  • Safety evaluation that only looks at final outputs will miss unsafe content hidden in reasoning traces, so long-CoT models need new evaluation protocols.
  • Reasoning-enhancing prompting in production should include safety reminders or demonstrations, since pure CoT prompts measurably lower jailbreak safety.
  • Fine-tuning on long-CoT datasets should be treated as a safety-critical intervention, with drops of at least 30% in the worst cases for both OpenO1 and Sky-T1.
  • Mixing small amounts of reflective safety data into long-CoT fine-tuning can reduce the safety drop by more than 30% while improving GSM8K accuracy over direct-refusal data.
  • The negative correlation extends to privacy: reasoning prompts and long-CoT fine-tuning also lower privacy scores, with Mistral's correlation dropping by 0.48.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported negative correlation may partly reflect judge bias: a judge trained on short responses could mis-classify long CoT outputs, so a human or full-response judge study would separate genuine safety degradation from evaluation artifact.
  • If the trade-off is real, it suggests a 'reasoning tax' in which capability gains in math and coding are bought at the expense of alignment, implying safety budgets should scale with reasoning training rather than being treated as constants.
  • The mitigation result hints that reflective safety data may act as a regularizer, keeping the model's reasoning schema from overfitting to harmful templates; this could be tested by ablating the reflection component.
  • The extension to privacy suggests the trade-off may generalize to other trustworthiness dimensions such as bias or truthfulness, testable by running the same prompt and fine-tuning sweep against additional trust benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper empirically examines whether improving LLM reasoning through prompting and fine-tuning trades off against safety. The authors evaluate 12 prompts on 7 open-weight LLMs, measuring GSM8K accuracy and HarmBench-based safety scores under GCG, AIR, and decomposition attacks, and report a strong negative Pearson correlation between prompt-averaged accuracy and jailbreak safety. They further fine-tune Llama-2-7B-Chat and Mistral-7B-Instruct on CoT-style, math/code, and long-CoT (OpenO1-SFT, Sky-T1) data, reporting that CoT-style responses and especially long-CoT data degrade direct-query safety (e.g., 0.844 to 0.331 for Llama-2 under CoT prompting after full OpenO1-SFT), and they propose safety-demo prompts and reflective safety data as mitigations. The paper also extends the analysis to privacy scores on TrustLLM.

Significance. If the empirical findings hold, the paper provides a useful and cautionary message: reasoning-enhancing methods are not safety-neutral, and long-CoT fine-tuning can produce large safety drops that final-output-only evaluations miss. The study is broad in scope, with complete per-model/per-prompt results in the appendices, several attack conditions, and concrete mitigation attempts that are directly actionable. The main results are, however, conditional on a single automated safety judge and a 1024-token truncation rule, so the quantitative claims—correlation magnitudes, 'catastrophic' drops, and scaling trends—need validation before they can be taken at face value.

major comments (4)
  1. [Appendix B.4, Fig. 2, Fig. 4, Tables 1-3, 12] All safety scores are produced by a single HarmBench-tuned Llama-2-13B judge applied to only the last 1024 tokens of each response, and the judge is never validated against human annotations or a second independent judge. This is load-bearing because every headline quantity—the Pearson correlations in Fig. 2, the CoT safety tax in Table 1, and the catastrophic drops in Fig. 4/Table 12—is a function of this judge. The truncation rule is particularly problematic for long-CoT responses: Appendix E documents 'toxic thought leakage' in which harmful content appears in the thinking trace, which may fall outside the 1024-token window, and the judge may also be format-sensitive to long structured outputs. Please validate the judge on a human-labeled subset, report agreement with a second judge (e.g., GPT-4 or Llama-3 based), and run a sensitivity analysis varying the truncation window; otherwise the reported trade-off and drops cannot be distinguished from judge artifacts.
  2. [Appendix B.2, Table 12] The 1% subset of NovaSky-Sky-T1 is omitted post hoc because 'models fine-tuned on this setting exhibit strong overfitting,' and the paper reports no repeated runs or error bars for any fine-tuning condition. This matters because the claimed U-shaped scaling trend for Sky-T1 in Fig. 4(b) is based on the remaining points, and the exclusion criterion was not pre-specified. Please provide the omitted numbers, justify the exclusion with a quantitative overfitting measure, or show that the trend is robust to inclusion; at minimum, report variance across seeds.
  3. [§3.1, Fig. 2] The negative-correlation claim is computed over 12 prompt-level averages (the caption says eleven prompts), with no confidence intervals and without accounting for the nested structure of the data (multiple models per prompt). With N≈12, a single prompt can drive the correlation, and the reported p-values (0.004, 0.008, 0.0002) treat prompts as independent observations. Please report per-model correlations, bootstrap confidence intervals, and the correlation after excluding the two safety-oriented prompts (ICD, Ref-S), which by construction occupy the low-accuracy/high-safety corner.
  4. [§4.1, footnote 1 and Table 10] The GCG suffix was generated on the base Llama-2-7B-Chat and, as the paper acknowledges, becomes 'less targeted' after fine-tuning because the weights change. Since Table 1 averages jailbreak safety over GCG, AIR, and Decom., the post-fine-tuning GCG scores are not comparable across models and can mask safety degradation; for example, GCG scores in Table 10 rise after fine-tuning on Alpaca. Please recompute the fine-tuning safety comparisons with attacks that are regenerated for each fine-tuned model, or report the averages without GCG and show that the 'CoT style is worst' conclusion still holds.
minor comments (6)
  1. [Figure 2 caption] The caption says 'eleven prompts' while the text and Appendix B.1 list twelve; please harmonize.
  2. [Table 11 caption] The caption states 'For Safety scores, lower values are better,' which contradicts the 'safety higher is better' convention used elsewhere; correct the caption.
  3. [§4.1] The text refers to 'Table 10 (Appendix 10)'; the appendix is C.2, not 'Appendix 10'.
  4. [Throughout] There are several typos: 'accuries' (Appendix D.1), 'it to comes' (§3.1), 'Anthrophic' (§4.3), and inconsistent capitalization of 'Cot' vs 'CoT'.
  5. [Appendix B.4] The paper does not state the number of harmful instructions in the safety evaluation set; please report it, along with the number of responses scored by the judge.
  6. [Appendix B.4] The GCG generation says 'ensembling three models' but footnote 1 says 'We ensemble Llama-2-7B-Chat'; please clarify which models were ensembled.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the core claims are measured correlations and safety-score drops, not derivations from fitted inputs; the one in-house citation (ICD prompt) is not load-bearing.

full rationale

The paper's central claims are empirical: a negative Pearson correlation between GSM8K accuracy and HarmBench safety scores across 12 prompts (Fig. 2), and drops in direct-query safety after CoT/long-CoT fine-tuning (Fig. 4, Tables 1-3). These are observations of measured quantities, not quantities derived from a model whose parameters were fitted to the same data. No fitted parameter is renamed as a prediction; no equation defines safety in terms of accuracy or vice versa; no uniqueness theorem is invoked. Safety scores use the external HarmBench LLM judge, and the reported 1024-token truncation is a stated evaluation protocol, not a definitional tie to the reasoning metric. The only in-house citation is Wei et al. 2023 for the ICD safety prompt and ICL-jailbreaking context; ICD is one of twelve prompts in the correlation plot and is not used to establish the trade-off, so removing it would not change the fine-tuning safety results or the general correlation. Limitations about judge truncation and GCG transfer are methodological validity concerns, not circularity. Score 1 reflects the non-load-bearing self-citation rather than any circular step.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities apply. The evaluation relies on standard benchmarks (HarmBench, GSM8K, TrustLLM) and standard attacks; the assumptions listed are the ones that, if false, would change the reported safety numbers.

assumptions (4)
  • domain assumption HarmBench's Llama-2-13B judge correctly classifies response safety
    The paper uses this judge for all safety scores; no human validation or alternate judge is reported (Appendix B.4).
  • domain assumption GSM8K accuracy is a sufficient proxy for reasoning ability
    All reasoning gains are measured only on GSM8K grade-school math; the 'reasoning' conclusion may not extend to other reasoning domains (Section 2.1).
  • ad hoc to paper The last 1024 tokens of a response are sufficient for safety classification
    Chosen for efficiency, not validated; could miss unsafe content in earlier tokens or in long CoT thoughts (Appendix B.4).
  • domain assumption The GCG suffix optimized on base Llama-2 remains a fair attack after fine-tuning
    The paper acknowledges in a footnote that the suffix becomes less targeted after weight changes, so some safety improvements may be attack transfer artifacts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuning." pith.science (2026). https://pith.science/paper/SHX6WXL7

@misc{pith2026250209673,
  author       = {Pith},
  title        = {Pith review of: Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SHX6WXL7}},
  note         = {Machine review of arXiv:2502.09673}
}
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable success across various NLP benchmarks. However, excelling in complex tasks that require nuanced reasoning and precise decision-making demands more than raw language proficiency--LLMs must reason, i.e., think logically, draw from past experiences, and synthesize information to reach conclusions and take action. To enhance reasoning abilities, approaches such as prompting and fine-tuning have been widely explored. While these methods have led to clear improvements in reasoning, their impact on LLM safety remains less understood. In this work, we investigate the interplay between reasoning and safety in LLMs. We highlight the latent safety risks that arise as reasoning capabilities improve, shedding light on previously overlooked vulnerabilities. At the same time, we explore how reasoning itself can be leveraged to enhance safety, uncovering potential mitigation strategies. By examining both the risks and opportunities in reasoning-driven LLM safety, our study provides valuable insights for developing models that are not only more capable but also more trustworthy in real-world deployments.

Figures

Figures reproduced from arXiv: 2502.09673 by the authors.

Figure 1
Figure 1. Safety challenges in improving LLM reasoning: Starting from the default Llama-2-7B￾Chat, we experiment with improving its reasoning abilities via (1) zero-shot Chain-of-Thought (CoT) prompting, (2) few-shot CoT, (3) fine-tuning on a coding and math dataset, and (4) fine-tuning on the Open-o1-SFT dataset. As the accuracy in￾creases from 16% to 41.4%, we see the model expe￾riences a catastrophic drop of 66% in safety … view at source ↗
Figure 2
Figure 2. The negative correlation between accuracy and safety. We evaluate seven LLMs with eleven prompts under three jail￾break attacks, GCG, AIR, and the decompo￾sition attack (Decom.). Each dot stands for a prompt positioned by its averaged accuracy and safety across the models. We present the correlation coefficient (C) between accuracy and safety against each attack, all of which show strong negative correlation between… view at source ↗
Figure 3
Figure 3. Effects of different prompting strategies. Few-shot CoTs are denoted as -n, with -M and -S representing math and safety demonstrations, respectively. Sys-t1 and Sys-ref are reflection-based system prompts. ICD and Ref-S are safety-oriented prompts. (a) Using CoT prompts and adding math-specific CoT examples decreases safety compared to the default system prompt. (b) Reflection￾based system prompts lower safety score… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Long CoT fine-tuning can lead to catastrophic damage to safety. We fine-tune Llama-2- [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Effects of inference-time prompt￾ing to privacy. We see that incorporating prompts to improve reasoning will also bring negative effects to privacy. In this section, we generalize our observations on the changes of safety to the privacy risks of LLMs. Following the exp…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMs

    cs.AI 2025-11 conditional novelty 6.0 of 10

    Benign PEFT fine-tuning changes LLM safety and fairness: adapter-based methods (LoRA, IA3) preserve alignment better than prompt-based methods, and the base model strongly moderates outcomes.

  2. JADES: A Universal Framework for Jailbreak Assessment via Decompositional Scoring

    cs.CR 2025-08 conditional novelty 6.0 of 10

    JADES judges jailbreak success by decomposing harmful prompts into weighted sub-questions and scoring each part, claiming 98.5% human agreement and showing prior attack success rates are inflated.

  3. VisCRA: A Visual Chain Reasoning Attack for Jailbreaking Multimodal Large Language Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    VisCRA jailbreaks multimodal LLMs by masking the most harmful image region and using a two-stage reasoning prompt to make the model infer and then comply.

  4. A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models

    cs.CL 2025-09 conditional novelty 4.0 of 10

    A structured literature survey concluding that reasoning capabilities do not automatically make LLMs more trustworthy and can introduce new vulnerabilities in safety, robustness, and privacy.

  5. Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Fine-tuning small LLMs on benign data raises harmfulness scores, but those scores vary widely across random seeds, temperatures, and repeated runs, making single-run safety comparisons unreliable.

Reference graph

Works this paper leans on

68 extracted references · 58 canonical work pages · cited by 5 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv, 2023

    2023

  2. [2]

    Bowman, Ethan Perez, Roger Baker Grosse, and David Duvenaud

    Cem Anil, Esin DURMUS, Nina Rimsky, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel J Ford, Francesco Mosconi, Rajashree Agrawal, Rylan Schaeffer, Naomi Bashkansky, Samuel Svenningsen, Mike Lambert, Ansh Radhakrishnan, Carson Denison, Evan J Hubinger, Yuntao Bai, Trenton Bricken, Timothy Maxwell, Nicholas Schiefer, Ja...

    work page 2024

  3. [3]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    2022

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020

    2020

  5. [5]

    Bowman, Julian Michael, Ethan Perez, and Miles Turpin

    James Chua, Edward Rees, Hunar Batra, Samuel R. Bowman, Julian Michael, Ethan Perez, and Miles Turpin. Bias-augmented consistency training reduces biased reasoning in chain-of-thought. arXiv, 2024

    work page 2024

  6. [6]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv, 2021

    work page 2021

  7. [7]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    work page 2025

  8. [8]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv, 2024

    work page 2024

Show all 68 references
  1. [9]

    Reasoning about knowledge

    Ronald Fagin, Joseph Y Halpern, Yoram Moses, and Moshe Vardi. Reasoning about knowledge. MIT press, 2004

    2004

  2. [10]

    Statistics (international student edition)

    David Freedman, Robert Pisani, and Roger Purves. Statistics (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York, 2007

    2007

  3. [11]

    Approaches to studying formal and everyday reasoning

    Kathleen M Galotti. Approaches to studying formal and everyday reasoning. Psychological bulletin, 105 0 (3): 0 331, 1989

    1989

  4. [12]

    Omni-math: A universal olympiad level mathematic benchmark for large language models

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-math: A universal ...

    2024

  5. [13]

    Bias runs deep: Implicit reasoning biases in persona-assigned LLM s

    Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. Bias runs deep: Implicit reasoning biases in persona-assigned LLM s. In ICLR, 2024

    2024

  6. [14]

    In-context learning may not elicit trustworthy reasoning: A-not- B errors in pretrained language models

    Pengrui Han, Peiyang Song, Haofei Yu, and Jiaxuan You. In-context learning may not elicit trustworthy reasoning: A-not- B errors in pretrained language models. In EMNLP, 2024

    2024

  7. [15]

    Multi-modal latent space learning for chain-of-thought reasoning in language models

    Liqi He, Zuchao Li, Xiantao Cai, and Ping Wang. Multi-modal latent space learning for chain-of-thought reasoning in language models. In AAAI, 2025

    2025

  8. [16]

    What is in your safe data? identifying benign data that breaks safety

    Luxi He, Mengzhou Xia, and Peter Henderson. What is in your safe data? identifying benign data that breaks safety. In COLM, 2024

    2024

  9. [17]

    Large language models cannot self-correct reasoning yet

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv, 2023

    2023

  10. [18]

    Trustllm: Trustworthiness in large language models

    Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, et al. Trustllm: Trustworthiness in large language models. In ICML, 2024

    2024

  11. [19]

    Alpaca-python-18k

    iamtarun. Alpaca-python-18k. https://huggingface.co/datasets/\ /python\_code\_instructions\_18k\_alpaca, 2023

    2023

  12. [20]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv, 2024

    2024

  13. [21]

    Multitask-bench: Unveiling and mitigating safety gaps in LLM s fine-tuning

    Essa Jan, Nouar Aldahoul, Moiz Ali, Faizan Ahmad, Fareed Zaffar, and Yasir Zaki. Multitask-bench: Unveiling and mitigating safety gaps in LLM s fine-tuning. In COLING, 2025

    2025

  14. [22]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv, 2023

    2023

  15. [23]

    Enhancing question answering for enterprise knowledge bases using large language models

    Feihu Jiang, Chuan Qin, Kaichun Yao, Chuyu Fang, Fuzhen Zhuang, Hengshu Zhu, and Hui Xiong. Enhancing question answering for enterprise knowledge bases using large language models. In DASFAA, 2024

    2024

  16. [24]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS, 2022

    2022

  17. [25]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In SOSP, 2023

    2023

  18. [26]

    Deceptive semantic shortcuts on reasoning chains: How far can models go without hallucination? In NAACL, 2024 a

    Bangzheng Li, Ben Zhou, Fei Wang, Xingyu Fu, Dan Roth, and Muhao Chen. Deceptive semantic shortcuts on reasoning chains: How far can models go without hallucination? In NAACL, 2024 a

    2024

  19. [27]

    D r A ttack: Prompt decomposition and reconstruction makes powerful LLM s jailbreakers

    Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. D r A ttack: Prompt decomposition and reconstruction makes powerful LLM s jailbreakers. In EMNLP, 2024 b

    2024

  20. [28]

    Retrieval-augmented multi-modal chain-of-thoughts reasoning for large language models

    Bingshuai Liu, Chenyang Lyu, Zijun Min, Zhanyu Wang, Jinsong Su, and Longyue Wang. Retrieval-augmented multi-modal chain-of-thoughts reasoning for large language models. arXiv, 2023

    2023

  21. [29]

    Intelligence and reasoning

    David F Lohman and Joni M Lakin. Intelligence and reasoning. The Cambridge handbook of intelligence, 2011

    2011

  22. [30]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, 2022

    2022

  23. [31]

    Keeping LLM s aligned after fine-tuning: The crucial role of prompt templates

    Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Keeping LLM s aligned after fine-tuning: The crucial role of prompt templates. In ICLR Workshop on Reliable and Responsible Foundation Models, 2024

    2024

  24. [32]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv, 2024

    2024

  25. [33]

    What is reasoning? Mind, 127 0 (505): 0 167--196, 2018

    Conor McHugh and Jonathan Way. What is reasoning? Mind, 127 0 (505): 0 167--196, 2018

    2018

  26. [34]

    Sky-t1: Train your own o1 preview model within \ 450

    Team NovaSky. Sky-t1: Train your own o1 preview model within \ 450. https://novasky-ai.github.io/posts/sky-t1, 2025. Accessed: 2025-01-09

    2025

  27. [35]

    O1-open/openo1-sft

    OpenO1. O1-open/openo1-sft. https://huggingface.co/datasets/O1-OPEN/OpenO1-SFT, 2025. Accessed: 2025-01-09

    2025

  28. [36]

    Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning

    Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning. In EMNLP, 2024

    2024

  29. [37]

    Fine-tuning aligned language models compromises safety, even when users do not intend to! In ICLR, 2024

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In ICLR, 2024

    2024

  30. [38]

    Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson E. Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, John Kernion, Kamil.e Lukovsiut.e, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Sam McCandlish, Sheer El Showk, Tamera Lanham, Tim Maxw...

    2023

  31. [39]

    On second thought, let`s not think step by step! bias and toxicity in zero-shot reasoning

    Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On second thought, let`s not think step by step! bias and toxicity in zero-shot reasoning. In ACL, 2023

    2023

  32. [40]

    Reflexion: an autonomous agent with dynamic memory and self-reflection

    Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv: Arxiv-2303.11366, 2023

    2023 arXiv

  33. [41]

    Automatic prompt augmentation and selection with chain-of-thought from labeled data

    KaShun Shum, Shizhe Diao, and Tong Zhang. Automatic prompt augmentation and selection with chain-of-thought from labeled data. arXiv, 2023

    2023

  34. [42]

    mattshumer/reflection-llama-3.1-70b

    Matt Shumer. mattshumer/reflection-llama-3.1-70b. https://huggingface.co/mattshumer/Reflection-Llama-3.1-70B, 2024

    2024

  35. [43]

    Barr, and Wei Le

    Benjamin Steenhoek, Md Mahbubur Rahman, Monoshi Kumar Roy, Mirza Sanjida Alam, Hengbo Tong, Swarna Das, Earl T. Barr, and Wei Le. To err is machine: Vulnerability detection challenges llm reasoning, 2025

    2025

  36. [44]

    Story centaur: Large language model few shot learning as a creative writing tool

    Ben Swanson, Kory Mathewson, Ben Pietrzak, Sherol Chen, and Monica Dinalescu. Story centaur: Large language model few shot learning as a creative writing tool. In EACL, 2021

    2021

  37. [45]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford-alpaca, 2023

    2023

  38. [46]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023

  39. [47]

    Rush, and Thomas Wolf

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alvaro Bartolome, Alexander M. Rush, and Thomas Wolf. The Alignment Handbook . URL https://github.com/huggingface/alignment-handbook

  40. [48]

    Decodingtrust: A comprehensive assessment of trustworthiness in gpt models

    Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In NeurIPS, 2023

    2023

  41. [49]

    Drt-o1: Optimized deep reasoning translation via long chain-of-thought

    Jiaan Wang, Fandong Meng, Yunlong Liang, and Jie Zhou. Drt-o1: Optimized deep reasoning translation via long chain-of-thought. arXiv, 2024 a

    2024

  42. [50]

    Openr: An open source framework for advanced reasoning with large language models

    Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, et al. Openr: An open source framework for advanced reasoning with large language models. arXiv, 2024 b

    2024

  43. [51]

    Stop reasoning! when multimodal LLM with chain-of-thought reasoning meets adversarial image

    Zefeng Wang, Zhen Han, Shuo Chen, Fan Xue, Zifeng Ding, Xun Xiao, Volker Tresp, Philip Torr, and Jindong Gu. Stop reasoning! when multimodal LLM with chain-of-thought reasoning meets adversarial image. In COLM, 2024 c

    2024

  44. [52]

    Reasoning about a rule

    Peter C Wason. Reasoning about a rule. Quarterly journal of experimental psychology, 20 0 (3): 0 273--281, 1968

    1968

  45. [53]

    Psychology of reasoning: Structure and content, volume 86

    Peter Cathcart Wason and Philip Nicholas Johnson-Laird. Psychology of reasoning: Structure and content, volume 86. Harvard University Press, 1972

    1972

  46. [54]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022

    2022

  47. [55]

    Jailbreak and guard aligned language models with only few in-context demonstrations

    Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv, 2023

    2023

  48. [56]

    Separate the wheat from the chaff: A post-hoc approach to safety re-alignment for fine-tuned language models

    Di Wu, Xin Lu, Yanyan Zhao, and Bing Qin. Separate the wheat from the chaff: A post-hoc approach to safety re-alignment for fine-tuned language models. arXiv, 2024 a

    2024

  49. [57]

    You know what i'm saying: Jailbreak attack via implicit reference

    Tianyu Wu, Lingrui Mei, Ruibin Yuan, Lujun Li, Wei Xue, and Yike Guo. You know what i'm saying: Jailbreak attack via implicit reference. arXiv, 2024 b

    2024

  50. [58]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via se...

    2024

  51. [59]

    Wordcraft: story writing with large language models

    Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. Wordcraft: story writing with large language models. In IUI, 2022

    2022

  52. [60]

    Prompting large language model for machine translation: A case study

    Biao Zhang, Barry Haddow, and Alexandra Birch. Prompting large language model for machine translation: A case study. In ICML, 2023

    2023

  53. [61]

    Grease LM : Graph REAS oning enhanced language models

    Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D Manning, and Jure Leskovec. Grease LM : Graph REAS oning enhanced language models. In ICLR, 2022 a

    2022

  54. [62]

    S afety B ench: Evaluating the safety of large language models

    Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. S afety B ench: Evaluating the safety of large language models. In ACL, 2024

    2024

  55. [63]

    Automatic chain of thought prompting in large language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv, 2022 b

    2022

  56. [64]

    Verify-and-edit: A knowledge-enhanced chain-of-thought framework

    Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. Verify-and-edit: A knowledge-enhanced chain-of-thought framework. arXiv, 2023

    2023

  57. [65]

    Marco-o1: Towards open reasoning models for open-ended solutions

    Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. Marco-o1: Towards open reasoning models for open-ended solutions. arXiv, 2024

    2024

  58. [66]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS, 2024

    2024

  59. [67]

    Rethinking machine ethics -- can LLM s perform moral reasoning through the lens of moral theories? In ACL, 2024

    Jingyan Zhou, Minda Hu, Junan Li, Xiaoying Zhang, Xixin Wu, Irwin King, and Helen Meng. Rethinking machine ethics -- can LLM s perform moral reasoning through the lens of moral theories? In ACL, 2024

    2024

  60. [68]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.