REVIEW 11 cited by
The Impact of Reasoning Step Length on Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Chain of Thought (CoT) is significant in improving the reasoning abilities of large language models (LLMs). However, the correlation between the effectiveness of CoT and the length of reasoning steps in prompts remains largely unknown. To shed light on this, we have conducted several empirical experiments to explore the relations. Specifically, we design experiments that expand and compress the rationale reasoning steps within CoT demonstrations while keeping all other factors constant. We have the following key findings. First, the results indicate that lengthening the reasoning steps in prompts, even without adding new information into the prompt, considerably enhances LLMs' reasoning abilities across multiple datasets. Alternatively, shortening the reasoning steps, even while preserving the key information, significantly diminishes the reasoning abilities of models. This finding highlights the importance of the number of steps in CoT prompts and provides practical guidance to make better use of LLMs' potential in complex problem-solving scenarios. Second, we also investigated the relationship between the performance of CoT and the rationales used in demonstrations. Surprisingly, the result shows that even incorrect rationales can yield favorable outcomes if they maintain the requisite length of inference. Third, we observed that the advantages of increasing reasoning steps are task-dependent: simpler tasks require fewer steps, whereas complex tasks gain significantly from longer inference sequences. The code is available at https://github.com/MingyuJ666/The-Impact-of-Reasoning-Step-Length-on-Large-Language-Models
Forward citations
Cited by 11 Pith papers
-
R2-Router: A New Paradigm for LLM Routing with Reasoning
R2-Router jointly selects the LLM and an output-token budget, modeling each model as a quality-cost curve rather than a fixed point, and reports 4-5x cost savings on its new R2-Bench.
-
Test-Time Scaling with Reflective Generative Model
MetaStone-S1 combines a shared policy and self-supervised process reward head to select high-quality reasoning traces, reaching o3-mini-level scores at 32B parameters.
-
ReCUT: Balancing Reasoning Length and Accuracy in LLMs via Stepwise Trails and Preference Optimization
ReCUT combines stepwise long-short sampling, dual DPO training, and DARE-Ties parameter interpolation to compress LLM reasoning chains by 30-50% without sacrificing accuracy on math benchmarks.
-
Overclocking LLM Reasoning: Monitoring and Controlling Thinking Path Lengths in LLMs
A fitted progress vector in LLM hidden states can monitor and, when added during decoding, shorten the thinking phase, often improving accuracy and cutting tokens.
-
Cell-o1: Training LLMs to Solve Single-Cell Reasoning Puzzles with Reinforcement Learning
A 7B model trained with reasoning distillation and reinforcement learning reaches 32.9% batch-level accuracy on a new single-cell annotation benchmark, versus 19.0% for OpenAI's o1.
-
Enhancing LLMs' Reasoning-Intensive Multimedia Search Capabilities through Fine-Tuning and Reinforcement Learning
A two-stage fine-tuning and reinforcement-learning method makes LLMs generate token-efficient natural-language search plans, reporting strong accuracy gains on financial and news search benchmarks.
-
ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark
ARB provides 1,356 Arabic multimodal questions with 5,119 human-reviewed reasoning steps and shows leading models score much higher on reasoning fluency than on correct answers.
-
Reasoning or Overthinking: Evaluating Large Language Models on Financial Sentiment Analysis
On financial sentiment classification, zero-shot LLMs match human labels better without chain-of-thought reasoning than with it.
-
Reasoning Can Hurt the Inductive Abilities of Large Language Models
Chain-of-thought reasoning can hurt LLMs' ability to infer hidden rules from gameplay transcripts, and structured interventions recover the lost accuracy.
-
Intelligent Assistants for the Semiconductor Failure Analysis with LLM-Based Planning Agents
An LLM-based planning agent using ReAct, retrieval tools, and RAG can autonomously answer semiconductor failure analysis queries in an industrial setting, with best average quality 3.6/5.
-
Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey
A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.
Discussion (0). Sign in to comment.