REVIEW 12 cited by
Enhancing LLM Reasoning with Reward-guided Tree Search
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recently, test-time scaling has garnered significant attention from the research community, largely due to the substantial advancements of the o1 model released by OpenAI. By allocating more computational resources during the inference phase, large language models~(LLMs) can extensively explore the solution space by generating more thought tokens or diverse solutions, thereby producing more accurate responses. However, developing an o1-like reasoning approach is challenging, and researchers have been making various attempts to advance this open area of research. In this paper, we present a preliminary exploration into enhancing the reasoning abilities of LLMs through reward-guided tree search algorithms. This framework is implemented by integrating the policy model, reward model, and search algorithm. It is primarily constructed around a tree search algorithm, where the policy model navigates a dynamically expanding tree guided by a specially trained reward model. The implemented framework is denoted as \textbf{STILL-1}. We thoroughly explore various design considerations necessary for implementing this framework and provide a detailed report of the technical aspects. To assess the effectiveness of our approach, we focus on mathematical reasoning tasks and conduct extensive evaluations on four challenging datasets, significantly enhancing the reasoning abilities of LLMs.
Forward citations
Cited by 12 Pith papers
-
Stepwise Reasoning Checkpoint Analysis: A Test Time Scaling Method to Enhance LLMs' Reasoning
A checkpoint-based search and candidate augmentation method improves small LLM mathematical reasoning accuracy over existing test-time scaling baselines.
-
Sticker-TTS: Learn to Utilize Historical Experience with a Sticker-driven Test-Time Scaling Framework
Sticker-TTS uses three collaborating models to distill and reuse compact summaries (stickers) of past reasoning attempts, improving math benchmark accuracy at a claimed equivalent inference cost.
-
Com$^2$: A Causal-Guided Benchmark for Exploring Complex Commonsense Reasoning in Large Language Models
Com2 is a causal-graph-guided benchmark with 3,754 questions showing that LLMs struggle with complex commonsense reasoning, particularly on intervention and transition tasks.
-
EquivPruner: Boosting Efficiency and Quality in LLM-Based Search via Action Pruning
EquivPruner detects mathematically equivalent reasoning steps during LLM tree search and keeps only one per group, cutting token use by up to half on GSM8K and MATH-500 without hurting accuracy.
-
TreeThink: A Modular Tree Search Library for Mathematical Reasoning with LLMs
TreeThink provides a modular, asynchronous tree-search library for neural theorem proving with unified REPL clients for Lean, Rocq, and Isabelle and up to 6.3× wall-clock speedup.
-
Let's Revise Step-by-Step: A Unified Local Search Framework for Code Generation with LLMs
The abstract claims a new local search framework for code generation, but the manuscript body is a different mathematics paper.
-
Ctrl-Z Sampling: Scaling Diffusion Sampling with Controlled Random Zigzag Explorations
Ctrl-Z Sampling improves text-to-image outputs by adaptively rolling back and re-exploring when a reward model flags a quality plateau, at roughly 3 to 9 times the usual compute.
-
VReST: Enhancing Reasoning in Large Vision-Language Models through Tree Search and Self-Reward Mechanism
VReST combines Monte Carlo tree search with a self-reward signal inside a vision-language model to get higher accuracy than CoT, ToT, or voting baselines on MathVista, MathVision, and CharXiv, while spending several t...
-
Reward Model Generalization for Compute-Aware Test-Time Reasoning
The paper claims PRM generalization error controls test-time compute efficiency and that the proposed CATS controller improves accuracy on MATH-500 and AIME24.
-
Why Does Reasoning Length Converge? Unveiling the Underfitting-Overfitting Trade-off in Chain-of-Thought
LLM chain-of-thought length converges to an optimum because of an underfitting-overfitting tradeoff, formalized in a continuous reasoning-space framework and tested with RL.
-
From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
A systematic analysis of LLM exploration in RLVR, introducing capability-boundary metrics and examining entropy-performance exchange across training stages and token levels.
-
Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning Capabilities Through Evaluation Design
Evaluation conditions like seed, dataset version, and answer ordering cause multi-point benchmark score swings in DeepSeek-R1-Distill and related reasoning models, undermining reliable comparison.
Discussion (0). Sign in to comment.