REVIEW 21 cited by
Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent works like Tree-of-Thought (ToT) and Reasoning via Planning (RAP) aim to augment the reasoning capabilities of LLMs by using tree-search algorithms to guide multi-step reasoning. These methods rely on prompting a pre-trained model to serve as a value function and focus on problems with low search depth. As a result, these methods will not work in domains where the pre-trained LLM does not have enough knowledge to serve as an effective value function or in domains that require long-horizon planning. To address these limitations, we present an AlphaZero-like tree-search learning framework for LLMs (termed TS-LLM), systematically illustrating how tree-search with a learned value function can guide LLM decoding. TS-LLM distinguishes itself in two key ways. (1) Leveraging a learned value function and AlphaZero-like algorithms, our approach can be generally adaptable to a wide range of tasks, language models of any size, and tasks of varying search depths. (2) Our approach can guide LLMs during both inference and training, iteratively improving the LLM. Empirical results across reasoning, planning, alignment, and decision-making tasks show that TS-LLM outperforms existing approaches and can handle trees with a depth of 64.
Forward citations
Cited by 21 Pith papers
-
APPO: Agentic Procedural Policy Optimization
APPO refines branching and credit assignment in agentic RL via a Branching Score and procedure-level scaling, improving baselines by nearly 4 points on 13 benchmarks.
-
LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacra
The LLM Economist framework couples persona-conditioned worker agents with an in-context RL planner to search US-bracket tax schedules, yet its Saez benchmark is derived from the planner's own solution and its headlin...
-
It's Not That Simple. An Analysis of Simple Test-Time Scaling
The apparent scaling behavior of the s1 model is mostly an artifact of truncating long outputs, while appending 'Wait' does not reliably improve answers.
-
Enhancing Test-Time Scaling of Large Language Models with Hierarchical Retrieval-Augmented MCTS
R2-LLMs retrieves hierarchical problem templates and step-level examples during Monte Carlo tree search, improving math reasoning accuracy of small LLMs at inference time without additional training.
-
Enhancing User Engagement in Socially-Driven Dialogue through Interactive LLM Alignments
Dialogue agents aligned via DPO on preference pairs mined from simulated conversations improve engagement scores against the same simulator, with smaller and partially inconsistent human evaluation evidence.
-
TreeRL: LLM Reinforcement Learning with On-Policy Tree Search
TreeRL combines entropy-guided tree search with on-policy RL and tree-derived process rewards, reporting modest accuracy gains over chain sampling on math and code benchmarks.
-
Kinetics: Rethinking Test-Time Scaling Laws
A memory-aware test-time scaling law shows small models are overestimated and sparse attention is needed for efficient scaling.
-
Structured Pruning for Diverse Best-of-N Reasoning Optimization
SPRINT learns to select which attention heads to prune per question, improving Pass@N over random head selection and multinomial sampling on MATH500 and GSM8K.
-
How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoning
On three controlled tasks, the optimal number of backtracks in SFT warm-up data grows with task difficulty, while trajectory correctness has little effect on final RL performance.
-
MMATH: A Multilingual Benchmark for Mathematical Reasoning
A new multilingual math benchmark shows that reasoning models often respond in the wrong language, and English-reasoning training improves both accuracy and language consistency.
-
Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning
A plan-then-reason SFT plus a plan-quality reward in GRPO improves math-reasoning accuracy by small but consistent margins over GRPO and DAPO.
-
CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
Adding actor perplexity and multi-head critic variance as intrinsic exploration bonuses improves RLVR math reasoning accuracy by roughly +2 to +3 points on AIME benchmarks.
-
Scale, Don't Fine-tune: Guiding Multimodal LLMs for Efficient Visual Place Recognition at Test-Time
Prompting a multimodal LLM to output a JSON similarity score directly, plus a mean-minus-standard-deviation consistency penalty, raises Tokyo247 R@1 from 77.1% (DINOv2 GeM) to 91.1% without fine-tuning.
-
Let's Revise Step-by-Step: A Unified Local Search Framework for Code Generation with LLMs
The abstract claims a new local search framework for code generation, but the manuscript body is a different mathematics paper.
-
Reasoning in machine vision by learning fast and slow thinking
A dual-process vision system improves segmentation accuracy by spending more inference-time compute, using a fast predictor and a slow self-play refiner, reporting gains on cancer localisation with only 8-16 labels.
-
Fast on the Easy, Deep on the Hard: Efficient Reasoning via Powered Length Penalty
Reward shaping with a powered length penalty makes LLMs answer easy questions with far fewer tokens while preserving or slightly improving accuracy on hard math benchmarks.
-
VReST: Enhancing Reasoning in Large Vision-Language Models through Tree Search and Self-Reward Mechanism
VReST combines Monte Carlo tree search with a self-reward signal inside a vision-language model to get higher accuracy than CoT, ToT, or voting baselines on MathVista, MathVision, and CharXiv, while spending several t...
-
Can Past Experience Accelerate LLM Reasoning?
LLMs can cut reasoning compute by up to 56% on similar questions when memory of past correct solutions is combined with adaptive early stopping, though the benchmark is small and synthetic.
-
Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.
-
Data Diversification Methods In Alignment Enhance Math Performance In LLMs
DTS, which generates diverse solution strategies before writing solutions, improves GSM8K by 7.1 points and MATH by 4.2 points over an untuned base model at 1.03x baseline compute.
-
DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models
A survey of DeepSeek's V3 and R1 models covering MLA, MoE, MTP, GRPO, and training engineering, with no new experimental results.
Discussion (0). Sign in to comment.