REVIEW 15 cited by
AlphaMath Almost Zero: Process Supervision without Process
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Although recent advancements in large language models (LLMs) have significantly improved their performance on various tasks, they still face challenges with complex and symbolic multi-step reasoning, particularly in mathematical reasoning. To bolster the mathematical reasoning capabilities of LLMs, most existing efforts concentrate on seeking assistance from either domain experts or GPT-4 for high-quality process-supervised data, which is not only expensive but also labor-intensive. In our study, we propose an innovative framework, AlphaMath, that bypasses the need for process annotations (from humans or GPTs) by leveraging Monte Carlo Tree Search (MCTS). This framework focuses on unleashing the potential of a well-pretrained LLM to autonomously enhance its mathematical reasoning. Specifically, we integrate a value model with the LLM, automatically generating both process supervision and step-level evaluation signals in MCTS. Furthermore, we propose an efficient inference strategy, step-level beam search, where the value model is crafted to assist the policy model (i.e., LLM) in navigating more effective reasoning paths, rather than solely relying on prior probabilities. The experimental results on both in-domain and out-of-domain datasets demonstrate that even without GPT-4 or human-annotated process supervision, our AlphaMath framework achieves comparable or superior results to previous state-of-the-art methods.
Forward citations
Cited by 15 Pith papers
-
Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
Distilled pretraining improves test-time scaling via generation diversity but impairs induction-head-based in-context learning, with the trade-off explained by a bigram model analysis.
-
Output-Space Search: Targeting LLM Generations in a Frozen Encoder-Defined Output Space
Retrieval-grounded reinforcement learning makes an autoregressive LLM hit requested coordinates in a frozen encoder-defined PCA space, giving an outer loop a low-dimensional target to sweep or optimize.
-
Sticker-TTS: Learn to Utilize Historical Experience with a Sticker-driven Test-Time Scaling Framework
Sticker-TTS uses three collaborating models to distill and reuse compact summaries (stickers) of past reasoning attempts, improving math benchmark accuracy at a claimed equivalent inference cost.
-
Teaching Language Models To Gather Information Proactively
Rewarding questions for eliciting genuinely new information trains a small model to outperform larger models at proactive clarification and downstream writing quality.
-
TreeRL: LLM Reinforcement Learning with On-Policy Tree Search
TreeRL combines entropy-guided tree search with on-policy RL and tree-derived process rewards, reporting modest accuracy gains over chain sampling on math and code benchmarks.
-
UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents
UI-Genie uses a specialized reward model and iterative self-improvement to generate synthetic training trajectories, achieving state-of-the-art results for mobile GUI agents on AndroidControl, AndroidLab, and Android Arena.
-
PIPA: Preference Alignment as Prior-Informed Statistical Estimation
A unified maximum-likelihood framework with prior constraints that recovers DPO and KTO as special cases and yields new PIPA-M/PIPA-N losses with 3-10% gains on GSM8K and MATH.
-
Iterative Deepening Sampling as Efficient Test-Time Scaling
ID-Sampling inserts self-correction trigger sentences at geometrically increasing token budgets during LLM generation, improving pass rates on MATH-500 and AIME over vanilla Best-of-N and majority voting.
-
CheMatAgent: Enhancing LLMs for Chemistry and Materials Science through Tree-Search Based Tool Learning
CheMatAgent uses hierarchical Monte Carlo tree search with separate policy and execution models, plus trained reward models, to improve tool selection and parameter filling on a new chemistry benchmark, ChemToolBench.
-
Embracing Imperfection: Simulating Students with Diverse Cognitive Levels Using LLM-based Agents
A knowledge-graph cognitive prototype plus beam-search self-refinement lets LLM agents simulate students' imperfect programming solutions more accurately than plain prompting.
-
Reward Model Generalization for Compute-Aware Test-Time Reasoning
The paper claims PRM generalization error controls test-time compute efficiency and that the proposed CATS controller improves accuracy on MATH-500 and AIME24.
-
Beyond the First Error: Process Reward Models for Reflective Mathematical Reasoning
A process reward model trained with Error Propagation and Error Cessation labels from an o1 judge outperforms existing PRMs on math selection and step-level scoring.
-
On the Emergence of Thinking in LLMs I: Searching for the Right Intuition
A post-training framework called RLSP, decoupling exploration from correctness rewards, induces backtracking and self-correction in LLMs and improves math reasoning performance.
-
Holistically Guided Monte Carlo Tree Search for Intricate Information Seeking
HG-MCTS guides Monte Carlo tree search with an adaptive checklist and self-generated rewards to improve multi-hop information seeking, claiming better benchmark accuracy.
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
Discussion (0). Continue with ORCID to comment.