Pith. sign in

REVIEW 16 cited by

ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.03816 v3 pith:QHDATL6P submitted 2024-06-06 cs.CL

Dan Zhang , Sining Zhoubian , Ziniu Hu , Yisong Yue , Yuxiao Dong , Jie Tang This is my paper
classification cs.CL
keywords rest-mctsprocessself-trainingcorrectpolicyrewardreasoningrewards
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Recent methodologies in LLM self-training mostly rely on LLM generating responses and filtering those with correct output answers as training data. This approach often yields a low-quality fine-tuning training set (e.g., incorrect plans or intermediate reasoning). In this paper, we develop a reinforced self-training approach, called ReST-MCTS*, based on integrating process reward guidance with tree search MCTS* for collecting higher-quality reasoning traces as well as per-step value to train policy and reward models. ReST-MCTS* circumvents the per-step manual annotation typically used to train process rewards by tree-search-based reinforcement learning: Given oracle final correct answers, ReST-MCTS* is able to infer the correct process rewards by estimating the probability this step can help lead to the correct answer. These inferred rewards serve dual purposes: they act as value targets for further refining the process reward model and also facilitate the selection of high-quality traces for policy model self-training. We first show that the tree-search policy in ReST-MCTS* achieves higher accuracy compared with prior LLM reasoning baselines such as Best-of-N and Tree-of-Thought, within the same search budget. We then show that by using traces searched by this tree-search policy as training data, we can continuously enhance the three language models for multiple iterations, and outperform other self-training algorithms such as ReST$^\text{EM}$ and Self-Rewarding LM. We release all code at https://github.com/THUDM/ReST-MCTS.

Discussion (0). Sign in to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LeAct: Learning to Reason from Expert Actions

    cs.LG 2026-07 conditional novelty 6.0 of 10

    An AI can learn to reason by sampling explanations for an expert's actions and keeping only the ones that help it predict those actions.

  2. UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A unified KG-plus-live-web data pipeline covering all three multimodal BrowseComp information-flow patterns, plus an exploration-degree filter, yields a 35B agent at 54.4 avg accuracy.

  3. Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A survey of 1,250 papers organizes AI self-improvement along two axes—what is improved and loop closure—finding that demonstrated self-improvement strength tracks a verification hierarchy from formal verifiers down to...

  4. Intern-S1-MO: Long-horizon Reasoning Agent for Olympiad?Level Mathematical Problem Solving

    cs.CL 2025-12 conditional novelty 6.0 of 10

    A multi-round, lemma-memory reasoning agent with hierarchical RL reaches reported gold-medal-level scores on Olympiad math benchmarks, though the proof-based scores are self-graded.

  5. Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL

    cs.AI 2025-09 conditional novelty 6.0 of 10

    DRER rewards CoT trajectories that increase the model's likelihood of the correct answer, plus a length penalty, and the new LogicTree benchmark reportedly lifts a 7B model's average accuracy from 0.13 to 0.60.

  6. Enhancing User Engagement in Socially-Driven Dialogue through Interactive LLM Alignments

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Dialogue agents aligned via DPO on preference pairs mined from simulated conversations improve engagement scores against the same simulator, with smaller and partially inconsistent human evaluation evidence.

  7. TreeRL: LLM Reinforcement Learning with On-Policy Tree Search

    cs.LG 2025-06 conditional novelty 6.0 of 10

    TreeRL combines entropy-guided tree search with on-policy RL and tree-derived process rewards, reporting modest accuracy gains over chain sampling on math and code benchmarks.

  8. Structured Pruning for Diverse Best-of-N Reasoning Optimization

    cs.CL 2025-06 reject novelty 6.0 of 10

    SPRINT learns to select which attention heads to prune per question, improving Pass@N over random head selection and multinomial sampling on MATH500 and GSM8K.

  9. Fostering Video Reasoning via Next-Event Prediction

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Next-event prediction, training video language models to caption unseen future frames, improves their scores on several temporal benchmarks while roughly preserving general video understanding.

  10. O$^2$-Searcher: A Searching-based Agent Model for Open-Domain Open-Ended Question Answering

    cs.CL 2025-05 conditional novelty 6.0 of 10

    O2-Searcher uses GRPO reinforcement learning to teach a 3B LLM to search a local corpus and answer open-ended and closed-ended questions, and introduces the O2-QA benchmark.

  11. CIGPO: Contextual Information-Gain Policy Optimization for Multi-Turn Evidence-Reading LLM Agents

    cs.LG 2026-06 conditional novelty 5.0 of 10

    Turn-level information-gain rewards prevent zero-advantage collapse in GRPO, improving HotpotQA F1 from 0.252 to 0.518 at the 3B scale.

  12. EduFlow: Advancing MLLMs' Problem-Solving Proficiency through Multi-Stage, Multi-Perspective Critique

    cs.AI 2025-07 conditional novelty 5.0 of 10

    A multi-stage framework (data filtering, MCTS-guided trajectory construction, PRM-based reranking) improves Qwen MLLMs' accuracy on K-12 multimodal science benchmarks.

  13. VReST: Enhancing Reasoning in Large Vision-Language Models through Tree Search and Self-Reward Mechanism

    cs.CV 2025-06 conditional novelty 5.0 of 10

    VReST combines Monte Carlo tree search with a self-reward signal inside a vision-language model to get higher accuracy than CoT, ToT, or voting baselines on MathVista, MathVision, and CharXiv, while spending several t...

  14. CheMatAgent: Enhancing LLMs for Chemistry and Materials Science through Tree-Search Based Tool Learning

    cs.LG 2025-06 conditional novelty 5.0 of 10

    CheMatAgent uses hierarchical Monte Carlo tree search with separate policy and execution models, plus trained reward models, to improve tool selection and parameter filling on a new chemistry benchmark, ChemToolBench.

  15. Beyond the First Error: Process Reward Models for Reflective Mathematical Reasoning

    cs.AI 2025-05 conditional novelty 5.0 of 10

    A process reward model trained with Error Propagation and Error Cessation labels from an o1 judge outperforms existing PRMs on math selection and step-level scoring.

  16. Inference Scaled GraphRAG: Improving Multi Hop Question Answering on Knowledge Graphs

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Allowing an LLM more inference steps and majority-voted graph actions improves multi-hop knowledge-graph QA scores on GRBench.

Pith tools