Pith. sign in

REVIEW 5 cited by

Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.06508 v1 pith:YY5BJV2J submitted 2024-10-09 cs.LG cs.CL

Xiyao Wang , Linfeng Song , Ye Tian , Dian Yu , Baolin Peng , Haitao Mi , Furong Huang , Dong Yu This is my paper · ORCID
classification cs.LGcs.CL
keywords mctsalphallm-cplllmsreasoningdistillationbehaviorlearningtraining
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Monte Carlo Tree Search (MCTS) has recently emerged as a powerful technique for enhancing the reasoning capabilities of LLMs. Techniques such as SFT or DPO have enabled LLMs to distill high-quality behaviors from MCTS, improving their reasoning performance. However, existing distillation methods underutilize the rich trajectory information generated by MCTS, limiting the potential for improvements in LLM reasoning. In this paper, we propose AlphaLLM-CPL, a novel pairwise training framework that enables LLMs to self-improve through MCTS behavior distillation. AlphaLLM-CPL efficiently leverages MCTS trajectories via two key innovations: (1) AlphaLLM-CPL constructs stepwise trajectory pairs from child nodes sharing the same parent in the search tree, providing step-level information for more effective MCTS behavior distillation. (2) AlphaLLM-CPL introduces curriculum preference learning, dynamically adjusting the training sequence of trajectory pairs in each offline training epoch to prioritize critical learning steps and mitigate overfitting. Experimental results on mathematical reasoning tasks demonstrate that AlphaLLM-CPL significantly outperforms previous MCTS behavior distillation methods, substantially boosting the reasoning capabilities of LLMs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Stepwise Reasoning Checkpoint Analysis: A Test Time Scaling Method to Enhance LLMs' Reasoning

    cs.CL 2025-05 conditional novelty 7.0 of 10

    A checkpoint-based search and candidate augmentation method improves small LLM mathematical reasoning accuracy over existing test-time scaling baselines.

  2. Enhancing User Engagement in Socially-Driven Dialogue through Interactive LLM Alignments

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Dialogue agents aligned via DPO on preference pairs mined from simulated conversations improve engagement scores against the same simulator, with smaller and partially inconsistent human evaluation evidence.

  3. MARCO: Meta-Reflection with Cross-Referencing for Code Reasoning

    cs.CL 2025-05 conditional novelty 6.0 of 10

    MARCO combines cross-problem knowledge accumulation with cross-agent lesson sharing to improve LLM code reasoning at inference time.

  4. CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Adding actor perplexity and multi-head critic variance as intrinsic exploration bonuses improves RLVR math reasoning accuracy by roughly +2 to +3 points on AIME benchmarks.

  5. On the Emergence of Thinking in LLMs I: Searching for the Right Intuition

    cs.AI 2025-02 conditional novelty 5.0 of 10

    A post-training framework called RLSP, decoupling exploration from correctness rewards, induces backtracking and self-correction in LLMs and improves math reasoning performance.

Pith tools