Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search

Chaofan Ding; Kexin Luan; Shuaihao Dong; Shuangtao Li; Xinhan Di

REVIEW 5 major objections 5 minor 3 cited by

Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Tree-search step scores lift LLM accuracy on MATH and GSM8K

desk verdict Plausible dense-scored MCTS self-training method, but test-set-based early stopping and one flat transfer cell weaken the reported gains. read the letter →

arxiv 2501.01478 v1 pith:3NNJMMT7 submitted 2025-01-02 cs.AI cs.CLcs.LG

Shuangtao Li , Shuaihao Dong , Kexin Luan , Xinhan Di , Chaofan Ding This is my paper · ORCID

classification cs.AIcs.CLcs.LG

keywords largelanguagemodelsreasoningprocesssupervisionMonteCarloTreeSearchrelativecorrectnessself-trainingmathematicalweightednegativelog-likelihood

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that an LLM can improve its own mathematical reasoning using process supervision it generates itself with Monte Carlo Tree Search. The method samples many next steps at each point of a solution, simulates completions, and assigns each step a continuous 'relative correctness' score measuring how often it leads to a correct final answer relative to its alternatives. Training the model on those scored steps, with a KL penalty against the previous model, improves accuracy on MATH and GSM8K for two instruction-tuned models, beyond rejection-sampling fine-tuning and step-level DPO. Because models trained on one dataset also improve on the other, the authors argue the gain is reasoning ability rather than memorization. The paper itself notes the gains saturate after a few iterations and that more data or iterations degrade performance.

What carries the argument

The object doing the work is the 'relative correctness' score in Eq. (1): for each candidate next step, the model's MCTS rolls out many completions, propagates a binary success reward back through the tree, and then scores the step as $\alpha$ times its visit count times the difference between its average reward and the average reward of its sibling steps. This turns whole-path success or failure into dense per-step supervision without human labels or a separate reward model. Training then minimizes a weighted negative log-likelihood of the step token sequence, weighted by that score, plus a KL-divergence penalty toward the previous iteration's model, which the paper frames as offline reinforcement learning with a distribution-shift regularizer.

What would settle it

Train two matched models: one with the Eq. (1) relative-correctness weights and one with the same steps weighted only by whether the sampled path ended correct; if the score-weighted model does not beat the whole-path-weighted control, the sibling-relative term is not doing the claimed work. A second check is to take a held-out set of traced solutions and ask whether the steps ranked highest by Eq. (1) match human or process-reward-model judgments; a large mismatch would indicate the supervision signal is miscalibrated.

Watch

Extended reading notes

Core claim

The paper's central claim is that a continuous step-level score generated by MCTS—the visit-weighted average success of a candidate next step minus the sibling-average success—can serve as process supervision for training an LLM, and that training with this score improves mathematical reasoning more than outcome-level or binary preference supervision. On Llama-3.1-8B-Instruct, MATH accuracy rises from 47.07 with zero-shot CoT to 51.92 after four iterations, and GSM8K rises from 80.77 to 85.80. On DeepSeek-Math-7B-Instruct, MATH rises from 41.20 to 44.57 and GSM8K from 78.79 to 82.02. Models trained on one dataset also outperform the zero-shot baseline on the other, suggesting the learned ability transfers rather than memorized answers. The paper also reports that binary Step-level DPO yields only marginal MATH improvements, which it interprets as evidence that continuous relative-correctness labels carry more information than chosen/rejected pairs.

Load-bearing premise

The results rest on the assumption that the Eq. (1) score measures a single step's quality, even though the score is built from whole-path success or failure, so a step can look good or bad mainly because of what later steps happened to do.

Editorial extensions

If this is right

MATH accuracy for Llama-3.1-8B-Instruct improves from 47.07 to 51.92 and GSM8K from 80.77 to 85.80 over four self-training iterations.
DeepSeek-Math-7B-Instruct improves from 41.20 to 44.57 on MATH and from 78.79 to 82.02 on GSM8K.
Continuous relative-correctness labels outperform binary best/worst step preferences, so denser process feedback is worth generating.
Models trained on one dataset beat the zero-shot baseline on the other, indicating transferable reasoning gains.
Process supervision via MCTS needs no human step labels, no reward model, and no stronger teacher, so it applies to the strongest available LLM.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper never checks its step scores against human labels or an external process reward model; if the score mostly reflects luck from later steps, a matched control that weights steps by whole-path success would reveal whether the sibling-relative term is the real driver.
The quick saturation and later degradation suggest a self-generated-data distribution collapse, and a temperature or confidence filter on which problems enter the training set might extend the improvement regime.
Because MATH and GSM8K are both math word problems, the reported transfer is a weak test of general reasoning; testing on planning, code, or non-math logic benchmarks would more sharply separate genuine reasoning gains from math-specific pattern learning.
Since training is done with LoRA, the paper's own limitation is that full fine-tuning might show larger or more stable gains than reported.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the dense MCTS step-score objective is a real, incremental step beyond binary-preference self-training, and the main MATH/GSM8K results look promising. But the evaluation has a load-bearing flaw: the stopping rule is test-set-based, and the transfer claim is overstated in one cell.

The paper's contribution is a training recipe: use MCTS rollouts to assign each reasoning step a real-valued 'relative correctness' score, then train the LLM with a KL-regularized weighted NLL, iterating. That is genuinely different from MCTS-DPO's binary chosen/rejected pairs and from ReST-MCTS*'s separate PRM. The method is simple, the write-up is clear, and the limitations section is honest about saturation and degradation with more iterations.

The biggest problem is evaluation protocol. 'We report the results of iterative training until the accuracies do not increase anymore'—with no validation set described. If the test set is used to pick the iteration and then to report that iteration's score, the gains (Llama MATH 47.07→51.92) are partly selection effects. The stress-test note is right that this is more damaging than the step-score criticism. Second, the transfer claim is not consistent: deepseek-math GSM8K→MATH is 41.15±0.31 versus 41.20±0.27 for Zero-shot-CoT—effectively flat, so 'consistently outperform' in the abstract is wrong. Third, the closest baselines are missing: no actual MCTS-DPO, ReST-MCTS*, or Math-Shepherd comparison. The Step-level DPO baseline is described as 'similar to' Xie et al., not a faithful implementation. Fourth, the Eq. (1) scores are never checked against human labels or an external PRM; noisy credit assignment is plausible. That concern is real but secondary if the eval were unbiased.

Missing alpha, MCTS simulation counts, temperatures, and no code or data. For the target audience—people working on process supervision and self-training—the dense-score idea is worth trying, but the reported numbers should be treated with caution. It deserves peer review because the method is novel enough and the core idea is sensible; the reviewers should demand a validation split, the missing baselines, and a corrected transfer sentence. I'd send it to review with major-revision expectations. For our reading group: maybe, as a case study in evaluation pitfalls.

Referee Report

5 major / 5 minor

Summary. The paper proposes an iterative self-training method that uses MCTS to assign per-step 'relative correctness' scores to reasoning steps generated by the target LLM, then fine-tunes the model with a weighted negative log-likelihood plus a KL penalty (Eq. (2)). Experiments on Llama-3.1-8B-Instruct and deepseek-math-7b-instruct report consistent gains on MATH and GSM8K over Zero-shot-CoT, RFT, and a Step-level DPO baseline, plus transfer between the two datasets. The core claim is that automatically generated process supervision from MCTS improves mathematical reasoning and transfers across benchmarks.

Significance. If the reported gains survive a properly controlled evaluation, the method is a useful addition to the process-supervision toolbox: it removes the need for human step labels or a trained PRM, uses graded step scores rather than binary preferences, and is a relatively simple generate-then-fine-tune loop. The paper also honestly reports convergence and degradation with more data, a useful negative result. However, the current manuscript does not yet establish the central empirical claim because the iteration-selection protocol is test-set-based and key hyperparameters are omitted, and the transfer claim is not supported in one of the four directions.

major comments (5)

[Main Results/Setup, Tables 1–2] The evaluation protocol is the most load-bearing threat to the headline numbers. The paper states, 'We report the results of iterative training until the accuracies do not increase anymore,' and no held-out validation split is described in Setup. Because the same test sets are used both to decide when to stop and to produce the reported accuracy, the reported gains (e.g., Llama-3.1-8B MATH 47.07 to 51.92, GSM8K 80.77 to 85.80) may be the maximum over the iteration curve rather than the accuracy of a predetermined training schedule. Please stop on a validation split, report all iterations with standard errors, and pre-specify the comparison.
[Eq. (1) and Limitations] The step score in Eq. (1) is an aggregated credit-assignment heuristic derived from binary whole-path success; the paper does not validate it against human step labels (e.g., PRM800K) or an external process reward model. A step that is itself wrong can receive a high score if later sampled continuations happen to lead to a correct answer, so the supervision signal may be noisy. The limitations section's observation that training for more iterations or using more problems 'actually degrades' performance is consistent with unstable supervision. Please add an external validation of the step scores, an ablation that replaces them with oracle step labels, and an analysis of the degradation.
[Related Work and Table 1] The closest MCTS-based self-training baselines are cited but not compared. MCTS-DPO and ReST-MCTS* are discussed in Related Work, but Table 1 only includes Zero-shot-CoT, RFT, and a Step-level DPO baseline that forms best/worst-step pairs. To support the claim that graded relative-correctness scores are superior to pairwise preference and to other MCTS self-training methods, the paper needs direct comparisons under matched compute and data conditions.
[Table 2 and Abstract] The transfer claim is not supported in all four directions. For deepseek-math-7b-instruct, GSM8K-to-MATH after Iteration 1 is 41.15 ± 0.31, which is no better than Zero-shot-CoT's 41.20 ± 0.27, and no significance test is reported for any transfer cell. The abstract's 'consistently outperform' therefore overstates the results. Please report significance tests and either weaken the claim or analyze why this direction fails.
[Setup] Several hyperparameters that control the data-generation and training loop are not reported: α in Eq. (1), the number of MCTS simulations, the sampling temperature, tree width, rollout length, LoRA rank, and learning rate. Without these values and a sensitivity analysis, the results are not reproducible and it is unclear whether the gains depend on a narrow set of settings.

minor comments (5)

[Experiments, Setup] The 'Rejective Sampling Fine-Tuning' baseline should be named 'Rejection Sampling Fine-Tuning'.
[Proposed Methods] The sentence 'we are not accessible to LLMs stronger than the target LLM' should read 'we do not have access to LLMs stronger than the target LLM'.
[Eq. (2)] In the loss in Eq. (2), the score r from Eq. (1) can be negative when a step's average reward is below the sibling average; please state explicitly whether negative weights are kept and how this interacts with the KL penalty.
[Tables 1–2] Table 1 and Table 2 would benefit from consistent column spacing and from a caption stating the number of seeds used for the mean±standard error estimates.
[Abstract] The sentence 'This generate-then-train process is repeated iteratively until convergence' in the abstract is not precise because the paper stops when test accuracy ceases to increase; please rephrase to match the actual stopping rule.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the step-score formula is a proposed MCTS heuristic, and the reported gains are empirical benchmark results rather than outputs derived from the method's own inputs by construction.

full rationale

The paper's claimed chain is an empirical training method: MCTS rollouts from the current LLM produce step scores via Eq. (1), and Eq. (2) trains the same LLM with weighted log-likelihood plus a KL penalty. Neither equation defines a target quantity in terms of the quantity it is used to predict; the score is a hand-designed Monte Carlo estimate of relative step quality, not a fitted parameter relabeled as a prediction. The central claim (improved MATH/GSM8K accuracy and transfer to the other dataset) is evaluated on held-out test sets and compared with Zero-shot-CoT, RFT, and Step-level DPO, so the derivation is not equivalent to its inputs by construction. The self-training nature of the loop (the model labels its own rollouts) is a bootstrapping strategy rather than a definitional circularity, and no load-bearing self-citations are present. The limitations section's observation that more iterations or more problems degrade performance is a substantive robustness concern, and the stopping criterion 'until the accuracies do not increase anymore' is described without an explicit validation split; these are correctness and evaluation risks, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central method rests on a hand-designed step-score statistic, a set of unreported hyperparameters, and the assumption that MCTS rollouts yield reliable per-step credit. No new physical or conceptual entities are introduced.

free parameters (3)

alpha (α) in score formula = not reported
Controls the scale of step scores in Eq. (1); the paper says it is a 'manually set constant' but gives no value, and it directly affects the loss weights.
MCTS hyperparameters (simulation count, temperature, tree width) = not reported
The quality of the step scores depends on these search parameters, which are not specified.
Number of training iterations selected = varies: Llama MATH 4, deepseek MATH 3, GSM8K 2
The stopping point is chosen on the test set as the accuracy peak, which is a hand-chosen selection that affects reported results.

assumptions (4)

domain assumption Ground-truth final answers are available and reliably detectable.
MCTS backpropagation uses a binary correct/incorrect reward based on the final answer, assuming the answer checker is exact and the model's answer format is parsable.
domain assumption The binary success of a simulated continuation is a valid signal for the quality of every preceding step.
Eq. (1) treats the visit-weighted average reward difference as 'relative correctness' of a step; this credit-assignment assumption is never validated.
ad hoc to paper Steps separated by newlines are atomic, commensurable units of reasoning.
The method treats each newline-separated segment as an independent step for scoring and training; this segmentation is an arbitrary modeling choice.
domain assumption The KL penalty against the previous iteration prevents distribution shift and training collapse.
The paper relies on the KL term for stability, but the limitations section shows performance degrades with more iterations and more data.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search." pith.science (2026). https://pith.science/paper/3NNJMMT7

@misc{pith2026250101478,
  author       = {Pith},
  title        = {Pith review of: Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3NNJMMT7}},
  note         = {Machine review of arXiv:2501.01478}
}

read the original abstract

Large language models (LLMs) have demonstrated their remarkable capacity across a variety of tasks. However, reasoning remains a challenge for LLMs. To improve LLMs' reasoning ability, process supervision has proven to be better than outcome supervision. In this work, we study using Monte Carlo Tree Search (MCTS) to generate process supervision data with LLMs themselves for training them. We sample reasoning steps with an LLM and assign each step a score that captures its "relative correctness," and the LLM is then trained by minimizing weighted log-likelihood of generating the reasoning steps. This generate-then-train process is repeated iteratively until convergence.Our experimental results demonstrate that the proposed methods considerably improve the performance of LLMs on two mathematical reasoning datasets. Furthermore, models trained on one dataset also exhibit improved performance on the other, showing the transferability of the enhanced reasoning ability.

Figures

Figures reproduced from arXiv: 2501.01478 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

On Almost Surely Safe Alignment of Large Language Models at Inference-Time
cs.LG 2025-02 conditional novelty 6.0 of 10

An inference-time beam-search method with a safety-state tracker and latent critic enforces a user-supplied safety cost model, with an almost-sure guarantee only relative to that model.
Efficient Reasoning on the Edge
cs.LG 2026-03 accept novelty 5.5 of 10

LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.
TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance
cs.IR 2025-10 conditional novelty 4.0 of 10

TaoSR-AGRL improves e-commerce search relevance by combining dense rule-aware reward shaping with adaptive ground-truth-guided replay in GRPO, reporting gains on Taobao's private offline and online evaluations.

Reference graph

Works this paper leans on

38 extracted references · 7 canonical work pages · cited by 3 Pith papers

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

B.; Powley, E.; Whitehouse, D.; Lucas, S

Browne, C. B.; Powley, E.; Whitehouse, D.; Lucas, S. M.; Cowling, P. I.; Rohlfshagen, P.; Tavener, S.; Perez, D.; Samothrakis, S.; and Colton, S. 2012. A survey of Monte Carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1): 1--43

work page 2012
[4]

Chen, G.; Liao, M.; Li, C.; and Fan, K. 2024. AlphaMath Almost Zero: process Supervision without process. arXiv preprint arXiv:2405.03553

arXiv 2024
[5]

Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

arXiv 2021
[6]

Fu, Y.; Peng, H.; Sabharwal, A.; Clark, P.; and Khot, T. 2022. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations

work page 2022
[7]

J.; Wang, Z.; Wang, D

Hao, S.; Gu, Y.; Ma, H.; Hong, J. J.; Wang, Z.; Wang, D. Z.; and Hu, Z. 2023. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992

arXiv 2023
[8]

Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

arXiv 2021

Show all 38 references

[9]

J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

2021 arXiv
[10]

Huang, J.; and Chang, K. C.-C. 2023. Towards Reasoning in Large Language Models: A Survey. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Findings of the Association for Computational Linguistics: ACL 2023, 1049--1065. Toronto, Canada: Association for Computational Linguistics

2023
[11]

Hwang, H.; Kim, D.; Kim, S.; Ye, S.; and Seo, M. 2024. Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards. In Findings of the Association for Computational Linguistics: EMNLP 2024, 1444--1466

2024
[12]

Kocsis, L.; and Szepesv \'a ri, C. 2006. Bandit based monte-carlo planning. In European conference on machine learning, 282--293. Springer

2006
[13]

S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y

Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199--22213

2022
[14]

Lai, X.; Tian, Z.; Chen, Y.; Yang, S.; Peng, X.; and Jia, J. 2024. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629

2024 arXiv
[15]

Li, C.; Wang, W.; Hu, J.; Wei, Y.; Zheng, N.; Hu, H.; Zhang, Z.; and Peng, H. 2024. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.04706

2024 arXiv
[16]

Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2023. Let's verify step by step. arXiv preprint arXiv:2305.20050

2023 arXiv
[17]

Liu, H.; Zhang, Y.; Luo, Y.; and Yao, A. C.-C. 2024. Augmenting math word problems via iterative question composing. arXiv preprint arXiv:2401.09003

2024 arXiv
[18]

Lu, Z.; Zhou, A.; Ren, H.; Wang, K.; Shi, W.; Pan, J.; Zhan, M.; and Li, H. 2024. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms. arXiv preprint arXiv:2402.16352

2024 arXiv
[19]

Luo, L.; Liu, Y.; Liu, R.; Phatale, S.; Lara, H.; Li, Y.; Shu, L.; Zhu, Y.; Meng, L.; Sun, J.; et al. 2024. Improve Mathematical Reasoning in Language Models by Automated Process Supervision. arXiv preprint arXiv:2406.06592

2024 arXiv
[20]

J.; Gur-Ari, G.; Michalewski, H.; Austin, J.; Bieber, D.; Dohan, D.; Lewkowycz, A.; Bosma, M.; Luan, D.; et al

Nye, M.; Andreassen, A. J.; Gur-Ari, G.; Michalewski, H.; Austin, J.; Bieber, D.; Dohan, D.; Lewkowycz, A.; Bosma, M.; Luan, D.; et al. 2021. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114

2021 arXiv
[21]

Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730--27744

2022
[22]

Y.; Yuan, W.; Cho, K.; He, H.; Sukhbaatar, S.; and Weston, J

Pang, R. Y.; Yuan, W.; Cho, K.; He, H.; Sukhbaatar, S.; and Weston, J. 2024. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733

2024 arXiv
[23]

L.; Yang, F.; and Yang, M

Qi, Z.; Ma, M.; Xu, J.; Zhang, L. L.; Yang, F.; and Yang, M. 2024. Mutual reasoning makes smaller llms stronger problem-solvers. arXiv preprint arXiv:2408.06195

2024 arXiv
[24]

D.; Ermon, S.; and Finn, C

Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Ermon, S.; and Finn, C. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

2024
[25]

Trung, L.; Zhang, X.; Jie, Z.; Sun, P.; Jin, X.; and Li, H. 2024. Reft: Reasoning with reinforced fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 7601--7614

2024
[26]

Valmeekam, K.; Olmo, A.; Sreedharan, S.; and Kambhampati, S. 2022. Large language models still can't plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop

2022
[27]

Wang, P.; Li, L.; Shao, Z.; Xu, R.; Dai, D.; Li, Y.; Chen, D.; Wu, Y.; and Sui, Z. 2024 a . Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo...

2024
[28]

Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; and Zhou, D. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

2022 arXiv
[29]

Wang, Z.; Li, Y.; Wu, Y.; Luo, L.; Hou, L.; Yu, H.; and Shang, J. 2024 b . Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision. arXiv preprint arXiv:2402.02658

2024 arXiv
[30]

V.; Zhou, D.; et al

Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 24824--24837

2022
[31]

P.; Kawaguchi, K.; and Shieh, M

Xie, Y.; Goyal, A.; Zheng, W.; Kan, M.-Y.; Lillicrap, T. P.; Kawaguchi, K.; and Shieh, M. 2024. Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning. arXiv preprint arXiv:2405.00451

2024 arXiv
[32]

Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; and Narasimhan, K. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

2024
[33]

T.; Li, Z.; Weller, A.; and Liu, W

Yu, L.; Jiang, W.; Shi, H.; Yu, J.; Liu, Z.; Zhang, Y.; Kwok, J. T.; Li, Z.; Weller, A.; and Liu, W. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284

2023 arXiv
[34]

Yuan, Z.; Yuan, H.; Li, C.; Dong, G.; Lu, K.; Tan, C.; Zhou, C.; and Zhou, J. 2023. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825

2023 arXiv
[35]

Zelikman, E.; Wu, Y.; Mu, J.; and Goodman, N. 2022. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35: 15476--15488

2022
[36]

Zhang, D.; Li, J.; Huang, X.; Zhou, D.; Li, Y.; and Ouyang, W. 2024 a . Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B. arXiv preprint arXiv:2406.07394

2024 arXiv
[37]

Zhang, D.; Zhoubian, S.; Hu, Z.; Yue, Y.; Dong, Y.; and Tang, J. 2024 b . Rest-mcts*: Llm self-training via process reward guided tree search. arXiv preprint arXiv:2406.03816

2024 arXiv
[38]

X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al

Zhao, W. X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223

2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

B.; Powley, E.; Whitehouse, D.; Lucas, S

Browne, C. B.; Powley, E.; Whitehouse, D.; Lucas, S. M.; Cowling, P. I.; Rohlfshagen, P.; Tavener, S.; Perez, D.; Samothrakis, S.; and Colton, S. 2012. A survey of Monte Carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1): 1--43

work page 2012

[4] [4]

Chen, G.; Liao, M.; Li, C.; and Fan, K. 2024. AlphaMath Almost Zero: process Supervision without process. arXiv preprint arXiv:2405.03553

arXiv 2024

[5] [5]

Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

arXiv 2021

[6] [6]

Fu, Y.; Peng, H.; Sabharwal, A.; Clark, P.; and Khot, T. 2022. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations

work page 2022

[7] [7]

J.; Wang, Z.; Wang, D

Hao, S.; Gu, Y.; Ma, H.; Hong, J. J.; Wang, Z.; Wang, D. Z.; and Hu, Z. 2023. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992

arXiv 2023

[8] [8]

Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

arXiv 2021