Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Beyond the First Error: Process Reward Models for Reflective Mathematical Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that process reward models can be trained on reflection-aware labels, using Error Propagation and Error Cessation rules with an LLM judge, to outperform existing PRMs on long chain-of-thought math reasoning.

desk verdict A useful, well-engineered PRM annotation method for long CoT with solid BoN gains; the step-level F1 is partly circular because o1 both annotates training and labels the test set, but that doesn't sink the central claim. read the letter →

arxiv 2505.14391 v1 pith:FIHGPDF7 submitted 2025-05-20 cs.AI

Zhaohui Yang , Chenghua He , Xiaowen Shi , Linjing Li , Qiyue Yin , Shihong Deng , Daxin Jiang This is my paper
classification cs.AI
keywords processrewardmodelslongchain-of-thoughtself-correctionerrorpropagationcessationLLM-as-judgemathematicalreasoningbest-of-Nevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Process reward models (PRMs) score each step of a solution, and how those step labels are produced determines whether the score is trustworthy. The paper argues that the standard annotation scheme—mark all steps after the first error as wrong—is wrong for long chain-of-thought reasoning, where models often make a mistake, notice it, and recover. It therefore introduces two labeling rules: Error Propagation, marking a step wrong when it builds on a prior error without fixing it, and Error Cessation, marking a step correct when it corrects a prior error or starts a fresh correct approach. Using o1 as the judge, the authors annotate 1.7 million steps and train a 7B PRM that reports the best PRM@64 on MATH500 (81.6% vs 77.8% for the second-best baseline) and the best balanced step-level F1 (0.828). A sympathetic reader would take away that process supervision for reflective reasoning can be both more accurate and far cheaper than Monte-Carlo rollout labeling.

What carries the argument

The mechanism is a two-rule annotation protocol applied by an LLM judge. Error Propagation says a step that continues from an earlier erroneous step without correcting it is itself incorrect; Error Cessation says a step that corrects an earlier error or introduces a new error-free approach is correct. These rules enter the judge prompt, so each step's label depends on the context before it rather than on absolute step content. A second supporting mechanism is a fine-tuned generator that segments long CoT responses into semantically coherent steps, because long CoT text does not come with reliable step separators; the segmentation is what makes step-level scoring possible. The PRM is then a standard binary classifier over "prefix up to step i", trained with cross-entropy against the judge-produced labels.

What would settle it

Take a set of long CoT solutions whose step correctness is fixed by independent human consensus (or by inserting known arithmetic errors into otherwise correct traces) and have the same o1 judge label them. If o1's labels disagree with the independently established labels on a nontrivial fraction of steps—particularly steps that follow a detected error—then both the training signal and the step-level test set carry that bias, and the reported 0.828 F1 is not a measure of mathematical truth. The paper's Appendix E reports 0.963 agreement between o1 and human annotators, but agreement is not ground truth; the decisive comparison is a step-level F1 evaluation whose reference labels come from human consensus rather than from the same judge used in training.

Watch

Extended reading notes

Core claim

The paper's central claim is that the "first-error cliff" is the reason existing PRMs underperform on reflective solutions. In long CoT, correct and incorrect steps alternate: a model may calculate wrongly, rank on that wrong value, then catch itself and recompute. Labeling every post-error step as wrong teaches a PRM that recovery is indistinguishable from compounding error. The paper's two rules give the judge a way to separate those cases, and the trained PRM inherits that separation. Concretely, the paper reports that on MATH500 its 7B PRM achieves PRM@64 of 0.816 against 0.778 for the second-best open model, that it matches or exceeds the best baseline on AIME24 (0.267 at PRM@64 and 0.167 at PRM@8-step), and that at the step level it reaches F1 0.828 with precision 0.850 and recall 0.806, the most balanced among the compared models. The same model also outperforms a 1M-step Monte-Carlo-trained PRM of equal data size, and the advantage grows when the data are scaled to 1.7M steps.

Load-bearing premise

The load-bearing premise is that the o1 judge's step labels are accurate ground truth rather than a model-specific opinion, because the same kind of labels are used both to train the PRM and to score it; if the judge is biased about reflection steps, the PRM inherits that bias and the reported step F1 mostly measures agreement with o1.

Editorial extensions

If this is right

  • If the central claim is correct, process reward models no longer have to treat the first error as a cliff: a PRM trained with Error Cessation can keep guiding search through a solution that stumbles and then recovers.
  • The paper's comparison with Monte-Carlo labeling implies that step supervision for long CoT can be obtained without per-step rollouts, removing the main computational bottleneck that has limited PRM data collection.
  • The reported 14B results and the AIME24 results under rule-based segmentation imply the method transfers beyond the exact generator and segmentation used in training.
  • The same annotation protocol, by the paper's own statement, can be carried over to other multi-step domains such as code repair and the 24-point game.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: because the step-level test labels come from the same o1-based protocol, the reported step F1 may partly measure agreement with o1's judgment rather than mathematical truth; an independent human-labeled step set would separate the two.
  • My inference: the method's advantage should track the density of reflection in the data, so a direct test is to measure PRM@64 gain against reflection-token frequency while holding dataset difficulty fixed—something the appendix correlation suggests but does not isolate.
  • My inference: applied to code repair, Error Cessation would label a corrected patch as correct even after a faulty diff; a PRM trained this way should improve patch search over one trained with first-error truncation, which is a testable extension the paper does not run.
  • My inference: the dependence on a strong judge is a deliberate trade: as open-source reflection models improve, the same pipeline should produce better PRMs at constant data-collection cost, an upside the authors mention only as a limitation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a data annotation method for training process reward models (PRMs) on long, reflective chain-of-thought (CoT) reasoning. The authors introduce two annotation rules, Error Propagation and Error Cessation, and use an o1-family LLM judge to label individual steps of segmented long CoT solutions. Using this method, they collect 1.7M labeled samples and train a 7B PRM. They evaluate at solution level via Best-of-N (PRM@N) and a proposed step-level online search metric (PRM@N-step), and at step level via precision/recall/F1 on a test set labeled by o1 cross-validation and manual annotation. The reported results show the proposed PRM outperforming open-source PRMs and MC-annotated PRMs on MATH500, AIME24, GSM8K, and OlympiadBench, with additional ablations on data efficiency, robustness to the generator model, and generalization to other segmentation methods and base models.

Significance. If the empirical claims hold, the paper offers a practical and data-efficient alternative to Monte-Carlo-based PRM annotation for long reflective reasoning, an important setting for test-time search and self-correction. The solution-level Best-of-N evaluation is grounded in final-answer correctness and is therefore not subject to the annotation-circularity concern that affects the step-level metric. The controlled comparison against MC annotation at matched training sizes, the first-error truncation ablation, and the generalization experiments across generator models and segmentation methods are valuable contributions. However, the step-level F1 result is weakened by the partial use of o1-derived labels in the test set, and the PRM@N-step metric is underspecified. Because the paper's central claim of "consistently outperforming" relies in part on these two pieces, the step-level and search-guidance conclusions need additional support before the paper can be accepted.

major comments (4)
  1. [Section 5.1.1 and Appendix E] The step-level evaluation is partly circular: the PRM training labels are produced by o1 (Section 4.3), and the step-level test labels are described as "determined through a combination of cross-validation with the o1 model and manual annotation" (Section 5.1.1). The reported F1=0.828 in Table 2 may therefore partly measure how well the PRM reproduces o1's judgments rather than true step correctness. Appendix E reports o1 annotation accuracy of 0.963 but does not state the number of annotated steps, the selection procedure, or inter-annotator agreement. I ask for a fully human-labeled held-out step set, or a clear report of the fraction of test labels that come from o1, together with a sensitivity analysis excluding those labels.
  2. [Section 5.1.3] The PRM@N-step metric is described in a single sentence: "During each step, N candidates are sampled and the highest-scoring step is chosen to continue the generation." This is not enough to reproduce the experiments or to interpret the PRM@8-step results in Table 2. Please specify the search algorithm in detail: how many rollouts are performed, how depth is controlled, how the final answer is selected from the continued trajectory, whether the same compute budget is used across all compared PRMs, and whether the metric averages over multiple runs. Without this, the claim that the proposed PRM provides better intermediate search guidance is not verifiable.
  3. [Section 4.2 and Appendix E] The Error Cessation rule depends on the judge's assessment that a step "introduces a new, error-free approach" or "corrects the previous mistakes." This is a subjective judgment, and the human annotation process in Appendix E trained annotators on the o1-style rubric, which can anchor their decisions to the o1 model's conventions. The reported 0.963 o1-human agreement thus does not establish that o1's labels are ground truth; it only establishes agreement under a shared rubric. I request a random sample of steps where o1's labels are independently reviewed by multiple annotators without being shown the o1 rationale, with Cohen's kappa or similar reported.
  4. [Table 2 and Section 5.2] All solution-level and step-level results are reported as point estimates without confidence intervals, significance tests, or repeated-seed variation. Given that the headline PRM@64 gain on MATH500 is 3.8 points over the second-best baseline and the F1 gain is 0.06, these differences may be within run-to-run noise. Please provide variance estimates over random seeds or bootstrap resampling over problems, and clarify whether the reported numbers are medians over multiple runs.
minor comments (5)
  1. [Section 8] The section heading contains a typo: "Limitaions" should be "Limitations."
  2. [Appendix G, Table 10] The "MATH" column in Table 10 should specify whether this is MATH500 or another subset; the value 0.964 for the proposed 14B PRM is far above the 7B result in Table 2 (0.816) and should be explained or verified.
  3. [Section 6.1] The text refers to "LLaMA3.1-8B-Base" and "LLaMA3.1-8B-SFT*" in the setup, but later says "LLaMA3.1-7B-SFT*" in the discussion of Table 3. Please correct the model-size inconsistency.
  4. [Table 5] The abbreviation "OBen" is used in the table caption but defined only in the text of Section 6.4; a full name in the caption would help readers.
  5. [Appendix B, prompt] The final-step rule in the judge prompt awards a score of 1 whenever the final answer matches the ground-truth short answer, even if earlier steps contain unresolved reasoning errors. This is a reasonable evaluation choice for solution selection, but it should be stated explicitly as a modeling decision in the main text, because it affects the step-level label distribution.

Circularity Check

1 steps flagged · score 5.0 of 10

Step-level evaluation is partly circular: the o1 model that labels the PRM training data also supplies the step-level test labels (with human validation checking o1's own rationales), so the reported F1 partly measures agreement with o1's scoring convention; solution-level BoN results remain independent.

  1. self definitional [Section 5.1.1 (Dataset), Section 4.3 (LLM Judgement), and Appendix E (Human Annotation Details)]
    "we employ the generator to produce 800 solutions for prompts in the test set, forming our step-level test set, which is used to evaluate the accuracy of PRM in assessing each step of the reasoning process. The labels for this set are determined through a combination of cross-validation with the o1 model and manual annotation. ... Therefore, we incorporate the rules in Section 4.2 into the prompt (see Appendix B), allowing the reflective LLM to evaluate the correctness of each step. ... A step is considered correctly scored if both the rationale and the scoring outcome are deemed reasonable."

    The same o1 model, using the same Error Propagation and Error Cessation rules, both annotates the PRM training data and supplies the step-level test labels. The manual validation step only checks whether o1's rationale and score are reasonable; it does not independently re-label the steps from first principles. The step-level F1 (0.828) therefore measures how faithfully the trained PRM reproduces o1's scoring convention, making the reported step-level prediction partly a self-consistency check rather than an external correctness test. The training objective and the step-level evaluation metric share the same annotator and rubric, so the F1 gain over baselines could reflect better mimicry of o1 rather than better true step-level accuracy.

full rationale

The paper's solution-level claims are grounded in final-answer correctness and are independent of the annotation method, so the central result is not wholly circular. However, the step-level evaluation is not independent: Section 5.1.1 states that the step-level test labels are determined 'through a combination of cross-validation with the o1 model and manual annotation,' while Section 4.3 shows that the same o1 model, using the same Error Propagation/Error Cessation rules, generated the training labels. Appendix E explains that human annotators judge whether o1's rationale and score are reasonable, rather than producing labels independently. Consequently, the step-level F1 and balanced precision/recall partly reduce to agreement with o1's scoring convention, which is the input used to train the PRM. This is a partial, not total, circularity: the BoN, PRM@N-step, GSM8K, OBen, and AIME experiments use ground-truth answers and support the method's effectiveness. The paper's own Section 8 limitation that labeling accuracy is 'limited by the inherent abilities of the model itself' acknowledges the dependency but does not repair the circular step-level benchmark.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim relies on the annotation rules and the judge's accuracy, not on fitted constants. The hand-designed Error Propagation and Error Cessation rules are the primary assumptions, together with the quality of the LLM judge and the segmentation procedure.

assumptions (5)
  • domain assumption Long CoT models exhibit self-correction: correct reasoning steps can follow incorrect steps.
    The paper's motivation in Section 3 and Appendix D; open-source PRMs show lower accuracy on Reflection-Based sets.
  • domain assumption The o1 model (or a sufficiently capable LLM judge) can accurately label each reasoning step as correct or incorrect.
    Section 4.3 and Appendix E; the paper reports 0.963 agreement with human annotation, but this does not prove ground truth.
  • ad hoc to paper Error Propagation: if previous steps are incorrect and the current step neither introduces a new approach nor corrects the previous mistakes, the current step is incorrect.
    Defined in Section 4.2 as an annotation rule; this is a hand-designed criterion, not derived from prior theory.
  • ad hoc to paper Error Cessation: if previous steps are incorrect but the current step introduces a new, error-free approach or corrects the previous mistakes, the current step is correct.
    Defined in Section 4.2 as an annotation rule; it is a modeling choice about how to treat steps after errors.
  • domain assumption LLM-based segmentation into semantically coherent steps preserves the correctness information of the original solution.
    Section 4.1 uses LLM-based resegmentation of long CoT data; no direct validation that segmentation does not alter step correctness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond the First Error: Process Reward Models for Reflective Mathematical Reasoning." pith.science (2026). https://pith.science/paper/FIHGPDF7

@misc{pith2026250514391,
  author       = {Pith},
  title        = {Pith review of: Beyond the First Error: Process Reward Models for Reflective Mathematical Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FIHGPDF7}},
  note         = {Machine review of arXiv:2505.14391}
}
read the original abstract

Many studies focus on data annotation techniques for training effective PRMs. However, current methods encounter a significant issue when applied to long CoT reasoning processes: they tend to focus solely on the first incorrect step and all preceding steps, assuming that all subsequent steps are incorrect. These methods overlook the unique self-correction and reflection mechanisms inherent in long CoT, where correct reasoning steps may still occur after initial reasoning mistakes. To address this issue, we propose a novel data annotation method for PRMs specifically designed to score the long CoT reasoning process. Given that under the reflection pattern, correct and incorrect steps often alternate, we introduce the concepts of Error Propagation and Error Cessation, enhancing PRMs' ability to identify both effective self-correction behaviors and reasoning based on erroneous steps. Leveraging an LLM-based judger for annotation, we collect 1.7 million data samples to train a 7B PRM and evaluate it at both solution and step levels. Experimental results demonstrate that compared to existing open-source PRMs and PRMs trained on open-source datasets, our PRM achieves superior performance across various metrics, including search guidance, BoN, and F1 scores. Compared to widely used MC-based annotation methods, our annotation approach not only achieves higher data efficiency but also delivers superior performance. Detailed analysis is also conducted to demonstrate the stability and generalizability of our method.

Figures

Figures reproduced from arXiv: 2505.14391 by the authors.

Figure 1
Figure 1. The overall framework of our method. Based on our categorization, we introduce two new annotation rules designed to capture these post￾error reasoning dynamics: • Error Propagation: If the previous steps are incorrect and the current step neither intro￾duces a new approach nor corrects the previ￾ous mistakes, but instead builds upon the erro￾neous steps, the current step is also considered incorrect. • Error Cessati… view at source ↗
Figure 2
Figure 2. PRM@N of Qwen2.5-7B-SFT∗ using PRMs trained on data annotated by MC-based and our method. 5.3 Comparison with MC-based Methods Although MC-based method has been widely used, it contains significant noise. Their effectiveness re￾lies on the ability of completion models. Correct steps may be misjudged as incorrect when comple￾tion models fail to produce correct solutions within limited rollouts. Reflective reasoning p… view at source ↗
Figure 3
Figure 3. We categorize 1,000 solutions into 10 equal [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Distribution of the number of steps in each solution and the number of tokens contained in each step [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: An Example of solution reformation (part 1). [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: An Example of solution reformation (part 2). [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: The prompt template for LLM judger. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: An example of an annotation result from the LLM judger (part 1). [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: An example of an annotation result from the LLM judger (part 2). [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: An example of an annotation result from the LLM judger (part 3). [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Not All Errors Are Created Equal: ASCoT Addresses Late-Stage Fragility in Efficient LLM Reasoning

    cs.CL 2025-08 reject novelty 5.0 of 10

    ASCoT claims later reasoning errors are more harmful than early ones and uses a position-weighted verifier to prune and correct CoT steps, but its key evidence is internally inconsistent.

Reference graph

Works this paper leans on

37 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024. Alphamath almost zero: process supervision without process. arXiv preprint arXiv:2405.03553

    arXiv 2024

  5. [5]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  6. [6]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  7. [7]

    Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. 2024. On designing effective rl reward at training time for llm reasoning. arXiv preprint arXiv:2410.15115

    arXiv 2024

  8. [8]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    arXiv 2025

Show all 37 references
  1. [9]

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv...

    2024 arXiv

  2. [10]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    2021 arXiv

  3. [11]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  4. [12]

    Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. 2023. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702

    2023 arXiv

  5. [13]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. arXiv preprint arXiv:2305.20050

    2023 arXiv

  6. [14]

    Haoxiong Liu, Yifan Zhang, Yifan Luo, and Andrew Chi-Chih Yao. 2024. Augmenting math word problems via iterative question composing. arXiv preprint arXiv:2401.09003

    2024 arXiv

  7. [15]

    Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang, Chao Du, and Min Lin. 2025. There may not be aha moment in r1-zero-like training — a pilot study. https://oatllm.notion.site/oat-zero. Notion Blog

    2025

  8. [16]

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. 2024. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592

    2024 arXiv

  9. [17]

    MAA. 2024. https://maa.org/math-competitions/american-invitational-mathematics-examination-aime American invitational mathematics examination - aime

    2024

  10. [18]

    Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. 2024. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412.09413

    2024 arXiv

  11. [19]

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393

    2025 arXiv

  12. [20]

    Skywork o1 Team. 2024. https://huggingface.co/Skywork Skywork-o1 open series . https://huggingface.co/Skywork

    2024

  13. [21]

    OpenAI. 2024. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/. Accessed: Month Day, Year

    2024

  14. [22]

    Sungjin Park, Xiao Liu, Yeyun Gong, and Edward Choi. 2024. Ensembling large language models with process reward-guided tree search for better complex reasoning. arXiv preprint arXiv:2412.15797

    2024 arXiv

  15. [23]

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. 2024. Rewarding progress: Scaling automated process verifiers for llm reasoning. arXiv preprint arXiv:2410.08146

    2024 arXiv

  16. [24]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

    2024 arXiv

  17. [25]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    2024 arXiv

  18. [26]

    Kimi Team. 2025. Kimi k1.5: Scaling reinforcement learning with llms

    2025

  19. [27]

    Qwen Team. 2024 a . https://qwenlm.github.io/blog/qwq-32b-preview/ Qwen team. qwq: Reflect deeply on the boundaries of the unknown, 2024b

    2024

  20. [28]

    Qwen Team. 2024 b . https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  21. [29]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  22. [30]

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275

    2022 arXiv

  23. [31]

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024 a . Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis...

    2024

  24. [32]

    Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang. 2024 b . Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision. arXiv preprint arXiv:2402.02658

    2024 arXiv

  25. [33]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  26. [34]

    Rest-mcts*: Llm self-training via process reward guided tree search, 2024a

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search, 2024a. URL https://arxiv. org/abs/2406.03816

    arXiv

  27. [35]

    Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301

    2025 arXiv

  28. [36]

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493

    2022 arXiv

  29. [37]

    Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2024. Processbench: Identifying process errors in mathematical reasoning. arXiv preprint arXiv:2412.06559

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.