Pith. sign in

REVIEW 5 major objections 5 minor 39 references

DuaShepherd: Integrating Stepwise Correctness and Potential Rewards for Mathematical Reasoning

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multiplying a step-correctness score by a forward-potential score yields a compound reward that beats either signal alone in LLM math verification.

desk verdict A simple, plausible combination of two PRM signals that is undercut by a non-standard ProcessBench threshold choice; worth a referee but not the SOTA claim as written. read the letter →

arxiv 2506.17533 v1 pith:Y32X4CST submitted 2025-06-21 cs.CL

Yuanhao Wu , Juntong Song , Hanning Zhang , Tong Zhang , Cheng Niu This is my paper
classification cs.CL
keywords processrewardmodelsmathematicalreasoningcorrectnesspotentialcompoundprobabilitymulti-tasklearningbest-of-Nverificationpseudo-labeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish that process reward models for mathematical reasoning judge better when each reasoning step gets two scores: one for whether the step itself is correct, and one for whether the path still leads to a correct final answer. The two scores are trained as separate heads on a shared base model and combined by multiplication, which the authors treat as a compound probability via the chain rule. On MATH500 best-of-64 with a Mistral-7B generator, the combined reward scores 0.526, versus 0.472 for either the correctness-only or the potential-only reward; on ProcessBench the full DuaShepherd model averages 65.5 percent, up from 57.5 percent for the correctness-only model. The point of caring is practical: the training labels come from two existing datasets plus pseudo-labeling, so the improvement does not require new human step annotations. If the claim is right, a cheap dual-signal recipe gives better verifiers for selecting among candidate math solutions.

What carries the argument

The load-bearing identity is Eq. (5), $R_{\mathrm{DuaShepherd}} = R_{\mathrm{correctness}} \cdot R_{\mathrm{potential}}$, applied to every step. Each factor is a sigmoid output from a linear head on a shared base model's hidden state; the heads are trained with summed binary cross-entropy on a dataset built by taking Math-Shepherd trajectories, keeping the Monte Carlo potential labels, and pseudo-labeling correctness with a reward model trained on PRM800K. The product is justified as a chain-rule compound probability, and the paper compares it against using either head alone, distilling the product into a single head, and multiplying two independently trained models' scores.

What would settle it

On the same MATH500 candidates, re-rank the solutions with the product $R_c R_p$, the minimum, and a weighted sum of the two heads; if the product does not beat the weighted sum specifically on trajectories where the two scores disagree (for example, a wrong step followed by a correct answer), then the claimed chain-rule mechanism is not what is carrying the gain. A more direct probe is to train a version of the model on synthetic data where correctness and potential are deliberately decoupled and see whether the product's advantage tracks that decoupling.

Watch

Extended reading notes

Core claim

The central claim is that step correctness and forward potential are complementary, measurable properties of a reasoning step, and that their product, $R_{\mathrm{DuaShepherd}} = R_{\mathrm{correctness}} \, R_{\mathrm{potential}}$, is a more reliable stepwise reward than either factor alone. Correctness looks backward, flagging whether any step so far is erroneous; potential looks forward, estimating whether the solution still ends at the right answer. The paper shows that multiplying two separately trained rewards already improves on both teachers, and that jointly training both heads in one multi-head model improves further, with each DuaShepherd head outperforming its single-task teacher. It also reports a weak-to-strong pattern in which a larger model trained on the product-style pseudo-labels does even better, which the authors take as evidence that the multiplication is surfacing a genuine underlying signal about reasoning quality.

Load-bearing premise

The premise the whole construction rests on is that 'this step is correct' and 'this step leads to the right answer' are separate enough that their scores can simply be multiplied into one probability; the paper's own reported correlation of 0.812 between the two scores shows that independence is not actually there, so the compound probability is a heuristic rather than a derived chain-rule product.

Editorial extensions

If this is right

  • Best-of-N solution selection improves on MATH500 without new human annotation, because the dual labels come from existing datasets plus pseudo-labels.
  • Multi-task training strengthens each individual head relative to its single-task teacher, so the benefit is not only in the final product but in the shared representation.
  • ProcessBench error-detection accuracy also rises, showing the dual signal helps locate the first wrong step, not just pick final answers.
  • The automated pipeline is scale-expandable, so the same dual-label recipe can be regenerated for newer generators as their solution styles drift from the original training data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper reports an overall Pearson correlation of 0.812 between the two rewards, the chain-rule interpretation is not strictly supported; a plausible reading is that the product acts as an AND-like gate that rejects candidates scoring low on either signal, and this heuristic is what yields the gains.
  • A testable consequence is that the product's edge over its factors should grow when the two signals are deliberately decorrelated; building a probe set where correctness and potential disagree and measuring the gap would separate the complementarity story from a simple weighting effect.
  • The weak-to-strong result suggests the pseudo-labeled product data may encode a generalizable preference, but the reported gains for stronger generators shrink; training on generator-aligned samples is the direct experiment that would show whether the dual-signal advantage survives distribution shift.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes DuaShepherd, a reward-modeling framework that combines two step-level signals: a correctness reward (trained on PRM800K, with neutral and positive labels collapsed to correct) and a potential reward (trained on Math-Shepherd's Monte-Carlo-derived labels). The two heads share a Qwen2.5-Math-7B base and are trained with multi-task binary cross-entropy; at inference the final score is the product R_correctness * R_potential, presented as a compound probability via a chain-rule argument (Eq. 5). The authors construct a pseudo-labeled training set by annotating Math-Shepherd steps with both a correctness model and a potential model, and evaluate on MATH500 best-of-64 selection and ProcessBench first-error detection, reporting gains over either single reward and over the separately trained teachers, and claiming state-of-the-art under comparable resource constraints (Tables 1 and 2). The paper also reports analyses of reward correlation, alternative mixing approaches, and a weak-to-strong scaling experiment.

Significance. The core idea—combining a backward-looking correctness signal with a forward-looking potential signal—is timely and practically motivated, and the automated pipeline that reuses PRM800K and Math-Shepherd without additional sampling is a genuine strength. The multi-head multi-task formulation is simple and plausible, and the head-level comparisons in Table 1 provide some evidence that joint training helps. The paper also contains useful analyses, including the reward-scatter diagnostics of Section 6.2 and the comparison of mixing strategies in Table 4. However, the headline ProcessBench claim is weakened by a protocol deviation, the compound-probability derivation in Eq. (5) is not actually derived, the potential-RM label validation in Table 3 is circular, and the reported numbers lack error bars or significance tests. If the ProcessBench comparison is re-run under the official protocol and the product rule is reframed as a heuristic combination, the contribution can be salvageable, but as written the central empirical claim is not fully supported.

major comments (5)
  1. [§4.1, Table 2] The ProcessBench evaluation protocol is not comparable to the published baselines. The paper fixes the threshold at 0.5 for all models that output a scalar, whereas the starred baselines taken from the ProcessBench paper use the official protocol of tuning the threshold on the GSM8K subset. First-error-detection F1 is threshold-dependent, so comparing a fixed-threshold DuaShepherd against tuned-threshold baselines can inflate or deflate the reported margin. The authors should re-evaluate all scalar-output models, including their own, under the official GSM8K-tuned protocol, or report the threshold sweep curve; without this, the state-of-the-art claim in the Abstract and Section 5.1 is not supported.
  2. [§3.3, Eq. (5)] Eq. (5) is presented as following from the chain rule of probability, but no conditioning event or independence assumption is stated. Moreover, Section 6.2 reports an overall Pearson correlation of 0.812 between the two reward scores, so the two signals are far from conditionally independent, and each score is a sigmoid output rather than a probability over a well-defined joint event. The product is a heuristic fusion, not a derived compound probability. The authors should either state the simplifying assumption explicitly and test it, or present the product as a heuristic aggregation with appropriate justification.
  3. [§6.1, Table 3] The validation of the potential reward model's pseudo-labels is circular. The potential RM was trained on Math-Shepherd binary labels and is then used to generate soft pseudo-labels on Math-Shepherd; comparing those pseudo-labels to the original Math-Shepherd labels measures consistency with the training target, not label quality. This part of Table 3 should be removed or replaced with an evaluation against an independent reference (e.g., the o1-mini annotations used for the correctness RM, or human judgments).
  4. [Tables 1 and 2] No error bars, multiple seeds, or significance tests are reported, yet the central claim is 'consistent performance improvements' and several reported margins are small (e.g., 0.498 vs 0.526 on MATH500 with Mistral, and 63.0% vs 65.5% on ProcessBench). Given the stochasticity of best-of-N evaluation and PRM training, the authors should provide at least bootstrap confidence intervals over the 500 MATH500 problems or averaged results over independent training runs.
  5. [§5.1, Table 1] The claimed consistency of the compound reward is not always observed. For DeepSeekMath-Instruct-7B, the DuaShepherd correctness head alone (0.626) outperforms the full DuaShepherd compound (0.622), and for Qwen-2.5-Math-Instruct-7B the potential head alone matches the compound (0.824 vs 0.824). The text should acknowledge these exceptions and discuss when the product helps versus when it merely tracks the stronger head.
minor comments (5)
  1. [§5.3] There is a typo in 'DuaShepherd!over majority voting' in the paragraph beginning 'As shown in Table 1'.
  2. [§1] The introduction contains a duplicated sentence: the two sentences beginning 'Although PRM800K requires steps labeled as positive...' are repeated verbatim.
  3. [Figure 3] The caption refers to 'Pcorrectness' and 'Ppotential', while the axis labels in the figure and the text use 'PRM Score' and 'Shepherd Score'; the notation should be unified to match Eqs. (2)–(3).
  4. [§4.1] The paper alternates between 'MATH500' and 'Math500'; standardize the capitalization.
  5. [§1] The contribution list has a typo: 'final answer..' should be 'final answer.'

Circularity Check

1 steps flagged · score 2.0 of 10

Minor in-sample validation of potential pseudo-labels; central benchmark claims remain independent.

  1. fitted input called prediction [Section 3.2 (dataset construction) and Section 6.1 / Table 3]
    "Since the Math-Shepherd dataset only provides binary hard labels, we used the potential reward model to generate corresponding pseudo-soft labels as potential rewards. ... For pseudo-labels generated by the potential reward model, we computed Precision, Recall, and F1 score against the original MathShepherd binary labels. ... The pseudo-labels from our model achieved an F1 score of 0.9346, demonstrating high consistency with the original labeling."

    The potential reward model used to generate the pseudo-soft labels was itself trained on the Math-Shepherd binary labels (Section 3.2: 'we used the Math-Shepherd dataset to train the potential reward model with its original binary label'). Comparing the resulting pseudo-labels to those same training labels therefore measures in-sample agreement; a high F1 is expected from a model fit to those targets, not independent evidence that the annotations generalize. The paper uses this as a quality claim ('These metrics indicate that our pseudo-labels are ... high label quality is crucial'), so the dataset-quality evidence for the potential signal is partly by construction.

full rationale

The central DuaShepherd claim—that multiplying correctness and potential scores improves verifier accuracy—is not circular: Eq. (5) is an explicit modeling choice, and the headline numbers (MATH500 best-of-64, ProcessBench F1) are computed on external benchmarks with independently trained baseline reward models. The DuaShepherd model is trained on pseudo-labels produced by those teachers, but this is distillation, not equivalence; the comparison to 'Mixing 1 & 2 with compound probability' is a real comparison. The one circular element is in Section 6.1/Table 3: the potential reward model's pseudo-labels are validated against the same Math-Shepherd binary labels used to train that model, so the reported 0.9346 F1 is an in-sample fit rather than independent evidence of label quality. This weakens the dataset-quality argument but does not reduce the empirical benchmark claim, because MATH500 and ProcessBench results are external and the correctness reward model is separately checked against o1-mini annotations. The ProcessBench fixed-threshold deviation and the high correlation between the two rewards are validity and correctness concerns, not circularity, and do not raise the circularity score.

Assumptions & free parameters 1 free parameters · 2 assumptions · 0 invented entities

No genuinely new physical or conceptual entities are introduced. The two free/assumed inputs are the fixed decision threshold and the reliability of teacher-generated pseudo-labels, with the product rule resting on an unjustified independence assumption.

free parameters (1)
  • ProcessBench decision threshold = 0.5
    The authors fix the scalar-output threshold at 0.5 for all models instead of selecting it on the GSM8K subset as done in the ProcessBench protocol. This choice affects the reported averages and comparability with published baselines.
assumptions (2)
  • ad hoc to paper Conditional independence of correctness and potential rewards
    Eq. (5) multiplies the two sigmoid scores as a 'compound probability' using the chain rule, but the paper reports a Pearson correlation of 0.812 between the rewards (Section 6.2), so the independence assumption does not hold. The product is a heuristic, not a derived probability.
  • domain assumption Teacher pseudo-labels are reliable enough for training
    The DuaShepherd dataset is labeled by two teacher reward models. The potential teacher is validated against its own training labels (Table 3), which is circular, so the quality of the training signal is not established independently.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DuaShepherd: Integrating Stepwise Correctness and Potential Rewards for Mathematical Reasoning." pith.science (2026). https://pith.science/paper/Y32X4CST

@misc{pith2026250617533,
  author       = {Pith},
  title        = {Pith review of: DuaShepherd: Integrating Stepwise Correctness and Potential Rewards for Mathematical Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y32X4CST}},
  note         = {Machine review of arXiv:2506.17533}
}
read the original abstract

In this paper, we propose DuaShepherd, a novel reward modeling framework that integrates two complementary reward signals, correctness and potential, to enhance the mathematical reasoning capabilities of Large Language Models (LLMs). While correctness-based signals emphasize identification of stepwise errors, potential-based signals focus on the likelihood of reaching the correct final answer. We developed an automated pipeline for constructing large-scale reward modeling dataset with both signals. A unified, multi-head architecture was explored to train the two reward models in a multi-task setup, demonstrating benefits from learning both correctness and potential in parallel. By combining these two signals into a compound probability, our model achieves consistent performance improvements across multiple benchmarks. Empirical evaluations on MATH500 and ProcessBench confirm that this combined reward significantly outperforms models trained on either reward type alone, achieving state-of-the-art performance under comparable resource constraints.

Figures

Figures reproduced from arXiv: 2506.17533 by the authors.

Figure 1
Figure 1. Overview of the DuaShepherdpipeline. We first trained reward models from PRM800K and Math [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance of Mistral-7B using different [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Reward distribution of the best solutions picked by different reward models in the MATH500 test set. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 7 canonical work pages

  1. [1]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

    arXiv 2020

  2. [2]

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. 2023. http://arxiv.org/abs/2312.09390 Weak-to-strong generalization: Eliciting strong capabilities with weak supervision

    arXiv 2023

  3. [3]

    Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024. http://arxiv.org/abs/2406.10858 Step-level value preference optimization for mathematical reasoning

    arXiv 2024

  4. [4]

    Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2024. http://arxiv.org/abs/2309.15402 Navigate through enigmatic labyrinth a survey of chain of thought reasoning: Advances, frontiers and future

    arXiv 2024

  5. [5]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. http://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems

    arXiv 2021

  6. [6]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

  7. [7]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huaj...

    arXiv 2024

  8. [8]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. 2024. http://arxiv.org/abs/2309.17452 Tora: A tool-integrated reasoning agent for mathematical problem solving

    arXiv 2024

Show all 39 references
  1. [9]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  2. [10]

    Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. 2025. http://arxiv.org/abs/2501.04519 rstar-math: Small llms can master math reasoning with self-evolved deep thinking

    2025 arXiv

  3. [11]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS

    2021

  4. [12]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  5. [13]

    Jie Huang and Kevin Chen-Chuan Chang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.67 Towards reasoning in large language models: A survey . In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049--1065, Toronto, Canada. Association for Comput...

    2023 doi

  6. [14]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  7. [15]

    Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. 2024. http://arxiv.org/abs/2410.01679 Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment

    2024 arXiv

  8. [16]

    Yanda Li, Dixuan Wang, Jiaqing Liang, Guochao Jiang, Qianyu He, Yanghua Xiao, and Deqing Yang. 2024. http://arxiv.org/abs/2404.04293 Reason from fallacy: Enhancing large language models' logical reasoning through logical fallacy understanding

    2024 arXiv

  9. [17]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. https://openreview.net/forum?id=v8L0pN6EOi Let's verify step by step . In The Twelfth International Conference on Learning Re...

    2024

  10. [18]

    Skywork o1 Team. 2024. https://huggingface.co/Skywork Skywork-o1 open series . https://huggingface.co/Skywork

    2024

  11. [19]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  12. [20]

    Team Qwen. 2024 a . https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  13. [21]

    Team Qwen. 2024 b . https://qwenlm.github.io/blog/qwq-32b-preview/ Qwq: Reflect deeply on the boundaries of the unknown

    2024

  14. [22]

    Matthew Renze and Erhan Guven. 2024. http://arxiv.org/abs/2405.06682 Self-reflection in llm agents: Effects on problem-solving performance

    2024 arXiv

  15. [23]

    Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. 2024 a . http://arxiv.org/abs/2406.14532 Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold

    2024 arXiv

  16. [24]

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. 2024 b . http://arxiv.org/abs/2410.08146 Rewarding progress: Scaling automated process verifiers for llm reasoning

    2024 arXiv

  17. [25]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. http://arxiv.org/abs/2402.03300 Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    2024 arXiv

  18. [26]

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. http://arxiv.org/abs/2211.14275 Solving math word problems with process- and outcome-based feedback

    2022 arXiv

  19. [27]

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024 a . http://arxiv.org/abs/2406.12845 Interpretable preferences via multi-objective reward modeling and mixture-of-experts

    2024 arXiv

  20. [28]

    Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2023. http://arxiv.org/abs/2310.03731 Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning

    2023 arXiv

  21. [29]

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.510 Math-shepherd: Verify and reinforce LLM s step-by-step without human annotations . In Proceedings of the 62nd Annual Meet...

    2024 doi

  22. [30]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. http://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models

    2023 arXiv

  23. [31]

    Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang. 2024. An implementation of generative prm. https://github.com/RLHFlow/RLHF-Reward-Modeling

    2024

  24. [32]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  25. [33]

    Fei Yu, Anningzhe Gao, and Benyou Wang. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.55 OVM , outcome-supervised value models for planning in mathematical reasoning . In Findings of the Association for Computational Linguistics: NAACL 2024, pages 858--875, Mexico City...

    2024 doi

  26. [34]

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024 a . https://proceedings.neurips.cc/paper_files/paper/2024/file/76ec4dc30e9faaf0e4b6093eaa377218-Paper-Conference.pdf Rest-mcts* : Llm self-training via process reward guided tree search . In Adva...

    2024

  27. [35]

    Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. 2024 b . http://arxiv.org/abs/2406.07394 Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b

    2024 arXiv

  28. [36]

    Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2024. http://arxiv.org/abs/2412.06559 Processbench: Identifying process errors in mathematical reasoning

    2024 arXiv

  29. [37]

    Qihuang Zhong, Kang Wang, Ziyang Xu, Juhua Liu, Liang Ding, and Bo Du. 2024. http://arxiv.org/abs/2404.14963 Achieving >97\

    2024 arXiv

  30. [38]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.