Dynamic and Generalizable Process Reward Modeling

Qinyuan Cheng; Qiushi Sun; Xipeng Qiu; Xuanjing Huang; Zhangyue Yin; Zhiyuan Zeng

REVIEW 4 major objections 5 minor 24 references

Dynamic and Generalizable Process Reward Modeling

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DG-PRM claims that storing reward criteria in a dynamic tree lifts process-reward accuracy to a new state of the art.

desk verdict Worth engaging: the reward-tree mechanism is genuinely new and the ablations around it are informative, but the headline SOTA rests on an apples-to-oranges judge comparison that needs matched-judge reruns. read the letter →

arxiv 2507.17849 v1 pith:AZEVUHCK submitted 2025-07-23 cs.CL

Zhangyue Yin , Qiushi Sun , Zhiyuan Zeng , Qinyuan Cheng , Xipeng Qiu , Xuanjing Huang This is my paper · ORCID

classification cs.CL

keywords processrewardmodeltreeLLM-as-a-judgeParetodominancedirectpreferenceoptimizationcross-domaingeneralizationstep-wisePRMBench

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that process reward models fail because they score every step against fixed, coarse criteria and give all errors the same negative label. It proposes DG-PRM, which stores many fine-grained reward criteria in a hierarchical reward tree, picks step-specific criteria dynamically, and uses Pareto dominance to build preference pairs. On PRMBench the framework raises DeepSeek-R1's overall PRM-Score from 69.5% to 76.5%, a new state of the art, and improves accuracy across general, science, and commonsense tasks. The claim, in short, is that dynamic selection of criteria from a reusable tree is what makes process rewards accurate and transferable across domains.

What carries the argument

The reward tree is the central object: a hierarchical clustering of embedded reward criteria, with coarse-grained parent nodes and fine-grained child nodes, built by the BIRCH algorithm on GPT-4o-generated judgments comparing positive and negative outputs. It carries the argument by allowing per-step retrieval: coarse criteria are selected with a context window of µ=20, fine criteria are matched when cosine distance is at most ζ=0.2, and similar criteria are merged when distance is at most ξ=0.25. Pareto dominance estimation converts the resulting multi-criterion scores into preference pairs, and DPO optimizes a step-wise log-ratio reward derived from the policy against a reference model.

What would settle it

Run a matched-judge experiment on PRMBench: use the same 7B model as both judge and scorer for DG-PRM and for Critic CoT and ORM baselines; if the DG-PRM advantage shrinks or reverses, the gains reflect judge strength. A second check is to re-tune the thresholds ξ, ζ, and µ on each target domain and see whether the QASC-tuned defaults still hold.

Watch

Extended reading notes

Core claim

DG-PRM establishes that process reward modeling can be made dynamic and generalizable by extracting comparative reward criteria from LLM judgments, organizing them into a hierarchical reward tree, and selecting per-step criteria by embedding similarity; Pareto dominance then picks discriminative positive and negative step pairs for step-wise DPO. The paper reports that this pipeline, with GPT-4o as judge and scorer, achieves state-of-the-art PRM-Score on PRMBench (DeepSeek-R1: 69.5% to 76.5%), lifts R1-Distill-Qwen-7B's Prerequisite Sensitivity from 45.6% to 62.4%, improves win rates on MT-Bench and Arena-Hard, and transfers out of distribution with smaller degradation than critic or ORM baselines.

Load-bearing premise

The reward pipeline's criteria and step scores come from GPT-4o by default, and the paper reports no matched-judge comparison where the same model scores both DG-PRM and the baselines, so part of the measured gain could come from the judge's strength rather than from the reward-tree mechanism itself.

Editorial extensions

If this is right

If DG-PRM is correct, process reward modeling no longer needs domain-specific hand-crafted labels; one constructed reward tree can be reused and updated across domains.
Step-level dense rewards built this way improve downstream policy models on math, general chat, science, and commonsense benchmarks, approaching or exceeding training on human-annotated process labels.
Out-of-distribution transfer holds: reward trees built on QASC or StrategyQA can score ChemistryQA and ARC-c with less degradation than ORM or critic baselines.
Training efficiency improves substantially, with DG-PRM reaching Critic CoT-level accuracy in about 30% of the training steps on QASC, StrategyQA, and ARC-c, and using half the compute of Critic CoT-SC(50) on PRMBench while gaining 4.8% overall.
The framework yields interpretable rewards, since each step is scored against explicitly selected criteria that explain why a positive step dominates a negative one.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the gains survive a matched-judge control, the framework suggests that criteria diversity and dynamic selection, not judge strength alone, drive PRM quality; a testable prediction is that swapping GPT-4o for a 7B judge would shrink but not erase the DG-PRM advantage.
The Pareto-based pair selection may transfer to other preference-optimization settings beyond process rewards, such as multi-objective RLHF, because it cleanly identifies non-dominated samples under diverse reward signals.
The reward tree's reusability points to a scaling path where a single general tree built on mixed domains serves as shared infrastructure, reducing per-domain annotation cost; the paper's unified-tree experiment on MT-Bench is initial evidence for this direction.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. The reward-tree idea is genuinely new and the paper gives it a fair internal test. But the headline PRMBench numbers and the MT-Bench/Arena-Hard wins are confounded, and the reader's stress-test is right: DG-PRM runs GPT-4o as scorer and validator while the Critic baselines judge with the policy model itself. The comparison does not isolate the framework.

What is actually good: the hierarchical organization of LLM-generated criteria and the Pareto-dominance pair construction are not in the cited prior work, and the internal ablations around them are the most useful part of the paper. The w/o-Pareto ablation (Table 2) shows a consistent 1.8-4.0 point gain across four datasets, and the D.1 judge-model analysis shows that better judges help, which is a partial acknowledgement but does not fix the headline comparison. The cost analysis and the unified reward tree experiment in D.5 also point in sensible directions.

The soft spots are serious but not fatal to the framework's potential. First, no matched-judge baseline appears anywhere. The authors need, at minimum, Critic CoT/Direct rerun with GPT-4o scoring, and DG-PRM with self-scoring, to attribute the gains to the reward tree rather than to the judge model. Second, there are no error bars or significance tests, and the o1-mini subset note is ambiguous. Third, no code or data release, which matters for a paper that is largely empirical. Fourth, the hyperparameters ξ, ζ, µ are tuned on QASC and transferred without testing the transfer assumption; sensitivity is shown only on QASC. These are fixable in revision, not signs of incoherence.

Citation pattern is fine: the related work is broad and the authors properly credit existing PRM and LLM-as-judge lines. The limitations section is honest about domain expansion and human oversight, though it does not flag the judge-confounding issue, which is the real omission.

Who this is for: people working on LLM-as-judge, process reward models, or preference optimization. It deserves a serious referee because the core mechanism is plausible and the empirical gaps are fixable. My recommendation: send to review, with instructions to insist on matched-judge controls and artifact release before any 'new state-of-the-art' claim is allowed to stand.

Referee Report

4 major / 5 minor

Summary. The paper proposes DG-PRM, a framework for process reward modeling that constructs a reward tree of fine-grained criteria via an LLM judge, dynamically selects criteria for each step, and uses Pareto dominance to form preference pairs for DPO training. Experiments on PRMBench, MT-Bench, Arena-Hard, QASC, StrategyQA, ChemistryQA, and ARC-c claim state-of-the-art process reward modeling and improved downstream accuracy, generalization, and training efficiency.

Significance. If the empirical claims are valid, DG-PRM would be a meaningful contribution: it advances beyond static, coarse-grained PRMs by storing and dynamically selecting multi-dimensional criteria, and its Pareto-based preference construction is a sensible way to handle conflicting reward signals. The paper includes extensive experiments, ablations, a human evaluation of reward allocation, and a computational cost analysis, which are strengths. However, the central SOTA and generalization claims are undermined by a judge-model confound in the experimental design.

major comments (4)

[Appendix B.1, Table 1, §5.2] The headline comparisons in Table 1 and the training-efficiency curves in Figure 6 do not control for the judge model. Appendix B.1 states 'By default, we use GPT-4o as the scorer S,' while the Critic baselines described in §5.2 prompt the policy model itself to produce step critiques. Thus every DG-PRM row receives scores from GPT-4o, whereas the 'Prompted as Critic Models' baselines in Table 1 are the official leaderboard results in which the model under evaluation is the critic. The reported gain (e.g., DeepSeek-R1 69.5→76.5) could therefore reflect GPT-4o's superior step-level judgment rather than the reward-tree/Pareto mechanism. The paper should add a matched-judge control: run the Critic Direct/CoT baselines with GPT-4o as the step scorer, and run DG-PRM with the policy model as scorer S, so that the judge is held fixed across conditions.
[§5.3, Appendix A, Figure 3] The MT-Bench and Arena-Hard evaluations use GPT-4o as the judge (Appendix A), and GPT-4o is also the model that generates the reward criteria (J), validates them, and scores each step (S). Consequently, the policy is optimized to match GPT-4o's assessment criteria and then evaluated by GPT-4o. This circularity is particularly problematic for the 'general' benchmarks because the reward tree for MT-Bench/Arena-Hard is built from LMSYS-Human-Preference-55k, which is derived from the same human preference distribution that MT-Bench/Arena-Hard sample; the evaluation is therefore not an independent test of generalization. The authors should report results with an independent judge (e.g., another LLM or human evaluation) to verify that the improvements are not an artifact of optimizing for the evaluator's preferences.
[Figures 4–6, §5.3] The same judge confound affects the Science and Commonsense results. The preference data for DG-PRM is produced by the GPT-4o pipeline, while the Critic baselines use the policy model's own judgments. Without a matched judge, the relative ranking of DG-PRM against Critic Direct/CoT in Figures 4–6 is uninterpretable as evidence that the reward-tree mechanism, rather than the stronger judge, drives the gains. The paper should include an additional condition in which the Critic baselines receive scores from the same judge (GPT-4o) used for DG-PRM.
[Appendix C, §4.2] The hyperparameters ξ=0.25, ζ=0.2, and µ=20 are tuned solely on QASC (Appendix C) and then applied to all other datasets and models without further tuning. Because these thresholds control tree construction, fine-grained matching, and context window, the paper's cross-domain generalization claim rests on their transferability. A sensitivity analysis for at least one additional domain (e.g., StrategyQA or ARC-c) would demonstrate that the fixed values are not an over-fit to QASC.

minor comments (5)

[§5.2] Typo: 'We obverse' should be 'We observe'.
[§5.3] Typo: 'Arean-Hard' should be 'Arena-Hard' in several places, and 'Critical Model' should be 'Critic Model'.
[Throughout] The benchmark name appears as 'PRMB ENCH' with a space in many locations, which is distracting.
[§5.3] The reference to 'Table 3 presents the results on MT-Bench and Arean-Hard' is incorrect: Table 3 in the appendix is the dataset description table. The actual results are in Figure 3, so the cross-reference should be fixed.
[Abstract and §5.3] The abstract and main text use non-quantitative superlatives such as 'stunning performance' and 'exceptional generalizability'; more measured language would be appropriate for a scientific report.

Circularity Check

1 steps flagged · score 4.0 of 10

General-benchmark gains are partly self-referential because GPT-4o supplies both the reward signal used to train DG-PRM and the judge used to evaluate it; the PRMBench and objective-accuracy results remain independent.

self definitional [Eq. (15) and Eq. (18) in Sec. 4; Appendix B.1; Appendix A; Fig. 3 caption]
"By default, we use GPT-4o as the scorer S. ... MT-Bench ... with evaluation scores output by GPT-4o. ... Arena-Hard ... utilizing GPT-4o as the judge model. ... The R1-Distill-Qwen-7B model is used as the backbone, with GPT-4o (OpenAI, 2024a) serving as the judge model."

DG-PRM's reward criteria come from the judge J=GPT-4o (Eq. 5; Appendix B.1), its step scores are s(t)_k = S(r_k, yhat(t), I_t) with S=GPT-4o (Eq. 15; Appendix B.1), and the DPO objective (Eqs. 18-20) trains the policy to increase the log-ratio for steps preferred under those GPT-4o scores. The claimed general-benchmark gains are then measured by GPT-4o ratings on MT-Bench and Arena-Hard (Appendix A; Fig. 3 caption). So for those benchmarks, the evaluation signal is the same model that proposed, validated, and scored the reward criteria; improving the GPT-4o-judged win rate is partly a self-agreement effect rather than an independent measure of process-reward quality or generalizability.

full rationale

The paper's core mechanism (reward tree + Pareto selection + DPO) is implemented with standard components and is not derived from a self-citation chain, a uniqueness theorem, or an ansatz smuggled in by citation; there is also no renaming of a known result. The central PRMBench claim uses objective error-position labels (PRM-Score) and the science/commonsense claims use objective accuracy, so those results are independent of the GPT-4o scoring loop. However, the general-benchmark claims (MT-Bench, Arena-Hard) are partially circular: GPT-4o generates, validates, and scores the reward criteria used for training and also serves as the judge in evaluation. Additionally, the PRMBench state-of-the-art comparison is confounded because DG-PRM rows receive GPT-4o judgments while the official-leaderboard Critic baselines use the policy model itself as critic; this is a comparison-control problem rather than a circular reduction. Overall, the load-bearing 'generalizable' claim has independent objective support, but the GPT-4o self-evaluation loop on general benchmarks warrants a moderate circularity score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on domain assumptions about LLM-generated criteria being faithful and embedding-space similarity being a valid matching signal, plus four hand-set parameters. No new physical or virtual entities are postulated.

free parameters (4)

Merge threshold ξ = 0.25
Controls merging of similar reward criteria in the reward tree; tuned on QASC (Figure 10a) and applied to all other datasets.
Matching threshold ζ = 0.2
Controls selection of fine-grained reward criteria via cosine distance; tuned on QASC (Figure 10b) and applied to all other datasets.
Context window µ = 20
Limits previous steps used in selection and scoring; set based on context-window constraints and performance (Appendix C).
Sampling frequency for coarse criteria = 5 samples, retain >3
Decision rule for reliable coarse-grained reward selection; chosen in Appendix B.1 without a dedicated ablation.

assumptions (4)

domain assumption LLM-generated reward criteria, when embedded with bge-en-icl, are semantically comparable via cosine distance so that hierarchical clustering yields meaningful coarse and fine criteria.
Central to reward tree construction (Section 4.1, equations 6-7); not independently verified beyond aggregate results.
domain assumption The automated validator (GPT-4o) reliably separates good from bad reward criteria, and its Good/Ordinary/Bad judgments align with human judgment outside the 100-sample consistency check.
Used to filter R (Appendix B.2); consistency is checked on 100 samples only (Figure 9).
ad hoc to paper Pareto-optimal steps provide better DPO training pairs than randomly paired high/low scoring steps.
The ablation in Table 2 supports it on four datasets, but it is a design hypothesis specific to DG-PRM, not a theorem.
ad hoc to paper The fixed thresholds ξ, ζ, µ, tuned on QASC, remain appropriate for all other domains and models.
Applied globally (Section 5.2, Appendix C) with no per-domain tuning or sensitivity analysis outside QASC.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Dynamic and Generalizable Process Reward Modeling." pith.science (2026). https://pith.science/paper/AZEVUHCK

@misc{pith2026250717849,
  author       = {Pith},
  title        = {Pith review of: Dynamic and Generalizable Process Reward Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AZEVUHCK}},
  note         = {Machine review of arXiv:2507.17849}
}

read the original abstract

Process Reward Models (PRMs) are crucial for guiding Large Language Models (LLMs) in complex scenarios by providing dense reward signals. However, existing PRMs primarily rely on heuristic approaches, which struggle with cross-domain generalization. While LLM-as-judge has been proposed to provide generalized rewards, current research has focused mainly on feedback results, overlooking the meaningful guidance embedded within the text. Additionally, static and coarse-grained evaluation criteria struggle to adapt to complex process supervision. To tackle these challenges, we propose Dynamic and Generalizable Process Reward Modeling (DG-PRM), which features a reward tree to capture and store fine-grained, multi-dimensional reward criteria. DG-PRM dynamically selects reward signals for step-wise reward scoring. To handle multifaceted reward signals, we pioneeringly adopt Pareto dominance estimation to identify discriminative positive and negative pairs. Experimental results show that DG-PRM achieves stunning performance on prevailing benchmarks, significantly boosting model performance across tasks with dense rewards. Further analysis reveals that DG-PRM adapts well to out-of-distribution scenarios, demonstrating exceptional generalizability.

Figures

Figures reproduced from arXiv: 2507.17849 by the authors.

**Figure 1.** An example from MT-Bench, where LLM judgments contain rich details, as highlighted in bold, but prevailing PRM methods fail to fully exploit. approach provides more dense reward signals (Li and Li, 2024), especially in complex problems where the model needs to reason, analyze, and explore different solution strategies (Wei et al., 2022). By evaluating intermediate steps, PRM is essential for improving the model’s a… view at source ↗

**Figure 2.** Overview of DG-PRM. DG-PRM consists of three main steps: (a) Automatic Process Reward Design, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Performance comparison on (a) MT-Bench and (b) Arena-Hard. The R1-Distill-Qwen-7B model is used [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Performance comparison on (a) QASC and (b) [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Generalization analysis on (a) ChemistryQA and (b) ARC-c. In the Out-of-Distribution setting, we [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Accuracy(%) variation with training steps on the QASC, StrategyQA, and ARC-c datasets. eter scales, demonstrating a clear advantage over other methods. Furthermore, DG-PRM achieves performance close to human annotation, even surpassing human-level performance on the S…

**Figure 8.** Figure 8: Token consumption and performance compar [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

**Figure 9.** Figure 9: Consistency between automated validator and [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

**Figure 10.** Figure 10: Ablation analysis of the hyperparameters on the QASC dataset, using R1-Distill-Qwen-7B as the [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

**Figure 11.** Figure 11: Impact of the judge model on ratings in the [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

**Figure 12.** Figure 12: Performance comparison of different PRM methods on the MT-Bench dataset, using R1-DistillQwen-7B as the backbone. plays a crucial role in the construction of the reward tree and the selection of fine-grained process reward criteria. We find that DG-PRM demonstrates…

**Figure 14.** Figure 14: Human evaluation of DG-PRM output, including the selection of reward criteria and scores. Optimization Algorithm Single-Turn Multi-Turn Baseline 6.01 5.66 DPO (Rafailov et al., 2023) 8.67 8.09 CPO (Xu et al., 2024) 8.58 7.87 SimPO (Meng et al., 2024) 8.62 7.95 [PITH…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 12 canonical work pages

[1]

Assistant A is significantly better: [[A»B]]

work page
[2]

Assistant A is slightly better: [[A>B]]
[3]

Tie, relatively the same: [[A=B]]
[4]

In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 4571–4585, Bangkok, Thailand

StepCoder: Improving code generation with reinforcement learning from compiler feedback. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 4571–4585, Bangkok, Thailand. Association for Computational Linguistics. Bofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Kemin...

arXiv 2024
[5]

Preprint, arXiv:2501.04519

rstar-math: Small llms can master math rea- soning with self-evolved deep thinking. Preprint, arXiv:2501.04519. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: In- centivizing reasoning capability in llms via reinforce- ment learning. arXiv preprint arXiv:2501...

arXiv 2025
[6]

In The Twelfth Inter- national Conference on Learning Representations

Let’s verify step by step. In The Twelfth Inter- national Conference on Learning Representations. Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. 2023. Deductive verification of chain-of-thought reasoning. In Thirty-seventh Conference on Neural Information Processing Systems. Chris Yuhao Liu, Liang Zeng, Jiacai Li...

arXiv 2023
[7]

In The Thirty-eighth Annual Conference on Neural Information Processing Systems

Rule based rewards for language model safety. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. OpenAI. 2024a. Hello gpt-4o. OpenAI. 2024b. New embedding models and api up- dates. Accessed: 2025-02-13. OpenAI. 2024c. Openai o1-mini: Advancing cost- efficient reasoning. OpenAI. 2025a. Introducing deep research. OpenAI. 2025b....

work page 2025
[9]

Jonathan Uesato, Nate Kushman, Ramana Kumar, Fran- cis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins

From live data to high-quality benchmarks: The arena-hard pipeline. Jonathan Uesato, Nate Kushman, Ramana Kumar, Fran- cis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process- and outcome- based feedback. Preprint, arXiv:2211.14275. Jonathan Uesato, Nate Kushman, Ramana Kumar, H...

arXiv 2022

Show all 24 references

[10]

Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui

Solving math word problems with process- based and outcome-based feedback. Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024a. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Proceed- i...

2023 arXiv
[12]

A Dataset Details In our experiments, we selected seven datasets en- compassing a wide variety of task types that require intricate and complex reasoning by the models

JudgeLM : Fine-tuned large language models are scalable judges. A Dataset Details In our experiments, we selected seven datasets en- compassing a wide variety of task types that require intricate and complex reasoning by the models. De- tailed information on the sample sizes, ...

2023
[13]

Step1” and “Step2

dataset to build the reward trees. B Experiment Details B.1 Implementation Details In the process, we segment steps using newline characters or explicit labels such as “Step1” and “Step2.” For each step, we construct positive and negative label pairs based on the reference ans...

2025
[18]

Assistant B is slightly better: [[B>A]]
[19]

My final verdict is tie: [[A=B]]

Assistant B is significantly better: [[B»A]] Example output: "My final verdict is tie: [[A=B]]". Table 15: Arena-Hard samples. Prompt: You are given a reward criterion used to evaluate a task or output from a model. Your task is to categorize the quality of this reward criteri...
[20]

Hawaii Five-O

In the ChemistryQA scenario, the reasoning process is more complex, which can exceed the model’s context window. Additionally, we observe that performance gains gradually diminish. There- fore, considering the cost overhead, we set µ = 20. 0.05 0.15 0.25 0.35 0.45 0.55 Merge H...

2024
[21]

Verify its correctness based on physics principles and equations

Carefully read and analyze the step. Verify its correctness based on physics principles and equations
[22]

If the step is correct, label it as Correct
[23]

Then modify it to the correct version

If the step is incorrect, label it as Incorrect, and provide a brief explanation of why it is wrong. Then modify it to the correct version
[24]

If you are unsure whether the step is incorrect, label it as Suspicious and provide a brief explanation for your decision
[25]

Good”, “Ordinary

Ensure that all modifications are clearly stated and the rationale for the correction is clear. Table 23: Instructions for annotators to modify incorrect steps and provide the correct version. Instruction: For each step in the problem and solution analysis, please evaluate whe...
[1997]

Data mining and knowledge discovery, 1:141–182

Birch: A new data clustering algorithm and its applications. Data mining and knowledge discovery, 1:141–182. Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jin- gren Zhou, and Junyang Lin. 2025. The lessons of developing process rewa...

2025 arXiv
[2021]

Preprint, arXiv:2110.14168

Training verifiers to solve math word prob- lems. Preprint, arXiv:2110.14168. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Ha...

arXiv
[2023]

In Thirty-seventh Conference on Neural Information Processing Sys- tems

Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Sys- tems. Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, a...

2025 arXiv
[2024]

In The Twelfth International Con- ference on Learning Representations

Safe RLHF: Safe reinforcement learning from human feedback. In The Twelfth International Con- ference on Learning Representations. Shihan Dou, Yan Liu, Haoxiang Jia, Enyu Zhou, Limao Xiong, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, R...
[2025]

Preprint, arXiv:2502.01456

Process reinforcement through implicit re- wards. Preprint, arXiv:2502.01456. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Assistant A is significantly better: [[A»B]]

work page

[2] [2]

Assistant A is slightly better: [[A>B]]

[3] [3]

Tie, relatively the same: [[A=B]]

[4] [4]

In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 4571–4585, Bangkok, Thailand

StepCoder: Improving code generation with reinforcement learning from compiler feedback. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 4571–4585, Bangkok, Thailand. Association for Computational Linguistics. Bofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Kemin...

arXiv 2024

[5] [5]

Preprint, arXiv:2501.04519

rstar-math: Small llms can master math rea- soning with self-evolved deep thinking. Preprint, arXiv:2501.04519. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: In- centivizing reasoning capability in llms via reinforce- ment learning. arXiv preprint arXiv:2501...

arXiv 2025

[6] [6]

In The Twelfth Inter- national Conference on Learning Representations

Let’s verify step by step. In The Twelfth Inter- national Conference on Learning Representations. Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. 2023. Deductive verification of chain-of-thought reasoning. In Thirty-seventh Conference on Neural Information Processing Systems. Chris Yuhao Liu, Liang Zeng, Jiacai Li...

arXiv 2023

[7] [7]

In The Thirty-eighth Annual Conference on Neural Information Processing Systems

Rule based rewards for language model safety. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. OpenAI. 2024a. Hello gpt-4o. OpenAI. 2024b. New embedding models and api up- dates. Accessed: 2025-02-13. OpenAI. 2024c. Openai o1-mini: Advancing cost- efficient reasoning. OpenAI. 2025a. Introducing deep research. OpenAI. 2025b....

work page 2025

[8] [9]

Jonathan Uesato, Nate Kushman, Ramana Kumar, Fran- cis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins

From live data to high-quality benchmarks: The arena-hard pipeline. Jonathan Uesato, Nate Kushman, Ramana Kumar, Fran- cis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process- and outcome- based feedback. Preprint, arXiv:2211.14275. Jonathan Uesato, Nate Kushman, Ramana Kumar, H...

arXiv 2022