Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Generative reward models that think for thousands of tokens before judging outperform Bradley-Terry and majority-voted GenRM baselines by 8% on RM-Bench.

desk verdict A solid 8B GenRM recipe showing long internal CoT beats vertical scaling on RM-Bench; the pairwise RLHF side is under-derived but not the load-bearing claim. read the letter →

arxiv 2505.16265 v1 pith:LLEWARJR submitted 2025-05-22 cs.LG

Ilgee Hong , Changlong Yu , Liang Qiu , Weixiang Yan , Zhenghao Xu , Haoming Jiang , Qingru Zhang , Qin Lu
show 3 more authors
Xin Liu Chao Zhang Tuo Zhao
This is my paper
classification cs.LG
keywords generativerewardmodelslong-horizonreasoningchain-of-thoughtRLHFGRPOpairwisepreferenceoptimizationmodelingRM-Bench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that reward modeling benefits more from a single long, self-guided reasoning trajectory than from many shallow ones. It introduces Think-RM, which trains an 8B instruct model to emit an internal chain-of-thought of thousands of tokens before giving a pairwise preference verdict, using only 6K training examples. The training recipe is two-stage: supervised warm-up on the longest correct traces generated by a stronger reasoning model, then rule-based GRPO that rewards only whether the final verdict matches the human label and shortens overly verbose reasoning. On RM-Bench, binary Think-RM reaches 75.06 average accuracy, beating a Bradley-Terry reward model (68.27) and the best 16-way majority-voted CoT-GenRM (68.55) by 8%. The paper further claims that pairwise RLHF can consume these preferences directly, without converting them into pointwise rewards.

What carries the argument

The carrying object is the internal thinking process: an unconstrained, self-generated chain-of-thought that can run thousands of tokens and exhibit capabilities like self-reflection, hypothetical reasoning, and divergent reasoning, as opposed to the short, externally structured rationales used by prior GenRMs. It is elicited by warm-up SFT on the longest correct reasoning trace per training instance, produced by a stronger pretrained reasoning model, and then refined by rule-based GRPO whose reward is 1.0 for an exact verdict, 0.5 for a correct preference direction in the multiclass case, and 0 otherwise. The second load-bearing component is the pairwise preference strength matrix D, a skew-symmetric matrix built from GenRM judgments between every pair of responses in a group, with binary judgments weighted by 1/|R| and multiclass judgments weighted by their strength. From this matrix the paper computes group-relative advantages directly as $\hat{A}_i = \frac{\sum_{j=1}^G d_{ij}}{\sqrt{\frac{G}{2(G-1)}\,\sum_{i,j} d_{ij}^2 + G\epsilon}}$, which reduces to standard GRPO advantages when $d_{ij}$ equals the difference of pointwise rewards $r_i - r_j$. This matrix is what lets pairwise RLHF optimize a policy without ever recovering scalar rewards.

What would settle it

Train the pairwise RLHF pipeline on synthetic preference data generated from known pointwise scores with injected non-transitive noise (rock-paper-scissors triples); if the proposed advantage estimator does not recover the correct gradient direction, measured by final policy win rate against the true score, the central assumption that pairwise strengths aggregate like reward differences fails.

Watch

Extended reading notes

Core claim

Think-RM claims that a generative reward model's judgment quality is set by the depth of a single internal reasoning trajectory, not by the number of trajectories sampled. Starting from Llama-3.1-8B-Instruct, the framework trains the model to produce free-form internal thinking over thousands of tokens before outputting a binary or multiclass preference. Warm-up SFT fits the model to the longest correct QwQ-32B trace for each HelpSteer2-Preference instance; rule-based GRPO then rewards only the correctness of the final verdict, which both raises accuracy and shortens responses. On RM-Bench, the binary Think-RM scores 75.06 average accuracy against 68.55 for the best 16-way majority-voted CoT-GenRM and 68.27 for a BT RM, an 8% margin, and both binary and multiclass Think-RMs also lead on RewardBench. For policy optimization, the paper introduces a pairwise RLHF pipeline: a skew-symmetric preference strength matrix D is assembled from GenRM pairwise evaluations, and group-relative advantages are estimated from that matrix directly, so no pointwise reward model is needed.

Load-bearing premise

The pairwise RLHF pipeline assumes the model's pairwise comparisons can be averaged into a group-relative advantage signal exactly as if each comparison value were a difference of an underlying per-response quality score, even though pairwise judgments need not be transitive or correspond to any single scalar score.

Editorial extensions

If this is right

  • If correct, long-horizon internal thinking lets a small reward model match or beat much larger or heavily sampled reward pipelines on reasoning-intensive evaluation, without requiring human-written rationales.
  • The rule-based RL stage both improves accuracy and cuts average response length, so the depth gain does not come with unbounded inference cost.
  • Pairwise RLHF removes the pointwise-reward conversion step, so any generative judge that emits relative judgments can drive policy optimization; policies trained this way achieve higher length-controlled and overall win rates on AlpacaEval2 than pointwise RLHF with a BT RM.
  • The same SFT-plus-rule-based-RL recipe can be reused for other evaluative heads, because the task instruction and evaluation criteria are modular and the warm-up traces are task-agnostic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension not tested here is an equal-token or equal-cost comparison across difficulty strata: the 1/|R| confidence weight suggests the authors expect long, uncertain chains to be down-weighted, but a learned confidence signal could improve advantage estimates when reasoning length does not track ambiguity.
  • If pairwise RLHF is stable without pointwise rewards, alignment data collection could shift from scalar ratings to relative judgments, which are cheaper and more natural for annotators; the non-transitivity caveat implies this will need a consistency filter on preference triples.
  • The warm-up traces come from a stronger reasoning model, so a policy-distillation interpretation would predict similar gains in other verifiable-answer tasks where long internal search can be distilled into a smaller model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Think-RM, a generative reward model (GenRM) that is trained to produce long internal chain-of-thought reasoning before emitting a pairwise preference. Training comprises (i) SFT on the longest correct CoT trajectories generated by QwQ-32B from HelpSteer2-Preference, and (ii) rule-based GRPO that rewards exact and direction-correct preference outputs. The authors also introduce a pairwise RLHF pipeline that converts pairwise GenRM evaluations directly into GRPO advantages without an intermediate pointwise reward. Experiments with Llama-3.1-8B-Instruct as backbone show that Think-RM outperforms Bradley-Terry RMs and CoT-GenRMs, with and without vertical inference-time scaling, on HelpSteer2/3, RewardBench, and especially RM-Bench, and that pairwise RLHF with GenRMs improves end-policy performance on HH-RLHF/AlpacaEval2 relative to pointwise RLHF with a BT RM.

Significance. The central empirical result is substantial and well-tabulated: a 10B-scale RM trained on 6K examples with long-horizon thinking outperforms BT RMs and vertically scaled CoT-GenRMs by a wide margin on reasoning-heavy benchmarks (e.g., binary Think-RM SFT+RL reaches 75.06 average on RM-Bench vs 68.27 for BT RM and 68.55 for the strongest vertically scaled baseline). The paper ships code, data, and models, and the recipe (long CoT warm-up + rule-based RL) is simple and reproducible. These strengths make the paper a valuable contribution even before the pairwise RLHF pipeline is fully validated. The pairwise RLHF contribution, however, currently rests on an unproven equivalence between pairwise preference strengths and pointwise reward differences, which the authors need to address before the broader 'new paradigm for preference-based policy optimization' claim can be accepted.

major comments (3)
  1. [Section 3.4, Eq. (2) and Eq. (3)] The pairwise advantage estimator is derived by substituting d_ij = r_i - r_j throughout. Specifically, the identities sum_j d_ij = G(r_i - rbar) and sum_{i,j} d_ij^2 = 2G(G-1) sigma_r^2 are only valid when D is a skew-symmetric difference matrix of a single pointwise reward. A trained pairwise GenRM is not constrained to produce transitive, reward-difference-compatible evaluations; pairwise comparisons can be cyclic, and the binary confidence weight 1/|R| is an untested heuristic. When D is not a difference matrix, the normalized row sums in Eq. (2) are a Borda-like aggregate whose alignment with the true policy-gradient direction is unknown. Please either prove that the GenRM training objective enforces the difference-matrix structure, or add a synthetic experiment with known pointwise ground-truth rewards to validate the recovered advantages, or restrict the method to a pointwise output head.
  2. [Section 3.4, Eq. (3)] There is a notation/scaling inconsistency in the denominator of the proposed advantage estimator. The formula shows sqrt( G/(2(G-1)) * sum_{i,j} d_ij^2 + G*epsilon ). After substituting d_ij = r_i - r_j, this simplifies to (r_i - rbar)/(sigma_r + epsilon/G) in the limit of small epsilon, not the stated standard GRPO advantage (r_i - rbar)/(sigma_r + epsilon). Please correct the epsilon scaling or clarify the definition; as written, the displayed equality to standard GRPO is not exact.
  3. [Section 4.2.2, Table 3] The text states that 'Think-RMs trained with both SFT and RL substantially outperform their SFT-only counterparts' across benchmarks, but the binary RM-Bench average column shows the opposite: Think-RM (SFT) averages 75.19 and Think-RM (SFT + RL) averages 75.06. This contradicts the stated conclusion and should be addressed, for example by reporting per-domain bootstrap intervals or explicitly acknowledging that the RL stage improves length efficiency without improving average RM-Bench accuracy in the binary setting.
minor comments (5)
  1. [Section 3.2, Eq. (1)] The SFT loss is written as -log G_theta(s | ...) - log G_theta(R | ...), but a single autoregressive GenRM generates (R, s) jointly. Please clarify whether this is a sum of two per-token loss terms on the full sequence or a genuine factorization, since the factorization as written does not correspond to the standard joint probability of the generated sequence.
  2. [Abstract, Section 4.2.2] The abstract reports an 8% improvement over BT RMs and vertically scaled GenRMs. Table 3 shows about 6.8 percentage points of absolute improvement on the binary RM-Bench average; please state whether the 8% figure is relative or absolute to avoid ambiguity.
  3. [Section 4.3, Figure 3] The ablation compares only longest versus shortest CoT trajectories for warm-up SFT. A comparison with random-correct selection or majority-agreement selection would better isolate the effect of length from the effect of correctness filtering.
  4. [Tables 1-4] All evaluation numbers come from single runs without confidence intervals or multiple seeds. Given the small training set (6K/4K) and the variability of RL training, please report at least the number of evaluation samples per domain and, where feasible, repeated-run variance.
  5. [Section 3.4] The binary confidence weight 1/|R| appears without an ablation or a theoretical justification. At minimum, please provide a sensitivity analysis varying the confidence weighting, or explain why longer reasoning chains should generally reduce confidence.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity; central RM accuracy results are benchmarked on held-out labels, and the §3.4 pairwise advantage formula is a conditional derivation, not a definitional loop.

full rationale

Think-RM’s main accuracy claim (Tables 1–3) is not circular: the model is trained on HelpSteer2-Preference with QwQ-32B long CoT traces filtered by ground-truth preference labels, then evaluated on held-out RewardBench and RM-Bench, whose labels are never used in SFT or the rule-based RL reward. The longest-vs-shortest CoT ablation (Fig. 3) is a controlled comparison, not a fitted prediction. The only self-citation in the paper is reference [21] (Yu et al., which includes co-author Chao Zhang), used in the related-work survey [18–22]; it is background and not load-bearing, so it does not raise circularity. The pairwise RLHF advantage in §3.4 is the one derivation that is conditional: the proposed A_i equals standard GRPO only when the pairwise matrix satisfies d_ij = r_i - r_j, as shown by the paper’s own equations. Since D is actually defined from GenRM preference outputs (-s or -s_tilde/|R|), this equality is an unverified modeling assumption (non-transitivity and the arbitrary 1/|R| confidence weight are not analyzed), not an equivalence forced by definition. That is a correctness/robustness risk, not circularity; the method is not fitting the benchmark labels and does not redefine the pointwise rewards it claims to avoid.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on several domain assumptions (label quality, benchmark validity, teacher quality) and four hand-chosen design parameters (confidence weight, M, longest selection, partial credit). No genuinely new physical or conceptual entities are introduced. The pairwise advantage assumption is the most fragile because it underpins the secondary RLHF claim and is not fully derived.

free parameters (4)
  • Confidence weight 1/|R| for binary pairwise advantages = 1/length
    Introduced in Section 3.4 without ablation or theoretical justification. It scales each pairwise comparison by the inverse reasoning length, changing the contribution of each comparison to the policy gradient. If this weight is miscalibrated, the pairwise RLHF results in Table 4 would change.
  • M, number of QwQ-32B CoT trajectories per training instance = 10
    Chosen in Section 4.1 without ablation. M determines how many trajectories are generated and, combined with the correctness filter, determines the resulting 6K (binary) and 4K (multiclass) training set sizes.
  • CoT selection rule: longest correct trajectory = longest
    Section 4.3 ablates longest versus shortest and shows the choice affects both accuracy and output length. The rule is a heuristic proxy for reasoning quality, not a principled criterion.
  • Rule-based RL partial credit 0.5 for matching sign in multiclass = 0.5
    Section 3.3 assigns partial reward for correct preference direction but wrong magnitude; no sensitivity analysis is provided for this value.
assumptions (4)
  • domain assumption HelpSteer2-Preference labels are reliable ground-truth preferences for training and evaluation.
    Used throughout Section 4.1 as both the supervision source for all models and the in-distribution test labels. If these labels contain noise or bias, all reported accuracies inherit it.
  • domain assumption RM-Bench and RewardBench preferences are valid proxies for real-world reward quality.
    The state-of-the-art claim in Section 4.2.2 rests on accuracy on these benchmarks. Neither benchmark is ground truth for downstream deployment quality, and high benchmark accuracy does not guarantee better alignment.
  • domain assumption QwQ-32B's long, correct CoT trajectories are a suitable teacher for reward-model reasoning.
    Section 3.2 uses these traces as SFT targets. If QwQ-32B's reasoning style transfers biases or if the correctness filter does not catch subtle errors, Think-RM inherits those problems.
  • ad hoc to paper Pairwise preference matrix D can substitute for pointwise rewards in GRPO advantage estimation.
    Section 3.4 claims a derivation of the advantage formula, but the reduction to standard GRPO requires dij = ri - rj, which is not established for binary preferences with the 1/|R| weighting. This is a modeling choice made specifically for this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models." pith.science (2026). https://pith.science/paper/LLEWARJR

@misc{pith2026250516265,
  author       = {Pith},
  title        = {Pith review of: Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LLEWARJR}},
  note         = {Machine review of arXiv:2505.16265}
}
read the original abstract

Reinforcement learning from human feedback (RLHF) has become a powerful post-training paradigm for aligning large language models with human preferences. A core challenge in RLHF is constructing accurate reward signals, where the conventional Bradley-Terry reward models (BT RMs) often suffer from sensitivity to data size and coverage, as well as vulnerability to reward hacking. Generative reward models (GenRMs) offer a more robust alternative by generating chain-of-thought (CoT) rationales followed by a final reward. However, existing GenRMs rely on shallow, vertically scaled reasoning, limiting their capacity to handle nuanced or complex (e.g., reasoning-intensive) tasks. Moreover, their pairwise preference outputs are incompatible with standard RLHF algorithms that require pointwise reward signals. In this work, we introduce Think-RM, a training framework that enables long-horizon reasoning in GenRMs by modeling an internal thinking process. Rather than producing structured, externally provided rationales, Think-RM generates flexible, self-guided reasoning traces that support advanced capabilities such as self-reflection, hypothetical reasoning, and divergent reasoning. To elicit these reasoning abilities, we first warm-up the models by supervised fine-tuning (SFT) over long CoT data. We then further improve the model's long-horizon abilities by rule-based reinforcement learning (RL). In addition, we propose a novel pairwise RLHF pipeline that directly optimizes policies using pairwise preference rewards, eliminating the need for pointwise reward conversion and enabling more effective use of Think-RM outputs. Experiments show that Think-RM achieves state-of-the-art results on RM-Bench, outperforming both BT RM and vertically scaled GenRM by 8%. When combined with our pairwise RLHF pipeline, it demonstrates superior end-policy performance compared to traditional approaches.

Figures

Figures reproduced from arXiv: 2505.16265 by the authors.

Figure 1
Figure 1. Examples of advanced reasoning abilities enabled by Think-RM. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the Think-RM training framework. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of two CoT filtering strategies for warm-up data selection. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VRPRM: Process Reward Modeling via Visual Reasoning

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    VRPRM combines visual reasoning with a two-stage SFT-plus-RL strategy to deliver higher-quality process reward modeling using far less annotated data than prior non-thinking PRMs.

Reference graph

Works this paper leans on

49 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  2. [2]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  3. [3]

    Self-instruct: Aligning language models with self-generated in- structions

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated in- structions. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, 2023

    2023

  4. [4]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  5. [5]

    Solving math word problems with process-and outcome-based feedback.arXiv preprint arXiv:2211.14275, 2022

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback.arXiv preprint arXiv:2211.14275, 2022

    arXiv 2022

  6. [6]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  7. [7]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations.arXiv preprint arXiv:2312.08935, 2023

    Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations.arXiv preprint arXiv:2312.08935, 2023. 10

    arXiv 2023

  8. [8]

    Teaching large language models to reason with reinforcement learning.arXiv preprint arXiv:2403.04642, 2024

    Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning.arXiv preprint arXiv:2403.04642, 2024

    arXiv 2024

Show all 49 references
  1. [9]

    Beavertails: Towards improved safety alignment of LLM via a human-preference dataset

    Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. InThirty-seventh Conference on Neural Information Processing Syste...

    2023

  2. [10]

    Safe rlhf: Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  3. [11]

    Rule based rewards for language model safety.arXiv preprint arXiv:2411.01111, 2024

    Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety.arXiv preprint arXiv:2411.01111, 2024

    2024 arXiv

  4. [12]

    Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    2017

  5. [13]

    Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize from human feedback. Advances in Neural Information Processing Systems, 2020

    2020

  6. [14]

    Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking.arXiv preprint arXiv:2312.09244, 2023

    Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvi- jotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking.arXiv preprint arXiv:23...

    2023 arXiv

  7. [15]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. InInternational Conference on Machine Learning, pages 10835–10866. PMLR, 2023

    2023

  8. [16]

    Direct language model alignment from online ai feedback.arXiv preprint arXiv:2402.04792, 2024

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexan- dre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback.arXiv preprint arXiv:2402.04792, 2024

    2024 arXiv

  9. [17]

    Rrm: Robust reward model training mitigates reward hacking

    Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, et al. Rrm: Robust reward model training mitigates reward hacking. arXiv preprint arXiv:2409.13156, 2024

    2024 arXiv

  10. [18]

    Improving reward models with synthetic critiques.arXiv preprint arXiv:2405.20850, 2024

    Zihuiwen Ye, Fraser Greenlee-Scott, Max Bartolo, Phil Blunsom, Jon Ander Campos, and Matthias Gallé. Improving reward models with synthetic critiques.arXiv preprint arXiv:2405.20850, 2024

    2024 arXiv

  11. [19]

    Critique-out-loud reward models.arXiv preprint arXiv:2408.11791, 2024

    Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. Critique-out-loud reward models.arXiv preprint arXiv:2408.11791, 2024

    2024 arXiv

  12. [20]

    Generative reward models.arXiv preprint arXiv:2410.12832, 2024

    Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models.arXiv preprint arXiv:2410.12832, 2024

    2024 arXiv

  13. [21]

    Self-generated critiques boost reward modeling for language models.arXiv preprint arXiv:2411.16646, 2024

    Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, et al. Self-generated critiques boost reward modeling for language models.arXiv preprint arXiv:2411.16646, 2024

    2024 arXiv

  14. [22]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  15. [23]

    Learning to reason with llms.OpenAI Blog, 2024

    OpenAI. Learning to reason with llms.OpenAI Blog, 2024

    2024

  16. [24]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  17. [25]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025. 11

    2025 arXiv

  18. [26]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  19. [27]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  20. [28]

    RM-bench: Benchmarking reward models of language models with subtlety and style

    Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. RM-bench: Benchmarking reward models of language models with subtlety and style. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  21. [29]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3/4):324–345, 1952

    1952

  22. [30]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  23. [31]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  24. [32]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  25. [33]

    xfinder: Robust and pinpoint answer extraction for large language models.arXiv preprint arXiv:2405.11874, 2024

    Qingchen Yu, Zifan Zheng, Shichao Song, Zhiyu Li, Feiyu Xiong, Bo Tang, and Ding Chen. xfinder: Robust and pinpoint answer extraction for large language models.arXiv preprint arXiv:2405.11874, 2024

    2024 arXiv

  26. [34]

    Chain of thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. Chain of thought prompting elicits reasoning in large language models. InNeurIPS, 2022

    2022

  27. [35]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    Xuezhi Wang, Jason Wei, Dale Schuurmans, et al. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  28. [36]

    Tree of thoughts: Deliberate problem solving with large language models.Ad- vances in neural information processing systems, 36:11809–11822, 2023

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.Ad- vances in neural information processing systems, 36:11809–11822, 2023

    2023

  29. [37]

    Introducing openai o3 and o4-mini.OpenAI Blog, 2025

    OpenAI. Introducing openai o3 and o4-mini.OpenAI Blog, 2025

    2025

  30. [38]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm.github.io/blog/qwq-32b/

    2025

  31. [39]

    Grok 3 beta — the age of reasoning agents.xAI Blog, 2025

    xAI. Grok 3 beta — the age of reasoning agents.xAI Blog, 2025

    2025

  32. [40]

    Helpsteer2-preference: Complementing ratings with prefer- ences

    Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpsteer2-preference: Complementing ratings with prefer- ences. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  33. [41]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  34. [42]

    High- dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438, 2015

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438, 2015

    2015 arXiv

  35. [43]

    Dedicated feedback and edit models empower inference- time scaling for open-ended general-domain tasks.arXiv preprint arXiv:2503.04378, 2025

    Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Daniel Egert, Ellie Evans, Hoo-Chang Shin, Felipe Soares, Yi Dong, and Oleksii Kuchaiev. Dedicated feedback and edit models empower inference- time scaling for open-ended general-domain tasks.arXiv preprint arXiv:2503.04378, 2025

    2025 arXiv

  36. [44]

    Rewardbench: Evaluating reward models for language modeling.arXiv preprint arXiv:2403.13787, 2024

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling.arXiv preprint arXiv:2403.13787, 2024

    2024 arXiv

  37. [45]

    Length-controlled alpacaeval: A simple debiasing of automatic evaluators

    Yann Dubois, Percy Liang, and Tatsunori Hashimoto. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. InFirst Conference on Language Modeling, 2024

    2024

  38. [46]

    Openrlhf: An easy-to-use, scalable and high-performance rlhf framework.arXiv preprint arXiv:2405.11143, 2024

    Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework.arXiv preprint arXiv:2405.11143, 2024. 12

    2024 arXiv

  39. [47]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  40. [48]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  41. [49]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 13 A Task Instruction Template Binary Think-RM System: You are an impartial judge, tasked with evaluating the quality of the two AI assistants’ responses to the conte...

    2017 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.