Pith. sign in

REVIEW 5 major objections 9 minor 7 cited by

Reward Reasoning Model

T0 review · 5 major / 9 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Chain-of-thought reward models beat static preference scorers

desk verdict A solid empirical demonstration that reasoning before reward judgment helps, with a good same-data control, but the headline PPE numbers carry a real train/eval overlap risk that needs auditing before they are taken at face value. read the letter →

arxiv 2505.14674 v1 pith:CH2JSDHC submitted 2025-05-20 cs.CL

Jiaxin Guo , Zewen Chi , Li Dong , Qingxiu Dong , Xun Wu , Shaohan Huang , Furu Wei This is my paper
classification cs.CL
keywords rewardmodelschain-of-thoughtreasoningreinforcementlearningtest-timecomputescalingpreferencemodelingLLM-as-a-judgebest-of-NinferenceGRPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that reward modeling should be treated as a reasoning task rather than a scoring task. It introduces Reward Reasoning Models, which think out loud before declaring which of two responses is better, and trains them with reinforcement learning using only the binary correctness of the final preference — no hand-written reasoning traces. The reported results show these models outperform scalar reward models and standard LLM judges on reward-modeling benchmarks across chat, reasoning, safety, and human-preference domains, and that they convert additional test-time compute into better judgments. If this holds, alignment pipelines could replace fixed scalar reward heads with reasoning policies whose compute budget can be tuned per query.

What carries the argument

The central object is the Reward Reasoning Model: a decoder-only language model that ingests a query and exactly two candidate responses, autoregressively writes a free-form chain of thought, and then commits to '\boxed{Assistant 1}' or '\boxed{Assistant 2}'. Training uses the Reward Reasoning via Reinforcement Learning framework: group relative policy optimization (GRPO) maximizes a rule-based reward that is +1 when the final choice matches the ground-truth preference and −1 otherwise, so no reasoning traces are ever shown to the model. To reward more than two responses, the paper adds two aggregation schemes — round-robin ELO rating and a knockout tournament — and combines either with majority voting over repeated samples to consume additional test-time compute.

What would settle it

Relabel a random sample of the synthesized training pairs with human annotators using the same forced-choice format; if human–GPT-4o agreement is near chance on the pairs the model is trained to get right, the rule-based reward is not a faithful preference proxy. Also compute exact and near-duplicate overlap between the training queries and the evaluation prompts; substantive overlap would make the reported benchmark gaps uninformative.

Watch

Extended reading notes

Core claim

The central claim is that injecting a deliberate reasoning phase before a pairwise preference judgment turns a reward model into a better judge, and that this reasoning ability can be induced by reinforcement learning with a rule-based reward that only checks the final choice. Concretely, the paper reports that a 32B parameter RRM reaches 91.2 overall on the RewardBench suite, an average of 77.4 (rising to 79.7 with voting@5) on the Preference Proxy Evaluations best-of-N tasks, and accuracy of 81.3 on MMLU-Pro, 95.4 on MATH, and 68.4 on GPQA in binary preference classification with voting@5, exceeding a 70B instruction-tuned LLM. The paper also shows that reward accuracy improves as the number of pairwise comparisons grows and as the thinking token budget grows, and that the trained model's reasoning exhibits more transition, reflection, and comparison patterns than the base model.

Load-bearing premise

The entire training signal and every headline number rest on the assumption that the GPT-4o-generated preference labels and the rule-based verifier labels used for training are unbiased proxies for what human raters would prefer, and that none of the training queries overlap with the RewardBench, PandaLM, or PPE evaluation sets.

Editorial extensions

If this is right

  • Reward-guided best-of-N inference selects better responses than scalar reward models and prompted frontier LLMs on reasoning-heavy benchmarks, with the largest RRM beating a 70B instruction-tuned model on binary preference classification.
  • RRM-generated rewards improve downstream post-training: reinforcement learning on unlabeled queries raises GPQA accuracy over training steps, and DPO with RRM labels outperforms DPO with labels from a frontier proprietary judge on an instruction-following benchmark.
  • Test-time compute helps in two orthogonal ways: more pairwise comparisons (parallel scaling) and a larger thinking-token budget (sequential scaling) both raise reward accuracy across model sizes.
  • The ELO aggregation is slightly more accurate than the knockout tournament, but the tournament needs only O(n) comparisons rather than O(n²), giving a practical efficiency–accuracy knob.
  • Trained RRMs shift their reasoning patterns toward transition, reflection, and comparison relative to the base model, indicating the reinforcement procedure actively reshapes evaluation behavior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If RRM-style reasoning generalizes, non-verifiable tasks that currently rely on scalar reward heads could be served by a configurable reasoning policy whose compute budget is chosen per query, making reward-guided decoding competitive with process-level supervision.
  • A natural extension not explored in the paper is a two-stage verifier that uses a cheap scalar model to filter candidates and an RRM only for the remaining hard comparisons; the reported scaling curves suggest this would preserve most of the accuracy gain at lower cost.
  • Because training rewards only the final choice, the reasoning trace itself is unconstrained; a stress test would be to mask or remove the thinking tokens at inference and measure whether accuracy collapses, which would reveal how much of the gain is genuinely due to reasoning.
  • The smaller gain from majority voting on GPQA compared with MATH hints that optimal test-time budgets should be domain-adaptive, which the paper's framework could support but does not implement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 9 minor

Summary. The paper proposes Reward Reasoning Models (RRMs), which are trained by rule-based reinforcement learning (GRPO) to emit a chain-of-thought before making a pairwise preference judgment between two responses. Training data are constructed from Skywork-Reward, Tülu 3 prompts labeled by GPT-4o, and verifiable math/reasoning sources, without explicit reasoning traces. The paper evaluates RRMs on RewardBench, PandaLM Test, and Preference Proxy Evaluations (MMLU-Pro, MATH, GPQA), reports best-of-N and binary preference gains over scalar and generative reward baselines, demonstrates RL post-training and DPO with RRM rewards, and studies parallel and sequential test-time scaling. The central claim is that reasoning before rewarding, trained without reasoning annotations, yields superior and test-time-scalable reward modeling.

Significance. If the central claim holds, RRMs offer a simple and reproducible recipe for improving reward models: initialize from a reasoning model, train with GRPO on pairwise preference accuracy, and spend more test-time compute on hard comparisons. The paper's controlled DirectJudge comparison (same base models and training data, no reasoning) is a valuable design that directly supports the specific claim that the reasoning phase, not merely the training data, drives the gains. The release of pretrained models and the demonstration of downstream RL/DPO use are additional strengths. The main caveats are evaluation-based: possible train/eval overlap with PPE and RewardBench, a data-mixture confound between the 7B and 32B models, and absent uncertainty quantification, so the significance of the headline numbers is currently bounded by those gaps.

major comments (5)
  1. [§4.1, §4.3] The training mixture described in §4.1 overlaps in content with the PPE evaluation used for the headline claims in Tables 2 and 3. The 180K synthetic pairs are constructed from WebInstruct-verified, Skywork-OR1, Big-Math-RL, and DAPO-Math, and the 80K Tülu 3 prompts are sampled from a post-training instruction set; PPE evaluates reward models on MMLU-Pro, MATH, and GPQA, and RewardBench contains chat, reasoning, and safety pairs. The paper reports no deduplication or overlap audit between these training sources and the evaluation sets, and Eq. (1) rewards exactly the pairwise preference judgment that PPE measures. Because DirectJudge is trained on the same mixture, it controls for the reasoning-vs-direct comparison but cannot control for absolute contamination, so the reported superiority over GPT-4o and J1-Llama in Tables 2–3 may be inflated by memorized or near-duplicate content. Please provide an n-gram or embedding overlap analysis and rerun the headline evaluations after strict deduplication, or evaluate on a fresh held-out reward benchmark.
  2. [§4.1] RRM-7B and RRM-32B are trained on different data mixtures: RRM-32B uses a 5:1:1:1 ratio that includes the 180K synthetic verifiable pairs, while RRM-7B uses a 5:1:1 ratio that omits them. This confounds model capacity with training data throughout Tables 1–3 and Figures 5–6, and it is not clear whether DirectJudge-7B and DirectJudge-32B also differ in this way. Please train both sizes on the identical mixture, or report a same-mixture ablation, so that the scaling and size comparisons isolate the intended variable.
  3. [§4.5.2, Appendix C] The sequential test-time scaling experiment is calibrated and evaluated on the same benchmark: Appendix C sets the post-thinking budget to 100 tokens using the measured post-thinking token-length distribution of RRM-32B on RewardBench, and §4.5.2 then reports longer-thinking gains on RewardBench. This is a form of evaluation-protocol tuning on the test set; the chosen budget can affect how often reasoning is truncated and therefore the apparent benefit of longer thinking. Please validate the sequential-scaling conclusion on a held-out set, or report results across a range of post-thinking budgets (e.g., 50, 100, 200 tokens) to show the conclusion is robust.
  4. [Tables 1–3] No variance or significance information is reported for the headline accuracy numbers, and Table 3 explicitly uses a single random permutation. Many adjacent comparisons are small (e.g., RRM-7B GPQA 49.0 vs 49.2 in Table 2; RRM-32B Chat Hard 81.1 vs 81.4 in Table 1), so the claims of consistent improvements are not statistically supported as reported. Please report standard errors over multiple permutation seeds for the PPE binary-preference and best-of-N results, and bootstrap confidence intervals or significance tests for the main RewardBench and PandaLM comparisons.
  5. [§4.1, Eq. (1)] The training framework treats GPT-4o annotations and rule-based verifier labels as ground truth for the reward in Eq. (1), but the paper provides no analysis of the quality or bias of these labels, e.g., agreement between GPT-4o and human judges on a sample of the Tülu 3 pairs. Since the stated goal is alignment with human preference, a systematic label-bias analysis, or at least a human-agreement sample, is needed to support the claim that the learned preferences generalize to human judgment rather than to GPT-4o's idiosyncrasies.
minor comments (9)
  1. [§4.2.2, Table 1] The text says RRM-32B attains an accuracy of 98.6 in the reasoning category, but the non-voting RRM-32B row in Table 1 shows 98.3; 98.6 appears in the voting@16 row. Please clarify which configuration is being cited.
  2. [Table 1] The DeepSeek-R1 row contains '73,7' with a comma instead of '73.7'; please fix the typo.
  3. [§4.2.1] The sentence 'providing robust ground truth for for evaluating alignment' has a duplicated 'for'.
  4. [§4.4.1] The word 'post-traineded' should be 'post-trained'.
  5. [Tables 2–3] The column header 'MA TH' should be 'MATH' in both tables.
  6. [§4.4.2] The Tülu dataset is cited as reference [34], but [34] is RewardBench; the Tülu 3 dataset is reference [33]. Please correct the citation.
  7. [§3.2] The sentence 'Unlike conventional supervised fine-tuning approaches, which relies on existing reasoning traces' has a subject-verb agreement error; 'which relies' should be 'which rely'.
  8. [Table 1] The caption states that underlined numbers indicate the second best, but the table as typeset does not show underlines; please ensure the formatting is visible.
  9. [Appendix C] The statement 'all the responses require fewer than 100 tokens' is based on the RRM-32B distribution; the corresponding distributions for RRM-7B and RRM-14B are not shown, so the fixed 100-token budget for all model sizes is not fully justified by the presented evidence.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: central claims are empirical; training/eval overlap is a contamination risk, and the sequential-scaling budget is a test-set-derived constant, neither of which reduces a result to its inputs.

full rationale

This paper makes no formal derivation whose conclusion is assumed in its inputs; it reports empirical comparisons. The training objective (Eq. 1) rewards the RRM for choosing the ground-truth-preferred response, and the evaluations measure exactly that ability, but this is standard supervised/RL alignment rather than circularity: the preference labels and evaluation benchmarks are external, and the paper's claim is that reasoning before the judgment improves the learned mapping. DirectJudge is trained on the same mixture without reasoning, so the reasoning-vs-direct gap is controlled for data identity. Two concerns are real but non-circular. First, Section 4.1 builds 180K pairs from WebInstruct-verified, Skywork-OR1, Big-Math-RL, and DAPO-Math plus 80K Tulu-3 prompts, and evaluates on MMLU-Pro/MATH/GPQA and RewardBench without any deduplication or overlap audit; this is a contamination/leakage risk that could inflate the headline gains, but it does not make any reported quantity equal to its own input by construction. Second, Appendix C calibrates the 100-token post-thinking budget by measuring RRM-32B responses on RewardBench and then Section 4.5.2 reports sequential-scaling results on RewardBench; this is a test-set-derived constant, an evaluation-protocol weakness, but longer thinking is measured empirically and the budget does not force the observed improvement. There are no load-bearing self-citations: baselines such as RM-R1, DeepSeek-GRM, Skywork-Reward, and J1 are external or concurrent work, and the base models and prompts come from outside sources. Accordingly, no step reduces to its own input by definition, and the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the validity of the preference labels used for training and on the assumption that the evaluation benchmarks are clean; the only hand-adjusted constant that affects reported results is the post-thinking budget derived from the evaluation set.

free parameters (2)
  • Post-thinking token budget = 100 tokens
    Fixed from the distribution of RRM-32B's post-thinking outputs on RewardBench (Appendix C), then applied to the same benchmark for sequential scaling experiments; this is an evaluation-set-derived constant.
  • Training data mixture ratio = 5:1:1:1 (Skywork:Tülu:GPT-4o:verifiable) for 32B; 5:1:1 for 7B
    Chosen by hand without ablation; the 7B and 32B models are trained on different mixtures, confounding size comparisons.
assumptions (3)
  • domain assumption The ground-truth labels in the training data are accurate proxies for human preference.
    Invoked in Section 4.1: GPT-4o annotations and rule-based verifiers are used as supervision; if these are biased, the model learns biased preferences.
  • domain assumption The evaluation benchmarks (RewardBench, PandaLM, PPE) are valid, uncontaminated measures of reward quality.
    Sections 4.2 and 4.3 assume these benchmarks reflect human preference and do not overlap with the training mixture (Tülu 3 prompts, Skywork-Reward, etc.).
  • standard math GRPO and the Qwen2/DeepSeek-R1-Distill base models are standard and function as described.
    The training framework relies on GRPO (verl library) and Qwen2 architecture without deriving or verifying them.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reward Reasoning Model." pith.science (2026). https://pith.science/paper/CH2JSDHC

@misc{pith2026250514674,
  author       = {Pith},
  title        = {Pith review of: Reward Reasoning Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CH2JSDHC}},
  note         = {Machine review of arXiv:2505.14674}
}
read the original abstract

Reward models play a critical role in guiding large language models toward outputs that align with human expectations. However, an open challenge remains in effectively utilizing test-time compute to enhance reward model performance. In this work, we introduce Reward Reasoning Models (RRMs), which are specifically designed to execute a deliberate reasoning process before generating final rewards. Through chain-of-thought reasoning, RRMs leverage additional test-time compute for complex queries where appropriate rewards are not immediately apparent. To develop RRMs, we implement a reinforcement learning framework that fosters self-evolved reward reasoning capabilities without requiring explicit reasoning traces as training data. Experimental results demonstrate that RRMs achieve superior performance on reward modeling benchmarks across diverse domains. Notably, we show that RRMs can adaptively exploit test-time compute to further improve reward accuracy. The pretrained reward reasoning models are available at https://huggingface.co/Reward-Reasoning.

Figures

Figures reproduced from arXiv: 2505.14674 by the authors.

Figure 1
Figure 1. Average accuracy of various reward models on Preference Proxy Evaluations [ [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of reward reasoning model (RRM). RRM adaptively leverages test-time compute through chain-of-though reasoning before producing rewards. outputting its final decision in the format ‘\boxed{Assistant 1}’ or ‘\boxed{Assistant 2}’. The detailed prompt template is provided in Appendix A.1. The input of RRMs is restricted to exactly two candidate responses, thereby reserving output length capacity for reward r… view at source ↗
Figure 3
Figure 3. GPQA accuracy of using RRM for RL post-training. 0 200 400 600 800 RL training steps 50 52 54 56 MMLU-Pro [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: MATH accuracy with varying number of pairwise comparisons. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Results on RewardBench vary￾ing thinking budgets. 0 200 400 600 Training steps 60 70 80 90 RewardBench (%) Chat Chat Hard Safety Reasoning Overall [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 8
Figure 8. Figure 8: Reward reasoning pattern analysis results. Compared to DeepSeek-R1-Distilled-Qwen-32B, [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Performance comparison of Skywork-Reward-Gemma-2-27B-v0.2, [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Performance comparison of Skywork-Reward-Gemma-2-27B-v0.2, [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Accuracy progression of the knockout tournament strategy on MATH as elimination [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Post-thinking token length distribution of R [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TAPAS: Throughput-adaptive Perception for Autonomous Systems

    cs.LG 2026-07 conditional novelty 6.0 of 10

    An RL-based scheduler chooses CPU/GPU/DLA clusters for each perception model to meet scene-derived frame-rate targets, reporting large energy savings, though the frame-rate targets are defined by a hand-tuned entropy mapping.

  2. AI Can Learn Scientific Taste

    cs.CL 2026-03 conditional novelty 6.0 of 10

    Reinforcement learning on citation-preference pairs teaches a model to predict which papers will be cited more and to propose ideas that LLM judges rate as likely to be cited more—but "taste" here means citation impact.

  3. OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning

    cs.CL 2025-10 reject novelty 6.0 of 10

    A tool-augmented reward model trained with GRPO on 27K synthetic pairs beats existing reward models on long-form QA judgment and improves downstream alignment.

  4. VRPRM: Process Reward Modeling via Visual Reasoning

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    VRPRM combines visual reasoning with a two-stage SFT-plus-RL strategy to deliver higher-quality process reward modeling using far less annotated data than prior non-thinking PRMs.

  5. GenSelect: A Generative Approach to Best-of-N

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Asking a reasoning LLM to compare and select the best among N math solutions at once beats pointwise scoring and majority voting on competition math benchmarks.

  6. Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings

    cs.AI 2025-05 conditional novelty 4.0 of 10

    DEEVO evolves better LLM prompts by debating outputs and selecting survivors with Elo ratings, without requiring labeled data or a hand-written fitness function.

  7. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

87 extracted references · 21 canonical work pages · cited by 7 Pith papers

  1. [1]

    GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models, 2025

    Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models, 2025. URLhttps://arxiv.org/abs/2502.17387

    arXiv 2025

  3. [3]

    Atla selene mini: A general purpose evaluation model.arXiv preprint arXiv:2501.17195, 2025

    Andrei Alexandru, Antonia Calvi, Henry Broomfield, Jackson Golden, Kyle Dai, Mathias Leys, Maurice Burger, Max Bartolo, Roman Engeler, Sashank Pisupati, et al. Atla selene mini: A general purpose evaluation model.arXiv preprint arXiv:2501.17195, 2025

    arXiv 2025

  4. [4]

    The claude 3 model family: Opus, sonnet, haiku.Claude-3 Model Card, 1:1, 2024

    AI Anthropic. The claude 3 model family: Opus, sonnet, haiku.Claude-3 Model Card, 1:1, 2024

    2024

  5. [5]

    Negar Arabzadeh, Siqing Huo, Nikhil Mehta, Qingyun Wu, Chi Wang, Ahmed Hassan Awadallah, Charles L. A. Clarke, and Julia Kiseleva. Assessing and verifying task util- ity in LLM-powered applications. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages ...

    work page doi:10.18653/v1/2024.emnlp-main.1219 2024

  6. [6]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  7. [7]

    Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    arXiv 2022

  8. [8]

    Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787, 2024

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787, 2024

    arXiv 2024

Show all 87 references
  1. [9]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  2. [10]

    Sparks of artificial general intelligence: Early experiments with GPT-4, 2023

    Sébastien Bubeck, Varun Chadrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4, 2023

    2023

  3. [11]

    Compassjudger-1: All-in-one judge model helps model evaluation and evolution.arXiv preprint arXiv:2410.16256, 2024

    Maosong Cao, Alexander Lam, Haodong Duan, Hongwei Liu, Songyang Zhang, and Kai Chen. Compassjudger-1: All-in-one judge model helps model evaluation and evolution.arXiv preprint arXiv:2410.16256, 2024. 13

    2024 arXiv

  4. [12]

    Judgelrm: Large reasoning models as a judge.arXiv preprint arXiv:2504.00050, 2025

    Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. Judgelrm: Large reasoning models as a judge.arXiv preprint arXiv:2504.00050, 2025

    2025

  5. [13]

    Seal: Steerable reasoning calibration of large language models for free

    Runjin Chen, Zhenyu (Allen) Zhang, Junyuan Hong, Souvik Kundu, and Zhangyang Wang. Seal: Steerable reasoning calibration of large language models for free. 2025. URL https: //api.semanticscholar.org/CorpusID:277741244

    2025

  6. [14]

    Rm-r1: Reward modeling as reasoning.arXiv preprint arXiv:2505.02387, 2025

    Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et al. Rm-r1: Reward modeling as reasoning.arXiv preprint arXiv:2505.02387, 2025

    2025

  7. [15]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. InProceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 4302–4310, Red Hook, NY , US...

    2017

  8. [16]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  9. [17]

    Elo.The Rating of Chessplayers, Past and Present

    Arpad E. Elo.The Rating of Chessplayers, Past and Present. Arco Pub., New York, 1978. ISBN 0668047216. URL http://www.amazon.com/Rating-Chess-Players-Past-Present/ dp/0668047216

    1978

  10. [18]

    Gonzalez, and Ion Stoica

    Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. How to evaluate reward models for RLHF. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://...

    2025

  11. [19]

    On designing effective rl reward at training time for llm reasoning.arXiv preprint arXiv:2410.15115, 2024

    Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for llm reasoning.arXiv preprint arXiv:2410.15115, 2024

    2024 arXiv

  12. [20]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023

    2023

  13. [21]

    A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  14. [22]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  15. [23]

    Mcranker: Generating diverse criteria on-the-fly to improve pointwise llm rankers

    Fang Guo, Wenyu Li, Honglei Zhuang, Yun Luo, Yafu Li, Le Yan, Qi Zhu, and Yue Zhang. Mcranker: Generating diverse criteria on-the-fly to improve pointwise llm rankers. InPro- ceedings of the Eighteenth ACM International Conference on Web Search and Data Min- ing, WSDM ’25, pag...

    2025

  16. [24]

    Skywork open reasoner series

    Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Yang Liu, and Yahui Zhou. Skywork open reasoner series. https://capricious-hydrogen-41c.notion.site/ S...

    2025

  17. [25]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (...

    2021

  18. [26]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  19. [27]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  20. [28]

    LLM-blender: Ensembling large language models with pairwise ranking and generative fusion

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-blender: Ensembling large language models with pairwise ranking and generative fusion. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61st Annual Meeting of the Association for Computational ...

    2023 doi

  21. [29]

    SWE-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/fo...

    2024

  22. [30]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  23. [31]

    Maple: Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. In2023 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 19113–19122, 2023. doi: 10.1109/CVPR52729.2023.01832

    2023

  24. [32]

    Prometheus 2: An open source language model specialized in evaluating other language models

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. In Yaser Al-Onaizan, Mohit Bansal, and ...

    2024

  25. [33]

    T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  26. [34]

    Rewardbench: Evaluating reward models for language modeling.arXiv preprint arXiv:2403.13787, 2024

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling.arXiv preprint arXiv:2403.13787, 2024

    2024 arXiv

  27. [35]

    Generative judge for evaluating alignment

    Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, hai zhao, and Pengfei Liu. Generative judge for evaluating alignment. InThe Twelfth International Conference on Learning Representations,

  28. [36]

    Gonzalez, and Ion Stoica

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline, 2024. URLhttps://arxiv.org/abs/2406.11939

    2024 arXiv

  29. [37]

    URLhttps://openreview.net/forum?id=gtkFw6sZGS

  30. [38]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004

    2004

  31. [39]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/fo...

    2024

  32. [40]

    Pairwise rm: Perform best-of-n sampling with knockout tournament.arXiv preprint arXiv:2501.13007, 2025

    Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Pairwise rm: Perform best-of-n sampling with knockout tournament.arXiv preprint arXiv:2501.13007, 2025

    2025 arXiv

  33. [41]

    Skywork-reward: Bag of tricks for reward modeling in llms.arXiv preprint arXiv:2410.18451, 2024

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms.arXiv preprint arXiv:2410.18451, 2024. 15

    2024 arXiv

  34. [42]

    General-reasoner: Advancing llm reasoning across all domains

    Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing llm reasoning across all domains. https://github.com/ TIGER-AI-Lab/General-Reasoner/blob/main/General_Reasoner.pdf, 2025

    2025

  35. [43]

    Inference-time scaling for generalist reward modeling.arXiv preprint arXiv:2504.02495, 2025

    Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling.arXiv preprint arXiv:2504.02495, 2025

    2025

  36. [44]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, an...

    2022

  37. [45]

    Generative reward models.arXiv preprint arXiv:2410.12832, 2024

    Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models.arXiv preprint arXiv:2410.12832, 2024

    2024 arXiv

  38. [46]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002

    2002

  39. [47]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  40. [48]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  41. [49]

    OffsetBias: Leveraging debiased data for tuning evaluators

    Junsoo Park, Seungyeon Jwa, Ren Meiying, Daeyoung Kim, and Sanghyuk Choi. OffsetBias: Leveraging debiased data for tuning evaluators. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1043...

    2024 doi

  42. [50]

    Bradley Knox, Chelsea Finn, and Scott Niekum

    Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, W. Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. InThe Thirty-eighth Annual Conference on Neural Information Processing Sys...

    2024

  43. [51]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openre...

    2023

  44. [52]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/ 2402.03300. 16

    2024 arXiv

  45. [53]

    The limits of automatic summarisation according to rouge

    Natalie Schluter. The limits of automatic summarisation according to rouge. InProceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 41–45. Association for Computational Linguistics, 2017

    2017

  46. [54]

    Skywork critic model se- ries

    Tu Shiwen, Zhao Liang, Chris Yuhao Liu, Liang Zeng, and Yang Liu. Skywork critic model se- ries. https://huggingface.co/Skywork, September 2024. URL https://huggingface. co/Skywork

    2024

  47. [55]

    HybridFlow: A flexible and efficient RLHF framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  48. [56]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  49. [57]

    Scaling LLM test- time compute optimally can be more effective than scaling parameters for reasoning

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test- time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.net/forum...

    2025

  50. [58]

    Secrets of rlhf in large language models part ii: Reward modeling.arXiv preprint arXiv:2401.06080, 2024

    Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling.arXiv preprint arXiv:2401.06080, 2024

    2024 arXiv

  51. [59]

    Foundational autoraters: Taming large language models for better automatic evaluation

    Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. Foundational autoraters: Taming large language models for better automatic evaluation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on ...

    2024 doi

  52. [60]

    Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization

    Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. InICLR,

  53. [61]

    Self-taught evaluators.arXiv preprint arXiv:2408.02666, 2024

    Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-taught evaluators.arXiv preprint arXiv:2408.02666, 2024

    2024 arXiv

  54. [62]

    Helpsteer: Multi-attribute helpfulness dataset for steerlm.arXiv preprint arXiv:2311.09528, 2023

    Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm.arXiv preprint arXiv:2311.09528, 2023

    2023 arXiv

  55. [63]

    URLhttps://openreview.net/forum?id=5Nn2BLV7SB

  56. [64]

    Reinforcement learning for reasoning in large language models with one training example

    Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. Reinforcement learning for reasoning in large language models with one training example. 202...

    2025

  57. [65]

    J1: Incentivizing thinking in llm-as-a-judge via reinforcement learning.arXiv preprint arXiv:2505.10320, 2025

    Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swar- nadeep Saha. J1: Incentivizing thinking in llm-as-a-judge via reinforcement learning.arXiv preprint arXiv:2505.10320, 2025. 17

    2025

  58. [66]

    Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev

    Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. InThe Thirty-eight Conference on Neu- ral Information Pr...

    2024

  59. [67]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in...

    2022

  60. [68]

    Inference scaling laws: An empirical analysis of compute-optimal inference for LLM problem-solving

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for LLM problem-solving. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.net/for...

    2025

  61. [69]

    Metametrics: Calibrating metrics for generation tasks using human preferences

    Genta Indra Winata, David Anugraha, Lucky Susanto, Garry Kuwanto, and Derry Tanti Wijaya. Metametrics: Calibrating metrics for generation tasks using human preferences. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/foru...

    2025

  62. [70]

    Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge.arXiv preprint arXiv:2407.19594, 2024

    Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge.arXiv preprint arXiv:2407.19594, 2024

    2024 arXiv

  63. [71]

    Learning LLM-as-a-judge for preference alignment

    Ziyi Ye, Xiangsheng Li, Qiuchi Li, Qingyao Ai, Yujia Zhou, Wei Shen, Dong Yan, and Yiqun LIU. Learning LLM-as-a-judge for preference alignment. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum? id=HZVIQE1MsJ

    2025

  64. [72]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  65. [73]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122, 2024

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via se...

    2024 arXiv

  66. [74]

    Self-rewarding language models

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    2024

  67. [75]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wa...

    2025 arXiv

  68. [76]

    Self-generated critiques boost reward modeling for language models

    Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, Melanie Kambadur, Dhruv Mahajan, and Rui Hou. Self-generated critiques boost reward modeling for language models. In Luis Chiruzzo, Alan...

    2025

  69. [77]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. InThirty-seventh Conference on Neural Inform...

    2023

  70. [78]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. InThe 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24, 2024. URL https://openreview.net/ forum?id=CxHRoTLmPX

    2024

  71. [79]

    The lessons of developing process reward models in mathematical reasoning.arXiv preprint arXiv:2501.07301, 2025

    Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning.arXiv preprint arXiv:2501.07301, 2025. 18

    2025 arXiv

  72. [80]

    JudgeLM: Fine-tuned large language mod- els are scalable judges

    Lianghui Zhu, Xinggang Wang, and Xinlong Wang. JudgeLM: Fine-tuned large language mod- els are scalable judges. InThe Thirteenth International Conference on Learning Representations,

  73. [81]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  74. [82]

    A comprehensive survey of reward models: Taxonomy, applications, challenges, and future.arXiv preprint arXiv:2504.12328, 2025

    Jialun Zhong, Wei Shen, Yanzeng Li, Songyang Gao, Hua Lu, Yicheng Chen, Yang Zhang, Wei Zhou, Jinjie Gu, and Lei Zou. A comprehensive survey of reward models: Taxonomy, applications, challenges, and future.arXiv preprint arXiv:2504.12328, 2025

    2025 arXiv

  75. [85]

    Partially Adhered

    Instruction Adherence:\n - Fully Adhered: The response fully complies with all instructions and requirements of the question.\n - Partially Adhered: The response meets most of the instructions but has some omissions or misunderstandings.\n - Basically Adhered: The response mee...

  76. [86]

    Useful but Incomplete

    Usefulness:\n - Highly Useful: The response provides comprehensive and accurate information, fully addressing the issue.\n - Useful but Incomplete: The response provides some useful information, but lacks details or accuracy.\n - Limited Usefulness: The response offers little ...

  77. [87]

    Not Detailed

    Level of Detail:\n - Very Detailed: The response includes ample details covering all aspects of the issue.\n - Detailed but Slightly Lacking: The response is fairly detailed but misses some important details.\n - Basically Detailed: The response provides some details but is no...

  78. [2024]

    doi: 10.18653/v1/2024.emnlp-main.248

    Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.248. URLhttps://aclanthology.org/2024.emnlp-main.248/

    2024 doi

  79. [2025]

    \boxed{Assistant 1}

    URLhttps://openreview.net/forum?id=xsELpEPn4A. 19 A RRMImplementation Details A.1 Prompt Template Here we provide the detailed prompt template used to guide our Reward Reasoning Models (RRMs) in performing systematic comparative analysis between two responses. We adapt the pro...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.