Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reward noise up to 40% flips barely hurts trained reasoning, and rewarding only phrase occurrences like 'first, I need to' reaches 70.21% on MATH-500, close to the 75.85% from exact verification.

desk verdict The robustness-to-flips result is real and worth engaging; the RPR/no-new-knowledge interpretation needs a leakage-free control before it can carry the paper. read the letter →

arxiv 2505.22653 v1 pith:CQ5BEX5V submitted 2025-05-28 cs.CL

Ang Lv , Ruobing Xie , Xingwu Sun , Zhanhui Kang , Rui Yan This is my paper · ORCID
classification cs.CL
keywords reinforcementlearningrewardnoisereasoningpatternLLMpost-trainingmodelcalibrationmathopen-endedtasksQwen-2.5
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what happens when the rewards used to post-train LLMs for reasoning are wrong, and what that reveals about where reasoning ability comes from. Its main claim is that models with strong pretrained reasoning can absorb large amounts of reward noise—flipping 40% of math rewards still yields 72.02% on MATH-500 versus 75.85% noiseless—and that the real driver of RL gains is not correctness feedback but the reinforcement of reasoning patterns the model already produced in pretraining. The decisive evidence is an experiment that rewards only the appearance of phrases like "first, I need to" and never verifies the answer, which reaches 70.21% accuracy. On open-ended tasks, rewarding those same reasoning patterns as a small add-on to a noisy reward model narrows the gap between a 65%-accurate and an 85%-accurate reward model from 25 net win points to 8. The practical consequence is that building ever-more-accurate reward models is not the only lever: strengthening pretrained reasoning is the complementary lever, since post-training is largely activation, not acquisition.

What carries the argument

Reasoning Pattern Reward (RPR), a rule-based reward that scores an output by how many of 40 pre-selected high-frequency reasoning phrases (e.g., "first, I need to", "we know that", "let me try") appear in the reasoning chain inside the <think> tags, scaled by $r = 1/40$, clipped to 1, and paired with an n-gram repetition penalty to stop the model from repeating phrases. RPR is the load-bearing instrument: because it gives no correctness signal, any downstream accuracy it produces must come from reinforcing reasoning behavior already latent in the pretrained model. In the calibration experiments the same RPR score is added to noisy reward-model scores below threshold $\tau = 0.5$ with weight $\alpha = 0.1$, compensating false negatives.

What would settle it

Run the RPR-only experiment again, replacing the 40 reasoning phrases with 40 equally frequent function words or length-matched filler phrases; if MATH-500 accuracy holds near 70%, RPR is a surface-format reward, not evidence that reasoning patterns alone drive RL gains. A second check is to force the same phrase list into a fixed template that prevents genuine step-by-step reasoning and see whether accuracy collapses.

Watch

Extended reading notes

Core claim

The paper's central discovery is that reward quality is far less important than the strength of the model's pretrained reasoning, and that the effective job of RL post-training is to reinforce output patterns the model already knows how to produce. The decisive experiment rewards the appearance of key reasoning phrases—such as "first, I need to", "we know that", and "let me check"—and never checks whether the final answer is right; this alone takes Qwen-2.5-7B to 70.21% on MATH-500, close to the 75.85% achieved with exact correctness rewards. The same logic holds for open-ended tasks: a reward model at 75% accuracy performs about as well as one at 85%, and a phrase-based compensatory reward lifts a 65%-accurate reward model from a 25 net-win gap behind an 85% model down to 8. The authors conclude that RL post-training explores and reinforces existing reasoning patterns rather than teaching much new knowledge, which is why robustness to reward noise is a property of models with strong pretrained reasoning.

Load-bearing premise

The paper's conclusion that RL adds no new knowledge depends on its 40 reasoning phrases being genuine causal markers of reasoning rather than proxies for output length, formatting, or other surface traits; if the phrases are only correlated with those traits, the pattern-reward results would not show what the paper claims.

Editorial extensions

If this is right

  • If RPR-only training matches strict verification near its peak, then answer-correctness verification is not the main source of RL gains for already-capable models; pattern reinforcement is sufficient to elicit near-verifier performance.
  • Reward-noise tolerance cannot be assumed: it depends on the model's pretrained reasoning strength, since a model with weaker foundational reasoning collapses at far lower noise.
  • Adding RPR as an auxiliary signal to a noisy reward model improves downstream win rates by up to 30 net points over the original reward model, and lets a smaller model (Qwen-2.5-3B) reason on open-ended tasks where vanilla RL collapses.
  • The boundary of noise robustness in math rewards is a 50% flip rate, where the reward becomes completely random and training collapses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If reward noise is as tolerable as the paper suggests, then for tasks with known reasoning patterns, development effort could be redirected from verifier accuracy to eliciting structured chains; RPR's fixed phrase list is one cheap embodiment, but the same logic could be tested with learned or model-generated pattern detectors.
  • The RPR experiment does not on its own separate "reasoning pattern" from "verbose, chain-shaped output"; a length-matched or format-matched control would test whether performance comes from semantic reasoning phrases or from merely producing longer, organized text.
  • A natural extension is to probe the transfer boundary of RPR-style pattern rewards across domains, languages, and prompt formats; the paper already shows cross-task transfer from math to HelpSteer3, but how far that transfer reaches is unexplored.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies RL post-training of LLMs under noisy rewards. In math tasks, the authors flip verification rewards with probability p and report that Qwen-2.5-7B remains robust up to 40% flips, reaching a peak MATH-500 accuracy of 72.02% versus 75.85% with clean rewards. They then introduce Reasoning Pattern Reward (RPR), which rewards the presence of 40 hand-collected reasoning phrases without checking answer correctness, and report a peak of 70.21%. They interpret this as direct evidence that RL mostly activates reasoning patterns learned in pretraining rather than teaching new knowledge, and they use RPR to calibrate noisy reward models on the HelpSteer3 open-ended task, reporting net win-rate improvements. The paper includes comparisons on Qwen-2.5-3B and Llama-3.1-8B, human evaluation, and a limitations section.

Significance. If the causal interpretation is accepted, the result would be practically significant: it would suggest that precise verification is less critical than previously assumed and that cheap phrase-based rewards can substitute for or calibrate noisy reward models. I credit the authors for several strong practices: they release code and scripts, they report human evaluation alongside GPT-4o evaluation with inter-annotator agreement, they include a limitations section, and they explicitly discuss an open research question about false-positive calibration. The empirical robustness to 40% reward flips is interesting even independently of the 'no new knowledge' interpretation. However, the headline interpretation rests on Experiment 2, whose design has an answer-leakage problem, and the quantitative support lacks variance control; both issues are fixable within the scope of a revision.

major comments (4)
  1. [Section 2.2, Experiment 2] The RPR phrase list was extracted from outputs of a model trained with p=0.3 reward flips on the same 57K math problems. The reward used in the RPR-only experiment is therefore built from features of a model whose output distribution was shaped by answer-verification rewards on this benchmark. Because the phrase reward can re-surface answer-derived behavior, the 70.21% peak does not by itself support Takeaway 2 ('LLMs do not require much new knowledge during RL'). I ask for a leakage-free control, such as extracting the phrase list from the base pretrained model before any RL, or constructing an equivalently sized phrase list from an unrelated domain and showing that it does not produce the same effect.
  2. [Section 2.1, Figure 3] The robustness claim is supported by single-run accuracy curves and peak values. No seeds or variance are reported, so the headline comparison of 72.02% versus 75.85% peak MATH-500 accuracy could be within run-to-run noise. I request mean and standard deviation over at least three independent seeds for the main conditions, and I recommend reporting mean final performance or a clearly defined convergence value rather than peak accuracy for the numbers in Figure 1 and the abstract.
  3. [Section 2.2, Figure 9] The RPR reward counts 40 generic surface n-grams (for example 'first,' and 'therefore') in the thought text. Without ablations that control for output length, formatting, or a random-phrase baseline, the conclusion that RL reinforces 'reasoning patterns' rather than length, format, or surface style is not established. I ask for at least one matched control, such as rewarding an equally frequent set of non-reasoning words, or a length-matched reward, to show that the observed peak is not merely a format or length signal.
  4. [Section 3.3 and Appendix E] The calibration weight alpha=0.1 was selected after observing its effect on training dynamics on the same HelpSteer3 task used for evaluation, and the win-rate comparisons in Figure 8 are based on a 200-sample GPT-4o evaluation without uncertainty estimates. The reported net win rates should be accompanied by confidence intervals or significance tests, and the selection of alpha should be validated on a held-out split to rule out overfitting to the evaluation set.
minor comments (4)
  1. [Figure 9] The printed code is not runnable as shown: line 13 uses the undefined variable 'think_pos' and line 15 immediately overwrites the string parsed from 'solution_str'. Please correct the code so it matches the described reward computation.
  2. [Figure 9] There is a typo in the code: 'makes sence' should be 'makes sense'.
  3. [Introduction and Section 2.2] The phrase 'no new knowledge was learned' is too strong even if the proposed controls are added; I recommend softening to 'little or no task-specific answer supervision' or similar, since RL can still induce output-format changes and selection behavior.
  4. [Section 2.2 and 3.3] The free parameters n, r, tau, alpha, and the repetition-penalty n-gram size should be summarized in a reproducibility table, since several are described as arbitrarily set or tuned on the target task.

Circularity Check

1 steps flagged · score 6.0 of 10

The RPR-only experiment's 'no new knowledge' conclusion is compromised because the rewarded phrase list is fitted to outputs of a model trained with answer-derived rewards; the robustness results themselves are independent.

  1. fitted input called prediction [Section 2.2, Experiment 2]
    "We first identified n high-frequency phrases that imply certain desired reasoning patterns, such as 'We know that' and 'First I need to,' in the outputs of a model trained with p=0.3 Next, we designed a rule-based reward function: instead of verifying the correctness of the answer, the model would receive a reward of value r each time a pre-identified reasoning phrase appeared in the output. ..."

    In the p=0.3 run, rewards are still derived from ground-truth correctness 70% of the time (Section 2.1: 'We train the model by randomly flipping the reward with a probability p, where a reward of 1 is transformed to 0, and vice versa'). The phrase list is therefore fitted to the output distribution of a model whose behavior was shaped by correctness supervision on these same 57K math problems. Using that fitted phrase list as the entire reward in a fresh run and then observing 70.21% on MATH-500 does not show that 'no new knowledge was learned' during RL: the reward itself was constructed from answer-influenced outputs, so correctness information can leak into the RPR signal.

full rationale

The paper's strongest self-contained results are the empirical robustness findings: reward flipping up to p=0.40 still yields rapid convergence (Experiment 1), and RPR-only training reaches 70.21% on MATH-500 while noting overthinking-induced decline. These numbers are not themselves circular. The circularity concerns the central interpretation that 'no correctness supervision signals were given, meaning no new knowledge was learned.' The RPR phrase list is extracted from outputs of a p=0.3 model, whose rewards were correct 70% of the time; the phrase set is thus a fitted representation of answer-supervised behavior. Rewarding those phrases and observing similar accuracy is a self-consistency result, not independent evidence that RL only activates pretrained patterns. The calibration experiment (Experiment 4) inherits this concern because it reuses the same phrase list, though its empirical transfer to HelpSteer3 is a genuine demonstration of cross-task utility. The self-citations in the paper (e.g., refs. [5], [16]) are minor and not load-bearing. The alpha=0.1 calibration weight is tuned on the target task, which is standard hyperparameter selection and not circular, though it limits the strength of the reported net-win-rate comparisons. Overall, the robustness claims stand independently, but the paper's headline conceptual claim leans on a fitted-input experiment and therefore receives a partial circularity score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on several free choices: the RPR phrase count and per-phrase weight, the calibration threshold and scaling factor, and the repetition-penalty parameters. The phrase list itself is a handcrafted artifact derived from RL-trained outputs. The main axioms are that Qwen models have strong pretrained reasoning, that question-wise flipping is a realistic noise model, and that GPT-4o evaluations track human judgment.

free parameters (5)
  • n (number of RPR phrases) = 40
    Set without tuning (Section 2.2: 'somewhat arbitrarily set to 40'). Affects reward scale r=1/n.
  • r (per-phrase reward) = 0.025
    Set as 1/n; a design choice rather than a learned value.
  • alpha (RPR calibration weight) = 0.1
    Tuned in Appendix E from 1 down to 0.1 based on response length dynamics on the target task.
  • tau (low-score threshold) = 0.5
    Natural for a binary classifier but still a manual choice for when to apply RPR compensation.
  • Repetition penalty n-gram size = 20
    Used in the RPR code snippet to prevent phrase repetition; chosen without reported sensitivity analysis.
assumptions (4)
  • domain assumption Qwen-2.5-7B and Qwen-2.5-3B have strong pretrained reasoning abilities that RL merely activates.
    Takeaway 1 restricts noise robustness to models with strong reasoning potential; this premise is supported by comparisons with Llama-3.1-8B but not independently measured.
  • domain assumption Question-wise reward flipping models realistic reward noise.
    Section 2.1: noise is applied per question, not per output; the paper argues output-wise noise is not effective, but the realism of question-wise noise is asserted.
  • domain assumption GPT-4o pairwise evaluation approximates human judgment of helpfulness.
    Main results use GPT-4o scores; human evaluation in Appendix B shows moderate/substantial agreement, supporting but not guaranteeing validity.
  • ad hoc to paper The 40 reasoning phrases capture the causal 'reasoning patterns' that drive RL gains.
    The phrase list is curated from outputs of a p=0.3 trained model and used both as the RPR reward and as evidence for the no-new-knowledge hypothesis; no independent validation that these phrases specifically cause reasoning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason." pith.science (2026). https://pith.science/paper/CQ5BEX5V

@misc{pith2026250522653,
  author       = {Pith},
  title        = {Pith review of: The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CQ5BEX5V}},
  note         = {Machine review of arXiv:2505.22653}
}
read the original abstract

Recent studies on post-training large language models (LLMs) for reasoning through reinforcement learning (RL) typically focus on tasks that can be accurately verified and rewarded, such as solving math problems. In contrast, our research investigates the impact of reward noise, a more practical consideration for real-world scenarios involving the post-training of LLMs using reward models. We found that LLMs demonstrate strong robustness to substantial reward noise. For example, manually flipping 40% of the reward function's outputs in math tasks still allows a Qwen-2.5-7B model to achieve rapid convergence, improving its performance on math tasks from 5% to 72%, compared to the 75% accuracy achieved by a model trained with noiseless rewards. Surprisingly, by only rewarding the appearance of key reasoning phrases (namely reasoning pattern reward, RPR), such as ``first, I need to''-without verifying the correctness of answers, the model achieved peak downstream performance (over 70% accuracy for Qwen-2.5-7B) comparable to models trained with strict correctness verification and accurate rewards. Recognizing the importance of the reasoning process over the final results, we combined RPR with noisy reward models. RPR helped calibrate the noisy reward models, mitigating potential false negatives and enhancing the LLM's performance on open-ended tasks. These findings suggest the importance of improving models' foundational abilities during the pre-training phase while providing insights for advancing post-training techniques. Our code and scripts are available at https://github.com/trestad/Noisy-Rewards-in-Learning-to-Reason.

Figures

Figures reproduced from arXiv: 2505.22653 by the authors.

Figure 1
Figure 1. All of (1) standard RL, (2) RL with 40% of the rewards manually flipped to the opposite, and (3) RL with only Reasoning Pat￾tern Rewards (RPR) (i.e., rewards are given whenever key reasoning phrases appear, with￾out verifying the final answer)—can improve Qwen-2.5-7B’s accuracy on MATH-500 from an initial 5% to over 70%. The performance gap between these three setups is minimal compared to the overall improvements. … view at source ↗
Figure 2
Figure 2. The prompt used in math training, where the “question” placeholder will be replaced with a specific question. Specifically: the dataset includes 57K high-quality, source-mixture math problems spanning various difficulty levels. The prompts used in these tasks are shown in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Accuracy on three test sets during training. Due to critic warmup, the actor model is not updated during the first 20 steps; thus, the x-axis begins at step 20. 2.2 Experiments Experiment 1. We train the model with the probability p of noise increasing from 0% to 50%, with intervals of 10%, corresponding to increasingly random reward flips. The results are shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (19 more)
Figure 4
Figure 4. Figure 4: An illustration of how the reason￾ing pattern reward works through two example outputs. Suppose the red text represents high￾frequency phrases that we have pre-identified as indicating key reasoning processes. In the first output, five key phrases are present, so the r…
Figure 6
Figure 6. Figure 6: The prompt used in the HelpSteer3 task, where the “question” and “chat history” place￾holders are filled accordingly. 3.1 Preliminaries: Training reward models with varying accuracy Dataset. We use the NVIDIA HelpSteer3 [30] dataset, which contains 40.5K multi-domain o…
Figure 5
Figure 5. Figure 5: Reward model’s accuracy across the training. Checkpoints at specific steps are used for RL experiments. The prediction head produces a scalar output s, and we optimize the model using the MSE between s and the corresponding binary label [16, 35, 7]. The model learns to…
Figure 7
Figure 7. Figure 7: Qwen-2.5-7B trained with an 85%- accurate RM performs similarly to using a 75%-accurate RM, but significantly better than using a 65%-accurate RM. The “Net Win” refers to the performance advantage of the for￾mer RM over the latter. Experiment 3. We compare the performa…
Figure 8
Figure 8. Figure 8: Reward noise calibration effectively enhances downstream performance. Given these considerations, we only introduce the first method to calibrate the RM model: When the RM outputs a low score (as determined by a thresh￾old τ ), we calculate an RPR score only for the th…
Figure 9
Figure 9. Figure 9: An example code of reasoning pattern reward without checking answer correctness. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: An example of output in the later stage of RL math training, where only the reasoning pattern reward is used without correctness verification. The model has arrived at the correct answer “A,” but due to the ongoing reasoning process, the <think> tag remains open, caus…
Figure 11
Figure 11. Figure 11: Qwen-2.5-3B results for Experiments 1 and 2: Accuracy on three test sets during training. In the HelpSteer3 task, vanilla RL fails to enable Qwen-2.5-3B to perform effective reasoning. We observe that response lengths initially increase but then rapidly collapse to ju…
Figure 12
Figure 12. Figure 12: (a) Average response length of Qwen-2.5-3B during training with original vs calibrated RMs. The calibrated RMs successfully enable this small-scale model to perform reasoning, whereas the original RMs fail. (b) Experiment 3 using Qwen-2.5-3B models trained with calibr…
Figure 13
Figure 13. Figure 13: Our calibrated RMs successfully elicit Qwen-2.5-3B’s reasoning ability, whereas the original RM fails to do so. This figure presents 1 of 2 output cases. The Chinese question translates to: “Please design a teaching PowerPoint for teaching elementary school students a…
Figure 14
Figure 14. Figure 14: This figure presents the second of two output cases from Qwen-2.5-3B, trained with the calibrated 85%-accurate RM. The model demonstrates reasoning to solve a physics problem. 20 40 60 80 100 120 140 20 40 60 80 100 120 140 20 40 60 80 100 120 140 0 5 10 15 20 20 15 1…
Figure 15
Figure 15. Figure 15: Llama-3.1-8B results for Experiments 1 and 2: Accuracy on three test sets during training (Llama-3.1-8B). B Human evaluation B.1 Guidelines We recruited three graduate students with expertise in model evaluation. Each evaluator spent approximately 8 hours completing a…
Figure 16
Figure 16. Figure 16: The evaluation prompt for GPT, designed according to the core guidelines for human annotators. The placeholders will be replaced with user-assistant chat history and two models’ responses. Your task is to determine which of the two responses better addresses the user’…
Figure 17
Figure 17. Figure 17: Human evaluation results and agreements. C Case studies Figures 19 and 20 show outputs from Qwen-2.5-7B trained with the calibrated and original 85%- accurate RMs, respectively. With RPR, the generated code and comments are more detailed, and both the main function an…
Figure 18
Figure 18. Figure 18: α = 0.1 enables Qwen’s effec￾tive reasoning in HelpSteer3 task. These experiments use the 85%-accurate RM. E RPR weight in calibration In Section 3.3, we set τ = 0.5 since the RMs are binary classifiers. For the scaling factor α, we initially used a value of 1, but ob…
Figure 19
Figure 19. Figure 19: How to create a chatbot using an LLM: the answer from Qwen-2.5-7B trained with the calibrated 85%-accurate RM. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 20
Figure 20. Figure 20: How to create a chatbot using an LLM: the answer from Qwen-2.5-7B trained with the original 85%-accurate RM. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_20.png]
Figure 21
Figure 21. Figure 21: How to create a chatbot using an LLM: the answer from Qwen-2.5-7B trained with the calibrated 65%-accurate RM. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_21.png]
Figure 22
Figure 22. Figure 22: How to create a chatbot using an LLM: the answer from Qwen-2.5-7B trained with the original 65%-accurate RM. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_22.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mirage or Method? How Model-Task Alignment Induces Divergent RL Conclusions

    cs.LG 2025-08 conditional novelty 6.0 of 10

    Counterintuitive RL phenomena in LLMs (one-shot, spurious reward, negative-only) appear only under strong model-task alignment measured by pass@k, not because of data contamination.

  2. StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason

    cs.AI 2025-07 conditional novelty 6.0 of 10

    StepHint improves RLVR math reasoning by giving the model multiple prefix-level hints drawn from correct chains generated by stronger models, beating several RLVR baselines on six math benchmarks and two out-of-domain sets.

  3. ASTRO: Teaching Language Models to Reason by Reflecting and Backtracking In-Context

    cs.AI 2025-07 conditional novelty 5.0 of 10

    ASTRO converts MCTS search trees into chain-of-thoughts with explicit self-reflection and backtracking, trains Llama-3.1-70B on them with SFT, then improves with RL, reaching 81.8% on MATH-500, 64.4% on AMC 2023, and ...

Reference graph

Works this paper leans on

38 extracted references · 17 canonical work pages · cited by 3 Pith papers

  1. [1]

    Rethinking reflection in pre-training, 2025

    Essential AI, :, Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Anthony Polloreno, Ashish Tanwer, Burhan Drak Sibai, Divya S Mansingka, Divya Shivaprasad, Ishaan Shah, Karl Stratos, Khoi Nguyen, Michael Callahan, Michael Pust, Mrinal Iyer, Phil...

    2025

  2. [2]

    Math- arena: Evaluating llms on uncontaminated math competitions, February 2025

    Mislav Balunovi´c, Jasper Dekoninck, Ivo Petrov, Nikola Jovanovi´c, and Martin Vechev. Math- arena: Evaluating llms on uncontaminated math competitions, February 2025

    work page 2025

  3. [3]

    Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025

    2025

  4. [4]

    The accuracy paradox in RLHF: When better reward models don‘t yield better language models

    Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, and Xiaoyu Shen. The accuracy paradox in RLHF: When better reward models don‘t yield better language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2980–2989, Miami, Florida, USA,...

    work page 2024

  5. [5]

    Fortify the shortest stave in attention: Enhancing context awareness of large language models for effective tool use

    Yuhan Chen, Ang Lv, Ting-En Lin, Changyu Chen, Yuchuan Wu, Fei Huang, Yongbin Li, and Rui Yan. Fortify the shortest stave in attention: Enhancing context awareness of large language models for effective tool use. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisti...

    work page 2024

  6. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  7. [7]

    Q*: Improving multi-step reasoning for LLMs with deliberative planning, 2024

    Yanchen Deng, Chaojie Wang, Zhiyi Lyu, Jujie He, Liang Zeng, Shuicheng Y AN, and Bo An. Q*: Improving multi-step reasoning for LLMs with deliberative planning, 2024

    work page 2024

  8. [8]

    Fleiss et al

    J.L. Fleiss et al. Measuring nominal scale agreement among many raters.Psychological Bulletin, 76(5):378–382, 1971

    work page 1971

Show all 38 references
  1. [9]

    Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E

    Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N. Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. How to evaluate reward models for rlhf, 2024

    2024

  2. [10]

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars, 2025

    2025

  3. [11]

    Training large language models to reason in a continuous latent space, 2024

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space, 2024

    2024

  4. [12]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (...

    2021

  5. [13]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model, 2025

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model, 2025

    2025

  6. [14]

    Human-centric dialog training via offline reinforcement learning

    Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of...

    2020

  7. [15]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. RewardBench: Evaluating reward models for language modeling. In Luis Chiru...

    2025

  8. [16]

    Skywork-reward: Bag of tricks for reward modeling in llms, 2024

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms, 2024

    2024

  9. [17]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024

    2024

  10. [18]

    RM-bench: Benchmarking reward models of language models with subtlety and style

    Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. RM-bench: Benchmarking reward models of language models with subtlety and style. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  11. [19]

    The llama 3 herd of models, 2024

    AI @ Meta Llama Team. The llama 3 herd of models, 2024

    2024

  12. [20]

    s1: Simple test-time scaling, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025

    2025

  13. [21]

    Webgpt: Browser-assisted question-answering with human feedback, 2022

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. We...

    2022

  14. [22]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  15. [23]

    Tinyzero

    Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https://github.com/Jiayi-Pan/TinyZero, 2025. Accessed: 2025-01-24

    2025

  16. [24]

    Lee, and Sanjeev Arora

    Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D. Lee, and Sanjeev Arora. What makes a reward model a good teacher? an optimization perspective, 2025

    2025

  17. [25]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023

    2023

  18. [26]

    High- dimensional continuous control using generalized advantage estimation, 2018

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation, 2018

    2018

  19. [27]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017

    2017

  20. [28]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  21. [29]

    Kimi k1.5: Scaling reinforcement learning with llms, 2025

    Kimi Team. Kimi k1.5: Scaling reinforcement learning with llms, 2025

    2025

  22. [30]

    Dedicated feedback and edit models empower inference- time scaling for open-ended general-domain tasks, 2025

    Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Daniel Egert, Ellie Evans, Hoo-Chang Shin, Felipe Soares, Yi Dong, and Oleksii Kuchaiev. Dedicated feedback and edit models empower inference- time scaling for open-ended general-domain tasks, 2025

    2025

  23. [31]

    Rethinking reward model evaluation: Are we barking up the wrong tree? InThe Thirteenth International Conference on Learning Representations, 2025

    Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, XingYu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, and Le Sun. Rethinking reward model evaluation: Are we barking up the wrong tree? InThe Thirteenth International Conference on Learning Representations, 2025. 11

    2025

  24. [32]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  25. [33]

    Demystifying long chain-of-thought reasoning in llms, 2025

    Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in llms, 2025

    2025

  26. [34]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025

    2025

  27. [35]

    ReST- MCTS*: LLM self-training via process reward guided tree search

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. ReST- MCTS*: LLM self-training via process reward guided tree search. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  28. [36]

    Found in the middle: How language models use long contexts better via plug-and-play positional encoding, 2024

    Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. Found in the middle: How language models use long contexts better via plug-and-play positional encoding, 2024

    2024

  29. [37]

    Rmb: Comprehensively benchmarking reward models in llm alignment, 2025

    Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. Rmb: Comprehensively benchmarking reward models in llm alignment, 2025

    2025

  30. [38]

    Assistant:␣<think>

    Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Proceedings of the 40th ...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.