Pith. sign in

REVIEW 4 major objections 3 minor 56 references

Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation

T0 review · 4 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 150M-parameter reward model trained on 19K human Likert ratings gives GRPO a semantic reward that makes 1.5B and 3B policies rival larger instruct models, beating ROUGE-L, BERTScore, and GRM-llama-3B on long-form generation.

desk verdict A sensible, cheap reward model for GRPO on long-form generation, but the paper never directly validates the reward on its target domains and overclaims what 'verifiable' means. read the letter →

arxiv 2506.15068 v1 pith:POHJ6WXT submitted 2025-06-18 cs.CL cs.LG

Zongxia Li , Yapei Chang , Yuhang Zhou , Xiyang Wu , Zichao Liang , Yoo Yeon Sung , Jordan Lee Boyd-Graber This is my paper · ORCID
classification cs.CLcs.LG
keywords PrefBERTGRPOrewardmodellong-formgenerationopen-endedhumanpreferencereinforcementlearningLikertrating
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes PrefBERT, a 150M-parameter encoder trained on 19K Likert-rated examples, and uses its predicted score as the reward signal in GRPO (Group Relative Policy Optimization) for open-ended long-form generation. The central claim is that this reward teaches policy models to produce answers humans prefer more than rewards from ROUGE-L, BERTScore, or the much larger GRM-llama-3B. Across ELI5, Alpaca, and LongForm, 1.5B and 3B policies trained with PrefBERT match or exceed Qwen2.5-7B-Instruct on LLM-as-a-judge metrics, and human annotators rank 3B-PrefBERT as the strongest 3B policy model. If right, this opens GRPO to tasks without verifiable correctness, using a small, cheap semantic signal instead of a large general reward model.

What carries the argument

PrefBERT is a ModernBERT encoder (150M parameters) with a linear head and sigmoid, trained by mean squared error on pairs of reference and generated answers to predict a normalized Likert score $r_i=(s_i-1)/4$. The input is the concatenation $x_{\text{pair}}=[\text{CLS}]\,x^{\text{ref}}\,[\text{SEP}]\,x^{\text{gen}}$, and the trained prediction $\hat r_i=\sigma(w^\top h_i+b)$ is the reward fed into GRPO, where it is normalized within a group of sampled responses to form the advantage $A(x,y_i)=(r_i-\bar r)/\sigma_r$. This gives the policy a graded, reference-grounding semantic signal rather than a lexical-overlap or length-based one.

What would settle it

Collect a random sample of final PrefBERT-trained responses from each of the three test sets, get human Likert ratings on the same rubric, and compute the rank correlation with PrefBERT scores. If the correlation is near zero or negative on these domains, the reward is not transferring. A second decisive check is to pad responses with additional words and see whether PrefBERT's score rises; if it does, it is rewarding length, repeating the GRM failure the paper claims to avoid.

Watch

Extended reading notes

Core claim

The paper's discovery is that a small reference-based semantic scorer can carry a GRPO training signal for free-form text. PrefBERT concatenates a reference answer and a generated answer, pools a ModernBERT representation, and regresses the pair to a normalized human Likert score; that scalar is then group-normalized into the GRPO advantage. Trained only on the Prometheus-preference and MOCHA evaluation datasets, the model's reward remains informative during optimization, unlike ROUGE-L and BERTScore, which barely move, and unlike GRM-llama-3B, whose reward tracks length and drives policies to 1,024-token verbose answers. The resulting 3B policy is the top 3B model in human ratings, and both 1.5B and 3B PrefBERT policies beat Qwen2.5-7B-Instruct on the automatic judge's Likert scale and win rate.

Load-bearing premise

The load-bearing premise is that PrefBERT's score, learned only from Prometheus-preference and MOCHA Likert data, transfers to ELI5, Alpaca, and LongForm and measures human quality there without being hacked by length or phrasing.

Editorial extensions

If this is right

  • GRPO can be applied to open-ended generation when the reward is a learned semantic model, not only when a rule-based correctness signal exists.
  • A 3B policy trained with PrefBERT can beat Qwen2.5-7B-Instruct on LLM-judged quality, cutting the parameter cost of open-ended alignment.
  • ROUGE-L and BERTScore are too weak as GRPO rewards for long-form text; their reward curves barely move and their policies produce generic, repetitive answers.
  • Reward models built for preference ranking, such as GRM-llama-3B, can be exploited by length in RL, and human evaluators downgrade the verbose outputs they induce.
  • PrefBERT-trained policies follow explicit length and format constraints more reliably than the base model, and are rated as more fluent and readable than SFT policies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is a length-controlled ablation: cap or penalize response length inside PrefBERT and see whether the quality gains survive; that would separate semantic signal from an implicit brevity prior.
  • The recipe should carry to other reference-available open-ended tasks, such as creative writing or summarization, but domains without a reference answer would need a prompt-derived or self-consistency reference.
  • The authors state that they never tried a 7B-scale reward provider; if a larger evaluator gives better rewards, the 150M size is a practical choice rather than an upper bound.
  • Because PrefBERT was trained on only 19K examples, the bottleneck may be the coverage of Likert supervision rather than model capacity, which suggests targeted collection of ratings in new domains could improve transfer.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes PrefBERT, a 150M-parameter ModernBERT regressor trained on roughly 19K Likert-rated (reference, response) pairs from Prometheus-preference and MOCHA, and uses its predicted score as the reward signal in GRPO for open-ended long-form generation. The policy models are Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct, trained on ELI5, Alpaca, and LongForm samples. The authors compare PrefBERT rewards against ROUGE-L, BERTScore, and GRM-llama-3B, using GPT-4-as-judge, a 150-prompt human evaluation, Bradley-Terry win rates, reward learning curves, and qualitative examples. The main claims are that PrefBERT provides better semantic reward feedback than token-overlap metrics, that PrefBERT-trained 1.5B/3B models rival or exceed Qwen2.5-7B-Instruct on open-ended generation, and that PrefBERT is more efficient than GRM-llama-3B while producing responses humans prefer.

Significance. If the central claim holds, the paper makes a useful empirical contribution: it demonstrates that a small, efficient reward model can be plugged into GRPO for a class of tasks where rule-based verifiable rewards are unavailable, and it provides a concrete training recipe and open-source code. The comparison against GRM-llama-3B and traditional metrics is informative, and the observation that GRM-llama-3B's reward tracks length while PrefBERT's does not is a valuable cautionary result. However, the significance is conditional on PrefBERT actually being a valid semantic reward on ELI5, Alpaca, and LongForm; the manuscript does not directly validate reward-model accuracy or human correlation on those distributions, and the human evaluation that would ground the preference-alignment claim is small and lacks significance testing. The paper also ships reproducible code and detailed prompts, which is a strength.

major comments (4)
  1. [§3.3, §4.1] The paper never reports any direct validation of PrefBERT on the target distributions ELI5, Alpaca, and LongForm. Section 3.3 trains PrefBERT on Prometheus-preference and MOCHA with the MSE objective of Eq. (4), but Section 4.1 applies it as a GRPO reward on three held-out datasets without reporting reward-model correlation with human ratings, per-dataset accuracy, calibration, or reliability on those domains. Since the central claim is that PrefBERT provides a semantically valid reward, this transfer is load-bearing; without such validation, the downstream policy improvements could reflect optimization of an unvalidated proxy rather than alignment with human quality. I request held-out accuracy on the 20% split described in §3.3 and human-rating correlation for PrefBERT scores on sampled outputs from each target dataset.
  2. [Table 1, Table 3, §5.3] The automatic-evaluation advantage for PrefBERT-trained models is confounded by surface style. Table 3 shows that 3B-PrefBERT uses markdown in 81.31% of responses and averages 258 words, whereas 3B-ROUGE-L uses markdown in 21.92% of responses and averages 182 words; the reward curves in §5.3 show that GRM-llama-3B exploits length, and the human evaluation in Table 2 contradicts the automatic ranking for that same model. Because GPT-4-as-judge is known to favor longer and more structured outputs, the observed Likert-score gains in Table 1 may reflect format and length rather than the semantic quality PrefBERT is claimed to measure. I request a length- and format-controlled analysis, for example by including length and markdown as covariates in a regression over judge scores or by evaluating truncated or plain-text versions of the responses.
  3. [§6.1, Table 2] The human evaluation is too small and insufficiently controlled to support the quantitative claim that PrefBERT-trained models are better aligned with human preferences. It uses 150 prompts, four author annotators, and no inter-annotator agreement or significance tests; the overall Likert gap between 3B-PrefBERT (3.36) and 3B-ROUGE-L (3.31) is only 0.05, and the success-rate differences are not accompanied by confidence intervals. I request significance testing (for example, bootstrap or a mixed-effects model), agreement statistics, and a blind protocol with independent annotators, or alternatively a downgraded claim that the human evidence is preliminary.
  4. [Appendix F, Figures 6–8] Several qualitative examples in Appendix F have mismatched prompts and reference answers, which weakens the qualitative evidence used in Section 6.2. Figure 6 shows a prompt asking "Who is Jagadeesh Kanna? Respond in 3 sentences" with a reference answer for Jagadeesh Kanna, yet the analysis praises a long-form entrepreneurial narrative; Figure 7 shows a Tesla Model 3 prompt with a Bruce Straley reference; Figure 8 shows a Tesla Model 3 prompt with a Big Bang reference. These mismatches must be corrected or the examples removed before the qualitative analysis can be used as support for the paper's claims.
minor comments (3)
  1. [Abstract, Introduction] There are missing spaces in several places (e.g., "inGRPOand," "rewardsand," "and-"), which should be fixed with a careful pass over the compiled PDF.
  2. [§4.2, §5.2] There are typos such as "with with a global batch size" in §4.2 and "success rate by 50%m" in §5.2; these should be corrected.
  3. [Table 8] The pairwise preference template instructs the judge to output "Final rating: (your rating, as an integer between 1 and 5)" even though the task is to choose Response A or B; this template inconsistency should be fixed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PrefBERT is a fitted reward regressor, and the paper's results are empirical comparisons, not derivations; the untested transfer and shared-rubric evaluation are validity concerns, not circularity.

full rationale

PrefBERT is trained as a supervised regressor on Likert-rated examples (Eq. 3-4), and the paper's central claims are empirical comparisons of policies optimized with different reward signals, not derivations from first principles. The reward model is not defined in terms of the evaluation outcome, nor is any evaluation metric identical to the training target by construction. The same rubric appears in the GRPO training prompt, the GPT-4 judge prompt, and the human annotation instrument, but PrefBERT was trained on Prometheus-preference and MOCHA rather than on that specific rubric, and the human evaluation is independent of PrefBERT's scores. This shared-rubric design is a possible evaluation-bias concern, not a definitional circularity. Self-citations (e.g., Li et al. 2024) are used only as supporting citations for known limitations of lexical metrics and are not load-bearing; no uniqueness theorem, ansatz, or prior result from the same authors is imported to force the paper's choices. The transfer of PrefBERT to ELI5, Alpaca, and LongForm is not directly validated, but that is an external-validity limitation consistent with the paper's own Limitations section, not a circular step. No equation reduces to another by construction, and no fitted parameter is renamed as a prediction. Therefore no circularity is found.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's contribution is a fitted reward model; the central claim rests on the transfer of that model to new domains and on evaluation proxies (GPT-4 and author-provided human ratings) whose validity is asserted but not demonstrated.

free parameters (4)
  • PrefBERT regressor weights (w, b) = Not reported
    Eq. 3: trained via MSE on 19K Likert-rated examples from Prometheus-preference and MOCHA; the central reward signal depends entirely on these fitted weights, yet the paper reports no held-out correlation of PrefBERT with human ratings.
  • GRPO group size G = 4
    Section 4.2: group size for advantage normalization; chosen by hand, affects reward variance and training dynamics.
  • GRPO learning rate = 1e-6
    Section 4.2: global batch 128, one epoch; hyperparameters chosen without ablation.
  • KL penalty coefficient beta = Not reported
    Eq. 5, Appendix A.1: KL regularization coefficient in GRPO objective, not specified in the paper; load-bearing for stable training.
assumptions (5)
  • domain assumption GRPO advantage normalization produces useful learning signal
    Eq. 1: The paper assumes group-normalized scalar rewards with KL penalty improve policy quality for open-ended generation, a standard RLHF assumption not proven in this setting.
  • domain assumption Prometheus-preference and MOCHA Likert labels are representative of general human quality judgments for long-form text
    Section 3.3: PrefBERT is trained only on these two datasets; transfer to ELI5/Alpaca/LongForm is assumed without measuring reward-model correlation on target domains.
  • domain assumption Reference answers are valid anchors for scoring open-ended generation
    Eq. 2: PrefBERT scores a generated response against a single reference; assumes distance to the reference is a proxy for quality, which is questionable for open-ended tasks with multiple valid answers.
  • domain assumption GPT-4 LLM-as-a-judge ratings are a reliable proxy for human preference
    Section 5.1: The main automatic evaluation uses GPT-4 with a fixed rubric; the paper cites prior work but does not validate GPT-4's agreement on these datasets.
  • ad hoc to paper The rubric dimensions (factual accuracy, relevance, completeness, clarity, conciseness) fully capture open-ended quality
    Tables 5, 7: These criteria are chosen by the authors and used in training prompts, reward model evaluation, and LLM and human evaluation, creating a closed loop.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation." pith.science (2026). https://pith.science/paper/POHJ6WXT

@misc{pith2026250615068,
  author       = {Pith},
  title        = {Pith review of: Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/POHJ6WXT}},
  note         = {Machine review of arXiv:2506.15068}
}
read the original abstract

Evaluating open-ended long-form generation is challenging because it is hard to define what clearly separates good from bad outputs. Existing methods often miss key aspects like coherence, style, or relevance, or are biased by pretraining data, making open-ended long-form evaluation an underexplored problem. To address this gap, we propose PrefBERT, a scoring model for evaluating open-ended long-form generation in GRPO and guiding its training with distinct rewards for good and bad outputs. Trained on two response evaluation datasets with diverse long-form styles and Likert-rated quality, PrefBERT effectively supports GRPO by offering better semantic reward feedback than traditional metrics ROUGE-L and BERTScore do. Through comprehensive evaluations, including LLM-as-a-judge, human ratings, and qualitative analysis, we show that PrefBERT, trained on multi-sentence and paragraph-length responses, remains reliable across varied long passages and aligns well with the verifiable rewards GRPO needs. Human evaluations confirm that using PrefBERT as the reward signal to train policy models yields responses better aligned with human preferences than those trained with traditional metrics. Our code is available at https://github.com/zli12321/long_form_rl.

Figures

Figures reproduced from arXiv: 2506.15068 by the authors.

Figure 1
Figure 1. Our proposed reward model PrefBERT better reflects human judgments by distinguishing the good responses from the bad ones than other traditional metrics, generalized reward model. pothesize that PrefBERT, by capturing human pref￾erences, quality, and semantics, provides stronger reward signals than traditional metrics, enabling GRPO to train models that generate higher-quality responses. Through extensive evaluation… view at source ↗
Figure 2
Figure 2. Reward curves during GRPO training show key differences across reward functions. Traditional metrics show minimal reward change—only 0.05 for ROUGE-L and 0.25 for BERTScore—indicating limited model improve￾ment across all global training steps. In contrast, 3B-GRM-llama-3B’s reward is strongly correlated with response length; by step 60, it already generates the maximum allowed tokens (1,024), causing reward values … view at source ↗
Figure 3
Figure 3. Our annotation tool for response quality annotation. Annotators will be displayed with the question [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative Analysis: Readability. We present showcases comparing the readability of PrefBERT against baseline models, accompanied by detailed analysis [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Qualitative Analysis: Content Logic. We present showcases comparing the content logic of PrefBERT against baseline models, accompanied by detailed analysis [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Qualitative Analysis: Instruction Following. We present showcases comparing the instruction following, of PrefBERT against baseline models, accompanied by detailed analysis [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Qualitative Analysis: Tone and Fluency. We present showcases comparing the tone and fluency of PrefBERT against baseline models, accompanied by detailed analysis [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Qualitative Analysis: failure of SFT. We present showcases comparing the readability of PrefBERT against baseline models, accompanied by detailed analysis [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 7 canonical work pages

  1. [1]

    Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157

    arXiv 2024

  2. [2]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    arXiv 2022

  3. [3]

    Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.751 Re-evaluating evaluation in text summarization . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347--9359, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.emnlp-main.751 2020

  4. [4]

    Ralph Allan Bradley and Milton E. Terry. 1952. http://www.jstor.org/stable/2334029 Rank analysis of incomplete block designs: I. the method of paired comparisons . Biometrika, 39(3/4):324--345

    arXiv 1952

  5. [5]

    Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. https://doi.org/10.18653/v1/D19-5817 Evaluating question answering evaluation . In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 119--124, Hong Kong, China. Association for Computational Linguistics

    doi:10.18653/v1/d19-5817 2019

  6. [6]

    Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.528 Mocha: A dataset for training and evaluating generative reading comprehension metrics . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics

    work page doi:10.18653/v1/2020.emnlp-main.528 2020

  7. [7]

    Cheng-Han Chiang and Hung yi Lee. 2023 a . https://arxiv.org/abs/2305.01937 Can large language models be an alternative to human evaluations? Preprint, arXiv:2305.01937

    arXiv 2023

  8. [8]

    Cheng-Han Chiang and Hung yi Lee. 2023 b . https://arxiv.org/abs/2310.05657 A closer look into automatic evaluation using large language models . Preprint, arXiv:2310.05657

    arXiv 2023

Show all 56 references
  1. [9]

    Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: an open platform for evaluating llms by human preference. In Proceedings of the 41s...

    2024

  2. [10]

    DeepSeek-AI, Daya Guo, and et al. 2025. https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning . Preprint, arXiv:2501.12948

    2025 arXiv

  3. [11]

    Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. https://arxiv.org/abs/2104.08758 Documenting large webtext corpora: A case study on the colossal clean crawled corpus . Preprint, arXiv:2104.08758

    2021 arXiv

  4. [12]

    Aarohi Srivastava et al. 2023. https://arxiv.org/abs/2206.04615 Beyond the imitation game: Quantifying and extrapolating the capabilities of language models . Preprint, arXiv:2206.04615

    2023 arXiv

  5. [13]

    Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev

    Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. https://arxiv.org/abs/2007.12626 Summeval: Re-evaluating summarization evaluation . Preprint, arXiv:2007.12626

    2021 arXiv

  6. [14]

    Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. https://arxiv.org/abs/1907.09190 Eli5: Long form question answering . Preprint, arXiv:1907.09190

    2019 arXiv

  7. [15]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. https://arxiv.org/abs/2411.15594 A survey on llm-as-a-judge . Preprint, a...

    2025 arXiv

  8. [16]

    Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. 2024. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143

    2024 arXiv

  9. [17]

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. https://arxiv.org/abs/2405.01535 Prometheus 2: An open source language model specialized in evaluating other language models ....

    2024 arXiv

  10. [18]

    Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. https://arxiv.org/abs/2103.06332 Hurdles to progress in long-form question answering . Preprint, arXiv:2103.06332

    2021 arXiv

  11. [19]

    Michael Krumdick, Charles Lovering, Varshini Reddy, Seth Ebner, and Chris Tanner. 2025. https://arxiv.org/abs/2503.05061 No free labels: Limitations of llm-as-a-judge without human grounding . Preprint, arXiv:2503.05061

    2025 arXiv

  12. [20]

    Kullback and R

    S. Kullback and R. A. Leibler. 1951. https://doi.org/10.1214/aoms/1177729694 On information and sufficiency . The Annals of Mathematical Statistics, 22(1):79--86

    1951

  13. [21]

    Abdullatif Köksal, Timo Schick, Anna Korhonen, and Hinrich Schütze. 2023. https://arxiv.org/abs/2304.08460 Longform: Effective instruction tuning with reverse instructions . Preprint, arXiv:2304.08460

    2023 arXiv

  14. [22]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. https://arxiv.org/abs/2403.13787 Rewardbench: Evaluating reward models for language mo...

    2024 arXiv

  15. [23]

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. 2022. https://arxiv.org/abs/2207.01780 Coderl: Mastering code generation through pretrained models and deep reinforcement learning . Preprint, arXiv:2207.01780

    2022 arXiv

  16. [24]

    Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016. https://arxiv.org/abs/1606.01541 Deep reinforcement learning for dialogue generation . Preprint, arXiv:1606.01541

    2016 arXiv

  17. [25]

    Zongxia Li, Ishani Mondal, Yijun Liang, Huy Nghiem, and Jordan Lee Boyd-Graber. 2024. https://arxiv.org/abs/2402.11161 Pedants: Cheap but effective and interpretable answer equivalence . Preprint, arXiv:2402.11161

    2024 arXiv

  18. [26]

    Zongxia Li, Xiyang Wu, Hongyang Du, Fuxiao Liu, Huy Nghiem, and Guangyao Shi. 2025 a . A survey of state of the art large vision language models: Benchmark evaluations and challenges. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR) Workshops, pa...

    2025

  19. [27]

    Zongxia Li, Xiyang Wu, Guangyao Shi, Yubin Qin, Hongyang Du, Tianyi Zhou, Dinesh Manocha, and Jordan Lee Boyd-Graber. 2025 b . https://arxiv.org/abs/2505.01481 Videohallu: Evaluating and mitigating multi-modal hallucinations on synthetic video understanding . Preprint, arXiv:2...

    2025

  20. [28]

    Chin-Yew Lin. 2004. Rouge: a package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out

    2004

  21. [29]

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451

    2024 arXiv

  22. [30]

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. https://arxiv.org/abs/2503.20783 Understanding r1-zero-like training: A critical perspective . Preprint, arXiv:2503.20783

    2025 arXiv

  23. [31]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. Notion Blog

    2025

  24. [32]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 Bleu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311...

    2002

  25. [33]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  26. [34]

    Paul Roit, Johan Ferret, Lior Shani, Roee Aharoni, Geoffrey Cideron, Robert Dadashi, Matthieu Geist, Sertan Girgin, Léonard Hussenot, Orgad Keller, Nikola Momchev, Sabela Ramos, Piotr Stanczyk, Nino Vieillard, Olivier Bachem, Gal Elidan, Avinatan Hassidim, Olivier Pietquin, an...

    2023 arXiv

  27. [35]

    Hamidreza Rouzegar and Masoud Makrehchi. 2024. Enhancing text classification through llm-driven active learning and human annotation. arXiv preprint arXiv:2406.12114

    2024 arXiv

  28. [36]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. https://arxiv.org/abs/1707.06347 Proximal policy optimization algorithms . Preprint, arXiv:1707.06347

    2017 arXiv

  29. [37]

    Kun Shao, Zhentao Tang, Yuanheng Zhu, Nannan Li, and Dongbin Zhao. 2019. https://arxiv.org/abs/1912.10944 A survey of deep reinforcement learning in video games . Preprint, arXiv:1912.10944

    2019 arXiv

  30. [38]

    Stack Exchange contributors . 2025. Stack Exchange . https:// site .stackexchange.com/questions/ question_id . [Online; accessed 5-May-2025]

    2025

  31. [39]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022. https://arxiv.org/abs/2009.01325 Learning to summarize from human feedback . Preprint, arXiv:2009.01325

    2022 arXiv

  32. [40]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023 a . Alpaca: A strong, replicable instruction-following model. https://crfm.stanford.edu/2023/03/13/alpaca.html. [Online; accessed 5-May-2025]

    2023

  33. [41]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023 b . Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  34. [42]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions

    2022

  35. [43]

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. https://arxiv.org/abs/2412.13663 Smarter, better, f...

    2024 arXiv

  36. [44]

    Wikipedia contributors . 2025. https://en.wikipedia.org/ Wikipedia , the free encyclopedia . [Online; accessed 5-May-2025]

    2025

  37. [45]

    Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. 2023. https://arxiv.org/abs/2310.00212 Pairwise proximal policy optimization: Harnessing relative feedback for llm alignment . Preprint, arXiv:2310.00212

    2023 arXiv

  38. [46]

    Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. 2024. https://arxiv.org/abs/2406.10216 Regularizing hidden states enables learning generalizable reward model for llms . Preprint, arXiv:2406.10216

    2024 arXiv

  39. [47]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. 2025. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476

    2025 arXiv

  40. [48]

    Jixiao Zhang and Chunsheng Zuo. 2025. Grpo-lead: A difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models. arXiv preprint arXiv:2504.09696

    2025

  41. [49]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

    2020 arXiv

  42. [50]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena . P...

    2023 arXiv

  43. [51]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. https://arxiv.org/abs/2305.11206 Lima: Less is more for alignment . Preprint, arXiv:2305.11206

    2023 arXiv

  44. [52]

    Yuhang Zhou and Wei Ai. 2024. Teaching-assistant-in-the-loop: Improving knowledge distillation from imperfect teacher models in low-budget scenarios. arXiv preprint arXiv:2406.05322

    2024 arXiv

  45. [53]

    Yuhang Zhou, Giannis Karamanolakis, Victor Soto, Anna Rumshisky, Mayank Kulkarni, Furong Huang, Wei Ai, and Jianhua Lu. 2025 a . Mergeme: Model merging techniques for homogeneous and heterogeneous moes. arXiv preprint arXiv:2502.00997

    2025 arXiv

  46. [54]

    Yuhang Zhou, Jing Zhu, Shengyi Qian, Zhuokai Zhao, Xiyao Wang, Xiaoyu Liu, Ming Li, Paiheng Xu, Wei Ai, and Furong Huang. 2025 b . Disco balances the scales: Adaptive domain-and difficulty-aware reinforcement learning on imbalanced data. arXiv preprint arXiv:2505.15074

    2025

  47. [55]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  48. [56]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.