Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A meta reward model that rewrites its own scoring rubric during training matches or beats hand-crafted reward prompts across four tasks.

desk verdict A novel and well-engineered RLAIF method with real promise, but the essay/oracle claims rest on a GPT-4o judge whose reliability is questionable where tested. read the letter →

arxiv 2504.20157 v2 pith:UWZSZDCN submitted 2025-04-28 cs.CL

Zae Myung Kim , Chanwoo Park , Vipul Raheja , Suin Kim , Dongyeop Kang This is my paper · ORCID
classification cs.CL
keywords MetaPolicyOptimizationrewardhackingLLM-as-a-judgepromptevolutionRLAIFPPOmodelevaluativethinking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reward-based alignment of language models usually fixes the reward model's scoring prompt, so the model can learn to exploit loopholes and the prompt must be re-engineered by hand when it fails. This paper proposes Meta Policy Optimization (MPO), in which a larger meta reward model periodically inspects sampled student responses, the current rubric, and the scores those responses received, then rewrites the rubric before training continues. The paper claims that this self-evolving evaluation signal matches or outperforms models trained with extensively hand-crafted reward prompts, and that it works across essay writing, summarization, ethical reasoning, and mathematical reasoning without task-specific prompt design. A sympathetic reader would care because it points to an alignment loop that maintains its own evaluation standards instead of relying on brittle manual prompt engineering.

What carries the argument

The central object is the Meta Reward Model (MRM), a senior-instructor LLM that operates on a wider context than the reward model: the task description, sampled prompt instructions and reference solutions, the policy's generated responses, the current evaluation rubric, and the reward model's assigned scores. Every k training batches it executes three steps—meta-analysis, which checks whether scores are inflated or criteria are missing; meta-refinement, which expands or rewrites rubric items with explicit point ranges and examples; and meta-merging, which combines the n sampled refinements into one coherent rubric that becomes the reward model's new prompt. This makes the reward function itself an evolving object, formalized in the paper as a sequence of progressively finer partitions of the state space, so that rewards move from coarse averaged judgments toward finer distinctions as training proceeds.

What would settle it

Take a fixed set of student responses with human quality ratings, run MPO's meta-analysis and refinement on a deliberately inflated scoring case, and test whether the refined rubric changes the reward model's ranking of those responses to agree with the human ratings; if refinements do not improve agreement with human judgments, the claimed anti-reward-hacking benefit is not supported.

Watch

Extended reading notes

Core claim

The central claim is that a policy trained by PPO against a reward model whose evaluation prompt is periodically refined by a meta reward model produces final outputs as good as or better than PPO against fixed, expert- or oracle-crafted prompts, and that the refinements catch reward-hacking behaviors that would otherwise collapse training. In the essay-writing experiment the MPO-aligned model beat every fixed-prompt baseline, including the PPO model using an expert-crafted evaluation prompt, and the 72B fixed-RM baseline collapsed into title-only responses while the MPO runs recovered from similar degenerate behaviors. In a direct comparison, MPO-evolved prompts outperformed an oracle prompt that cost more than 60 PPO runs to engineer, produced within a single training epoch. The framework also improved accuracy over the base and fixed-prompt PPO models on the Scruples ethics benchmark and the MATH benchmark, and improved ROUGE scores on BillSum summarization.

Load-bearing premise

The load-bearing premise is that the meta reward model can reliably tell when a score is inflated and rewrite the rubric so the reward model's scores track true output quality; the paper never checks this meta-judgment against human ratings or an independent quality measure.

Editorial extensions

If this is right

  • Rubric prompts do not need to be finalized before training; starting from a generic initial prompt and letting the meta reward model refine it can replace expert prompt engineering for reward models.
  • Because refinements are triggered by observed score-inflation failures, MPO offers a concrete mechanism for catching reward hacking mid-run rather than only detecting it after training collapses.
  • The same task-agnostic meta-level prompts work across four tasks, so a single MPO pipeline can be deployed without per-task rubric design.
  • The framework is modular: the authors note it can be attached to other reward-based optimizers such as GRPO, not only PPO.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension is to validate each MRM refinement against human judgments or an independent quality metric; the paper does not do this, so the meta-judgment's reliability is currently measured only by downstream task scores.
  • The observation that most rubric changes happen in the first 5–10 iterations suggests a dynamic MPO schedule—more frequent refinements early, sparser later—could cut compute without losing the benefit.
  • If the meta reward model's refinements generalize, the same loop could serve as a continuous evaluation monitor after training, flagging when a deployed policy drifts into behaviors the current rubric rewards too generously.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Meta Policy Optimization (MPO), a framework that augments reinforcement learning from AI feedback (RLAIF) with a meta-reward model (MRM) that periodically rewrites the evaluation rubric (prompt) used by the reward model during PPO training. The central claim is that this dynamic rubric evolution yields policies that match or exceed those trained with extensively hand-crafted reward prompts, mitigates reward hacking, and generalizes across essay writing, summarization, ethical reasoning, and mathematical reasoning. The empirical evidence consists of GPT-4o head-to-head Elo ratings (Tables 1 and 4), ROUGE scores on BillSum (Table 2), accuracy on Scruples and MATH (Table 3), an analysis of rubric evolution (Section 3.4, Figure 5), and an oracle-prompt comparison (Section 3.5).

Significance. If the results hold, the paper addresses a real and timely problem: the brittleness and prompt-engineering cost of LLM-based reward models in alignment. The framework is simple, modular, and the authors release code, which is a strength. The task diversity is commendable, and the reported Elo gains are large (e.g., 1168 vs 984 in Table 1). However, the current evidence base is insufficient to support the headline claims because the principal evaluation metric (GPT-4o Elo) is shown to disagree with an objective reference metric in the one task where such a check exists, and the reward-hacking mitigation claim rests on a single anecdote. The theoretical formulation in Appendix B is also not connected to the actual implementation, as the authors acknowledge in Section 3.2.

major comments (4)
  1. [Section 3.3.1, Table 2] The only task with an objective reference metric contradicts the ranking used elsewhere: MPO 32b_32b achieves the best ROUGE scores, yet PPO 32b_AP receives the higher Elo rating. The authors' own hypothesis, that the GPT-4o judge favors outputs from models aligned using rubrics it helped generate, directly undermines the validity of the Elo-based comparisons in Tables 1 and 4, which use the same judge. Without human evaluation or an independent quality measure for the essay task, the central claim that MPO beats expert and oracle prompts is not established.
  2. [Section 3.2 and Appendix C] The reward-hacking mitigation claim is not supported by quantitative evidence. Appendix C provides a single anecdote of a degenerate response receiving a score of 3/5 and a subsequent rubric refinement, and Section 3.2 reports that MPO models also displayed instances of reward hacking during training. No measurement of hacking frequency or severity before versus after MPO, and no comparison with the PPO baselines, is provided, so the claim that MPO mitigates reward hacking remains unsubstantiated.
  3. [Section 2.2, Appendix B, and Section 3.2] The formal framework is disconnected from the implementation. Section 2.2 defines rewards as averages over observation partitions, R_t(o) = E_{s in O_{o,t}}[r(s)], and relies on the refinement property O_{o,t} ⊆ O_{o',t-1}. However, Section 3.2 explicitly states that raw RL reward values are used 'despite a slight deviation from our formal theoretical framework.' Since the implemented reward is a raw LLM score under an evolving textual rubric, neither the partition structure nor the refinement property is actually instantiated, so the theory does not justify the central mechanism as implemented.
  4. [Section 2.3 and Section 3.4] The meta reward model is never validated as a reliable judge of score inflation. The entire pipeline depends on the MRM's meta-analysis, meta-refinement, and meta-merging producing rubrics that make the reward model's scores track true output quality. The paper reports only that rubrics lengthen (Figure 5a) and that scores shift (Figure 5b); it never compares MRM-refined rubric scores against human judgments or an independent quality metric. The one attempt at validation, Appendix C, is an anecdote.
minor comments (5)
  1. [Section 3.2, Table 1] The right panel of Table 1 reports a 'failed training run' for PPO with the 72B RM and expert prompt, but the table entry is only marked with a value; annotating it explicitly (e.g., with a dagger) would make the failure easier to identify at a glance.
  2. [Figure 5b] The x-axis is labeled 'Checkpoint' while the legend lists 'Prompt Version' values (0, 1, 2, 5, 7, 20, 40); reconciling these two notions in the caption or axes would improve interpretability.
  3. [Appendix F.4] The iteration-20 ethical reasoning examples for 'Depth of Moral Insight' and 'Alignment with Established Ethical Principles' are near-duplicates of the 'Logical Coherence' example, with only the verdict tag changed; this suggests either template degeneration in the MRM or a copying error in the appendix and should be checked.
  4. [Section 3.5] The comparison against the oracle prompt is framed as a compute savings, but MPO training itself incurs substantial MRM inference overhead; reporting the total compute (PPO steps plus MRM calls) would make the efficiency claim more precise.
  5. [Sections 3.1 and 3.5] The 'expert' prompt used in Section 3.1 and the 'oracle' prompt used in Section 3.5 appear to be two different hand-built prompts, but the relationship between them is not explained; clarifying whether they are distinct resources would avoid confusion.

Circularity Check

1 steps flagged · score 2.0 of 10

No by-construction circularity in the main results; only a mild self-referential loop in the reward-hacking evidence.

  1. other [Section 3.2, reward-hacking paragraph; Appendix C]
    "However, these cases were identified and addressed during the MPO procedure. An example of such reward hacking and how it was mitigated through the MPO steps is provided in Appendix C."

    The only evidence offered for the claim that MPO detects and corrects reward hacking is the meta-analysis text generated by the MRM itself (Appendix C: 'The junior instructor’s assigned score of 3 out of 5 points for the student’s response is misleading...'). The MRM both defines the rubric that determines what is scored as hacking and produces the self-report that the loophole was closed. No independent human rating, external quality metric, or pre/post quantitative measure of hacking is provided, so the mitigation claim is self-referential rather than validated against an external yardstick.

full rationale

MPO is an empirical method paper. Its headline comparisons are anchored by external benchmarks (ROUGE on BillSum, accuracy on Scruples/MATH) and by GPT-4o Elo for essay; none of these is a fitted quantity or an equation that reduces to the method's inputs. The self-citations to Kim et al. [2025] supply the essay training set and the oracle prompt; the oracle prompt's strength is asserted from prior work, but the current paper's central MPO-vs-baseline comparisons do not depend on a uniqueness theorem or a derived identity. The only mild circular element is the reward-hacking-mitigation claim: the MRM's own meta-analysis is presented as evidence that reward hacking was detected and corrected, with no independent human or objective check that the refined rubric actually fixed the problem. That is a self-referential validation loop, but it does not collapse the main empirical claims, which are separately supported by objective metrics on three tasks.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

MPO introduces one new component, the MRM, and several hand-chosen hyperparameters that the results depend on. The theoretical derivation in Appendix B assumes observation partitions refine monotonically and that average rewards converge to the true reward, but the experiments use raw rewards, so the theory is not actually exercised. The most important unvalidated premise is that MRM refinements improve the fidelity of the RM to an independent notion of quality.

free parameters (4)
  • MPO refinement interval k = 10 for essay/ethics, 20 for summarization, 30 for math
    The frequency of rubric refinement is a hand-chosen hyperparameter that varies by task and directly controls how often the reward signal changes. No sensitivity analysis is provided.
  • Number of sampled responses n per MPO step = not specified in the paper
    The MRM consumes n student responses, n reference solutions, and n scores to produce refinements. The value of n is a free design choice that affects the quality and variance of the refined rubric, but it is not reported.
  • MRM output length cap (max_words) = not specified in the paper
    The meta prompts in Appendix E ask for refinement within a max_words budget. This budget is hand-chosen and is not reported for the experiments.
  • Initial evaluation prompts per task = one per task, shown in Appendix D
    Each task starts from a manually written initial prompt. The paper compares MPO against the iter0 version of these prompts, but the choice of this starting point is itself a design decision.
assumptions (4)
  • domain assumption A reward model with a refined rubric produces scores that are closer to an unobserved golden reward r(s) over time.
    This is the central assumption behind the partition-refinement formalism in Appendix B and behind the claim that MPO mitigates reward hacking. It is never directly validated.
  • domain assumption GPT-4o head-to-head Elo ratings are a valid proxy for output quality and human preference.
    All headline results in Tables 1, 2, and 4 are based on GPT-4o pairwise judgments. The summarization result, where GPT-4o prefers AutoPrompt despite lower ROUGE, calls this assumption into question.
  • ad hoc to paper The MRM's meta-prompts are task-agnostic and transferable across domains.
    The paper asserts that the meta-level prompts are designed to be broadly applicable, but no experiments isolate prompt transfer or show that the same prompts work without task-specific tuning.
  • domain assumption A single policy model, Qwen2-1.5B-Instruct, and single epoch of training are sufficient to demonstrate the framework's general effectiveness.
    The authors acknowledge this limitation in Appendix H. The empirical scope is narrow, and the claim of generality across models and training durations rests on this assumption.
invented entities (1)
  • Meta Reward Model (MRM), the 'senior instructor'
    purpose: A fixed pretrained LLM that monitors training context and rewrites the reward model's evaluation rubric at fixed intervals.
    The MRM is a new component introduced by the paper. It has no falsifiable handle outside the paper: the only evidence for its usefulness is the downstream task performance of policies trained with it, and its refinement quality is never measured independently.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models." pith.science (2026). https://pith.science/paper/UWZSZDCN

@misc{pith2026250420157,
  author       = {Pith},
  title        = {Pith review of: Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UWZSZDCN}},
  note         = {Machine review of arXiv:2504.20157}
}
read the original abstract

Reward-based alignment methods for large language models (LLMs) face two key limitations: vulnerability to reward hacking, where models exploit flaws in the reward signal; and reliance on brittle, labor-intensive prompt engineering when LLMs are used as reward models. We introduce Meta Policy Optimization (MPO), a framework that addresses these challenges by integrating a meta-reward model that dynamically refines the reward model's prompt throughout training. In MPO, the meta-reward model monitors the evolving training context and continuously adjusts the reward model's prompt to maintain high alignment, providing an adaptive reward signal that resists exploitation by the policy. This meta-learning approach promotes a more stable policy optimization, and greatly reduces the need for manual reward prompt design. It yields performance on par with or better than models guided by extensively hand-crafted reward prompts. Furthermore, we show that MPO maintains its effectiveness across diverse tasks, from essay writing to mathematical reasoning, without requiring specialized reward designs. Beyond standard RLAIF, MPO's meta-learning formulation is readily extensible to higher-level alignment frameworks. Overall, this method addresses theoretical and practical challenges in reward-based RL alignment for LLMs, paving the way for more robust and adaptable alignment strategies. The code and data can be accessed at: https://github.com/minnesotanlp/mpo

Figures

Figures reproduced from arXiv: 2504.20157 by the authors.

Figure 1
Figure 1. In standard RLAIF, the reward model used during proximal policy optimization (PPO) remains fixed throughout RL alignment. In con￾trast, MPO framework (in green) introduces a meta reward model that dynamically evolves the reward model based on the current training context, in￾cluding the task prompt, sampled generations with associated scores, and the latest evaluation prompt. MPO leverages this contextual informatio… view at source ↗
Figure 2
Figure 2. Dimensions of Evaluative Thinking: Depth and Breadth. While ECB through ET provides a foundation for improv￾ing the RM, it does not fully capture the dynamics of RL, where learning is driven by signals from an ECB-enhanced RM. To address this gap, we introduce a mathematical framework that formalizes how the RM evolves under the influence of ET and how this evolving RM can be inte￾grated into the training process of… view at source ↗
Figure 3
Figure 3. The three Meta Policy Optimization steps— [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Training curves for eight essay-writing policy models, each pairing different-sized reward models (RM) and meta-reward models (MRM). The RL Reward and Normalized RL Reward plots show how reward values evolve over global batch steps, capturing the quality of generated r…
Figure 5
Figure 5. Figure 5: (a) Mean length of rubric items for essay writing task across the MPO-aligned models. (b) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparison between initial and last versions of MPO rubrics for all tasks. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Comparison between hand-crafted and MPO-evolved rubrics for essay writing task. [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Comparison of discourse motifs found in rubric [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]
Figure 9
Figure 9. Figure 9: Comparison of discourse motifs found in rubric prompts at the initial stage, after the first, [PITH_FULL_IMAGE:figures/full_fig_p036_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Training single-layer attention with squared regret loss has stationary points that implement smoothed fictitious play (external regret) and, via a new swap-regret loss, the Blum–Mansour no-swap-regret algorithm.

  2. Improving Generalization Robustness of Multimodal RLVR

    cs.AI 2026-08 conditional novelty 6.0 of 10

    PIRL, a trinary reward plus embedding-space adversarial consistency regularizer, reduces prompt-rewording degradation of RLVR-trained multimodal LLMs to about 1% average, versus about 3% for GRPO.

Reference graph

Works this paper leans on

78 extracted references · 40 canonical work pages · cited by 2 Pith papers

  1. [1]

    Ahmadian, C

    A. Ahmadian, C. Cremer, M. Gall \'e , M. Fadaee, J. Kreutzer, O. Pietquin, A. \"U st \"u n, and S. Hooker. Back to basics: Revisiting REINFORCE -style optimization for learning from human feedback in LLM s. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: ...

    doi:10.18653/v1/2024.acl-long.662 2024

  2. [2]

    Amodei, C

    D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in ai safety, 2016 a

    work page 2016

  3. [3]

    Amodei, C

    D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in ai safety, 2016 b . URL https://arxiv.org/abs/1606.06565

    arXiv 2016

  4. [4]

    Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  5. [5]

    Buckley, T

    J. Buckley, T. Archibald, M. Hargraves, and W. M. Trochim. Defining and teaching evaluative thinking: Insights from research on critical thinking. American Journal of Evaluation, 36 0 (3): 0 375--388, 2015

    work page 2015

  6. [6]

    L. Chen, C. Zhu, D. Soselia, J. Chen, T. Zhou, T. Goldstein, H. Huang, M. Shoeybi, and B. Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf, 2024 a . URL https://arxiv.org/abs/2402.07319

    arXiv 2024

  7. [7]

    Z. Chen, Y. Deng, H. Yuan, K. Ji, and Q. Gu. Self-play fine-tuning convertsweak language models to strong language models. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024 b

    work page 2024

  8. [8]

    Chiang, L

    W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024. URL https://arxiv.org/abs/2403.04132

    arXiv 2024

Show all 78 references
  1. [9]

    Coste, U

    T. Coste, U. Anwar, R. Kirk, and D. Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=dcjtMYkpXx

    2024

  2. [10]

    E. De Bono. The Use of Lateral Thinking. Pelican books. Penguin Books, 1971. ISBN 9780140214468. URL https://books.google.com/books?id=2Fd-AAAAMAAJ

    1971

  3. [11]

    DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li...

    2025 arXiv

  4. [12]

    Denison, M

    C. Denison, M. MacDiarmid, F. Barez, D. Duvenaud, S. Kravec, S. Marks, N. Schiefer, R. Soklaski, A. Tamkin, J. Kaplan, et al. Sycophancy to subterfuge: Investigating reward-tampering in large language models. arXiv preprint arXiv:2406.10162, 2024

    2024 arXiv

  5. [13]

    M. Ding, S. Chakraborty, V. Agrawal, Z. Che, A. Koppel, M. Wang, A. Bedi, and F. Huang. SAIL : Self-improving efficient online alignment of large language models. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models, 2024. URL https://openreview.net/forum?id=9...

    2024

  6. [14]

    Efklides

    A. Efklides. Metacognition and affect: What can metacognitive experiences tell us about the learning process? Educational research review, 1 0 (1): 0 3--14, 2006

    2006

  7. [15]

    Eisenstein, C

    J. Eisenstein, C. Nagpal, A. Agarwal, A. Beirami, A. N. D'Amour, K. D. Dvijotham, A. Fisch, K. A. Heller, S. R. Pfohl, D. Ramachandran, P. Shaw, and J. Berant. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. In First Conference on Langu...

    2024

  8. [16]

    Ethayarajh, W

    K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  9. [17]

    Everitt, M

    T. Everitt, M. Hutter, R. Kumar, and V. Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese, 198 0 (Suppl 27): 0 6435--6467, 2021

    2021

  10. [18]

    J. H. Flavell. Metacognition and cognitive monitoring: A new area of cognitive--developmental inquiry. American psychologist, 34 0 (10): 0 906, 1979

    1979

  11. [19]

    Fluri, L

    L. Fluri, L. Lang, A. Abate, P. Forr \'e , D. Krueger, and J. Skalse. The perils of optimizing learned reward functions: Low training error does not guarantee low regret. arXiv preprint arXiv:2406.15753, 2024

    2024 arXiv

  12. [20]

    J. Fu, X. Zhao, C. Yao, H. Wang, Q. Han, and Y. Xiao. Reward shaping to mitigate reward hacking in rlhf. arXiv preprint arXiv:2502.18770, 2025

    2025 arXiv

  13. [21]

    L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machin...

    2023

  14. [22]

    Hamner, J

    B. Hamner, J. Morgan, lynnvandev, M. Shermis, and T. V. Ark. The hewlett foundation: Automated essay scoring. https://kaggle.com/competitions/asap-aes, 2012. Kaggle

    2012

  15. [23]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and B...

    2021

  16. [24]

    Z. M. Kim, K. Lee, P. Zhu, V. Raheja, and D. Kang. Threads of subtlety: Detecting machine-generated texts through discourse motifs. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volum...

    2024 doi

  17. [25]

    Z. M. Kim, A. Ramachandran, F. Tavazoee, J.-K. Kim, O. Rokhlenko, and D. Kang. Align to structure: Aligning large language models with structural information, 2025. URL https://arxiv.org/abs/2504.03622

    2025

  18. [26]

    Kornilova and V

    A. Kornilova and V. Eidelman. B ill S um: A corpus for automatic summarization of US legislation. In L. Wang, J. C. K. Cheung, G. Carenini, and F. Liu, editors, Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 48--56, Hong Kong, China, Nov. 2019. Associ...

    2019 doi

  19. [27]

    Krakovna, J

    V. Krakovna, J. Uesato, V. Mikulik, M. Rahtz, T. Everitt, R. Kumar, Z. Kenton, J. Leike, and S. Legg. Specification gaming: the flip side of ai ingenuity. DeepMind Blog, 3, 2020

    2020

  20. [28]

    L. L. D. Langosco, J. Koch, L. D. Sharkey, J. Pfau, and D. Krueger. Goal misgeneralization in deep reinforcement learning. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning,...

    2022

  21. [29]

    H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35: 0 21314--21328, 2022

    2022

  22. [31]

    E. Levi, E. Brosh, and M. Friedmann. Intent-based prompt calibration: Enhancing prompt optimization with synthetic boundary cases, 2024

    2024

  23. [32]

    C.-Y. Lin. ROUGE : A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74--81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013/

    2004

  24. [33]

    H. Liu, C. Sferrazza, and P. Abbeel. Chain of hindsight aligns language models with feedback. In The Twelfth International Conference on Learning Representations, 2024 a . URL https://openreview.net/forum?id=6xfe4IVcOu

    2024

  25. [34]

    T. Liu, W. Xiong, J. Ren, L. Chen, J. Wu, R. Joshi, Y. Gao, J. Shen, Z. Qin, T. Yu, D. Sohn, A. Makarova, J. Liu, Y. Liu, B. Piot, A. Ittycheriah, A. Kumar, and M. Saleh. Rrm: Robust reward model training mitigates reward hacking. ArXiv, abs/2409.13156, 2024 b . URL https://ap...

    2024 arXiv

  26. [35]

    C. G. Lord, L. Ross, and M. R. Lepper. Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of personality and social psychology, 37 0 (11): 0 2098, 1979

    1979

  27. [36]

    Lourie, R

    N. Lourie, R. L. Bras, and Y. Choi. Scruples: A corpus of community ethical judgments on 32,000 real-life anecdotes. arXiv e-prints, 2020

    2020

  28. [37]

    W. C. Mann and S. A. Thompson. Rhetorical structure theory: A theory of text organization. University of Southern California, Information Sciences Institute Los Angeles, 1987

    1987

  29. [38]

    C. B. McCormick. Metacognition and learning. Handbook of psychology, pages 79--102, 2003

    2003

  30. [39]

    Y. Meng, M. Xia, and D. Chen. Sim PO : Simple preference optimization with a reference-free reward. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=3Tzcot1LKb

    2024

  31. [40]

    Metcalfe and N

    J. Metcalfe and N. Kornell. A region of proximal learning model of study time allocation. Journal of Memory and Language, 52 0 (4): 0 463--477, 2005. ISSN 0749-596X. doi:https://doi.org/10.1016/j.jml.2004.12.001. URL https://www.sciencedirect.com/science/article/pii/S0749596X0...

    2005 doi

  32. [41]

    Y. Miao, S. Zhang, L. Ding, R. Bao, L. Zhang, and D. Tao. Info RM : Mitigating reward hacking in RLHF via information-theoretic reward modeling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=3XnBVK9sD6

    2024

  33. [42]

    Y. Miao, S. Zhang, L. Ding, Y. Zhang, L. Zhang, and D. Tao. The energy loss phenomenon in rlhf: A new perspective on mitigating reward hacking, 2025. URL https://arxiv.org/abs/2501.19358

    2025 arXiv

  34. [43]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions ...

    2022

  35. [44]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022 b

    2022

  36. [45]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions ...

    2022

  37. [46]

    A. Pan, K. Bhatia, and J. Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=JYtwGwIL7ye

    2022

  38. [47]

    R. Y. Pang, V. Padmakumar, T. Sellam, A. Parikh, and H. He. Reward gaming in conditional text generation. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pag...

    2023 doi

  39. [48]

    R. Y. Pang, W. Yuan, K. Cho, H. He, S. Sukhbaatar, and J. Weston. Iterative reasoning preference optimization. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 116...

    2024

  40. [49]

    C. Park, M. Liu, D. Kong, K. Zhang, and A. Ozdaglar. Rlhf from heterogeneous feedback via personalization and preference aggregation. arXiv preprint arXiv:2405.00254, 2024

    2024 arXiv

  41. [50]

    C. Park, S. Han, X. Guo, A. Ozdaglar, K. Zhang, and J.-K. Kim. Maporl: Multi-agent post-co-training for collaborative large language models with reinforcement learning. arXiv preprint arXiv:2502.18439, 2025

    2025 arXiv

  42. [51]

    Perez, S

    E. Perez, S. Ringer, K. Lukosiute, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, A. Jones, A. Chen, B. Mann, B. Israel, B. Seethor, C. McKinnon, C. Olah, D. Yan, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, G. Khundadze, J. Kernion, J. ...

    2023

  43. [52]

    Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, ...

    2025 arXiv

  44. [53]

    A. Rame, N. Vieillard, L. Hussenot, R. Dadashi-Tazehozi, G. Cideron, O. Bachem, and J. Ferret. WARM : On the benefits of weight averaged reward models. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the ...

    2024

  45. [54]

    S. Saha, X. Li, M. Ghazvininejad, J. Weston, and T. Wang. Learning to plan & reason for evaluation with thinking-llm-as-a-judge, 2025. URL https://arxiv.org/abs/2501.18099

    2025 arXiv

  46. [55]

    Saito, A

    K. Saito, A. Wachi, K. Wataoka, and Y. Akimoto. Verbosity bias in preference labeling by large language models, 2023. URL https://arxiv.org/abs/2310.10076

    2023 arXiv

  47. [57]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024 b . URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  48. [58]

    Sharma, M

    M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, E. DURMUS, Z. Hatfield-Dodds, S. R. Johnston, S. M. Kravec, T. Maxwell, S. McCandlish, K. Ndousse, O. Rausch, N. Schiefer, D. Yan, M. Zhang, and E. Perez. Towards understanding sycophancy in language models. ...

    2024

  49. [59]

    Singhal, T

    P. Singhal, T. Goyal, J. Xu, and G. Durrett. A long way to go: Investigating length correlations in RLHF , 2024. URL https://openreview.net/forum?id=sNtDKdcI1f

    2024

  50. [60]

    J. M. V. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger. Defining and characterizing reward gaming. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=yb3HOXO3lX2

    2022

  51. [61]

    Stiennon, L

    N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, vol...

    2020

  52. [62]

    K. Tian, E. Mitchell, H. Yao, C. D. Manning, and C. Finn. Fine-tuning language models for factuality. In The Twelfth International Conference on Learning Representations, 2024

    2024

  53. [63]

    H. Tran, C. Glaze, and B. Hancock. Iterative dpo alignment. Technical report, Snorkel AI, 2023. URL https://snorkel.ai/new-benchmark-results-demonstrate-value-of-snorkel-ai-approach-to-llm-alignment

    2023

  54. [64]

    M. V. Veenman, B. H. Van Hout-Wolters, and P. Afflerbach. Metacognition and learning: Conceptual and methodological considerations. Metacognition and learning, 1: 0 3--14, 2006

    2006

  55. [65]

    von Werra, Y

    L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Rasul, and Q. Gallouédec. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl, 2020

    2020

  56. [66]

    Y. Wang, H. Ivison, P. Dasigi, J. Hessel, T. Khot, K. Chandu, D. Wadden, K. MacMillan, N. A. Smith, I. Beltagy, and H. Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources. In Thirty-seventh Conference on Neural Information Processing ...

    2023

  57. [67]

    J. Wen, R. Zhong, A. Khan, E. Perez, J. Steinhardt, M. Huang, S. R. Bowman, H. He, and S. Feng. Language models learn to mislead humans via RLHF . In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=xJljiPE6dg

    2025

  58. [68]

    T. Wu, W. Yuan, O. Golovneva, J. Xu, Y. Tian, J. Jiao, J. Weston, and S. Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594, 2024 a

    2024 arXiv

  59. [69]

    Y. Wu, Z. Sun, H. Yuan, K. Ji, Y. Yang, and Q. Gu. Self-play preference optimization for language model alignment. In Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning, 2024 b . URL https://openreview.net/forum?id=Z1PDdGekgn

    2024

  60. [70]

    Xiong, H

    W. Xiong, H. Dong, C. Ye, Z. Wang, H. Zhong, H. Ji, N. Jiang, and T. Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, 2024. URL https://arxiv.org/abs/2312.11456

    2024 arXiv

  61. [71]

    J. Xu, A. Lee, S. Sukhbaatar, and J. Weston. Some things are more cringe than others: Iterative preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682, 2023

    2023 arXiv

  62. [72]

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Yang, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. ...

    2024 arXiv

  63. [73]

    W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. Weston. Self-rewarding language models, 2025. URL https://arxiv.org/abs/2401.10020

    2025 arXiv

  64. [74]

    Zhang, C

    C. Zhang, C. Tang, D. Chong, K. Shi, G. Tang, F. Jiang, and H. Li. TS -align: A teacher-student collaborative framework for scalable iterative finetuning of large language models. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Findings of the Association for Computation...

    2024 doi

  65. [75]

    Zhang, Z

    S. Zhang, Z. Chen, S. Chen, Y. Shen, Z. Sun, and C. Gan. Improving reinforcement learning from human feedback with efficient reward model ensemble, 2024 b . URL https://arxiv.org/abs/2401.16635

    2024 arXiv

  66. [76]

    Zheng, L

    L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng. Sglang: Efficient execution of structured language model programs, 2024. URL https://arxiv.org/abs/2312.07104

    2024 arXiv

  67. [78]

    D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences, 2020. URL https://arxiv.org/abs/1909.08593

    2020 arXiv

  68. [79]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  69. [80]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  70. [81]

    Back to Basics: Revisiting REINFORCE -Style Optimization for Learning from Human Feedback in LLM s

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    1971 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.