Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

Reward hacking in language-model RL is reduced by penalizing the gradient norm, which steers training to flatter, more accurate reward regions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 22:01 UTC pith:R6APKS7T

load-bearing objection GR beats KL across the board in sensible experiments; but the flatness-to-accuracy theory is a one-sided lower bound the authors over-interpret, so treat the mechanism as unproven. the 3 major comments →

arxiv 2602.18037 v2 pith:R6APKS7T submitted 2026-02-20 cs.LG cs.AIcs.CL

Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards

Johannes Ackermann , Michael Noukhovitch , Takashi Ishida , Masashi Sugiyama This is my paper
classification cs.LG cs.AIcs.CL
keywords reward hackinggradient regularizationRLHFRLVRKL penaltyflat minimaproxy reward accuracyLLM-as-a-judge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that reward hacking is often the exploitation of overly sharp peaks in a proxy reward, and that the standard fix—staying close to the reference policy with a KL penalty—is neither necessary nor the best choice. The central claim is that adding a penalty on the squared norm of the policy gradient during RL training biases updates toward flat optima, and that flat optima of the proxy reward are ones where the reward model stays accurate. The paper supports this with a theoretical chain: parameter-space flatness implies robustness in action space, and non-robust policies incur a provable excess in the Bradley-Terry loss used to measure reward accuracy. Empirically, the paper shows gradient norm tracks both sharpness and reward-model accuracy, that reference-reset training works by the same implicit mechanism, and that explicit gradient regularization outperforms the KL penalty across RLHF and math RLVR tasks, including preventing format hacking and judge hacking. A sympathetic reader would care because it points to a replacement for the KL penalty that does not constrain the policy as tightly while achieving better final behavior.

Core claim

The paper sets out to show that a policy update rule of the form maximize expected proxy reward minus a weighted squared norm of the policy gradient—gradient regularization—produces policies that land on flat optima, and that flat optima of a proxy reward are exactly the optima at which the proxy remains accurate. The formal argument has three steps: gradient regularization biases optimization toward flat parameter-space maxima; flat parameter maxima imply policies that rarely sample a pair of near-identical actions whose proxy reward differs sharply; and under a Lipschitz true reward, every such 'sharp pair' forces an excess Bradley-Terry loss, so fewer sharp pairs means the proxy reward is

What carries the argument

The central object is gradient regularization (GR): adding a term proportional to the squared norm of the policy gradient to the RL objective so that updates resist steep reward landscapes. The proposed implementation uses a finite-difference estimate of the gradient of the gradient norm, reusing the same sampled actions for both gradient computations to keep overhead small. The theoretical bridge is a pairwise-robustness definition: a policy is (δ,K,ρ)-robust when near-identical action pairs rarely have proxy-reward gaps larger than K; the paper proves flat parameter maxima imply such robustness, and that non-robustness lower-bounds the excess Bradley-Terry loss of the proxy reward. That ch

Load-bearing premise

The load-bearing premise is the existence of a meaningful distance on the action space in which the true reward is Lipschitz and the proxy reward is smooth; the paper itself notes this is hard to define for discrete token sequences, so if no such distance exists the theoretical guarantee does not cover language models.

What would settle it

A run where the proxy reward is deliberately inaccurate on a broad, flat basin: if the policy's gradient norm stays small while true reward still collapses, the claimed flatness-accuracy link fails. Equivalently, a single trajectory where reward hacking begins while the gradient norm is decreasing would contradict the paper's observed correlation.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Training with explicit gradient regularization outperforms a KL penalty on judged win rate in RLHF and can replace the KL term altogether.
  • In rule-based math rewards, it stops the policy from overfitting to easier formatting and correctness components, improving final test accuracy.
  • With an LLM-as-a-judge reward, it prevents the model from formatting its way to high scores and preserves true accuracy better than KL regularization.
  • During training, gradient norm is empirically coupled to both parameter sharpness and reward-model Bradley-Terry loss, giving a practical early signal for reward hacking.
  • Resetting the KL reference works because it induces implicit gradient regularization, not simply because it effectively weakens the KL penalty.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The argument suggests gradient regularization could also stabilize agentic and code RLVR, where rewards are sparse and noisy and the KL term is often dropped; the paper does not test this.
  • A risk left open by the paper is that GR may favor flat but semantically wrong proxy optima; pairing GR with periodic reward-model updates is a natural extension.
  • Because the finite-difference implementation reuses actions, it biases the gradient estimate; an importance-sampling-corrected version is a direct testable refinement.
  • If a semantic distance on token sequences can be defined, for instance in the LM's hidden representation, the theoretical chain would extend to discrete actions; that is the key open theoretical target.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes gradient regularization (GR) as an alternative to KL penalties for preventing reward hacking in RLHF/RLVR. It argues that flat maxima of the proxy reward correspond to regions where the proxy reward is more accurate, and that adding a gradient-norm penalty to the policy objective biases optimization toward such flat regions. The theoretical argument proceeds in three steps: GR favors flat parameter-space optima, flat parameter optima imply action-space pairwise robustness under a Gaussian policy, and pairwise robustness implies a lower bound on excess Bradley-Terry loss. The paper then interprets Reference Resets as implicit GR, proposes an explicit finite-difference GR implementation, and reports experiments on TL;DR, AlpacaFarm, GSM8K, MATH, and LLM-as-a-Judge settings, concluding that GR can match or outperform KL penalties and in some cases replace them.

Significance. If the empirical findings are reproducible, this is a practically useful contribution: it offers a simple, low-overhead regularizer that may reduce reward hacking and relax the need for KL constraints in RL post-training. The empirical breadth is a genuine strength, especially the controlled gold-reward evaluation on TL;DR and the rule-based/LLM-judge math experiments. However, the theoretical contribution as currently stated is not established. The central flatness-to-accuracy mechanism rests on a one-sided lower bound that does not imply the directional claim made in the paper, and the formal assumptions exclude the discrete token-level action space of the actual LMs. The paper's value would be materially increased by scaling back the theoretical claims and by reporting seed/error-bar information for the headline tables.

major comments (3)
  1. [Section 3, Proposition 3.4 (Eq. 10)] The step from Eq. (10) to the claim that reducing P(S_K,δ) or K 'obtain[s] a policy that induces a smaller excess BT loss' is not valid. Eq. (10) is a lower bound: L_BT(R̃) − L_BT(R*) ≥ 2(σ(K) − σ(Lδ))² P(S_K,δ). Shrinking the right-hand side does not constrain the left-hand side. A constant proxy reward has P(S_K,δ)=0 yet can incur arbitrarily large excess BT loss; conversely, large P(S_K,δ) need not imply large excess loss if the bound is loose. The limitation stated in Section 3 ('we do not show whether or not GR prevents convergence to flat but incorrect regions') is narrower than this issue: even the sharpness component of the mechanism is not established. Consequently, the abstract's and Section 6's claim that GR 'maintains reward accuracy' through flatness is unsupported by the derivation as written.
  2. [Section 3 / Assumptions B.2–B.4, Appendix B] Propositions 3.3 and 3.4 rely on a Gaussian policy with fixed covariance, a metric on the action space, and L-Lipschitz/β-smooth rewards in that metric. The paper itself acknowledges, at the end of Section 3, that 'our theoretical argument assumes a Gaussian policy and requires a distance function, which is difficult to define for LMs.' Since the empirical method is applied to autoregressive LMs generating discrete token sequences, the theoretical derivation does not cover the actual setting. The correlations in Section 4 and Figure 3 are suggestive, but they do not supply the missing parameter-space-to-action-space link for LMs. Without a meaningful action metric, the flatness-to-accuracy mechanism remains unverified in the domain where the method is recommended.
  3. [Tables 1, 2, and 3] The headline empirical claim that 'GR consistently performs best' and 'can replace the KL penalty' is reported without variance information. Tables 1–3 show single runs per configuration; Figure 5 provides three seeds only for the 1.5B AlpacaFarm setting, where the GR advantage over KL is small in some RM-accuracy regimes. Given the modest margins in several comparisons (e.g., 29.2% vs 27.6% in Table 2 for 1.5B; 50.9% vs 44.3% in Table 3 for rule-based GSM8K at 0.5B), the consistency claim needs error bars or additional seeds. I am not asking for large-scale multi-seed runs across every cell, but the paper should quantify run-to-run variance for the main comparisons, or explicitly state which numbers are single runs and temper the strength of the claim accordingly.
minor comments (4)
  1. [Table 3 heading] 'Test accuracies on GSM8K after with GRPO' is ungrammatical; should read 'after training with GRPO'.
  2. [Eq. (4)] The notation Δϕ∥∇ϕL(ϕ)∥² is ambiguous. Suggest writing the finite-difference operator explicitly, e.g., (∇L(ϕ+ε∇L(ϕ))−∇L(ϕ))/ε, and reserving Δ for the parameter update.
  3. [Section 4] The claim that Reference Resets 'implicitly regularize the gradient' is supported by an empirical correlation rather than a derivation. Please mark this as a hypothesis or provide the derivation; as written, it reads as a stronger statement than the evidence supports.
  4. [Figure 4] The x-axis label uses θ for the policy parameters while the text uses ϕ; please unify the notation.

Circularity Check

0 steps flagged

No circularity: the flatness-to-accuracy chain is a formal bound plus independently measured correlations; the acknowledged inference gap is a correctness concern, not a reduction to inputs.

full rationale

I find no equation-level circularity. The claimed derivation is: (i) GR biases optimization toward flat parameter-space optima (cited to external work: Zhao et al. 2022; Karakida et al. 2023); (ii) flat parameter optima imply action-space pairwise robustness (Prop. 3.3, built on Lee & Yoon 2025); and (iii) sharp action pairs incur excess BT loss (Prop. 3.4). None of these steps defines its conclusion into its premise: Eq. (7) penalizes the gradient norm of the policy return, while reward accuracy is measured as the BT loss L_BT(θ,φ); the gradient norm is not constructed from L_BT. The empirical correlations in Figures 3, 8, and 11 are measurements of independently obtained quantities, not fitted parameters renamed as predictions. The one genuinely load-bearing gap is logical rather than circular: Prop. 3.4 (Eq. 10) is only a lower bound, yet Section 3 asserts, 'By changing the policy φ to decrease the ratio of violating pairs P(S_K,δ) or the magnitude of the violations K, we obtain a policy that induces a smaller excess BT loss.' Decreasing a lower bound does not force the left-hand side down. The paper's own Limitations admit this scope: 'we only address excess BT loss incurred by overly sharp maxima... We do not show whether or not GR prevents convergence to flat but incorrect regions.' That under-support is a correctness risk, not a circularity. The only overlapping-author citation (Ackermann et al. 2025) is used for a historical note and dataset filtering, not to justify the central claim; per the rules it does not raise the circularity score.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No new physical or conceptual entities are invented. The free parameters are ordinary hyperparameters of the proposed method and baselines. The main axiom load is the continuous-action smoothness/Lipschitz framework, which the paper itself flags as not directly applicable to discrete token-level action spaces.

free parameters (4)
  • GR strength γ = 1e-3 (math tasks); 1e-2–0.1 (RLHF, after grid search)
    Controls the trade-off between reward maximization and gradient regularization; tuned per setting in RLHF, fixed conservatively in reasoning experiments.
  • Finite-difference perturbation ε = 1e-3
    Chosen after initial RLHF experiments and used throughout; the paper reports it did not need per-task tuning.
  • KL penalty strength β (baseline) = 0.03–0.4 across experiments
    Baseline hyperparameter, tuned per experiment to compare against GR.
  • Reference-reset interval R = 200 gradient steps (default)
    A method hyperparameter; values 25–400 were tested, with larger R generally better but more expensive.
axioms (5)
  • domain assumption Gaussian policy with fixed covariance and full row-rank Jacobian (Assumption B.2)
    Used in Proposition 3.3 and Appendix B.5 to relate parameter-space perturbations to action-space shifts; does not hold for autoregressive language models.
  • domain assumption Proxy reward is β-smooth in action space (Assumption B.3)
    Invoked in Lemma B.6/B.7 to control gradient norms; no evidence is given that learned reward models or LLM judges are β-smooth.
  • domain assumption True reward is L-Lipschitz in a meaningful action-space distance (Assumption B.4)
    Needed for Proposition 3.4 and the excess-BT-loss bound; the paper acknowledges this distance is difficult to define for discrete LM action spaces.
  • standard math GR biases optimization toward flat minima in parameter space
    Treated as known, citing Zhao et al. 2022 and reproducing the Lipschitz argument in Appendix B.3.
  • domain assumption Flat minima generalize better and, by extension, correlate with proxy-reward accuracy
    Imported from supervised learning (Hochreiter & Schmidhuber 1997); the paper provides an empirical correlation for RLHF but the transfer from parameter-space flatness to BT-loss accuracy is a substantive assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 28531 in / 10689 out tokens · 92953 ms · 2026-08-02T22:01:15.906651+00:00 · methodology

0 comments
read the original abstract

Reinforcement Learning from Human Feedback (RLHF) or Verifiable Rewards (RLVR) are two key steps in the post-training of modern Language Models (LMs). A common problem is reward hacking, where the policy may exploit inaccuracies of the reward and learn an unintended behavior. Most previous works address this by limiting the policy update with a Kullback-Leibler (KL) penalty towards a reference model. We propose a different framing: Train the LM in a way that biases policy updates towards regions in which the reward is more accurate. First, we derive a theoretical connection between the accuracy of a reward model and the flatness of an optimum at convergence. Gradient regularization (GR) can then be used to bias training to flatter regions and thereby maintain reward model accuracy. We confirm these results by showing that the gradient norm and reward accuracy are empirically correlated in RLHF. We then empirically show that Reference Resets of the KL penalty find flatter regions with a higher reward accuracy. We further improve on this by proposing to use explicit GR with an efficient finite-difference estimate. Empirically, GR performs better than a KL penalty across a diverse set of RL experiments with LMs. GR achieves a higher GPT-judged win-rate in RLHF, avoids overly focusing on the format in rule-based math rewards, and prevents hacking the judge in LLM-as-a-Judge math tasks.

Figures

Figures reproduced from arXiv: 2602.18037 by Johannes Ackermann, Masashi Sugiyama, Michael Noukhovitch, Takashi Ishida.

Figure 1
Figure 1. Figure 1: We argue that reward hacking often corresponds to exploiting sharp maxima in action space, as illustrated by the conceptual figure (left). For example, an LLM judge may be confused and assign a high reward to a wrong answer with specific formatting. In the LLM-as-a-Judge training run shown on the right, the increase in gradient norm coincides with reward hacking, resulting in true reward collapsing. By usi… view at source ↗
Figure 2
Figure 2. Figure 2: Conceptual illustration of our theoretical argument: (left) Regularizing the gradient norm biases optimization toward flat basins in parameter space, and (right) under action-smoothness, a flat maximum makes δ-close pairs unlikely to have a reward gap larger than K, i.e. decreases the probability of overly sharp action pairs a1, a2 : ∥a1 − a2∥ ≤ δ, |Re(s, a1) − Re(s, a2)| > K. Under the assumption of a Lip… view at source ↗
Figure 4
Figure 4. Figure 4: Reference Resets outperform all possible weights β of KL penalty. Oracle evaluation (Gold Model Score) vs KL from initial model for Pythia 1B on the TL;DR test set [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: GR prevents overly focusing on formatting reward. Qwen2.5-0.5B on GSM8K, test set accuracy (left) and formatting reward (right), the dashed line shows the optimal formatting reward. Without regularization, the policy focuses overly on the formatting reward, resulting in worse accuracy. regime where the RM is more accurate than on its training distribution, see Appendix D.2. 5.2. GR Prevents Focus on Easy R… view at source ↗
Figure 8
Figure 8. Figure 8: Gradient norm increases as reward hacking occurs. True accuracy and gradient norm when training Qwen2.5 0.5B￾Instruct on GSM8K with Qwen2.5 1.5B-Instruct LLM judge. PR. To investigate, we re-run the previous GSM8k experi￾ments but replace the rule-based correctness reward with an LLM judge based on Qwen2.5 1.5B-Instruct (Qwen et al., 2025). The judge receives the problem description, true answer, model res… view at source ↗
Figure 7
Figure 7. Figure 7: GR prevents reward hacking with LLM-as-a-Judge Results when training Qwen2.5-0.5B-Inst. on GSM8K with Qwen2.5 1.5B-Inst. as judge. Left: LLM-Judge and rule-based ac￾curacy over time, showing reward hacking without regularization. train a Qwen-2.5-1.5B-Instruct model with a rule-based reward on MATH (Hendrycks et al., 2021). The base model achieves 46.3% pass@1 accuracy and GRPO+GR (57.6%) clearly outperfor… view at source ↗
Figure 9
Figure 9. Figure 9: Implementation of finite-difference gradient regularization with GRPO in PyTorch Since we can use the inequality DKL(Bernoulli(p); Bernoulli(q)) ≥ 2|p−q| 2 for p, q ∈ (0, 1), combining with equation 26 yields: DKL(Bernoulli(p); Bernoulli(q)) ≥ 2(σ(K) − σ(Lδ))2 for every (s, a1, a2) ∈ SK,δ . (27) Let 1S := 1{Z ∈ SK,δ} indicate membership of SK,δ. We can make equation 27 valid for all (s, a1, a2) by multiply… view at source ↗
Figure 11
Figure 11. Figure 11: Strong GR can decrease BT loss below initial value. BT loss LBT(ϕ, θ) during training of a Qwen 2.5 0.5B model on the TL;DR task with GR, using the Gold reward model. 2 4 6 Learning Rate 1e 6 22 24 26 28 GPT 4.1-Nano Winrate (%) SFT Model GR KL [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 13
Figure 13. Figure 13: GR allows usage of cheaper judges to reach same performance. Test accuracy when training a Qwen2.5-0.5B model on GSM8K with different judges. D. Additional Experiments D.1. Gradient Regularization γ, ε, and learning rate sweeps To evaluate the sensitivity of GR to the hyper-parameters controlling strength of the regularization γ and the strength of the perturbation ε, we performed experiments on the Qwen2… view at source ↗
Figure 14
Figure 14. Figure 14: Pythia 1B on TL;DR task. Left: Reference Resets schedule ablation. A scheduled β performs better than a constant value, however, it does not match the performance of full Reference Resets. Right: Steps per reset R for GRPO + Reference Resets. A larger R is generally beneficial, but requires significantly more gradient steps. D.3. LLM judge ablation In the main text we are using Qwen2.5 1.5B-Instruct as ju… view at source ↗
Figure 15
Figure 15. Figure 15: Proxy and gold reward for no resets (blue, β = 0.06) and resets (orange, β = 0.3). The proxy reward should thus match after 5 resets which here with R = 200 corresponds to 1200 training steps. The achieved proxy reward is relatively similar at training step t = 1200, however, the gold reward with resets is significantly higher. \text{initial}} = 500, \text{spent_on_clothes}} = 20/100 \times 500 = 100, \te… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Multimodal Reward Hacking in Reinforcement Learning

    cs.AI 2026-07 conditional novelty 6.5

    Imperfect multimodal RL rewards systematically create new failures (NRFR > RHR); scaling and answer-aware rewards help but do not eliminate hacking, and unreliable visual verifiers actively increase it.

  2. Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priors

    cs.LG 2026-06 unverdicted novelty 6.0

    Argues for shifting to diagnosis-driven tension management of offline priors in online RL, supported by a framework on prior roles, experiments showing help-or-hurt reversals, and cross-domain evidence.

  3. Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Models

    cs.LG 2026-05 unverdicted novelty 5.0

    Trusted-direction projection constrains RL gradient updates in language models to a low-dimensional clean subspace, reducing reward hacking on mathematical reasoning tasks.

Reference graph

Works this paper leans on

8 extracted references · 5 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Proxy and gold reward for no resets (blue, β = 0.06) and resets (orange, β = 0.3)

    Given values: \\[ 23 Gradient Regularization Prevents Reward Hacking 0 500 1000 1500 2000 Gradient Steps 0 1 2 3 4RM Model Score GRPO GRPO+Reset 0 500 1000 1500 2000 Gradient Steps 14 13 12 11 10 9 8 7 Gold Model Score 0 500 1000 1500 2000 Gradient Steps 0.58 0.60 0.62 0.64 0.66 0.68 0.70RM BT Loss 0.60 0.65 0.70 RM BT Loss 14 13 12 11 10 9 8 7 Gold Model...

    2000

  2. [4]

    URLhttps://proceedings.mlr.press/ v202/karakida23a.html. Kimi Team, Bai, Y ., Bao, Y ., Chen, G., Chen, J., Chen, N., Chen, R., Chen, Y ., Chen, Y ., Chen, Y ., Chen, Z., Cui, J., Ding, H., Dong, M., Du, A., Du, C., Du, D., Du, Y ., Fan, Y ., Feng, Y ., Fu, K., Gao, B., Gao, H., Gao, P., Gao, T., Gu, X., Guan, L., Guo, H., Guo, J., Hu, H., Hao, X., He, T....

    Pith/arXiv arXiv 2025

  3. [6]

    PPO + New KL

    URLhttps://proceedings.mlr.press/ v162/nikishin22a.html. Noukhovitch, M., Lavoie, S., Strub, F., and Courville, A. Language Model Alignment with Elastic Reset. In NeurIPS, 2023. URL https://openreview.net/ forum?id=6lgugutkin. Olmo Team, Ettinger, A., Bertsch, A., Kuehl, B., Gra- ham, D., Heineman, D., Groeneveld, D., Brahman, F., Timbers, F., Ivison, H.,...

    Pith/arXiv arXiv 2023

  4. [2021]

    flat maximum

    has been shown to correspond to gradient regularization with a specific choice of hyper-parameters (Karakida et al., 2023). (Bahri et al., 2022; Zhang et al., 2022) apply SAM to transformer pre-training in order to improve generalization. SAM has also been used in RL, particularly by Lee et al. (2023) to improve sample efficiency when training a policy fo...

    2023

  5. [2022]

    acl-long.508/

    URL https://aclanthology.org/2022. acl-long.508/. Barrett, D. G. T. and Dherin, B. Implicit Gradient Regular- ization. In ICLR, 2021. URL http://arxiv.org/ abs/2009.11162. Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and Wal, O. v. d. Pyt...

    Pith/arXiv arXiv 2022

  6. [2023]

    URLhttps://proceedings.mlr.press/ v202/gao23h/gao23h.pdf. GLM-4.5 Team, Zeng, A., Lv, X., Zheng, Q., Hou, Z., Chen, B., Xie, C., Wang, C., Yin, D., Zeng, H., Zhang, J., Wang, K., Zhong, L., Liu, M., Lu, R., Cao, S., Zhang, X., Huang, X., Wei, Y ., Cheng, Y ., An, Y ., Niu, Y ., Wen, Y ., Bai, Y ., Du, Z., Wang, Z., Zhu, Z., Zhang, B., Wen, B., Wu, B., Xu,...

    Pith/arXiv arXiv 2025

  7. [2024]

    acl-long.662.pdf

    URL https://aclanthology.org/2024. acl-long.662.pdf. Bahri, D., Mobahi, H., and Tay, Y . Sharpness-Aware Minimization Improves Language Model Generaliza- tion. In ACL, pp. 7360–7371, Dublin, Ireland,

    2024

  8. [2025]

    URL https://openreview.net/forum? id=4OaO3GjP7k. Liu, C. Y ., Zeng, L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y ., and Zhou, Y . Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs, 2024. URL http://arxiv.org/abs/2410.18451. Liu, M., Diao, S., Lu, X., Hu, J., Dong, X., Choi, Y ., Kautz, J., and Dong, Y . ProRL: Prolonged Reinforcement Le...

    Pith/arXiv arXiv 2024