Pith. sign in

REVIEW 4 major objections 6 minor 78 references

Bradley-Terry and Multi-Objective Reward Modeling Are Complementary

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Bradley–Terry preference training and multi-attribute regression training are complementary when done jointly on one shared backbone, and a quantitative lower bound captures that complementarity.

desk verdict A genuinely useful empirical recipe—joint BT and multi-attribute regression on a shared embedding—supports a broken theoretical proof, and the headline 7B-vs-70B claim is thinner than it looks. read the letter →

arxiv 2507.07375 v1 pith:DGCR3OOM submitted 2025-07-10 cs.LG cs.CL

Zhiwei Zhang , Hui Liu , Xiaomin Li , Zhenwei Dai , Jingying Zeng , Fali Wang , Minhua Lin , Ramraj Chandradevan
show 5 more authors
Zhen Li Chen Luo Xianfeng Tang Qi He Suhang Wang
This is my paper
classification cs.LGcs.CL
keywords rewardhackingRLHFBradley-Terrymodelingmulti-objectivemodelout-of-distributiongeneralizationsharedembeddingspaceoveroptimizationLLMalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the two standard styles of reward modeling for RLHF — pairwise Bradley–Terry (BT) preference classification and multi-attribute score regression — are complementary, and that training them jointly on one shared embedding space yields both defense against reward hacking and accurate scoring in a single forward pass. Its central claim is that the regression task, even with only small amounts of fine-grained attribute data, hardens the BT head against reward hacking in out-of-distribution settings, while the BT task corrects the embedding so the multi-attribute head scores competitively without new annotations. The paper supports this with a theoretical coupling bound and with experiments in which joint training lets a 7B reward model outscore a 70B baseline on RewardBench while resisting overoptimization in in-distribution and out-of-distribution PPO and best-of-n sampling. If the claim is right, the data-scarcity bottleneck that has kept multi-objective reward models weak is removable without collecting more attribute labels.

What carries the argument

The central object is SMORM (Single and Multi-Objective Reward Model): a decoder-only LLM feature extractor $f_\theta$ whose final hidden state is shared by two linear heads — a single-objective head $w_S$ trained with the Bradley–Terry logistic loss on chosen/rejected pairs, and a multi-objective head $w_M$ trained with squared-error regression against attribute score vectors $r \in \mathbb{R}^K$. The argument is carried by the coupling inequality $r_m \ge c r_s - \varepsilon$, whose constants come from the closed-form population minimizers $w_S = \Sigma_S^{-1}\mu_S$ and $w_M = \Sigma_M^{-1} C_M$ of squared-loss versions of the two objectives. The proof route is what does the work: it substitutes squared losses for the logistic loss (asserting minimizer directions are preserved), whitens the feature space, and projects each column of the multi-objective weight matrix onto the single-objective direction; the bounded-feature assumption bounds the residual as $\varepsilon$. Theorem 2 supplies the opposite direction: the combined task set enlarges the Fisher information matrix, shrinking the MLE covariance and hence the asymptotic MSE of both heads, with Lemma 1 bridging reduced MSE to smaller pairwise preference error.

What would settle it

Train SMORM and its two isolated baselines on identical data, then on held-out pairs plot the average attribute score $r_m$ against the BT score $r_s$ and fit the line $c r_s - \varepsilon$: if the empirical gap $r_m - (c r_s - \varepsilon)$ goes negative for a non-negligible share of high-scoring responses, the claimed monotone coupling is absent. The proof's substitution step can be tested directly: on a linear head with skewed real features, compute the population minimizers under logistic and squared loss with unit target — if they disagree in direction, the derivation of $w_S = \Sigma_S^{-1}\mu_S$ and $w_M = \Sigma_M^{-1} C_M$ does not follow.

Watch

Extended reading notes

Core claim

The paper's central claim is that a reward model carrying two heads on one shared decoder — a single-objective head trained with the Bradley–Terry preference loss and a multi-objective head trained with squared-error regression on fine-grained attribute scores — produces complementary gains that neither training alone can match. The formal statement is Theorem 1, the 'Implicit Multi-Attribute Effect': once both heads converge to their population minimizers, the average multi-attribute score $r_m(x,y) = \frac{1}{K}\sum_{i=1}^K w_{M,i}^\top f_\theta(x,y)$ is bounded below by $c r_s(x,y) - \varepsilon$, a positive affine function of the single-objective score $w_S^\top f_\theta(x,y)$, under a positive-correlation assumption between the aggregated attribute labels and the chosen/rejected preference direction. From this the paper draws two consequences: a high single-objective score alone guarantees a respectable level of fine-grained quality, which explains why policies trained with only the single-objective head perform comparably to those using both; and the BT signal positions responses correctly in the shared embedding space, so the multi-objective head reaches competitive scoring with far less multi-attribute data. A companion theorem argues that joint training strictly reduces the asymptotic mean squared error of both heads because the combined task set enlarges the shared Fisher information, and a lemma converts that reduction into a bound on pairwise preference error.

Load-bearing premise

The central inequality rests on the claim, made in the proof of Theorem 1, that replacing the pairwise preference (logistic) loss with a squared-error loss leaves the direction of the optimal head weights unchanged; that equivalence fails in general — with skewed features the squared-loss minimizer stays finite while the logistic minimizer can drift — so the central bound is not established by the supplied proof.

Editorial extensions

If this is right

  • A single forward pass through one backbone yields both a hack-resistant BT score and an accurate multi-attribute score, removing the two-pass inference cost of separately trained single- and multi-objective ensembles.
  • Multi-objective reward models can reach competitive scoring with only a small human-annotated attribute dataset (the paper uses 20K HelpSteer2 samples), because the BT head corrects embedding-space positioning; the paper's 7B SMORM outscores a 70B reward model on RewardBench.
  • Policies optimized against the jointly trained single-objective head resist reward hacking in OOD PPO and best-of-n settings, improving all five measured attribute dimensions — helpfulness, correctness, coherence, complexity, verbosity — rather than gaming only style dimensions.
  • If the variance argument in Theorem 2 holds, joint training improves both heads over their isolated counterparts at equal data, making the data-scarcity bottleneck of multi-objective reward models removable without new annotation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The projection machinery behind Theorem 1 is task-agnostic: any pair of objectives sharing an embedding space, one supplying a coarse global signal and the other fine-grained local labels, should exhibit the same type of coupling — a testable prediction for settings like step-level process reward modeling where attribute labels are even scarcer.
  • The coupling constant $c$ from Theorem 1 could serve as a training-time diagnostic the paper does not develop: monitoring $c$ during joint training would reveal when the multi-attribute dataset is misaligned with the preference direction and joint training should be expected to fail.
  • The 7B-beats-70B result suggests that model size and joint-task training are partly substitutable inputs to scoring quality; mapping that trade-off across base-model sizes and attribute taxonomies would show how far the substitution extends.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SMORM, a unified reward-modeling framework that jointly trains a Bradley-Terry single-objective head and a multi-objective regression head on a shared transformer embedding. The authors claim that this joint training yields complementary benefits: the regression task improves the single-objective head's robustness against reward hacking in out-of-distribution settings, while the BT task improves the scoring capability of the multi-objective head, allowing a 7B model to outperform a 70B baseline. The paper supports these claims with Theorem 1 (a lower bound relating the multi-objective average score to the single-objective score), Lemma 1 (an MSE-to-preference-error bound), Theorem 2 (an asymptotic MSE reduction under joint training), and experiments on RewardBench, RM-Bench, PPO, and Best-of-N sampling.

Significance. The problem addressed is timely and important: reward hacking under out-of-distribution prompts is a real limitation of current reward models, and the idea of sharing an embedding between a BT head and a multi-attribute regression head is simple and plausible. The experimental scope is broad, covering multiple base models, data scales, and evaluation protocols, and the paper is honest about the difficulty of obtaining high-quality multi-attribute annotations. However, the theoretical results are currently the main weak point: the proof of Theorem 1 rests on an invalid equivalence between logistic and squared losses, and the proof of Theorem 2 is incomplete. If the theoretical claims were repaired or appropriately softened, the empirical recipe would still be a useful contribution; as written, the 'theoretically grounded' framing is not supported.

major comments (4)
  1. [Appendix C.2, Eq. (11)-(14)] The proof of Theorem 1 replaces the logistic BT loss with a squared loss and asserts that 'any strictly convex proper surrogate has the same first-order optimality conditions up to a positive scalar factor.' This claim is false for the losses in question. For a linear head w on features u = f_c - f_r, the logistic/BT population minimizer satisfies E[u(σ(w^T u) - p(u))] = 0, whereas the squared-loss minimizer satisfies E[uu^T]w = E[u t] with t = 1; these minimizers coincide only under special distributional assumptions, such as Gaussian class-conditional features with equal covariance. The derivation of w_S = Σ_S^{-1} μ_S, the decomposition in Eq. (13), and the constants c and ε in Eq. (14) therefore apply to a different objective than the SMORM objective in Eq. (5). Since Theorem 1 is the paper's stated basis for the 'implicit multi-attribute effect' and for the claim that SMORM-F matches SMORM-M, the central theoretical result is not established by the proof. In addition, the theorem's conclusion is structurally dependent on assumption (3), 1^T α ≥ 0, because c is defined directly from α; the lower bound is therefore closer to a restatement of the assumption than an independent derivation, and the paper does not verify this assumption on the trained models.
  2. [Appendix C.4, Theorem 2 proof] The proof of Theorem 2 is incomplete in several steps. First, the argument that the hybrid Fisher matrix is strictly larger than the single-head Fisher matrix shows only that g0^T I_hybrid g0 > g0^T I_single g0 for one direction g0; this does not imply the Loewner inequality I_hybrid - I_single ≻ 0, which would require v^T(I_hybrid - I_single)v > 0 for all v. Second, the proof invokes 'positive correlation ρ0k > 0' without defining ρ0k or proving that the relevant inner products are strictly positive. Third, the step from Cov(θ̂_hybrid) ≺ Cov(θ̂_single) to the head-specific MSE inequalities in Eq. (9) assumes that the head weights w_S and w_M are fixed, whereas in SMORM they are jointly estimated; the standard Cramér-Rao argument does not directly compare MSE of predictions under two different models with different parameter spaces. The inequalities in Eq. (9) are therefore not established.
  3. [Section 5.3 and Figures 2-5] The RLHF experiments report no error bars, no number of random seeds, and no statistical significance tests. Claims such as 'SMORM-F and SMORM-M exhibit a consistent increase in gold score throughout training' and 'SMORM significantly outperforms all baselines' are based on single trajectories, which are known to be noisy in PPO and BoN experiments. This is a load-bearing empirical claim, because the paper's main narrative is that SMORM mitigates reward hacking more reliably than prior methods.
  4. [Table 3 and Abstract] The abstract's claim that SMORM 'enables a 7B model to outperform a 70B baseline' rests on the RewardBench average 89.0 for SMORM-L 7B versus 88.8 for Llama-3-70B-RM, a 0.2-point difference. No variance, confidence interval, or repeated-run comparison is reported, and the comparison involves different base models, training sets, and data sizes. As stated, this headline comparison is not statistically supported.
minor comments (6)
  1. [Appendix G.1] The label smoothing loss is written as L_smooth(θ) = -E[(1-ε) log σ(...) - ε log σ(...)], where the two logarithmic terms are identical. As written, this reduces to the standard BT loss and provides no smoothing; presumably one term should be log σ(-(...)) or an equivalent modification. If the implementation matches the formula, the Label Smooth baseline in Table 1 is indistinguishable from the Baseline.
  2. [Appendix C.2] In the proof of Theorem 1, the sentence describing the bound on the orthogonal part ends with 'the square root of the smallest eigenvalue of the .' — an incomplete sentence that should reference Σ_S.
  3. [Appendix C.1] The line 'Assuming E[r] = E[w_S^T] = 0' is ill-typed: E[w_S^T] is a vector mean, not a scalar, and the intended statement is presumably E[r] = 0 and E[w_S^T f] = 0.
  4. [Appendix C.2] The proof states 'By assumption (3) we have 1^T α ≥ 0 and thus 1^T β > 0'; the latter should be ≥, since 1^T α = 0 is allowed. The subsequent assertion c > 0 should be c ≥ 0 unless strict positivity is separately assumed.
  5. [Title page] The affiliation for Xiaomin Li is listed as 'Harvard University'; this should be corrected to 'Harvard University'.
  6. [Theorem 1 statement] In Eq. (6), the equality defining r_m is followed by the inequality; writing 'r_m = ... ≥ ...' is confusing. The definition and the inequality should be presented separately.

Circularity Check

2 steps flagged · score 6.0 of 10

Theorem 1 and Theorem 2 restate the assumed positive correlation between single- and multi-objective heads as the derived 'complementary benefit'; the theoretical grounding reduces to its own assumptions.

  1. self definitional [Section 3, Theorem 1 (Eq. 6); proof in Appendix C.2 (Eqs. 10-14)]
    "Positive correlation: Let µS := E(xs,yc,yr)∼DS [fθ(xs, yc) − fθ(xs, yr)] and let CM := E(xm,ym,r)∼DM [fθ(xm, ym) r⊤] ∈ Rd×K. Then α := µ⊤SΣ−1M CM has non-negative sum, i.e. 1⊤α ≥ 0. As the optimization of both reward heads converge to their population minimizers, there exist constants c = 1⊤α K (µ⊤S Σ−1S µS) and ε ≥ 0—depending only on B and second-order moments—such that for every pair (x, y): rm(x, y) = 1 K PK i=1 w⊤M,ifθ(x, y) ≥ c(w⊤S fθ(x, y)) − ε = crs(x, y) − ε."

    Assumption (3) of the theorem is exactly 1^T α ≥ 0, where α = μ_S^T Σ_M^{-1} C_M. The proof then defines c := 1^T β/K = 1^T α/(K||μ~_S||^2) and uses the assumption to assert c > 0, giving r_m ≥ c r_s − ε. Thus the claimed 'implicit multi-attribute effect' is the positive-correlation hypothesis rewritten as an inequality; the slope c is the assumed correlation, normalized. Moreover, w_S = Σ_S^{-1} μ_S and w_M = Σ_M^{-1} C_M are the ordinary population least-squares formulas for any two linear heads on the same features; nothing in the derivation depends on joint SMORM training, so the theorem cannot establish the complementary benefit it is invoked to explain.

  2. self definitional [Section 4, Theorem 2 (Eq. 9); proof in Appendix C.4 (Eqs. 17-19)]
    "Strict positivity of difference term. In fact, give the assumed positive correlation between head 0 and other attribute heads, we can show that the overall Fisher matrix can be strictly larger... The positive correlation assumption ρ0k > 0 implies that, on average, the gradients ∇θrk(yi) tend to point in a similar direction to g0(yi)... Therefore, we can get I(hybrid) = I(single) + ∆, ∆ ≻ 0... Hence MSESMORM S < MSEsingle S, MSESMORM M < MSEmulti M."

    The advertised strict inequalities MSESMORM_S < MSEsingle_S and MSESMORM_M < MSEsingle_M are obtained in the proof only after the additional assumption ρ0k > 0, i.e., that each attribute head's gradients are positively correlated with the single-objective head. That positive correlation is precisely the 'complementary benefit' that the paper says BT-based joint training produces. Without an independent argument that SMORM training creates ρ0k > 0, the strict MSE reduction is the targeted effect assumed as a hypothesis, not derived. The generic PSD step I_hybrid = I_single + Δ, Δ ⪰ 0 gives only a weak ordering and does not imply the strict inequalities in Eq. (9).

full rationale

Most of the empirical content of the paper is self-contained: RewardBench/RM-Bench and the PPO/BoN evaluations are external benchmarks, and Tables 1-3 compare against independent baselines; no fitted parameter from those benchmarks is recycled as a prediction. Self-citations are not load-bearing. However, the paper's advertised theoretical grounding ('theoretically establish a connection', 'theoretically and empirically establish complementary benefits') reduces to two assumption-restating moves. Theorem 1's condition (3) is 1^T α ≥ 0, and its only nontrivial conclusion c>0 uses c := 1^T α/(K||μ~_S||^2); the lower bound is therefore the positive-correlation assumption expressed as an inequality, and the derivation uses only population least-squares formulas that hold for any two heads on the same features, with no term representing joint training. Theorem 2 obtains its strict MSE reductions only after assuming ρ0k >0, i.e., the same positive coupling whose creation by BT training is the paper's central claim; the generic PSD Fisher-information addition would give only a non-strict comparison. Separately, the proof in C.2 replaces the actual logistic BT loss by a squared loss on the assertion that minimizer directions coincide; this is a mathematical correctness issue rather than circularity, but it means Theorem 1 is not about the SMORM objective as defined in Eq. (5). Because the two central theorems' advertised 'complementary benefits' are restatements of assumed correlations, the circularity score is 6, while the standalone empirical results would support a much lower score.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central theoretical claim rests on a false loss-equivalence assumption and on a positive-correlation assumption that already contains the conclusion. The empirical method itself uses no invented entities, and the only hand-set hyperparameter is the loss weight ratio.

free parameters (1)
  • single/multi loss weight ratio = 1.0 (default)
    Set to 1.0 by default (Table 10). Appendix H shows sensitivity: at ratio 0.01, SMORM-L falls below the multi-objective baseline, and at ratio 10, SMORM-F falls below the single-objective baseline. The central results use the default, so the method's advantage depends on this hand choice.
assumptions (4)
  • ad hoc to paper Logistic Bradley-Terry loss and squared loss have the same population minimizer directions up to a positive scalar factor.
    Invoked in Appendix C.2, first paragraph, to replace the BT loss with squared loss. This is false in general, so the head solutions w_S and w_M are not justified for the BT-trained head.
  • ad hoc to paper The covariance between the single-objective preference direction and the aggregate multi-attribute score is non-negative, i.e., 1^T alpha >= 0.
    Assumed in Theorem 1 and Theorem 2 (Appendix C.1, C.2). It is not derived; the constant c = 1^T alpha / (K mu_S^T Sigma_S^{-1} mu_S) is positive exactly when this holds, so the main inequality r_m >= c r_s - epsilon largely restates the assumption.
  • domain assumption Sigma_S and Sigma_M are positive definite covariance matrices for the features.
    Assumed in Theorem 1. Likely violated for high-dimensional LLM embeddings with limited data, though the paper argues that samples not lying in a hyperplane suffice.
  • standard math Cramer-Rao bound and asymptotic normality of the MLE, plus zero asymptotic bias.
    Used in Theorem 2 to convert larger Fisher information into smaller prediction MSE. Standard, but requires model correctness and identifiability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bradley-Terry and Multi-Objective Reward Modeling Are Complementary." pith.science (2026). https://pith.science/paper/DGCR3OOM

@misc{pith2026250707375,
  author       = {Pith},
  title        = {Pith review of: Bradley-Terry and Multi-Objective Reward Modeling Are Complementary},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DGCR3OOM}},
  note         = {Machine review of arXiv:2507.07375}
}
read the original abstract

Reward models trained on human preference data have demonstrated strong effectiveness in aligning Large Language Models (LLMs) with human intent under the framework of Reinforcement Learning from Human Feedback (RLHF). However, RLHF remains vulnerable to reward hacking, where the policy exploits imperfections in the reward function rather than genuinely learning the intended behavior. Although significant efforts have been made to mitigate reward hacking, they predominantly focus on and evaluate in-distribution scenarios, where the training and testing data for the reward model share the same distribution. In this paper, we empirically show that state-of-the-art methods struggle in more challenging out-of-distribution (OOD) settings. We further demonstrate that incorporating fine-grained multi-attribute scores helps address this challenge. However, the limited availability of high-quality data often leads to weak performance of multi-objective reward functions, which can negatively impact overall performance and become the bottleneck. To address this issue, we propose a unified reward modeling framework that jointly trains Bradley--Terry (BT) single-objective and multi-objective regression-based reward functions using a shared embedding space. We theoretically establish a connection between the BT loss and the regression objective and highlight their complementary benefits. Specifically, the regression task enhances the single-objective reward function's ability to mitigate reward hacking in challenging OOD settings, while BT-based training improves the scoring capability of the multi-objective reward function, enabling a 7B model to outperform a 70B baseline. Extensive experimental results demonstrate that our framework significantly improves both the robustness and the scoring performance of reward models.

Figures

Figures reproduced from arXiv: 2507.07375 by the authors.

Figure 1
Figure 1. Illustration of SMORM training and its advantages over baseline methods. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Proxy and gold scores from (a)(b) PPO and (c)(d) BoN experiments under the OOD setting, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Proxy scores and gold scores of PPO experiments for reward model based on (a)(b) [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Proxy scores and gold scores of BoN experiments for base models of (a)(b) gemma-2b-it [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Proxy and gold scores from (a)(b) PPO and (c)(d) BoN experiments under the OOD setting. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Hyperparameter analysis. I Interpretation of Why SORM Fail in OOD Setting 0 10000 20000 30000 40000 Training Samples 1.0 0.8 0.6 0.4 0.2 0.0 0.2 0.4 Scores Golden Score Proxy Score (r=-0.99) Helpfulness (r=0.98) Correctness (r=0.98) Coherence (r=0.93) Complexity (r=-0.…
Figure 7
Figure 7. Figure 7: Fine-grained attribute scores of the optimized policy model using (a) the baseline classifier [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]
Figure 8
Figure 8. Figure 8: Visualization of results on RM-Bench. The size of each marker indicates the model’s [PITH_FULL_IMAGE:figures/full_fig_p028_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

78 extracted references · 31 canonical work pages

  1. [1]

    A survey on evaluation of large language models

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1–45, 2024

    2024

  2. [2]

    Using an llm to help with code understanding

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13, 2024

    work page 2024

  3. [3]

    A survey of large language models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2), 2023

    arXiv 2023

  4. [4]

    A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llms, and trustworthiness

    Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, et al. A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llms, and trustworthiness. arXiv preprint arXiv:2411.03350, 2024

    arXiv 2024

  5. [5]

    A survey on large language models for code generation

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515, 2024

    arXiv 2024

  6. [6]

    Nguyen, Quang Pham, and Nghi D

    Dung Manh Nguyen, Thang Chau Phan, Nam Le Hai, Tien-Thong Doan, Nam V . Nguyen, Quang Pham, and Nghi D. Q. Bui. CodeMMLU: A multi-task benchmark for assessing code understanding & reasoning capabilities of codeLLMs. In The Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  7. [7]

    Rational decision-making agent with learning internal utility judgment

    Yining Ye, Xin Cong, Shizuo Tian, Yujia Qin, Chong Liu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Rational decision-making agent with learning internal utility judgment. In The Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  8. [8]

    How Far are LLMs from Real Search? A Comprehensive Study on Efficiency, Completeness, and Inherent Capabilities

    Minhua Lin, Hui Liu, Xianfeng Tang, Jingying Zeng, Zhenwei Dai, Chen Luo, Zheng Li, Xiang Zhang, Qi He, and Suhang Wang. How far are llms from real search? a comprehensive study on efficiency, completeness, and inherent capabilities. arXiv preprint arXiv:2502.18387, 2025

    work page Pith review arXiv 2025

Show all 78 references
  1. [9]

    Safe RLHF: Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024

    2024

  2. [10]

    Data-adaptive safety rules for training reward models

    Xiaomin Li, Mingye Gao, Zhiwei Zhang, Jingxuan Fan, and Weiyu Li. Data-adaptive safety rules for training reward models. arXiv preprint arXiv:2501.15453, 2025

    2025 arXiv

  3. [11]

    Large language model alignment: A survey

    Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey. arXiv preprint arXiv:2309.15025, 2023

    2023 arXiv

  4. [12]

    Catastrophic failure of LLM unlearning via quantization

    Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. Catastrophic failure of LLM unlearning via quantization. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  5. [13]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    2022 arXiv

  6. [14]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  7. [15]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 10

    2017 arXiv

  8. [16]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024

    2024 arXiv

  9. [17]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  10. [18]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023

    2023

  11. [19]

    Regularizing hidden states enables learning generalizable reward model for LLMs

    Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  12. [20]

    Reinforced self-training (rest) for language modeling

    Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023

    2023 arXiv

  13. [21]

    RAFT: Reward ranked finetuning for generative foundation model alignment

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023

    2023

  14. [22]

    BoNBon alignment for large language models and the sweetness of best-of-n sampling

    Lin Gui, Cristina Garbacea, and Victor Veitch. BoNBon alignment for large language models and the sweetness of best-of-n sampling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  15. [23]

    Reward model ensembles help mitigate overoptimization

    Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representa- tions, 2024

    2024

  16. [24]

    Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking

    Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvi- jotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2...

    2023 arXiv

  17. [25]

    Im- proving reinforcement learning from human feedback with efficient reward model ensemble

    Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, and Chuang Gan. Im- proving reinforcement learning from human feedback with efficient reward model ensemble. arXiv preprint arXiv:2401.16635, 2024

    2024 arXiv

  18. [26]

    Reward-robust rlhf in llms

    Yuzi Yan, Xingzhou Lou, Jialian Li, Yiping Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, and Yuan Shen. Reward-robust rlhf in llms. arXiv preprint arXiv:2409.15360, 2024

    2024 arXiv

  19. [27]

    Warm: On the benefits of weight averaged reward models

    Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.12187, 2024

    2024 arXiv

  20. [28]

    Mitigating reward overoptimization via lightweight uncertainty estimation

    Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Mitigating reward overoptimization via lightweight uncertainty estimation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  21. [29]

    Confronting reward model overoptimization with constrained RLHF

    Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca Dragan, and Stephen Marcus McAleer. Confronting reward model overoptimization with constrained RLHF. In The Twelfth International Conference on Learning Representations , 2024

    2024

  22. [30]

    Provably mitigating overoptimization in RLHF: Your SFT loss is implicitly an adversarial regularizer

    Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in RLHF: Your SFT loss is implicitly an adversarial regularizer. In The Thirty-eighth Annual Conference on Neural Information Processi...

    2024

  23. [31]

    Overcom- ing reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation

    Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Overcom- ing reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation. arXiv preprint arXiv:2403.05171, 2024. 11

    2024 arXiv

  24. [32]

    Correlated proxies: A new definition and improved mitigation for reward hacking

    Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Correlated proxies: A new definition and improved mitigation for reward hacking. arXiv preprint arXiv:2403.03185, 2024

    2024 arXiv

  25. [33]

    Odin: Disentangled reward mitigates hacking in rlhf

    Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. In International Conference on Machine Learning, pages 7935–7952. PMLR, 2024

    2024

  26. [34]

    Mitigating reward over- optimization in RLHF via behavior-supported regularization

    Juntao Dai, Taiye Chen, Yaodong Yang, Qian Zheng, and Gang Pan. Mitigating reward over- optimization in RLHF via behavior-supported regularization. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  27. [35]

    Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev

    Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. In The Thirty-eight Conference on Neural Informa- tion P...

    2024

  28. [36]

    Interpretable prefer- ences via multi-objective reward modeling and mixture-of-experts

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable prefer- ences via multi-objective reward modeling and mixture-of-experts. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2024, pages 10582–10592, 2024

    2024

  29. [37]

    Ultrafeedback: Boosting language models with high-quality feedback

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. 2023

    2023

  30. [38]

    Prometheus: Inducing fine-grained evaluation capability in language models

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representa...

    2023

  31. [39]

    From generation to judgment: Opportunities and challenges of llm-as-a-judge

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594, 2024

    2024

  32. [40]

    A survey on llm-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  33. [41]

    Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev

    Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models, 2024

    2024

  34. [42]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024

    2024

  35. [43]

    RM-bench: Benchmarking reward models of language models with subtlety and style

    Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. RM-bench: Benchmarking reward models of language models with subtlety and style. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  36. [44]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952

    1952

  37. [45]

    Helpsteer: Multi-attribute helpfulness dataset for steerlm

    Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.09528, 2023

    2023 arXiv

  38. [46]

    Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards

    Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards. In ACL, 2024

    2024

  39. [47]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020. 12

    2020

  40. [48]

    Pairwise proximal policy optimization: Language model alignment with comparative RL

    Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. Pairwise proximal policy optimization: Language model alignment with comparative RL. In First Conference on Language Modeling, 2024

    2024

  41. [49]

    Reward shaping to mitigate reward hacking in rlhf

    Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, and Yanghua Xiao. Reward shaping to mitigate reward hacking in rlhf. arXiv preprint arXiv:2502.18770, 2025

    2025

  42. [50]

    Skywork-reward: Bag of tricks for reward modeling in llms

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451, 2024

    2024 arXiv

  43. [51]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  44. [52]

    Reward model ensembles help mitigate overoptimization

    Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743, 2023

    2023 arXiv

  45. [53]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023

  46. [54]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  47. [55]

    Secrets of rlhf in large language models part ii: Reward modeling

    Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06080, 2024

    2024 arXiv

  48. [56]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  49. [57]

    Rlhf workflow: From reward modeling to online rlhf

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024

    2024 arXiv

  50. [58]

    Rethinking reward model evaluation: Are we barking up the wrong tree? In The Thirteenth International Conference on Learning Representations, 2025

    Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, XingYu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, and Le Sun. Rethinking reward model evaluation: Are we barking up the wrong tree? In The Thirteenth International Conference on Learning Representations, 2025

    2025

  51. [59]

    What makes a reward model a good teacher? an optimization perspective

    Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D Lee, and Sanjeev Arora. What makes a reward model a good teacher? an optimization perspective. arXiv preprint arXiv:2503.15477, 2025

    2025

  52. [60]

    Mitigating the alignment tax of rlhf

    Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, et al. Mitigating the alignment tax of rlhf. arXiv preprint arXiv:2309.06256, 2023

    2023 arXiv

  53. [61]

    Rethinking reward modeling in preference-based large language model alignment

    Hao Sun, Yunyi Shen, and Jean-Francois Ton. Rethinking reward modeling in preference-based large language model alignment. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  54. [62]

    Starling-7b: Improving llm helpfulness & harmlessness with rlaif

    Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving llm helpfulness & harmlessness with rlaif. 2023, 2023

    2023

  55. [63]

    Nemotron-4 340b technical report

    Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprint arXiv:2406.11704, 2024. 13

    2024 arXiv

  56. [64]

    Fine-grained human feedback gives better rewards for language model training

    Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems , 36:59008–59...

    2023

  57. [65]

    Beyond imitation: Leveraging fine-grained quality signals for alignment

    Geyang Guo, Ranchi Zhao, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. Beyond imitation: Leveraging fine-grained quality signals for alignment. arXiv preprint arXiv:2311.04072, 2023

    2023 arXiv

  58. [66]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference...

    2024

  59. [67]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  60. [68]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:2312.08935, 2023

    2023 arXiv

  61. [69]

    Improve mathematical reasoning in language models by automated process supervision

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592, 2024

    2024 arXiv

  62. [70]

    The lessons of developing process reward models in mathematical reasoning

    Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301, 2025

    2025 arXiv

  63. [71]

    Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf

    Banghua Zhu, Michael I Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv preprint arXiv:2401.16335, 2024

    2024 arXiv

  64. [72]

    RRM: Robust reward model training mitigates reward hacking

    Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, Daniel Sohn, Anastasia Makarova, Jeremiah Zhe Liu, Yuan Liu, Bilal Piot, Abe Ittycheriah, Aviral Kumar, and Mohammad Saleh. RRM: Robust reward model training mit...

    2025

  65. [73]

    Beyond reward hacking: Causal rewards for large language model alignment

    Chaoqi Wang, Zhuokai Zhao, Yibo Jiang, Zhaorun Chen, Chen Zhu, Yuxin Chen, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Hao Ma, et al. Beyond reward hacking: Causal rewards for large language model alignment. arXiv preprint arXiv:2501.09620, 2025

    2025 arXiv

  66. [74]

    Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling

    Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  67. [75]

    Convexity, classification, and risk bounds

    Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138–156, 2006

    2006

  68. [76]

    Fundamentals of statistical signal processing: estimation theory

    Steven M Kay. Fundamentals of statistical signal processing: estimation theory. Prentice-Hall, Inc., 1993

    1993

  69. [77]

    Args: Alignment as reward-guided search

    Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search. arXiv preprint arXiv:2402.01694, 2024

    2024 arXiv

  70. [78]

    Transformers: State- of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in n...

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.