Pith. sign in

REVIEW 4 major objections 7 minor 59 references

Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Preference signals in LLM alignment are concentrated in the opening tokens, so training on truncated responses matches or beats full-data training on standard benchmarks.

desk verdict Truncation works, but 'shallow preference signals' is not yet separated from length debiasing—still a legit paper for review. read the letter →

arxiv 2505.17122 v1 pith:FD6I66JQ submitted 2025-05-21 cs.CL

Xuan Qi , Jiahao Qiu , Xinzhe Juan , Yue Wu , Mengdi Wang This is my paper
classification cs.CL
keywords shallowpreferencesignalsdatatruncationrewardmodeldirectoptimizationLLMalignmenthackingdecodingstrategyBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the signal telling which of two LLM responses is preferred is concentrated in the opening tokens, a property it names shallow preference signals. It truncates preference datasets to the first 25–50% of each response, then trains reward models and Direct Preference Optimization (DPO) models on the shortened data. On standard benchmarks these truncated-data models match or beat models trained on the full responses; for instance, a reward model trained on the Skywork data truncated to 40% of its token length scores 76.35 on RewardBench versus 75.85 for the full dataset. The authors argue this shows alignment pipelines are mostly shaping only the beginning of a response, so focusing training on early tokens can reduce cost without sacrificing benchmark performance. The paper explicitly limits itself to the empirical claim, noting in its Limitations section that it lacks a strong theoretical account of where preference information lives.

What carries the argument

The carrying mechanism is the token-level reward decomposition: the total preference signal is approximated by the cumulative sum of per-token log-probability differences between chosen and rejected responses up to a truncation point $t_k$, with later tokens treated as less informative or noisy. This identity motivates both the truncation training procedure and the two decoding strategies, Length Control Decoding and KL Threshold Control Decoding, which generate the first tokens from the DPO policy and later tokens from the reference policy, improving the reward-KL tradeoff.

What would settle it

Construct a preference set in which chosen and rejected responses share the same first half and differ only in the second half; a reward model trained on 40%-truncated Skywork data should score close to chance on such pairs if preference signals truly live in early tokens. A length-controlled baseline on full data would further separate truncation from length-debiasing.

Watch

Extended reading notes

Core claim

The central discovery is the shallow preference signal: for a chosen response $y_1,\ldots,y_T$ and a rejected response $z_1,\ldots,z_T$, the per-token signal $R_t(i)=\log p(y_t\mid x,y_{1:t-1})-\log p(z_t\mid x,z_{1:t-1})$ is strong near the start of the response and weakens later, so the cumulative sum over the first $t_k$ tokens approximates the full-response reward. Truncation experiments bear this out across synthetic preference data and human-generated SHP data: keeping only the first 40–50% of tokens yields reward-model and DPO performance comparable to or better than full-data training on RewardBench and AlpacaEval 2.0. The paper concludes that current alignment methods achieve only shallow alignment, improving metrics while leaving most of a generated response unaligned with real human preferences.

Load-bearing premise

The paper's argument rests on the premise that the difference in quality between a chosen and a rejected response is genuinely concentrated in the opening tokens, so cutting each response to the same short prefix keeps the true preference signal intact rather than merely removing confounds such as response length.

Editorial extensions

If this is right

  • Preference datasets can be truncated to roughly half their token length before training, cutting compute and storage while retaining or improving RewardBench and AlpacaEval scores.
  • DPO models trained only on the opening tokens still generate full responses at inference time, and at 50% truncation achieve a higher AlpacaEval length-controlled win rate (25.19) than the full-data DPO model (24.90).
  • Because the KL divergence between aligned and reference models is concentrated in early tokens, sampling from the aligned model for only the first part of a response and from the reference model afterward preserves reward while reducing divergence.
  • Evaluation of alignment should not treat a whole response as uniformly aligned; the paper's results imply current pipelines mostly alter the opening tokens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark improvements may come partly from removing response-length confounds, because truncating chosen and rejected responses to equal length strips the length signal that reward models are known to exploit; the paper does not run a length-controlled full-data baseline.
  • The per-token signal definition depends on the model $p$ used to score responses, so where the preference signal appears is model-dependent; the same truncation recipe could behave differently for a stronger base model.
  • The decoding strategies imply a practical inference-time saving: if later tokens contribute little preference signal, models can switch to a cheaper reference policy after the first few tokens.
  • The shallow-alignment diagnosis suggests a testable extension: build preference pairs whose opening tokens are identical and whose later tokens decide quality, and measure whether trained reward models still recover the preference; the paper's account predicts they would fail.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces the notion of 'shallow preference signals,' arguing that the distinguishing information between chosen and rejected responses in preference data is concentrated in the early tokens. To test this, the authors truncate preference datasets at various ratios (50%, 40%, 33%, 25%) and train reward models and DPO models on the truncated data, reporting that models trained on 40–50% of the tokens achieve comparable or even superior benchmark performance to models trained on full responses. They also propose two decoding strategies—Length Control Decoding and KL Threshold Control Decoding—that use the early-token concentration to improve the reward–KL trade-off. Experiments span Skywork, UltraFeedback, RLHFlow, and human-written SHP data, with evaluation on RewardBench, AlpacaEval 2.0, and in-house accuracy measures.

Significance. If the central claim holds, the paper has practical value: truncating preference data to the first half of each response could substantially reduce training cost while maintaining alignment quality, and the shallow-signal perspective could inform why RLHF/DPO sometimes produces shallow alignment. The paper includes a useful breadth of experiments, code release, and a human-generated-data control, which strengthens the plausibility of the core observation. However, the 'comparable or even superior' claim is currently not separated from a known confound—removing length differences between chosen and rejected responses—and the statistical evidence is limited to single runs without error bars or significance tests. The theoretical framing in Section 3.1 is a definitional identity rather than an empirical measurement, so the paper's interpretive claim outruns its evidence.

major comments (4)
  1. [4.2.1 / Table 1] The headline 'superior' claim is not statistically supported. The Skywork total difference (76.35 vs. 75.85 at 40% vs. full) is only 0.5 percentage points, which is likely within seed noise, and the direction is inconsistent across datasets: UltraFeedback total drops from 73.91 to 71.94 at 40%. The RLHFlow Reasoning jump from 59.84 to 83.74 at 50% is anomalously large and unexplained. The paper reports single runs with no error bars or significance tests. Please provide multiple seeds with standard deviations and appropriate significance tests, and temper the 'even superior' wording unless the effect is consistently significant.
  2. [3.2 / 4.2] The truncation procedure forces both chosen and rejected responses to the same length tk, thereby removing any length difference between them. Prior work cited as [10] (Park et al.) shows that reward models and DPO readily exploit response length as a shortcut. The paper never includes a length-controlled or length-debiased baseline on full-length responses, so the improved performance at 40–50% truncation could be caused by eliminating the length confound rather than by concentration of preference information in early tokens. To support the shallow-signal interpretation, add a baseline such as training on full-length data with length-matched pairs, or truncating only the longer response to the length of the shorter response, and show that the early-token effect persists.
  3. [3.1] Equation (1) defines R(i) = log p(y1:tk|x) − log p(z1:tk|x) as the cumulative sum of per-token log-probability differences. This is a definitional identity for any model p and does not, by itself, establish that preference signal is concentrated in early tokens. The model p is never specified, and no experiment directly measures per-token reward contribution for an actual reward model or policy. The decoding experiments in Section 5 measure DPO-vs-reference KL divergence, which is a related but different quantity. Please clarify that the formula is a motivating identity, or provide a direct empirical decomposition of token-level reward contribution that controls for length.
  4. [4.2.3 / Table 3] The DPO experiments are limited to a single dataset (Skywork) and single seeds. The reported gains over full-data training are small (e.g., LCWR 24.90 vs. 25.19 at 50%), and the evaluation uses the length-controlled LCWR metric even though the training data are truncated without any length debias. Given the small differences and lack of significance testing, the claim that truncated DPO training performs 'similarly or better' is not yet established. Please report multiple seeds and ideally include a length-debiased full-data DPO baseline.
minor comments (7)
  1. [Abstract / Section 1] The dataset name 'RLHFlow-pair-data-v2-80K-wsafety' is duplicated in the introduction ('RLHFlow-pair-data-v2-80K-wsafetyRLHFlow-pair-data-v2-80K-wsafety'); please fix.
  2. [3.2] There is a typo: 'truncated preference datas' should be 'truncated preference data'.
  3. [References] Reference [54], cited for Llama-3.1-8B-Instruct, points to Patterson et al. (carbon footprint), which appears to be the wrong reference; the Llama 3 model card or a similar source should be cited.
  4. [References] References [30] and [53] are duplicates of the same Bradley–Terry paper; consolidate them.
  5. [5.1] The sentence 'The KL divergence between two policy at a token is computed' has a grammar error and should be 'between the two policies at a token'.
  6. [Figure 3] The figure caption says 'KL-Divergence vs Generated Length' but the x-axis is labeled 'Token Position (t)'; align the caption and axis label.
  7. [3.3.2] In the KL Threshold Control Decoding description, the definition of y(i)_t is confusing ('the i-th sampled token from the DPO model at the t-th position'); clarify the notation for the sampling index and the generated token.

Circularity Check

1 steps flagged · score 2.0 of 10

One definitional identity in the Section 3.1 formalization, but the central truncation claim is tested on external benchmarks and is not load-bearing circularity.

  1. self definitional [Section 3.1 (Formulation of Reward Signal Location)]
    "We define the reward signal at each token position t as ... Rt(i) = logp(yt | x, y1:t−1) − log p(zt | x, z1:t−1) ... We argue that the total reward signal R(i) can be approximated as the cumulative sum of the reward signals up to a truncation point tk: R(i) = ... = logp(y1:tk | x) − log p(z1:tk | x) ... This leads to the hypothesis that truncated responses up to position tk preserve most of the reward signal."

    By defining the per-token signal as a log-probability difference, the cumulative sum to tk is exactly log p(y1:tk|x) − log p(z1:tk|x) through the chain rule; the word 'approximated' hides that this is an identity. Therefore the statement that truncation preserves the reward signal up to tk is true by construction. However, this identity does not entail that early tokens contain most of the total signal, and the paper's headline result (e.g., Table 1: 40% Skywork 76.35 vs. full 75.85) is an external benchmark observation, so the circularity is confined to the framing and is not load-bearing.

full rationale

The paper's central assertion—that preference data truncated to 40–50% yields reward and DPO models comparable to or better than full-data training—is not forced by any parameter fit or by the authors' prior work. Tables 1–4 and Figure 2 are external evaluations (RewardBench, AlpacaEval LCWR/WR, human preference labels), and the truncation ratios are data manipulations, not variables fitted to those benchmarks. The only definitional element is Section 3.1, where the per-token 'reward signal' is defined as a log-probability difference; the cumulative sum to tk is then identical to the log-probability difference of the truncated prefix via the chain rule. This makes the claim that truncation retains the prefix signal true by construction, but it does not establish that early tokens contain 'most' of the total signal—that requires the empirical comparisons, which the paper supplies. The self-citations (e.g., [28], [29]) are used only as related-work context and are not the premise of the truncation claim. Section 7's admitted 'absence of a strong theoretical foundation' is a support gap, not circularity. A more serious concern, but a correctness risk rather than a circular one, is the missing length-debiased baseline: truncating both chosen and rejected responses to equal length removes the response-length signal that prior work [10] shows DPO can exploit, so the improvements could stem from length debiasing rather than early-token concentration. Overall, the derivation is not circular in its load-bearing parts; the definitional identity is a minor framing issue.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities. Its free parameters are the hand-chosen truncation ratios and decoding thresholds; none are fitted to data. The main unstated assumption is the identity of the model p in Eq. (1), which is load-bearing for the interpretation but not for the empirical results.

free parameters (3)
  • Truncation ratios = 50%, 40%, 33%, 25% (and 30%, 20%, 10% for UltraFeedback tasks)
    Hand-chosen scanning values for the independent variable; not fitted to data.
  • Length t in Length Control Decoding = swept across values shown in Figure 4
    Hand-chosen switch point from DPO to reference policy; not fitted to a single value.
  • KL threshold b in KL Threshold Control Decoding = swept across values shown in Figure 4
    Hand-chosen threshold for switching between DPO and reference policies; not fitted to a single value.
assumptions (3)
  • domain assumption Bradley-Terry preference model for reward learning
    Used in Section 3.2 and Appendix A.2; standard in RLHF, but it is an assumption about how preferences are generated.
  • domain assumption The per-token reward signal R_t is defined with an unspecified model p whose log-probabilities reflect preference-relevant content
    Eq. (1) in Section 3.1 defines R_t using log p(y_t|...) - log p(z_t|...), but the paper never specifies which model p is used; the claim that early tokens carry the signal depends on this model being a faithful proxy for human preference.
  • standard math Autoregressive token-level MDP with additive rewards
    Appendix A.1 frames generation as an MDP; this is standard and not controversial, but underlies the per-token decomposition.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?." pith.science (2026). https://pith.science/paper/FD6I66JQ

@misc{pith2026250517122,
  author       = {Pith},
  title        = {Pith review of: Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FD6I66JQ}},
  note         = {Machine review of arXiv:2505.17122}
}
read the original abstract

Aligning large language models (LLMs) with human preferences remains a key challenge in AI. Preference-based optimization methods, such as Reinforcement Learning with Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on human-annotated datasets to improve alignment. In this work, we identify a crucial property of the existing learning method: the distinguishing signal obtained in preferred responses is often concentrated in the early tokens. We refer to this as shallow preference signals. To explore this property, we systematically truncate preference datasets at various points and train both reward models and DPO models on the truncated data. Surprisingly, models trained on truncated datasets, retaining only the first half or fewer tokens, achieve comparable or even superior performance to those trained on full datasets. For example, a reward model trained on the Skywork-Reward-Preference-80K-v0.2 dataset outperforms the full dataset when trained on a 40\% truncated dataset. This pattern is consistent across multiple datasets, suggesting the widespread presence of shallow preference signals. We further investigate the distribution of the reward signal through decoding strategies. We consider two simple decoding strategies motivated by the shallow reward signal observation, namely Length Control Decoding and KL Threshold Control Decoding, which leverage shallow preference signals to optimize the trade-off between alignment and computational efficiency. The performance is even better, which again validates our hypothesis. The phenomenon of shallow preference signals highlights potential issues in LLM alignment: existing alignment methods often focus on aligning only the initial tokens of responses, rather than considering the full response. This could lead to discrepancies with real-world human preferences, resulting in suboptimal alignment performance.

Figures

Figures reproduced from arXiv: 2505.17122 by the authors.

Figure 1
Figure 1. An example illustrating the phenomenon of shallow preference signals. It demonstrates how the relative [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 2
Figure 2. The x-axis represents the response truncation length and ratio, while the y-axis shows the accuracy of DPO [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: KL Divergence between the DPO model and the reference model at each token position. The plot shows that [PITH_FULL_IMAGE:figures/full_fig_p009_3.png]
Figure 4
Figure 4. Figure 4: Reward and corresponding KL Divergence for the baseline and two different control strategies. The blue [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Accuracy of DPO implicit reward in predicting the relative quality of responses on the human-generated [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 24 canonical work pages

  1. [10]

    Disentangling length from quality in direct preference optimization

    Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 4998–5017. Association for Computa...

    work page 2024

  2. [1]

    Aligning large language models with human: A survey

    Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966, 2023

    arXiv 2023

  3. [2]

    Skywork-reward: Bag of tricks for reward modeling in llms

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. CoRR, abs/2410.18451, 2024

    arXiv 2024

  4. [3]

    Ultrafeedback: Boosting language models with high-quality feedback

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. CoRR, abs/2310.01377, 2023

    arXiv 2023

  5. [4]

    Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as ...

    arXiv 2021

  6. [5]

    Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    arXiv 2022

  7. [6]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human fee...

    work page 2022

  8. [7]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference ...

    work page 2023

Show all 59 references
  1. [8]

    From lists to emojis: How format bias affects model alignment

    Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, and Tong Zhang. From lists to emojis: How format bias affects model alignment. CoRR, abs/2409.11704, 2024

    2024 arXiv

  2. [9]

    Offsetbias: Leveraging debiased data for tuning evaluators

    Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Leveraging debiased data for tuning evaluators. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Fl...

    2024

  3. [11]

    KTO: model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: model alignment as prospect theoretic optimization. CoRR, abs/2402.01306, 2024

    2024 arXiv

  4. [12]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling. CoRR, abs/2403.13787, 2024

    2024 arXiv

  5. [13]

    Predictive pipelined decoding: A compute-latency trade-off for exact LLM decoding

    Seongjun Yang, Gibbeum Lee, Jaewoong Cho, Dimitris Papailiopoulos, and Kangwook Lee. Predictive pipelined decoding: A compute-latency trade-off for exact LLM decoding. Trans. Mach. Learn. Res., 2024, 2024

    2024

  6. [14]

    Asano, and Babak Ehteshami Bejnordi

    Benjamin Bergner, Andrii Skliar, Amelie Royer, Tijmen Blankevoort, Yuki M. Asano, and Babak Ehteshami Bejnordi. Think big, generate quick: Llm-to-slm for fast autoregressive decoding. CoRR, abs/2402.16844, 2024

    2024 arXiv

  7. [15]

    SAM decoding: Speculative decoding via suffix automaton

    Yuxuan Hu, Ke Wang, Xiaokang Zhang, Fanjin Zhang, Cuiping Li, Hong Chen, and Jing Zhang. SAM decoding: Speculative decoding via suffix automaton. CoRR, abs/2411.10666, 2024

    2024 arXiv

  8. [16]

    S2D: sorted speculative decoding for more efficient deployment of nested large language models

    Parsa Kavehzadeh, Mohammadreza Pourreza, Mojtaba Valipour, Tinashu Zhu, Haoli Bai, Ali Ghodsi, Boxing Chen, and Mehdi Rezagholizadeh. S2D: sorted speculative decoding for more efficient deployment of nested large language models. CoRR, abs/2407.01955, 2024. 12

    2024 arXiv

  9. [17]

    The unlocking spell on base llms: Rethinking alignment via in-context learning

    Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Raghavi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Repre...

    2024

  10. [18]

    Safety alignment should be made more than just a few tokens deep

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. CoRR, abs/2406.05946, 2024

    2024 arXiv

  11. [19]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett, editors,Ad...

    2017

  12. [20]

    Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Re...

    arXiv

  13. [21]

    Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler,...

    2023 arXiv

  14. [22]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  15. [23]

    On the weaknesses of reinforcement learning for neural machine translation

    Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforcement learning for neural machine translation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020

    2020

  16. [24]

    Implementation matters in deep policy gradients: A case study on PPO and TRPO

    Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on PPO and TRPO. CoRR, abs/2005.12729, 2020. 13

    2005 arXiv

  17. [25]

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. CoRR, abs/2305.10425, 2023

    2023 arXiv

  18. [26]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Rémi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, International ...

    2024

  19. [27]

    Generalized preference optimization: A unified approach to offline alignment

    Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Har- vey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. In Forty-first Internationa...

    2024

  20. [28]

    Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling

    Jiahao Qiu, Yifu Lu, Yifan Zeng, Jiacheng Guo, Jiayi Geng, Huazheng Wang, Kaixuan Huang, Yue Wu, and Mengdi Wang. Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling. CoRR, abs/2410.16033, 2024

    arXiv

  21. [29]

    Maxmin-rlhf: Alignment with diverse human preferences

    Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. Maxmin-rlhf: Alignment with diverse human preferences. arXiv preprint arXiv:2402.08925, 2024

    2024 arXiv

  22. [30]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952

    1952

  23. [31]

    Enable language models to implicitly learn self-improvement from data

    Ziqi Wang, Le Hou, Tianjian Lu, Yuexin Wu, Yunxuan Li, Hongkun Yu, and Heng Ji. Enable language models to implicitly learn self-improvement from data. CoRR, abs/2310.00898, 2023

    2023 arXiv

  24. [32]

    Mankowitz, Doina Precup, and Bilal Piot

    Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, an...

    2024

  25. [33]

    A minimaximalist approach to reinforcement learning from human feedback

    Gokul Swamy, Christoph Dann, Rahul Kidambi, Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024

    2024

  26. [34]

    A theoretical analysis of nash learning from human feedback under general kl-regularized preference

    Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. A theoretical analysis of nash learning from human feedback under general kl-regularized preference. CoRR, abs/2402.07314, 2024

    2024 arXiv

  27. [35]

    Llm-blender: Ensembling large language models with pairwise ranking and generative fusion

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61st Annual Meeting of the Association for Computation...

    2023

  28. [36]

    Liu, and Jialu Liu

    Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Op...

    2024

  29. [38]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  30. [39]

    Helpsteer: Multi-attribute helpfulness dataset for steerlm

    Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. Helpsteer: Multi-attribute helpfulness dataset for steerlm. In Kevin Duh, Helena Gómez-Adorno, and ...

    2024

  31. [40]

    Interpretable preferences via multi- objective reward modeling and mixture-of-experts

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi- objective reward modeling and mixture-of-experts. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EM...

    2024

  32. [41]

    Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. CoRR, abs/2308.09583, 2023

    2023 arXiv

  33. [42]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-1...

    2024

  34. [43]

    Process reward model with q-value rankings

    Wendi Li and Yixuan Li. Process reward model with q-value rankings. CoRR, abs/2410.11287, 2024

    2024 arXiv

  35. [44]

    Glore: When, where, and how to improve LLM reasoning via global and local refinements

    Alexander Havrilla, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Raileanu. Glore: When, where, and how to improve LLM reasoning via global and local refinements. In Forty-first International Conference on Mac...

    2024

  36. [45]

    Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. CoRR, abs/2209.13085, 2022

    2022 arXiv

  37. [46]

    Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter

    Jose A. Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. RUDDER: return decomposition for delayed rewards. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman G...

    2019

  38. [47]

    The effects of reward misspecification: Mapping and mitigating misaligned models

    Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022

    2022

  39. [48]

    Ball, Oleh Rybkin, Stephen Roberts, Tim Rocktäschel, and Edward Grefenstette

    Yingchen Xu, Jack Parker-Holder, Aldo Pacchiano, Philip J. Ball, Oleh Rybkin, Stephen Roberts, Tim Rocktäschel, and Edward Grefenstette. Learning general world models in a handful of reward-free deployments. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, a...

    2022

  40. [49]

    Correlated proxies: A new definition and improved mitigation for reward hacking, 2024

    Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Correlated proxies: A new definition and improved mitigation for reward hacking, 2024

    2024

  41. [50]

    Inform: Mitigating reward hacking in RLHF via information-theoretic reward modeling

    Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Inform: Mitigating reward hacking in RLHF via information-theoretic reward modeling. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, ...

    2024

  42. [51]

    Yuchen Zhu, Daniel Augusto de Souza, Zhengyan Shi, Mengyue Yang, Pasquale Minervini, Alexander D’Amour, and Matt J. Kusner. When can proxies improve the sample complexity of preference learning? CoRR, abs/2412.16475, 2024

    2024 arXiv

  43. [52]

    Rlhf workflow: From reward modeling to online rlhf

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024

    2024 arXiv

  44. [53]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324, 1952

    1952

  45. [54]

    Patterson, Joseph Gonzalez, Urs Hölzle, Quoc V

    David A. Patterson, Joseph Gonzalez, Urs Hölzle, Quoc V . Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean. The carbon footprint of machine learning training will plateau, then shrink. Computer, 55(7):18–28, 2022

    2022

  46. [55]

    Openrlhf: An easy-to-use, scalable and high-performance rlhf framework

    Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143, 2024. 15

    2024 arXiv

  47. [56]

    Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, 2024

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, 2024

    2024

  48. [57]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github. com/tatsu-lab/alpaca_eval, 5 2023

    2023

  49. [58]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/ tatsu-lab/stanford_alpaca, 2023

    2023

  50. [59]

    Understanding dataset difficulty with V-usable information

    Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machin...

    2022

  51. [2024]

    OpenReview.net, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.