Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling

Houqiang Li; Jianfeng Cai; Jinhua Zhu; Li Li; Ruopei Sun; Wengang Zhou; Yue Wang

REVIEW 4 major objections 4 minor 57 references

Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that training a reward model on response-conditioned pairs — comparing a prompt against a length-augmented version of itself with the response held fixed — converts implicit length bias into explicit length…

desk verdict Rc-BT is a genuinely new idea — same response compared under length-augmented and original prompts — and the RM evidence is worth taking seriously, but missing dataset-size controls and a length-biased DPO judge keep the central claim from being fully settled. read the letter →

arxiv 2502.00814 v2 pith:SPCWOQ7B submitted 2025-02-02 cs.LG cs.CL

Jianfeng Cai , Jinhua Zhu , Ruopei Sun , Yue Wang , Li Li , Wengang Zhou , Houqiang Li This is my paper · ORCID

classification cs.LGcs.CL

keywords lengthbiasrewardhackingpreferencelearningRLHFdirectoptimizationBradley-Terrymodelinstructionfollowing

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Length bias — reward models favoring longer responses regardless of quality — is usually treated as noise to be suppressed. This paper argues instead that the bias can be converted into explicit, usable understanding of length, and proposes a Response-conditioned Bradley-Terry (Rc-BT) framework that does so. The training data compare a prompt against a length-augmented version of itself while holding the response fixed: for a chosen response $y_w$ that violates an added length constraint the original prompt is preferred, and for a rejected response $y_l$ that satisfies the constraint the constrained prompt is preferred. On a debiased quality evaluation set the resulting reward model beats the baseline by 10.4 to 16.9 accuracy points and the ODIN baseline by up to 13.4 points, and the DPO variant lifts the AlpacaEval Quality Win Ratio by up to 21.8 points while matching or beating LIFT-plus on length-instruction adherence.

What carries the argument

The Response-conditioned Bradley-Terry (Rc-BT) objective, built from two constructed preference pair types. For a chosen response $y_w$, a length-augmented prompt $x^1_l$ is generated so that $y_w$ violates its length constraint, and $(x, y_w)$ is labeled preferable to $(x^1_l, y_w)$ (Eq. 2). For a rejected response $y_l$, a length-augmented prompt $x^2_l$ is generated so that $y_l$ satisfies its constraint, and $(x^2_l, y_l)$ is labeled preferable to $(x, y_l)$ (Eq. 3). Maximum-likelihood training on the union of these pairs (Eq. 4) forces the model to score the same response differently depending on whether the prompt demands a length the response can meet — the mechanism the paper credits with both mitigating length bias and learning to follow length instructions. The same pairs plug into reward modeling (Eq. 6) and, after the DPO-style derivation in Appendix B, into direct policy optimization (Eq. 8).

What would settle it

Have human annotators judge a random sample of the constructed pairs $D_{Rc}$ — comparing $(x, y_w)$ against $(x^1_l, y_w)$ and $(x^2_l, y_l)$ against $(x, y_l)$ — and measure their agreement with the labels that Eqs. (2) and (3) assign. The central claim predicts agreement comparable to the 90.5% human consistency the paper reports for $D^q_{\mathrm{eval}}$; substantially lower agreement would mean the training signal is an artifact of construction and the reported gains would not transfer to true human preference.

Watch

Extended reading notes

Core claim

The paper's central claim is that a reward model trained on response-conditioned preference pairs can distinguish human semantic preference from length requirement, so that length bias is mitigated without sacrificing the model's awareness of length. The construction holds the response fixed and asks which prompt — the original or a length-augmented variant — the response fits better, inverting the standard preference format in which the prompt is fixed and the response varies. The authors argue that this inversion is what turns an implicit, exploitable bias into explicit length understanding, and they show the same mechanism works at the reward-model stage (Rc-RM) and the policy stage (Rc-DPO), yielding markedly higher scores on a debiased quality set while matching a dedicated length-instruction method on length adherence.

Load-bearing premise

The load-bearing premise is that the constructed preference labels are genuine: a chosen response that violates an added length constraint really should be judged worse under the constrained prompt, and a rejected response that satisfies the constraint really should be judged better — if that is not how humans actually evaluate these pairs, the model learns an artificial ordering rather than a true separation of length from quality.

Editorial extensions

If this is right

Reward models trained with Rc-RM beat the Baseline by 10.4 to 16.9 accuracy points and ODIN by up to 13.4 points on the debiased quality evaluation set $D^q_{\mathrm{eval}}$, across Qwen2-1.5B, Qwen2.5-7B, and Llama-3.1-8B (Table 3).
Rc-DPO raises the AlpacaEval Quality Win Ratio by up to 21.8 points over Baseline (Llama-3.1-8B-Instruct: 42.52 to 64.34) while producing substantially shorter responses (Table 2).
Length instruction adherence matches or beats LIFT-plus: Rc-RM reaches 88.14% and 92.31% Length Eval Acc on Qwen2.5-7B-Base and Qwen2.5-7B-Instruct versus LIFT-plus's 84.94% and 83.97% (Table 8).
The two constructed pair types are complementary: using only $D^c_{Rc}$ or only $D^r_{Rc}$ drops Quality Eval Acc toward Baseline and Length Eval Acc toward chance (Table 5).
The same response-conditioned construction transfers to format bias, improving accuracy on FormatBiasEval by up to 10.9 points over the prompt-conditioned baseline (Table 14).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's 90.5% human-agreement check validates the debiased evaluation set $D^q_{\mathrm{eval}}$, not the constructed training pairs $D_{Rc}$; a direct human preference study on $D_{Rc}$ would confirm whether the Eq. (2)-(3) labels are genuine length-conditioned preferences or an ordering the model learns to satisfy mechanically.
The response-conditioned template should transfer to other spurious correlates that can be appended to a prompt — format, citation requirements, politeness register — with the FormatBiasEval result as a first piece of evidence, and a testable prediction that the same accuracy gains appear on debiased evaluation sets for those attributes.
The contrast with LIFT-plus's 'short bias' suggests that the response-conditioned format itself, not the length instruction, is the active ingredient; a controlled ablation that keeps the same augmented pairs but trains in the standard prompt-conditioned format would isolate that factor.
If the gains hold at larger scales, evaluation practice would need adjusting, since length-imbalanced evaluation sets such as the original $D_{\mathrm{eval}}$ (59.78% of chosen responses longer than rejected ones) systematically understate a debiased model's quality accuracy.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The short version: this is a real idea, not a repackaging. The Rc-BT construction — hold the response fixed, compare the original prompt against a length-augmented version, (x, yw) preferred to (x1_l, yw) when yw violates the added constraint, and (x2_l, yl) preferred to (x, yl) when yl satisfies it — reframes length bias as an instruction-following problem rather than a nuisance to regress out. That framing is new relative to ODIN, R-DPO, and LIFT, and it is the right kind of new: simple, plugging into both RM and DPO training, and giving a principled reason to keep length sensitivity instead of suppressing it.

The paper does several things well beyond the headline. The preliminary diagnostics (empty and random prompts still score ~60% accuracy, 85%+ preference consistency, the length-reward correlation) are clean. The Dq_eval construction uses programmatic length verification plus human agreement of 90.5%, which is a reasonable debiased eval set. The ablation showing Dc_Rc and Dr_Rc are individually near-useless but jointly effective is genuinely informative. The "short bias" critique of LIFT-plus is fair.

Soft spots, in proportion. The biggest is the data-size confound: Rc-RM trains on roughly twice the preference pairs of Baseline, with no matched control. A 2× Drm baseline is needed before the 10–17 point Quality Eval Acc gains can be attributed to the construction rather than the extra data. The gains are large enough that I suspect the construction matters, but that is not yet established. Fixable. Second is the DPO judge: GPT-4o with an explicit conciseness criterion, and Rc-DPO produces the shortest responses. That inflates its Quality Win Ratios. The paper cites Length-Controlled AlpacaEval and then doesn't use it — a visible omission for a paper about length bias. The RM results don't share this flaw; the DPO claims should be rechecked with a length-controlled judge. Third, I think the stress-test overshoots on the unvalidated DRc labels. Preferring (x, yw) over (x1_l, yw) when yw violates an explicit length constraint is close to a mechanical truth — humans would agree with that ordering at very high rates. A small agreement study on DRc would still be cheap and worth doing, but I don't regard the training signal as a fabrication risk. Fourth, the Rc-DPO derivation: the π(y) marginals in Eq. 18 do cancel since the same y appears in both ratios, so that step is fine. The real gap is the final loss's πθ(x, y) — an autoregressive LM parameterizes πθ(y|x), not the joint, so computing the loss requires a prompt prior the paper never defines. The natural implementation presumably uses response likelihoods and lets the prior drop; that's a reasonable heuristic, but it should be stated.

Who this is for: anyone in reward modeling or direct preference optimization. It deserves a serious referee — novel, relevant, and the problems are addressable rather than structural. Send it out; ask for the matched-data baseline, a length-controlled judge, and a short human check on DRc. With those, it could be a solid paper.

Recommendation: send to review; expect major revision.

Referee Report

4 major / 4 minor

Summary. The paper proposes Response-conditioned Bradley-Terry (Rc-BT), a preference-learning framework that constructs additional preference pairs by adding length constraints to the original prompt while holding the response fixed. The resulting pairs (x, x1_l, yw) and (x2_l, x, yl) are used together with the original dataset Drm to train Rc-RM and Rc-DPO. The authors report that Rc-RM improves accuracy on a debiased quality evaluation set Dq_eval by 10.4–16.9 points over Baseline (Table 3), that Rc-DPO improves Quality Win Ratio on AlpacaEval by up to 21.8 points (Table 2), and that the method matches or beats LIFT-plus on length-instruction adherence (Tables 8, 10, 11). The paper also documents length-bias diagnostics, ablations for the two augmented subsets, and extensions to HH-RLHF and format bias.

Significance. If the reported effects are real, the framework is a simple and potentially useful way to make reward models and DPO policies aware of length instructions without sacrificing semantic quality. The paper's preliminary diagnostics of length bias (empty/random prompts, consistency scores, length-score correlation) are clear and reproducible in spirit, and the experiments cover multiple base models, reward modeling, PPO, DPO, and cross-dataset generalization. The central contribution, however, rests on three load-bearing points that are not yet established: the validity of the synthetic training preferences, the comparability of training data sizes, and the appropriateness of the automatic judge used for the headline DPO results.

major comments (4)

[4.2, Eqs. (2)–(3); Appendix D.2] The constructed training pairs in DRc are never validated against human judgments. The paper reports human agreement only for the evaluation dataset Dq_eval (Appendix D.2, 20 volunteers × 10 samples, 90.5% agreement), not for the pairs (x, x1_l, yw) and (x2_l, x, yl) that define the Rc-BT objective (Eq. 4). Because the whole method assumes that violating a length constraint makes a response worse and satisfying it makes it better, a direct human-agreement test on a sample of DRc is needed. Without it, the gains on Dq_eval could reflect the model learning the construction rule rather than a genuine separation of length from semantic quality.
[5.1, Table 13, Limitations] Rc-RM and Rc-DPO train on Drm ∪ DRc, which roughly doubles the number of preference pairs compared with Baseline, and the paper does not include a dataset-size-matched Baseline. Table 13 shows a monotonic improvement from 60.75% to 71.47% as the fraction of DRc increases from 0% to 100%, so the large gains in Table 3 may be partly or wholly due to seeing more preference pairs rather than to the response-conditioned construction. The authors state the doubling explicitly in the Limitations section; the revision should add a control trained on twice as many original-format pairs (e.g., repeating Drm or adding held-out preference data) and report the comparison.
[Appendix E.1, Figure 6; Table 2] The DPO judge used for Quality Win Ratio explicitly includes 'Conciseness: the response should avoid redundancy, unnecessary verbosity, or inclusion of irrelevant information' as a scoring criterion. Since Rc-DPO produces substantially shorter outputs (e.g., 208 vs. 517 tokens on Qwen2.5-7B-Base in Table 2), the judge rewards the very dimension the method changes, confounding the claim that Rc-DPO improves semantic quality. The paper should validate the judge against human ratings on the evaluated pairs, use a judge prompt without length/conciseness criteria, or report length-controlled win ratios such as AlpacaEval LC.
[Appendix B, Eq. (18); Section 4.4, Eq. (8)] The Rc-DPO derivation is incomplete. In Eq. (18) Bayes' rule introduces marginal distributions π*(y) and πref(y), which are dropped without comment; they do cancel between the two terms of the sigmoid for a fixed y, but the paper should state this. More importantly, Eq. (8) is written with joint probabilities πθ(x,y), yet a language-model policy only defines πθ(y|x), and no definition of the prompt prior πθ(x) is given. As a result, the objective actually optimized in the Rc-DPO experiments is unspecified. The revision should either provide a complete derivation with an explicit prompt-prior assumption or state the approximation used in the implementation.

minor comments (4)

[General] There are several typographical errors: 'underlining' should be 'underlying' in Section 4.1, 'preforming' should be 'performing' in Appendix B, 'Tabel 10' should be 'Table 10' in Appendix E.4, and 'Figture 9' should be 'Figure 9' in Appendix E.6.
[Appendix E.2, Figure 7] The text refers to 'Figure 7(c) and 7(d)', but the figure has only panels (a) and (b); the cross-references should be corrected.
[E.3, Table 9] The authors note that the LIFT-plus result on Qwen2.5-1.5B-Instruct is 'somewhat unreasonable' and defer analysis to future work; this is honest, but the reader would benefit from a sanity check (e.g., confirming the training did not diverge) before the table is presented as a comparison.
[4.2, Section 5.1] The construction recipe for DRc is not described: the paper never states how the length-augmented prompts x1_l and x2_l are generated, what word_num values are used, or how the 'violates/satisfies' conditions are verified. This is needed for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training objective, evaluation sets, and comparison baselines are independently grounded.

full rationale

The core derivation is self-contained and non-circular. Rc-BT trains on Drm ∪ DRc, where Drm carries human preference labels from OpenAssistant and DRc is a constructed response-conditioned set (Eqs. 2–3). The reported quality gains are not forced by this construction: Dq_eval (Table 3) is built from the original human-preferred responses with GPT-4o rewrites plus programmatic length balancing, and Appendix D.2 reports 90.5% human agreement on Dq_eval, an external check independent of the model's training objective. The AlpacaEval Quality Win Ratio (Table 2) is judged by GPT-4o against SFT/Instruct baselines; because Rc-DPO outputs are shorter, any residual length bias in the judge would work against Rc-DPO, so the favorable result cannot be explained as the training rule being rediscovered in evaluation. Length Eval Acc (Table 8) measures whether the model learned the explicit length-constraint preference that DRc was designed to teach; reporting success there is the intended training outcome, not a circular prediction masquerading as an independent finding. The Rc-DPO derivation (Appendix B) is a standard DPO-style algebraic manipulation; the Bayes reparameterization in Eq. 18 is algebraically valid because the π(y) factors cancel, and it does not smuggle the conclusion into the assumptions. There are no load-bearing self-citations, no imported uniqueness theorem, and no fitted parameter renamed as a prediction. The unvalidated nature of the synthetic DRc labels is a legitimate correctness/robustness concern, but it is a question of whether the training signal reflects human preferences, not a circularity in the paper's derivation chain.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central method relies on one new synthetic data-construction rule (the response-conditioned preference pairs), one unstated assumption about joint probability factorization in the DPO derivation, and the standard RLHF/DPO assumptions. The evaluation datasets are also constructed by the authors, which adds a mild circularity burden. There are no invented physical entities or ungrounded scales beyond the λ and β hyperparameters.

free parameters (2)

lambda (λ) = not reported (implicitly 1.0)
Balances the Dc and Dr terms in Eq. 6; the paper does not state the value used in main experiments.
DPO beta (β) = not reported
Standard DPO temperature; value not given for Rc-DPO experiments.

assumptions (5)

domain assumption Bradley-Terry model with an underlying reward r*(x,y) for human preferences (Eq. 1)
Standard RLHF assumption, cited to Ouyang et al. 2022.
ad hoc to paper Constructed preference pairs (x, x1_l, yw) and (x2_l, x, yl) are valid: violating a length constraint makes a response worse under the constrained prompt, and satisfying it makes it better (Eqs. 2-3)
This is the load-bearing premise of the data augmentation; if the labels do not reflect human preferences, the training signal is wrong.
standard math The optimal policy under the response-conditioned RL objective is π* ∝ π_ref exp(r/β) (Eq. 13)
Standard DPO derivation; assumes the KL-constrained RL solution, as in Rafailov et al. 2024.
domain assumption π(x|y) can be reparameterized via Bayes' rule to π(x,y)/π(y), and π(y) cancels in the ratio, allowing the DPO loss to be written with joint probabilities π(x,y) (Eq. 18)
Assumes the language model defines a well-formed joint distribution over prompt-response pairs and that the prompt prior π(x) is unchanged during DPO training; the paper does not state or justify this.
domain assumption Dq_eval and Dl_eval are valid proxies for semantic quality and length adherence
The evaluation datasets are constructed with GPT-4o and length templates; human verification covers only 200 samples in Dq_eval, and no external validation is provided for Dl_eval.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling." pith.science (2026). https://pith.science/paper/SPCWOQ7B

@misc{pith2026250200814,
  author       = {Pith},
  title        = {Pith review of: Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SPCWOQ7B}},
  note         = {Machine review of arXiv:2502.00814}
}

abstract

Reinforcement Learning from Human Feedback (RLHF) has achieved considerable success in aligning large language models (LLMs) by modeling human preferences with a learnable reward model and employing a reinforcement learning algorithm to maximize the reward model's scores. However, these reward models are susceptible to exploitation through various superficial confounding factors, with length bias emerging as a particularly significant concern. Moreover, while the pronounced impact of length bias on preference modeling suggests that LLMs possess an inherent sensitivity to length perception, our preliminary investigations reveal that fine-tuned LLMs consistently struggle to adhere to explicit length instructions. To address these two limitations, we propose a novel framework wherein the reward model explicitly differentiates between human semantic preferences and response length requirements. Specifically, we introduce a $\textbf{R}$esponse-$\textbf{c}$onditioned $\textbf{B}$radley-$\textbf{T}$erry (Rc-BT) model that enhances the model's capability in length bias mitigating and length instruction following, through training on our augmented dataset. Furthermore, we propose the Rc-RM and Rc-DPO algorithm to leverage the Rc-BT model for reward modeling and direct policy optimization (DPO) of LLMs, simultaneously mitigating length bias and promoting adherence to length instructions. Extensive experiments across various foundational models and datasets demonstrate the effectiveness and generalizability of our approach.

Figures

Figures reproduced from arXiv: 2502.00814 by the authors.

**Figure 2.** To illustrate the distinct data formats across different methods, we present the maximum length instruction [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Analysis of reward scores across models on [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: (a). Response rewriting template for the chosen response y (i) w ; (b). Response expansion template for the rejected response y (i) l ; (c). Response compression template for the rejected response y (i) l ; (d). Quality consistency verification template for assessing t…

**Figure 5.** Figure 5: The trajectories of Quality Eval Acc, Length Eval Acc, and Training Loss (Smoothed) for different reward [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

**Figure 6.** Figure 6: Response comparison template for evaluating the semantic quality between responses. [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]

**Figure 7.** Figure 7: Comparison of score differences between LIFT-plus and Rc-RM on Dmls eval under varying word_num constraints. An ideal reward model should show an initial increase in score difference followed by a return to its initial value. when word_num < lw, the length constraint …

**Figure 8.** Figure 8: The trajectories in Length Eval Acc and Quality Eval Acc of Qwen2.5-1.5B-Instruct with training steps. [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

**Figure 9.** Figure 9: The relationship between response length and [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

**Figure 10.** Figure 10: Variations in Rc-RM’s Length Eval Acc and Quality Eval Acc as Dr Rc increases. The green dashed line denotes the Baseline Quality Eval Acc, while the blue dashed line represents the Length Eval Acc of LIFT-plus [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]

**Figure 11.** Figure 11: Response length range generated by LIFT-plus trained on Llama-3.1-8B-Instruct using length instruction [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 14 canonical work pages

[1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[2]

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022 a . Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

arXiv 2022
[3]

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022 b . Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073

arXiv 2022
[4]

Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324--345

1952
[5]

Loredana Caruccio, Stefano Cirillo, Giuseppe Polese, Giandomenico Solimando, Shanmugam Sundaramurthy, and Genoveffa Tortora. 2024. Claude 2.0 large language model: Tackling a real-world classification problem with a new iterative prompt engineering approach. Intelligent Systems with Applications, 21:200336

2024
[6]

Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, and 13 others

Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, J \'e r \'e my Scheurer, Javier Rando, Rachel Freedman, Tomek Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip J.K. Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, and 13 others. 2023. https://openrevie...

work page 2023
[7]

Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, and Jun Zhu. 2024 a . Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.05369

arXiv 2024
[8]

Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. 2024 b . Odin: Disentangled reward mitigates hacking in rlhf. In Forty-first International Conference on Machine Learning

work page 2024

Show all 57 references

[9]

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, and 1 others. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6

2023
[10]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

2024 arXiv
[11]

Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024 a . Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475

2024 arXiv
[12]

Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2024 b . Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36

2024
[13]

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306

2024 arXiv
[14]

Leo Gao, John Schulman, and Jacob Hilton. 2023. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835--10866. PMLR

2023
[15]

Yang Gu, Yuhu Cheng, C. L. Philip Chen, and Xuesong Wang. 2022. https://doi.org/10.1109/TSMC.2021.3098451 Proximal policy optimization with policy feedback . IEEE Transactions on Systems, Man, and Cybernetics: Systems, 52(7):4600--4610

2022
[16]

Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.507 Reasoning with language model is planning with world model . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro...

2023 doi
[17]

Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170--11189

2024
[18]

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

2024 arXiv
[19]

Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 37--42

2023
[20]

Smith, Yejin Choi, and Hannaneh Hajishirzi

Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. 2024. https://openreview.net/forum?id=JMBWTlazjW Unpacking DPO and PPO : Disentangling best practices for learning from preference feedba...

2024
[21]

Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. In Advances in Neural Information Processing Systems, volume ...

2023
[22]

Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. 2024. https://doi.org/10.1145/3672456 Self-planning code generation with large language models . ACM Trans. Softw. Eng. Methodol., 33(7)

2024 doi
[23]

o pf, Yannic Kilcher, Dimitri von R \

Andreas K \"o pf, Yannic Kilcher, Dimitri von R \"u tte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Rich \'a rd Nagyfi, and 1 others. 2024. Openassistant conversations-democratizing large language model alignment. Advances i...

2024
[24]

Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. 2018. https://doi.org/10.18653/v1/P18-1165 Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning . In Proceedings of the 56th Annual Meeting of the Association for Computational ...

2018 doi
[25]

Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. 2019. https://openreview.net/forum?id=rkGG6s0qKQ The GAN landscape: Losses, architectures, regularization, and normalization

2019
[26]

Nathan Lambert and Roberto Calandra. 2023. The alignment ceiling: Objective mismatch in reinforcement learning from human feedback. arXiv preprint arXiv:2311.00168

2023 arXiv
[27]

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, and 1 others. 2024. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787

2024 arXiv
[28]

Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267

2023 arXiv
[29]

Fei Liu and 1 others. 2020. Learning to summarize from human feedback. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

2020
[30]

Do Xuan Long, Hai Nguyen Ngoc, Tiviatis Sim, Hieu Dao, Shafiq Joty, Kenji Kawaguchi, Nancy F Chen, and Min-Yen Kan. 2024. Llms are biased towards output formats! systematically evaluating and mitigating output format bias of llms. arXiv preprint arXiv:2408.08656

2024 arXiv
[31]

Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583

2023 arXiv
[32]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing sys...

2022
[33]

Alexander Pan, Kush Bhatia, and Jacob Steinhardt. 2022. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations

2022
[34]

Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. 2024. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159

2024 arXiv
[35]

Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. https://doi.org/10.18653/v1/2022.findings-acl.165 BBQ : A hand-built bias benchmark for question answering . In Findings of the Association for ...

2022 doi
[36]

Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. https://openreview.net/forum?id=KmtVD97J43e Synchromesh: Reliable code generation from pre-trained language models . In International Conference on Learning Representations

2022
[37]

James Queeney, Yannis Paschalidis, and Christos G Cassandras. 2021. https://proceedings.neurips.cc/paper_files/paper/2021/file/63c4b1baf3b4460fa9936b1a20919bec-Paper.pdf Generalized proximal policy optimization with sample reuse . In Advances in Neural Information Processing S...

2021
[38]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

2024
[39]

Alexandre Ram \'e , Nino Vieillard, L \'e onard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. 2024. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.12187

2024 arXiv
[40]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

2017 arXiv
[41]

Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. arXiv preprint arXiv:2310.05199

2023 arXiv
[42]

Solomon Eyal Shimony. 1993. The role of relevance in explanation i: Irrelevance as statistical independence. International Journal of Approximate Reasoning, 8(4):281--324

1993
[43]

Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2023. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716

2023 arXiv
[44]

Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460--9471

2022
[45]

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008--3021

2020
[46]

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

2023 arXiv
[47]

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, and 1 others. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empiri...

2020
[48]

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024 a . https://arxiv.org/abs/2401.08417 Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation . Preprint, arXi...

2024 arXiv
[49]

Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. 2024 b . https://openreview.net/forum?id=6XH8R7YrSk Is DPO superior to PPO for LLM alignment? a comprehensive study . In Forty-first International Conference on Machine Learning

2024
[50]

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, and 1 others. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

2024 arXiv
[51]

Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, and 1 others. 2023. Deepspeed-chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales. arXiv...

2023 arXiv
[52]

Weizhe Yuan, Ilia Kulikov, Ping Yu, Kyunghyun Cho, Sainbayar Sukhbaatar, Jason Weston, and Jing Xu. 2024. Following length constraints in instructions. arXiv preprint arXiv:2406.17744

2024 arXiv
[53]

Zirui Zhao, Wee Sun Lee, and David Hsu. 2024. Large language models as commonsense knowledge for large-scale task planning. Advances in Neural Information Processing Systems, 36

2024
[54]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

2023
[55]

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

2019 arXiv
[56]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[57]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[2] [2]

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022 a . Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

arXiv 2022

[3] [3]

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022 b . Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073

arXiv 2022

[4] [4]

Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324--345

1952

[5] [5]

Loredana Caruccio, Stefano Cirillo, Giuseppe Polese, Giandomenico Solimando, Shanmugam Sundaramurthy, and Genoveffa Tortora. 2024. Claude 2.0 large language model: Tackling a real-world classification problem with a new iterative prompt engineering approach. Intelligent Systems with Applications, 21:200336

2024

[6] [6]

Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, and 13 others

Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, J \'e r \'e my Scheurer, Javier Rando, Rachel Freedman, Tomek Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip J.K. Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, and 13 others. 2023. https://openrevie...

work page 2023

[7] [7]

Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, and Jun Zhu. 2024 a . Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.05369

arXiv 2024

[8] [8]

Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. 2024 b . Odin: Disentangled reward mitigates hacking in rlhf. In Forty-first International Conference on Machine Learning

work page 2024