Pith. sign in

REVIEW 4 major objections 6 minor 49 references

Debiasing Online Preference Learning via Preference Feature Preservation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Online preference learning collapses the variety of human preferences onto whichever feature dominates the binary labels; PFP counters this by extracting the preference features behind the labels, training a classifier to map each…

desk verdict PFP is a genuinely new mechanism for preserving feature diversity in online preference learning, but the paper's own variance table undermines the headline comparison and the debiasing metric is circular, so the core claims need referee attention and human-validated labels. read the letter →

arxiv 2506.11098 v1 pith:CEUDEHRP submitted 2025-06-06 cs.LG cs.AI

Dongyoung Kim , Jinsung Yoon , Jinwoo Shin , Jaehyung Kim This is my paper
classification cs.LGcs.AI
keywords preferencelearningLLMalignmentonlineoptimizationDirectfeaturedebiasingsystempromptconditioninglengthbiasdistributionpreservation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard preference learning for LLMs compresses human preferences into binary pairwise comparisons and scalar rewards. The paper argues that this compression lets online iterative training drift toward the most dominant response features, so a model trained over several rounds becomes stylistically and behaviorally narrower, more verbose, and less safe. PFP is offered as a fix: it infers a five-dimensional preference feature for each offline preference pair, trains a feature classifier, and then, at every online iteration, assigns each new instruction features whose overall distribution is forced to match the original human feature distribution. Those features are turned into system prompts, so the model generates and judges responses with the full range of preferences in mind. The experiments report that PFP improves AlpacaEval 2.0 length-controlled win rate to 15.24%, versus 14.23% for SPA and 13.13% for Iterative DPO, with corresponding MT-Bench gains, while keeping the feature distribution close to the initial model's.

What carries the argument

The load-bearing mechanism is a preference-feature classifier combined with a distribution-preserving relabeling step. The paper defines preference features as one selection from each of five dimensions—style, tone, harmlessness, user background knowledge, and informativeness—so each preference is a five-tuple of sub-features. A DeBERTa-v3-large classifier predicts these features for new instructions, and the relabeling step solves a constrained optimization (Eq. 3) with the Sinkhorn-Knopp algorithm so that the aggregate distribution of assigned features across an online batch matches the empirical distribution extracted from the offline human preference data. A system-prompt synthesizer then converts sampled features into natural-language system prompts, and two system prompts are sampled per instruction (with temperature decreasing across iterations) so the model sees contrasting preference instantiations without collapsing response diversity.

What would settle it

Collect human annotations of the preference features on a random sample of UltraFeedback pairs and on the responses each method generates for AlpacaEval 2.0 prompts. Compute the agreement between GPT-4o's feature labels and human labels, and re-run the KL-divergence comparison with human-labeled distributions; if GPT-4o and humans disagree substantially, or if PFP no longer beats SPA under human-judged feature preservation, the debiasing claim is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that the bias accumulated in online preference learning is a feature-distribution bias, not just a reward-model bias, and that preserving the distribution of preference features across iterations removes it. On the paper's own terms, PFP demonstrates this: the KL divergence between the feature distribution of generated responses and the initial model's distribution stays nearly flat across four iterations, whereas SPA and Iterative DPO drift. This preservation is what the authors credit for the higher AlpacaEval 2.0 and MT-Bench scores, for the sustained improvement in Anthropic-HHH harmlessness and honestness, and for the side effect of shortening responses (average length 1187 characters versus 2412 for SPA and 1709 for Iterative DPO).

Load-bearing premise

The load-bearing premise is that GPT-4o's feature assignments for the training data and its feature judgments for the debiasing metric capture the features that actually determined the human preference labels; if those labels are noisy or style-driven, the preservation target is the wrong target.

Editorial extensions

If this is right

  • If PFP is correct, online preference learning can be run for more iterations without the usual collapse toward one dominant style, because the feature distribution is held fixed rather than allowed to drift.
  • Models trained with PFP should maintain or improve safety-relevant attributes such as harmlessness and honestness across iterations, where SPA and Iterative DPO decline.
  • Length bias is reduced as a side effect of feature preservation: PFP's average response length stays near 1,187 characters, far below the 2,412 of SPA and 1,709 of Iterative DPO, without explicit length penalties.
  • The framework is compatible with existing offline preference optimizers: any method such as DPO can be run inside the loop once system-prompt-augmented preference data is constructed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would test PFP with a cheaper feature extractor: the paper notes its reliance on GPT-4o as a practical cost, and if smaller models can provide feature labels, the framework becomes more deployable.
  • The feature taxonomy is a fixed human choice, so the method's debiasing only covers the five dimensions and twenty-five sub-features it tracks; biases outside that vocabulary would remain invisible to the preservation step.
  • Because the debiasing metric and the training labels both come from the same model family (GPT-4o), an independent test would be to measure whether PFP's preserved distribution matches human-labeled feature distributions; the paper reports no such validation.
  • The system-prompt scheduling idea could be studied separately as a curriculum: if temperature reduction is what drives the gain, then the feature-preservation step and the curriculum step may be disentangled in future ablations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes PFP, an online preference learning framework that extracts five-dimensional preference features from a seed dataset using GPT-4o, trains DeBERTa-based classifiers to assign features to new instructions, adjusts the assignments via Sinkhorn-Knopp to match the seed feature distribution, and conditions response generation on system prompts synthesized from the assigned features. Training uses SPA/DPO as the base algorithm. Experiments on UltraFeedback with Mistral-7B report improvements on AlpacaEval 2.0, MT-Bench, and Anthropic-HHH, together with a reduced KL divergence between the model's response feature distribution and the seed distribution, supporting the claim that PFP mitigates preference-feature bias during online learning.

Significance. If the debiasing effect is real, PFP addresses a genuine problem: online preference learning can collapse response diversity and amplify dominant features, and conditioning on system prompts is a plausible mechanism. The paper provides component ablations (Tables 2 and 3), a public code link, results on a second backbone, and a multi-seed robustness table, which are concrete strengths. However, the evidence for the central debiasing claim is currently incomplete: the feature labels used for both training and evaluation are generated by the same model without human validation, and the head-to-head performance gain over SPA is not statistically established. With human-validated feature labels, an independent bias metric, and a proper significance analysis, this would be a valuable contribution to online preference learning.

major comments (4)
  1. [§4.1, §5.1, Eq. (4)] The debiasing claim rests on GPT-4o feature labels that are used both to train the classifier q_phi (Sec. 4.1) and to compute the KL-divergence metric (Eq. 4) on model responses. No human validation of these labels is reported, and the Limitations section only mentions compute cost and SFT-stage impact. Table 8 shows per-class classifier accuracy of 0.496–0.688, so per-instruction labels are noisy; Eq. (3) can only match the aggregate seed distribution and cannot correct noise in the target distribution itself. Consequently, the shrinking KL divergence in Fig. 1(a) may partly reflect that both the reference and PFP responses are labeled by the same extractor that produced the training labels. I request a human-annotation study on a subset of extracted features (with agreement rates) and an independent feature judge or a different model/prompt for the debiasing metric.
  2. [Table 9, Appendix E] The claim that the 1.46% mean AlpacaEval LC win-rate advantage over SPA is statistically supported is not justified. With three seeds, PFP's win rates are 15.24, 14.38, 14.22 and SPA's are 14.23, 12.58, 12.64; the best SPA run (14.23) essentially equals the worst PFP run (14.22). The reported variances (0.29 vs 0.84) come from n=3, and no significance test is given. Table 1 reports single runs without error bars. Please provide per-seed results for the main tables, a paired test or confidence interval, and a clear statement of whether the mean gap is actually significant.
  3. [References, §3] The SPA baseline is cited as "Kim et al., 2025. Debiasing online preference learning via preference feature preservation," which is the title and author list of the present paper. As written, the paper cites itself as the baseline, so the original SPA source is missing and the comparison is not reproducible. The reference must be corrected to the actual prior work. In addition, Sec. 5.1 states that the self-refine step is removed from SPA "to reduce the number of hyperparameters"; this deviation from the original algorithm should be disclosed explicitly in the comparison, as it can affect performance.
  4. [Algorithm 1, §4.3, §5.1] There is an inconsistency about what the initial DPO model is trained on. Algorithm 1 writes π0 ← DPO(πinit, πinit, DFE), where DFE (Sec. 4.1) contains tuples (p, x, y_l, y_w) without system prompts, whereas Sec. 5.1 says the seed data "with added system prompts" are used for initial DPO training. Please clarify the exact training data for π0. Also specify whether PFP is evaluated on AlpacaEval 2.0 and MT-Bench with or without system prompts; if system prompts are used at evaluation, the benchmark protocol differs from the baselines, and if not, the model faces a train/eval mismatch.
minor comments (6)
  1. [Table 10] In Table 10 the SFT row reports "xx.xx" for the average Anthropic-HHH accuracy; please fill in the value.
  2. [Table 12] The title of Table 12 reads "Valuation results on various benchmark"; it should be "Evaluation results."
  3. [Page 2, Abstract] There are typos, including "exibits" for "exhibits" and "prefernece" for "preference".
  4. [Table 1] Table 1 reports single runs without specifying the number of seeds; please state the number of random seeds and whether the reported values are means or single runs.
  5. [Fig. 3] The evaluation of Anthropic-HHH in Fig. 3 should specify whether the model's internal preference (Eq. 1) is computed with the initial DPO or the current policy as reference, and whether system prompts are included during benchmark scoring.
  6. [Table 7] The LLaMA3.1-8B results include only SFT, DPO, and PFP; adding SPA and Iterative DPO on this backbone would strengthen the comparison.

Circularity Check

3 steps flagged · score 6.0 of 10

PFP's debiasing evidence is partly circular: GPT-4o supplies both the feature labels used to build system prompts and the KL-divergence metric used to measure debiasing, Eq. 3 forces the input feature distribution by construction, and the SPA subroutine/baseline is cited to the authors' own paper.

  1. self citation load bearing [Sec. 3 (Online preference learning and SPA) and References]
    "Since choosing the proper reward model is non-trivial in our problem, we adopt SPA (Kim et al., 2025) as the online preference learning algorithm. ... (yw, yl) = (y1, y2) if pθt−1(y1 ≻ y2|x) > 0.5, (yw, yl) = (y2, y1) else (2). References: Dongyoung Kim, Jaehyung Kim, Jinsung Yoon, and Jinwoo Shin. 2025. Debiasing online preference learning via preference feature preservation."

    The only bibliographic support given for SPA—the core online-preference subroutine used to construct every PFP dataset and the primary baseline in every comparison—is a citation whose author list and title are exactly the present paper. Thus the algorithm's definition, its implicit-reward labeling rule (Eq. 2), and its expected behavior are attributed to a self-citation, not to an independently verifiable external source. Because all PFP runs are built 'with SPA' and all headline comparisons are against 'SPA', the comparison inherits the unverified self-referential definition. This is load-bearing: if the citation does not point to a real external method, the PFP derivation chain and the SPA baseline both reduce to the authors' own unpublished/self-referential text.

  2. other [Sec. 5.1 (Evaluations), Eq. 4; Sec. 4.1 (Feature extraction)]
    "In addition, to measure the debiasing effect on preference features, we extract the preference features from the responses generated for the test instructions in AlpacaEval 2.0. Then, we use GPT-4o (OpenAI, 2024a) to infer the most prominent preference feature in each response. After obtaining the feature distribution, we measure how the KL divergence between this and the feature distribution of the responses of the initial model... DKL(PInit.Model ∥ Ptarget) = Σx PInit.Model(x) log(PInit.Model(x)/Ptarget(x)). (4)"

    The same GPT-4o-based feature extractor is used in two roles: (i) in Sec. 4.1 it produces the preference-feature labels p that train the classifier and generate the system prompts; (ii) in Sec. 5.1 it scores the responses whose feature distribution is the debiasing metric in Eq. 4. Because PFP's system prompts are synthesized from GPT-4o's own feature vocabulary and explicitly name features such as 'formal', 'accuracy', and 'depth', the KL divergence can shrink simply because both the target responses and the evaluator share the same LLM-produced feature labels. The claimed 'bias mitigation' is therefore measured with a yardstick that is itself one of the method's inputs; no human validation or independent feature ground truth anchors the reported KL values.

1 more flagged steps
  1. self definitional [Sec. 4.2, Eq. 3]
    "Our goal is to find the adjusted output probability q̃i(x) ∈ [0,1]^5, Σ q̃i(x) = 1 for x ∈ Xt that yields the identical empirical distribution with Pi while minimizing the deviation from the original probability qϕi(x). ... min_q CE(qϕi, q) s.t. ... Σ_{x∈Xt} q(x)/|Xt| = Pi. (3)"

    Equation 3 imposes, as a hard constraint, that the aggregate adjusted feature assignment over all online instructions exactly equals the seed feature distribution Pi. Therefore the input-side 'distribution preservation' is not an empirical finding but a construction: the sampled features p̃i ∼ q̃i(x) match the seed distribution in aggregate by definition. The paper's conclusion that 'the preference feature distribution is successfully preserved when applying both feature classifier and relabeling' is thus guaranteed on the assigned-feature side. What remains empirical is only the response-side KL measured with GPT-4o, and that measurement is already entangled with the same feature extractor used to create the system prompts.

full rationale

The paper's contribution has real, independently checkable engineering content: a classifier, a Sinkhorn-Knopp distribution-matching step, system-prompt synthesis, and online DPO training, with external benchmarks (AlpacaEval 2.0, MT-Bench, Anthropic-HHH) that do not themselves depend on the debiasing metric. Those benchmark gains are not circular. However, the central claim that PFP 'successfully mitigates the bias in preference features' rests on a partly circular evidence chain. The feature vocabulary is defined by GPT-4o prompts (Sec. 4.1); the same GPT-4o model is then used to measure the KL divergence that is offered as proof of debiasing (Eq. 4). Since PFP's system prompts explicitly contain GPT-4o's feature names, the evaluator and the generator are not independent, and no human validation of the extracted feature labels is reported. In addition, Eq. 3 makes the aggregate assigned feature distribution equal to the seed distribution by construction, so 'preservation' on the input side is definitional rather than demonstrated. Finally, the SPA online-learning method and baseline is cited to the authors' own paper (identical title and author list), making that load-bearing component a self-citation rather than an independent reference. These issues are partial—the external benchmark improvements and the response-side measurements are not purely forced—but they are sufficient to mark the debiasing evaluation as substantially circular. Score: 6.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the seed feature distribution estimate, the classifier, and the GPT-4o labeling; these are the main inputs not independently verified. The feature taxonomy itself is taken from prior work, so it is not an invented entity here.

free parameters (4)
  • Seed feature distribution Pi = Empirical distribution over 5 sub-features per dimension from 10K UltraFeedback samples
    Computed from the seed dataset and used as the target distribution in Eq. 3; the entire distribution-preservation step is calibrated to this estimate.
  • System prompt temperature schedule = 1.25, decreasing by 0.25 per iteration
    Chosen by hand; ablation (Table 3) shows it improves AlpacaEval and MT-Bench scores. It directly controls response diversity and difficulty.
  • Number of online iterations = 4
    Chosen for experiments; robustness to more iterations not fully explored.
  • Sinkhorn-Knopp iteration count = 100 (default), ablated to 10,000
    Controls how aggressively the assigned feature distribution is matched to Pi; the paper tests two values and picks 100.
assumptions (4)
  • domain assumption Human preference can be decomposed into five dimensions (style, tone, harmlessness, background knowledge, informativeness), each with five exclusive sub-features.
    Adopted from Lee et al. (2024) and used to define the feature space; if the true preference space is not captured by these 25 categories, the method's bias mitigation is incomplete.
  • domain assumption GPT-4o's zero-shot CoT feature extraction from seed pairs and from generated responses accurately reflects the features that determined human preference.
    This underpins both the training labels for the classifier and the KL-divergence evaluation metric; no validation against human-annotated features is provided.
  • domain assumption The implicit DPO reward (Eq. 1) is a reliable preference judge for online labeling (SPA).
    PFP inherits SPA's labeling rule; if the implicit reward is biased, the online labels are biased.
  • standard math Sinkhorn-Knopp converges to the optimal solution of Eq. 3.
    Standard optimal transport result; used to project classifier outputs onto the target marginal distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Debiasing Online Preference Learning via Preference Feature Preservation." pith.science (2026). https://pith.science/paper/CEUDEHRP

@misc{pith2026250611098,
  author       = {Pith},
  title        = {Pith review of: Debiasing Online Preference Learning via Preference Feature Preservation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CEUDEHRP}},
  note         = {Machine review of arXiv:2506.11098}
}
read the original abstract

Recent preference learning frameworks for large language models (LLMs) simplify human preferences with binary pairwise comparisons and scalar rewards. This simplification could make LLMs' responses biased to mostly preferred features, and would be exacerbated during the iterations of online preference learning steps. To address these challenges, we propose a novel framework coined PFP (Preference Feature Preservation). The key idea of PFP is maintaining the distribution of human preference features and utilizing such rich signals throughout the online preference learning process. Specifically, PFP first extract preference features from offline pairwise human preference data and trains a feature classifier. Then, using trained classifier and the distribution preserving optimization, PFP maps appropriate preference features for a new input instruction during online learning. Lastly, PFP trains LLM using the existing preference learning method, by incorporating the preference feature into system prompts and enabling LLM to explicitly handle various human preferences. Our experiments demonstrate that PFP successfully mitigates the bias in preference features during online learning, and hence achieves superior performance compared to previous preference learning methods on standard benchmarks to evaluate LLM alignment.

Figures

Figures reproduced from arXiv: 2506.11098 by the authors.

Figure 1
Figure 1. Motivation for debiasing online preference learning. (a) For existing online preference learning methods (SPA, Iterative DPO), the preference feature distribution of trained models, obtained by inversely asking GPT-4o, progressively diverges from the original preference distribution, captured by the initial DPO model. (b) To preserve the distribution, we propose to map each input instruction with the specific prefer… view at source ↗
Figure 2
Figure 2. Illustration of the proposed PFP framework. (a) PFP first extracts the preference feature among the predefined categories for a given human preference dataset using an LLM-based feature extractor. (b) With the extracted features, PFP trains the feature classifier. (c) The trained feature classifier along with am additional distribution adjustment step assigns the preference feature for a new instruction in a online … view at source ↗
Figure 3
Figure 3. Change of Accuracy on Anthropic-HHH. Accuracy of the model’s internal preference (Eq. 1) on Anthropic-HHH. For the reference model for Eq. 1, the initial DPO model is used. As training progresses, PFP exhibits significant improvements in harmlessness and honestness, whereas other baselines show notable declines. as used SFT and DPO trained on initial prefer￾ence dataset. Notably, PFP achieves the higher performance … view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Analyses. (a,b) KL divergence with feature distribution from different methods for ablation, (c) Change of average response length on AlpacaEval 2.0 with different methods [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: Initial DPO Analysis II. LLMs trained by DPO using human feedback data with a system prompt have higher performance in honestness on Anthropic￾HHH is measured from the SFT response distribution, the DPO model with the preference feature ex￾hibits significantly reduced …
Figure 5
Figure 5. Figure 5: Initial DPO Analysis I. LLMs trained by DPO using human feedback data with system prompt has less length and feature distribution bias. B Pre-defined Preference Feature Set [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 7
Figure 7. Figure 7: Change of preference features. KL divergence comparison by class, showing how the feature distribution of the initial DPO model’s response evolves during the online learning process. PFP, unlike other iterative learning algorithms, shows minimal change in distribution …
Figure 8
Figure 8. Figure 8: Preference feature distribution captured in seed dataset Clarity Conciseness Consistency Format Vividness 0 20 40 60 80 Proportion (%) Style initial iter1 iter4 Authoritative Engaging Familiar Formal Sophisticated Tone Accuracy Morality Safety Sensitivity Trustworthine…
Figure 9
Figure 9. Figure 9: Preference feature distribution captured in responses generated from PFP Clarity Conciseness Consistency Format Vividness 0 20 40 60 80 Proportion (%) Style initial iter1 iter4 Authoritative Engaging Familiar Formal Sophisticated Tone Accuracy Morality Safety Sensitivi…
Figure 10
Figure 10. Figure 10: Preference feature distribution captured in responses generated from SPA Clarity Conciseness Consistency Format Vividness 0 20 40 60 80 Proportion (%) Style initial iter1 iter4 Authoritative Engaging Familiar Formal Sophisticated Tone Accuracy Morality Safety Sensitiv…
Figure 11
Figure 11. Figure 11: Preference feature distribution captured in responses generated from Iterative DPO [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Prompt for feature extraction. Input prompt for the feature extraction form pairwise pref￾erence data. prompt Read the following two responses to the same prompt. After reading, determine why the preferred response is chosen over the dispreferred response, focusing on…
Figure 15
Figure 15. Figure 15: Prompt for feature extraction. Input prompt for the feature extraction form single response of LLM, focusing on user’s background knowledge [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 17
Figure 17. Figure 17: Result of double system prompt sampling. Two system prompts are sampled differently from the same preference features [PITH_FULL_IMAGE:figures/full_fig_p018_17.png]
Figure 16
Figure 16. Figure 16: Prompt for system prompt generation. Input prompt for the system prompt generation from preference feature. Preference Features ’Format’, ’Formal’, ’Accuracy’, ’Basic’, ’Depth’ System prompt 1 You are a formal and precise ex￾pert, dedicated to providing accurate and h…
Figure 19
Figure 19. Figure 19: Response in AlpacaEval2.0 . For a quali￾tative comparison between Iterative DPO and PFP, we generate the responses from the same prompt [PITH_FULL_IMAGE:figures/full_fig_p019_19.png]
Figure 20
Figure 20. Figure 20: Response in AlpacaEval2.0 . For a quali￾tative comparison between Iterative DPO and PFP, we generate the responses from the same prompt [PITH_FULL_IMAGE:figures/full_fig_p020_20.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 27 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  4. [4]

    Anthropic. 2024. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet

    work page 2024

  5. [5]

    Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. 2020. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations (ICLR)

    work page 2020

  6. [6]

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861

    arXiv 2021

  7. [7]

    Yoshua Bengio, J \'e r \^o me Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the International Conference on Machine Learning (ICML)

    work page 2009

  8. [8]

    Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324--345

    1952

Show all 49 references
  1. [9]

    Daniele Calandriello, Zhaohan Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. 2024. Human alignment of large language models through online preference optimisation. In Proceedin...

    2024

  2. [10]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS)

    2017

  3. [11]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377

    2023 arXiv

  4. [12]

    Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems (NeurIPS)

    2013

  5. [13]

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233

    2023 arXiv

  6. [14]

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863

    2024 arXiv

  7. [15]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  8. [16]

    Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475

    2024 arXiv

  9. [17]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. In International Conference on Learning Representations (ICLR)

    2023

  10. [18]

    Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. In Conference on Empirical Methods in Natural Language Processing (EMNLP)

    2024

  11. [19]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023 a . Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  12. [20]

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023 b . Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. In Annual Meeting of the Association for Computational Linguistics (ACL)

    2023

  13. [21]

    Dongyoung Kim, Jaehyung Kim, Jinsung Yoon, and Jinwoo Shin. 2025. https://openreview.net/forum?id=OV0rZx8jr1 Debiasing online preference learning via preference feature preservation

    2025

  14. [22]

    Jaehyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. 2020. Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS)

    2020

  15. [23]

    Diederik P Kingma. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR)

    2015

  16. [24]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems (NeurIPS)

    2022

  17. [25]

    Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. 2024. Aligning to thousands of preferences via system message generalization. In Advances in Neural Information Processing Systems (NeurIPS)

    2024

  18. [26]

    Junlong Li, Fan Zhou, Shichao Sun, Yikai Zhang, Hai Zhao, and Pengfei Liu. 2024. Dissecting human and llm preferences. arXiv preprint arXiv:2402.11296

    2024 arXiv

  19. [27]

    Ilya Loshchilov, Frank Hutter, et al. 2017. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5

    2017 arXiv

  20. [28]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems (NeurIPS)

    2024

  21. [29]

    Juhyun Oh, Eunsu Kim, Jiseon Kim, Wenda Xu, Inha Cha, William Yang Wang, and Alice Oh. 2024. Uncovering factor level preferences to improve human-model alignment. arXiv preprint arXiv:2410.06965

    2024

  22. [30]

    OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    2023 arXiv

  23. [31]

    OpenAI. 2024 a . Hello gpt-4o. https://openai.com/index/hello-gpt-4o/

    2024

  24. [32]

    OpenAI. 2024 b . Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/

    2024

  25. [33]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems...

    2022

  26. [34]

    Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. 2024. Disentangling length from quality in direct preference optimization. In Findings of Annual Meeting of the Association for Computational Linguistics (ACL)

    2024

  27. [35]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS)

    2023

  28. [36]

    Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. 2024. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715

    2024 arXiv

  29. [37]

    Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2023. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716

    2023 arXiv

  30. [38]

    Snorkel. 2024. New benchmark results demonstrate value of snorkel ai approach to llm alignment. https://snorkel.ai/new-benchmark-results-demonstrate-value-of-snorkel-ai-approach-to-llm-alignment

    2024

  31. [39]

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems (NeurIPS)

    2020

  32. [40]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  33. [41]

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Cl \'e mentine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944

    2023 arXiv

  34. [42]

    Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. 2023. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Proc...

    2023

  35. [43]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS)

    2022

  36. [44]

    Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. 2024. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675

    2024 arXiv

  37. [45]

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. 2024. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Proceedings of the International Conference on Machine Learning (ICML)

    2024

  38. [46]

    Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. 2023. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682

    2023 arXiv

  39. [47]

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. 2023. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425

    2023 arXiv

  40. [48]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS)

    2023

  41. [49]

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

    2019 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.