Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that preference examples vary in difficulty and that examples harder than a model can handle actively hurt DPO alignment, so filtering them out by validation loss raises AlpacaEval 2 win rates by 9–16% over standard DPO.

desk verdict Solid empirical demonstration of difficulty ordering in DPO preference data, but the headline AlpacaEval gains are not cleanly identified until an equal-compute random-subset control settles the confound. read the letter →

arxiv 2502.09650 v2 pith:FYM2NAGG submitted 2025-02-11 cs.CL cs.AIcs.LG

Chengqian Gao , Haonan Li , Liu Liu , Zeke Xie , Peilin Zhao , Zhiqiang Xu This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords preferencealignmentdataselectionexampledifficultydirectoptimizationmodelcapacityvalidationlosscurriculumlearninglargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that preference data used to align language models are not uniformly useful: examples vary in difficulty, and examples harder than the model can handle actively hurt alignment rather than merely adding nothing. It proposes the principle that data difficulty must be matched to model capacity, and supports it with three claims: preference examples show a stable learning order, removing difficult examples improves performance across four models and two datasets, and larger models tolerate higher fractions of difficult data. As an instantiation, Selective DPO ranks examples by validation loss from reference DPO models trained on random halves of the data and trains only on the easiest examples, reporting 9–16% higher win rates on AlpacaEval 2 than standard DPO and beating several DPO variants. A reader should care because data selection in alignment has mostly been about removing errors, whereas this paper says even clean but too-hard examples should be filtered.

What carries the argument

The load-bearing object is the validation-loss difficulty score. For each preference example $(x, y_w, y_l)$, the paper defines the learned step as the earliest training step at which the implicit DPO reward margin $\beta\log(\pi_\theta(y_w|x)/\pi_{\mathrm{ref}}(y_w|x))-\beta\log(\pi_\theta(y_l|x)/\pi_{\mathrm{ref}}(y_l|x))$ stays above a threshold $\delta=0.4$, and then uses the negative log-sigmoid of that margin (validation loss, Eq. 3) as a cheaper proxy, computed by reference models trained on random halves of the dataset and averaged over three runs. The validation loss produces a difficulty ranking with Spearman correlation about 0.92 against the learned-step ranking. Selective DPO averages this score from six reference models, selects the easiest $\tau\%$ of examples (here $\tau=50$), and trains the target model on them in easy-to-difficult order; the filtering step, not the ordering, is what the experiments attribute the gains to.

What would settle it

A direct falsifying experiment: on a fixed dataset, rank examples by validation loss and by an independent difficulty score such as human-rated reasoning complexity. If filtering by the independent score does not reproduce the win-rate gains, or if the hardest examples by validation loss are rated easy by the independent score yet still degrade DPO, then the capacity mechanism is not the cause.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that DPO alignment is harmed when training examples exceed the current model's capacity, and that this harm can be removed by a cheap difficulty filter. The authors show that examples are learned in a consistent order across random seeds and training subsets, that the examples learned last (highest DPO validation loss) cause the largest performance drops, and that the optimal amount of difficult data grows with model size: the sweet spot moves from 64% of the training set for a 3B model to 81% for a 14B model. Label flipping, distribution-shift remedies, and learning-rate changes do not remove the drop, which the paper takes as evidence that difficulty is not simply mislabeling or optimization error. From this, Selective DPO filters out the hardest examples identified by validation loss and aligns the model on the remaining easy-to-medium examples.

Load-bearing premise

Preference examples have an intrinsic difficulty that is accurately measured by the DPO validation loss computed with reference models trained on random halves of the same dataset, rather than by artifacts of the DPO training procedure itself.

Editorial extensions

If this is right

  • If the principle holds, alignment pipelines should measure example difficulty against the target model before training, not assume all clean preference data is equally useful.
  • The sweet-spot result implies that as models scale up, the same dataset can be reused with a higher difficulty threshold, so data collection and filtering schedules should be tied to model size.
  • Because the gains come from exclusion rather than ordering, practitioners can shuffle the selected easy examples and still recover most of the benefit.
  • A curriculum built from a smaller model's difficulty scores is weaker than one built from the target model's own scores, so difficulty should be measured on the model that will be aligned.
  • Selective DPO's win-rate gains on AlpacaEval 2 also beat several algorithmic modifications to DPO such as SimPO and WPO, suggesting data selection is a complementary or alternative axis of improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that the validation-loss ranking may partly encode response-length preferences; the paper reports higher raw win rates but only comparable length-controlled win rates, so a testable extension is to run Selective DPO with a length-normalized difficulty score and see if the length-controlled gap closes.
  • The principle is stated for DPO; a natural extension is to measure difficulty with a policy-gradient or RLHF-style loss, which would test whether the capacity-matching rule transfers beyond direct preference optimization.
  • A cheaper practical corollary: if difficulty is a property of the example-model pair, one could predict the optimal tau from model size and pretraining data instead of training six reference models, and the 3B-to-14B sweet-spot curve provides a calibration target.
  • Because the hardest examples include some mislabeled ones, combining Selective DPO with label-noise correction might recover additional gains; the paper's label-flipping experiment suggests these are separate effects, so the combination is an untested hypothesis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that preference data vary in intrinsic difficulty and that examples exceeding a model's capacity actively harm DPO-based alignment. It proposes a validation-loss proxy for difficulty (Eq. 3), validates the proxy against a learned-step metric (Eq. 2), and introduces Selective DPO, which trains only on the easiest τ% of examples. Experiments across four SFT models and two datasets report 9–16% win-rate gains on AlpacaEval 2 over standard DPO and several variants, with additional results on Arena-Hard and MT-Bench. The central claims are that difficulty ordering is consistent across runs, that difficult examples degrade alignment, and that the optimal difficulty threshold grows with model size.

Significance. If the causal claim is correct, the paper identifies a simple, cheap, and widely applicable data-selection principle for preference alignment, with a concrete algorithm that appears to beat several strong DPO variants. The manuscript's strengths include a broad experiment matrix (four models, two datasets, three benchmarks), explicit controls for label noise, distribution shift, and learning-rate sensitivity (Figure 4), and release of code. The validation-loss proxy is computationally attractive, and the weak-to-strong curriculum study (Figure 9) is a useful negative result. The main significance is conditional on ruling out the equal-compute confound and on establishing that the difficulty measure is not merely a DPO-internal artifact.

major comments (3)
  1. [Algorithm 1, Step 3; Table 1; Figure 3] The headline comparison lacks an equal-compute random-subset control. Selective DPO trains for one epoch on the easiest τ% of data, so with a fixed batch size it takes roughly τ% of the gradient steps of the full-data DPO baseline. Table 1 and Figure 7 therefore compare a model trained on 50% of the dataset with a model trained on 100%, and Figure 5 varies example composition and training length along the same x-axis. The only 'random' comparison in Figure 3 is a mid-training checkpoint of full-data DPO, not a full one-epoch run on a random τ% subset. If a random 50% subset trained for the same number of steps with the same learning-rate schedule reproduces most of Selective DPO's gains on AlpacaEval 2 and Arena-Hard, the causal attribution to 'overly difficult examples exceeding model capacity' is not identified. Please add random-subset DPO baselines with matched compute (and matched learning-rate schedule) to Figure 3, Table 1, and Figure 7.
  2. [Section 3.2, Definition 3.1, Eq. (2), Eq. (3)] The difficulty construct is defined entirely through the DPO objective. Both difficulty metrics — the learned step (Eq. 2) and the validation loss (Eq. 3) — are DPO-loss-based, using reference models trained on random halves of the same dataset, and the ranking is validated only against itself (Spearman correlations around 0.92). This creates a circularity risk: 'difficult examples hinder alignment' is partly 'examples that DPO fails to learn, when trained on, hurt a DPO-trained model.' The claim would be substantially stronger if the difficulty ranking were validated against an external measure (for example, a held-out reward model, human difficulty ratings, or accuracy on a separate task), or if the filtering benefit were shown to transfer to a non-DPO alignment objective (e.g., SimPO or IPO) using the same ranking.
  3. [Remark 5.1; Figure 3; Table 1] The selection threshold τ is tuned on the same benchmark family used for the headline claims. Remark 5.1 states that τ=50 for UltraFeedback-binarized is chosen 'based on insights from Figure 3,' and Figure 3 reports WR' on AlpacaEval 2 (with ArmoRM as the evaluator). The abstract and Table 1 then report gains on AlpacaEval 2. This benchmark-selected hyperparameter inflates the reported effect size and makes the 9–16% figure less informative. Please report sensitivity of the headline results to τ (e.g., τ ∈ {40, 50, 60}) on both AlpacaEval 2 and Arena-Hard, or tune τ on a separate validation split and report the fixed-τ results.
minor comments (5)
  1. [Figure 3 caption] The caption contains a typo: '1-eopch' should be '1-epoch.'
  2. [Appendix C.3, Table 5 caption] The caption reads 'Figure 3, bottom' but the table describes hyper-parameters for Table 1 experiments; also 'ultrafeedback-bianrized' is misspelled as 'ultrafeedback-bianrized.'
  3. [Algorithm 1, Step 2] The pseudo-code says 'Compute the validation loss using the three held-out reference alignment models' while the text and Figure 6 describe six reference models; please clarify whether each example is scored by three models or six, and how the mean is taken.
  4. [Appendix D, Table 8] The notable GSM8K drop for Mistral-7B Selective DPO (14.48 vs. 30.40 for DPO) deserves more analysis than the two bullet points given; the formatting explanation is plausible but the reader should be pointed to the exact failure mode with examples.
  5. [Section 3.2 and Appendix E.1] Appendix E.1 states that 'Validation loss serves as the ground truth for difficulty due to its strong correlation with the learned step measure.' This phrasing is circular given that both measures are DPO-based; please reword to describe validation loss as a DPO-internal proxy whose external validity is an open question.

Circularity Check

2 steps flagged · score 4.0 of 10

Difficulty is operationalized entirely through DPO's implicit reward margin (validation loss is validated only against learned step, another DPO-internal metric), and the headline AlpacaEval 2 gain uses tau=50 selected on AlpacaEval-2-family WR'; the core benchmark results otherwise remain external empirical findings.

  1. self definitional [Section 3.2, Eq. (2), Eq. (3), Definition 3.1, Figure 2]
    "the high correlation coefficients between average learned step and average validation loss across the four models (0.9258, 0.9227, 0.9336, and 0.9283) validate the effectiveness of validation loss as a computationally efficient proxy for learned step."

    Eq. (2) defines learned step as the earliest step at which the DPO implicit reward margin beta*log(pi(yw)/pi_ref(yw)) - beta*log(pi(yl)/pi_ref(yl)) exceeds delta, while Eq. (3) defines validation loss as -log sigma of that same margin under a reference model. Both are monotone functions of the same DPO-internal quantity, so their high Spearman correlation is close to a self-consistency check rather than an external validation of 'difficulty.' Definition 3.1 then equates 'difficult' with high DPO validation loss, so the paper's difficulty construct is internal to DPO; the claim that difficult examples 'exceed the model's capacity' rests on this DPO-internal operationalization rather than on an independent difficulty measure.

  2. fitted input called prediction [Remark 5.1 (Section 5); Section 4.1 evaluation; Table 1]
    "In practice, tau can be tuned using a third-party evaluator such as AlpacaEval 2 (Dubois et al., 2024). For the evaluation in the next section, we set tau = 50 for the UltraFeedback-binarized dataset, based on insights from Figure 3."

    Figure 3's y-axis is WR', defined in Section 4.1 as 'the win rate against gpt-4-turbo on 805 testing examples from AlpacaEval 2' with ArmoRM. The abstract and Table 1 headline ('9-16% higher win rate on AlpacaEval 2') use this tau=50 choice. Thus the flagship AlpacaEval 2 gain is reported on the same benchmark family used to select the key hyperparameter, making it a benchmark-tuned result rather than a parameter-free prediction. Arena-Hard, MT-Bench, and the model-size sweet-spot pattern are independent and soften the circularity.

full rationale

The central empirical comparisons are not definitionally forced: AlpacaEval 2, Arena-Hard, and MT-Bench scores, as well as the Figure 5 sweet-spot shift, are external measurements, and it is logically possible that removing high-validation-loss examples would not improve them. No load-bearing self-citation chain is present, and the paper is self-contained against external benchmarks. The two circular elements are: (i) the difficulty construct is defined and validated entirely through DPO's implicit reward margin, so 'difficult examples hinder alignment' is partly 'examples DPO finds hard, when removed, help a DPO-trained model'; and (ii) the headline AlpacaEval 2 result uses tau=50 selected on an AlpacaEval-2-family metric. A separate validity concern, the absence of an equal-compute random-50%-subset DPO control, weakens the causal attribution to example difficulty, but that is a confound rather than a definitional reduction, so it does not by itself raise the circularity score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The principle's operational core rests on two DPO-internal difficulty metrics (Eqs. 2 and 3) and on benchmark judgments (ArmoRM, AlpacaEval 2). The main free parameters are tau (selection threshold), delta (diagnostic threshold), and the fitted sweet spots that carry the capacity claim. No invented entities. A reader who rejects the DPO-internal difficulty construct loses both the mechanism and the method's ranking criterion.

free parameters (4)
  • Selection threshold tau = 50% (UF-binarized); 60% tested in Appendix D
    Tuned on AlpacaEval-2 WR' curves (Figure 3) and then used for the headline AlpacaEval 2 results (Table 1); a free parameter of the method.
  • Learned-step threshold delta = 0.4
    Set by hand for all experiments (Section 2.2, Eq. 2) with no sensitivity analysis; it defines the difficulty measure used to validate the validation-loss proxy.
  • Number of reference models = 6
    Chosen as a cost/accuracy balance (Section 6.2, Figure 8a); performance varies with 2, 6, or 10 reference models.
  • Sweet-spot data percentages = 64% (3B) and 81% (14B); 7B intermediate
    Peaks of second-degree polynomial fits to win-rate curves in Figure 5; direct evidence for the capacity-scaling claim, with no confidence intervals reported.
assumptions (5)
  • domain assumption The DPO objective and implicit reward model (Eq. 1) validly model preference alignment.
    The two difficulty metrics (Eqs. 2-3) are defined through this objective, so the principle is operationalized entirely inside DPO; the paper restricts the principle to the DPO setting (Section 8).
  • domain assumption AlpacaEval 2 win rates (WR, LC) and ArmoRM scores measure alignment quality.
    ArmoRM WR' is used to tune tau (Figure 3) and AlpacaEval 2 is the headline benchmark (Table 1); the paper concedes length exploitation makes WR an optimistic metric (Section 8).
  • domain assumption Validation loss from reference DPO models trained on random halves is a stable, transferable difficulty proxy.
    Section 3.2, Eq. 3; validated only against the learned step (Eq. 2), which is itself DPO-internal, and never against an external difficulty measure.
  • domain assumption Difficulty ordering transfers from LoRA-based analytics runs to full fine-tuning and across model sizes.
    Figures 3 and 5 use LoRA (Appendix C.4) while Table 1 uses FPFT; Appendix E.2 shows cross-size overlap heatmaps but no formal transfer guarantee.
  • standard math Standard logistic and DPO mathematics from Rafailov et al. (2024).
    Eqs. 1-3 build on DPO's derivation without re-proving it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples." pith.science (2026). https://pith.science/paper/FYM2NAGG

@misc{pith2026250209650,
  author       = {Pith},
  title        = {Pith review of: Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FYM2NAGG}},
  note         = {Machine review of arXiv:2502.09650}
}
read the original abstract

The alignment of large language models (LLMs) often assumes that using more clean data yields better outcomes, overlooking the match between model capacity and example difficulty. Challenging this, we propose a new principle: Preference data vary in difficulty, and overly difficult examples hinder alignment, by exceeding the model's capacity. Through systematic experimentation, we validate this principle with three key findings: (1) preference examples vary in difficulty, as evidenced by consistent learning orders across alignment runs; (2) overly difficult examples significantly degrade performance across four LLMs and two datasets; and (3) the capacity of a model dictates its threshold for handling difficult examples, underscoring a critical relationship between data selection and model capacity. Building on this principle, we introduce Selective DPO, which filters out overly difficult examples. This simple adjustment improves alignment performance by 9-16% in win rates on the AlpacaEval 2 benchmark compared to the DPO baseline, suppressing a series of DPO variants with different algorithmic adjustments. Together, these results illuminate the importance of aligning data difficulty with model capacity, offering a transformative perspective for improving alignment strategies in LLMs. Code is available at https://github.com/glorgao/SelectiveDPO.

Figures

Figures reproduced from arXiv: 2502.09650 by the authors.

Figure 1
Figure 1. Overly difficult examples hinder the alignment. Training on difficult examples, identified by high valida￾tion loss, adversely affects alignment and decreases overall performance by 9.4% in win rate. The results are from experiments with four SFT models on the UltraFeedback￾binarized dataset, i.e., [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Examples are learned in consistent orders across different runs of the same LLM, despite variations in the training data and random seeds. Left: The learned step (ranging from 1 to 948) represents the step at which the implicit reward model distinguishes between preferred and rejected responses (see Eq. (2), threshold δ = 0.4). X-axis: 40 unique combinations of model size (4 total) and training data subset (10 per m… view at source ↗
Figure 3
Figure 3. Direct Preference Optimization (DPO) struggles with difficult examples, broadly and significantly. We present the defined WR′ evolution for four models trained on the argilla-mix-dpo-7k and ultrafeedback-binarized datasets. The results are based on checkpoints from three 1-eopch runs with different seeds. Random Ordering (DPO): Training data are presented in a randomized sequence. Sorted by VL (From Easy to Difficul… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Difficulty examples are not necessarily data errors. (a): flipping the last 40% examples with higher validation loss. (b): sorting the examples with the ϵ-greedy sorting algorithm. In this case, each mini-batch data contains (1-ϵ) part of easy-to-difficult examples and…
Figure 5
Figure 5. Figure 5: Difficult examples benefit larger models with greater capacities. Examples are sorted by their validation loss, ranging from easy to difficult. We fit the measured WR′ (scatter points) using a second-degree polynomial (dashed line), identifying the peak of each parabol…
Figure 6
Figure 6. Figure 6: The pipeline of Selective DPO. It extends DPO (Rafailov et al., 2024) with a principled data selection process: selecting preference examples within the model’s capacity. Specifically, Selective DPO comprises three steps: (1) Train a set of reference models using the D…
Figure 7
Figure 7. Figure 7: Comparison results against SimPO and WPO, with all methods tuned for their learning rates. Selective DPO [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Hyper-parameter study and in-depth analysis of Selective DPO. (a): Relationship between the number of reference models and performance. (b): Performance with different percentages of selected easy examples. (c): Negative log-likelihoods distributions on the generated r…
Figure 9
Figure 9. Figure 9: Weak-to-strong curriculum under-performs. Aligning a 7B model with examples ordered by 3B reference models yields compromised results. indicating higher confidence in generated responses. Ad￾ditionally, the implicit reward model learned by Selective DPO exhibits better…
Figure 10
Figure 10. Figure 10: Comparison of response length and reward margin measures with validation loss across three difficulty levels: Easy, Medium, and Difficult. The dataset examples are partitioned into these levels based on increasing validation loss. While validation loss increases consi…
Figure 11
Figure 11. Figure 11: Easy examples identified by smaller models are likely also recognized as easy by larger models, and difficult [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Comparison of our difficulty metric validation loss against alternative sorting criteria: perplexity gap, completion length, and reward margin. Perplexity Gap is defined as the difference in perplexity between the chosen and rejected responses given the same prompt. P…
Figure 13
Figure 13. Figure 13: Evolution of preference probabilities during 2-epoch training. We track the trajectory of [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Weak-to-strong curriculum under-performs. Aligning a 7B model with examples ordered by 3B reference models [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationale

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A preference optimization strategy using confidence-based hard example mining, similarity retrieval, and synthetic counterfactual rationales improves chest X-ray VQA accuracy by 8.93% relative over supervised fine-tuning.

Reference graph

Works this paper leans on

94 extracted references · 39 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Llama 3 model card

    AI@Meta. Llama 3 model card. web page, 2024

    2024

  3. [3]

    M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L., Jeong, H., Raffel, C., Chang, S., Hashimoto, T., and Wang, W

    Albalak, A., Elazar, Y., Xie, S. M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L., Jeong, H., Raffel, C., Chang, S., Hashimoto, T., and Wang, W. Y. A survey on data selection for language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=XfHWcNTSHp. Survey Certification

    2024

  4. [4]

    L., and Paul, M

    Ankner, Z., Blakeney, C., Sreenivasan, K., Marion, M., Leavitt, M. L., and Paul, M. Perplexed by perplexity: Perplexity-based data pruning with small reference models. arXiv preprint arXiv:2405.20541, 2024

    arXiv 2024

  5. [5]

    Ultrafeedback binarized preferences-cleaned

    Argilla. Ultrafeedback binarized preferences-cleaned. https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned, 2024

    2024

  6. [6]

    A general language assistant as a laboratory for alignment

    Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021

    arXiv 2021

  7. [7]

    Teaching by examples: Implications for the process of category acquisition

    Avrahami, J., Kareev, Y., Bogot, Y., Caspi, R., Dunaevsky, S., and Lerner, S. Teaching by examples: Implications for the process of category acquisition. The Quarterly Journal of Experimental Psychology Section A, 50 0 (3): 0 586--606, 1997

    1997

  8. [8]

    G., Guo, Z

    Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp.\ 4447--4455. PMLR, 2024

    2024

Show all 94 references
  1. [9]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    2022 arXiv

  2. [10]

    Bartolome, A., Martin, G., and Vila, D. Notus. https://github.com/argilla-io/notus, 2023

    2023

  3. [11]

    Open LLM leaderboard

    Beeching, E., Fourrier, C., Habib, N., Han, S., Lambert, N., Rajani, N., Sanseviero, O., Tunstall, L., and Wolf, T. Open LLM leaderboard. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023

    2023

  4. [12]

    Curriculum learning

    Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp.\ 41--48, 2009

    2009

  5. [13]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  6. [14]

    Curriculum learning for language modeling

    Campos, D. Curriculum learning for language modeling. arXiv preprint arXiv:2108.02170, 2021

    2021 arXiv

  7. [15]

    Instruction mining: When data mining meets large language model finetuning

    Cao, Y., Kang, Y., Wang, C., and Sun, L. Instruction mining: When data mining meets large language model finetuning. arXiv preprint arXiv:2307.06290, 2023

    2023 arXiv

  8. [16]

    Does the order of training samples matter? improving neural data-to-text generation with curriculum learning

    Chang, E., Yeh, H.-S., and Demberg, V. Does the order of training samples matter? improving neural data-to-text generation with curriculum learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp....

    2021

  9. [17]

    Alpagasus: Training a better alpaca with fewer data

    Chen, L., Li, S., Yan, J., Wang, H., Gunaratna, K., Yadav, V., Tang, Z., Srinivasan, V., Zhou, T., Huang, H., and Jin, H. Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations, 2024 a . URL https://openreview.n...

    2024

  10. [18]

    Skill-it! a data-driven skills framework for understanding and training language models

    Chen, M., Roberts, N., Bhatia, K., Wang, J., Zhang, C., Sala, F., and R \'e , C. Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36, 2024 b

    2024

  11. [19]

    Sdformer: Similarity-driven discrete transformer for time series generation

    Chen, Z., Feng, S., Zhang, Z., Xiao, X., Gao, X., and Zhao, P. Sdformer: Similarity-driven discrete transformer for time series generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 c

    2024

  12. [20]

    R., Kini, A., and Natarajan, N

    Chowdhury, S. R., Kini, A., and Natarajan, N. Provably robust dpo: Aligning language models with noisy feedback. In Forty-first International Conference on Machine Learning, 2024

    2024

  13. [21]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  14. [22]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  15. [23]

    T., and Shah, M

    Croitoru, F.-A., Hondru, V., Ionescu, R. T., and Shah, M. Reverse stable diffusion: What prompt was used to generate this image? arXiv preprint arXiv:2308.01472, 2023

    2023 arXiv

  16. [24]

    T., Sebe, N., and Shah, M

    Croitoru, F.-A., Hondru, V., Ionescu, R. T., Sebe, N., and Shah, M. Curriculum direct preference optimization for diffusion and consistency models. arXiv preprint arXiv:2405.13637, 2024

    2024 arXiv

  17. [25]

    Mods: Model-oriented data selection for instruction tuning

    Du, Q., Zong, C., and Zhang, J. Mods: Model-oriented data selection for instruction tuning. arXiv preprint arXiv:2311.15653, 2023

    2023 arXiv

  18. [26]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  19. [27]

    Length-controlled alpacaeval: A simple debiasing of automatic evaluators

    Dubois, Y., Liang, P., and Hashimoto, T. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=CybBmzWBX0

    2024

  20. [28]

    Dsdm: Model-aware dataset selection with datamodels

    Engstrom, L., Feldmann, A., and Madry, A. Dsdm: Model-aware dataset selection with datamodels. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=GC8HkKeH8s

    2024

  21. [29]

    Understanding dataset difficulty with \ V \ -usable information

    Ethayarajh, K., Choi, Y., and Swayamdipta, S. Understanding dataset difficulty with \ V \ -usable information. In International Conference on Machine Learning, pp.\ 5988--6008. PMLR, 2022

    2022

  22. [30]

    Model alignment as prospect theoretic optimization

    Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Model alignment as prospect theoretic optimization. In Forty-first International Conference on Machine Learning, 2024

    2024

  23. [31]

    Impact of preference noise on the alignment performance of generative language models

    Gao, Y., Alon, D., and Metzler, D. Impact of preference noise on the alignment performance of generative language models. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=nMAaCsCTCI

    2024

  24. [32]

    Understanding finetuning for factual knowledge extraction

    Ghosal, G., Hashimoto, T., and Raghunathan, A. Understanding finetuning for factual knowledge extraction. In Proceedings of the 41st International Conference on Machine Learning, pp.\ 15540--15558, 2024

    2024

  25. [33]

    ORPO : Monolithic preference optimization without reference model

    Hong, J., Lee, N., and Thorne, J. ORPO : Monolithic preference optimization without reference model. ArXiv, abs/2403.07691, 2024

    2024 arXiv

  26. [34]

    J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al

    Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022

    2022

  27. [35]

    R., and Kwon, D

    Ji12, S., Sinulingga, H. R., and Kwon, D. Robust guidance for unsupervised data selection: Capturing perplexing named entities for domain-specific machine translation. LREC-COLING 2024, pp.\ 307, 2024

    2024

  28. [36]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  29. [37]

    Jiang, L., Meng, D., Mitamura, T., and Hauptmann, A. G. Easy samples first: Self-paced reranking for zero-example multimedia search. In Proceedings of the 22nd ACM international conference on Multimedia, pp.\ 547--556, 2014

    2014

  30. [38]

    Fasttext

    Joulin, A. Fasttext. zip: Compressing text classification models. arXiv preprint arXiv:1612.03651, 2016

    2016 arXiv

  31. [39]

    and Lee, J

    Kim, J. and Lee, J. Strategic data ordering: Enhancing large language model performance through curriculum learning. arXiv preprint arXiv:2405.07490, 2024

    2024 arXiv

  32. [40]

    Lambert, N., Pyatkin, V., Morrison, J., Miranda, L. J. V., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. R eward B ench: Evaluating reward models for language modeling. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Findi...

    2025

  33. [41]

    The bigscience roots corpus: A 1.6 tb composite multilingual dataset

    Lauren c on, H., Saulnier, L., Wang, T., Akiki, C., Villanova del Moral, A., Le Scao, T., Von Werra, L., Mou, C., Gonz \'a lez Ponferrada, E., Nguyen, H., et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Sys...

    2022

  34. [42]

    Deduplicating training data makes language models better

    Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021

    2021 arXiv

  35. [43]

    From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning

    Li, M., Zhang, Y., Li, Z., Chen, J., Chen, L., Cheng, N., Wang, J., Zhou, T., and Xiao, J. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Asso...

    2024

  36. [44]

    E., and Stoica, I

    Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, I. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939, 2024 b

    2024 arXiv

  37. [45]

    Not all tokens are what you need for pretraining

    Lin, Z., Gou, Z., Gong, Y., Liu, X., yelong shen, Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., and Chen, W. Not all tokens are what you need for pretraining. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/for...

    2024

  38. [46]

    High dimensional robust m-estimation: Arbitrary corruption and heavy tails

    Liu, L. High dimensional robust m-estimation: Arbitrary corruption and heavy tails. The University of Texas at Austin, 2021

    2021

  39. [47]

    What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning

    Liu, W., Zeng, W., He, K., Jiang, Y., and He, J. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=BTKAeLqLMw

    2024

  40. [48]

    Simpo: Simple preference optimization with a reference-free reward

    Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37: 0 124198--124235, 2024

    2024

  41. [49]

    and Sachdeva, B

    Mishra, S. and Sachdeva, B. S. Do we need to create big datasets to learn a task? In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pp.\ 169--173, 2020

    2020

  42. [50]

    A note on dpo with noisy preferences and relationship to ipo, 2023

    Mitchell, E. A note on dpo with noisy preferences and relationship to ipo, 2023. URL https://ericmitchell.ai/cdpo.pdf

    2023

  43. [51]

    Orca: Progressive learning from complex explanation traces of gpt-4

    Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., and Awadallah, A. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707, 2023

    2023 arXiv

  44. [52]

    Length-based curriculum learning for efficient pre-training of language models

    Nagatsuka, K., Broni-Bediako, C., and Atsumi, M. Length-based curriculum learning for efficient pre-training of language models. New Generation Computing, 41 0 (1): 0 109--134, 2023

    2023

  45. [53]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  46. [54]

    Disentangling length from quality in direct preference optimization

    Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentangling length from quality in direct preference optimization. In Findings of the Association for Computational Linguistics ACL 2024, pp.\ 4998--5017, 2024

    2024

  47. [55]

    Pattnaik, P., Maheshwary, R., Ogueji, K., Yadav, V., and Madhusudhan, S. T. Curry-dpo: Enhancing alignment using curriculum learning & ranked preferences. arXiv preprint arXiv:2403.07230, 2024

    2024 arXiv

  48. [56]

    The refinedweb dataset for falcon llm: Outperforming curated corpora with web data only

    Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Alobeidli, H., Cappelli, A., Pannier, B., Almazrouei, E., and Launay, J. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data only. Advances in Neural Information Processing Systems, 36: 0 79155...

    2023

  49. [57]

    A., Stretcu, O., Neubig, G., Poczos, B., and Mitchell, T

    Platanios, E. A., Stretcu, O., Neubig, G., Poczos, B., and Mitchell, T. M. Competence-based curriculum learning for neural machine translation. arXiv preprint arXiv:1903.09848, 2019

    1903 arXiv

  50. [58]

    Unleashing the power of data tsunami: A comprehensive survey on data assessment and selection for instruction tuning of language models

    Qin, Y., Yang, Y., Guo, P., Li, G., Shao, H., Shi, Y., Xu, Z., Gu, Y., Li, K., and Sun, X. Unleashing the power of data tsunami: A comprehensive survey on data assessment and selection for instruction tuning of language models. Transactions on Machine Learning Research, 2025

    2025

  51. [59]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  52. [60]

    Rampp, S., Milling, M., Triantafyllopoulos, A., and Schuller, B. W. Does the definition of difficulty matter? scoring functions and their role for curriculum learning. arXiv preprint arXiv:2411.00973, 2024

    2024 arXiv

  53. [61]

    Unintentional unalignment: Likelihood displacement in direct preference optimization

    Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., and Hanin, B. Unintentional unalignment: Likelihood displacement in direct preference optimization. In International Conference on Learning Representations, 2025

    2025

  54. [62]

    and Xing, E

    Sachan, M. and Xing, E. Easy questions first? a case study on curriculum learning for question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 453--463, 2016

    2016

  55. [63]

    and Sanghavi, S

    Shen, Y. and Sanghavi, S. Learning with bad training data via iterative trimmed loss minimization. In International conference on machine learning, pp.\ 5739--5748. PMLR, 2019

    2019

  56. [64]

    less is more

    Spitkovsky, V. I., Alshawi, H., and Jurafsky, D. From baby steps to leapfrog: How “less is more” in unsupervised dependency parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp.\ ...

    2010

  57. [65]

    A., and Choi, Y

    Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H., Smith, N. A., and Choi, Y. Dataset cartography: Mapping and diagnosing datasets with training dynamics. arXiv preprint arXiv:2009.10795, 2020

    2009 arXiv

  58. [66]

    Txt360: A top-quality llm pre-training dataset requires the perfect blend, 2024

    Tang, L., Ranjan, N., Pangarkar, O., Liang, X., Wang, Z., An, L., Rao, B., Jin, L., Wang, H., Cheng, Z., Sun, S., Mu, C., Miller, V., Ma, X., Peng, Y., Liu, Z., and Xing, E. Txt360: A top-quality llm pre-training dataset requires the perfect blend, 2024

    2024

  59. [67]

    T., Fu, J., Phan, M

    Tay, Y., Wang, S., Luu, A. T., Fu, J., Phan, M. C., Yuan, X., Rao, J., Hui, S. C., and Zhang, A. Simple and effective curriculum pointer-generator networks for reading comprehension over long narratives. In Proceedings of the 57th Annual Meeting of the Association for Computat...

    2019

  60. [68]

    G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al

    Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  61. [69]

    P., and Ferrari, V

    Tudor Ionescu, R., Alexe, B., Leordeanu, M., Popescu, M., Papadopoulos, D. P., and Ferrari, V. How hard can it be? estimating the difficulty of visual search in an image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 2157--2166, 2016

    2016

  62. [70]

    Zephyr: Direct distillation of lm alignment

    Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023

    2023 arXiv

  63. [71]

    Secrets of rlhf in large language models part ii: Reward modeling

    Wang, B., Zheng, R., Chen, L., Liu, Y., Dou, S., Huang, C., Shen, W., Jin, S., Zhou, E., Shi, C., et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06080, 2024 a

    2024 arXiv

  64. [72]

    Step-on-feet tuning: Scaling self-alignment of LLM s via bootstrapping

    Wang, H., Ma, G., Meng, Z., Qin, Z., Shen, L., Zhang, Z., Wu, B., Liu, L., Bian, Y., Xu, T., Wang, X., and Zhao, P. Step-on-feet tuning: Scaling self-alignment of LLM s via bootstrapping. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment, 2024 b . URL https://...

    2024

  65. [73]

    Interpretable preferences via multi-objective reward modeling and mixture-of-experts

    Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 10582--10592, 2024 c

    2024

  66. [74]

    A., Beltagy, I., et al

    Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36: 0 74764--...

    2023

  67. [75]

    Ethical and social risks of harm from language models

    Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021

    2021 arXiv

  68. [76]

    Ccnet: Extracting high quality monolingual datasets from web crawl data

    Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzm \'a n, F., Joulin, A., and Grave, E. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359, 2019

    1911 arXiv

  69. [77]

    QuRating : Selecting high-quality data for training language models

    Wettig, A., Gupta, A., Malik, S., and Chen, D. QuRating : Selecting high-quality data for training language models. In International Conference on Machine Learning (ICML), 2024

    2024

  70. [78]

    Curriculum learning with quality-driven data selection

    Wu, B., Meng, F., and Chen, L. Curriculum learning with quality-driven data selection. arXiv preprint arXiv:2407.00102, 2024

    2024 arXiv

  71. [79]

    Self-evolved diverse data sampling for efficient instruction tuning

    Wu, S., Lu, K., Xu, B., Lin, J., Su, Q., and Zhou, C. Self-evolved diverse data sampling for efficient instruction tuning. arXiv preprint arXiv:2311.08182, 2023

    2023 arXiv

  72. [80]

    When do curricula work? In ICLR, 2021

    Wu, X., Dyer, E., and Neyshabur, B. When do curricula work? In ICLR, 2021

    2021

  73. [81]

    Rethinking data selection at scale: Random selection is almost all you need

    Xia, T., Yu, B., Dang, K., Yang, A., Wu, Y., Tian, Y., Chang, Y., and Lin, J. Rethinking data selection at scale: Random selection is almost all you need. arXiv preprint arXiv:2410.09335, 2024

    2024 arXiv

  74. [82]

    Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme, B., Murray, K., and Kim, Y. J. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. In International Conference on Machine Learning, pp.\ 55204--55224. PMLR, 2024

    2024

  75. [83]

    Rethinking the instruction quality: Lift is what you need

    Xu, Y., Yao, Y., Huang, Y., Qi, M., Wang, M., Gu, B., and Sundaresan, N. Rethinking the instruction quality: Lift is what you need. CoRR, 2023

    2023

  76. [84]

    Qwen2.5 technical report

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  77. [85]

    Mates: Model-aware data selection for efficient pretraining with data influence models

    Yu, Z., Das, S., and Xiong, C. Mates: Model-aware data selection for efficient pretraining with data influence models. Advances in Neural Information Processing Systems, 37: 0 108735--108759, 2024

    2024

  78. [86]

    Rrhf: Rank responses to align language models with human feedback

    Yuan, H., Yuan, Z., Tan, C., Wang, W., Huang, S., and Huang, F. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36: 0 10935--10950, 2023

    2023

  79. [87]

    and Sutskever, I

    Zaremba, W. and Sutskever, I. Learning to execute. arXiv preprint arXiv:1410.4615, 2014

    2014 arXiv

  80. [88]

    X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023 a

    2023 arXiv

  81. [89]

    Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. SLiC-HF : Sequence likelihood calibration with human feedback. ArXiv, abs/2305.10425, 2023 b

    2023 arXiv

  82. [90]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2023

    2023

  83. [91]

    Lima: Less is more for alignment

    Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2023

    2023

  84. [92]

    R., Zhao, S., Song, K., Xu, S., and Zhu, C

    Zhou, W., Agrawal, R., Zhang, S., Indurthi, S. R., Zhao, S., Song, K., Xu, S., and Zhu, C. Wpo: Enhancing rlhf with weighted preference optimization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 8328--8340, 2024

    2024

  85. [93]

    Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf

    Zhu, B., Jordan, M., and Jiao, J. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. In International Conference on Machine Learning, pp.\ 62405--62428. PMLR, 2024 a

    2024

  86. [94]

    Exploring the learning difficulty of data: Theory and measure

    Zhu, W., Wu, O., Su, F., and Deng, Y. Exploring the learning difficulty of data: Theory and measure. ACM Transactions on Knowledge Discovery from Data, 18 0 (4): 0 1--37, 2024 b

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.