Pith. sign in

REVIEW 4 major objections 6 minor 52 references

R.I.P.: Better Models by Survival of the Fittest Prompts

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RIP filters training prompts by the quality of their rejected response, and models trained on the kept prompts beat unfiltered baselines on three benchmarks.

desk verdict RIP is a surprisingly effective prompt-filtering recipe with real benchmark gains; the central result is likely real, but the ArmoRM coupling needs one clean replication before the numbers are taken at face value. read the letter →

arxiv 2501.18578 v2 pith:JNOQN3KA submitted 2025-01-30 cs.CL cs.AIcs.LG

Ping Yu , Weizhe Yuan , Olga Golovneva , Tianhao Wu , Sainbayar Sukhbaatar , Jason Weston , Jing Xu This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords preferenceoptimizationdatafilteringpromptqualityrejectedresponserewardgapdirectsyntheticinstructiongenerationRLHF
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that in preference optimization, a training prompt's quality is legible in the responses it provokes: low-quality prompts produce low-quality, short rejected responses and a large gap between the best and worst response. The authors propose Rejecting Instruction Preferences (RIP), which keeps only prompts whose rejected response reward $r(y_l|x)$ and length $\mathrm{len}(y_l)$ clear a percentile threshold and whose reward gap $r(y_w|x)-r(y_l|x)$ stays below a percentile threshold. With these thresholds, DPO training on 4,538 curated WildChat prompts beats training on all 20,000, and the same recipe applied to synthetic prompt generation (Self-RIP) outperforms both unfiltered human prompts and standard Self-Instruct data. This matters because data curation is a central lever on model quality, and RIP extracts a signal from response pairs that rivals or beats expensive LLM-based prompt judging.

What carries the argument

The central object is the preference pair $(x, y_w, y_l)$ with its three response-derived statistics: the rejected response reward $r(y_l|x)$, the rejected response length $\mathrm{len}(y_l)$, and the reward gap $r(y_w|x)-r(y_l|x)$. RIP thresholds each statistic, with lower bounds on the first two and an upper bound on the third, to decide whether the prompt $x$ stays in the training set. The mechanism behind the thresholds is the hypothesis that bad prompts elicit bad, short rejected responses and a wide spread among sampled responses, so the pair's tail statistics reveal prompt quality without needing an external judge to read the prompt itself.

What would settle it

Run the identically-thresholded RIP pipeline on WildChat-20k, but replace the scores used to build chosen and rejected pairs with random numbers; if DPO on the random-scored filtered subset still reproduces the 9.4-point AlpacaEval2 LC gain, the filter's power lies in the pair statistics themselves rather than in response-quality ranking.

Watch

Extended reading notes

Core claim

RIP treats the preference pair, not just the prompt, as the unit of data-quality judgment. Given $N$ sampled responses to a prompt, the paper uses best-vs-worst pairing to define a chosen response $y_w$ and a rejected response $y_l$, then filters on three statistics: the rejected response's reward, the rejected response's length, and the reward gap $r(y_w|x)-r(y_l|x)$. The claim is that low values of the first two and high values of the third mark prompts that are ambiguous, unclear, unsafe, or otherwise unhelpful for preference optimization. Applying the filter to WildChat-20k keeps 4,538 of 20,000 prompts and raises Llama 3.1-8B-Instruct's AlpacaEval2 LC win rate from 48.4 to 57.8, Arena-Hard from 37.9 to 43.1, and WildBench from 41.5 to 45.6; the same filter on Llama 3.3-70B-Instruct lifts Arena-Hard from 70.5 to 82.9. The paper also shows the same signal can filter synthetically generated prompts, and that a weak 8B model can filter useful data for a strong 70B model.

Load-bearing premise

The load-bearing premise is that the reward model's scores faithfully rank response quality for the seed model, because if the judge mis-ranks responses, all three metrics $r(y_l|x)$, $\mathrm{len}(y_l)$, and $r(y_w|x)-r(y_l|x)$ inherit that error and the thresholds select the wrong prompts.

Editorial extensions

If this is right

  • On WildChat with Llama 3.1-8B-Instruct, RIP filtering raises AlpacaEval2 LC win rate from 48.4 to 57.8, Arena-Hard from 37.9 to 43.1, and WildBench from 41.5 to 45.6 while training on only 4,538 of the original 20,000 prompts.
  • RIP works with different reward sources: ArmoRM, LLM-as-a-judge, and human reward annotations on HelpSteer2 all show gains over unfiltered preference data.
  • A weaker model can filter data for a stronger model: Llama 3.1-8B-Instruct-filtered WildChat improves Llama 3.3-70B-Instruct's Arena-Hard from 70.5 to 76.7, while self-filtering pushes it to 82.9.
  • Self-RIP, which uses RIP-curated prompts as few-shot seeds and applies RIP again after generation, beats both Self-Instruct and unfiltered human-written prompts, reaching a 60.2 AlpacaEval2 LC win rate.
  • Prompt-only filtering methods (InsTag, IFD, perplexity, LLM-as-prompt-judge) mostly fail to beat no filtering on preference data, indicating that response-pair statistics carry the signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper tunes its thresholds on WildChat with ArmoRM, so the transferable claim is the metric family, not the specific 50th-percentile cutoffs; applying RIP to a new dataset or reward model would require re-tuning the quantiles.
  • The reward-gap criterion likely functions as a specificity detector as well as a quality detector: prompts that admit many good but divergent answers are removed even when the rejected response is long and high-scored, which could make RIP useful for ambiguity measurement outside DPO.
  • A natural test the paper does not run is the same filtering applied to supervised fine-tuning or other RL objectives; if the hypothesis is right, unclear prompts should hurt those settings too, and a rejected-response statistic derived from on-policy samples should predict SFT data quality as well.
  • The presence of unsafe prompts among the filtered examples hints at a safety-by-data-curation effect, so measuring downstream refusal and safety behavior on RIP-trained models, not just instruction-following benchmarks, would test whether the safety improvement observed qualitatively is systematic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes RIP (Rejecting Instruction Preferences), a data-curation method for preference optimization. The method filters training prompts based on three metrics derived from preference pairs: the reward of the rejected response, the length of the rejected response, and the reward gap between chosen and rejected responses. On WildChat, using ArmoRM to construct and score pairs, RIP filtering improves Llama 3.1-8B-Instruct on AlpacaEval2 LC from 48.4 to 57.8, Arena-Hard from 37.9 to 43.1, and WildBench from 41.5 to 45.6 while using only 4,538 of 20,000 prompts. The paper also introduces Self-RIP for generating synthetic prompts, reports gains on Llama 3.3-70B-Instruct, and shows weak-to-strong transfer. Experiments with human-annotated HelpSteer2 rewards and an LLM-as-a-Judge annotator are included as evidence of generality.

Significance. If the central claim holds, RIP is a simple, practical method for improving preference-optimization data, with especially notable results on AlpacaEval2 where the 8B model surpasses GPT-4 Omni. The paper's strengths are its breadth of evaluation (three benchmarks, two model sizes, multiple reward annotators), the release of curated datasets, and ablations on the number of sampled responses and on pairing methods. The main weakness is that the same reward model (ArmoRM) is used for pair construction, metric computation, threshold selection, and early stopping, which risks a form of proxy overfitting; the HelpSteer2 experiment does not fully break this coupling because early stopping still uses ArmoRM and only a single threshold is applied. The significance is contingent on an independent-reward-model replication.

major comments (4)
  1. [§4.1.1, §3.3.1, Appendix Table 10] The same reward model ArmoRM is used in four connected roles: constructing best-vs-worst pairs, computing the rejected-reward and reward-gap metrics, selecting the filtering thresholds on the validation set, and choosing the early-stopping checkpoint. This coupling means the thresholds are effectively optimized to improve a proxy that is the same signal that defines the filter, so a systematic bias in ArmoRM could inflate the reported gains without actually selecting higher-quality prompts. The external benchmarks (AlpacaEval2, Arena-Hard, WildBench) are reassuring but do not fully break the coupling because their judges are trained on the same general human-preference objective, and the HelpSteer2 experiment still uses ArmoRM for early stopping and applies only a single threshold. The authors should replicate the core WildChat experiment with an independent reward model for pairing, threshold selection, and checkpoint selection, or provide a fully human-annotated checkpoint selection.
  2. [§4.1.2 and Table 10] The HelpSteer2 experiment, which is the main evidence that RIP works with human-annotated rewards, does not actually implement the full three-metric RIP: Table 10 shows the best model is obtained with only the rejected-length threshold (Rejected Length ≥ 1303), not the conjunction of rejected reward, rejected length, and reward gap. As a result, the claim that the full RIP signal transfers across reward annotators is not directly tested. The authors should either run the full RIP on HelpSteer2 (with thresholds selected on a held-out split using human rewards) or explicitly justify and analyze the single-metric version.
  3. [§4.1.1 and Appendix A.2] The three filtering thresholds are selected via coordinate-wise sweeps and a three-parameter hyperparameter search on a validation set of only 470 examples (253 Humpback plus 218 Evol-Instruct). This is a relatively small basis for choosing three parameters, and because the same validation set is used for early stopping, the reported numbers may reflect overfitting to this particular split. The authors should report threshold-sensitivity curves, results across multiple validation splits or bootstraps, and ideally a pre-registered or default threshold choice.
  4. [Table 20 and Table 2] Table 20 shows that on the WildChat validation set, filtering by rejected reward alone at the 75-100 percentile achieves an Armo score of 0.18795, while the full three-metric RIP achieves 0.1898; the incremental gains from adding rejected length and reward gap are small. The paper should report final AlpacaEval2/Arena-Hard/WildBench results for each individual RIP metric and for the full conjunction, so that the contribution of the reward-gap component (and the 'combination' claim in the abstract) can be assessed.
minor comments (6)
  1. [Table 10] In Table 10, the line for Wildchat 20k with RIP says 'Reward Gap > 0.042' but the text in Section 5 and Section 3.3.1 say the gap should be small (below the 50% percentile); the inequality should be '<' or '≤'. This inconsistency makes the method ambiguous.
  2. [Throughout] There are several typos and inconsistent spellings: 'LLama' should be 'Llama' (Sections 4.1.1, 4.4, Table 9), 'AplacaEval-2' in Appendix A.1, 'commpared' in A.1, 'InstT ag' in Figure 1 caption, and inconsistent use of 'Alpaca-Eval' vs 'AlpacaEval2'.
  3. [Table 19] Table 19 has malformed entries ('58.3 3.2' and '51,2') that appear to be typographical errors; please correct them.
  4. [§3.3.1] The set-builder notation S = {x | τ_k ≤ m_k(x, yw, yl)} is written uniformly for all metrics, but for the reward gap the desired inequality is the opposite; please use separate notation for lower-bounded and upper-bounded thresholds.
  5. [Figure 2] The x-axis of Figure 2 is labeled 'Training Data Size (in log)' but the tick labels appear to be linear counts; clarify the scale.
  6. [§6.1] The analysis of unsafe-prompt removal in Table 7 is post-hoc and uses GPT-4 labels that were not used in threshold selection; the text should describe this explicitly as a post-hoc understanding rather than as a property the method was optimized for.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are evaluated on held-out external benchmarks and are not forced by the ArmoRM-based filtering definitions.

full rationale

RIP is an empirical filtering method, not a derivation. Its three metrics (rejected reward, rejected length, reward gap) are defined from sampled responses, and the paper claims that these predict prompt quality; the claim is tested by training DPO models on filtered WildChat, HelpSteer2, and synthetic data and measuring AlpacaEval2, Arena-Hard, and WildBench. None of these benchmark scores appears in the definition of the metrics or in the threshold-selection objective, and the thresholds and checkpoints are chosen on a separate 470-example validation set (Section 4.1.1, Appendix A.2, Tables 20-21). The same reward model, ArmoRM, is reused for pair construction, two of the three metrics, threshold sweeps, and early stopping, which creates a legitimate risk of selection on a proxy, but it does not make the external-benchmark result true by construction. The HelpSteer2 experiment uses human reward annotations for the filter metrics, and the LLM-as-a-Judge WildChat variant uses a different annotator, further showing the filtering signal is not definitionally tied to ArmoRM. Self-citations (e.g., Wu et al. 2024b, Yuan et al. 2024b, Xu et al. 2023b, Li et al. 2024c) support setup choices or background observations and are not invoked as uniqueness theorems or as the sole evidence for the central claim; the paper provides its own ablations and external evaluations. Therefore no circular step satisfying the quoted-reduction standard is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or model entities; RIP and Self-RIP are procedures over existing prompts and responses. The main uncharged costs are the three filtering thresholds, the sampling count N, and the reward-model/validation assumptions listed above.

free parameters (2)
  • Filtering thresholds (rejected length, rejected reward, reward gap) = WildChat 8B: rejected length >= 1878, rejected Armo >= 0.126, reward gap > 0.042; HelpSteer2: rejected length >= 1303…
    Selected by coordinate-wise experiments and hyperparameter search on a 470-example validation set (Section 3.3.1; Appendix A.2, Table 10). The central result depends on these specific cutoffs, and they are tuned rather than derived.
  • Number of sampled responses N for best-vs-worst pairing = 64 for Llama 3.1-8B main runs, 32 for Llama 3.3-70B, 8/16/32/64 in ablation
    N controls the quality of preference pairs and therefore the RIP metrics. The paper chooses N per experiment rather than deriving it, and larger N improves gains (Table 18).
assumptions (4)
  • domain assumption Low-quality prompts produce low-quality rejected responses and larger response variance (Hypotheses 1 and 2)
    Stated in Section 3.2 as the motivation for all three metrics. The paper tests the consequences indirectly but does not prove the underlying causal claim.
  • domain assumption ArmoRM reward scores are faithful proxies for response quality in WildChat and transfer to downstream benchmarks
    Used in Section 4.1.1 to construct preference pairs and to select thresholds and checkpoints. The paper does not independently calibrate ArmoRM against the downstream benchmark judges.
  • domain assumption The 470-example validation set (253 Humpback + 218 Evol-Instruct, overlaps removed) is representative for threshold and checkpoint selection
    Used for early stopping and threshold tuning in Section 4.1.1. The small size may make selected thresholds noisy or overfit.
  • domain assumption Best-vs-worst pairing of N on-policy samples yields valid preference pairs for DPO
    The method constructs yw and yl as max/min reward responses from the seed model (Section 3.1). This follows prior work but is assumed to be a useful preference signal for training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of R.I.P.: Better Models by Survival of the Fittest Prompts." pith.science (2026). https://pith.science/paper/JNOQN3KA

@misc{pith2026250118578,
  author       = {Pith},
  title        = {Pith review of: R.I.P.: Better Models by Survival of the Fittest Prompts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JNOQN3KA}},
  note         = {Machine review of arXiv:2501.18578}
}
read the original abstract

Training data quality is one of the most important drivers of final model quality. In this work, we introduce a method for evaluating data integrity based on the assumption that low-quality input prompts result in high variance and low quality responses. This is achieved by measuring the rejected response quality and the reward gap between the chosen and rejected preference pair. Our method, Rejecting Instruction Preferences (RIP) can be used to filter prompts from existing training sets, or to make high quality synthetic datasets, yielding large performance gains across various benchmarks compared to unfiltered data. Using Llama 3.1-8B-Instruct, RIP improves AlpacaEval2 LC Win Rate by 9.4%, Arena-Hard by 8.7%, and WildBench by 9.9%. Using Llama 3.3-70B-Instruct, RIP improves Arena-Hard from 67.5 to 82.9, which is from 18th place to 6th overall in the leaderboard.

Figures

Figures reproduced from arXiv: 2501.18578 by the authors.

Figure 1
Figure 1. Our method Rejecting Instruction Preferences (RIP) for curating data, and Self-RIP for creating syn￾thetic data. The x-axis represents the effective training set size (after filtering). At every data size training on unfil￾tered WildChat prompts is significantly outperformed by RIP. RIP also outperforms various other curation baselines. Synthetic data built by Self-RIP improves results further. to curate data – most… view at source ↗
Figure 2
Figure 2. Results on DPO Training with Varying WildChat Data Sizes. Using different sizes of WildChat data for DPO training on LLaMA 3.1-8B-Instruct, the performance, measured by Armo rewards on the validation set, gradually saturates as the data size increases. 0 10k 20k 30k 40k 50k Data Size 0.177 0.178 0.179 0.180 0.181 0.182 0.183 0.184 Armo Rewards 0.1774 0.1794 0.1812 0.1823 0.1831 0.1831 0.1829 Rewards vs Data Size Arm… view at source ↗
Figure 3
Figure 3. GPT4 eval prompt. I have a collection of prompts that I need to evaluate for their effectiveness in fine-tuning a language model. A useful prompt should: - Clearly ask a question - Be concise and specific - Directly relate to the topic of interest or follow given instructions Please assess each prompt and assign a score from 1 to 5 based on its usefulness: - 1: Pretty useful - 2: Somewhat useful - 3: Neutral (neithe… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Self-Instruct few-shot prompt template. Below are sample tasks from user. 1. <begin>{INSTRUCTION 1}</end> 2. <begin>{INSTRUCTION 2}</end> 3. <begin>{INSTRUCTION 3}</end> 4. <begin>{INSTRUCTION 4}</end> 5. <begin>{INSTRUCTION 5}</end> 6. <begin>{INSTRUCTION 6}</end> 7. …
Figure 5
Figure 5. Figure 5: t-SNE plots on instructions before and after filtering by rewards and lengths of rejected responses. Red dots represent unfiltered instructions, while blue dots are instructions curated by filtering out those with low-reward and shorter rejected responses. 20 [PITH_FU…
Figure 6
Figure 6. Figure 6: t-SNE plots on instructions before and after filtering by reward gaps. Blue dots represent instructions filtered only by rejected response, while yellow dots are instructions curated with smaller gap. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 6 canonical work pages

  1. [1]

    fairseq2, 2023

    Balioglu, C. fairseq2, 2023. URL http://github.com/facebookresearch/fairseq2

    work page 2023

  2. [2]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  3. [3]

    Instruction mining: Instruction data selection for tuning large language models

    Cao, Y., Kang, Y., Wang, C., and Sun, L. Instruction mining: Instruction data selection for tuning large language models. arXiv preprint arXiv:2307.06290, 2023

    arXiv 2023

  4. [4]

    AlpaGasus : Training a better alpaca with fewer data

    Chen, L., Li, S., Yan, J., Wang, H., Gunaratna, K., Yadav, V., Tang, Z., Srinivasan, V., Zhou, T., Huang, H., et al. AlpaGasus : Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=FdVXgSJhvz

    work page 2024

  5. [5]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  6. [6]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    arXiv 2022

  7. [7]

    Unnatural instructions: Tuning language models with (almost) no human labor

    Honovich, O., Scialom, T., Levy, O., and Schick, T. Unnatural instructions: Tuning language models with (almost) no human labor. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 14409--14428, Toronto, Canada, July 2023. Association ...

    doi:10.18653/v1/2023.acl-long.806 2023

  8. [8]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    arXiv 2001

Show all 52 references
  1. [9]

    Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models

    Khaki, S., Li, J., Ma, L., Yang, L., and Ramachandra, P. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. arXiv preprint arXiv:2402.10038, 2024

    2024 arXiv

  2. [10]

    Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., et al. T " ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  3. [11]

    From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning

    Li, M., Zhang, Y., Li, Z., Chen, J., Chen, L., Cheng, N., Wang, J., Zhou, T., and Xiao, J. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308.12032, 2023 a

    2023 arXiv

  4. [12]

    Superfiltering: Weak-to-strong data filtering for fast instruction-tuning

    Li, M., Zhang, Y., He, S., Li, Z., Zhao, H., Wang, J., Cheng, N., and Zhou, T. Superfiltering: Weak-to-strong data filtering for fast instruction-tuning. arXiv preprint arXiv:2402.00530, 2024 a

    2024 arXiv

  5. [13]

    E., and Stoica, I

    Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Zhu, B., Gonzalez, J. E., and Stoica, I. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024 b . URL https://lmsys.org/blog/2024-04-19-arena-hard/

    2024

  6. [14]

    Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023 b

    2023

  7. [15]

    Self-alignment with instruction backtranslation

    Li, X., Yu, P., Zhou, C., Schick, T., Zettlemoyer, L., Levy, O., Weston, J., and Lewis, M. Self-alignment with instruction backtranslation. In The Twelfth International Conference on Learning Representations, 2024 c . URL https://openreview.net/forum?id=1oijHJBRsT

    2024

  8. [16]

    Y., Deng, Y., Chandu, K., Brahman, F., Ravichander, A., Pyatkin, V., Dziri, N., Bras, R

    Lin, B. Y., Deng, Y., Chandu, K., Brahman, F., Ravichander, A., Pyatkin, V., Dziri, N., Bras, R. L., and Choi, Y. Wildbench: Benchmarking llms with challenging tasks from real users in the wild, 2024. URL https://arxiv.org/abs/2406.04770

    2024 arXiv

  9. [17]

    What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning

    Liu, W., Zeng, W., He, K., Jiang, Y., and He, J. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312.15685, 2023

    2023 arXiv

  10. [18]

    \# instag: Instruction tagging for analyzing supervised fine-tuning of large language models

    Lu, K., Yuan, H., Yuan, Z., Lin, R., Lin, J., Tan, C., Zhou, C., and Zhou, J. \# instag: Instruction tagging for analyzing supervised fine-tuning of large language models. In The Twelfth International Conference on Learning Representations, 2023

    2023

  11. [19]

    Simpo: Simple preference optimization with a reference-free reward

    Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024

    2024 arXiv

  12. [20]

    Cross-task generalization via natural language crowdsourcing instructions

    Mishra, S., Khashabi, D., Baral, C., and Hajishirzi, H. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773, 2021

    2021 arXiv

  13. [21]

    West-of-n: Synthetic preference generation for improved reward modeling

    Pace, A., Mallinson, J., Malmi, E., Krause, S., and Severyn, A. West-of-n: Synthetic preference generation for improved reward modeling. arXiv preprint arXiv:2401.12086, 2024

    2024 arXiv

  14. [22]

    W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al

    Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021

    2021 arXiv

  15. [23]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=HPuSIXJaa9

    2023

  16. [24]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  17. [25]

    Dolma: An open corpus of three trillion tokens for language model pretraining research

    Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159, 2024

    2024 arXiv

  18. [26]

    Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 0 3008--3021, 2020

    2020

  19. [27]

    Toneva, M., Sordoni, A., Combes, R. T. d., Trischler, A., Bengio, Y., and Gordon, G. J. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159, 2018

    2018 arXiv

  20. [28]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  21. [29]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  22. [30]

    Interpretable preferences via multi-objective reward modeling and mixture-of-experts

    Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845, 2024 a

    2024 arXiv

  23. [31]

    Y., Fazel-Zarandi, M., Weston, J., and Li, X

    Wang, T., Kulikov, I., Golovneva, O., Yu, P., Yuan, W., Dwivedi-Yu, J., Pang, R. Y., Fazel-Zarandi, M., Weston, J., and Li, X. Self-taught evaluators. arXiv preprint arXiv:2408.02666, 2024 b

    2024 arXiv

  24. [32]

    S., Naik, A., Stap, D., et al

    Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Arunkumar, A., Ashok, A., Dhanasekaran, A. S., Naik, A., Stap, D., et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.07705, 2022

    2022 arXiv

  25. [33]

    A., Khashabi, D., and Hajishirzi, H

    Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association ...

    2023 doi

  26. [34]

    J., Sreedhar, M

    Wang, Z., Dong, Y., Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J. J., Sreedhar, M. N., and Kuchaiev, O. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673, 2024 c

    2024 arXiv

  27. [35]

    Y., Guu, K., Yu, A

    Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021

    2021 arXiv

  28. [36]

    -dpo: Direct preference optimization with dynamic

    Wu, J., Xie, Y., Yang, Z., Wu, J., Gao, J., Ding, B., Wang, X., and He, X. -dpo: Direct preference optimization with dynamic . arXiv preprint arXiv:2407.08639, 2024 a

    2024 arXiv

  29. [37]

    Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge

    Wu, T., Yuan, W., Golovneva, O., Xu, J., Tian, Y., Jiao, J., Weston, J., and Sukhbaatar, S. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594, 2024 b

    2024 arXiv

  30. [38]

    Less: Selecting influential data for targeted instruction tuning

    Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333, 2024

    2024 arXiv

  31. [39]

    Wizardlm: Empowering large language models to follow complex instructions

    Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023 a

    2023 arXiv

  32. [40]

    Some things are more cringe than others: Preference optimization with the pairwise cringe loss

    Xu, J., Lee, A., Sukhbaatar, S., and Weston, J. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682, 2023 b

    2023 arXiv

  33. [41]

    Dataset pruning: Reducing training data by examining generalization influence

    Yang, S., Xie, Z., Peng, H., Xu, M., Sun, M., and Li, P. Dataset pruning: Reducing training data by examining generalization influence. arXiv preprint arXiv:2205.09329, 2022

    2022 arXiv

  34. [42]

    Alma: Alignment with minimal annotation

    Yasunaga, M., Shamis, L., Zhou, C., Cohen, A., Weston, J., Zettlemoyer, L., and Ghazvininejad, M. Alma: Alignment with minimal annotation. arXiv preprint arXiv:2412.04305, 2024

    2024 arXiv

  35. [43]

    Following length constraints in instructions

    Yuan, W., Kulikov, I., Yu, P., Cho, K., Sukhbaatar, S., Weston, J., and Xu, J. Following length constraints in instructions. arXiv preprint arXiv:2406.17744, 2024 a

    2024 arXiv

  36. [44]

    Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J

    Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024 b

    2024 arXiv

  37. [45]

    Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning

    Zhao, H., Andriushchenko, M., Croce, F., and Flammarion, N. Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning. arXiv preprint arXiv:2402.04833, 2024 a

    2024 arXiv

  38. [46]

    Wildchat: 1m chatgpt interaction logs in the wild

    Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470, 2024 b

    2024 arXiv

  39. [47]

    E., and Stoica, I

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM -as-a-judge with MT -bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Da...

    2023

  40. [48]

    Lima: Less is more for alignment

    Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024

    2024

  41. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  42. [50]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  43. [51]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  44. [52]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.