Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that balancing the number of prompts and response pairs in alignment data, then selecting the most informative pairs by their gradient signatures, lets a preference-optimized LLM match or beat standard DPO, RS-DPO, and…

desk verdict BPO's dynamic per-prompt depth augmentation is a real idea, but the paper's '≤10% data' claim is contradicted by its own Table 1 and needs a fix before publication. read the letter →

arxiv 2411.10914 v2 pith:J2MS2JEP submitted 2024-11-16 cs.CL

Sizhe Wang , Yongqi Tong , Hengyuan Zhang , Dawei Li , Xin Zhang , Tianlong Chen This is my paper · ORCID
classification cs.CL
keywords knowledgebreadthdepthpreferencedataoptimizationDPOgradientclusteringcompressionalignmentrejectionsampling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that alignment tuning data is imbalanced: many prompts but only two responses each, leading to over-learning of breadth and under-learning of depth. It proposes Balanced Preference Optimization (BPO), which first compresses prompts to a representative subset and then dynamically augments knowledge depth per prompt, using gradient-based clustering to pick the response pairs closest to cluster centroids. The central claim is that BPO with no more than 10% of the total data matches or exceeds vanilla DPO, RS-DPO, and Curry-DPO on MT-Bench and AlpacaEval, while reducing training cost. A reader should care because it suggests preference data can be aggressively compressed and selectively augmented without sacrificing alignment quality.

What carries the argument

The central object is knowledge depth allocation driven by gradient-based clustering. Concretely: after SFT with LoRA, each candidate (prompt, response pair) is assigned a projected gradient vector $\hat\nabla\ell(z;\theta_t)$ obtained by random projection of the LoRA gradient; K-means groups these into clusters, and within each cluster the pairs closest to the centroid (smallest Euclidean distance to $\mu_g$) are selected. This selection rule — Eq. (6) — is what converts "informativeness" into a concrete depth allocation: prompts with pairs near centroids get more response pairs, and the augmented set $D_{dyn}$ is then fed to DPO. The machinery also includes the embedding-based K-means prompt selection (Eq. 1) that compresses breadth by keeping the prompts nearest to prompt-embedding centroids.

What would settle it

Replace the gradient-centroid selection in Eq. (6) with random selection of the same number of response pairs per prompt, keeping every other component fixed; if the random-selection run matches BPO's MT-Bench and AlpacaEval scores, then the claim that centroid proximity tracks informativeness is falsified. A second check is to train BPO on a held-out distribution (e.g., a different domain) and see whether the performance advantage over random depth allocation persists.

Watch

Extended reading notes

Core claim

On the paper's own terms: we introduce knowledge breadth and knowledge depth as complementary measures of what an alignment dataset teaches, and we argue that the typical dataset structure — tens of thousands of prompts with only two responses each — allocates most learning to breadth and starves depth. We show that a simple uniform rebalancing, compressing prompts to 1–10% of the original set and augmenting the remaining prompts with extra response pairs, improves DPO results on SafeRLHF and HH-RLHF. Building on this, BPO makes depth augmentation dynamic: it generates multiple responses per prompt from an SFT policy, scores them with an LLM judge, and uses K-means on randomly projected LoRA gradients to select the response pairs closest to the gradient centroids, allocating more response pairs to prompts whose features sit near a centroid. We report that BPO with no more than 10% of the overall data achieves comparable or better MT-Bench and AlpacaEval scores than vanilla DPO, RS-DPO, and Curry-DPO across Llama-2-13B and Llama-3-8B, and that gradient-based depth allocation outperforms allocation based on response length or semantic similarity.

Load-bearing premise

The load-bearing premise is that K-means clustering of projected LoRA gradients identifies the most informative response pairs, with the samples closest to a cluster centroid being the ones worth extra learning resources; the paper's evidence for this is the final benchmark scores, which were also used to pick the cluster counts and selection ratio.

Editorial extensions

If this is right

  • If BPO is right, alignment pipelines can be run on roughly one-tenth of the original preference data, cutting training time and memory while keeping or improving benchmark wins.
  • The breadth/depth framing implies that simply increasing the number of responses per retained prompt is a reusable lever for alignment quality, independent of the base DPO objective.
  • Because gradient features, rather than surface statistics like response length or similarity, determine which samples get more depth, the method suggests that optimizing in gradient space is a better proxy for what the model needs to learn.
  • BPO's reported success on SafeRLHF, HH-RLHF, and UltraFeedback suggests the depth-allocation recipe transfers across safety, helpfulness, and general instruction-following data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's claim that centroid-adjacent gradient samples are more informative is a heuristic; it would be worth testing whether selecting the same number of pairs per prompt but uniformly at random from the gradient clusters (rather than from the centroids) collapses the performance gap, which would indicate the centroid rule is doing less work than claimed.
  • The 10% data compression suggests that most preference data may be redundant for DPO-style updates; if so, scaling laws for alignment data could be much shallower than scaling laws for pre-training.
  • A natural extension is to make the depth allocation adaptive during training, recomputing gradients and re-clustering after each epoch rather than once, which the paper does not explore.
  • The jailbreak-based response generation for safety data hints that depth augmentation only helps when the augmented pairs are actually separable (winning vs losing); a testable prediction is that the benefit of BPO shrinks when the judge scores have low variance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces knowledge breadth and knowledge depth as complementary notions for alignment data, arguing that preference datasets over-represent breadth (many prompts, two responses each) and under-represent depth. It proposes a simple balancing method (Knowledge Breadth Compression plus Knowledge Depth Augmentation) and then a full method, Balanced Preference Optimization (BPO), which selects a representative subset of prompts via embedding clustering, generates multiple candidate responses per prompt with an SFT policy, scores them with GPT-4, and dynamically allocates more response pairs to prompts whose gradient features lie near cluster centroids after random projection of LoRA gradients. The final DPO training is performed on the selected pairs. Experiments on SafeRLHF and HH-RLHF with Llama-3-8B and Llama-2-13B, plus an additional UltraFeedback experiment, report that BPO outperforms vanilla DPO, Curry-DPO, and RS-DPO on MT-Bench and AlpacaEval while using roughly 10% of the training data, and ablations compare clustering algorithms, cluster counts, and alternative depth-measurement signals.

Significance. If the central data-efficiency claim holds, BPO would be a practically valuable result: it would show that aggressive prompt selection plus per-prompt depth augmentation can match or beat full-data preference optimization at a fraction of the compute. The paper has real strengths: it tests two model families and multiple benchmarks, includes an extra UltraFeedback evaluation, provides detailed ablations of clustering choices and depth measures, and candidly acknowledges in the Limitations section the reliance on GPT-4 scoring and the limited gains from pure self-optimization. The code release is promised, which would help reproducibility. However, the evidence as presented does not yet establish the headline claim: the main tables contain no error bars or significance tests, the reported data sizes in Table 1 do not consistently satisfy the 'no more than 10%' statement, and the gradient-centroid informativeness mechanism is asserted rather than validated against matched random selection with statistical controls. These issues are fixable in revision, but they are load-bearing for the paper's central contribution.

major comments (4)
  1. [Table 1 and Section 4.2] The headline claim that BPO uses 'no more than 10% overall data' is not supported by the Data Size column under the natural reading. On HH-RLHF, BPO reports 3,312 pairs for Llama-3-8B and 2,753 pairs for Llama-2-13B, while vanilla DPO uses 27,000 pairs; these are 12.3% and 10.2% of the vanilla budget, respectively. The KBC (s=10%) row also reports 2,744 pairs, slightly above 10% of 27,000. If 'overall data' instead refers to the full 170K HH-RLHF set, then the comparison is uncontrolled because the vanilla baseline itself trains on only a 27K subsample. The paper should state the reference set explicitly, report exactly how many preference pairs are constructed from the k=16 generated responses before the 'top 10% of pairs' filter (Appendix H), and either cap the data budget at 10% of the same reference or revise the claim.
  2. [Section 2.3 and Appendix A, Table 4] The statement that selecting only 1% to 10% of prompts via embedding clustering 'can achieve performance comparable to, and even surpassing, alignment on the full dataset' is contradicted by the HH-RLHF rows in Table 4. At a 10% scaling ratio, MT-Bench is 4.11 versus 4.71 for the full set, and at 20% the AlpacaEval win rate is 16.77% versus 18.14% for the full set. The claim should be restricted to the specific settings where the result actually holds, or rephrased as a mixed outcome that depends on dataset and metric.
  3. [Section 4.2, Tables 1, 5, and 6] All reported comparisons are single-run point estimates without error bars, confidence intervals, or significance tests. Several differences are small enough to be within evaluation noise: for example, on SafeRLHF with Llama-2-13B, MT-Bench is 5.85 for BPO versus 5.79 for RS-DPO, and on HH-RLHF with Llama-2-13B, AlpacaEval is 26.96% for BPO versus 25.26% for RS-DPO. The paper should report multiple seeds with means and standard deviations, or otherwise quantify variability, for the main tables before claiming consistent superiority.
  4. [Section 3.2, Eq. (6), and Section 5.1] The core mechanism of BPO is that gradient vectors closest to cluster centroids mark the most informative samples and therefore deserve more depth resources. This is an asserted heuristic, and the supporting evidence is Figure 6 plus final benchmark scores that were also used to choose the number of gradient clusters and the selection ratio eta (Figure 5 and Table 5). Because G and eta are tuned on the same evaluation benchmarks, the improvement over random selection could partly reflect selection on the test set. I recommend adding a matched-compute comparison with random selection across multiple seeds and, ideally, using a held-out validation split for hyperparameter selection, so the benefit of the gradient signal can be separated from tuning effects.
minor comments (4)
  1. [Throughout] There are numerous typos and duplicated words, including 'hierichical' (Section 4.2), 'conducteded' and 'anselectedcted' (Appendix I), 'obtian' (Section 3.2), and 'B BPO achieves' (Section 4.2). The paper needs a careful copyedit.
  2. [Eq. (1) and Figure 5] The notation for the selection ratio is inconsistent: Eq. (1) defines the breadth-compression ratio as s, while the caption of Figure 5 says 'The top eta = 10% of data points are selected based on Equation 1.' Use one symbol for the proportion selected in each stage.
  3. [Appendix H] The pair-construction convention is not specified. From k=16 generated responses, one could form 15 pairs (one chosen response versus each rejected response) or 120 ordered pairs, and the final Data Size after the 'top 10% of pairs' filter depends strongly on this choice. This should be stated explicitly so the data-efficiency claim can be checked.
  4. [Appendix A, Table 4] The caption says 'almost only 1% to 10% of the prompts can lead to promising outcomes,' but the HH-RLHF columns show a non-monotonic pattern, with 20% worse than 1% on AlpacaEval. The caption overstates the regularity of the trend.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity found: BPO is an empirically evaluated selection heuristic, and the Table 1 data-efficiency inconsistency is a correctness concern rather than a circular derivation.

full rationale

The paper's load-bearing claim is that gradient-clustered dynamic augmentation of response pairs improves DPO (Section 3.2, Eq. 6). This is not equivalent to its inputs: the selection rule is an explicitly stated heuristic, and its support comes from external benchmarks (MT-Bench, AlpacaEval, and the UltraFeedback experiments in Appendix I) plus a direct random-selection control (Figure 6). No parameter is fit to a target and then reported as a prediction of that target; the s=10% and eta=10% ratios are fixed selection hyperparameters, and the final DPO model is evaluated on held-out instruction-following benchmarks. The concepts 'knowledge breadth' and 'knowledge depth' are operationalized as counts of prompts and response pairs (Sections 2.1 and 2.2), which makes the label 'depth' partly definitional, but the empirical claim that balancing these counts helps alignment is tested rather than assumed, so this is a construct-validity concern, not a circular derivation. The only self-citation, Zhang et al. (2024), supports a motivational claim about LLM depth limitations and is not load-bearing. The reported 'no more than 10% overall data' claim is not consistently supported by Table 1 (e.g., 3,312/27,000 = 12.3% for Llama-3-8B on HH-RLHF), and the pair-construction chain is under-specified; this is an internal-consistency or accounting problem, not circularity. Likewise, tuning cluster counts and eta on the same evaluation benchmarks (Figure 5, Table 5) is a test-set-selection risk, but it does not make the evaluated outcome equal to the input by construction.

Assumptions & free parameters 8 free parameters · 7 assumptions · 0 invented entities

The method rests on several domain assumptions about what knowledge depth means and how gradients identify useful samples. The free parameters are hyperparameters selected by hand and benchmark tuning, not fitted constants, but they affect the reported data-efficiency gains. No new physical or structural entities are introduced; knowledge breadth and depth are conceptual labels rather than entities with independent evidence.

free parameters (8)
  • scaling ratio s = 0.1
    Chosen in Section 4.1 to balance breadth and depth; validated only by benchmark outcomes and central to the claim of using 10% of the data.
  • embedding cluster count C = 100
    Chosen in Section 5.1; Figure 5 shows sensitivity to this value.
  • gradient cluster count G = 50
    Chosen in Section 5.1; Figure 5 and Table 5 show sensitivity.
  • gradient selection ratio eta = 10%
    Used in Eq. 6; Table 5 shows that 1% and 5% perform worse on HH-RLHF.
  • generated responses per prompt k = 16
    Appendix H; determines the pool of candidate responses for depth augmentation.
  • gradient projection dimension d = 8192
    Appendix C and H; follows LESS settings and is chosen by hand.
  • top-pair score-difference cutoff = top 10%
    Appendix H; only pairs with the highest GPT-4 score differences are kept before gradient computation.
  • response sampling temperature, top-k, max tokens = 1.0, 50, 512
    Appendix H; controls diversity of generated responses during augmentation.
assumptions (7)
  • domain assumption Knowledge depth of an aligned model can be increased by adding more response pairs per prompt, i.e., K scales with response count.
    Invoked in Section 2.2 when mapping L=(n,2) to Lbal=(n*s,2/s); not proven, and alternative depth measures (length, semantic similarity) are tested and rejected.
  • domain assumption Prompts closest to K-means centroids in embedding space are the most representative and sufficient for knowledge breadth.
    Eq. 1 in Section 3.1; the core of Knowledge Breadth Compression.
  • domain assumption Samples whose LoRA gradient features are closest to gradient-cluster centroids are the most informative and should receive more depth resources.
    Eq. 6 in Section 3.2; the central mechanism of BPO, asserted without derivation.
  • domain assumption GPT-4-as-a-judge correctly identifies chosen and rejected responses for pair construction.
    Section 3.2 and the Limitations section; the authors acknowledge dependence on GPT-4.
  • domain assumption Random projection of LoRA gradients preserves enough geometry for meaningful K-means clustering.
    Appendix C, following LESS (Xia et al. 2024); adopted without new validation.
  • domain assumption MT-Bench and AlpacaEval scores are valid proxies for alignment quality and knowledge depth.
    Section 4.1; only two automatic benchmarks are used for the main comparisons.
  • domain assumption SFT-policy generations, including jailbreak-prompted responses on SafeRLHF, form valid preference pairs for safety alignment.
    Section 5.3; unusual method invoked only because normal generations were too safe to produce contrastive pairs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment." pith.science (2026). https://pith.science/paper/J2MS2JEP

@misc{pith2026241110914,
  author       = {Pith},
  title        = {Pith review of: BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J2MS2JEP}},
  note         = {Machine review of arXiv:2411.10914}
}
read the original abstract

Reinforcement Learning with Human Feedback (RLHF) is the key to the success of large language models (LLMs) in recent years. In this work, we first introduce the concepts of knowledge breadth and knowledge depth, which measure the comprehensiveness and depth of an LLM or knowledge source respectively. We reveal that the imbalance in the number of prompts and responses can lead to a potential disparity in breadth and depth learning within alignment tuning datasets by showing that even a simple uniform method for balancing the number of instructions and responses can lead to significant improvements. Building on this, we further propose Balanced Preference Optimization (BPO), designed to dynamically augment the knowledge depth of each sample. BPO is motivated by the observation that the usefulness of knowledge varies across samples, necessitating tailored learning of knowledge depth. To achieve this, we introduce gradient-based clustering, estimating the knowledge informativeness and usefulness of each augmented sample based on the model's optimization direction. Our experimental results across various benchmarks demonstrate that BPO outperforms other baseline methods in alignment tuning while maintaining training efficiency. Furthermore, we conduct a detailed analysis of each component of BPO, providing guidelines for future research in preference data optimization.

Figures

Figures reproduced from arXiv: 2411.10914 by the authors.

Figure 1
Figure 1. Overview of knowledge breadth and depth, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Preliminary experiment results on SafeRLHF using simple balance. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of BPO pipeline. BPO first selects representative prompts to reduce knowledge breadth through embedding-based clustering. Next, it generates responses using the SFT policy and employs GPT-4 to score these responses to uniformly construct response pairs. Subsequently, BPO samples response pairs to dynamically augment knowledge depth through gradient-based clustering. Finally, DPO is applied to the sampled da… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Dynamic allocation of response pairs based on [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Experimental results for varying numbers of clusters during embedding-based and gradient-based [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Experimental results for different clustering algorithms and random selection used in knowledge breadth [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Fragments to Facts: A Curriculum-Driven DPO Approach for Generating Hindi News Veracity Explanations

    cs.CL 2025-07 unverdicted novelty 6.0 of 10

    DeFactoX, a curriculum-driven DPO variant with Actuality and Finesse loss weighting, improves automatic and human scores for Hindi news explanation generation over existing preference optimization baselines.

Reference graph

Works this paper leans on

49 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [1]

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. 2024. A survey on data selection for language models. arXiv preprint arXiv:2402.16827

    arXiv 2024

  2. [2]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  3. [3]

    Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. 2024. Benchmarking foundation models with language-model-as-an-examiner. Advances in Neural Information Processing Systems, 36

    work page 2024

  4. [4]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30

    2017

  5. [5]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. 2024. Ultrafeedback: Boosting language models with scaled ai feedback. In Forty-first International Conference on Machine Learning

    2024

  6. [6]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback

    2023

  7. [7]

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2023. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773

    arXiv 2023

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

Show all 49 references
  1. [9]

    Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475

    2024 arXiv

  2. [10]

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306

    2024 arXiv

  3. [11]

    Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human-like summarization evaluation with chatgpt. arXiv preprint arXiv:2304.02554

    2023 arXiv

  4. [12]

    Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, and Tianlu Wang. 2023. Understanding in-context learning via supportive pretraining data. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long ...

    2023

  5. [13]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  6. [14]

    William B Johnson. 1984. Extensions of lipshitz mapping into hilbert space. In Conference modern analysis and probability, 1984, pages 189--206

    1984

  7. [15]

    Adam Tauman Kalai and Santosh S. Vempala. 2024. https://arxiv.org/abs/2311.14648 Calibrated language models must hallucinate . Preprint, arXiv:2311.14648

    2024 arXiv

  8. [17]

    Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. 2024 b . https://arxiv.org/abs/2402.10038 Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models . Preprint, arXiv:2402.10038

    2024 arXiv

  9. [18]

    Diederik P Kingma. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  10. [19]

    Dawei Li, Shu Yang, Zhen Tan, Jae Young Baik, Sunkwon Yun, Joseph Lee, Aaron Chacko, Bojian Hou, Duy Duong-Tran, Ying Ding, et al. 2024 a . Dalk: Dynamic co-augmentation of llms and kg to answer alzheimer's disease questions with scientific literature. arXiv preprint arXiv:2405.04819

    2024 arXiv

  11. [20]

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024 b . From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939

    2024 arXiv

  12. [21]

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models

    2023

  13. [22]

    Liu, and Jialu Liu

    Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. 2024. https://arxiv.org/abs/2309.06657 Statistical rejection sampling improves preference optimization . Preprint, arXiv:2309.06657

    2024 arXiv

  14. [23]

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2023. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312.15685

    2023 arXiv

  15. [24]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734

    2024 arXiv

  16. [25]

    Tetsuro Morimura, Mitsuki Sakamoto, Yuu Jinnai, Kenshi Abe, and Kaito Air. 2024. Filtered direct preference optimization. arXiv preprint arXiv:2404.13846

    2024 arXiv

  17. [28]

    Pulkit Pattnaik, Rishabh Maheshwary, Kelechi Ogueji, Vikas Yadav, and Sathwik Tejaswi Madhusudhan. 2024 c . https://arxiv.org/abs/2403.07230 Curry-dpo: Enhancing alignment using curriculum learning & ranked preferences . Preprint, arXiv:2403.07230

    2024 arXiv

  18. [29]

    Alina Petukhova, Joao P Matos-Carvalho, and Nuno Fachada. 2024. Text clustering with llm embeddings. arXiv preprint arXiv:2403.15112

    2024 arXiv

  19. [30]

    Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. 2020. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920--19930

    2020

  20. [31]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  21. [32]

    Amir Saeidi, Shivanshu Verma, and Chitta Baral. 2024. Insights into alignment: Evaluating dpo and its variants across multiple tasks. arXiv preprint arXiv:2404.14723

    2024 arXiv

  22. [33]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  23. [34]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. https://arxiv.org/abs/2308.03825 "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models . Preprint, arXiv:2308.03825

    2024 arXiv

  24. [36]

    Feifan Song, Bowen Yu, Hao Lang, Haiyang Yu, Fei Huang, Houfeng Wang, and Yongbin Li. 2024 b . https://arxiv.org/abs/2403.11124 Scaling data diversity for fine-tuning language models in human alignment . Preprint, arXiv:2403.11124

    2024 arXiv

  25. [37]

    Zhen Tan, Alimohammad Beigi, Song Wang, Ruocheng Guo, Amrita Bhattacharjee, Bohan Jiang, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation: A survey. arXiv preprint arXiv:2402.13446

    2024 arXiv

  26. [38]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  27. [39]

    Tianduo Wang, Shichen Li, and Wei Lu. 2024. Self-training with direct preference optimization improves chain-of-thought reasoning. arXiv preprint arXiv:2407.18248

    2024 arXiv

  28. [40]

    Lai Wei, Zihao Jiang, Weiran Huang, and Lichao Sun. 2023. Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4. arXiv preprint arXiv:2308.12067

    2023 arXiv

  29. [41]

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333

    2024 arXiv

  30. [42]

    Siheng Xiong, Ali Payani, Yuan Yang, and Faramarz Fekri. 2024. Deliberate reasoning for llms as structure-aware planning with accurate world model. arXiv preprint arXiv:2410.03136

    2024 arXiv

  31. [43]

    Hengyuan Zhang, Yanru Wu, Dawei Li, Zacc Yang, Rui Zhao, Yong Jiang, and Fei Tan. 2024. Balancing speciality and versatility: a coarse to fine framework for supervised fine-tuning large language model. arXiv preprint arXiv:2404.10306

    2024 arXiv

  32. [44]

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. 2023 a . Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425

    2023 arXiv

  33. [45]

    Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. 2022. Calibrating sequence likelihood improves conditional language generation. In The eleventh international conference on learning representations

    2022

  34. [46]

    Yingxiu Zhao, Bowen Yu, Binyuan Hui, Haiyang Yu, Fei Huang, Yongbin Li, and Nevin L Zhang. 2023 b . A preliminary study of the intrinsic relationship between complexity and alignment. arXiv preprint arXiv:2308.05696

    2023 arXiv

  35. [47]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023 a . Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  36. [48]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023 b . https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena ...

    2023 arXiv

  37. [49]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36

    2024

  38. [50]

    B Zhu, E Frick, T Wu, H Zhu, and J Starling Jiao. 2023 a . 7b: Improving llm helpfulness & harmlessness with rlaif

    2023

  39. [51]

    Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael I Jordan, and Jiantao Jiao. 2023 b . Fine-tuning language models with advantage-induced policy alignment. arXiv preprint arXiv:2306.02231

    2023 arXiv

  40. [52]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  41. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.