Pith. sign in

REVIEW 4 major objections 5 minor 31 references

SGPO: Self-Generated Preference Optimization based on Self-Improver

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single LLM refines its own answers into preference data and beats DPO alignment.

desk verdict Clever self-improvement loop with a serious baseline-tuning flaw: the DPO comparison uses a 50x different beta, so the headline 'significantly outperforms' claim is not yet supported. read the letter →

arxiv 2507.20181 v1 pith:CGWZ3LAX submitted 2025-07-27 cs.CL cs.AI

Hyeonji Lee , Daejin Jo , Seohwan Yun , Sungwoong Kim This is my paper
classification cs.CLcs.AI
keywords self-improvingalignmentdirectpreferenceoptimizationon-policydatasyntheticself-improverLLMresponserefinementSFTreference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that preference alignment can be driven entirely by a model's own improved responses, removing the need for human-written preference pairs. It proposes SGPO, in which one model first answers an instruction and then refines that answer into a clearly better version by referencing an SFT response, and the pair is used as on-policy preference data for DPO. The improver is trained once, on incremental refinements of the initial model's outputs generated by a stronger external LLM, constrained to stay close to the initial distribution. If the claim holds, alignment becomes cheaper and more online: no external preference annotation, no separate reward model, and preference data that always match the current policy's output distribution. The reported results on AlpacaEval 2.0 and Arena-Hard support this claim across three base model sizes.

What carries the argument

The central mechanism is the integrated improver-policy model: one shared model, invoked with different prompts, first produces an answer and then a revised version of that same answer. The revision is trained with a supervised objective on targets from an external LLM that satisfy two constraints—an edit-distance cap keeping the revision close to the original response, and a quality ceiling at the SFT reference level—so the improvement signal is incremental rather than aggressive. A perplexity-based interquartile-range filter removes targets that lie far outside the initial policy's output distribution, and DPO then treats the revision as the chosen response and the original as the rejected response, making every preference pair on-policy.

What would settle it

Concrete check: take the post-Step-2 policy $\pi_{\theta^{**}}$ and have the original improver $\pi_{\theta^*}$ refine 500 of its responses; if GPT-4 Turbo judges the refinements as better than the originals at a rate near 50%, or if the chosen responses in $\mathcal{D}_P$ are not rated above the rejected ones, the central transfer claim collapses. A cleaner test: retrain the improver on refinements of the updated policy's responses and compare SGPO's benchmark scores, since unchanged scores would indicate that the one-time improver training is not the load-bearing component.

Watch

Extended reading notes

Core claim

SGPO claims that a single language model can act as both generator and improver: it samples a response to an instruction, then refines that same response toward an SFT reference, and the refined-versus-original pair serves as an on-policy preference pair for direct preference optimization. Trained on target refinements produced by GPT-4 Turbo that are edit-distance-constrained to stay close to the initial policy's outputs and are judged to be no better than the SFT reference, the improver learns gradual, attainable improvements rather than large leaps. The paper reports that this design significantly outperforms DPO and prior self-improving approaches on AlpacaEval 2.0 and Arena-Hard across Qwen2.5-Base (7B), Llama3-Base (8B), and Qwen2-Base (1.5B), with gains up to 16.18 points in length-controlled win rate and 17.3 points in Arena-Hard win rate, all without human preference labels.

Load-bearing premise

The improver, trained once on refinements of the initial policy's responses, remains effective when asked to refine the responses of the updated policy during preference optimization.

Editorial extensions

If this is right

  • SGPO outperforms DPO on AlpacaEval 2.0 by up to 16.18 points in length-controlled win rate and on Arena-Hard by up to 17.3 points in win rate, without using any human preference annotations.
  • Both chosen and rejected responses come from the current policy, and replacing either side with SFT or initial-policy responses degrades performance, showing that on-policy pairing is the main driver of the gains.
  • The two design choices—unifying improver and policy into one model, and training on improved responses rather than only external-LLM responses—each contribute; the full SGPO beats both ablations on every benchmark.
  • SGPO can be iterated: applying a second Step-2 preference optimization without retraining the improver further improves most reported numbers, suggesting the loop is self-sustaining.
  • The method needs only a small SFT dataset for improver training and the SFT split of UltraFeedback for policy updates, avoiding preference labels and separate reward models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the one-time improver training is the method's likely ceiling; a direct diagnostic comparing refinement win rates on initial versus updated policy responses would reveal how quickly the improver's advantage decays as the policy shifts.
  • Editorial inference: SGPO still depends on a stronger external LLM and SFT references for its training targets, so 'without external preference data' should not be read as 'without external supervision'; a fully self-contained upgrade would need the model to propose its own refinement targets.
  • Editorial inference: the edit-distance and 'no better than SFT' constraints are verified only on sampled win rates and histograms; an ablation that removes the edit-distance cap while holding everything else fixed could isolate how much of the gain comes from incremental targets versus from on-policy pairing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SGPO, an alignment framework in which a single language model acts as both the policy and a response improver. In Step 1, the initial policy model generates responses, and GPT-4 Turbo is prompted to produce improved responses under explicit constraints: the improved response should stay close to the policy response in edit distance and should not exceed the SFT reference in quality. The resulting targets, filtered by perplexity-based IQR, are used to train the unified improver-policy model. In Step 2, the trained model generates both a policy response and an improved response, which are used as rejected and chosen samples, respectively, in a DPO-style objective. The method is evaluated on AlpacaEval 2.0 and Arena-Hard with Qwen2.5-Base (7B), Llama3-Base (8B), and Qwen2-Base (1.5B), and it is compared against DPO, SPIN, and two SGPO variants. The central claim is that SGPO significantly outperforms DPO and prior self-improving methods without human preference annotations.

Significance. If the results are upheld after a controlled comparison, SGPO is a practically interesting contribution: it removes the need for human preference pairs, uses a single shared model for both generation and refinement, and provides a concrete recipe for constructing gradually improving targets. The manuscript has notable strengths: the ablation structure isolating the improver-policy integration and the improved-response supervision (Tables 4 and 5) is well designed and internally consistent; Figure 8 provides a useful transfer diagnostic showing that the trained improver refines current-policy responses better than the initial-policy improver; and Table 6 demonstrates one additional self-boosting step without retraining the improver. The main weakness is that the headline comparison against DPO is built on a hyperparameter mismatch and on single-run results with no variance estimates, so the significance claim is currently not supported. The method also relies on GPT-4 Turbo as both teacher and judge, which should be discussed as a potential benchmark-confounding factor.

major comments (4)
  1. [§4.2.1, §4.2.2, Table 3] The DPO baseline is trained with beta=0.01 (§4.2.1), while SGPO's preference-optimization step uses beta=0.5 (§4.2.2), with both using one epoch and a learning rate of 5e-7. Because the DPO objective in Eq. (6) scales multiplicatively with beta, the baseline is trained under 50x weaker preference-update pressure, so the large LC/WR gaps in Table 3 may reflect an under-tuned baseline rather than a property of SGPO. The Section 1 claim that SGPO 'significantly outperforms baseline preference optimization methods including DPO' is therefore not established by the reported comparison. Please add a beta sweep for DPO (including beta=0.5, with the same learning rate and epoch count) or otherwise demonstrate that the reported superiority is not dominated by the beta mismatch.
  2. [Tables 3, 4, 6; §4.4.1] All reported numbers appear to come from a single run, with no seeds, standard deviations, or statistical tests. The word 'significantly' in the abstract and Section 1 is therefore not statistically grounded. Please report results over multiple seeds (at least three) with means and standard deviations for the main comparisons and key ablations, or provide a paired significance test over the 805/500 benchmark prompts. This is necessary to support the consistency claims made throughout Section 4.4.
  3. [§3.1.1, §3.1.1.4, Table 2] GPT-4 Turbo is both the teacher that produces the improver training targets in Eq. (2) and the judge on AlpacaEval 2.0 (as GPT-4-Preview-1106, it is also the judge on Arena-Hard, per Table 2). This train-judge overlap means the reported improvements may partly reflect the model imitating GPT-4 Turbo's stylistic response preferences rather than aligning better with human preferences. Since no human evaluation or independent judge is provided, the practical claim that SGPO improves model quality is at risk of being confounded by judge-model alignment. Please add a discussion of this limitation and, if feasible, an evaluation with an independent LLM judge or a small human study.
  4. [§4.4.3, Table 6, §5] The claim in Section 5 that 'the self-improver does not suffer from degraded refinement performance during the policy updates' is supported by only one additional Step 2 and by Figure 8, which evaluates the improver on responses from pi_theta* rather than on responses from later policies such as pi_theta**. The discussion of 'fully iterative learning' in Section 5 is therefore extrapolated beyond the tested regime. Please either narrow the claim to the number of iterations actually measured or add a diagnostic that evaluates improver win rate on responses from the later policy in SGPO + Step 2.
minor comments (5)
  1. [§3.1.1, reference [31]] The edit-distance constraint is attributed to reference [31], but that reference (TLCR) is about token-level continuous rewards, not edit distance; the citation appears mismatched and should be corrected or replaced.
  2. [References [25] and [30]] Reference [25] is labeled as the Qwen2 technical report but the arXiv ID 2412.15115 corresponds to the Qwen2.5 technical report; please correct the bibliographic data to avoid confusion with reference [30].
  3. [Tables 10-12] Several SGPO responses in Tables 10-12 contain stray '2019s' artifacts (e.g., 'Here's 2019s a suggested agenda', 'the day's 2019s priorities', 'Yesterday's 2019s Achievements'); these appear to be formatting or encoding errors that should be cleaned before publication.
  4. [§3.1.1, Table 1] The phrase 'approximately 80% higher quality' is imprecise given the reported win rates of 87.00/86.50 for improved-versus-initial and 50.50/45.00 for improved-versus-SFT; stating the percentage-point differences directly would be clearer.
  5. [General] No code or data release is mentioned; including a reproducibility statement with training/inference code or model checkpoints would be valuable given the many nonstandard choices (prompt templates, IQR filtering, edit-distance constraints).

Circularity Check

2 steps flagged · score 4.0 of 10

Partial circularity: GPT-4 Turbo is both the teacher that generates SGPO's improvement targets and the judge for the reported benchmark wins, so the central gains and the improver evaluation partly measure self-agreement with the evaluator.

  1. self definitional [Sec. 3.1 (Eq. 2) + Sec. 4.3 (Table 2)]
    "we first generate a response ˆyθ0,i by the initial policy πθ0 and obtain its improved response ˜yφ,i from an external LLM πφ with prompt r ... we use an external LLM (GPT-4 Turbo [21]) to generate the target improved responses ... AlpacaEval 2.0 ... Judge Model GPT-4 Turbo ... Arena-Hard ... Judge Model GPT-4 Turbo"

    The same GPT-4 Turbo that produces the target improved responses for training the self-improver (Eq. 2) is also the judge on both benchmarks used for the main results (Table 2). Step 2 constructs chosen responses as refinements generated by the trained self-improver, which has been trained to imitate GPT-4 Turbo's refinements; the benchmark win rate then measures how often the judge prefers outputs modeled on its own preferences. The claimed 'significant outperformance' over DPO is therefore partly a self-agreement score rather than an independent test of preference quality. The benchmark prompts and baselines are external, so the result is not forced by construction, but the training signal and the evaluation metric share one source, which is a partial circularity in the central claim.

  2. other [Sec. 4.4.2 (Fig. 8); cf. Sec. 3.1.1 (Table 1)]
    "Using GPT-4 Turbo as an automated annotator, we perform pairwise comparisons on 500 samples ... the responses refined by the trained self-improver πθ∗ achieve a win rate of approximately 70% when compared to both ˜yθ0 and y, demonstrating better refinement."

    The demonstration that the self-improver refines better is conducted by asking GPT-4 Turbo to judge responses produced by a model trained to imitate GPT-4 Turbo's refinements. This is self-evaluation: the teacher model is used as the examiner of its own student, so the ~70% win rate largely reflects stylistic agreement with the teacher. The same pattern appears in Sec. 3.1.1, where GPT-4 Turbo validates its own generated improvements. It does not by itself prove that the refined responses are better according to independent human preference, so this supporting evidence is circular.

full rationale

The derivation chain of SGPO is not circular in the fitted-parameter sense: Eq. (6) is a standard DPO objective and no parameter is fitted to benchmark outcomes and then reported as a prediction. The improver/policy roll-out and IQR filtering are independent procedural steps. However, there is a real train-judge overlap. The external LLM GPT-4 Turbo produces the target improved responses used to train the self-improver (Eq. 2), and the same GPT-4 Turbo is the judge on AlpacaEval 2.0 and Arena-Hard (Table 2). Because the chosen responses in Step 2 are generated by the trained improver, they are implicitly matched to the judge's refinement preferences; the reported SGPO-vs-DPO win rates therefore partly reflect how closely the model imitates the evaluator, not purely independent response quality. Additionally, Section 4.4.2's claim that the self-improver is better is verified by GPT-4 Turbo, the same model whose outputs trained it, making that demonstration self-confirmatory. This is partial circularity, so score 4: the benchmark prompts, baseline models, and DPO update remain external and not forced by construction, so the central result has independent content. The skeptic's beta mismatch (β=0.01 DPO vs β=0.5 SGPO) is a baseline-fairness concern, not a circularity, and does not raise this score. The self-citation to [31] for edit-distance prompting is a design choice with ablations, not a load-bearing self-citation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a standard DPO objective, but the quality of the self-generated preference data depends on hand-set prompts, the IQR filter, and the transfer of GPT-4 Turbo refinement behavior into a smaller policy model. No new particles or physical entities are introduced.

free parameters (3)
  • DPO temperature beta = 0.5 for SGPO, 0.01 for DPO baseline
    Controls deviation from the reference policy in the DPO objective (Eq. 6) and is set by hand; the paper does not sweep or justify it beyond a reference to beta-DPO.
  • IQR filter multiplier = 1.5
    Standard Tukey IQR criterion used to remove perplexity outliers from improver training data (Section 3.1.2).
  • Edit distance constraint threshold = Unspecified (prompt-based)
    The prompt instructs the external LLM to keep improvements close to the initial response in edit distance, but no quantitative threshold is specified; this hand-crafted constraint shapes the preference gap.
assumptions (5)
  • standard math Bradley-Terry model of preferences as used in DPO
    The preference optimization objective Eq. (6) inherits the Bradley-Terry assumption that pairwise preferences satisfy the logistic model.
  • domain assumption SFT responses are high-quality reference outputs
    The improver and refinement prompts use SFT responses as a quality ceiling (response-A), assuming they are reliably better than policy outputs; this assumption underlies the training signal in Section 3.1.1.
  • domain assumption GPT-4 Turbo's edit-constrained refinements are learnable by a 7B/8B model in one SFT step
    Step 1 (Eq. 4) trains the policy as improver on GPT-4 Turbo targets, assuming a single training pass distills this refinement behavior into the smaller model.
  • domain assumption Perplexity under the initial policy is a valid proxy for on-policy alignment
    IQR filtering of the improver training data removes samples whose perplexity is extreme under the initial policy, assuming these are outliers harmful to learning rather than informative signals.
  • ad hoc to paper The prompt constraint on edit distance and quality ceiling produces learnable improvement targets
    This instruction is introduced specifically to make the preference gap incremental; no independent evidence shows this is the optimal wording.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SGPO: Self-Generated Preference Optimization based on Self-Improver." pith.science (2026). https://pith.science/paper/CGWZ3LAX

@misc{pith2026250720181,
  author       = {Pith},
  title        = {Pith review of: SGPO: Self-Generated Preference Optimization based on Self-Improver},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CGWZ3LAX}},
  note         = {Machine review of arXiv:2507.20181}
}
read the original abstract

Large language models (LLMs), despite their extensive pretraining on diverse datasets, require effective alignment to human preferences for practical and reliable deployment. Conventional alignment methods typically employ off-policy learning and depend on human-annotated datasets, which limits their broad applicability and introduces distribution shift issues during training. To address these challenges, we propose Self-Generated Preference Optimization based on Self-Improver (SGPO), an innovative alignment framework that leverages an on-policy self-improving mechanism. Specifically, the improver refines responses from a policy model to self-generate preference data for direct preference optimization (DPO) of the policy model. Here, the improver and policy are unified into a single model, and in order to generate higher-quality preference data, this self-improver learns to make incremental yet discernible improvements to the current responses by referencing supervised fine-tuning outputs. Experimental results on AlpacaEval 2.0 and Arena-Hard show that the proposed SGPO significantly improves performance over DPO and baseline self-improving methods without using external preference data.

Figures

Figures reproduced from arXiv: 2507.20181 by the authors.

Figure 1
Figure 1. Overview of the Self-Generated Preference Optimization (SGPO) framework. The SGPO framework consists of two steps: [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Prompt for an external LLM to obtain target improved responses. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Histogram of edit distances comparing the improved response and the SFT response (response-A) against the initial policy [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Histogram of edit distances comparing the improved response and the SFT response (response-A) against the initial policy [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Histograms of perplexities of improved responses before and after IQR filtering (outliers removed). The top plot corresponds to [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Prompt 𝑟 for training self-improver, as denoted in Eq. (4). outliers using the interquartile range (IQR) [27] method: the lower quartile (Q1) and upper quartile (Q3) of the perplexity values, corresponding to the 25th and 75th percentiles respectively, are calculated, …
Figure 7
Figure 7. Figure 7: Prompt 𝑟 for refining the response using self-improver, as denoted in Eq. (1). When the trained self-improver generates an improved response, we utilize SFT response y to guide the current policy. The details of the refinement prompt used in this process are provided i…
Figure 8
Figure 8. Figure 8: Performance evaluation of the self-improver. A represents the improved responses [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Histogram of log-scaled perplexity values measured by the self-improver for each set of responses. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Performance evaluation of the self-improver under different training prompt settings. The graphs compare the improved [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 10 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Allenai. 2023. allenai-ultrafeedback-binarized-cleaned. Retrieved June 25, 2025 from https://huggingface.co/datasets/allenai/ultrafeedback_binarized_ cleaned

    work page 2023

  3. [3]

    Allenai. 2024. argilla-dpo-mix-7k. Retrieved June 23, 2025 from https://huggingface.co/datasets/argilla/dpo-mix-7k

    work page 2024

  4. [4]

    Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39, 3/4 (1952), 324–345

    work page 1952

  5. [5]

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335 (2024)

    arXiv 2024

  6. [6]

    Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai, Tianhao Hu, Peixin Cao, Nan Du, and Xiaolong Li. 2023. Adversarial preference optimization: Enhancing your alignment via rm-llm game. arXiv preprint arXiv:2311.08045 (2023)

    arXiv 2023

  7. [7]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017)

    2017

  8. [8]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. UltraFeedback: Boosting Language Models with High-quality Feedback. arXiv:2310.01377 [cs.CL]

    arXiv 2023

Show all 31 references
  1. [9]

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233 (2023)

    2023 arXiv

  2. [10]

    Qingxiu Dong, Li Dong, Xingxing Zhang, Zhifang Sui, and Furu Wei. 2024. Self-boosting large language models with synthetic preference data. arXiv preprint arXiv:2410.06961 (2024)

    2024 arXiv

  3. [11]

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 (2024)

    2024 arXiv

  4. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  5. [13]

    Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691 (2024)

    2024 arXiv

  6. [14]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  7. [15]

    Sangkyu Lee, Sungdong Kim, Ashkan Yousefpour, Minjoon Seo, Kang Min Yoo, and Youngjae Yu. 2024. Aligning Large Language Models by On-Policy Self-Judgment. arXiv preprint arXiv:2402.11253 (2024)

    2024 arXiv

  8. [16]

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024. From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline. arXiv preprint arXiv:2406.11939 (2024)

    2024 arXiv

  9. [17]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. AlpacaEval: An Automatic Evaluator of Instruction-following Models. https://github.com/tatsu-lab/alpaca_eval

    2023

  10. [18]

    Ziniu Li, Tian Xu, and Yang Yu. 2023. Policy optimization in rlhf: The impact of out-of-preference data. arXiv preprint arXiv:2312.10584 (2023)

    2023 arXiv

  11. [19]

    Yong Lin, Skyler Seto, Maartje Ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, Chen Huang, and Tong Zhang. 2024. On the limited generalization capability of the implicit reward model induced by direct preference optimization. arXiv preprint arXiv:240...

    2024 arXiv

  12. [20]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward.Advances in Neural Information Processing Systems 37 (2024), 124198–124235

    2024

  13. [21]

    OpenAI. 2023. GPT-4 Turbo. Retrieved June 25, 2025 from https://help.openai.com/en/articles/8555510-gpt-4-turbo-in-the-openai-api SGPO: Self-Generated Preference Optimization based on Self-Improver 25

    2023

  14. [22]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35...

    2022

  15. [23]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36 (2023), 53728–53741

    2023

  16. [24]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    2023 arXiv

  17. [25]

    Qwen Team. 2024. Qwen2 technical report. arXiv preprint arXiv:2412.15115 (2024)

    2024 arXiv

  18. [26]

    Gonzalez Ion Stoica Tianle Li*, Wei-Lin Chiang*

    Evan Frick Lisa Dunlap Banghua Zhu Joseph E. Gonzalez Ion Stoica Tianle Li*, Wei-Lin Chiang*. 2024. From Live Data to High-Quality Benchmarks: The Arena-Hard Pipeline. https://lmsys.org/blog/2024-04-19-arena-hard/

    2024

  19. [27]

    John Wilder Tukey et al. 1977. Exploratory data analysis. Vol. 2. Springer

    1977

  20. [28]

    Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. 2024. beta-DPO: Direct Preference Optimization with Dynamic beta. Advances in Neural Information Processing Systems 37 (2024), 129944–129966

    2024

  21. [29]

    Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. 2024. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594 (2024)

    2024 arXiv

  22. [30]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  23. [31]

    Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Wontae Nam, Daejin Jo, Kyoung-Woon On, Mark A Hasegawa-Johnson, Sungwoong Kim, and Chang D Yoo. 2024. Tlcr: Token-level continuous reward for fine-grained reinforcement learning from human feedback. arXiv preprint arX...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.