Pith. sign in

REVIEW 4 major objections 7 minor 17 references

A tiny set of on-policy answers that multi-evaluator judges agree on can beat much larger preference datasets for alignment.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

On-policy multi-evaluator consensus filtering retains ~3.5% of candidates and, with KTO on 1.9k labels, improves Mistral-7B preference metrics over stronger data-heavy baselines, with mixed transfer to Llama.

T0 review reviewed 2026-07-31 challenge →

load-bearing objection Clean data-centric recipe with real ablations: ~3.5% on-policy consensus labels work on Mistral, but the 0.08 MT-Bench edge over ORPO is thin and partly test-selected, and Llama does not transfer cleanly. the 4 major comments →

arxiv 2607.25136 v1 pith:ON76YAMC submitted 2026-07-27 cs.AI

Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization

Zhengtao Yao , Runhao Li , Xupeng Chen , Jiayi Cheng , Chenqian Le , Michael Yue , Siheng Wang , Haoyan Xu
show 7 more authors
Yuqi Li Chenhao Wei Zhengdao Li Rongchao Zhang Guang Yang Yidong Wang Junhao Dong
This is my paper
classification cs.AI
keywords preference optimizationdata-centric alignmentmulti-evaluator consensuson-policy generationKTOLLM-as-judgeconfidence gatinginstruction following
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most preference-optimization work swaps the loss while keeping the preference data fixed. This paper asks the opposite question: can a very small, high-confidence set of answers produced by the model itself give a strong enough learning signal? DMAPO generates several responses per prompt from the target model, scores each on helpfulness, factuality, and conciseness with specialized judges, applies a small process-critic penalty for reasoning flaws, and keeps only examples where the judges strongly agree that the answer is clearly good or clearly bad. On Mistral-7B that gate accepts about 3.45% of candidates—1,871 labeled rows. Training KTO on that set improves MT-Bench, length-controlled win rate, instruction-following accuracy, and independent pairwise judgments against stronger baselines trained on far more data. The same aggressive filter behaves similarly on a second backbone, though gains there are smaller. The practical claim is that consensus filtering is a data-efficient path for general-instruction alignment, paid for with extra curation compute and reliance on automated judges.

Core claim

A small on-policy preference set retained by multi-evaluator consensus gating supplies a reliable binary learning signal: KTO trained on 1,871 gated Mistral-7B examples (3.45% of 54,236 candidates) reaches MT-Bench 7.50, 95.5% length-controlled win rate versus a text-davinci-003 reference, 57.3% IFEval accuracy, and substantial independent net win rates over SimPO, ORPO, and REINFORCE++.

What carries the argument

DMAPO’s confidence/variance gate: after rubric-specialized scores for helpfulness, factuality, and conciseness are adjusted by a capped process-critic penalty, a candidate is labeled desirable only if every adjusted score is high and variance is low, undesirable only if every score is low and variance is low, and discarded otherwise.

Load-bearing premise

Agreement among automated judges on helpfulness, factuality, and conciseness really marks answers that improve true preference-aligned behavior, not just answers that match the judges’ shared biases.

What would settle it

Human preference labels on the full candidate pool, or a held-out human pairwise study, showing that the gated desirable/undesirable sets do not rank above discarded answers or that models trained on them lose to same-size random or single-judge baselines under human raters.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Preference work can usefully move compute from longer optimization runs into offline on-policy generation and multi-judge filtering.
  • Binary objectives like KTO can exploit aggressively filtered tails without needing continuous reward scores at train time.
  • The same curated binary set can be reused across preference losses (KTO, DPO, ORPO, SimPO) for a fixed backbone.
  • Changing judge model or rubric can reshuffle which examples pass while leaving downstream scores largely intact, if consensus structure is preserved.
  • Strict multi-dimension gating can trade some response diversity and long-form math style for clearer general-instruction gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If judge consensus is the bottleneck, investing in cheaper or distilled multi-rubric judges may matter more than inventing new preference losses.
  • The method may under-serve tasks that need long chains of reasoning precisely because conciseness is a co-vetoing dimension.
  • On-policy gating per backbone suggests curated sets will not transfer cleanly across model families without re-generation.
  • Publishing per-example gate decisions would let others test whether human raters recover the same tails the judges kept.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The manuscript proposes DMAPO, a data-centric preference-optimization pipeline. A target policy generates four responses per prompt; three Qwen3-8B instances score helpfulness, factuality, and conciseness; a bounded process critic adjusts the scores; and a joint quality/variance gate retains only confident desirable and undesirable examples for KTO. On Mistral-7B-Instruct-v0.2, 1,871 of 54,236 candidates are retained. The reported model reaches 7.50 on MT-Bench, 95.5% length-controlled AE-style win rate, and 57.3% IFEval accuracy, with positive GPT-4o and Claude pairwise margins over SimPO, ORPO, and REINFORCE++. The study includes seven training baselines, size-matched filtering controls, gating and critic sweeps, cross-objective reuse, evaluator/rubric substitutions, an out-of-distribution LMSYS test, and a mixed Llama replication. The authors appropriately label Qwen-family win rate as diagnostic rather than independent evidence.

Significance. If the central result survives a cleaner model-selection protocol, the paper provides useful evidence that a small, on-policy, consensus-filtered preference set can match or exceed much larger fixed preference datasets for general instruction following. Particularly valuable strengths are the size-matched filtering ablations, cross-objective reuse, rule-based IFEval evidence, two non-Qwen pairwise judges, an explicitly untouched LMSYS evaluation, evaluator/rubric substitutions, and candid treatment of evaluator circularity. These features make the claims more falsifiable than a study relying only on the judge family used for curation. The work does not reduce total compute—curation requires 13.8 A100-hours—and the mixed Llama result and Math regression appropriately bound its generality.

major comments (4)
  1. [§4.4, §5.1; Tables 1, 16, 19] MT-Bench is purely a test benchmark, but the manuscript does not specify a separate model-selection protocol. In Appendix H the default 3.45% acceptance rate is also the MT-Bench maximum; in Appendix L the default critic cap 0.15 is also the maximum. Moreover, the 129-prompt “held-out” pairwise set is the 5% validation split, while only LMSYS-Chat is explicitly excluded from all selection. The headline 7.50 versus ORPO's 7.42 and the +23.3 validation-pairwise margin are therefore vulnerable to benchmark-guided selection and noise. Please document the selection chronology, tune on a non-test proxy, and report a frozen untouched evaluation with paired confidence intervals; otherwise frame the comparative claim as exploratory.
  2. [§4.2, Table 1] The key filtering controls do not specify how KTO labels are produced. A uniform Random-1.9k sample from the full 54,236-candidate pool should consist mostly of examples labeled “discard” under Eq. (3); are labels assigned randomly, by another score rule, or is the sample actually drawn from an already labeled/accepted pool? Similarly, Single-Evaluator-1.9k selects the “top” 1,871 candidates and therefore appears to contain only desirable examples, confounding multi-evaluator consensus with negative supervision and class balance. Please give the exact labeling rule and desirable/undesirable counts for every control, and add controls matched on both dataset size and label composition.
  3. [§6.4, Table 6] The claim that changing the evaluator or rubric has “little effect” is important because it addresses shared-judge bias, but Table 6 reports no seed dispersion, confidence intervals, or paired tests. Its MT-Bench differences of 0.02–0.05 are smaller than the roughly 0.05 seed standard deviations in Table 1. Please state whether these runs use the same four seeds, report mean±std or confidence intervals for every metric, and provide uncertainty for the pairwise margins. As written, the robustness conclusion is stronger than the displayed evidence supports.
  4. [§4.4, §5.1, §7] The manuscript states that the AE-style and IFEval prompts do not overlap the curation pool, but gives no corresponding exact or near-duplicate decontamination result for MT-Bench, despite MT-Bench carrying the headline comparison. This should be reported for all evaluation sets. Relatedly, §7 says that rubrics, evaluation prompts, parsers, per-example decisions, and clustering settings are necessary for an audit, but no artifact or repository is supplied. Because small prompt or parser changes can alter the 3.45% gate, these materials and the raw benchmark judgments are needed to audit the central empirical result.
minor comments (7)
  1. [§4.1 / References] Jiang et al. (2024) is the Mixtral paper, not the primary citation for Mistral-7B-Instruct-v0.2. Please cite the appropriate Mistral-7B source and give the exact model revision or checkpoint date.
  2. [Eq. (3), §3.5] Define the variance estimator used in Var(s̃), whether it is a population or sample variance, and the tie behavior at the threshold. With only three evaluator scores, these details can affect gate decisions.
  3. [§3.6, Eq. (4)] Specify the KTO desirable/undesirable loss weights, how z0 is estimated and updated, and the reference-model implementation. Table 15 should also state whether “Margin” is measured on training or validation data and over which label classes.
  4. [§6.2, Table 4] Identify the evaluator used in the one-evaluator row and the pair used in the two-evaluator row, and report class counts. Because the critic changes the selected set by only 172 examples relative to α=0, a count of decision flips by severity would make the component ablation easier to interpret.
  5. [§4.4, Appendix B, Table 2] Identify the judge model and prompt used for the AE-style pairwise evaluation, and report GPT/Claude model versions, generation settings, and whether response order was swapped or randomized in Table 2.
  6. [Appendix K, Table 18] Clarify the compute comparison: baseline dataset construction and reward-model training are excluded, whereas DMAPO generation and scoring are included. The table is useful, but the accounting boundary should be explicit.
  7. [Abstract, §5.3] The abstract should mention the Math-category regression alongside the mixed Llama result so that the scope of “general instructions” is clear upfront.

Circularity Check

1 steps flagged

Minor by-construction diagnostic only: Qwen-gated WR reuses curation labels; primary claims rest on independent judges and rule-based metrics.

specific steps
  1. fitted input called prediction [§4.4 Evaluation; §5.1 / Table 1 Qwen diag. WR; Appendix D Table 12]
    "The Qwen-gated diagnostic WR instead compares log probabilities against the base model on the 129-prompt held-out split. Because its labels come from the evaluator family used for curation, it measures fit to the gated signal rather than independent response quality. ... + DMAPO ... 90.7±1.2"

    Desirable/undesirable labels are defined by the Qwen multi-evaluator gate (Eq. 3). KTO is trained only on those binary labels. The diagnostic WR then asks whether the trained policy assigns higher log-probability to held-out examples labeled desirable by the same family. High diagnostic WR is therefore largely forced by successful fit to the curation signal; the paper correctly labels it non-independent, so it does not underwrite the main external claims.

full rationale

DMAPO is an empirical filtering-and-training pipeline, not a first-principles derivation. The sole clear by-construction loop is the Qwen-gated diagnostic win rate: binary labels are produced by Qwen3-8B rubric evaluators (Eqs. 1–3), KTO is trained on those labels, and the diagnostic then scores log-prob preference on the same evaluator family’s held-out gated set. The paper explicitly demotes this metric to a fit check (“measures fit to the gated signal rather than independent response quality”) and does not treat it as external evidence. Headline results instead use GPT-4o MT-Bench, rule-based IFEval, length-controlled AE-style evaluation, and pairwise GPT-4o/Claude never used in curation, plus size-matched filtering ablations. No self-definitional identity, no uniqueness theorem imported from overlapping authors, no ansatz smuggled via self-citation, and no renaming of a known closed-form result. Test-set hyperparameter peaking on MT-Bench (acceptance-rate and critic-cap sweeps) is a selection-validity concern, not a derivation that reduces prediction to input by equation. Proportionate score is therefore 2: one admitted non-load-bearing diagnostic circularity; central empirical claims remain independently scored.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 2 invented entities

The central empirical claim rests on hand-chosen gating thresholds and critic caps, the assumption that specialized LLM judges are adequate proxies for preference quality, and the on-policy support assumption—not on a parameter-free theory of alignment. Invented machinery is the DMAPO gate assembly rather than a new physical entity. Free parameters dominate what must be re-tuned for a new backbone or judge family.

free parameters (7)
  • desirable threshold τ+ = 7
    Minimum adjusted score on every dimension for desirable label; chosen as 7 on a 1–10 scale and controls acceptance tail.
  • undesirable threshold τ− = 4
    Maximum adjusted score on every dimension for undesirable label; set to 4.
  • inter-evaluator variance cap σ² = 2.5
    Maximum allowed variance across the three adjusted scores; set to 2.5 and is load-bearing for the consensus story (No-Variance-Gate ablation).
  • process-critic penalty cap α_max = 0.15
    Caps total critic subtraction so the critic cannot dominate primary scores; default 0.15 selected via sweep.
  • critic severity penalties p_i = 0.05 / 0.10 / 0.15
    Fixed minor/moderate/severe penalties applied per detected flaw before capping.
  • KTO β and LoRA (r, α) = β=0.1; r=16, α=32
    Preference loss scale and adapter capacity used for all main comparisons; standard but claim-dependent training knobs.
  • candidates per prompt k and sampling (T, top-p) = k=4; T=0.8; top-p=0.95
    On-policy pool size and nucleus settings that define the candidate distribution before gating.
axioms (5)
  • domain assumption Rubric-specialized LLM judges produce scores whose joint high/low consensus is a useful proxy for preference-optimization targets on general instructions.
    Invoked throughout §3.3–3.5 and Limitations; sensitivity study varies judge/rubric but never replaces judges with humans on the full pool.
  • domain assumption On-policy candidates from the base instruct model are the right support for preference feedback (no need to inject off-policy expert demonstrations for the claimed gains).
    §3.2 and §6.1 ‘On-policy relevance’; method cannot add knowledge outside base support by construction.
  • domain assumption Binary KTO on gated des/und labels is an appropriate primary objective for the constructed supervision.
    §3.6; cross-objective table shows pairs also work but main headline uses KTO on all binary rows.
  • domain assumption Standard automatic benchmarks (MT-Bench with GPT-4o, IFEval rules, LC win rate vs text-davinci-003, third-party pairwise) track the intended alignment improvement.
    §4.4–5.2; AE-style is nonstandard vs AlpacaEval 2.0 as the paper notes.
  • ad hoc to paper Arithmetic mean of critic-adjusted dimension scores and joint min/max+variance rules are valid aggregation for keep/discard.
    Eqs. 1–3 define the gate; ablations support pieces of it but the exact functional form is design choice.
invented entities (2)
  • DMAPO consensus gate (multi-evaluator + process critic + confidence/variance ternary labeler) no independent evidence
    purpose: Map on-policy responses to sparse binary des/und/discard supervision before preference optimization.
    Named method assembling known pieces into a specific retain-3.45% procedure; independent evidence is only the paper’s downstream metrics, not an external validated construct.
  • Bounded process-critic severity taxonomy (minor/moderate/severe → fixed penalties) no independent evidence
    purpose: Down-weight responses with detected reasoning/factual flaws without overriding primary dimension scores.
    §3.4 operational definition; cap sweep is internal. No external calibration to human flaw labels is provided.

reviewed 2026-07-31 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization." pith.science (2026). https://pith.science/paper/ON76YAMC

@misc{pith2026260725136,
  author       = {Pith},
  title        = {Pith review of: Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ON76YAMC}},
  note         = {Machine review of arXiv:2607.25136}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Research on preference optimization often varies the training objective while holding the data fixed. We instead ask whether a small, high-confidence set of on-policy responses can provide a reliable learning signal. Our method, DMAPO (Data-centric Multi-evaluator Agreement for Preference Optimization), generates candidate responses from the target policy, evaluates helpfulness, factuality, and conciseness with rubric-specialized evaluators, applies a process-critic correction, and retains only high-consensus desirable or undesirable examples. This procedure accepts 1,871 of 54,236 Mistral-7B candidates (3.45%). KTO trained on this set reaches 7.50 on MT-Bench, 95.5% length-controlled win rate against a text-davinci-003 reference, and 57.3% IFEval prompt accuracy. Independent pairwise evaluation also favors DMAPO over SimPO: GPT-4o yields a net win rate of 23.3 points on 129 held-out prompts and 24.0 points on 200 out-of-distribution LMSYS-Chat prompts; Claude Opus 4.7 yields 24.1 points on the held-out set. Changing the evaluator model or rubric alters the selected examples but has little effect on downstream performance. A second-backbone study yields a similar 3.41% acceptance rate, although its performance gains are more modest. Across these experiments, consensus filtering offers a data-efficient route to preference optimization for general instructions, at the cost of additional curation compute and dependence on evaluator judgments.

Figures

Figures reproduced from arXiv: 2607.25136 by Chenhao Wei, Chenqian Le, Guang Yang, Haoyan Xu, Jiayi Cheng, Junhao Dong, Michael Yue, Rongchao Zhang, Runhao Li, Siheng Wang, Xupeng Chen, Yidong Wang, Yuqi Li, Zhengdao Li, Zhengtao Yao.

Figure 1
Figure 1. Figure 1: Overview of the DMAPO pipeline and primary MT-Bench result. (A) From a pool of 14,272 prompts, the 13,559-prompt training split yields 54,236 on-policy candidates and 1,871 binary-labeled training examples (3.45% acceptance) after rubric-specialized evaluator scoring, a bounded process critic, and confidence/variance gating. (B) On Mistral-7B-Instruct-v0.2, DMAPO reaches an MT-Bench score of 7.50 using 1,8… view at source ↗
Figure 2
Figure 2. Figure 2: Score distribution across 54,236 candidates. The confidence gate retains only the tails. [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training loss curves. DMAPO’s loss drops rapidly to [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 15 linked inside Pith

  1. [4]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  2. [5]

    Orpo: Monolithic preference optimization without reference model

    11 Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11170–11189,

    2024

  3. [7]

    Mixtral of experts.arXiv preprint arXiv:2401.04088,

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088,

    Pith/arXiv arXiv

  4. [8]

    Rlaif vs

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback.arXiv preprint arXiv:2309.00267,

    Pith/arXiv arXiv

  5. [9]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  6. [11]

    Helpsteer2: Open-source dataset for training top- performing reward models.arXiv preprint arXiv:2406.08673,

    Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top- performing reward models.arXiv preprint arXiv:2406.08673,

    Pith/arXiv arXiv

  7. [12]

    Self-play preference optimization for language model alignment.arXiv preprint arXiv:2405.00675,

    Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment.arXiv preprint arXiv:2405.00675,

    Pith/arXiv arXiv

  8. [13]

    Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333,

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333,

    Pith/arXiv arXiv

  9. [14]

    Wizardlm: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244,

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244,

    Pith/arXiv arXiv

  10. [15]

    Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387,

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387,

    Pith/arXiv arXiv

  11. [16]

    Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    Pith/arXiv arXiv

  12. [17]

    Avg. Len

    13 A Full experimental setup Table 9: Complete experimental setup. Component Details Policy backbone (primary) Mistral-7B-Instruct-v0.2 (7.24B) Policy backbone (transfer) Llama-3.1-8B-Instruct (8.03B) Curation evaluator Qwen3-8B (/no_thinkmode) Independent evaluators GPT-4o; Claude Opus 4.7 LoRA rank / alpha 16 / 32 LoRA dropout 0.05 Target modules q, k, ...

    2024

  13. [2017]

    Ultrafeedback: Boosting language models with scaled ai feedback.arXiv preprint arXiv:2310.01377,

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with scaled ai feedback.arXiv preprint arXiv:2310.01377,

    Pith/arXiv arXiv

  14. [2022]

    Reinforce++: Stabilizingcritic-freepolicyoptimization with global advantage normalization.arXiv preprint arXiv:2501.03262,

    JianHu, JasonKleinLiu, HaotianXu, andWeiShen. Reinforce++: Stabilizingcritic-freepolicyoptimization with global advantage normalization.arXiv preprint arXiv:2501.03262,

    Pith/arXiv arXiv

  15. [2023]

    Proximal policy optimiza- tion algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  16. [2024]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Pith/arXiv arXiv

  17. [2025]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475,

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475,

    Pith/arXiv arXiv

This paper was first reviewed by grok-4.5 on July 31, 2026.