Pith. sign in

REVIEW 4 major objections 5 minor 37 references

Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A small aligned model writes the opening of a large base model's response, and the large model then continues by itself; the paper reports this decoding-time scheme beats all tested low-resource alignment baselines and leaves math and…

desk verdict Missing control of the draft model alone undercuts the headline 'outperforms all baselines,' but the method is clear, novel, and worth referee time. read the letter →

arxiv 2506.07434 v1 pith:MARYGN2X submitted 2025-06-09 cs.CL cs.AI

Feifan Song , Shaohang Wei , Wen Luo , Yuxuan Fan , Tianyu Liu , Guoyin Wang , Houfeng Wang This is my paper
classification cs.CLcs.AI
keywords weak-to-strongdecodingpreferencealignmentdecoding-timelow-resourcetaxdraftmodelauto-switchmechanismGenerAlign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central proposal is that the hardest part of aligning a large language model is the first few tokens: once a response begins the right way, the large model's own distribution makes continuing in an aligned style easy. To exploit this, it introduces Weak-to-Strong Decoding (WSD), in which a small model fine-tuned on preference data drafts the beginning and the large base model takes over automatically once its window-smoothed confidence in the draft crosses a threshold. The authors build a 3-billion-parameter draft model, Pilot-3B, trained on a new dataset, GenerAlign, and report that WSD beats every low-resource alignment baseline they compare on five preference benchmarks, across three base models up to 70 billion parameters, while GSM8K and HumanEval scores stay flat or improve. The practical consequence would be that preference alignment can be had without fine-tuning the large model at all, only a small one.

What carries the argument

The load-bearing mechanism is the auto-switch. After the draft model $m$ produces a prefix $y_m$, the base model $M$ computes its own probability of that prefix, smoothed over a window of size $w$ by a geometric mean of per-token probabilities, and stops the draft at the first token index $k$ where $$P'_M = \left(\prod_{j=i-w}^{i-1} P_M($y^{{m}}$_{j+1}\mid $y^{{m}}$_{\le j}, x)\right)^{1/w} \ge \gamma.$$ From that point $M$ generates the rest itself. The paper's supporting observation is that as an aligned prefix grows, the base model's perplexity on the next tokens drops sharply, so the switch point marks where the base model has, in the paper's view, drifted into the aligned region of its distribution.

What would settle it

Run WSD on the same prompts with the auto-switch replaced by a fixed switch at several token positions (for instance 20, 50, 100, 200), and compare preference-alignment win-rates. If no fixed position is worse than the auto-switch, or if the smoothed-confidence criterion does not track response quality across prompts, the claim that the confidence signal carries the alignment gain is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that decoding-time collaboration between a weak aligned model and a strong base model is enough to produce responses that are more aligned than any tested low-resource baseline, and that the collaboration can be controlled by a simple confidence rule. The base model scores the draft model's prefix with its own token probabilities, smoothed over a window, and switches to itself at the first token where that smoothed confidence reaches threshold $\gamma$ (default $0.8$, window $w=6$). The paper presents this as evidence that a well-aligned beginning reduces the base model's perplexity on later tokens, so the role of the draft model is to put the base model on the right path rather than to supply the whole answer. It reports that this holds for Llama-3-70B, Llama-3.1-70B, and Gemma-2-27B, that WSD dominates the baselines on HH-RLHF, TruthfulQA, AlpacaEval 2, ArenaHard, and MT-Bench, and that downstream GSM8K and HumanEval performance does not degrade.

Load-bearing premise

The load-bearing premise is that the base model's window-smoothed probability of the draft tokens reliably indicates that the base model has actually absorbed the aligned style, so the switch happens at a point where the base model will keep producing good content on its own; the paper's evidence for this is post hoc categorization of where switches occurred, not a predictive validation of the criterion.

Editorial extensions

If this is right

  • WSD gives a low-resource alignment recipe: fine-tune a small model once, then attach it to any base model of the same family; the paper shows the same Pilot-3B draft model working with three different large base models.
  • Because the large model's parameters are never changed, the paper's results imply that the alignment tax can be decoupled from alignment itself: GSM8K and HumanEval scores do not drop under WSD, unlike fine-tuning.
  • Tightening the switch condition (larger window $w$, higher threshold $\gamma$, longer max draft length) improves AlpacaEval 2 scores up to a point, while GSM8K stays nearly flat, so the switch policy can be tuned for alignment without a downstream penalty.
  • The switch happens early for GSM8K and later for AlpacaEval 2, which the paper reads as evidence that for downstream tasks the draft only needs to point the base model at the correct solving path, while for open-ended alignment more drafted structure is useful.
  • WSD's per-token time overhead is the lowest among the compared methods (as low as 0.84x on Llama-3-70B), making it a practical substitute for rejection-sampling and reward-guided decoding baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not pursue is to treat the auto-switch as a general prefix-steering control: the same small-drafts/large-finishes design could steer base models toward other target properties, such as a specified style, factuality standard, or safety policy, whenever a small model can supply a good opening.
  • The paper's post-hoc analysis of switch points suggests a testable hypothesis—that switching is essentially the base model adopting the draft's helpful style—which, if confirmed, would allow a cheaper style or format detector to replace the smoothed-confidence threshold.
  • The authors note that they did not implement WSD end-to-end with efficient inference engines; whether the reported time ratios survive a production integration is an open question the paper leaves for future work.
  • Because WSD leaves the base model untouched, it could be composed with later fine-tuning of the base model or with a faster inference scheme, without conflicting mechanisms.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Weak-to-Strong Decoding (WSD), a decoding-time method in which a small aligned model (Pilot-3B, DPO-trained on the new GenerAlign dataset) drafts the beginning of a response and a large base model continues after an automatic switch controlled by the base model's window-smoothed confidence in the draft tokens (Eq. 5). The authors evaluate WSD with Llama-3-70B, Llama-3.1-70B, and Gemma-2-27B on HH-RLHF, TruthfulQA, AlpacaEval 2, ArenaHard, and MT-Bench against several low-resource baselines, and on GSM8K and HumanEval to test for alignment tax. They also ablate hyperparameters, draft-model choice, model scale, and time efficiency, and analyze switch-point behavior.

Significance. WSD is a simple and plausible way to use a small aligned model to steer a large base model at decoding time, with the practical advantage of avoiding large-scale fine-tuning. The paper's empirical scope is broad: three base models, five preference benchmarks, and several low-resource baselines (URIAL, CARDS, ARGS, Best-of-N, Aligner, RE-Control), and the artifact release (code, dataset, Pilot-3B) supports reproducibility. If the central claims survive the missing-control and statistical-rigor concerns below, the method would be a useful addition to the low-resource alignment toolbox.

major comments (4)
  1. [Section 3.4, Table 1; Figure 4; Section 3.3] The evaluation omits the most direct control: Pilot-3B generating the full response by itself. Because the default maximum draft length (512 tokens, Section 3.3) equals the total response-length cap for HH-RLHF, TruthfulQA, and MT-Bench, WSD can in principle return a response that is entirely Pilot-3B's output whenever the switch condition in Eq. (5) does not fire. Figure 4 reports that Pilot-3B alone achieves an AlpacaEval 2 LC win-rate of 43.79, whereas WSD with Llama-3-70B scores 20.13 in Table 1. Without a full-response Pilot-3B condition on all five benchmarks, the paper cannot establish that the weak-to-strong continuation, rather than the draft model itself, is responsible for the reported wins over URIAL, CARDS, ARGS, and the other baselines.
  2. [Section 3.5, Table 2] The no-alignment-tax conclusion is based on two downstream tasks and on point estimates without any stochasticity information. GSM8K for Llama-3-70B is identical across Base and WSD (82.18), and the remaining deltas (e.g., +0.15 GSM8K on Llama-3.1-70B, +1.83 HumanEval on Llama-3-70B) are small enough that they could be run-to-run variation in sampling-based decoding. No seeds, repeated runs, or confidence intervals are reported anywhere in the paper, so the abstract-level claim that WSD avoids the alignment tax is not supported at the precision stated.
  3. [Section 2.3, Eq. (5); Section 3.10] The auto-switch criterion is not validated predictively. Section 3.10 categorizes where switches occur after the fact, but it does not show that the smoothed-confidence rule is better than a fixed-length draft or an oracle switch point, nor does it measure the marginal contribution of the base continuation once the draft ends. Without such comparisons, the paper's mechanistic claim that the confidence signal in Eq. (5), rather than simply the presence of a well-aligned prefix, drives the gains is unsupported.
  4. [Tables 1 and 3] Several supporting claims are made without statistical support. For example, Table 3 reports time ratios that differ by a few hundredths (WSD 0.84/1.03/0.99 versus URIAL 0.90/1.03/1.00), and the text states that WSD has the lowest average overhead, but no variance or test accompanies these numbers. Since the time-ratio measurements involve only 20 prompts, these near-ties should not be presented as definitive without additional runs or error estimates.
minor comments (5)
  1. [Section 2.4] The text says the Pilot-3B preliminary results are shown in Figure 3, but Figure 3 is the WSD framework illustration; the alignment-tax bar chart is Figure 4. Please fix the cross-reference.
  2. [Section 2.1, Figure 2] The preliminary experiment reports only qualitative statements about perplexity ranks and reward values. Reporting the actual rank statistics and reward differences would make the motivating claim more precise and reproducible.
  3. [Section 3.1, Appendix B] The evaluation section does not specify the decoding hyperparameters (temperature, top-p, number of samples) used for the base model, WSD, and each baseline, nor the exact judge prompts for the reward-model and LLM-as-judge evaluations. These details are needed for reproducibility.
  4. [Section 3.10] The term 'stylization' is introduced to explain why the base model becomes confident, but it is not defined or measured. Please clarify what evidence in the categorized cases supports this interpretation.
  5. [Figure 6] The caption contains the misspelling 'Cummulative'; it should be 'Cumulative'. There are also similar typographical issues elsewhere (e.g., 'mathmatical' in Section 3.5).

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: reported gains rest on external benchmark evaluations, with only a mild non-load-bearing self-reference in dataset construction.

full rationale

WSD is defined by Eqs. 1-2 as a draft prefix from a small model followed by a base-model continuation, with the switch point fixed by the base model's smoothed confidence in Eqs. 4-5; none of these equations takes the reported evaluation scores as inputs. Pilot-3B is trained on GenerAlign via DPO, and GenerAlign also supplies the motivating preliminary experiment, but that experiment only motivates the design and does not enter the measured HH-RLHF, TruthfulQA, AlpacaEval 2, ArenaHard, MT-Bench, GSM8K, or HumanEval numbers. The window size and threshold are fixed hyperparameters, not fitted values renamed as predictions. The self-citation in Appendix A for prompt-diversity selection is an implementation detail rather than a load-bearing premise or uniqueness claim. The absence of a Pilot-3B-alone control is an experimental-validity concern about attribution, not a circular reduction of a result to its input. No circular step is exhibited.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The WSD framework introduces only hyperparameters and a trained draft model, not new theoretical entities. The main assumptions are about the behavior of LLMs and the validity of automatic evaluators.

free parameters (4)
  • window size w = 6
    Set by default in Section 3.3; not derived. Ablation (Section 3.6, Figure 5) shows larger w improves AlpacaEval 2 performance, so the choice affects the reported gains.
  • threshold gamma = 0.8
    Set by default in Section 3.3; not derived. Ablation (Section 3.6, Figure 5) shows larger gamma improves AlpacaEval 2 performance, so this choice affects the headline results.
  • max draft length = 512
    Set by default in Section 3.3; ablation (Section 3.6, Figure 5) shows larger values help within a range, so the choice is not neutral.
  • decoding temperature
    The paper does not report the sampling temperature for either the draft or base model; results may depend on this unreported setting.
assumptions (3)
  • domain assumption A well-aligned prefix substantially reduces the difficulty of continuing aligned generation for base LLMs.
    This is the core 'well begun is half done' hypothesis, supported by the preliminary experiment in Section 2.1 but not proven; if false, WSD's mechanism loses its basis.
  • domain assumption The base model's smoothed next-token probability of draft tokens is a reliable proxy for when to switch to the base model.
    Section 2.3 defines the auto-switch based on smoothed confidence (Eq. 5). No theoretical justification; empirical analysis in Section 3.10 is post hoc.
  • domain assumption Reward models and LLM judges used for evaluation reflect human preference.
    HH-RLHF and TruthfulQA are scored by reward/judge models (Section 3.1), and AlpacaEval/ArenaHard/MT-Bench use GPT-4-turbo. These are proxy metrics, not ground truth human judgments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding." pith.science (2026). https://pith.science/paper/MARYGN2X

@misc{pith2026250607434,
  author       = {Pith},
  title        = {Pith review of: Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MARYGN2X}},
  note         = {Machine review of arXiv:2506.07434}
}
read the original abstract

Large Language Models (LLMs) require alignment with human preferences to avoid generating offensive, false, or meaningless content. Recently, low-resource methods for LLM alignment have been popular, while still facing challenges in obtaining both high-quality and aligned content. Motivated by the observation that the difficulty of generating aligned responses is concentrated at the beginning of decoding, we propose a novel framework, Weak-to-Strong Decoding (WSD), to enhance the alignment ability of base models by the guidance of a small aligned model. The small model first drafts well-aligned beginnings, followed by the large base model to continue the rest, controlled by a well-designed auto-switch mechanism. We also collect a new dataset, GenerAlign, to fine-tune a small-sized Pilot-3B as the draft model, which effectively enhances different base models under the WSD framework to outperform all baseline methods, while avoiding degradation on downstream tasks, termed as the alignment tax. Extensive experiments are further conducted to examine the impact of different settings and time efficiency, as well as analyses on the intrinsic mechanisms of WSD in depth.

Figures

Figures reproduced from arXiv: 2506.07434 by the authors.

Figure 1
Figure 1. Illustration of our motivation. (1) It is hard for [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Results of the preliminary experiment. 2 Methodology In this section, we propose our method, WSD, which aims to enhance the performance of a base LLM in a low-resource manner. 2.1 Well Begun is Half Done Given a prompt x, the base LLM is often unable to directly generate a corresponding response y that is well-aligned with human preference. The dif￾ficulty lies in the fact that the base LLM models the whole text dis… view at source ↗
Figure 3
Figure 3. The illustration of the WSD framework. Given a user query in (a), the draft model [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The illustrations of alignment tax in fine [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Results of ablation study on Llama-3-70B. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Cummulative distribution of AlpacaEval 2 [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Evaluating the effect of different draft models. (a) Llama-3-70B; (b) Llama-3.1-70B; (c) Gemma-2-27B. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Scalability of WSD on models of different [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Case study. (a) Model switching happens after the partial answer is drafted. (b) Model switching at the [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Results of ablation study on Llama-3.1-70B. [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Results of ablation study on Gemma-2-27B. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Additional results. In Section 3.6, we take Llama-3-70B as the example to show the influence of different hyperparameter settings, as well as change in the number of accepted draft with the increase of decoding steps. In this section, we provide such results of the ab…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 9 canonical work pages

  1. [1]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. https://arxiv.org/abs/2204.05862 Training a helpful and harmless assistant with reinforcement learning from human feedback . arXiv preprint arXiv:2204.05862

    arXiv 2022

  2. [2]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  3. [3]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6

    2023

  4. [4]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  5. [5]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. https://openreview.net/forum?id=BOorDpKHiJ ULTRAFEEDBACK : Boosting language models with scaled AI feedback . In Forty-first International Conference on Machine Learning

    work page 2024

  6. [6]

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.183 Enhancing chat language models by scaling high-quality instructional conversations . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3029--3051, Singapore...

    doi:10.18653/v1/2023.emnlp-main.183 2023

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  8. [8]

    Bofei Gao, Feifan Song, Yibo Miao, Zefan Cai, Zhe Yang, Liang Chen, Helan Hu, Runxin Xu, Qingxiu Dong, Ce Zheng, et al. 2024. Towards a unified view of preference learning for large language models: A survey. arXiv preprint arXiv:2409.02795

    arXiv 2024

Show all 37 references
  1. [9]

    Gemma Team , Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    2024 arXiv

  2. [10]

    Shengding Hu, Yifan Luo, Huadong Wang, Xingyi Cheng, Zhiyuan Liu, and Maosong Sun. 2023. https://doi.org/10.18653/v1/2023.acl-long.309 Won`t get fooled again: Answering questions with false premises . In Proceedings of the 61st Annual Meeting of the Association for Computation...

    2023 doi

  3. [11]

    Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Tianyi Qiu, Juntao Dai, and Yaodong Yang. 2024. https://openreview.net/forum?id=kq166jACVP Aligner: Efficient alignment by learning to correct . In The Thirty-eighth Annual Conference on Neural Inform...

    2024

  4. [12]

    Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. 2024. https://openreview.net/forum?id=shgx0eqdw6 ARGS : Alignment as reward-guided search . In The Twelfth International Conference on Learning Representations

    2024

  5. [13]

    Lingkai Kong, Haorui Wang, Wenhao Mu, Yuanqi Du, Yuchen Zhuang, Yifei Zhou, Yue Song, Rongzhi Zhang, Kai Wang, and Chao Zhang. 2024. https://openreview.net/forum?id=yTTomSJsSW Aligning large language models with representation editing: A control perspective . In The Thirty-eig...

    2024

  6. [14]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  7. [15]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274--19286. PMLR

    2023

  8. [16]

    Bolian Li, Yifan Wang, Ananth Grama, and Ruqi Zhang. 2024 a . Cascade reward sampling for efficient decoding-time alignment. arXiv preprint arXiv:2406.16306

    2024 arXiv

  9. [17]

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024 b . From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939

    2024 arXiv

  10. [18]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval

    2023

  11. [19]

    Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. 2024 c . https://openreview.net/forum?id=pETSfWMUzy RAIN : Your language models can align themselves without finetuning . In The Twelfth International Conference on Learning Representations

    2024

  12. [20]

    Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. 2024. https://openreview.net/forum?id=wxJ0eXwwda The unlocking spell on base LLM s: Rethinking alignment via in-context learning . In The Twelfth...

    2024

  13. [21]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://doi.org/10.18653/v1/2022.acl-long.229 T ruthful QA : Measuring how models mimic human falsehoods . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    2022 doi

  14. [22]

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451

    2024 arXiv

  15. [23]

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. https://proceedings.mlr.press/v202/longpre23a.html The flan collection: Designing data and methods for effective instruction tuning . I...

    2023

  16. [24]

    Yougang Lyu, Lingyong Yan, Zihan Wang, Dawei Yin, Pengjie Ren, Maarten de Rijke, and Zhaochun Ren. 2024. Macpo: Weak-to-strong alignment via multi-agent contrastive preference optimization. arXiv preprint arXiv:2410.07672

    2024 arXiv

  17. [25]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. https://openreview.net/forum?id=3Tzcot1LKb Sim PO : Simple preference optimization with a reference-free reward . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  18. [26]

    Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. 2024. https://openreview.net/forum?id=bVIcZb7Qa0 Controlled decoding from language mode...

    2024

  19. [27]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  20. [28]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  21. [29]

    Feifan Song, Yuxuan Fan, Xin Zhang, Peiyi Wang, and Houfeng Wang. 2025. https://aclanthology.org/2025.naacl-long.8/ Instantly learning preference alignment via in-context DPO . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for ...

    2025

  22. [30]

    Feifan Song, Bowen Yu, Hao Lang, Haiyang Yu, Fei Huang, Houfeng Wang, and Yongbin Li. 2024. https://aclanthology.org/2024.lrec-main.1251/ Scaling data diversity for fine-tuning language models in human alignment . In Proceedings of the 2024 Joint International Conference on Co...

    2024

  23. [31]

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024 a . Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845

    2024 arXiv

  24. [32]

    Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. 2024 b . https://arxiv.org/abs/2410.01257 Helpsteer2-preference: Complementing ratings with preferences . Preprint, arXiv:2410.01257

    2024 arXiv

  25. [33]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  26. [34]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024 a . Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36

    2024

  27. [35]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024 b . https://doi.org/10.18653/v1/2024.acl-demos.38 L lama F actory: Unified efficient fine-tuning of 100+ language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational ...

    2024 doi

  28. [36]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  29. [37]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.