Pith. sign in

REVIEW 5 major objections 6 minor 49 references

Contrastive Weak-to-strong Generalization

T0 review · 5 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that contrastive decoding between a weak model's pre- and post-alignment versions produces higher-quality supervision for training stronger models, yielding consistent gains without reward models or human feedback.

desk verdict Empirically promising, theoretically broken: ConG's pipeline looks useful, but Appendix D's proof is algebraically wrong and the claimed guarantees don't follow. read the letter →

arxiv 2510.07884 v3 pith:IZBE7EIE submitted 2025-10-09 cs.CL cs.AI

Houcheng Jiang , Junfeng Fang , Jiaxin Wu , Tianyu Zhang , Chen Gao , Xiang Wang , Xiangnan He , Yang Deng This is my paper · ORCID
classification cs.CLcs.AI
keywords weak-to-stronggeneralizationcontrastivedecodingimplicitrewarddirectpreferenceoptimizationLLMalignmentself-alignmentinstructionfollowing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that the noise and bias in a weak aligned model's outputs—not the gap in capability—is what limits weak-to-strong generalization, and that this noise can be removed by decoding with the difference between the weak model's pre-alignment and post-alignment versions. The key identification is that the 'implicit reward' used in direct preference optimization, a log-likelihood ratio between aligned and reference policies, is structurally the same quantity that contrastive decoding uses to rerank tokens. On that basis the authors construct ConG, a two-stage recipe: SFT a strong model on contrastively decoded responses from the weak model, then run DPO with those CD responses as chosen and the strong model's own samples as rejected. Across the Qwen2.5 and Llama3 families, including self-alignment and cross-family transfer, they report average gains around 16.5% over the unaligned base on AlpacaEval2 and Arena-Hard, while other weak-to-strong baselines often degrade. The sympathetic reader should care because it suggests that supervision quality, not supervision source, is the bottleneck, and that a weak model's contrast with itself can stand in for expensive reward labels.

What carries the argument

The CD–Implicit Reward Equivalence. Token-level implicit reward is defined as the per-token log-probability gap β[log π_r(y_t|x,y<t)−log π_ref(y_t|x,y<t)]. Contrastive decoding's scoring function for candidate tokens is a weighted sum of exactly this gap and the aligned model's own log-probability, so decoding under CD is exponential tilting of π_r by the implicit reward statistic; α controls the tilt. This equivalence is doing the work: it turns a decoding heuristic into a sample-quality signal, letting ConG treat CD responses as approximate maximizers of implicit reward and thus as chosen data for SFT and DPO.

What would settle it

On a held-out set of prompts, compute the implicit reward r̂ directly for pairs (y_w, y_l) produced by ConG's Stage II using the aligned weak model as the reward proxy; if a nontrivial fraction of pairs have r̂(y_l) ≥ r̂(y_w), the ordering assumption is violated. Alternatively, train π_s_SFT to near-perfect fit on the CD responses; if DPO then stops improving or hurts, the projected-tilt premise is the reason.

Watch

Extended reading notes

Core claim

ConG rests on the CD–Implicit Reward Equivalence: for a token-level factorization, the implicit reward r̂(x,y)=β log π_r(y|x)/π_ref(y|x) is exactly the contrastive term in contrastive decoding, so the CD sampling distribution p_α(y_t) ∝ π_r(y_t) exp((1−α) r̂_t(y_t)) is an exponential tilt of the aligned policy toward high-implicit-reward tokens. Consequently, samples drawn under CD approximately maximize implicit reward and carry a denser preference signal than naive samples from the aligned weak model. The paper then uses the pre-alignment weak model π_w_ref and post-alignment weak model π_w_r as the two poles of CD to generate chosen responses y_w, fine-tunes the strong model on them (ConG

Load-bearing premise

Stage II assumes that the implicit reward of a CD response exceeds that of a standard sample from the SFT-tuned strong model, E[r̂(y_w)] > E[r̂(y_l)]; the paper asserts in Appendix D that KL projection of the CD distribution onto the strong model attenuates the tilting effect enough to guarantee this, but supplies no proof, and if the inequality fails the DPO stage is trained on inverted preference pairs.

Editorial extensions

If this is right

  • Weak-to-strong alignment can be improved with no reward model or human labels: the weak model's own pre/post contrast is the only preference signal needed.
  • Because CD responses concentrate at high implicit reward, the strong model is trained on less noisy supervision, so it can outperform naive weak-to-strong SFT and even the base strong model.
  • Tuning α directly controls the strength of the preference signal; moderate values (≈0.3–0.5) give the best alignment, while α>0.5 over-weights the contrast term and erodes quality.
  • The recipe transfers across model families and scales, and the same machinery works when weak and strong models coincide (self-alignment).
  • Alignment gains do not come at the cost of general ability: downstream benchmarks stay within roughly 1 point of the base model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the equivalence suggests a general principle—any reward expressible as a log-ratio between two model states can be turned into a decoding-time sampler, so ConG-style contrast is a template for denoising weak supervision beyond instruction following, e.g., for reasoning or factuality tasks.
  • Editorial inference: if the exponential-tilt view is right, then the same weak model could serve as an inference-time filter on the strong model's own generations, not just as a training-data generator; Stage II's pairing is a training-time version of that.
  • Editorial inference: a near-perfect SFT fit should close the implicit-reward gap between y_w and y_l, predicting that ConG's gain shrinks as the strong model memorizes the CD data; this is testable by varying ConG-S epochs or data duplication.
  • Editorial inference: a head-to-head ablation against reward-model-ranked selection of weak samples, using the same compute budget, would isolate whether CD's benefit comes from the contrast mechanism itself or merely from selecting higher-quality samples.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes ConG, a two-stage weak-to-strong alignment method. Stage I (ConG-S) uses contrastive decoding (CD) between a post-alignment weak model pi_w_r and its pre-alignment checkpoint pi_w_ref to generate SFT samples for a stronger model. Stage II (ConG) applies DPO with the CD response as y_w and a standard-decoding sample from the SFT'd strong model as y_l. The paper claims a CD-Implicit Reward Equivalence: CD approximately maximizes the DPO implicit reward log(pi_r/pi_ref), and this equivalence is used to justify both training stages. Empirically, ConG reports large improvements over Weak SFT, AuxConf, and WSPO on AlpacaEval2 and Arena-Hard for Qwen2.5 and Llama3 families, including cross-family transfer, with only small degradation on downstream tasks.

Significance. If the empirical results are reproducible, ConG is a practically important contribution: it converts a decoding-time technique into a training-signal denoiser for weak-to-strong generalization, and the reported gains (roughly 16.5 average points over base) are substantially larger than those of prior weak-to-strong baselines. The paper's strengths include re-implemented baselines, reported error bars, cross-family experiments, downstream capability checks, and released code. However, the theoretical foundation is not sound as written: Appendix D contains an algebraic error, and the central reward-gap inequality is false in general. Because both stages of ConG are motivated by that inequality, the manuscript requires substantial revision before the theoretical claims can be accepted.

major comments (5)
  1. [Appendix D, Eq. (13)] The distribution defined in Eq. (7) is p_alpha(y_t) proportional to exp((1-alpha)(log pi_r - log pi_ref) + alpha log pi_r) = pi_r(y_t)^alpha exp((1-alpha) r_hat_t(y_t)), not pi_r(y_t) exp((1-alpha) r_hat_t(y_t)) as claimed in the 'rewritten' form. Consequently, the partition function Z(eta)=Sum_v pi_r(v) exp(eta r_hat_t(v)) is not the correct normalizer, and the convexity of log Z does not imply monotonicity of E_{p_alpha}[r_hat] in alpha. The claim can fail outright: with pi_r=(0.99,0.01), pi_ref=(0.9,0.1), alpha=0, p_alpha is proportional to [1.1,0.1], giving E_{pi_r}[r_hat]=0.071 but E_{p_alpha}[r_hat]=-0.104. Therefore 'smaller alpha increases implicit reward' is not a theorem; the empirical trend in Figure 2 may hold for the tested models but is not established by this proof.
  2. [Section 3.2, Eq. (9)] Even if the per-token tilting were algebraically correct, the claim that CD 'approximately maximizes implicit reward' conflates per-token softmax sampling with sequence-level argmax. Eq. (9) is a greedy per-token argmax, not a characterization of sequences sampled from the CD distribution, and the V_head pruning in Eq. (8) changes the effective distribution. The paper needs a formal statement of what 'approximately maximizes' means, and a proof or a clear label that this is a heuristic. The current wording overstates the theoretical support.
  3. [Section 4, Stage II / Appendix D] The preference ordering used in DPO requires E_{pi_s_SFT}[r_hat] <= E_{p_w_alpha}[r_hat]. Appendix D asserts this by saying that KL projection 'attenuates the tilting effect', but no proof is supplied. Minimizing D_KL(p_w_alpha || pi_s) does not in general preserve or reduce the mean of r_hat; the projection can shift the mean in either direction, and the minimizer need not lie 'between' the tilting distribution and pi_r in reward space. If this inequality fails, Eq. (12) is trained on inverted preferences. Either provide a rigorous argument under explicit, verifiable assumptions or remove the theoretical claim and justify the pair construction empirically.
  4. [Section 5.1 vs. Abstract/Introduction] The abstract and introduction claim that ConG works 'without requiring human feedback or explicit reward modeling.' However, the experimental protocol in Section 5.1 explicitly uses a reward model, ArmoRM-Llama3-8B-v0.1, to rank responses and construct the DPO dataset for the weak model. As stated, the method assumes access to an explicit reward model at the data-construction stage. The paper should clarify whether the 'no explicit reward' claim applies only to ConG's own supervision signal, and should disclose this upstream dependency in the abstract and introduction.
  5. [Section 3, CD-Implicit Reward Equivalence] The so-called CD-Implicit Reward Equivalence is, as written, a rewriting of the definition of F in Eq. (7): the term (1-alpha) log(pi_r/pi_ref) is literally the implicit reward. The non-tautological part of the claim is the monotonicity/concentration prediction, and that is precisely the part that fails in Appendix D. The novelty claim should be recalibrated: the connection is a useful interpretive observation, not a proven equivalence with new content.
minor comments (6)
  1. [Section 1] 'effect paradigm' should read 'effective paradigm'.
  2. [Figure 1(c)] The underline convention in the caption is hard to parse. Please state explicitly which row/color corresponds to Qwen2.5-7B-Instruct and which to Llama3-8B-Instruct.
  3. [Eq. (9)] The argmax should explicitly include the V_head constraint from Eq. (8); as written, the pruning step is omitted from the optimization statement.
  4. [Section 3.3] 'In Section 3.2, we established the theoretical correspondence' is inaccurate because Section 3.2 does not establish the central monotonicity claim. Use 'we now empirically examine'.
  5. [Appendix D] The sentence 'Summing over t extends the result to full responses' assumes token-level independence or a particular factorization; this is not justified and should be discussed, especially since the stage-II argument concerns full-response rewards.
  6. [Table 3] The caption mentions 'standard preference optimization methods' but the table only lists weak-to-strong baselines; either add those baselines or correct the caption.

Circularity Check

2 steps flagged · score 6.0 of 10

CD–Implicit Reward Equivalence is a definitional identity; the α-dependent 'prediction' is read off the decoding objective, so the theoretical foundation is partially circular (score 6).

  1. self definitional [Section 3.2, Eqn. 7 and Eqn. 9 (CD–Implicit Reward Equivalence)]
    "Comparing Eqn. 7 with the token-level implicit reward in Eqn. 4 shows that the contrastive term log πr(yt|x,y<t) πref(yt|x,y<t) is exactly the implicit reward up to a scaling factor (1−α). Therefore, under the contrastive decoding distribution pα, the decoding objective is to find y∗ that approximately maximizes the implicit reward"

    CD is defined in Eqn. 7 with F(yt) = (1−α) log(πr/πref) + α log πr, while the token-level implicit reward is defined in Eqn. 6 as the same log-ratio log(πr/πref). Thus the 'equivalence' and the conclusion that CD decoding maximizes implicit reward are restatements of the definition of the decoding objective: the statistic said to be maximized was placed into F by construction. No independent quantity is derived; the paper presents this definitional identity as a theoretical discovery ('CD–Implicit Reward Equivalence').

  2. self definitional [Section 3.2 (after Eqn. 9); Appendix D, 'Reward Gap between CD and Naive Sampling']
    "Based on this derivation, samples generated with contrastive decoding are expected to have higher implicit rewards than standard decoding, and α offers a direct control over the implicit reward level by adjusting the weight of the contrastive term (see Appendix D for a detailed proof)."

    The claimed 'prediction' that smaller α yields higher implicit reward is read off the definition of F: enlarging (1−α) increases the coefficient on the log-ratio term that is itself the implicit reward. Appendix D attempts an independent derivation by rewriting p_α as π_r exp((1−α)r̂), but that algebra is inconsistent with Eqn. 7: exp((1−α)r̂ + α log π_r) is not π_r exp((1−α)r̂), since the α log π_r term is absorbed incorrectly. So the monotonicity is either a definitional weighting statement or, if taken as a statistical expectation, unproved (and can fail, e.g. π_r=(0.99,0.01), π_ref=(0.9,0.1), α=0 gives E_{p_α}[r̂] < E_{π_r}[r̂]). It does not provide independent validation of the construction.

full rationale

Step 1 is a genuine definitional circularity: the paper's CD–Implicit Reward Equivalence says CD maximizes implicit reward, while the CD logit in Eqn. 7 is defined with the same log-ratio that Eqn. 6 calls implicit reward. Step 2 extends this: the α-trend is read off the weight on that log-ratio in F, and the Appendix D proof attempts to give an independent exponential-tilting derivation but algebraically misstates p_α relative to Eqn. 7, so the 'prediction' is either definitional or unsupported. I do not count the ConG method itself as circular: Stage I and Stage II are trained and evaluated externally on AlpacaEval2 and Arena-Hard, and the empirical gains are not fitted to those benchmarks; no load-bearing self-citation chain was found. The Stage II assumption E_{π_s_SFT}[r̂] ≤ E_{p_w_α}[r̂] (Appendix D, 'Reward Ordering between CD and ConG-S Generations') is asserted without proof; that is an omitted proof and a correctness risk, not a circularity, so it is weighed in the verdict but not scored as a separate circular step. Overall score 6: the central theoretical motivation reduces by construction, while the empirical contribution remains externally benchmarked.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method depends on standard RLHF/DPO assumptions plus two paper-specific assumptions that are not proven: token-level greedy reward maximization approximates sequence-level maximization, and SFT projection attenuates the CD teacher's implicit reward. Free parameters α and β are tuned per model and affect the reported results.

free parameters (4)
  • contrastive coefficient α = 0.5 (Qwen2.5-3B), 0.4 (Llama3.2-3B)
    Tuned per weak model via grid search; the paper does not report a held-out validation split, so α may have been selected using the evaluation benchmarks (Section 5.1, Appendix C.2).
  • DPO preference sharpness β = 0.5 for ConG (vs 0.1 for DPO baselines)
    Manually adjusted upward for ConG to compensate for approximate distribution matching of (y_w, y_l) pairs (Appendix C.2).
  • SFT learning rate = 1e-5
    Selected from {5e-6, 1e-5, 2e-5} (Appendix C.2).
  • DPO learning rate = 6e-7 for ConG
    Selected from {4e-7, 6e-7, 8e-7} (Appendix C.2).
assumptions (4)
  • domain assumption Bradley-Terry preference model governs pairwise human preferences
    Used to derive the DPO reward reparameterization (Section 2.1, Eq. 1). Standard in preference optimization.
  • domain assumption DPO implicit reward rhat = β log(π_θ/π_ref) is a valid proxy for explicit reward
    Section 2.2, Eq. 4. Standard DPO result; its use as a dense per-token reward for generation is an extension.
  • ad hoc to paper Token-level greedy maximization of implicit reward approximates sequence-level reward maximization
    Section 3.2 and Appendix D assume per-token CD decoding approximately maximizes total implicit reward; the paper does not prove this for full autoregressive sequences (Eq. 9).
  • ad hoc to paper SFT KL projection attenuates the CD teacher's implicit reward, so E_{π_s_SFT}[rhat] ≤ E_{p_w_α}[rhat]
    Appendix D, 'Reward Ordering between CD and ConG-S Generations'; unproved and load-bearing for the DPO pairing in Stage II.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Contrastive Weak-to-strong Generalization." pith.science (2026). https://pith.science/paper/IZBE7EIE

@misc{pith2026251007884,
  author       = {Pith},
  title        = {Pith review of: Contrastive Weak-to-strong Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IZBE7EIE}},
  note         = {Machine review of arXiv:2510.07884}
}
read the original abstract

Weak-to-strong generalization provides a promising paradigm for scaling large language models (LLMs) by training stronger models on samples from aligned weaker ones, without requiring human feedback or explicit reward modeling. However, its robustness and generalization are hindered by the noise and biases in weak-model outputs, which limit its applicability in practice. To address this challenge, we leverage implicit rewards, which approximate explicit rewards through log-likelihood ratios, and reveal their structural equivalence with Contrastive Decoding (CD), a decoding strategy shown to reduce noise in LLM generation. Building on this connection, we propose Contrastive Weak-to-Strong Generalization (ConG), a framework that employs contrastive decoding between pre- and post-alignment weak models to generate higher-quality samples. This approach enables more reliable capability transfer, denoising, and improved robustness, substantially mitigating the limitations of traditional weak-to-strong methods. Empirical results across different model families confirm consistent improvements, demonstrating the generality and effectiveness of ConG. Taken together, our findings highlight the potential of ConG to advance weak-to-strong generalization and provide a promising pathway toward AGI.

Figures

Figures reproduced from arXiv: 2510.07884 by the authors.

Figure 1
Figure 1. Overview of our proposed ConG. (a) Paradigm illustration comparing traditional weak-to [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance comparison of contrastive decoding with different contrastive coefficients [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Performance of contrastive weak-to-strong generalization. (a) and (b) Results across [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 19 linked inside Pith

  1. [1]

    Mathqa: Towards interpretable math word problem solving with operation-based formalisms

    Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319, 2019

    arXiv 1905

  2. [2]

    Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield - Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant a...

    arXiv 2021

  3. [3]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  4. [4]

    Weak-to-strong generalization: Eliciting strong capabilities with weak supervision

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeffrey Wu. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In ICML . OpenReview.net, 2024

    2024

  5. [5]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NIPS , pp.\ 4299--4307, 2017

    2017

  6. [6]

    Glass, and Pengcheng He

    Yung - Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In ICLR . OpenReview.net, 2024

    2024

  7. [7]

    Think you have solved question answering? try arc, the AI2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018

    arXiv 2018

  8. [8]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021

    arXiv 2021

Show all 49 references
  1. [9]

    ULTRAFEEDBACK: boosting language models with scaled AI feedback

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. ULTRAFEEDBACK: boosting language models with scaled AI feedback. In ICML . OpenReview.net, 2024

    2024

  2. [10]

    Process reinforcement through implicit rewards

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process...

    2025 arXiv

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024 arXiv

  4. [12]

    Hashimoto

    Yann Dubois, Bal \' a zs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. CoRR, abs/2404.04475, 2024

    2024 arXiv

  5. [13]

    Artificial general intelligence: Concept, state of the art, and future prospects

    Ben Goertzel. Artificial general intelligence: Concept, state of the art, and future prospects. Journal of Artificial General Intelligence, 5 0 (1): 0 1, 2014

    2014

  6. [14]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR . OpenReview.net, 2021

    2021

  7. [15]

    ORPO: monolithic preference optimization without reference model

    Jiwoo Hong, Noah Lee, and James Thorne. ORPO: monolithic preference optimization without reference model. In EMNLP , pp.\ 11170--11189. Association for Computational Linguistics, 2024

    2024

  8. [16]

    AI alignment: A comprehensive survey

    Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O'Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, S...

    2023 arXiv

  9. [17]

    The road to artificial superintelligence: A comprehensive survey of superalignment

    HyunJin Kim, Xiaoyuan Yi, Jing Yao, Jianxun Lian, Muhua Huang, Shitong Duan, JinYeong Bak, and Xing Xie. The road to artificial superintelligence: A comprehensive survey of superalignment. CoRR, abs/2412.16468, 2024

    2024 arXiv

  10. [18]

    RLAIF vs

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. RLAIF vs. RLHF: scaling reinforcement learning from human feedback with AI feedback. In ICML . OpenReview.net, 2024

    2024

  11. [19]

    Scalable agent alignment via reward modeling: a research direction

    Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. CoRR, abs/1811.07871, 2018

    2018 arXiv

  12. [20]

    Gonzalez, and Ion Stoica

    Tianle Li, Wei - Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. CoRR, abs/2406.11939, 2024

    2024 arXiv

  13. [21]

    Contrastive decoding: Open-ended text generation as optimization

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In ACL (1) , pp.\ 12286--12312. Association for Computational Linguistics, 2023

    2023

  14. [22]

    Strong empowered and aligned weak mastered annotation for weak-to-strong generalization

    Yongqi Li, Xin Miao, Mayi Xu, and Tieyun Qian. Strong empowered and aligned weak mastered annotation for weak-to-strong generalization. In AAAI , pp.\ 27437--27445. AAAI Press, 2025

    2025

  15. [23]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In ACL (1) , pp.\ 3214--3252. Association for Computational Linguistics, 2022

    2022

  16. [24]

    MACPO: weak-to-strong alignment via multi-agent contrastive preference optimization

    Yougang Lyu, Lingyong Yan, Zihan Wang, Dawei Yin, Pengjie Ren, Maarten de Rijke, and Zhaochun Ren. MACPO: weak-to-strong alignment via multi-agent contrastive preference optimization. In ICLR . OpenReview.net, 2025

    2025

  17. [25]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. In NeurIPS, 2024

    2024

  18. [26]

    Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D. Manning. An emulator for fine-tuning large language models using small language models. In ICLR . OpenReview.net, 2024

    2024

  19. [27]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023

    2023 arXiv

  20. [28]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leik...

    2022

  21. [29]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2023

    2023

  22. [30]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017

    2017 arXiv

  23. [31]

    Large language model alignment: A survey

    Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey. CoRR, abs/2309.15025, 2023

    2023 arXiv

  24. [32]

    A transfer learning framework for weak to strong generalization

    Seamus Somerstep, Felipe Maia Polo, Moulinath Banerjee, Yaacov Ritov, Mikhail Yurochkin, and Yuekai Sun. A transfer learning framework for weak to strong generalization. In ICLR . OpenReview.net, 2025

    2025

  25. [33]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In NeurIPS, 2020

    2020

  26. [34]

    Eleutherai/lm-evaluation-harness: v0.4.3, July 2024

    Lintang Sutawika, Hailey Schoelkopf, Leo Gao, Baber Abbasi, Stella Biderman, Jonathan Tow, ben fattori, Charles Lovering, farzanehnakhaee70, Jason Phang, Anish Thite, Fazz, Aflah, Niklas Muennighoff, Thomas Wang, sdtblck, nopperl, gakada, tttyuntian, researcher2, Julen Etxaniz...

    2024 doi

  27. [35]

    Interpretable preferences via multi-objective reward modeling and mixture-of-experts

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In EMNLP (Findings) , pp.\ 10582--10592. Association for Computational Linguistics, 2024

    2024

  28. [36]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  29. [37]

    Super(ficial)-alignment: Strong models may deceive weak models in weak-to-strong generalization

    Wenkai Yang, Shiqi Shen, Guangyao Shen, Wei Yao, Yong Liu, Gong Zhi, Yankai Lin, and Ji - Rong Wen. Super(ficial)-alignment: Strong models may deceive weak models in weak-to-strong generalization. In ICLR . OpenReview.net, 2025

    2025

  30. [38]

    Understanding the capabilities and limitations of weak-to-strong generalization

    Wei Yao, Wenkai Yang, Ziqiao Wang, Yankai Lin, and Yong Liu. Understanding the capabilities and limitations of weak-to-strong generalization. CoRR, abs/2502.01458, 2025

    2025 arXiv

  31. [39]

    Free process rewards without process labels

    Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kai Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. CoRR, abs/2412.01981, 2024

    2024 arXiv

  32. [40]

    Hellaswag: Can a machine really finish your sentence? In ACL (1) , pp.\ 4791--4800

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In ACL (1) , pp.\ 4791--4800. Association for Computational Linguistics, 2019

    2019

  33. [41]

    Redefining superalignment: From weak-to-strong alignment to human-ai co-alignment to sustainable symbiotic society

    Yi Zeng, Feifei Zhao, Yuwei Wang, Enmeng Lu, Yaodong Yang, Lei Wang, Chao Liu, Yitao Liang, Dongcheng Zhao, Bing Han, et al. Redefining superalignment: From weak-to-strong alignment to human-ai co-alignment to sustainable symbiotic society. arXiv preprint arXiv:2504.17404, 2025

    2025 arXiv

  34. [42]

    Alleviating hallucinations of large language models through induced hallucinations

    Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi. Alleviating hallucinations of large language models through induced hallucinations. arXiv preprint arXiv:2312.15710, 2023

    2023 arXiv

  35. [43]

    Emulated disalignment: Safety alignment for large language models may backfire! In ACL (1) , pp.\ 15810--15830

    Zhanhui Zhou, Jie Liu, Zhichen Dong, Jiaheng Liu, Chao Yang, Wanli Ouyang, and Yu Qiao. Emulated disalignment: Safety alignment for large language models may backfire! In ACL (1) , pp.\ 15810--15830. Association for Computational Linguistics, 2024 a

    2024

  36. [44]

    Weak-to-strong search: Align large language models via searching over small language models

    Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang, and Yu Qiao. Weak-to-strong search: Align large language models via searching over small language models. In NeurIPS, 2024 b

    2024

  37. [45]

    Weak-to-strong preference optimization: Stealing reward from weak aligned model

    Wenhong Zhu, Zhiwei He, Xiaofeng Wang, Pengfei Liu, and Rui Wang. Weak-to-strong preference optimization: Stealing reward from weak aligned model. In ICLR . OpenReview.net, 2025

    2025

  38. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  39. [47]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  40. [48]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  41. [49]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.