Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

Online Preference Alignment for Language Models via Count-based Exploration

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read COPO adds a count-based exploration bonus to online preference optimization, giving large language models a provably efficient way to explore novel responses while aligning to human preferences.

desk verdict COPO is an honest, well-specified empirical paper adding CFN pseudo-counts to iterative DPO, but its theoretical spine is broken: Eq. (15), the cited bridge from UCB to count bonus, fails even in a minimal tabular case, so the O(sqrt(T)) regret claim does not follow as written. read the letter →

arxiv 2501.12735 v3 pith:AVTCXNBD submitted 2025-01-22 cs.LG

Chenjia Bai , Yang Zhang , Shuang Qiu , Qiaosheng Zhang , Kang Xu , Xuelong Li This is my paper · ORCID
classification cs.LG
keywords onlineRLHFcount-basedexplorationpseudo-countdirectpreferenceoptimizationcoin-flippingnetworkLLMalignmentregretboundoptimism
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper addresses a central gap in online RLHF: how an LLM should explore when iteratively collecting preference data. The authors argue that naive online DPO, which samples responses from the current policy and learns from them, does not systematically encourage coverage of the prompt-response space, so the policy may stall or overfit to already-seen regions. They propose COPO, which augments the standard DPO objective with an exploration bonus derived from an upper confidence bound on the reward estimate. In tabular settings, this UCB term reduces to a count-based bonus, and the paper shows that optimizing the resulting optimistic objective yields a suboptimality bound of order $\tilde{O}(\sqrt{T})$. Empirically, COPO improves AlpacaEval 2.0 and MT-Bench scores over online DPO and SELM on Zephyr-7B and Llama-3-8B, with especially large gains when only 20% of the preference data is available. If the claim is right, then count-based exploration is a cheap, practical addition to iterative preference alignment that does not require a learned reward model or expensive density estimation.

What carries the argument

The key machinery is the equivalence, stated as Eq. (15) and attributed to a lemma from Bai et al. (2022a), that in a tabular MDP with one-hot features, the UCB norm $\|\mathbb{E}_{x\sim\rho}[\phi(x, \pi(x))]\|_{(\Sigma_{D_t} + \lambda I)^{-1}}$ equals $\mathbb{E}_{x\sim\rho, y\sim\pi}[1/\sqrt{N_{D_t}(x,y)+\lambda}]$. This identity turns an abstract confidence-set bonus into a simple function of visitation counts. The second load-bearing component is the coin-flipping network (CFN): by training $f_\vartheta$ on pairs $(s_i, c_i)$ with $c_i$ a random vector in $\{-1, +1\}^d$, the squared norm of the average prediction approximates $1/N(s)$, giving a differentiable pseudo-count for continuous, high-dimensional prompt-response spaces.

What would settle it

A direct test: take a small finite prompt-response space where true counts are known, run COPO with the CFN and with a perfect oracle counter, and compare the learned policy's regret against the theoretical $\tilde{O}(\sqrt{T})$ bound. If the CFN version does not track the oracle-counter version, or if the regret grows faster than $\tilde{O}(\sqrt{T})$ when the CFN is used, then the count-based bonus is not faithfully implementing the UCB objective and the central claim fails.

Watch

Extended reading notes

Core claim

The paper claims that optimistic exploration for online preference alignment can be realized by the objective $\max_\pi J_{\mathrm{copo}}(\pi, D_t) = -L_{\mathrm{DPO}}(\pi; D_t) + \alpha \mathbb{E}_{x \sim D_t, y \sim \pi}[1/\sqrt{N_{D_t}(x,y;\vartheta)+\lambda}]$, where $N_{D_t}$ is a pseudo-count estimated by a coin-flipping network. This objective is derived by starting from a linear-reward model with a UCB confidence set, showing that the optimistic value function incurs at most $\tilde{O}(\sqrt{T})$ regret over $T$ online iterations, and then converting the UCB norm to a count-based bonus in the tabular case. The practical algorithm trains a small fully-connected network to predict the average of random Rademacher labels for each prompt-response pair, yielding an unbiased estimate of the inverse visitation count, and adds this bonus to the DPO loss during each iteration of online data collection.

Load-bearing premise

The whole argument rests on the identity that the UCB bonus from the reward-confidence set can be replaced by the inverse square root of visitation counts; this identity comes from an external lemma and is not proved in the paper, and the covariance in the lemma is built from differences between chosen and rejected responses, not from plain state-action visit counts.

Editorial extensions

If this is right

  • If the central claim holds, adding a count-based bonus to DPO is sufficient to make iterative preference alignment explore systematically, without reward-model ensembles, Thompson sampling, or likelihood-ratio penalties.
  • The $\tilde{O}(\sqrt{T})$ regret guarantee implies that a policy that collects one new preference pair per iteration and optimizes the COPO objective converges to the optimal KL-regularized policy at a rate comparable to standard online RL algorithms.
  • The CFN-based pseudo-count is lightweight and architecture-agnostic, so it can be glued onto any DPO-style alignment loss, including KTO as demonstrated in the paper, and onto LoRA-tuned LLMs without full-parameter retraining.
  • The improvement with only 20% of the preference data suggests that count-based exploration matters most when the initial coverage is poor, which is the regime where offline RLHF is known to fail.
  • Because the bonus is inverse in the visitation count, the objective naturally reduces to plain DPO as the policy converges and covers the space, so the exploration term fades in late iterations rather than destabilizing the aligned policy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the pseudo-count could be computed at the token or segment level rather than the whole-response level, which would give a denser exploration signal and may be a natural next step for long-form generation where whole-sequence counts are almost always one.
  • Editorial extension: the same optimistic objective could be used to guide prompt selection as well as response generation; the paper fixes the prompt set and only explores responses, but exploration over prompts is an obvious extension that the conclusion mentions as future work.
  • Editorial extension: the claim implies that any method that estimates inverse visitation counts accurately, such as hash-based counting with good collision controls, should reproduce COPO's gains without need for a learned CFN; a direct test would be swapping CFN for a perfect oracle counter on a small finite action set.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Count-based Online Preference Optimization (COPO), an iterative RLHF method that combines a DPO-style preference objective with a count-based exploration bonus estimated by a coin-flipping network. The authors first present a theoretical motivation in which an optimistic reward with a UCB term is shown to yield an O~(√T) regret bound, then claim that in tabular settings the UCB term equals a count-based exploration bonus (Lemma 2, Eq. (15)). This equivalence is used to justify the COPO objective in Eq. (16). The empirical section reports AlpacaEval 2.0, MT-Bench, and academic-benchmark results on Zephyr-7B and Llama-3-8B, arguing that COPO improves over online DPO and SELM.

Significance. If the theoretical connection were valid, the paper would make a useful contribution: it connects optimistic RLHF to count-based exploration and provides a cheap, easy-to-train pseudo-count module. The CFN implementation is clearly described and the code is released. The empirical results are suggestive, especially the 20%-data adversarial setting in Appendix C.1. However, the central theoretical bridge is false: Eq. (15) does not hold even in the minimal tabular case, and the proof of Theorem 3 contains a further covariance-mismatch gap. The provable-efficiency claim for COPO is therefore not established, and the paper's main novelty is substantially weakened.

major comments (3)
  1. [Section 3.2, Eq. (15)] Lemma 2 is false as stated, and it is load-bearing because Eq. (16) is derived from it. Consider X={x}, Y={y1,y2}, D_t={(x,y1,y2)}, λ=4, and π uniform over the two responses. Then Σ_Dt=(e1-e2)(e1-e2)^T, and the left-hand side of Eq. (15) is ∥(1/2,1/2)∥_{(Σ_Dt+4I)^{-1}} = sqrt(1/8) ≈ 0.354. The right-hand side, with N_Dt(x,y1)=N_Dt(x,y2)=1, is (1/2)(1/sqrt(5))+(1/2)(1/sqrt(5)) = 1/sqrt(5) ≈ 0.447. These are unequal. Even for a deterministic policy the equality fails. The problem is structural: the norm of an expected one-hot feature vector is not the expectation of the norm, and the covariance Σ_Dt is built from chosen-minus-rejected difference vectors, not from per-action visit counts. Since Eq. (15) is the only bridge between the UCB objective Eq. (12) and the count-based objective Eq. (16), Theorem 3's regret bound does not apply to the COPO algorithm as written.
  2. [Appendix A.3, proof of Theorem 3, after Eq. (35)] The regret proof applies Corollary 1 to the expected feature vectors E_{x~ρ}[φ(x,π_t(x))] while the covariance (Σ_Dt+λI) that appears in the quadratic form is assembled from the difference vectors φ(x,y_w)-φ(x,y_l). Corollary 1 bounds a sum of quadratic forms involving the same vectors that are used to update the matrix Λ_t. Here the vectors in the sum and the vectors defining Σ_Dt are different, so the determinant-ratio inequality does not follow. Setting the Corollary-1 vectors to the expected policy features would require Λ_t to be updated with those expected features; setting them to the preference-difference vectors gives the wrong quadratic form. This is a second, independent gap in the proof of the O~(√T) regret bound.
  3. [Section 5.2, Table 1] The empirical claim that COPO 'significantly increases' performance is not statistically grounded: no error bars, confidence intervals, or number of seeds are reported, and several reported trends are nonmonotonic (e.g., COPO Zephyr LC win rate is 26.43, 27.21, then 26.91 across iterations; Llama-3 COPO win rate drops at Iter 3). Since the theoretical guarantee is the primary justification for the algorithm, the empirical section would need much stronger statistical support to carry the paper on its own.
minor comments (4)
  1. [Throughout] There are several typos and notation slips: 'covert' should be 'convert' in Section 3.1; the conclusion writes 'AlphaEval' instead of 'AlpacaEval'; and Eq. (15) writes 'E_{s~ρ}' where 'E_{x~ρ}' is intended.
  2. [Appendix B] The paper states that COPO uses LoRA while the baseline results are obtained with full-parameter tuning. This is a potential confound in the comparison and should be disclosed in the main text; ideally the baselines should use the same training budget and parameterization as COPO.
  3. [Section 3.2, Eq. (17)] The displayed gradient of the count-bonus term omits the partition function Z(r,x) that relates π_φ to π_ref exp(r/β); as written it is not exactly the gradient of Eq. (16). If this expression is meant only as intuition, that should be stated explicitly.
  4. [Section 5.3, Table 3] The ablation of the exploration coefficient α is reported only for Llama-3, while the main experiments use different α values for Zephyr (0.01) and Llama (0.1). Reporting sensitivity on both models, or at least stating why the selected values differ, would make the hyperparameter choice more transparent.

Circularity Check

1 steps flagged · score 5.0 of 10

COPO's central theoretical bridge from UCB optimism to count-based bonus is imported from the authors' own prior work and is not derived in the paper.

  1. self citation load bearing [Section 3.2, Lemma 2 / Eq. (15), and the sentence following it]
    "Lemma 2. (Bai et al., 2022a) ... ∥E_{s∼ρ}[ϕ(x, π(x))]∥_{(Σ_{D_t}+λI)^{-1}} = E_{x∼ρ,y∼π(·|x)}[1/\sqrt{N_{D_t}(x,y)+λ}] ... We refer to Bai et al. (2022a; 2024) for a detailed proof."

    This identity is the only step connecting the provably efficient UCB objective (12) to the implemented COPO objective (16), so it is load-bearing. It is not proved in this paper; it is imported from prior work whose first author overlaps with the present paper. The matrix Σ_{D_t} in Eq. (10) and Eq. (12) is the covariance of preference differences (ϕ(x,y_w)−ϕ(x,y_l))(ϕ(x,y_w)−ϕ(x,y_l))^T, not diag(N_{x,y}). The tabular identity cited from Bai et al. (2022a) holds for covariances of state-action features, so the equality to E[1/\sqrt{N_{D_t}(x,y)+λ}] is an unproved adaptation rather than a derivation. Without Eq. (15), the claim that COPO realizes the optimistic UCB objective and inherits the O~(√T) regret bound of Theorem 3 does not follow from the paper's own equations.

full rationale

The empirical evaluation is self-contained and is not produced by fitted constants: the regret bound contains no fitted parameter, and the benchmark gains are independent of the theory. The DPO part of Eq. (16) is an established reduction from Rafailov et al. (2023). However, the paper's central theoretical claim—that the count-based bonus in Eq. (16) implements the UCB objective in Eq. (12)—rests entirely on Lemma 2 / Eq. (15), which is cited to the authors' own prior work (Bai et al., 2022a; 2024) and not proved or derived in the present paper. Moreover, the cited identity appears mismatched to this setting: Σ_{D_t} is a covariance of chosen-minus-rejected feature differences, not a diagonal matrix of visit counts, so the reduction to 1/\sqrt{N(x,y)+λ} is not a straightforward tabular consequence. In the minimal tabular case X={x}, Y={y1,y2}, D_t={(x,y1,y2)}, λ=4, uniform π, the left side of Eq. (15) is √(1/8)≈0.354 while the right side is 1/√5≈0.447, so the asserted equivalence is numerically false. This is a load-bearing self-citation rather than a circular fit: no benchmark score is defined in terms of the theory, and the CFN pseudo-count is an estimate, not a fitted constant that forces the reported results. The theoretical guarantee for COPO therefore has an unverified link, raising the circularity score to 5—the central derivation depends on an imported self-cited identity, while the empirical content remains independent.

Assumptions & free parameters 2 free parameters · 6 assumptions · 1 invented entities

The central algorithm uses two hand-set constants (α, λ), relies on the Bradley-Terry preference model, imports the tabular UCB-to-count equivalence from the authors' prior work without proof, and assumes a learned pseudo-count generalizes. The theory further assumes linear reward and one preference pair per iteration, which the practical DPO+CFN setup does not satisfy.

free parameters (2)
  • Exploration coefficient α = 0.1 (Llama-3), 0.01 (Zephyr)
    Controls strength of the count-based bonus in Eq. (16); chosen per model and shown in Table 3 to strongly affect results.
  • Count-bonus regularizer λ = 0.01 (practical), 4 (theory)
    Appears in the denominator of the exploration bonus; set by hand in Appendix B and set to 4 in the theoretical regret bound.
assumptions (6)
  • domain assumption Assumption 1: Reward is linear in a known feature map φ(x,y) with bounded parameter θ*.
    Used to derive the confidence set and regret bounds; not verified empirically and not needed by the practical algorithm.
  • domain assumption Preferences follow the Bradley-Terry model: P(y_w ≻ y_l) = σ(r(x,y_w)-r(x,y_l)).
    Standard RLHF assumption used for reward MLE and DPO loss.
  • standard math Lemma 1 (MLE concentration, Zhu et al. 2023) holds for the logistic preference loss.
    Imported from cited prior work to define the confidence set Θ; not reproved.
  • ad hoc to paper Lemma 2 (Bai et al. 2022a): in tabular preference settings the UCB norm equals E[1/sqrt(N+λ)].
    Central bridge from UCB to count-based bonus; cited without proof and applied to a covariance built from chosen-rejected differences, which is not the setting of the original lemma.
  • domain assumption The CFN pseudo-count f_ϑ satisfies (1/d)||f_ϑ(s)||² ≈ 1/N(s) and generalizes to unseen responses.
    Holds for repeated states by Eq. (19), but generalization to the vast LLM response space is assumed, not demonstrated.
  • standard math Elliptical potential lemma (Abbasi-Yadkori et al. 2011) and Corollary 1 for determinant bounds.
    Used in the proof of Theorem 3; the proof applies it to vectors that are not clearly the ones composing the covariance.
invented entities (1)
  • Coin Flipping Network pseudo-count f_ϑ
    purpose: Estimates the visit count of prompt-response pairs for the exploration bonus
    Trained on Rademacher labels from self-generated data; no external validation that its outputs correspond to true counts in the response space.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Online Preference Alignment for Language Models via Count-based Exploration." pith.science (2026). https://pith.science/paper/AVTCXNBD

@misc{pith2026250112735,
  author       = {Pith},
  title        = {Pith review of: Online Preference Alignment for Language Models via Count-based Exploration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AVTCXNBD}},
  note         = {Machine review of arXiv:2501.12735}
}
read the original abstract

Reinforcement Learning from Human Feedback (RLHF) has shown great potential in fine-tuning Large Language Models (LLMs) to align with human preferences. Existing methods perform preference alignment from a fixed dataset, which can be limited in data coverage, and the resulting reward model is hard to generalize in out-of-distribution responses. Thus, online RLHF is more desirable to empower the LLM to explore outside the support of the initial dataset by iteratively collecting the prompt-response pairs. In this paper, we study the fundamental problem in online RLHF, i.e. \emph{how to explore} for LLM. We give a theoretical motivation in linear reward assumption to show that an optimistic reward with an upper confidence bound (UCB) term leads to a provably efficient RLHF policy. Then, we reformulate our objective to direct preference optimization with an exploration term, where the UCB-term can be converted to a count-based exploration bonus. We further propose a practical algorithm, named \emph{Count-based Online Preference Optimization (COPO)}, which leverages a simple coin-flip counting module to estimate the pseudo-count of a prompt-response pair in previously collected data. COPO encourages LLMs to balance exploration and preference optimization in an iterative manner, which enlarges the exploration space and the entire data coverage of iterative LLM policies. We conduct online RLHF experiments on Zephyr and Llama-3 models. The results on instruction-following and standard academic benchmarks show that COPO significantly increases performance.

Figures

Figures reproduced from arXiv: 2501.12735 by the authors.

Figure 1
Figure 1. Exploration rewards in 3 iterations with different [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Representation-Based Exploration for Language Models: From Test-Time to Post-Training

    cs.LG 2025-10 conditional novelty 6.0 of 10

    Representation-based elliptical bonuses improve inference-time and post-training pass@k for LLM reasoning, but the headline AIME result is tainted by validation/test overlap.

  2. Outcome-based Exploration for LLM Reasoning

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Outcome-based exploration bonuses (UCB-Con and Batch) improve pass@1 and pass@32 for LLM math reasoning while slowing diversity collapse, supported by a bandit model with a strong generalization assumption.

  3. CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Adding actor perplexity and multi-head critic variance as intrinsic exploration bonuses improves RLVR math reasoning accuracy by roughly +2 to +3 points on AIME benchmarks.

Reference graph

Works this paper leans on

80 extracted references · 19 canonical work pages · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Improved algorithms for linear stochastic bandits

    Yasin Abbasi-Yadkori, D \'a vid P \'a l, and Csaba Szepesv \'a ri. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011

    2011

  3. [3]

    Reinforcement learning: Theory and algorithms

    Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32: 0 96, 2019

    2019

  4. [4]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gall \'e , Marzieh Fadaee, Julia Kreutzer, Ahmet \"U st \"u n, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024

    arXiv 2024

  5. [5]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp.\ 4447--4455. PMLR, 2024

    2024

  6. [6]

    Dynamic bottleneck for robust self-supervised exploration

    Chenjia Bai, Lingxiao Wang, Lei Han, Animesh Garg, Jianye Hao, Peng Liu, and Zhaoran Wang. Dynamic bottleneck for robust self-supervised exploration. Advances in Neural Information Processing Systems, 34: 0 17007--17020, 2021 a

    work page 2021

  7. [7]

    Principled exploration via optimistic bootstrapping and backward induction

    Chenjia Bai, Lingxiao Wang, Lei Han, Jianye Hao, Animesh Garg, Peng Liu, and Zhaoran Wang. Principled exploration via optimistic bootstrapping and backward induction. In International Conference on Machine Learning, pp.\ 577--587. PMLR, 2021 b

    work page 2021

  8. [8]

    Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning

    Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations, 2022 a

    work page 2022

Show all 80 references
  1. [9]

    Pessimistic value iteration for multi-task data sharing in offline reinforcement learning

    Chenjia Bai, Lingxiao Wang, Jianye Hao, Zhuoran Yang, Bin Zhao, Zhen Wang, and Xuelong Li. Pessimistic value iteration for multi-task data sharing in offline reinforcement learning. Artificial Intelligence, 326: 0 104048, 2024

    2024

  2. [10]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022 b

    2022 arXiv

  3. [11]

    Unifying count-based exploration and intrinsic motivation

    Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016

    2016

  4. [12]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  5. [13]

    Exploration by random network distillation

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018

    2018 arXiv

  6. [14]

    Value-incentivized preference optimization: A unified approach to online and offline rlhf

    Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Value-incentivized preference optimization: A unified approach to online and offline rlhf. arXiv preprint arXiv:2405.19320, 2024

    2024 arXiv

  7. [15]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  8. [16]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  9. [17]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  10. [18]

    Ultrafeedback: Boosting language models with high-quality feedback

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023

    2023 arXiv

  11. [19]

    Rlhf workflow: From reward modeling to online rlhf

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024

    2024 arXiv

  12. [20]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  13. [21]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators

    Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

    2024 arXiv

  14. [22]

    Efficient exploration for llms

    Vikranth Dwaracherla, Seyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy. Efficient exploration for llms. arXiv preprint arXiv:2402.00396, 2024

    2024 arXiv

  15. [23]

    Kto: Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. In International Conference on Machine Learning, 2024 a

    2024

  16. [24]

    Kto: Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024 b

    2024 arXiv

  17. [25]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp.\ 10835--10866. PMLR, 2023

    2023

  18. [26]

    A framework for few-shot language model evaluation, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  19. [27]

    Direct language model alignment from online ai feedback

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024

    2024 arXiv

  20. [28]

    Exploration in deep reinforcement learning: From single-agent to multiagent domain

    Jianye Hao, Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Zhaopeng Meng, Peng Liu, and Zhen Wang. Exploration in deep reinforcement learning: From single-agent to multiagent domain. IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  21. [29]

    VIME: variational information maximizing exploration

    Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: variational information maximizing exploration. In Advances in Neural Information Processing Systems, 2016

    2016

  22. [30]

    Lo RA : Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  23. [31]

    Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback

    Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. In Advances in neural information processing sy...

    2024

  24. [32]

    Llm-blender: Ensembling large language models with pairwise ranking and generative fusion

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561, 2023

    2023 arXiv

  25. [33]

    Provably efficient reinforcement learning with linear function approximation

    Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on learning theory, pp.\ 2137--2143. PMLR, 2020

    2020

  26. [34]

    Kearns and Satinder P

    Michael J. Kearns and Satinder P. Singh. Near-optimal reinforcement learning in polynomial time. Machine Learning, 49 0 (2-3): 0 209--232, 2002

    2002

  27. [35]

    Rewardbench: Evaluating reward models for language modeling

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024

    2024 arXiv

  28. [36]

    Bandit algorithms

    Tor Lattimore and Csaba Szepesv \'a ri. Bandit algorithms. Cambridge University Press, 2020

    2020

  29. [37]

    A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity

    Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. In International Conference on Machine Learning, 2024 a

    2024

  30. [38]

    Aligning large language models by on-policy self-judgment

    Sangkyu Lee, Sungdong Kim, Ashkan Yousefpour, Minjoon Seo, Kang Min Yoo, and Youngjae Yu. Aligning large language models by on-policy self-judgment. arXiv preprint arXiv:2402.11253, 2024 b

    2024 arXiv

  31. [39]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021

    2021 arXiv

  32. [40]

    Flipping coins to estimate pseudocounts for exploration in reinforcement learning

    Sam Lobel, Akhil Bagaria, and George Konidaris. Flipping coins to estimate pseudocounts for exploration in reinforcement learning. In International Conference on Machine Learning, pp.\ 22594--22613. PMLR, 2023

    2023

  33. [41]

    The sample complexity of exploration in the multi-armed bandit problem

    Shie Mannor and John N Tsitsiklis. The sample complexity of exploration in the multi-armed bandit problem. Journal of Machine Learning Research, 5 0 (Jun): 0 623--648, 2004

    2004

  34. [42]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024

    2024 arXiv

  35. [43]

    Introducing meta llama 3: The most capable openly available llm to date

    AI Meta. Introducing meta llama 3: The most capable openly available llm to date. Meta AI, 2024

    2024

  36. [44]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  37. [45]

    Nash learning from human feedback

    R \'e mi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:2312.00886, 2023

    2023 arXiv

  38. [46]

    Count-based exploration with neural density models

    Georg Ostrovski, Marc G Bellemare, A \"a ron Oord, and R \'e mi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp.\ 2721--2730. PMLR, 2017

    2017

  39. [47]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  40. [48]

    Eq-bench: An emotional intelligence benchmark for large language models

    Samuel J Paech. Eq-bench: An emotional intelligence benchmark for large language models. arXiv preprint arXiv:2312.06281, 2023

    2023 arXiv

  41. [49]

    Curiosity-driven exploration by self-supervised prediction

    Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp.\ 2778--2787. PMLR, 2017

    2017

  42. [50]

    Contrastive ucb: Provably efficient contrastive self-supervised learning in online reinforcement learning

    Shuang Qiu, Lingxiao Wang, Chenjia Bai, Zhuoran Yang, and Zhaoran Wang. Contrastive ucb: Provably efficient contrastive self-supervised learning in online reinforcement learning. In International Conference on Machine Learning, pp.\ 18168--18210. PMLR, 2022

  43. [51]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2023

    2023

  44. [52]

    Scaling laws for reward model overoptimization in direct alignment algorithms

    Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. arXiv preprint arXiv:2406.02900, 2024 a

    2024 arXiv

  45. [53]

    From r to q*: Your language model is secretly a q-function

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q*: Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358, 2024 b

    2024 arXiv

  46. [54]

    Optimistic exploration even with a pessimistic initialisation

    Tabish Rashid, Bei Peng, Wendelin Boehmer, and Shimon Whiteson. Optimistic exploration even with a pessimistic initialisation. In International Conference on Learning Representations, 2020

    2020

  47. [55]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  48. [56]

    A long way to go: Investigating length correlations in rlhf

    Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023

    2023 arXiv

  49. [57]

    D2po: Discriminator-guided dpo with response evaluation models

    Prasann Singhal, Nathan Lambert, Scott Niekum, Tanya Goyal, and Greg Durrett. D2po: Discriminator-guided dpo with response evaluation models. arXiv preprint arXiv:2405.01511, 2024

    2024 arXiv

  50. [58]

    An analysis of model-based interval estimation for markov decision processes

    Alexander L Strehl and Michael L Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74 0 (8): 0 1309--1331, 2008

    2008

  51. [59]

    A minimaximalist approach to reinforcement learning from human feedback

    Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.04056, 2024

    2024 arXiv

  52. [60]

    \# exploration: A study of count-based exploration for deep reinforcement learning

    Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. \# exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017

    2017

  53. [61]

    Generalized preference optimization: A unified approach to offline alignment

    Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, R \'e mi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo \'A vila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2...

    2024 arXiv

  54. [62]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  55. [63]

    Zephyr: Direct distillation of lm alignment

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Cl \'e mentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023

    arXiv

  56. [64]

    Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf

    Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf. arXiv preprint arXiv:2405.21046, 2024

    2024 arXiv

  57. [65]

    Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning, 2024

    2024

  58. [66]

    Is dpo superior to ppo for llm alignment? a comprehensive study

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study. In International Conference on Machine Learning, 2024

    2024

  59. [67]

    Rorl: Robust offline reinforcement learning via conservative smoothing

    Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. Advances in neural information processing systems, 35: 0 23851--23866, 2022

    2022

  60. [68]

    Yi: Open foundation models by 01

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024

    2024 arXiv

  61. [69]

    Regularized conditional diffusion model for multi-task preference alignment

    Xudong Yu, Chenjia Bai, Haoran He, Changhong Wang, and Xuelong Li. Regularized conditional diffusion model for multi-task preference alignment. arXiv preprint arXiv:2404.04920, 2024

    2024 arXiv

  62. [70]

    Self-rewarding language models

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024 a

    2024 arXiv

  63. [71]

    Preference aligned diffusion planner for quadrupedal locomotion control

    Xinyi Yuan, Zhiwei Shang, Zifan Wang, Chenkai Wang, Zhao Shan, Zhenchao Qi, Meixin Zhu, Chenjia Bai, and Xuelong Li. Preference aligned diffusion planner for quadrupedal locomotion control. arXiv preprint arXiv:2410.13586, 2024 b

    arXiv

  64. [72]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  65. [73]

    Self-exploring language models: Active preference elicitation for online alignment

    Shenao Zhang, Donghan Yu, Hiteshi Sharma, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhaoran Wang. Self-exploring language models: Active preference elicitation for online alignment. arXiv preprint arXiv:2405.19332, 2024

    2024 arXiv

  66. [74]

    Slic-hf: Sequence likelihood calibration with human feedback

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023

    2023 arXiv

  67. [75]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2023

    2023

  68. [76]

    Principled reinforcement learning with human feedback from pairwise or k-wise comparisons

    Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pp.\ 43037--43067. PMLR, 2023

    2023

  69. [77]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

  70. [78]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  71. [79]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  72. [80]

    a&!Ï "l=-BpEMUs J5ū ?bvtCy O (^rڜH - Y`J* aH/'V t@Ys ;ӓj(u B FBa 竑 6 ^mN OB Y>X 5 >D Q=h .+' A Ի, _|k P(qd/T) nV P C/ۿA+ڽ.W b< ydQx> gQ`U Ԡ < a

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2000

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.