Pith. sign in

REVIEW 4 major objections 7 minor 61 references

Thompson Sampling in Online RLHF with General Function Approximation

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Model-free Thompson sampling for online RLHF is claimed to achieve O(sqrt(T)) regret when the completed value-function class has low Bellman eluder dimension, with the proof resting on a new MLE-based squared Bellman error concentration…

desk verdict The core theorem is unsupported because the posterior draw is never shown to lie in the confidence set used by the Bellman-error concentration, but the new combination is worth a serious referee's time. read the letter →

arxiv 2505.23927 v1 pith:B7T6I6XL submitted 2025-05-29 cs.LG

Songtao Feng , Jie Fu This is my paper
classification cs.LG
keywords onlineRLHFThompsonsamplingBellmaneluderdimensiongeneralfunctionapproximationposteriorregretboundpreference-basedreinforcementlearningmaximumlikelihoodestimator
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a model-free Thompson sampling algorithm for online reinforcement learning from human feedback (RLHF) with general function approximation, and it claims an O(sqrt(T)) regret bound. This matters because most theoretical RLHF algorithms rely on confidence-set or model-based methods, while posterior sampling is more computationally tractable in practice. The paper's key move is to derive a concentration inequality for squared Bellman error from a maximum likelihood estimator generalization bound, then feed that inequality into a Bellman eluder dimension argument. If correct, the result shows that posterior sampling can match the statistical efficiency of confidence-set methods in preference-based RL.

What carries the argument

The load-bearing objects are the completed function class G_F (the closure of F under all possible posterior distributions, introduced so that realizability and completeness hold under posterior sampling), the Bellman eluder dimension of the residual class F_h - T_h F_{h+1}, and the confidence set V_1^F built from the MLE generalization bound. The MLE generalization bound converts preference log-likelihood into a bound on the squared distance between posterior draws and the MLE; applied to Bellman residuals, it yields the concentration inequality that makes the eluder dimension argument go through. The algorithm itself draws f^t from the posterior in each round, plays the greedy policy of f^t against the greedy policy of the previous round, and updates the dataset with the resulting trajectory comparison.

What would settle it

Take a finite function class F whose completion G_F has log-covering number growing like T log 2, as the paper notes for finite classes, and run Algorithm 1; if the cumulative squared Bellman error of the posterior draws exceeds the claimed O(beta_F) bound on a positive fraction of runs, the concentration step in the proof fails. More directly, inspect the posterior draws f^t: any round t where f^t falls outside the set V_1^F defined in Eq. (4) with non-negligible probability refutes the assertion that greedy selection plays the role of the MLE maximizer.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that model-free posterior sampling can achieve the same eluder-type regret guarantee as confidence-set RLHF. With probability at least 1-delta, Algorithm 1 satisfies Regret(t) <= O(H $\sqrt$(d t beta_G)) for all t, where d is the Bellman eluder dimension of the completed function class G_F and beta_G is a log-bracketing complexity term. The proof works in three steps: it simplifies the Bayesian regret using the fact that the posterior draw f^t and the true model f* are identically distributed given history, it bounds the regret by cumulative Bellman error through a standard policy loss decomposition, and it controls that error using a new squared-Bellman-error concentration inequality obtained from the MLE generalization bound. The paper presents this as the first eluder-type regret bound for Thompson sampling in online RLHF.

Load-bearing premise

The posterior sample f^t in round t lies in the MLE-based confidence set V_1^F, so the squared Bellman error bound applies to the functions actually played; the paper supports this only by asserting that greedy selection plays the role of the MLE maximizer, which is not proven.

Editorial extensions

If this is right

  • Posterior sampling with a planning oracle can match the statistical efficiency of confidence-set methods in online RLHF, achieving O(sqrt(T)) regret when the Bellman eluder dimension and log-bracketing constants are fixed.
  • The comparator policy can simply be the previous round's greedy policy, so no separate exploration policy or reward-model training loop is required by the algorithm.
  • The squared Bellman error concentration inequality holds uniformly for every function in the confidence set, not only for the function executed in a given round, which strengthens the standard eluder argument.
  • An epsilon-optimal policy with O(H^2 d beta_G / epsilon^2)-style sample complexity follows directly from the regret bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gap between posterior sampling and the confidence set is repaired, the same MLE-based concentration argument would likely transfer to Thompson sampling for standard RL, as the paper itself hints.
  • The paper's own finite-class calculation suggests a stress test: for a finite function class, completion can inflate the log-covering number to order T log 2, so the sqrt(T) bound may degrade; checking whether natural parameterized classes such as linear or low-rank classes preserve low Bellman eluder dimension under completion would determine the practical scope of the result.
  • A testable extension is to replace the sigmoid link with other monotone link functions in the preference model and see whether the regret bound's dependence on kappa tracks the inverse derivative lower bound as the paper predicts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper studies online RLHF with general function approximation and proposes Algorithm 1, a model-free Thompson sampling algorithm that samples an action-value function from the posterior and plays greedy policies. The main result (Theorem 4.1) claims a high-probability regret bound O(H sqrt(d t beta_G)) for all t, where d is the Bellman eluder dimension of a completed function class G_F and beta_G is a log-bracketing complexity term. The analysis introduces a confidence set V_1^F based on an MLE generalization bound and derives a squared Bellman error concentration inequality (Lemma B.6) to feed into the Bellman eluder dimension argument.

Significance. If the main theorem were proved, the paper would provide the first eluder-type regret bound for model-free Thompson sampling in RLHF, a useful complement to the confidence-set method of Chen et al. [7] and the model-based randomized algorithm of Wu and Sun [48]. The MLE-based concentration inequality (Lemma B.4/B.6) is a potentially interesting technical contribution, and the paper openly discusses limitations of the function-class completion. However, the central proof step—linking the posterior sample drawn in Algorithm 1 to the MLE-based confidence set—is missing, and the definition of V_1^F is ill-posed. These issues undermine Theorem 4.1 as stated.

major comments (4)
  1. [Appendix B.2, Eq. (4)] The confidence set V_1^F is ill-defined. The membership condition contains an inner expectation E_{f_h}[ (f_h(x_t) - hat f_h(x_t))^2 | H_{t-1} ], where f_h is sampled from the posterior; this expression does not depend on the candidate function (f_1,...,f_H) being tested. Consequently V_1^F is either empty or equal to all of F, making the 'for any f in V_1^F' statements in Lemma B.6 vacuous or trivial. The definition must be repaired before the argument can be assessed.
  2. [Appendix B.2 and Step III of Appendix C] The proof never establishes that the posterior sample f^t used in Algorithm 1 lies in V_1^F. Algorithm 1 draws f^t from p(f|D_{t-1}), but Lemma B.6 controls the cumulative squared Bellman error only for functions in V_1^F, and Step III invokes Lemma B.2(1) with phi_t = f^t_h - T_h f^t_{h+1} for the realized f^t. The single bridging sentence, 'f obtained by greedy selection in the algorithm plays the role of MLE maximizer hat f', conflates sampling from the posterior with maximizing the likelihood; a posterior draw is not the MLE. Since Lemma B.4 bounds posterior-averaged squared deviations rather than the deviation of a realized draw, the needed containment f^t in V_1^F is unproven and the eluder argument collapses.
  3. [Section 4.4 and Theorem 4.1] The paper defines regret as a frequentist quantity in Section 2, but the proof in Step I of Section 4.4 and Appendix C replaces it with Bayesian regret by writing Regret(T) = E_{f*} [ V^{pi^t_0}_{1;f^t} - V^{pi^t_0}_{1;f^*} ] (s_1) and using posterior exchangeability of f^t and f*. That identity holds only for the Bayesian regret, not for the high-probability regret bound stated in Theorem 4.1. The connection between the Bayesian argument and the claimed 'with probability at least 1-delta' statement is not made explicit, so the theorem's statement is not supported by the proof as written.
  4. [Section 3.1] The completed function class G_F is an assumption object, not a construction, and the paper itself notes that for finite F the completion may have log-covering number Theta(T), potentially yielding linear regret. The bound in Theorem 4.1 depends on d = dim_BE(G_F), but the paper does not show that d is small for nontrivial classes beyond the linear case. This makes the practical scope of the claimed O(sqrt(T)) regret unclear, though it is an assumption rather than an internal inconsistency.
minor comments (7)
  1. [Abstract and Section 5] 'crucial rules' should be 'crucial roles' in both occurrences.
  2. [Appendix E] 'Thomson sampling' should be 'Thompson sampling'.
  3. [Related Works] 'exiting sample-efficient RL algorithms' should be 'existing sample-efficient RL algorithms'.
  4. [Section 4.3, Eq. (2)] Equation (2) sums over t=1 to T, while Step II in Appendix C sums over t=0 to T; the indexing should be unified.
  5. [Algorithm 1, line 3] The notation 'p(·|S_{t-1})' is confusing; it should be 'p(·|D_{t-1})' or 'p(·|H_{t-1})'.
  6. [Section 2.1] The display 'r(τ^i) = P H h=1 = r(...)' contains an extra equality sign and should be corrected.
  7. [Lemma B.6] The statement quantifies 'for all f in V_1^F' but the left-hand side uses f^t and the right-hand side uses beta terms; the connection between the quantified f and the sequence f^t is not defined, which makes the lemma hard to parse.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the proof gap between posterior sampling and the MLE confidence set is a correctness issue, not a circular reduction.

full rationale

The paper's derivation is not circular in the sense of fitting a parameter to data and then presenting it as a prediction, nor does it reduce a claimed result to its own inputs by construction. The regret bound is stated under explicit assumptions (realizability, completeness, and bounded Bellman eluder dimension of the completed class), and the MLE generalization bound is imported from the external work of Wu and Sun [48], not from the authors' own prior results. The function class completion is an explicit complexity assumption, with the paper itself warning that finite classes may become intractable after completion, so it is not a hidden reuse of the conclusion. The only load-bearing bridge that is questionable is the statement in Section B.2 that the function obtained by greedy selection in the algorithm 'plays the role of MLE maximizer', which is used to connect the posterior draw in Algorithm 1 to the confidence set V_1^F in Eq. (4). This is an unsupported logical identification, and the confidence set definition in (4) is ill-posed as written because the candidate function f does not enter the defining condition. However, these are correctness gaps that would need repair before the theorem is established; they are not circular reductions in which the conclusion is assumed or the prediction is equivalent to the input. No load-bearing self-citations appear in the argument, and no step in the proof chain is defined in terms of the result it is meant to prove. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 6 assumptions · 1 invented entities

The central claim rests on standard structural assumptions (realizability, completeness, link function regularity), a cited MLE generalization bound, and two assumptions specific to this paper: the completed function class G_F has low Bellman eluder dimension, and (in the main theorem) the transition kernel P_h is known. There are no fitted free parameters in the theoretical claim; the simulation hyperparameters (n=5, m=20, l=50, omega=20) are chosen by hand but do not affect the theorem. The completed function class G_F is an invented construct with no independent evidence.

assumptions (6)
  • domain assumption Realizability: Q*_h in F_h for all h (Assumption 2.3, generalized as Assumption 3.2)
    Ensures the function class contains the optimal action-value function; standard but unverifiable in practice.
  • domain assumption Completeness: T_h F_{h+1} subset of F_h (Assumption 2.4, generalized as Assumption 3.3)
    Ensures Bellman backups stay in the class; standard but restrictive for nonlinear classes.
  • domain assumption Link function Phi is differentiable with kappa^{-1} <= Phi' <= bar-kappa^{-1} (Assumption 2.2)
    Controls how much preference feedback reveals about reward differences.
  • ad hoc to paper The completed function class G_F has low Bellman eluder dimension d and satisfies generalized realizability and completeness
    The theorem's regret is stated in terms of d = dim_BE(G_F, Pi, 1/sqrt(T)); the paper itself notes the completion can destroy tractability for finite classes (Section 3.1).
  • domain assumption Transition kernel P_h is known in the main analysis
    The likelihood in Eq. (1) uses r(tau) = sum_h (f_h - P_h f_{h+1})(s_h,a_h), requiring P_h; the paper claims estimation is possible but does not incorporate it into the theorem.
  • standard math MLE generalization bound (Lemma B.4, cited from Wu and Sun 2023)
    Provides the concentration inequality for posterior samples used to define the confidence set; imported from prior work.
invented entities (1)
  • Completed function class G_F
    purpose: Enlarges the function class so that any posterior distribution over models remains representable, enabling the Thompson sampling analysis.
    Constructed ad hoc in Definition 3.1; the paper leaves open whether known classes remain tractable after completion, and admits finite classes may incur linear regret.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Thompson Sampling in Online RLHF with General Function Approximation." pith.science (2026). https://pith.science/paper/B7T6I6XL

@misc{pith2026250523927,
  author       = {Pith},
  title        = {Pith review of: Thompson Sampling in Online RLHF with General Function Approximation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B7T6I6XL}},
  note         = {Machine review of arXiv:2505.23927}
}
abstract

Reinforcement learning from human feedback (RLHF) has achieved great empirical success in aligning large language models (LLMs) with human preference, and it is of great importance to study the statistical efficiency of RLHF algorithms from a theoretical perspective. In this work, we consider the online RLHF setting where the preference data is revealed during the learning process and study action value function approximation. We design a model-free posterior sampling algorithm for online RLHF inspired by Thompson sampling and provide its theoretical guarantee. Specifically, we adopt Bellman eluder (BE) dimension as the complexity measure of the function class and establish $O(\sqrt{T})$ regret bound for the proposed algorithm with other multiplicative factor depending on the horizon, BE dimension and the $log$-bracketing number of the function class. Further, in the analysis, we first establish the concentration-type inequality of the squared Bellman error bound based on the maximum likelihood estimator (MLE) generalization bound, which plays the crucial rules in obtaining the eluder-type regret bound and may be of independent interest.

Figures

Figures reproduced from arXiv: 2505.23927 by the authors.

Figure 1
Figure 1. Average regret and value of the function over iterations. [PITH_FULL_IMAGE:figures/full_fig_p019_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 37 canonical work pages

  1. [7]

    Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation.arXiv e-prints, page arXiv:2205.11140, May 2022

    Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation.arXiv e-prints, page arXiv:2205.11140, May 2022

    arXiv 2022

  2. [48]

    Making RL with Preference-based Feedback Efficient via Randomization

    Runzhe Wu and Wen Sun. Making RL with Preference-based Feedback Efficient via Randomization. arXiv e-prints, page arXiv:2310.14554, October 2023. 24

    arXiv 2023

  3. [1]

    Analysis of thompson sampling for the multi-armed bandit problem

    Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Shie Mannor, Nathan Srebro, and Robert C. Williamson, editors,Proceedings of the 25th Annual Conference on Learning Theory, volume 23 ofProceedings of Machine Learning Research, pages 39.1–39.26, Edinburgh, Scotland, 25–27 Jun 2012. PMLR

    work page 2012

  4. [2]

    Thompson sampling for contextual bandits with linear payoffs

    Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In Sanjoy Dasgupta and David McAllester, editors,Proceedings of the 30th International Conference on Machine Learning, volume 28 ofProceedings of Machine Learning Research, pages 127–135, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR

    work page 2013

  5. [3]

    Near-optimal regret bounds for thompson sampling.J

    Shipra Agrawal and Navin Goyal. Near-optimal regret bounds for thompson sampling.J. ACM, 64(5), September 2017

    work page 2017

  6. [4]

    Preference-based online learning with dueling bandits: a survey.J

    Viktor Bengs, Róbert Busa-Fekete, Adil El Mesaoudi-Paul, and Eyke Hüllermeier. Preference-based online learning with dueling bandits: a survey.J. Mach. Learn. Res., 22(1), January 2021

    work page 2021

  7. [5]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: The method of paired comparisons.Biometrika, 39(3-4):324–345, 12 1952

    work page 1952

  8. [6]

    Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell

    Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel- Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jaco...

    arXiv 2023

Show all 61 references
  1. [8]

    On the Weaknesses of Reinforcement Learning for Neural Machine Translation.arXiv e-prints, page arXiv:1907.01752, July 2019

    Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the Weaknesses of Reinforcement Learning for Neural Machine Translation.arXiv e-prints, page arXiv:1907.01752, July 2019

    1907 arXiv

  2. [9]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. InProceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 4302–4310, Red Hook, NY, USA...

    2017

  3. [10]

    RAFT: Reward ranked finetuning for generative foundation model alignment.Transactions on Machine Learning Research, 2023

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment.Transactions on Machine Learning Research, 2023

    2023

  4. [11]

    Schapire, Aleksandrs Slivkins, and Masrour Zoghi

    Miroslav Dudík, Katja Hofmann, Robert E. Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual dueling bandits. In Peter Grünwald, Elad Hazan, and Satyen Kale, editors,Proceedings of The 28th Conference on Learning Theory, volume 40 ofProceedings of Machine Learning Res...

    2015

  5. [12]

    Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO.arXiv e-prints, page arXiv:2005.12729, May 2020

    Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO.arXiv e-prints, page arXiv:2005.12729, May 2020

    2005 arXiv

  6. [13]

    Foster and Alexander Rakhlin

    Dylan J. Foster and Alexander Rakhlin. Beyond ucb: optimal and efficient contextual bandits with regression oracles. InProceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020

    2020

  7. [14]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023

    2023

  8. [15]

    A General Theoretical Paradigm to Understand Learning from Human Preferences

    Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A General Theoretical Paradigm to Understand Learning from Human Preferences. arXiv e-prints, page arXiv:2310.12036, October 2023

    arXiv

  9. [16]

    Reinforced Self-Training (ReST) for Language Modeling.arXiv e-prints, page arXiv:2308.08998, August 2023

    Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced Self-Training (ReST) for Language Mode...

    2023 arXiv

  10. [17]

    Haque Ishfaq, Qiwen Cui, Viet Nguyen, Alex Ayoub, Zhuoran Yang, Zhaoran Wang, Doina Precup, and Lin F. Yang. Randomized Exploration for Reinforcement Learning with General Value Function Approximation.arXiv e-prints, page arXiv:2106.07841, June 2021

    2021 arXiv

  11. [18]

    Rupam Mahmood, Doina Precup, Anima Anandkumar, and Kamyar Azizzadenesheli

    Haque Ishfaq, Qingfeng Lan, Pan Xu, A. Rupam Mahmood, Doina Precup, Anima Anandkumar, and Kamyar Azizzadenesheli. Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo.arXiv e-prints, page arXiv:2305.18246, May 2023

    2023 arXiv

  12. [19]

    Learning trajectory preferences for manipulators via iterative improvement

    Ashesh Jain, Brian Wojcik, Thorsten Joachims, and Ashutosh Saxena. Learning trajectory preferences for manipulators via iterative improvement. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors,Advances in Neural Information Processing Systems, ...

    2013

  13. [20]

    Schapire

    Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E. Schapire. Contextual decision processes with low Bellman rank are PAC-learnable. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Learning, volume ...

    2017

  14. [21]

    Bellman eluder dimension: new rich classes of rl problems, and sample-efficient algorithms

    Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: new rich classes of rl problems, and sample-efficient algorithms. InProceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY, USA, 2021. Curran Assoc...

    2021

  15. [22]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. An Introduction to Variational Autoencoders.arXiv e-prints, page arXiv:1906.02691, June 2019

    1906 arXiv

  16. [23]

    Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism.arXiv e-prints, page arXiv:2305.18438, May 2023

    Zihao Li, Zhuoran Yang, and Mengdi Wang. Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism.arXiv e-prints, page arXiv:2305.18438, May 2023. 21

    2023 arXiv

  17. [24]

    ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models

    Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. arXiv e-prints, page arXiv:2310.10505, October 2023

    arXiv

  18. [25]

    Liu, and Jialu Liu

    Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. Statistical Rejection Sampling Improves Preference Optimization.arXiv e-prints, page arXiv:2309.06657, September 2023

    2023 arXiv

  19. [26]

    Roberts, Matthew E

    James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L. Roberts, Matthew E. Taylor, and Michael L. Littman. Interactive learning from policy-dependent human feedback. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17...

    2017

  20. [27]

    Novoseller, Yanan Sui, Yisong Yue, and Joel W

    Ellen R. Novoseller, Yanan Sui, Yisong Yue, and Joel W. Burdick. Dueling posterior sampling for preference-based reinforcement learning.ArXiv, abs/1908.01289, 2019

    1908 arXiv

  21. [28]

    Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, and Alec Radford

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jef...

    2023 arXiv

  22. [29]

    Randomized prior functions for deep reinforcement learning

    Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. InProceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 8626–8638, Red Hook, NY, USA, 2018. Curran Associates Inc

    2018

  23. [30]

    Approximate thompson sampling via epistemic neural networks

    Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Approximate thompson sampling via epistemic neural networks. In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, UAI ’2...

    2023

  24. [31]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  25. [32]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  26. [33]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model.arXiv e-prints, page arXiv:2305.18290, May 2023

    2023 arXiv

  27. [34]

    Worst-case regret bounds for exploration via randomized value functions.ArXiv, abs/1906.02870, 2019

    Daniel Russo. Worst-case regret bounds for exploration via randomized value functions.ArXiv, abs/1906.02870, 2019

    1906 arXiv

  28. [35]

    Learning to optimize via posterior sampling.Math

    Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling.Math. Oper. Res., 39(4):1221–1243, November 2014

    2014

  29. [36]

    Optimal algorithms for stochastic contextual preference bandits

    Aadirupa Saha. Optimal algorithms for stochastic contextual preference bandits. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, volume 34, pages 30050–30062. Curran Associates, Inc., 2021

    2021

  30. [37]

    Efficient and optimal algorithms for contextual dueling bandits under realizability

    Aadirupa Saha and Akshay Krishnamurthy. Efficient and optimal algorithms for contextual dueling bandits under realizability. In Sanjoy Dasgupta and Nika Haghtalab, editors,Proceedings of The 33rd International Conference on Algorithmic Learning Theory, volume 167 ofProceedings...

    2022

  31. [38]

    Dueling rl: Reinforcement learning with trajectory preferences

    Aadirupa Saha, Aldo Pacchiano, and Jonathan Lee. Dueling rl: Reinforcement learning with trajectory preferences. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors,Proceedings of The 23 26th International Conference on Artificial Intelligence and Statistics, ...

    2023

  32. [39]

    Proximal Policy Optimization Algorithms.arXiv e-prints, page arXiv:1707.06347, July 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms.arXiv e-prints, page arXiv:1707.06347, July 2017

    2017 arXiv

  33. [40]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. InProceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, ...

    2020

  34. [41]

    Thompson

    William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples.Biometrika, 25(3/4):285–294, 1933

    1933

  35. [42]

    Llama 2: Open Foundation and Fine-Tuned Chat Models.arXiv e-prints, page arXiv:2307.09288, July 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  36. [43]

    van de Geer

    Sara A. van de Geer. Empirical processes in m-estimation. 2000

    2000

  37. [44]

    Beyond Reverse KL: Gen- eralizing Direct Preference Optimization with Diverse Divergence Constraints.arXiv e-prints, page arXiv:2309.16240, September 2023

    Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond Reverse KL: Gen- eralizing Direct Preference Optimization with Diverse Divergence Constraints.arXiv e-prints, page arXiv:2309.16240, September 2023

    2023 arXiv

  38. [45]

    Thompson sampling for combinatorial semi-bandits

    Siwei Wang and Wei Chen. Thompson sampling for combinatorial semi-bandits. In Jennifer Dy and Andreas Krause, editors,Proceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 5114–5122. PMLR, 10–15 Jul 2018

    2018

  39. [46]

    Is rlhf more difficult than standard rl? a theoretical perspective

    Yuanhao Wang, Qinghua Liu, and Chi Jin. Is rlhf more difficult than standard rl? a theoretical perspective. InProceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc

    2023

  40. [47]

    A survey of preference-based reinforcement learning methods.Journal of Machine Learning Research, 18(136):1–46, 2017

    Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference-based reinforcement learning methods.Journal of Machine Learning Research, 18(136):1–46, 2017

    2017

  41. [49]

    Borda regret minimization for generalized linear dueling bandits

    Yue Wu, Tao Jin, Qiwei Di, Hao Lou, Farzad Farnoud, and Quanquan Gu. Borda regret minimization for generalized linear dueling bandits. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    2024

  42. [50]

    Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint.arXiv e-prints, page arXiv:2312.11456, December 2023

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint.arXiv e-prints, page arXiv:2312.11456, December 2023

    2023 arXiv

  43. [51]

    Near-optimal randomized exploration for tabular markov decision processes

    Zhihan Xiong, Ruoqi Shen, Qiwen Cui, Maryam Fazel, and Simon S Du. Near-optimal randomized exploration for tabular markov decision processes. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing Systems, volume...

    2022

  44. [52]

    Yang, Aarti Singh, and Artur Dubrawski

    Yichong Xu, Ruosong Wang, Lin F. Yang, Aarti Singh, and Artur Dubrawski. Preference-based reinforcement learning with finite-time guarantees. InProceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA, 2020. Curran ...

    2020

  45. [53]

    RRHF: Rank Responses to Align Language Models with Human Feedback without tears.arXiv e-prints, page arXiv:2304.05302, April 2023

    Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: Rank Responses to Align Language Models with Human Feedback without tears.arXiv e-prints, page arXiv:2304.05302, April 2023

    2023 arXiv

  46. [54]

    The k-armed dueling bandits problem.Journal of Computer and System Sciences, 78(5):1538–1556, 2012

    Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem.Journal of Computer and System Sciences, 78(5):1538–1556, 2012. JCSS Special Issue: Cloud Computing 2011

    2012

  47. [55]

    Frequentist Regret Bounds for Randomized Least-Squares Value Iteration.arXiv e-prints, page arXiv:1911.00567, November 2019

    Andrea Zanette, David Brandfonbrener, Emma Brunskill, Matteo Pirotta, and Alessandro Lazaric. Frequentist Regret Bounds for Randomized Least-Squares Value Iteration.arXiv e-prints, page arXiv:1911.00567, November 2019

    1911 arXiv

  48. [56]

    Lee, and Wen Sun

    Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D. Lee, and Wen Sun. Provable Offline Preference-Based Reinforcement Learning.arXiv e-prints, page arXiv:2305.14816, May 2023

    2023 arXiv

  49. [57]

    Lee, and Wen Sun

    Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D. Lee, and Wen Sun. Provable offline preference- based reinforcement learning. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  50. [58]

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. SLiC-HF: Sequence Likelihood Calibration with Human Feedback.arXiv e-prints, page arXiv:2305.10425, May 2023

    2023 arXiv

  51. [59]

    Jordan, and Jiantao Jiao

    Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael I. Jordan, and Jiantao Jiao. Fine-Tuning Language Models with Advantage-Induced Policy Alignment.arXiv e-prints, page arXiv:2306.02231, June 2023

    2023 arXiv

  52. [60]

    Efficient active learning with abstention

    Yinglun Zhu and Robert Nowak. Efficient active learning with abstention. InProceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA,

  53. [2022]

    Curran Associates Inc. 25

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.