Pith. sign in

REVIEW 3 major objections 5 minor 74 references

Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysis

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proves that augmenting any confidence-based online RL algorithm with an offline dataset yields order-optimal sub-optimality gap and regret, and that the right offline data differs by objective.

desk verdict Clean and insightful framework for hybrid RL, but the sub-optimality gap upper bound has a proof gap that makes the main theorem unproven as written. read the letter →

arxiv 2505.13768 v3 pith:IRTYQVBQ submitted 2025-05-19 cs.LG stat.ML

Ruiquan Huang , Donghao Li , Chengshuai Shi , Cong Shen , Jing Yang This is my paper · ORCID
classification cs.LGstat.ML MSC 68Q3268T05
keywords hybridreinforcementlearningoffline-to-onlineRLconfidence-basedalgorithmsconcentrabilitycoefficientuncertaintylevelregretminimizationsub-optimalitygaplinearcontextualbandits
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Hybrid reinforcement learning—combining a fixed offline dataset with online interaction—has many ad hoc recipes; this paper claims one generic recipe already attains the best possible rates. The recipe is to take any confidence-based online algorithm that outputs a value estimate and a high-probability uncertainty bound, rerun it on the pooled dataset $D_0 \cup D_{t-1}$ at every episode, play the upper-confidence-bound policy, and, for gap minimization, return the lower-confidence-bound policy at the end. The proved sub-optimality gap is $\tilde{O}(C_{\mathrm{Alg}}/\sqrt{N_0/C(\pi^*|\rho)+N_1})$ and the regret is $\tilde{O}(C_{\mathrm{Alg}}\sqrt{N_1}\sqrt{N_1/(N_0/C(\pi^-_\epsilon|\rho)+N_1)})$, with matching lower bounds on a two-arm bandit instance, so offline data helps exactly according to a new concentrability coefficient $C(\pi|\rho)$ and never asymptotically hurts. A practically useful consequence is a separation: gap minimization wants offline data that covers the optimal policy, while regret minimization wants coverage of sub-optimal policies. The authors instantiate the framework for tabular MDPs and linear contextual bandits and report experiments consistent with these predictions.

What carries the argument

The load-bearing identity is the harmonic-mean shortcut $\min\{C(\pi^*|\rho)/N_0, 1/N_1\} \le 2/(N_0/C(\pi^*|\rho)+N_1)$ used at the end of Theorem 3.1's proof, which lets the offline uncertainty term and the online Eluder term be merged into a single effective-sample-size rate. The objects feeding that identity are the uncertainty level $U_{M^*}(\pi)$, the concentrability coefficient $C(\pi|\rho)$, and the Eluder-type condition $\sum_{t=1}^{N_1}\hat{U}^{\pi_t}_{\mathrm{Alg}}(D_{t-1})^2 \le C_{\mathrm{Alg}}^2$, which plays the role of a pigeonhole or elliptical-potential lemma controlling how much exploration can cost. Together they express offline and online statistical difficulty in the same units, so the two bounds can be compared and combined.

What would settle it

Run Algorithm 1 on a two-state tabular MDP or a two-arm linear bandit with an offline dataset chosen so that after some round $t$ the UCB-selected policy has $\hat{U}^{\pi_t}_{\mathrm{Alg}}(D_{t-1}) < \hat{U}^{\pi^*}_{\mathrm{Alg}}(D_{t-1})$ while both uncertainty bounds remain valid upper bounds. That single inequality contradicts the proof's step (b) in Theorem B.1; if it occurs with non-negligible probability, the claimed $\tilde{O}(1/\sqrt{N_1})$ online component of the sub-optimality gap bound does not follow from the stated assumptions.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that confidence-based online RL needs no new algorithmic machinery to benefit from offline data: initialize with $D_0$, keep it in the training set, act optimistically, and finish pessimistically. Defining $U_{M^*}(\pi)$ as the minimax expected error of the best oracle in estimating $V^\pi_{M^*}$ from $N_0$ draws under the behavior policy $\rho$, and $C(\pi|\rho)=(U_{M^*}(\pi)/U_{M^*}(\rho))^2$, the paper proves (Theorem 3.1) that the final policy's gap is $\tilde{O}(C_{\mathrm{Alg}}/\sqrt{N_0/C(\pi^*|\rho)+N_1})$ and (Theorem 3.2) that the cumulative regret is $\tilde{O}(C_{\mathrm{Alg}}\sqrt{N_1}\sqrt{N_1/(N_0/C(\pi^-_\epsilon|\rho)+N_1)})$ whenever the oracle satisfies an Eluder-type condition. Theorem 5.1 gives matching $\tilde{\Omega}$ lower bounds on a two-arm linear contextual bandit, establishing order-optimality. The conceptual discovery is the coefficient: $N_0$ offline trajectories under $\rho$ act like $N_0/C(\pi|\rho)$ virtual online episodes, and which coefficient matters flips depending on whether the learning goal is a final near-optimal policy or low cumulative regret.

Load-bearing premise

The proof of the sub-optimality gap bound assumes that, whenever the algorithm picks the UCB-optimistic policy, that policy's estimation uncertainty about the optimal policy is at least as large as the optimal policy's own uncertainty, i.e. $\hat{U}(\pi^*|D_{t-1})\le \hat{U}(\pi_t|D_{t-1})$; the argument derives this from the policy being UCB-optimal, but maximizing an estimated value plus uncertainty bound does not by itself force that inequality.

Editorial extensions

If this is right

  • A behavior policy $\rho$ with good coverage of the optimal policy makes the hybrid sub-optimality gap behave like an online learner with $N_0/C(\pi^*|\rho)+N_1$ episodes, so the gap strictly improves over both pure online and pure offline whenever the coefficient is finite.
  • For regret, the relevant coefficient is $C(\pi^-_\epsilon|\rho)$, the worst concentrability among policies that are at least $\epsilon$ sub-optimal; offline data that explores sub-optimal policies yields a constant-factor speed-up over pure online, while a dataset of only optimal trajectories yields no asymptotic regret improvement.
  • The two lower bounds show that the dependence on $N_0$, $N_1$, and the concentrability coefficients is order-tight, so the simple plug-in design cannot be improved asymptotically by more elaborate hybrid algorithms.
  • In tabular MDPs the bounds become $\tilde{O}(\sqrt{H^4|X||A|/(N_0/C(\pi^*|\rho)+N_1)})$ for the gap and the corresponding regret; in linear contextual bandits they become $\tilde{O}(d\sqrt{1/(N_0/C+N_1)})$ and $\tilde{O}(d\sqrt{N_1}\sqrt{N_1/(N_0/C+N_1)})$, giving explicit dimension dependence.
  • The separation result translates to data-collection guidance: for a final-policy objective, collect offline data concentrated on good actions; for a regret objective, collect a broad exploratory dataset, and a single dataset cannot simultaneously optimize both unless it covers both kinds of policies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not pursue is to optimize the offline data-collection policy under a budget: the two coefficients make the tradeoff explicit, and a mixture policy that covers both $\pi^*$ and a spread of $\epsilon$-suboptimal policies should dominate either extreme; this is testable in the paper's own linear-bandit setup.
  • The proof gap in the appendix suggests the rate may still be true but needs one more argument: replacing the asserted inequality $\hat{U}(\pi^*|D_{t-1})\le \hat{U}(\pi_t|D_{t-1})$ with a variance-weighted or exploration-conditioned bound would likely preserve the claimed order while making the proof self-contained.
  • For practitioners using deep RL, where exact confidence sets are unavailable, the framework suggests a concrete heuristic: train the value network and an epistemic-uncertainty estimate on $D_0 \cup D_{t-1}$ at every round, act by UCB, and output by LCB; the rate guarantee would not transfer formally, but the effective-sample-size interpretation gives a principled reason to expect similar qualitati
  • The lower-bound instance is a two-arm linear bandit, so the order-optimality claim has not been tested against richer MDP structure; constructing a tabular MDP instance where the gap and regret coefficients genuinely differ would strengthen the separation beyond the current simulation evidence.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a generic hybrid reinforcement-learning framework in which any confidence-based online RL algorithm is run with the offline dataset included at every online update. The main theoretical claims are: (i) the sub-optimality gap of the final pessimistic policy is O~(C_Alg / sqrt(N0/C(pi*|rho) + N1)); and (ii) the online regret is O~(C_Alg sqrt(N1) sqrt(N1/(N0/C(pi^-|rho)+N1))), where C(pi|rho) is a new concentrability coefficient defined as the squared ratio of uncertainty levels. The paper also proves matching lower bounds and instantiates the framework in tabular MDPs and linear contextual bandits, with experiments supporting the predicted separation between coverage requirements for gap minimization and regret minimization.

Significance. If the upper bounds are established, the paper gives an appealing unified template: offline data acts as an effective additional sample budget, and the results are order-optimal with respect to the stated lower bounds. The regret theorem, the lower bounds, and the coverage-separation insight are valuable and appear coherent. The paper also benefits from concrete instantiations with tabular MDPs and linear bandits and from experimental validation. However, the significance is currently conditional: the proof of the sub-optimality gap theorem contains an unjustified inference, and the new concentrability coefficient is defined in terms of the very uncertainty quantities being bounded, so its interpretation as an independent coverage measure needs clarification.

major comments (3)
  1. [Appendix B, Theorem B.1] The step labelled (b) in the proof of Theorem B.1, namely Uhat(pi*|D_{t-1}) <= Uhat(pi_t|D_{t-1}) 'due to the optimality of pi_t', is not justified. Since pi_t maximizes Vhat + Uhat, UCB optimality only implies Vhat(pi*) + Uhat(pi*) <= Vhat(pi_t) + Uhat(pi_t), equivalently Uhat(pi*) - Uhat(pi_t) <= Vhat(pi_t) - Vhat(pi*). The right-hand side need not be non-positive, and neither the confidence-interval definition in Section 3.1 nor the Eluder-type condition in Definition 3.3 (which bounds only the executed policies pi_t) rules this out. This step is what converts the average of Uhat(pi*|D_{t-1}) into the Eluder bound, so the O~(C_Alg/sqrt(N1)) online component of Theorem 3.1 is not established as written. The regret proof in Theorem B.2 does not use this step and appears unaffected. A possible repair is to use UCB selection to show V* - V(pi_t) <= 2 Uhat(pi_t|D_{t-1}) and then compare the final LCB against the LCB of a visited policy, but the authors should supply such an argument or add an explicit assumption.
  2. [Definition 3.2 and proof of Theorem 3.1] The offline component of the sub-optimality gap bound is largely definitional. Since C(pi*|rho) = (U(pi*)/U(rho))^2 and U(rho) = O~(1/sqrt(N0)), the bound O~(C_Alg sqrt(C(pi*|rho)/N0)) is just O~(C_Alg U(pi*)), i.e., the target policy's own offline estimation error. Thus the theorem's offline term partly restates the new coefficient rather than relating C(pi|rho) to an independent data-coverage quantity such as a density-ratio concentrability. The paper should either prove concrete inequalities comparing C(pi|rho) with standard concentrability coefficients in the examples (the short arguments in Section 4 only give inequalities in the opposite direction) or explicitly state that C(pi|rho) is a normalization of the target policy's minimax estimation error. Without this, the claim that the paper introduces a new and meaningful concentrability coefficient is weakened.
  3. [Section 3.1 and Appendix B] The proofs in Theorem B.1 and Theorem B.2 use the monotonicity property that Uhat(pi|D) <= Uhat(pi|D') whenever D' is a subset of D, but this property is not stated among the assumptions on the oracle algorithm. For a generic 'confidence-based algorithm', enlarging the dataset does not automatically shrink the confidence bound; the definition of the oracle in Section 3.1 only guarantees that Uhat is an upper confidence bound. This monotonicity should be made an explicit assumption of the framework, or verified for each instantiation, since both main upper bounds rely on it.
minor comments (5)
  1. [Corollary 4.2] The stated sub-optimality gap uses C(pi^{-epsilon}|rho) in the denominator, but for gap minimization the relevant coefficient is C(pi*|rho); the same symbol appears to be copied from the regret statement.
  2. [Definition 3.1] The uncertainty level is denoted both U_M*(pi) and U_M(pi) at different places; please use one notation consistently and define all arguments in the minimax definition of Alg0.
  3. [Remark 3.2] The remark writes C(pi^{-epsilon}|rho) = 0, but Definition 3.2 states that C(pi|rho) is in [1, infinity]; the boundary case should be reconciled or the definition should allow zero.
  4. [Theorem 5.1] The lower-bound formulas are typeset ambiguously: e.g., Omega(1/sqrt(N0/C(pi*|rho)+N1)) should be written with clear parentheses so that the denominator is N0/C(pi*|rho) + N1, not 1/sqrt(N0) divided by C plus N1.
  5. [Section 6] The text says Figure 1(a) and (f) list C(pi^{-epsilon}|rho), but the tables in the figure appear to list C(pi*|rho); please make the caption and text consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the new concentrability coefficient is a definitional parameter, while the harmonic-mean combination and the matching lower bound provide independent content.

full rationale

The derivation is self-contained and does not reduce any central prediction to its inputs. Definition 3.2 defines C(pi|rho) as a squared ratio of minimax offline uncertainties, so the offline branch of Theorem 3.1 (Appendix B, Theorem B.1) is partly a repackaging of that definition together with the Azuma-Hoeffding bound U(rho|D0)=O(1/sqrt(N0)). I do not count this as circular because the coefficient is a problem-dependent parameter rather than a fitted quantity, and the substantive content of the theorem lies in the harmonic-mean combination of the offline and online uncertainty terms and in the matching lower bound of Theorem 5.1. The lower bound is built on the external two-arm linear bandit construction of He et al. (2022), not on the authors' own prior results, and no load-bearing self-citations appear in the proofs. I do flag a correctness gap that is not circularity: in Theorem B.1 step (b), the inequality Uhat(pi*|D_{t-1}) <= Uhat(pi_t|D_{t-1}) is attributed to the optimality of pi_t, but UCB optimality only gives Vhat^{pi_t}+Uhat^{pi_t} >= Vhat^{pi*}+Uhat^{pi*}, which does not imply the uncertainty ordering. This affects the validity of the online component as written, but it is a missing implication, not a reduction of the result to its inputs, so it does not change the circularity score.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the assumed existence of a confidence-based oracle satisfying the Eluder-type condition and a constant-factor relation to the minimax uncertainty level. No free parameters are fitted to data in the theoretical derivation; the concentrability coefficient is a problem-dependent definition rather than a fitted quantity. No new physical entities are introduced.

assumptions (5)
  • domain assumption Existence of a confidence-based oracle Alg that, for any dataset D, outputs value estimates V_hat^pi and uncertainty U_hat^pi with |V^pi_M* - V_hat^pi| <= U_hat^pi with high probability.
    Core to Algorithm 1 and to Theorems 3.1/3.2; Section 3.1 assumes such an oracle.
  • domain assumption The oracle and the UCB policy sequence satisfy the Eluder-type condition sum_t U_hat^{pi_t}(D_{t-1})^2 <= C_Alg^2.
    Definition 3.3; used in the online component of both bounds, verified for tabular MDPs and LinUCB in Appendix C.
  • domain assumption The oracle's uncertainty on the offline dataset is within a constant factor of the minimax uncertainty level: U_hat^pi_Alg(D0) <= C_Alg U_M(pi) for any M, D0.
    Assumption after Definition 3.3; used to convert offline uncertainty into the concentrability coefficient.
  • domain assumption In the examples, U_hat_Alg is used as a proxy for the minimax uncertainty level U_M*(pi); the ratio U_hat_Alg/U_M* is bounded by problem-dependent factors.
    Sections 4.1/4.2; used to state Corollaries 4.1/4.2 with the new coefficient.
  • standard math Standard concentration tools: Azuma-Hoeffding, elliptical potential lemma, and model-based confidence bounds from Azar et al. 2017.
    Used throughout proofs; cited and invoked in Appendices B and C.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysis." pith.science (2026). https://pith.science/paper/IRTYQVBQ

@misc{pith2026250513768,
  author       = {Pith},
  title        = {Pith review of: Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IRTYQVBQ}},
  note         = {Machine review of arXiv:2505.13768}
}
abstract

This paper investigates a hybrid learning framework for reinforcement learning (RL) in which the agent can leverage both an offline dataset and online interactions to learn the optimal policy. We present a unified algorithm and analysis and show that augmenting confidence-based online RL algorithms with the offline dataset outperforms any pure online or offline algorithm alone and achieves state-of-the-art results under two learning metrics, i.e., sub-optimality gap and online learning regret. Specifically, we show that our algorithm achieves a sub-optimality gap $\tilde{O}(\sqrt{1/(N_0/\mathtt{C}(\pi^*|\rho)+N_1}) )$, where $\mathtt{C}(\pi^*|\rho)$ is a new concentrability coefficient, $N_0$ and $N_1$ are the numbers of offline and online samples, respectively. For regret minimization, we show that it achieves a constant $\tilde{O}( \sqrt{N_1/(N_0/\mathtt{C}(\pi^{-}|\rho)+N_1)} )$ speed-up compared to pure online learning, where $\mathtt{C}(\pi^-|\rho)$ is the concentrability coefficient over all sub-optimal policies. Our results also reveal an interesting separation on the desired coverage properties of the offline dataset for sub-optimality gap minimization and regret minimization. We further validate our theoretical findings in several experiments in special RL models such as linear contextual bandits and Markov decision processes (MDPs).

Figures

Figures reproduced from arXiv: 2505.13768 by the authors.

Figure 1
Figure 1. Experimental results on sub-optimality gap (SOG) and regret for different behavior policies and [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Experimental results on sub-optimality gap (SOG) and regret for different behavior policies and [PITH_FULL_IMAGE:figures/full_fig_p022_2.png] view at source ↗
Figure 3
Figure 3. Experimental results on sub-optimality gap (SOG) and regret for different offline coefficient [PITH_FULL_IMAGE:figures/full_fig_p023_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 33 canonical work pages

  1. [1]

    Improved algorithms for linear stochastic bandits

    Yasin Abbasi-Yadkori, D \'a vid P \'a l, and Csaba Szepesv \'a ri. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011

    2011

  2. [2]

    Analysis of thompson sampling for the multi-armed bandit problem

    Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory, pages 39--1. JMLR Workshop and Conference Proceedings, 2012

    2012

  3. [3]

    Thompson sampling for contextual bandits with linear payoffs

    Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning, pages 127--135. PMLR, 2013

    2013

  4. [4]

    Optimal best-arm identification in bandits with access to offline data

    Shubhada Agrawal, Sandeep Juneja, Karthikeyan Shanmugam, and Arun Sai Suggala. Optimal best-arm identification in bandits with access to offline data. arXiv preprint arXiv:2306.09048, 2023

    arXiv 2023

  5. [5]

    Exploration--exploitation tradeoff using variance estimates in multi-armed bandits

    Jean-Yves Audibert, R \'e mi Munos, and Csaba Szepesv \'a ri. Exploration--exploitation tradeoff using variance estimates in multi-armed bandits. Theoretical Computer Science, 410 0 (19): 0 1876--1902, 2009

    1902

  6. [6]

    P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Mach. Learn., 47 0 (2-3): 0 235--256, May 2002

    work page 2002

  7. [7]

    Minimax regret bounds for reinforcement learning

    Mohammad Gheshlaghi Azar, Ian Osband, and R \'e mi Munos. Minimax regret bounds for reinforcement learning. In International conference on machine learning, pages 263--272. PMLR, 2017

    2017

  8. [8]

    Stochastic linear bandits robust to adversarial attacks

    Ilija Bogunovic, Arpan Losalka, Andreas Krause, and Jonathan Scarlett. Stochastic linear bandits robust to adversarial attacks. In International Conference on Artificial Intelligence and Statistics, pages 991--999. PMLR, 2021

    2021

Show all 74 references
  1. [9]

    Offline contextual bandits with overparameterized models

    David Brandfonbrener, William Whitney, Rajesh Ranganath, and Joan Bruna. Offline contextual bandits with overparameterized models. In International Conference on Machine Learning, pages 1049--1058. PMLR, 2021

    2021

  2. [10]

    Regret analysis of stochastic and nonstochastic multi-armed bandit problems

    S \'e bastien Bubeck and Nicolo Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning, 5 0 (1): 0 1--122, 2012

    2012

  3. [11]

    Kullback-leibler upper confidence bounds for optimal sequential allocation

    Olivier Capp \'e , Aur \'e lien Garivier, Odalric-Ambrym Maillard, R \'e mi Munos, and Gilles Stoltz. Kullback-leibler upper confidence bounds for optimal sequential allocation. The Annals of Statistics, pages 1516--1541, 2013

    2013

  4. [12]

    The elliptical potential lemma revisited

    Alexandra Carpentier, Claire Vernade, and Yasin Abbasi-Yadkori. The elliptical potential lemma revisited. arXiv preprint arXiv:2010.10182, 2020

    2010 arXiv

  5. [13]

    An empirical evaluation of thompson sampling

    Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. Advances in neural information processing systems, 24, 2011

    2011

  6. [14]

    Information-theoretic considerations in batch reinforcement learning

    Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning, pages 1042--1051. PMLR, 2019

    2019

  7. [15]

    Leveraging (biased) information: Multi-armed bandits with offline data

    Wang Chi Cheung and Lixing Lyu. Leveraging (biased) information: Multi-armed bandits with offline data. arXiv preprint arXiv:2405.02594, 2024

    2024

  8. [16]

    Contextual bandits with linear payoff functions

    Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 208--214. JMLR Workshop and Conference Proceedings, 2011

    2011

  9. [17]

    Stochastic linear optimization under bandit feedback

    Varsha Dani, Thomas P Hayes, and Sham M Kakade. Stochastic linear optimization under bandit feedback. 21st Annual Conference on Learning Theory, pages 355--366, 2008

    2008

  10. [18]

    Minimax-optimal off-policy evaluation with linear function approximation

    Yaqi Duan, Zeyu Jia, and Mengdi Wang. Minimax-optimal off-policy evaluation with linear function approximation. In International Conference on Machine Learning, pages 2701--2709. PMLR, 2020

    2020

  11. [19]

    The kl-ucb algorithm for bounded stochastic bandits and beyond

    Aur \'e lien Garivier and Olivier Capp \'e . The kl-ucb algorithm for bounded stochastic bandits and beyond. In Proceedings of the 24th annual conference on learning theory, pages 359--376. JMLR Workshop and Conference Proceedings, 2011

    2011

  12. [20]

    Guidelines for reinforcement learning in healthcare

    Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Doshi-Velez, and Leo Anthony Celi. Guidelines for reinforcement learning in healthcare. Nature medicine, 25 0 (1): 0 16--18, 2019

    2019

  13. [21]

    The movielens datasets: History and context

    F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5 0 (4): 0 1--19, 2015

    2015

  14. [22]

    A reduction from linear contextual bandits lower bounds to estimations lower bounds

    Jiahao He, Jiheng Zhang, and Rachel Q Zhang. A reduction from linear contextual bandits lower bounds to estimations lower bounds. In International Conference on Machine Learning, pages 8660--8677. PMLR, 2022

    2022

  15. [23]

    Deep q-learning from demonstrations

    Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  16. [24]

    Optimal best-arm identification in linear bandits

    Yassir Jedra and Alexandre Proutiere. Optimal best-arm identification in linear bandits. Advances in Neural Information Processing Systems, 33: 0 10007--10017, 2020

    2020

  17. [25]

    Provably efficient reinforcement learning with linear function approximation

    Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137--2143. PMLR, 2020

    2020

  18. [26]

    Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms

    Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems, 34: 0 13406--13418, 2021 a

    2021

  19. [27]

    Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, pages 5084--5096

    Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, pages 5084--5096. PMLR, 2021 b

    2021

  20. [28]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 1179--1191, 2020

    2020

  21. [29]

    Asymptotically efficient adaptive allocation rules

    Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Adv. Appl. Math., 6 0 (1): 0 4--22, March 1985

    1985

  22. [30]

    Bandit algorithms

    Tor Lattimore and Csaba Szepesv \'a ri. Bandit algorithms. Cambridge University Press, 2020

    2020

  23. [31]

    Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble

    Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1702--1712. PMLR, 2022

    2022

  24. [32]

    Lee, Yuejie Chi, and Yuxin Chen

    Gen Li, Wenhao Zhan, Jason D. Lee, Yuejie Chi, and Yuxin Chen. Reward-agnostic fine-tuning: Provable statistical benefits of hybrid reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=Nd3FennRJZ

    2023

  25. [33]

    Settling the sample complexity of model-based offline reinforcement learning

    Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi, and Yuting Wei. Settling the sample complexity of model-based offline reinforcement learning. The Annals of Statistics, 52 0 (1): 0 233--260, 2024

    2024

  26. [34]

    Pessimism for offline linear contextual bandits using l _p confidence sets

    Gene Li, Cong Ma, and Nati Srebro. Pessimism for offline linear contextual bandits using l _p confidence sets. Advances in Neural Information Processing Systems, 35: 0 20974--20987, 2022

    2022

  27. [35]

    A contextual-bandit approach to personalized news article recommendation

    Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pages 661--670, 2010

    2010

  28. [36]

    Fast active learning for pure exploration in reinforcement learning

    Pierre M \'e nard, Omar Darwiche Domingues, Anders Jonsson, Emilie Kaufmann, Edouard Leurent, and Michal Valko. Fast active learning for pure exploration in reinforcement learning. In International Conference on Machine Learning, pages 7599--7608. PMLR, 2021

    2021

  29. [37]

    Efficient memory-based learning for robot control

    Andrew William Moore. Efficient memory-based learning for robot control. Technical report, University of Cambridge, 1990

    1990

  30. [38]

    Collaborative-filtering

    Keval Morabia. Collaborative-filtering. https://github.com/kevalmorabia97/Collaborative-Filtering, 2019

    2019

  31. [39]

    Finite-time bounds for fitted value iteration

    R \'e mi Munos and Csaba Szepesv \'a ri. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9 0 (5), 2008

    2008

  32. [40]

    Overcoming exploration in reinforcement learning with demonstrations

    Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pages 6292--6299. IEEE, 2018

    2018

  33. [41]

    Awac: Accelerating online reinforcement learning with offline datasets

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  34. [42]

    Offline neural contextual bandits: Pessimism, optimization and generalization

    Thanh Nguyen-Tang, Sunil Gupta, A Tuan Nguyen, and Svetha Venkatesh. Offline neural contextual bandits: Pessimism, optimization and generalization. arXiv preprint arXiv:2111.13807, 2021

    2021 arXiv

  35. [43]

    Cutting to the chase with warm-start contextual bandits

    Bastian Oetomo, R Malinga Perera, Renata Borovica-Gajic, and Benjamin IP Rubinstein. Cutting to the chase with warm-start contextual bandits. Knowledge and Information Systems, pages 1--33, 2023

    2023

  36. [44]

    Learning complex dexterous manipulation with deep reinforcement learning and demonstrations

    Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087, 2017

    2017 arXiv

  37. [45]

    Bridging offline reinforcement learning and imitation learning: A tale of pessimism

    Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34: 0 11702--11716, 2021

    2021

  38. [46]

    Agnostic system identification for model-based reinforcement learning

    Stephane Ross and J Andrew Bagnell. Agnostic system identification for model-based reinforcement learning. arXiv preprint arXiv:1203.1007, 2012

    2012 arXiv

  39. [47]

    Warm starting bandits with side information from confounded data

    Nihal Sharma, Soumya Basu, Karthikeyan Shanmugam, and Sanjay Shakkottai. Warm starting bandits with side information from confounded data. arXiv preprint arXiv:2002.08405, 2020

    2002 arXiv

  40. [48]

    Multi-armed bandit problems with history

    Pannagadatta Shivaswamy and Thorsten Joachims. Multi-armed bandit problems with history. In Artificial Intelligence and Statistics, pages 1046--1054. PMLR, 2012

    2012

  41. [49]

    User cold-start problem in multi-armed bandits: When the first recommendations guide the user’s experience

    Nicollas Silva, Thiago Silva, Heitor Werneck, Leonardo Rocha, and Adriano Pereira. User cold-start problem in multi-armed bandits: When the first recommendations guide the user’s experience. ACM Transactions on Recommender Systems, 1 0 (1): 0 1--24, 2023

    2023

  42. [50]

    Best-arm identification in linear bandits

    Marta Soare, Alessandro Lazaric, and R \'e mi Munos. Best-arm identification in linear bandits. Advances in Neural Information Processing Systems, 27, 2014

    2014

  43. [51]

    Hybrid rl: Using both offline and online data can make rl efficient

    Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.06718, 2022

    2022 arXiv

  44. [52]

    R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, USA, 1998

    1998

  45. [53]

    Algorithms for reinforcement learning

    Csaba Szepesv \'a ri. Algorithms for reinforcement learning. Springer nature, 2022

    2022

  46. [54]

    A natural extension to online algorithms for hybrid rl with limited coverage

    Kevin Tan and Ziping Xu. A natural extension to online algorithms for hybrid rl with limited coverage. arXiv preprint arXiv:2403.09701, 2024

    2024 arXiv

  47. [55]

    Hybrid reinforcement learning breaks sample size barriers in linear mdps

    Kevin Tan, Wei Fan, and Yuting Wei. Hybrid reinforcement learning breaks sample size barriers in linear mdps. arXiv preprint arXiv:2408.04526, 2024

    2024 arXiv

  48. [56]

    Predictive off-policy policy evaluation for nonstationary decision problems, with applications to digital marketing

    Philip Thomas, Georgios Theocharous, Mohammad Ghavamzadeh, Ishan Durugkar, and Emma Brunskill. Predictive off-policy policy evaluation for nonstationary decision problems, with applications to digital marketing. In Proceedings of the AAAI Conference on Artificial Intelligence,...

    2017

  49. [57]

    On the likelihood that one unknown probability exceeds another in view of the evidence of two samples

    William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25 0 (3-4): 0 285--294, 1933

    1933

  50. [58]

    Gymnasium: A standard interface for reinforcement learning environments

    Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goul \ a o, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024

    2024 arXiv

  51. [59]

    Pessimistic model-based offline reinforcement learning under partial coverage

    Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=tyrJsbKAe6

    2022

  52. [60]

    Representation learning for online and offline rl in low-rank mdps

    Masatoshi Uehara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline rl in low-rank mdps. arXiv preprint arXiv:2110.04652, 2021

    2021 arXiv

  53. [61]

    Instance-dependent near-optimal policy identification in linear mdps via online experiment design

    Andrew Wagenmaker and Kevin G Jamieson. Instance-dependent near-optimal policy identification in linear mdps via online experiment design. Advances in Neural Information Processing Systems, 35: 0 5968--5981, 2022

    2022

  54. [62]

    Leveraging offline data in online reinforcement learning

    Andrew Wagenmaker and Aldo Pacchiano. Leveraging offline data in online reinforcement learning. In International Conference on Machine Learning, pages 35300--35338. PMLR, 2023

    2023

  55. [63]

    Experimental design for regret minimization in linear bandits

    Andrew Wagenmaker, Julian Katz-Samuels, and Kevin Jamieson. Experimental design for regret minimization in linear bandits. In International Conference on Artificial Intelligence and Statistics, pages 3088--3096. PMLR, 2021

    2021

  56. [64]

    Oracle-efficient pessimism: Offline policy optimization in contextual bandits

    Lequn Wang, Akshay Krishnamurthy, and Aleksandrs Slivkins. Oracle-efficient pessimism: Offline policy optimization in contextual bandits. arXiv preprint arXiv:2306.07923, 2023

    2023 arXiv

  57. [65]

    Bellman-consistent pessimism for offline reinforcement learning

    Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34: 0 6683--6694, 2021 a

    2021

  58. [66]

    Policy finetuning: Bridging sample-efficient offline and online reinforcement learning

    Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34: 0 27395--27407, 2021 b

    2021

  59. [67]

    Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game

    Wei Xiong, Han Zhong, Chengshuai Shi, Cong Shen, Liwei Wang, and Tong Zhang. Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game. arXiv preprint arXiv:2205.15512, 2022

    2022 arXiv

  60. [68]

    Minimax optimal fixed-budget best arm identification in linear bandits

    Junwen Yang and Vincent Tan. Minimax optimal fixed-budget best arm identification in linear bandits. Advances in Neural Information Processing Systems, 35: 0 12253--12266, 2022

    2022

  61. [69]

    Offline reinforcement learning for wireless network optimization with mixture datasets

    Kun Yang, Cong Shen, Jing Yang, Shu-ping Yeh, and Jerry Sydir. Offline reinforcement learning for wireless network optimization with mixture datasets. arXiv preprint arXiv:2311.11423, 2023

    2023 arXiv

  62. [70]

    Provable benefits of actor-critic methods for offline reinforcement learning

    Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34: 0 13626--13640, 2021

    2021

  63. [71]

    Warm-starting contextual bandits: Robustly combining supervised and bandit feedback

    Chicheng Zhang, Alekh Agarwal, Hal Daum \'e III, John Langford, and Sahand N Negahban. Warm-starting contextual bandits: Robustly combining supervised and bandit feedback. arXiv preprint arXiv:1901.00301, 2019

    1901 arXiv

  64. [72]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  65. [73]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  66. [74]

    page @startpage numbered @text Submitted to @long ( @short)

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.