REVIEW 2 major objections 4 minor 300 references
Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions
T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read This paper proves that the minimax sample complexity of learning epsilon-optimal robust policies in average-reward MDPs with total-variation uncertainty sets is SA/epsilon^2 times min{H0,Hsigma}, plus an extra SA sigma Hsigma^2/epsilon^2…
desk verdict Strong theoretical package with a real gap: the claimed matching minimax rate is unproven in an intermediate tolerance regime. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the optimal bias span, defined as the minimum span seminorm among solutions of the average-reward Bellman optimality equations $\rho^\star\mathbf{1}+h=T_0h$ and $\rho^{\star,\sigma}\mathbf{1}+h=T_\sigma h$, giving $H_0$ and $H_\sigma$. The argument is a reduction from robust average-reward to robust discounted MDPs: at $\gamma=1-\varepsilon/(3H_\sigma)$, a policy that is $H_\sigma$-suboptimal in discounted value is $\varepsilon$-optimal in robust average reward (Corollary 1). The proof sharpens generic discounted plug-in bounds with a nominal-anchor comparison that isolates $\min\{H_0,H_\sigma\}$, span control of discounted values by $H_\sigma$ (Lemma 3), and span-localized concentration in place of the worst-case horizon $(1-\gamma)^{-1}$. For unknown spans, a first data batch certifies a nominal anchor and a dyadic discount grid with lower-confidence bounds selects the reduction and horizon.
What would settle it
Construct the hard instances of Appendix C with known $H_0,H_\sigma,\sigma$ and compute the minimax risk; if any algorithm succeeds with $NSA$ below the claimed bound by a large constant factor, the lower bound is wrong, and a concrete check is to run the span-agnostic algorithm on the two-state family of Proposition 1 and verify that the empirical per-state-action sample size $N_{95}$ bends at $\varepsilon\sim\sigma H_0$ with slope $+1$ in $\min\{H_0,H_\sigma\}$ above the threshold and an additive dependence linear in $\sigma$ and quadratic in $H_\sigma$ below it.
Extended reading notes
Core claim
The paper's central claim is that the statistical price of distributional robustness in average-reward MDPs is exactly captured by the scale $\sigma H_0$ and the robust bias span $H_\sigma$. Over $(s,a)$-rectangular TV uncertainty sets of radius at most $\sigma$, the minimax number of samples per state-action pair is, up to log factors, $N \asymp \varepsilon^{-2}\min\{H_0,H_\sigma\}$ when $\varepsilon\gtrsim\sigma H_0$, and $N\asymp \varepsilon^{-2}(\min\{H_0,H_\sigma\}+\sigma H_\sigma^2)$ when $\varepsilon\lesssim\sigma H_0$. The lower bound (Theorem 1) decomposes into the linear min-span term inherited from standard AMDPs and a robustness-specific $\sigma H_\sigma^2$ term active in low tolerance; the upper bounds (Theorems 2 and 3) attain both rates by reducing to discounted problems, choosing the nominal or robust reduction and a discount factor from known spans or adaptively from data. The paper also shows the two span parameters are independent (Proposition 1), so both must appear, and that a nominal optimal policy is $\sigma H_0$-optimal for the robust problem (Proposition 2), which is why the threshold appears.
Load-bearing premise
The load-bearing premise is the unichain condition (Assumption 1): for every stationary policy and every transition kernel in the uncertainty set, the induced Markov chain has exactly one recurrent class, so the robust average reward is a single number independent of the starting state; if this fails, the problem formulation itself becomes initial-state dependent and the reduction framework used for all upper and lower bounds no longer applies.
Editorial extensions
If this is right
- When the target tolerance is at least a constant times $\sigma H_0$, robustness is statistically free: planning in the nominal MDP already achieves the minimax rate $SA\min\{H_0,H_\sigma\}/\varepsilon^2$, with the robust reduction winning when $H_\sigma\leq H_0$.
- When $\varepsilon\lesssim\sigma H_0$, an extra $SA\sigma H_\sigma^2/\varepsilon^2$ samples are necessary and sufficient; this is the first demonstration that the low-tolerance regime has a distinct robustness-specific cost.
- If only the robust span $H_\sigma$ is bounded and $H_0$ may be unbounded, the minimax rate collapses to $SA(H_\sigma+\sigma H_\sigma^2)/\varepsilon^2$, and a robust-only variant attains it without knowing the span.
- The new bounds uniformly improve on earlier robust AMDP guarantees, replacing quadratic dependence on larger robust span parameters with $\min\{H_0,H_\sigma\}$ plus the $\sigma H_\sigma^2$ term.
- The matching lower bound shows both terms in the rate are unavoidable, so any algorithm with sample complexity below this rate would contradict minimax optimality.
Reading between the lines
- The authors do not pursue it, but the $\sigma H_0$ threshold suggests a general design principle for other uncertainty geometries (e.g., $L_p$ or Wasserstein balls): robustness is free until the perturbation's value loss is comparable to the allowed suboptimality, with the bias span replaced by a suitable continuity modulus of the value function.
- A testable practical consequence is that simulators should first estimate $H_0$ from a small pilot batch and only switch to robust planning when $\varepsilon/\sigma$ is on the order of $H_0$; this could save orders of magnitude in sample budget in high-tolerance settings.
- The lower-bound construction, which encodes which of two actions has a larger perturbed transition probability, suggests that the $\sigma H_\sigma^2$ term is an action-identification cost; non-rectangular uncertainty sets, where perturbations can be coordinated across states, might yield a different robustness term rather than a simple additive one.
- The lower-confidence-bound selection over a dyadic grid is a generic recipe: any average-reward reduction whose guarantee depends on an unknown horizon or span parameter can be made adaptive at a constant factor by maintaining candidate policies on a geometric grid and choosing by valid confidence penalties.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies distributionally robust average-reward Markov decision processes under (s,a)-rectangular total-variation uncertainty sets, in a generative-model setting. It defines nominal and robust optimal bias spans H_0 and H_sigma, and claims that the minimax total sample complexity is, up to logarithms, (SA/epsilon^2) times min{H_0,H_sigma} in the high-tolerance regime epsilon ≳ sigma H_0, and min{H_0,H_sigma} + sigma H_sigma^2 in the low-tolerance regime epsilon ≲ sigma H_0. The main contributions are a lower bound (Theorem 1), a span-informed plug-in reduction with matching upper bound (Theorem 2), a span-agnostic adaptive procedure (Theorem 3), and simulations supporting the predicted rates. The paper also proves supporting results on robust Bellman equations, span comparison with prior robust span parameters, and a perturbation bound relating nominal and robust optimal rewards.
Significance. If the stated minimax characterization is correct, this is a substantial contribution: it gives the first tight sample-complexity characterization for robust average-reward MDPs and improves on prior quadratic-in-span upper bounds. The central rates are parameter-free up to universal constants, the lower-bound constructions instantiate prescribed span values, and the experimental section varies H_0, H_sigma, and sigma independently and checks the predicted slopes, which are all genuine strengths. The proof structure is extensive, with explicit verification lemmas, a discounted plug-in theorem with localized concentration, and a span-agnostic selection argument. The main reservation is a gap in the regime coverage of the lower bound, described below, which prevents the headline rate (1) from being fully established as stated.
major comments (2)
- [Theorem 1(b) vs Theorem 2(b), Eq. (1)] The claimed minimax rate (1) is not established across the full low-tolerance regime. The robust-specific term sigma H_sigma^2 in the lower bound is proved only under the condition epsilon <= 0.01 sigma H_0, whereas the matching upper bound in Theorem 2(b) is asserted for the complementary condition 7 sigma H_0 > epsilon, i.e., epsilon < 7 sigma H_0. For instances with H_0 < H_sigma and sigma H_sigma^2 >> H_0, the lower bound supplies only Omega(SA H_0/epsilon^2) in the intermediate regime 0.01 sigma H_0 < epsilon < 7 sigma H_0, while the upper bound is O(SA(H_0 + sigma H_sigma^2)/epsilon^2); these differ by a nonconstant factor. Proposition 1 shows that such span pairs are realizable, and the Appendix C hard instances respect the span constraints, so this is not an empty region. Please either extend the lower bound to all epsilon <= c sigma H_0 for a universal constant c, or rescope Eq. (1) and Theorems 1–3 so that the robust-specific lower bound is stated only for epsilon sufficiently far below sigma H_0, with the intermediate regime explicitly left open.
- [Section C.2, combination argument] The proof of Theorem 1 says it suffices to verify the min-span and robustness-specific components separately, but the combination is not complete for the regime in which both components must be active. The robustness-specific construction is run under epsilon <= 0.01 sigma min{H_0,H_sigma}, and the absorption argument in Eq. (26) covers only the case where this condition fails while H_sigma < H_0. The remaining case H_0 < H_sigma with 0.01 sigma H_0 < epsilon < 7 sigma H_0 is not addressed by either construction, so the lower bound does not support the rate displayed in Eq. (1) for that region. This is the same gap as the previous comment, but it is worth making explicit that the issue is in the proof's reduction to two separate sample-size components, not merely in the theorem statement.
minor comments (4)
- [Abstract, display equation] The abstract's display for NSA appears to be missing a closing delimiter for the cases environment; the full-text Eq. (1) is correctly typeset, but the abstract version should be fixed for consistency.
- [Table 1 caption] The caption refers to a dashed outline marking regimes where the robust reduction is used, but no dashed outline is visible in the table as rendered; please add the outline or revise the caption.
- [Section 5.1, Figure 1] The experiments check the low-tolerance rate at isolated values of sigma and H_sigma, but they do not probe the intermediate regime 0.01 sigma H_0 < epsilon < 7 sigma H_0. If the lower-bound gap is closed, an experimental check in that regime would strengthen confidence in the claimed threshold behavior.
- [Notation, Section 2.2] The notation H0 and H_sigma is defined with max{1, ...}, but Proposition 1 is stated for any H_0,H_sigma >= 1; the proof uses values at least one, so this is consistent, but the proposition statement would be cleaner if it explicitly noted that the constructed spans are exactly the prescribed values after the max-with-one convention.
Circularity Check
No significant circularity: the central minimax rates are derived from independent span definitions and explicit hard-instance and plug-in arguments, not from fitted inputs or self-referential reductions.
full rationale
The paper's central sample-complexity claim (Eq. (1)) is not circular with respect to its inputs. The parameters H0 and Hsigma are defined in Eqs. (3a)-(3b) as minimum spans of nominal and robust Bellman optimality equations, independently of any sample-complexity claim. The lower bound in Theorem 1 is established by explicit two-instance constructions whose nominal and robust spans are verified directly in Appendix C (Lemmas 6 and 8), and Proposition 1 shows that arbitrary prescribed span pairs are realizable, so the rate's parameters are not fitted from the result. The upper bounds in Theorems 2 and 3 are proven through reduction-based plug-in analyses with sample-size conditions derived from concentration inequalities; they do not assume Eq. (1). No fitted constant is renamed as a prediction. The paper does cite prior work with overlapping authorship, notably Shi et al. (2026) for TV strong-duality and Bernstein-type concentration lemmas and Li et al. (2024) for discounted DMDP complexity, but these are technical proof tools with stated hypotheses that do not include the target minimax rate, and they are used as proof components rather than as self-supporting assertions of the main theorem. The robust-specific lower-bound term is proven under the condition epsilon <= 0.01 sigma H0, while Eq. (1) displays it for epsilon <= sigma H0; to the extent this leaves an intermediate regime unproven, that is a completeness or correctness question, not circularity: the lower bound does not redefine the sample complexity nor presume the upper bound. Accordingly, no circular steps are identified.
Assumptions & free parameters
assumptions (5)
- domain assumption Assumption 1: for every stationary policy and every kernel in the uncertainty set, the induced Markov chain has one recurrent class (unichain).
- domain assumption (s,a)-rectangular total-variation uncertainty sets with local radii bounded by sigma.
- domain assumption Generative model providing N independent next-state samples per state-action pair from the nominal kernel, with known rewards.
- standard math Standard finite-state unichain MDP theory, including Poisson equations, deviation matrices, and compactness arguments for Bellman solutions.
- domain assumption Access to a discounted robust MDP solver that returns a policy with value error at most epsilon_opt.
Cite this review
Pith. "Pith review of Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions." pith.science (2026). https://pith.science/paper/DOMSK7LW
@misc{pith2026260806545,
author = {Pith},
title = {Pith review of: Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions},
year = {2026},
howpublished = {\url{https://pith.science/paper/DOMSK7LW}},
note = {Machine review of arXiv:2608.06545}
}
abstract
Distributionally robust Markov decision processes provide a principled framework for sequential decision making under model uncertainty. We study how many samples are necessary and sufficient to learn an $\varepsilon$-optimal robust policy under the average-reward criterion. A generative model provides samples from the nominal transition kernel, whereas policy performance is evaluated over $(s,a)$-rectangular total-variation uncertainty sets of radius at most $\sigma$. Let $H_0$ and $H_\sigma$ denote the nominal and robust optimal bias spans, respectively. We identify $\sigma H_0$ as the perturbation scale separating high- and low-tolerance regimes. Our matching upper and lower bounds show that, up to logarithmic factors, the minimax total sample complexity is $$ NSA \asymp \frac{SA}{\varepsilon^2}\begin{cases} \min\{H_0,H_\sigma\}, & \varepsilon\gtrsim\sigma H_0,\\ \min\{H_0,H_\sigma\}+\sigma H_\sigma^2, & \varepsilon\lesssim\sigma H_0. \end{cases} $$ Here $S$ and $A$ are the numbers of states and actions, and $N$ is the number of samples per state-action pair. The sample complexity consists of a linear-span term that resembles the nominal AMDP results and a robustness-specific term that appears only in the low-tolerance regime. We attain these rates using reduction-based plug-in procedures that select the reduction---nominal or robust---and its discount factor: a span-informed procedure that makes these choices using known span parameters, and a span-agnostic procedure that calibrates both choices from data.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Towards Tight Bounds on the Sample Complexity of Average-Reward
Jin, Yujia and Sidford, Aaron , booktitle =. Towards Tight Bounds on the Sample Complexity of Average-Reward. 2021 , organization =
2021
-
[2]
Foundations and Trends in Machine Learning , volume =
Continual Learning as Computationally Constrained Reinforcement Learning , author =. Foundations and Trends in Machine Learning , volume =. 2025 , doi =
2025
-
[3]
The curious price of distributional robustness in reinforcement learning with a generative model , author =. Operations Research , year =. doi:10.1287/opre.2025.2240 , note =
-
[4]
Near Sample-Optimal Reduction-Based Policy Learning for Average Reward
Wang, Jinghan and Wang, Mengdi and Yang, Lin F , journal =. Near Sample-Optimal Reduction-Based Policy Learning for Average Reward. 2022 , url =
2022
-
[5]
Span-Based Optimal Sample Complexity for Weakly Communicating and General Average Reward
Zurek, Matthew and Chen, Yudong , journal =. Span-Based Optimal Sample Complexity for Weakly Communicating and General Average Reward
-
[6]
and Tewari, Ambuj , booktitle =
Bartlett, Peter L. and Tewari, Ambuj , booktitle =. 2009 , publisher =
2009
-
[7]
Proceedings of the 35th International Conference on Machine Learning , pages =
Efficient Bias-Span-Constrained Exploration-Exploitation in Reinforcement Learning , author =. Proceedings of the 35th International Conference on Machine Learning , pages =. 2018 , volume =
2018
-
[8]
Proceedings of the 42nd International Conference on Machine Learning , pages =
A Reduction Framework for Distributionally Robust Reinforcement Learning under Average Reward , author =. Proceedings of the 42nd International Conference on Machine Learning , pages =. 2025 , volume =
2025
Show all 300 references
-
[9]
arXiv preprint arXiv:2505.12462 , year =
Model-Free Robust Average-Reward Reinforcement Learning with Sample Complexity Analysis , author =. arXiv preprint arXiv:2505.12462 , year =
-
[10]
High-Dimensional Statistics: A Non-Asymptotic Viewpoint , year =
Wainwright, Martin J , publisher =. High-Dimensional Statistics: A Non-Asymptotic Viewpoint , year =
-
[11]
arXiv preprint arXiv:2603.00945 , year =
Shengbo Wang and Nian Si , title =. arXiv preprint arXiv:2603.00945 , year =
-
[12]
Sample Complexity of Distributionally Robust Average-Reward Reinforcement Learning , year =
Chen, Zijun and Wang, Shengbo and Si, Nian , journal =. Sample Complexity of Distributionally Robust Average-Reward Reinforcement Learning , year =
-
[13]
Efficiently Solving
Jin, Yujia and Sidford, Aaron , booktitle =. Efficiently Solving. 2020 , organization =
2020
-
[14]
The Twelfth International Conference on Learning Representations , year =
Optimal Sample Complexity for Average Reward Markov Decision Processes , author =. The Twelfth International Conference on Learning Representations , year =
-
[15]
The Plugin Approach for Average-Reward and Discounted
Matthew Zurek and Yudong Chen , booktitle =. The Plugin Approach for Average-Reward and Discounted. 2025 , url =
2025
-
[16]
Span-Agnostic Optimal Sample Complexity and Oracle Inequalities for Average-Reward
Zurek, Matthew and Chen, Yudong , booktitle =. Span-Agnostic Optimal Sample Complexity and Oracle Inequalities for Average-Reward. 2025 , volume =
2025
-
[17]
Sharper Model-Free Reinforcement Learning for Average-Reward
Zhang, Zihan and Xie, Qiaomin , booktitle =. Sharper Model-Free Reinforcement Learning for Average-Reward. 2023 , organization =
2023
-
[18]
Journal of Machine Learning Research , volume =
Near-optimal regret bounds for reinforcement learning , author =. Journal of Machine Learning Research , volume =. 2010 , url =
2010
-
[19]
Model-free Reinforcement Learning in Infinite-horizon Average-reward
Wei, Chen-Yu and Jahromi, Mehdi Jafarnia and Luo, Haipeng and Sharma, Hiteshi and Jain, Rahul , booktitle =. Model-free Reinforcement Learning in Infinite-horizon Average-reward. 2020 , volume =
2020
-
[20]
Learning Infinite-horizon Average-reward
Wei, Chen-Yu and Jahromi, Mehdi Jafarnia and Luo, Haipeng and Jain, Rahul , booktitle =. Learning Infinite-horizon Average-reward. 2021 , volume =
2021
-
[21]
Efficient
Xu, Yang and Ganesh, Swetha and Aggarwal, Vaneet , journal =. Efficient. 2026 , note =
2026
-
[22]
Distributionally Robust
Xu, Huan and Mannor, Shie , journal =. Distributionally Robust. 2012 , publisher =
2012
-
[23]
The International Journal of Robotics Research , volume =
Reinforcement learning in robotics: A survey , author =. The International Journal of Robotics Research , volume =. 2013 , publisher =
2013
-
[24]
Nature , volume =
Human-Level Control through Deep Reinforcement Learning , author =. Nature , volume =
-
[25]
Mastering the Game of
Silver, David and Huang, Aja and Maddison, Chris J and Guez, Arthur and Sifre, Laurent and Van Den Driessche, George and Schrittwieser, Julian and Antonoglou, Ioannis and Panneershelvam, Veda and Lanctot, Marc and others , journal =. Mastering the Game of. 2016 , publisher =
2016
-
[26]
International Conference on Artificial Intelligence and Statistics , pages =
Distributionally robust model-based reinforcement learning with large state spaces , author =. International Conference on Artificial Intelligence and Statistics , pages =. 2024 , organization =
2024
-
[27]
2020 , organization =
Sinha, Aman and O'Kelly, Matthew and Zheng, Hongrui and Mangharam, Rahul and Duchi, John and Tedrake, Russ , booktitle =. 2020 , organization =
2020
-
[28]
Advances in Neural Information Processing Systems , volume =
Robust reinforcement learning using offline data , author =. Advances in Neural Information Processing Systems , volume =
-
[29]
International Conference on Artificial Intelligence and Statistics , pages =
Sample complexity of robust reinforcement learning with a generative model , author =. International Conference on Artificial Intelligence and Statistics , pages =. 2022 , organization =
2022
-
[30]
COLT 2009 - The 22nd Conference on Learning Theory , year =
Empirical Bernstein bounds and sample variance penalization , author =. COLT 2009 - The 22nd Conference on Learning Theory , year =
2009
-
[31]
Data-Driven Distributionally Robust Optimization Using the
Mohajerin Esfahani, Peyman and Kuhn, Daniel , journal =. Data-Driven Distributionally Robust Optimization Using the. 2018 , number =
2018
-
[32]
Distributionally robust convex optimization , year =
Wiesemann, Wolfram and Kuhn, Daniel and Sim, Melvyn , journal =. Distributionally robust convex optimization , year =
-
[33]
Distributionally robust optimization and its tractable approximations , year =
Goh, Joel and Sim, Melvyn , journal =. Distributionally robust optimization and its tractable approximations , year =
-
[34]
Learning models with uniform performance via distributionally robust optimization , year =
Duchi, John C and Namkoong, Hongseok , journal =. Learning models with uniform performance via distributionally robust optimization , year =
-
[35]
Robust Average-Reward
Wang, Yue and Velasquez, Alvaro and Atia, George and Prater-Bennette, Ashley and Zou, Shaofeng , journal =. Robust Average-Reward. 2023 , number =
2023
-
[36]
and Prater-Bennette, Ashley and Zou, Shaofeng , booktitle =
Wang, Yue and Velasquez, Alvaro and Atia, George K. and Prater-Bennette, Ashley and Zou, Shaofeng , booktitle =. Model-Free Robust Average-Reward Reinforcement Learning , year =
-
[37]
Toward Theoretical Understandings of Robust
Yang, Wenhao and Zhang, Liangyu and Zhang, Zhihua , journal =. Toward Theoretical Understandings of Robust. 2022 , number =
2022
-
[38]
Sample Complexity of Variance-Reduced Distributionally Robust
Wang, Shengbo and Si, Nian and Blanchet, Jose and Zhou, Zhengyuan , journal =. Sample Complexity of Variance-Reduced Distributionally Robust. 2024 , number =
2024
-
[39]
Near-Optimal Distributionally Robust Reinforcement Learning with General
Clavier, Pierre and Shi, Laixi and Le Pennec, Erwan and Mazumdar, Eric and Wierman, Adam and Geist, Matthieu , journal =. Near-Optimal Distributionally Robust Reinforcement Learning with General. 2024 , pages =
2024
-
[40]
Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity , year =
Shi, Laixi and Chi, Yuejie , journal =. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity , year =
-
[41]
Sample Complexity of Offline Distributionally Robust Linear
Wang, He and Shi, Laixi and Chi, Yuejie , journal =. Sample Complexity of Offline Distributionally Robust Linear. 2024 , volume =
2024
-
[42]
1994 , publisher =
Markov Decision Processes: Discrete Stochastic Dynamic Programming , author =. 1994 , publisher =
1994
-
[43]
Tsybakov , publisher =
Alexandre B. Tsybakov , publisher =. Introduction to Nonparametric Estimation , year =
-
[44]
Machine learning , volume=
Convergence results for single-step on-policy reinforcement-learning algorithms , author=. Machine learning , volume=. 2000 , publisher=
2000
-
[45]
International Conference on Machine Learning , pages=
Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertainty , author=. International Conference on Machine Learning , pages=. 2024 , organization=
2024
-
[46]
arXiv preprint arXiv:2505.12202 , year=
Near-Optimal Sample Complexities of Divergence-based S -rectangular Distributionally Robust Reinforcement Learning , author=. arXiv preprint arXiv:2505.12202 , year=
-
[47]
Forty-second International Conference on Machine Learning , year=
Breaking the Curse of Multiagency in Robust Multi-Agent Reinforcement Learning , author=. Forty-second International Conference on Machine Learning , year=
-
[48]
The blessing of heterogeneity in federated
Woo, Jiin and Joshi, Gauri and Chi, Yuejie , journal=. The blessing of heterogeneity in federated
-
[49]
Advances in Neural Information Processing Systems , volume=
Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage , author=. Advances in Neural Information Processing Systems , volume=
-
[50]
Thirty-seventh Conference on Neural Information Processing Systems , year=
Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation , author=. Thirty-seventh Conference on Neural Information Processing Systems , year=
-
[51]
Operations research , volume=
Distributionally robust optimization under moment uncertainty with application to data-driven problems , author=. Operations research , volume=. 2010 , publisher=
2010
-
[52]
Robust control of
Nilim, Arnab and El Ghaoui, Laurent , journal=. Robust control of. 2005 , publisher=
2005
-
[53]
arXiv preprint arXiv:2006.09118 , year=
Q -learning with Logarithmic Regret , author=. arXiv preprint arXiv:2006.09118 , year=
2006 arXiv
-
[54]
arXiv preprint arXiv:2007.03760 , year=
Near Optimal Provable Uniform Convergence in Off-Policy Evaluation for Reinforcement Learning , author=. arXiv preprint arXiv:2007.03760 , year=
2007 arXiv
-
[55]
Advances in neural information processing systems , pages=
A convergent form of approximate policy iteration , author=. Advances in neural information processing systems , pages=
-
[56]
arXiv preprint arXiv:2002.10043 , year=
Complete Dictionary Learning via _p -norm Maximization , author=. arXiv preprint arXiv:2002.10043 , year=
2002 arXiv
-
[57]
Journal of Applied Probability , volume=
Sensitivity and convergence of uniformly ergodic Markov chains , author=. Journal of Applied Probability , volume=. 2005 , publisher=
2005
-
[58]
arXiv preprint arXiv:2005.03557 , year=
Non-asymptotic Convergence Analysis of Two Time-scale (Natural) Actor-Critic Algorithms , author=. arXiv preprint arXiv:2005.03557 , year=
2005 arXiv
-
[59]
Advances in Neural Information Processing Systems , volume=
Improving sample complexity bounds for (natural) actor-critic algorithms , author=. Advances in Neural Information Processing Systems , volume=
-
[60]
arXiv preprint arXiv:2005.06392 , year=
On the Global Convergence Rates of Softmax Policy Gradient Methods , author=. arXiv preprint arXiv:2005.06392 , year=
2005 arXiv
-
[61]
ICML , volume=
Approximately optimal approximate reinforcement learning , author=. ICML , volume=
-
[62]
Optimality and approximation with policy gradient methods in
Agarwal, Alekh and Kakade, Sham M and Lee, Jason D and Mahajan, Gaurav , journal=. Optimality and approximation with policy gradient methods in
-
[63]
Advances in neural information processing systems , volume=
A natural policy gradient , author=. Advances in neural information processing systems , volume=
-
[64]
Advances in neural information processing systems , pages=
Policy gradient methods for reinforcement learning with function approximation , author=. Advances in neural information processing systems , pages=
-
[65]
Rummery, Gavin A and Niranjan, Mahesan , volume=. On-line. 1994 , publisher=
1994
-
[66]
arXiv preprint arXiv:2102.07301 , year=
Nearly Minimax Optimal Regret for Learning Infinite-horizon Average-reward MDPs with Linear Function Approximation , author=. arXiv preprint arXiv:2102.07301 , year=
-
[67]
Probability Theory and Related Fields , volume=
Information geometry and sufficient statistics , author=. Probability Theory and Related Fields , volume=. 2015 , publisher=
2015
-
[68]
Proceedings of the 27th international conference on international conference on machine learning , pages=
Learning fast approximations of sparse coding , author=. Proceedings of the 27th international conference on international conference on machine learning , pages=
-
[69]
International Conference on Learning Representations (ICLR) , year=
ALISTA: Analytic weights are as good as learned weights in LISTA , author=. International Conference on Learning Representations (ICLR) , year=
-
[70]
arXiv preprint arXiv:1808.10038 , year=
Theoretical linear convergence of unfolded ISTA and its practical weights and thresholds , author=. arXiv preprint arXiv:1808.10038 , year=
-
[71]
arXiv preprint arXiv:2001.08456 , year=
Ada-lista: Learned solvers adaptive to varying models , author=. arXiv preprint arXiv:2001.08456 , year=
2001 arXiv
-
[72]
arXiv preprint arXiv:1609.00285 , year=
Understanding trainable sparse coding via matrix factorization , author=. arXiv preprint arXiv:1609.00285 , year=
-
[73]
International Conference on Artificial Intelligence and Statistics , pages=
Finite-Sample Regret Bound for Distributionally Robust Offline Tabular Reinforcement Learning , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2021 , organization=
2021
-
[74]
Mathematics of Operations Research , volume=
Robust dynamic programming , author=. Mathematics of Operations Research , volume=. 2005 , publisher=
2005
-
[75]
Advances in Neural Information Processing Systems , volume=
Online Robust Reinforcement Learning with Model Uncertainty , author=. Advances in Neural Information Processing Systems , volume=
-
[76]
Twice regularized
Derman, Esther and Geist, Matthieu and Mannor, Shie , journal=. Twice regularized
-
[77]
International Conference on Machine Learning , pages=
Distributionally robust policy evaluation and learning in offline contextual bandits , author=. International Conference on Machine Learning , pages=. 2020 , organization=
2020
-
[78]
arXiv preprint arXiv:2212.06355 , year=
A review of off-policy evaluation in reinforcement learning , author=. arXiv preprint arXiv:2212.06355 , year=
-
[79]
International Conference on Machine Learning , pages=
Robust reinforcement learning using least squares policy iteration with provable performance guarantees , author=. International Conference on Machine Learning , pages=. 2021 , organization=
2021
-
[80]
arXiv preprint arXiv:1908.05659 , year=
Distributionally robust optimization: A review , author=. arXiv preprint arXiv:1908.05659 , year=
1908 arXiv
-
[81]
Finite-sample guarantees for
Gao, Rui , journal=. Finite-sample guarantees for. 2023 , publisher=
2023
-
[82]
arXiv preprint arXiv:2201.12440 , year=
Certifying Model Accuracy under Distribution Shifts , author=. arXiv preprint arXiv:2201.12440 , year=
-
[83]
Advances in neural information processing systems , volume=
Robustness in markov decision problems with uncertain transition matrices , author=. Advances in neural information processing systems , volume=
-
[84]
arXiv preprint arXiv:2204.05275 , year=
Settling the sample complexity of model-based offline reinforcement learning , author=. arXiv preprint arXiv:2204.05275 , year=
-
[85]
Available at Optimization Online , pages=
Kullback-Leibler divergence constrained distributionally robust optimization , author=. Available at Optimization Online , pages=
-
[86]
Pessimistic
Shi, Laixi and Li, Gen and Wei, Yuting and Chen, Yuxin and Chi, Yuejie , booktitle =. Pessimistic. 2022 , volume =
2022
-
[87]
The Bell system technical journal , volume=
A comparison of signalling alphabets , author=. The Bell system technical journal , volume=. 1952 , publisher=
1952
-
[88]
The mathematics of data , volume=
Introductory lectures on stochastic optimization , author=. The mathematics of data , volume=
-
[89]
The Journal of finance , volume=
Reinforcement learning and savings behavior , author=. The Journal of finance , volume=. 2009 , publisher=
2009
-
[90]
, author=
Finding locally optimal, collision-free trajectories with sequential convex optimization. , author=. Robotics: science and systems , volume=. 2013 , organization=
2013
-
[91]
Mathematical Programming , volume=
Data-driven robust optimization , author=. Mathematical Programming , volume=. 2018 , publisher=
2018
-
[92]
Mathematics of Operations Research , volume=
Quantifying distributional model risk via optimal transport , author=. Mathematics of Operations Research , volume=. 2019 , publisher=
2019
-
[93]
Robust control of uncertain
Wolff, Eric M and Topcu, Ufuk and Murray, Richard M , booktitle=. Robust control of uncertain. 2012 , organization=
2012
-
[94]
Wiesemann, Wolfram and Kuhn, Daniel and Rustem, Ber. Robust. Mathematics of Operations Research , volume=. 2013 , publisher=
2013
-
[95]
INFORMS Journal on Computing , volume=
Robust modified policy iteration , author=. INFORMS Journal on Computing , volume=. 2013 , publisher=
2013
-
[96]
Ho, Chin Pang and Petrik, Marek and Wiesemann, Wolfram , booktitle=. Fast. 2018 , organization=
2018
-
[97]
arXiv preprint arXiv:1902.08708 , year=
Distributionally robust reinforcement learning , author=. arXiv preprint arXiv:1902.08708 , year=
1902 arXiv
-
[98]
Journal of Machine Learning Research , volume=
Partial policy iteration for L1-robust Markov decision processes , author=. Journal of Machine Learning Research , volume=
-
[99]
Goyal, Vineet and Grand-Clement, Julien , journal=. Robust. 2023 , publisher=
2023
-
[100]
arXiv preprint arXiv:2003.02894 , year=
Distributional robustness and regularization in reinforcement learning , author=. arXiv preprint arXiv:2003.02894 , year=
2003 arXiv
-
[101]
Scaling up robust
Tamar, Aviv and Mannor, Shie and Xu, Huan , booktitle=. Scaling up robust. 2014 , organization=
2014
-
[102]
2001 , publisher=
Fixed point theory and applications , author=. 2001 , publisher=
2001
-
[103]
Conference on Learning Theory , pages=
Model-based reinforcement learning with a generative model is minimax optimal , author=. Conference on Learning Theory , pages=. 2020 , organization=
2020
-
[104]
Minimax-optimal multi-agent
Li, Gen and Chi, Yuejie and Wei, Yuting and Chen, Yuxin , journal=. Minimax-optimal multi-agent
-
[105]
Operations Research , volume=
Breaking the sample size barrier in model-based reinforcement learning with a generative model , author=. Operations Research , volume=. 2024 , publisher=
2024
-
[106]
2018 , publisher=
High-dimensional probability: An introduction with applications in data science , author=. 2018 , publisher=
2018
-
[107]
Some notes on concentration for
Sambale, Holger , journal=. Some notes on concentration for
-
[108]
Advances in Neural Information Processing Systems , volume=
Near-Optimal Distributionally Robust Reinforcement Learning with General L_p Norms , author=. Advances in Neural Information Processing Systems , volume=
-
[109]
Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pages=
Minimax group fairness: Algorithms and experiments , author=. Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pages=
2021
-
[110]
International Conference on Machine Learning , pages=
Optidice: Offline policy optimization via stationary distribution correction estimation , author=. International Conference on Machine Learning , pages=. 2021 , organization=
2021
-
[111]
Online Policy Optimization for Robust
Dong, Jing and Li, Jingwei and Wang, Baoxiang and Zhang, Jingzhao , journal=. Online Policy Optimization for Robust
-
[112]
International Conference on Artificial Intelligence and Statistics , pages=
Improved sample complexity bounds for distributionally robust reinforcement learning , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2023 , organization=
2023
-
[113]
Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence , pages=
Towards minimax optimality of model-based robust reinforcement learning , author=. Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence , pages=
-
[114]
arXiv preprint arXiv:2302.01248 , year=
Avoiding Model Estimation in Robust Markov Decision Processes with a Generative Model , author=. arXiv preprint arXiv:2302.01248 , year=
-
[115]
A finite sample complexity bound for distributionally robust
Wang, Shengbo and Si, Nian and Blanchet, Jose and Zhou, Zhengyuan , booktitle=. A finite sample complexity bound for distributionally robust. 2023 , organization=
2023
-
[116]
arXiv preprint arXiv:1909.08593 , year=
Fine-tuning language models from human preferences , author=. arXiv preprint arXiv:1909.08593 , year=
1909 arXiv
-
[117]
arXiv preprint arXiv:1907.09475 , year=
Deep reinforcement learning for clinical decision support: a brief survey , author=. arXiv preprint arXiv:1907.09475 , year=
1907 arXiv
-
[118]
Advances in Neural Information Processing Systems , volume=
Medical dead-ends and learning to identify high-risk states and treatments , author=. Advances in Neural Information Processing Systems , volume=
-
[119]
arXiv preprint arXiv:1312.5602 , year=
Playing atari with deep reinforcement learning , author=. arXiv preprint arXiv:1312.5602 , year=
-
[120]
2023 , publisher=
GPT-4 Technical Report , author=. 2023 , publisher=
2023
-
[121]
International Conference on Machine Learning , pages=
Robust adversarial reinforcement learning , author=. International Conference on Machine Learning , pages=. 2017 , organization=
2017
-
[122]
Conference on robot learning , pages=
Benchmarking reinforcement learning algorithms on real-world robots , author=. Conference on robot learning , pages=. 2018 , organization=
2018
-
[123]
International conference on machine learning , pages=
Trust region policy optimization , author=. International conference on machine learning , pages=. 2015 , organization=
2015
-
[124]
Advances in Neural Information Processing Systems , volume=
Dynamic bottleneck for robust self-supervised exploration , author=. Advances in Neural Information Processing Systems , volume=
-
[125]
arXiv preprint arXiv:2110.06539 , year=
On covariate shift of latent confounders in imitation and reinforcement learning , author=. arXiv preprint arXiv:2110.06539 , year=
-
[126]
arXiv preprint arXiv:2212.02705 , year=
What is the Solution for State Adversarial Multi-Agent Reinforcement Learning? , author=. arXiv preprint arXiv:2212.02705 , year=
-
[127]
ICML 2021 Workshop on Adversarial Machine Learning , year=
Strategically-timed state-observation attacks on deep reinforcement learning agents , author=. ICML 2021 Workshop on Adversarial Machine Learning , year=
2021
-
[128]
arXiv preprint arXiv:2109.08776 , year=
Exploring the Training Robustness of Distributional Reinforcement Learning against Noisy State Observations , author=. arXiv preprint arXiv:2109.08776 , year=
-
[129]
arXiv preprint arXiv:2206.07188 , year=
Defending Observation Attacks in Deep Reinforcement Learning via Detection and Denoising , author=. arXiv preprint arXiv:2206.07188 , year=
-
[130]
arXiv preprint arXiv:2303.06614 , year=
Synthetic Experience Replay , author=. arXiv preprint arXiv:2303.06614 , year=
-
[131]
arXiv preprint arXiv:2303.06818 , year=
Backdoor Defense via Deconfounded Representation Learning , author=. arXiv preprint arXiv:2303.06818 , year=
-
[132]
arXiv preprint arXiv:2011.04065 , year=
Bait and switch: Online training data poisoning of autonomous driving systems , author=. arXiv preprint arXiv:2011.04065 , year=
2011 arXiv
-
[133]
arXiv preprint arXiv:2212.01433 , year=
Avoiding spurious correlations via logit correction , author=. arXiv preprint arXiv:2212.01433 , year=
-
[134]
International Conference on Machine Learning , pages=
Curl: Contrastive unsupervised representations for reinforcement learning , author=. International Conference on Machine Learning , pages=. 2020 , organization=
2020
-
[135]
2009 , publisher=
Causality , author=. 2009 , publisher=
2009
-
[136]
Advances in Neural Information Processing Systems , volume=
Provably efficient causal reinforcement learning with confounded observational data , author=. Advances in Neural Information Processing Systems , volume=
-
[137]
arXiv preprint arXiv:2210.11287 , year=
MoCoDA: Model-based Counterfactual Data Augmentation , author=. arXiv preprint arXiv:2210.11287 , year=
-
[138]
Machine Learning and Knowledge Extraction , volume=
Robust reinforcement learning: A review of foundations and recent advances , author=. Machine Learning and Knowledge Extraction , volume=. 2022 , publisher=
2022
-
[139]
International Conference on Machine Learning , pages=
Action robust reinforcement learning and applications in continuous control , author=. International Conference on Machine Learning , pages=. 2019 , organization=
2019
-
[140]
2020 American control conference (ACC) , pages=
Robustifying reinforcement learning agents via action space adversarial training , author=. 2020 American control conference (ACC) , pages=. 2020 , organization=
2020
-
[141]
arXiv preprint arXiv:2101.08452 , year=
Robust reinforcement learning on state observations with learned optimal adversary , author=. arXiv preprint arXiv:2101.08452 , year=
-
[142]
arXiv preprint arXiv:2209.10579 , year=
First-order policy optimization for robust Markov decision process , author=. arXiv preprint arXiv:2209.10579 , year=
-
[143]
arXiv preprint arXiv:2301.13589 , year=
Policy Gradient for s-Rectangular Robust Markov Decision Processes , author=. arXiv preprint arXiv:2301.13589 , year=
-
[144]
arXiv preprint arXiv:2306.11626 , year=
Regularized Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexity , author=. arXiv preprint arXiv:2306.11626 , year=
-
[145]
arXiv preprint arXiv:2309.02236 , year=
Distributionally Robust Model-based Reinforcement Learning with Large State Spaces , author=. arXiv preprint arXiv:2309.02236 , year=
-
[146]
Distributionally Robust
Liu, Zijian and Bai, Qinxun and Blanchet, Jose and Dong, Perry and Xu, Wei and Zhou, Zhengqing and Zhou, Zhengyuan , booktitle=. Distributionally Robust. 2022 , organization=
2022
-
[147]
arXiv preprint arXiv:2301.11721 , year=
Single-Trajectory Distributionally Robust Reinforcement Learning , author=. arXiv preprint arXiv:2301.11721 , year=
-
[148]
arXiv preprint arXiv:2209.06620 , year=
Distributionally robust offline reinforcement learning with linear function approximation , author=. arXiv preprint arXiv:2209.06620 , year=
-
[149]
arXiv preprint arXiv:2311.09018 , year=
On the Foundation of Distributionally Robust Reinforcement Learning , author=. arXiv preprint arXiv:2311.09018 , year=
-
[150]
arXiv preprint arXiv:2310.04323 , year=
Adjustable Robust Reinforcement Learning for Online 3D Bin Packing , author=. arXiv preprint arXiv:2310.04323 , year=
-
[151]
arXiv preprint arXiv:2307.15890 , year=
First-order Policy Optimization for Robust Policy Evaluation , author=. arXiv preprint arXiv:2307.15890 , year=
-
[152]
Annals of statistics , volume=
Batch policy learning in average reward markov decision processes , author=. Annals of statistics , volume=. 2022 , publisher=
2022
-
[153]
IEEE Transactions on Automation Science and Engineering , volume=
Robot packing with known items and nondeterministic arrival order , author=. IEEE Transactions on Automation Science and Engineering , volume=. 2020 , publisher=
2020
-
[154]
International Transactions in Operational Research , volume=
A greedy search for the three-dimensional bin packing problem: the packing static stability case , author=. International Transactions in Operational Research , volume=. 2003 , publisher=
2003
-
[155]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Online 3D bin packing with constrained deep reinforcement learning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[156]
International conference on learning representations , year=
Learning efficient online 3D bin packing on packing configuration trees , author=. International conference on learning representations , year=
-
[157]
Management Science , volume=
Adaptive distributionally robust optimization , author=. Management Science , volume=. 2019 , publisher=
2019
-
[158]
arXiv preprint arXiv:2310.18434 , year=
Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage , author=. arXiv preprint arXiv:2310.18434 , year=
-
[159]
Operations Research , volume=
Scalable reinforcement learning for multiagent networked systems , author=. Operations Research , volume=. 2022 , publisher=
2022
-
[160]
Operations Research , volume=
Adaptive execution: Exploration and learning of price impact , author=. Operations Research , volume=. 2015 , publisher=
2015
-
[161]
Mathematics of Operations Research , volume=
Distributionally robust Markov decision processes and their connection to risk measures , author=. Mathematics of Operations Research , volume=. 2022 , publisher=
2022
-
[162]
Mathematics of Operations Research , volume=
Robust MDPs with k-rectangular uncertainty , author=. Mathematics of Operations Research , volume=. 2016 , publisher=
2016
-
[163]
Operations Research , volume=
Distributionally robust optimization with infinitely constrained ambiguity sets , author=. Operations Research , volume=. 2019 , publisher=
2019
-
[164]
Operations Research , volume=
Recovering best statistical guarantees via the empirical divergence-based distributionally robust optimization , author=. Operations Research , volume=. 2019 , publisher=
2019
-
[165]
arXiv preprint arXiv:2404.03578 , year=
Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithm , author=. arXiv preprint arXiv:2404.03578 , year=
-
[166]
Science advances , volume=
Adapting to disruptions: Managing supply chain resilience through product rerouting , author=. Science advances , volume=. 2024 , publisher=
2024
-
[167]
Sample complexity of variance-reduced distributionally robust
Wang, Shengbo and Si, Nian and Blanchet, Jose and Zhou, Zhengyuan , journal=. Sample complexity of variance-reduced distributionally robust
-
[168]
Advances in Neural Information Processing Systems , volume=
Distributionally robust Markov decision processes , author=. Advances in Neural Information Processing Systems , volume=
-
[169]
Operations research , volume=
Combined pricing and inventory control under uncertainty , author=. Operations research , volume=. 1999 , publisher=
1999
-
[170]
3rd Student Conference on Operational Research (2012) , pages=
Revenue maximization through dynamic pricing under unknown market behaviour , author=. 3rd Student Conference on Operational Research (2012) , pages=. 2012 , organization=
2012
-
[171]
Manufacturing & Service Operations Management , volume=
An overview of pricing models for revenue management , author=. Manufacturing & Service Operations Management , volume=. 2003 , publisher=
2003
-
[172]
arXiv preprint arXiv:2404.00940 , year=
Sequential decision-making under uncertainty: A robust mdps review , author=. arXiv preprint arXiv:2404.00940 , year=
-
[173]
The Annals of Statistics , pages=
Convergence of estimates under dimensionality restrictions , author=. The Annals of Statistics , pages=. 1973 , publisher=
1973
-
[174]
Proceedings of the National Academy of Sciences of the United States of America , volume=
On the theory of dynamic programming , author=. Proceedings of the National Academy of Sciences of the United States of America , volume=. 1952 , publisher=
1952
-
[175]
2025 , publisher=
Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Zhang, Ruoyu and Ma, Shirong and Bi, Xiao and others , journal=. 2025 , publisher=
2025
-
[176]
arXiv preprint arXiv:2403.12338 , year=
Stochastic halpern iteration in normed spaces and applications to reinforcement learning , author=. arXiv preprint arXiv:2403.12338 , year=
-
[177]
Span-Agnostic Optimal Sample Complexity and Oracle Inequalities for Average-Reward
Zurek, Matthew and Chen, Yudong , booktitle =. Span-Agnostic Optimal Sample Complexity and Oracle Inequalities for Average-Reward. 2025 , editor =
2025
-
[178]
Advances in Neural Information Processing Systems , volume=
On gap-dependent bounds for offline reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=
-
[179]
Near-Optimal Sample Complexity for
Lee, Jongmin and Bravo, Mario and Cominetti, Roberto , booktitle =. Near-Optimal Sample Complexity for. 2025 , volume =
2025
-
[180]
SIAM Journal on Mathematics of Data Science , volume=
Is temporal difference learning optimal? an instance-dependent analysis , author=. SIAM Journal on Mathematics of Data Science , volume=. 2021 , publisher=
2021
-
[181]
On Convergence of Average-Reward
Wan, Yi and Yu, Huizhen and Sutton, Richard S , journal=. On Convergence of Average-Reward
-
[182]
Non-asymptotic guarantees for average-reward
Chen, Zaiwei , journal=. Non-asymptotic guarantees for average-reward
-
[183]
Feasible
Jin, Ying and Gummadi, Ramki and Zhou, Zhengyuan and Blanchet, Jose , booktitle=. Feasible. 2024 , organization=
2024
-
[184]
Federated
Zheng, Zhong and Gao, Fengyu and Xue, Lingzhou and Yang, Jing , booktitle=. Federated
-
[185]
Distributed
Liu, Rui and Olshevsky, Alex , journal=. Distributed. 2023 , publisher=
2023
-
[186]
2023 , publisher=
Sivashangaran, Shathushan and Khairnar, Apoorva and Eskandarian, Azim , journal=. 2023 , publisher=
2023
-
[187]
Field and Service Robotics: Results of the 11th International Conference , pages=
Airsim: High-fidelity visual and physical simulation for autonomous vehicles , author=. Field and Service Robotics: Results of the 11th International Conference , pages=. 2018 , organization=
2018
-
[188]
arXiv preprint arXiv:1910.08639 , year=
Offworld gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research , author=. arXiv preprint arXiv:1910.08639 , year=
1910 arXiv
-
[189]
arXiv preprint arXiv:2310.06114 , volume=
Learning interactive real-world simulators , author=. arXiv preprint arXiv:2310.06114 , volume=
-
[190]
Journal of artificial intelligence research , volume=
The arcade learning environment: An evaluation platform for general agents , author=. Journal of artificial intelligence research , volume=
-
[191]
The Twelfth International Conference on Learning Representations , year=
Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning , author=. The Twelfth International Conference on Learning Representations , year=
-
[192]
One-Shot Averaging for Distributed
Tian, Haoxing and Paschalidis, Ioannis Ch and Olshevsky, Alex , journal=. One-Shot Averaging for Distributed. 2024 , publisher=
2024
-
[193]
Federated Reinforcement Learning: Linear Speedup Under
Khodadadian, Sajad and Sharma, Pranay and Joshi, Gauri and Maguluri, Siva Theja , booktitle =. Federated Reinforcement Learning: Linear Speedup Under
-
[194]
Forty-first International Conference on Machine Learning , year=
Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices , author=. Forty-first International Conference on Machine Learning , year=
-
[195]
The blessing of heterogeneity in federated
Woo, Jiin and Joshi, Gauri and Chi, Yuejie , booktitle=. The blessing of heterogeneity in federated. 2023 , organization=
2023
-
[196]
Sample Complexity of Average-Reward
Jiao, Yuchen and Woo, Jiin and Li, Gen and Joshi, Gauri and Chi, Yuejie , journal=. Sample Complexity of Average-Reward. 2026 , url=
2026
-
[197]
Proceedings of the Fourth Connectionist Models Summer School , year=
Issues in using function approximation for reinforcement learning , author=. Proceedings of the Fourth Connectionist Models Summer School , year=
-
[198]
Operations Research , volume=
A Finite Time Analysis of Temporal Difference Learning with Linear Function Approximation , author=. Operations Research , volume=. 2021 , publisher=
2021
-
[199]
Foundations and Trends
Spectral methods for data science: A statistical perspective , author=. Foundations and Trends. 2021 , publisher=
2021
-
[200]
The Plug-in Approach for Average-Reward and Discounted
Zurek, Matthew and Chen, Yudong , booktitle=. The Plug-in Approach for Average-Reward and Discounted
-
[201]
Yu, Tianhe and Thomas, Garrett and Yu, Lantao and Ermon, Stefano and Zou, James Y and Levine, Sergey and Finn, Chelsea and Ma, Tengyu , journal=
-
[202]
Advances in Neural Information Processing Systems , volume=
Reward-agnostic fine-tuning: Provable statistical benefits of hybrid reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=
-
[203]
arXiv preprint arXiv:2304.07278 , year=
Minimax-optimal reward-agnostic exploration in reinforcement learning , author=. arXiv preprint arXiv:2304.07278 , year=
-
[204]
Conference on Learning Theory , pages=
Is reinforcement learning more difficult than bandits? a near-optimal algorithm escaping the curse of horizon , author=. Conference on Learning Theory , pages=. 2021 , organization=
2021
-
[205]
arXiv preprint arXiv:2307.13586 , year=
Settling the sample complexity of online reinforcement learning , author=. arXiv preprint arXiv:2307.13586 , year=
-
[206]
Advances in neural information processing systems , volume=
Logarithmic online regret bounds for undiscounted reinforcement learning , author=. Advances in neural information processing systems , volume=
-
[207]
International Conference on Machine Learning , pages=
Tightening exploration in upper confidence reinforcement learning , author=. International Conference on Machine Learning , pages=. 2020 , organization=
2020
-
[208]
2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages=
Optimism in reinforcement learning and Kullback-Leibler divergence , author=. 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages=. 2010 , organization=
2010
-
[209]
Improved analysis of
Fruit, Ronan and Pirotta, Matteo and Lazaric, Alessandro , journal=. Improved analysis of
-
[210]
International Conference on Machine Learning , pages=
Minimax regret bounds for reinforcement learning , author=. International Conference on Machine Learning , pages=. 2017 , organization=
2017
-
[211]
Advances in neural information processing systems , volume=
Near-optimal regret bounds for reinforcement learning , author=. Advances in neural information processing systems , volume=
-
[212]
Exploration bonus for regret minimization in discrete and continuous average reward
Qian, Jian and Fruit, Ronan and Pirotta, Matteo and Lazaric, Alessandro , journal=. Exploration bonus for regret minimization in discrete and continuous average reward
-
[213]
Algorithmic Learning Theory , pages=
Variance-aware regret bounds for undiscounted reinforcement learning in mdps , author=. Algorithmic Learning Theory , pages=. 2018 , organization=
2018
-
[214]
Conference on Learning Theory , pages=
Provably efficient reinforcement learning with linear function approximation , author=. Conference on Learning Theory , pages=. 2020 , organization=
2020
-
[215]
Kidambi, Rahul and Rajeswaran, Aravind and Netrapalli, Praneeth and Joachims, Thorsten , journal=
-
[216]
NIPS , volume=
Improved Algorithms for Linear Stochastic Bandits , author=. NIPS , volume=
-
[217]
2018 , edition=
Reinforcement Learning: An Introduction , author=. 2018 , edition=
2018
-
[218]
International Conference on Machine Learning , pages=
Minimax-optimal off-policy evaluation with linear function approximation , author=. International Conference on Machine Learning , pages=. 2020 , organization=
2020
-
[219]
Uncertainty in Artificial Intelligence , pages=
Minimax sample complexity for turn-based stochastic game , author=. Uncertainty in Artificial Intelligence , pages=. 2021 , organization=
2021
-
[220]
Off-policy evaluation via the regularized
Yang, Mengjiao and Nachum, Ofir and Dai, Bo and Li, Lihong and Schuurmans, Dale , journal=. Off-policy evaluation via the regularized
-
[221]
arXiv preprint arXiv:1409.3653 , year=
On minimax optimal offline policy evaluation , author=. arXiv preprint arXiv:1409.3653 , year=
-
[222]
Advances in neural information processing systems , volume=
Provable benefits of actor-critic methods for offline reinforcement learning , author=. Advances in neural information processing systems , volume=
-
[223]
Representation Learning for Online and Offline
Uehara, Masatoshi and Zhang, Xuezhou and Sun, Wen , booktitle=. Representation Learning for Online and Offline
-
[224]
International Conference on Learning Representations , year=
Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism , author=. International Conference on Learning Representations , year=
-
[225]
Advances in neural information processing systems , volume=
Towards instance-optimal offline reinforcement learning with pessimism , author=. Advances in neural information processing systems , volume=
-
[226]
The Annals of Statistics , volume=
Settling the sample complexity of model-based offline reinforcement learning , author=. The Annals of Statistics , volume=. 2024 , publisher=
2024
-
[227]
Concentration inequalities and model selection: Ecole d'Et
Massart, Pascal , year=. Concentration inequalities and model selection: Ecole d'Et
-
[228]
arXiv preprint arXiv:2102.11866 , year=
Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality , author=. arXiv preprint arXiv:2102.11866 , year=
-
[229]
Sample Complexity of Offline Reinforcement Learning with Deep
Nguyen-Tang, Thanh and Gupta, Sunil and Venkatesh, Svetha , journal=. Sample Complexity of Offline Reinforcement Learning with Deep
-
[230]
International Conference on Machine Learning , pages=
Information-theoretic considerations in batch reinforcement learning , author=. International Conference on Machine Learning , pages=. 2019 , organization=
2019
-
[231]
Advances in Neural Information Processing Systems , volume=
Toward the Fundamental Limits of Imitation Learning , author=. Advances in Neural Information Processing Systems , volume=
-
[232]
Is Pessimism Provably Efficient for Offline
Jin, Ying and Yang, Zhuoran and Wang, Zhaoran , booktitle=. Is Pessimism Provably Efficient for Offline
-
[233]
arXiv preprint arXiv:2106.06926 , year=
Bellman-consistent Pessimism for Offline Reinforcement Learning , author=. arXiv preprint arXiv:2106.06926 , year=
-
[234]
International Conference on Learning Representations , year=
The Importance of Pessimism in Fixed-Dataset Policy Optimization , author=. International Conference on Learning Representations , year=
-
[235]
Advances in neural information processing systems , volume=
Policy finetuning: Bridging sample-efficient offline and online reinforcement learning , author=. Advances in neural information processing systems , volume=
-
[236]
IEEE Transactions on Information Theory , volume=
Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism , author=. IEEE Transactions on Information Theory , volume=. 2022 , publisher=
2022
-
[237]
arXiv preprint arXiv:2005.01643 , year=
Offline reinforcement learning: Tutorial, review, and perspectives on open problems , author=. arXiv preprint arXiv:2005.01643 , year=
2005 arXiv
-
[238]
Conservative
Kumar, Aviral and Zhou, Aurick and Tucker, George and Levine, Sergey , booktitle =. Conservative
-
[239]
International Conference on Artificial Intelligence and Statistics , pages=
Q-learning with Logarithmic Regret , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2021 , organization=
2021
-
[240]
arXiv preprint arXiv:2006.11911 , year=
On optimism in model-based reinforcement learning , author=. arXiv preprint arXiv:2006.11911 , year=
2006 arXiv
-
[241]
arXiv preprint arXiv:1905.11527 , year=
Tight regret bounds for model-based reinforcement learning with greedy policies , author=. arXiv preprint arXiv:1905.11527 , year=
1905 arXiv
-
[242]
A Theoretical Analysis of Deep
Fan, Jianqing and Wang, Zhaoran and Xie, Yuchen and Yang, Zhuoran , journal=. A Theoretical Analysis of Deep
-
[243]
The asymptotic convergence-rate of
Szepesv. The asymptotic convergence-rate of. NIPS , volume=. 1997 , organization=
1997
-
[244]
2010 , publisher=
Auer, Peter and Ortner, Ronald , journal=. 2010 , publisher=
2010
-
[245]
arXiv preprint arXiv:2006.03864 , year=
Model-free reinforcement learning: from clipped pseudo-regret to sample complexity , author=. arXiv preprint arXiv:2006.03864 , year=
2006 arXiv
-
[246]
Strehl, Alexander L and Li, Lihong and Wiewiora, Eric and Langford, John and Littman, Michael L , booktitle=
-
[247]
arXiv preprint arXiv:1705.07041 , year=
Posterior sampling for reinforcement learning: worst-case regret bounds , author=. arXiv preprint arXiv:1705.07041 , year=
-
[248]
Unifying
Dann, Christoph and Lattimore, Tor and Brunskill, Emma , journal=. Unifying
-
[249]
Momentum
Weng, Bowen and Xiong, Huaqing and Zhao, Lin and Liang, Yingbin and Zhang, Wei , journal=. Momentum
-
[250]
Finite-time analysis for double
Xiong, Huaqing and Zhao, Lin and Liang, Yingbin and Zhang, Wei , journal=. Finite-time analysis for double
-
[251]
arXiv preprint arXiv:2009.13503 , year=
Is reinforcement learning more difficult than bandits? a near-optimal algorithm escaping the curse of horizon , author=. arXiv preprint arXiv:2009.13503 , year=
2009 arXiv
-
[252]
arXiv preprint arXiv:2203.01387 , year=
A survey on offline reinforcement learning: Taxonomy, review, and open problems , author=. arXiv preprint arXiv:2203.01387 , year=
-
[253]
Machine Learning for Healthcare Conference , pages=
Model selection for offline reinforcement learning: Practical considerations for healthcare settings , author=. Machine Learning for Healthcare Conference , pages=. 2021 , organization=
2021
-
[254]
arXiv preprint arXiv:1812.00568 , year=
Visual foresight: Model-based deep reinforcement learning for vision-based robotic control , author=. arXiv preprint arXiv:1812.00568 , year=
-
[255]
arXiv preprint arXiv:2111.11097 , year=
Umbrella: Uncertainty-aware model-based offline reinforcement learning leveraging planning , author=. arXiv preprint arXiv:2111.11097 , year=
-
[256]
The Sample-Communication Complexity Trade-off in Federated
Salgia, Sudeep and Chi, Yuejie , booktitle=. The Sample-Communication Complexity Trade-off in Federated
-
[257]
arXiv preprint arXiv:2407.18878 , year=
A sharper global convergence analysis for average reward reinforcement learning via an actor-critic approach , author=. arXiv preprint arXiv:2407.18878 , year=
-
[258]
A Provably Efficient Algorithm for Linear
Gao, Minbo and Xie, Tianle and Du, Simon S and Yang, Lin F , journal=. A Provably Efficient Algorithm for Linear
-
[259]
IEEE Robotics and Automation Letters , volume=
Badgr: An autonomous self-supervised learning-based navigation system , author=. IEEE Robotics and Automation Letters , volume=. 2021 , publisher=
2021
-
[260]
Exponential lower bounds for planning in
Weisz, Gellert and Amortila, Philip and Szepesv. Exponential lower bounds for planning in. Algorithmic Learning Theory , pages=. 2021 , organization=
2021
-
[261]
Dong, Kefan and Wang, Yuanhao and Chen, Xiaoyu and Wang, Liwei , journal=
-
[262]
Mathematics of Operations Research , volume=
Efficient reinforcement learning in deterministic systems with value function generalization , author=. Mathematics of Operations Research , volume=. 2017 , publisher=
2017
-
[263]
International Conference on Machine Learning , pages=
Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound , author=. International Conference on Machine Learning , pages=. 2020 , organization=
2020
-
[264]
Proceedings of the National Conference on Artificial Intelligence , volume=
Error bounds for approximate value iteration , author=. Proceedings of the National Conference on Artificial Intelligence , volume=. 2005 , organization=
2005
-
[265]
arXiv preprint arXiv:2103.10897 , year=
Bilinear classes: A structural framework for provable generalization in rl , author=. arXiv preprint arXiv:2103.10897 , year=
-
[266]
Advances in applied mathematics , volume=
Asymptotically efficient adaptive allocation rules , author=. Advances in applied mathematics , volume=. 1985 , publisher=
1985
-
[267]
2020 , publisher=
Bandit algorithms , author=. 2020 , publisher=
2020
-
[268]
International Conference on Learning Representations , year=
Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning? , author=. International Conference on Learning Representations , year=
-
[269]
arXiv preprint arXiv:2103.12690 , year=
An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gap , author=. arXiv preprint arXiv:2103.12690 , year=
-
[270]
Tohoku Mathematical Journal, Second Series , volume=
Weighted sums of certain dependent random variables , author=. Tohoku Mathematical Journal, Second Series , volume=. 1967 , publisher=
1967
-
[271]
National climatic data center , howpublished =
-
[272]
Hasselt, Hado , journal=. Double. 2010 , publisher=
2010
-
[273]
Proceedings of the 32nd Annual Conference on Neural Information Processing Systems , volume=
Graph oracle models, lower bounds, and gaps for parallel stochastic optimization , author=. Proceedings of the 32nd Annual Conference on Neural Information Processing Systems , volume=
-
[274]
Proceedings of the 34th Conference on Learning Theory , pages=
The Min-max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communication , author=. Proceedings of the 34th Conference on Learning Theory , pages=. 2021 , organization=
2021
-
[275]
Advances in Neural Information Processing Systems , volume=
Breaking the sample size barrier in model-based reinforcement learning with a generative model , author=. Advances in Neural Information Processing Systems , volume=
-
[276]
A Generalization Error for
Murphy, Susan , journal=. A Generalization Error for
-
[277]
The annals of mathematical statistics , pages=
A stochastic approximation method , author=. The annals of mathematical statistics , pages=. 1951 , publisher=
1951
-
[278]
Chen, Zaiwei and Maguluri, Siva Theja and Sanjay Shakkottai and Karthikeyan Shanmugam , journal=. A
-
[279]
Advances in Neural Information Processing Systems , volume=
Finite-sample analysis of contractive stochastic approximation using smooth convex envelopes , author=. Advances in Neural Information Processing Systems , volume=
-
[280]
Electronic Communications in Probability , volume=
Freedman's inequality for matrix martingales , author=. Electronic Communications in Probability , volume=. 2011 , publisher=
2011
-
[281]
Machine learning , volume=
Learning to predict by the methods of temporal differences , author=. Machine learning , volume=. 1988 , publisher=
1988
-
[282]
the Annals of Probability , pages=
On tail probabilities for martingales , author=. the Annals of Probability , pages=. 1975 , publisher=
1975
-
[283]
Sample-optimal parametric
Yang, Lin and Wang, Mengdi , booktitle=. Sample-optimal parametric
-
[284]
Mastering the game of
Silver, David and Schrittwieser, Julian and Simonyan, Karen and Antonoglou, Ioannis and Huang, Aja and Guez, Arthur and Hubert, Thomas and Baker, Lucas and Lai, Matthew and Bolton, Adrian and others , journal=. Mastering the game of. 2017 , publisher=
2017
-
[285]
arXiv preprint arXiv:2006.06555 , year=
Distributed Reinforcement Learning in Multi-Agent Networked Systems , author=. arXiv preprint arXiv:2006.06555 , year=
2006 arXiv
-
[286]
Azar, Mohammad G and Kappen, Hilbert J and Ghavamzadeh, Mohammad and Munos, R. Speedy. Advances in neural information processing systems , pages=
-
[287]
Finite-sample analysis for
Zou, Shaofeng and Xu, Tengyu and Liang, Yingbin , booktitle=. Finite-sample analysis for
-
[288]
Markov chains:
Br. Markov chains:. 2013 , publisher=
2013
-
[289]
2013 , publisher=
Concentration inequalities: A nonasymptotic theory of independence , author=. 2013 , publisher=
2013
-
[290]
Bernstein's inequality for general
Jiang, Bai and Sun, Qiang and Fan, Jianqing , journal=. Bernstein's inequality for general
-
[291]
Provably efficient
Du, Simon S and Luo, Yuping and Wang, Ruosong and Zhang, Hanrui , booktitle=. Provably efficient
-
[292]
Reinforcement learning with a near optimal rate of convergence , author=
-
[293]
Finite Time Analysis of Linear Two-timescale Stochastic Approximation with
Kaledin, Maxim and Moulines, Eric and Naumov, Alexey and Tadic, Vladislav and Wai, Hoi-To , journal=. Finite Time Analysis of Linear Two-timescale Stochastic Approximation with
-
[294]
Conference On Learning Theory , pages=
Finite Sample Analysis of Two-Timescale Stochastic Approximation with Applications to Reinforcement Learning , author=. Conference On Learning Theory , pages=
-
[295]
Finite sample analyses for
Dalal, Gal and Sz. Finite sample analyses for. Thirty-Second AAAI Conference on Artificial Intelligence , year=
-
[296]
Yuanhao Wang and Kefan Dong and Xiaoyu Chen and Liwei Wang , booktitle=
-
[297]
A theoretical analysis of deep
Jianqing Fan and Zhaoran Wang and Xie, Yuchen and Yang, Zhuoran , journal=. A theoretical analysis of deep
-
[298]
arXiv preprint arXiv:2002.10301 , year=
Q-learning with uniformly bounded variance: Large discounting is not a barrier to fast learning , author=. arXiv preprint arXiv:2002.10301 , year=
2002 arXiv
-
[299]
arXiv preprint arXiv:2002.00874 , year=
Finite-Sample Analysis of Stochastic Approximation Using Smooth Convex Envelopes , author=. arXiv preprint arXiv:2002.00874 , year=
2002 arXiv
-
[300]
Advances in Neural Information Processing Systems , volume=
Variance reduced policy evaluation with smooth function approximation , author=. Advances in Neural Information Processing Systems , volume=
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.