Pith. sign in

REVIEW 3 major objections 5 minor 64 references

Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The performance of model-based offline RL is bounded by the Wasserstein distance between behavioral data and the optimal policy's occupancy, and diverse unsupervised data with buffer selection can approach optimality on unknown tasks.

desk verdict A useful Wasserstein reframing of offline data quality, but the headline bound is an unproven approximation and the experiments are thin; worth refereeing nonetheless. read the letter →

arxiv 2506.19643 v1 pith:7HC6YYIR submitted 2025-06-24 cs.LG

Shuncheng He , Hongchang Zhang , Jianzhun Shao , Yuhang Jiang , Xiangyang Ji This is my paper · ORCID
classification cs.LG
keywords offlinereinforcementlearningunsuperviseddatagenerationWassersteindistancemodel-basedRLtask-agnosticpolicydiversitybufferselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that, in model-based offline reinforcement learning, the important property of a batch dataset is how close its state-action occupancy is to the occupancy of the optimal policy, measured by the 1-Wasserstein distance. Under Lipschitz continuity and a nearest-neighbor learned transition model, the return gap between the learned policy and the optimal policy is bounded by a constant times that distance, so data generation becomes a distance-minimization problem. Because the task reward may be unknown when data are collected, the paper trains a family of policies with diversity rewards and later picks the buffer that scores best under the revealed reward. The resulting framework, UDG, outperforms data produced by a single supervised policy in the locomotion tasks tested.

What carries the argument

The argument is carried by two objects. First, the telescoping performance-difference identity (Lemma 4.1), which expresses the return gap between true and learned dynamics as an expectation over the learned occupancy; combined with Lipschitz dynamics and value functions, the integrand is bounded by a model error that can be measured as a 1-Wasserstein distance. A non-parametric nearest-neighbor transition model (Assumption 4.6) keeps the support of model rollouts inside the behavioral buffer, and that is what lets the triangle inequality turn the bound into the distance $W_1(\rho^{\pi_\beta}_T, \rho^{\pi^*}_T)$. Second, the surrogate diversity objective of Equation 12: maximize the minimum pairwise Wasserstein distance between the occupancy distributions of the candidate policies, which the paper argues approximates the minimax regret of Equation 11.

What would settle it

Construct a Lipschitz MDP with a multi-modal transition (two very different next states both likely) and collect a behavioral buffer close to the optimal occupancy in $W_1$; if the measured gap $\eta_M(\pi^*) - \eta_M(\hat\pi)$ repeatedly exceeds $2C\,W_1(\rho^{\pi_\beta}_T, \rho^{\pi^*}_T)$, then the $D_1 \approx D_2$ supposition behind Equation 8 fails in that setting.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that MOPO's model-error lower bound can be rewritten as a data-distribution bound: $\eta_M(\hat\pi) \ge \eta_M(\pi^*) - 2C\,W_1(\rho^{\pi_\beta}_T, \rho^{\pi^*}_T)$, with $C = 2c\gamma L_r L_T$, whenever the unknown model error under the optimal policy is comparable to the behavioral-to-optimal distance. This is Theorem 4.8 together with the $D_1 \approx D_2$ approximation behind Equation 8. The theorem also gives a mixture version (Equation 9), showing that blending many diverse buffers can widen the bound unless the mixture stays close to the optimal occupancy. The paper further claims that the unsupervised diversity objective $\max \min_{i\neq j} W_1(\rho^{\pi_i}_T, \rho^{\pi_j}_T)$ approximately optimizes the worst-case regret $\min_{\pi_1,\dots,\pi_N}\max_{\pi^*}\min_i L(\pi_i,\pi^*)$ over unknown tasks, and that choosing the highest-scoring buffer is therefore the right data-selection rule.

Load-bearing premise

The entire chain collapses if the learned model's error on states visited by the optimal policy is not comparable to the Wasserstein distance from the behavioral data to the optimal policy, because the paper asserts this comparability is natural without offering a quantitative bound.

Editorial extensions

If this is right

  • Data collection for offline RL should be judged by the occupancy distance to the intended task's optimal policy, not by raw state coverage; a single close buffer can beat a broad mixture.
  • When the task is unknown, unsupervised diversity is a rational strategy: a sufficiently diverse policy set makes it likely that at least one buffer is close to the optimal policy, and selecting that buffer by average return yields near-optimal offline behavior.
  • Mixing all generated buffers can hurt performance compared with selecting the best one, because the bound in Equation 9 depends on the mixture's distance as well as the nearest policy's distance; the Ant-Angle experiments confirm this.
  • Model-based offline RL with uncertainty penalties inherits its performance directly from data quality, so improving data collection and buffer selection is an alternative route to better offline RL that does not require changing the conservative algorithm.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The bound suggests a cheap data-quality diagnostic: estimate $W_1$ between a candidate buffer and the occupancy of a reward-optimized policy, and pre-select buffers by that distance rather than by rollout evaluation, which would extend UDG to settings where the reward is only partially known.
  • The packing/covering relation between the surrogate diversity objective and the minimax regret (which the paper acknowledges is hard to analyze) implies that the number of policies $K$ needed for a given worst-case regret in continuous state spaces may scale with the metric entropy of policy-occupancy space; quantifying that scaling is a natural open problem.
  • Because the proof relies on a non-parametric nearest-neighbor model, the same buffer-selection recipe should be tested with neural dynamics models and model-free offline algorithms to see whether the Wasserstein distance remains the controlling quantity when the proof device is removed.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates how the distribution of the offline batch data affects the performance of model-based offline reinforcement learning. It builds on the MOPO performance lower bound and derives, under Lipschitz and nearest-neighbor model assumptions, a bound (Theorem 4.8) on the return of the MOPO-trained policy in terms of Wasserstein distances between the behavioral occupancy distribution and the optimal policy occupancy distribution. It then introduces an approximate bound (Eq. 8) that drops the model-error term by assuming D1≈D2, and uses this to formulate a worst-case regret objective (Eq. 11) and a diversity surrogate objective (Eq. 12) for task-agnostic data generation. The proposed framework, UDG, trains a set of diverse policies, collects buffers from their rollouts, relabels rewards once the task is revealed, selects the buffer with the highest average return, and trains MOPO on that buffer. Experiments on Ant-Angle and Cheetah-Jump compare UDG with supervised data, mixed buffers, and non-diverse baselines.

Significance. If the theoretical chain were rigorous, the paper would provide a useful and interpretable data-centric principle: in model-based offline RL, the gap to the optimal policy is governed by the Wasserstein distance between the behavioral and optimal occupancy distributions, so unsupervised diversity-based data generation is a principled choice for task-agnostic settings. The paper deserves credit for grounding the discussion in MOPO's bound, for transparently listing its own limitations (Lipschitz assumptions, non-parametric model, neural-model gap), and for including buffer-selection ablations that test the data-distance hypothesis. However, the headline result is currently an unquantified approximation, the relation between the minimax regret objective and the diversity surrogate is asserted rather than proved, and the experiments use a different model class than the theory. The contribution is therefore suggestive and potentially valuable, but the central theoretical claims are not yet established.

major comments (3)
  1. [Section 4, Eq. (8) and text after Eq. (7)] The step from the first line of Theorem 4.8 to Eq. (8) replaces D1 = W1(ρπ*_T, ρπ*_\hatT) with D2 = W1(ρπβ_T, ρπ*_T) on the basis of the assertion that ρπ*_\hatT 'will not be too distinct' from the behavioral distribution. This is not a theorem step: the support inclusion supp(ρπ*_\hatT(s)) ⊆ supp(ρπβ_T(s)) only constrains the states emitted by the episodic memory, not the action marginals or the frequencies of ρπ*_\hatT. D1 can therefore be large even when D2 is small, and since D1 is the model-error term under the optimal policy's occupancy, the claim that W1(ρπβ_T, ρπ*_T) controls the performance gap is not established without a quantitative bound on |D1−D2|. The authors should either prove such a bound under a stated additional assumption or present Eq. (8) explicitly as a heuristic and soften the corresponding claims in the abstract and interpretation.
  2. [Section 4, Eq. (12) versus Eq. (11)] The surrogate diversity objective (Eq. 12) is admitted, in the text following Eq. (11), to have an optimal solution that does not match the minimax regret objective (Eq. 11) in general, and the paper says 'we assume the gap will be small' with an analogy to covering/packing and computational graphics. This is not a proof, and the third contribution is explicitly 'an objective of minimal worst-case regret'. Without a quantitative relation between the packing-type objective (Eq. 12) and the covering/regret objective (Eq. 11), the statement that UDG approximately minimizes worst-case regret is unsupported. The authors should either prove a bound under a stated condition on the policy class Π or clearly label Eq. (12) as a heuristic proxy and remove or qualify the regret-optimality claims.
  3. [Section 5 and Section 4, Assumption 4.6] The main theorem, Theorem 4.8, relies on Assumption 4.6 that the prediction model is a non-parametric nearest-neighbor model, but the experiments train MOPO with neural network dynamics models, as the Limitations section acknowledges. This mismatch means the experiments do not directly test Theorem 4.8. A claim that 'experimental results match the findings in theoretical analysis' needs either a nearest-neighbor variant of MOPO in the experiments or a separate argument that the bound extends to neural regressors, such as a Lipschitz or regression-error bound for the learned model.
minor comments (5)
  1. [Section 4, after Theorem 4.8] The statement that substituting π* with πβ yields ηM(πhat) ≥ ηM(πβ) does not follow from Eq. (6), since the bound still contains W1(ρπβ_T, ρπβ_\hatT), and MOPO's own guarantee includes the model-error term 2γϵ_u(πβ).
  2. [Definition 4.3] Definition 4.3 defines the radius of the minimal enclosing ball, not the diameter; this makes the quantity Bπβ in Eq. (6) ambiguous and should be renamed 'covering radius' or redefined.
  3. [Assumption 4.7] The assumption uses three '≈' relations (\hatρπβ_T ≈ ρπβ_T ≈ ρπβ_\hatT) without quantifying the Wasserstein error; if these approximations affect the final bound, they should be stated as explicit assumptions with error terms.
  4. [Table 1] For Cheetah-Crawl, the diverse result (1348.19±274) overlaps heavily with the random result (1239.00±57), so the claim that diverse data performs better is not statistically supported by this table.
  5. [Throughout] There are several typos and minor wording issues, including 'connenction' (Section 4 heading), 'polices' (Figure 4 caption), 'an policy' (Section 5), and inconsistent use of 'Mujoco' vs. 'MuJoCo'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the main bound is derived from MOPO plus Lipschitz/nearest-neighbor assumptions, with D1≈D2 explicitly an approximation, and the surrogate objective is acknowledged to be a proxy rather than a fitted prediction.

full rationale

The paper's central bound in Eq. 8 is obtained by combining MOPO's Theorem 4.2 (an external, cited result) with Lipschitz assumptions 4.4-4.5, a nearest-neighbor model assumption 4.6, and triangle inequality on Wasserstein distances; no free constant is fitted to experimental data. The step that replaces D1 with D2 is explicitly flagged as an assumption ('We can naturally suppose that ρπ*_T(s,a) will not be too distinct from ρπβ_T(s,a)'), and the paper even notes D1>D2 is possible under non-smooth dynamics or multi-modal situations; this is an unproven approximation, not a reduction of the conclusion to its inputs. The surrogate objective Eq. 12 is acknowledged not to match Eq. 11 in general and is justified only as a covering/packing proxy via prior literature; again the paper does not present it as a fitted equivalence. The use of WURL (He et al. 2022, same authors) as the diversity base is an implementation choice consistent with Eq. 12; WURL is prior published work and its objective is not invoked as an external uniqueness theorem that forbids alternatives. The empirical selection of the best buffer (Algorithm 1, line 5) is part of the method and is not used to validate the theoretical bound. No load-bearing result reduces by construction to a fitted parameter, a self-citation chain, or a definition, so no circular step is exhibited.

Assumptions & free parameters 4 free parameters · 7 assumptions · 0 invented entities

No new physical entities, mediators, or conserved quantities are introduced; UDG is a procedural framework composed of existing components (WURL-style diversity policies, relabeling, buffer selection).

free parameters (4)
  • diversity weight lambda = not reported
    Reweights task reward and diversity pseudo-reward in semi-supervised training; no value or tuning protocol is given.
  • number of policies K = 10 for Ant-Angle, 5 for Cheetah
    Controls the number of data buffers generated; no sensitivity analysis is provided.
  • diversity metric projection = x-y position for Ant-Angle
    Diversity reward is computed on a low-dimensional task-relevant subspace, which requires prior knowledge of the reward structure; the paper acknowledges this limitation.
  • maximum buffer size N = not specified
    The number of transitions per generated buffer is not specified in the main text.
assumptions (7)
  • domain assumption State and action spaces are bounded subsets of Euclidean spaces, and the transition is deterministic and continuous (Assumption 4.4).
    Boundedness and determinism are used to define diameters and to make the model error collapse to a point-wise distance.
  • domain assumption The transition function and all value functions are Lipschitz continuous (Assumption 4.5).
    Lipschitz constants L_T and L_r enter the constant C in the main bound; value-function Lipschitzness for all policies is strong.
  • ad hoc to paper The prediction model is a non-parametric nearest-neighbor model with an episodic memory (Assumption 4.6).
    This makes the model deterministic and ties model error to distances in memory, but does not describe the neural dynamics model used in MOPO experiments.
  • domain assumption Batch data is sufficient so empirical occupancy equals true occupancy, and the behavioral occupancy has bounded support (Assumption 4.7).
    Finite-sample effects are explicitly set aside, so the bound is asymptotic in dataset size.
  • ad hoc to paper D1 ≈ D2: W1(ρπ∗_T, ρπ∗_ˆT) ≈ W1(ρπβ_T, ρπ∗_T).
    Unproven approximation that converts the rigorous bound (Eq. 6) into the clean bound (Eq. 8); if false, the central conclusion does not follow.
  • ad hoc to paper The surrogate diversity objective (Eq. 12) is a satisfactory proxy for the minimax regret objective (Eq. 11).
    The paper states the gap between covering and packing formulations is assumed to be small; no quantitative conditions are given.
  • domain assumption MOPO's admissible error estimator u(s,a) upper-bounds the IPM model error and its penalized-MDP lower bound applies (Theorem 4.2).
    The whole analysis starts from MOPO's lower bound; this is an external result from cited prior work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model." pith.science (2026). https://pith.science/paper/7HC6YYIR

@misc{pith2026250619643,
  author       = {Pith},
  title        = {Pith review of: Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7HC6YYIR}},
  note         = {Machine review of arXiv:2506.19643}
}
read the original abstract

Offline reinforcement learning (RL) recently gains growing interests from RL researchers. However, the performance of offline RL suffers from the out-of-distribution problem, which can be corrected by feedback in online RL. Previous offline RL research focuses on restricting the offline algorithm in in-distribution even in-sample action sampling. In contrast, fewer work pays attention to the influence of the batch data. In this paper, we first build a bridge over the batch data and the performance of offline RL algorithms theoretically, from the perspective of model-based offline RL optimization. We draw a conclusion that, with mild assumptions, the distance between the state-action pair distribution generated by the behavioural policy and the distribution generated by the optimal policy, accounts for the performance gap between the policy learned by model-based offline RL and the optimal policy. Secondly, we reveal that in task-agnostic settings, a series of policies trained by unsupervised RL can minimize the worst-case regret in the performance gap. Inspired by the theoretical conclusions, UDG (Unsupervised Data Generation) is devised to generate data and select proper data for offline training under tasks-agnostic settings. Empirical results demonstrate that UDG can outperform supervised data generation on solving unknown tasks.

Figures

Figures reproduced from arXiv: 2506.19643 by the authors.

Figure 1
Figure 1. Rendered trajectories of offline trained policies in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The framework of UDG. First a series of K policies are trained simultaneously with diversity rewards. Second, collect rollout experience (s, a, s′ ) from each policy and con￾struct a corresponding data buffer. Third, relabel the reward in the batch data with a designated reward function, and se￾lect the data buffer with the maximal average return. Finally train the agent on the chosen data by offline RL approaches. … view at source ↗
Figure 3
Figure 3. Results on Ant-Angle tasks. The data buffers of all [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 26 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Agarwal, R.; Schuurmans, D.; and Norouzi, M. 2019. Striving for simplicity in off-policy deep reinforcement learning. arXiv preprint arXiv:1907.04543

    arXiv 2019

  4. [4]

    Arjovsky, M.; and Bottou, L. 2017. Towards principled methods for training generative adversarial networks. arXiv preprint arXiv:1701.04862

    arXiv 2017

  5. [5]

    Arjovsky, M.; Chintala, S.; and Bottou, L. 2017. Wasserstein generative adversarial networks. In International conference on machine learning, 214--223. PMLR

    2017

  6. [6]

    o r \"o czky Jr, K.; B \

    B \"o r \"o czky Jr, K.; B \"o r \"o czky, K.; et al. 2004. Finite packing and covering, volume 154. Cambridge University Press

    work page 2004

  7. [7]

    Brockman, G.; Cheung, V.; Pettersson, L.; Schneider, J.; Schulman, J.; Tang, J.; and Zaremba, W. 2016. Openai gym. arXiv preprint arXiv:1606.01540

    arXiv 2016

  8. [8]

    Burda, Y.; Edwards, H.; Storkey, A.; and Klimov, O. 2018. Exploration by random network distillation. arXiv preprint arXiv:1810.12894

    arXiv 2018

Show all 64 references
  1. [9]

    Campos, V.; Trott, A.; Xiong, C.; Socher, R.; Giro-i Nieto, X.; and Torres, J. 2020. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, 1317--1327. PMLR

    2020

  2. [10]

    Chen, J.; and Jiang, N. 2019. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning, 1042--1051. PMLR

    2019

  3. [11]

    Chen, L.; and Xu, J.-c. 2004. Optimal delaunay triangulations. Journal of Computational Mathematics, 299--308

    2004

  4. [12]

    Chen, X.-H.; Yu, Y.; Li, Q.; Luo, F.-M.; Qin, Z.; Shang, W.; and Ye, J. 2021. Offline model-based adaptable policy learning. Advances in Neural Information Processing Systems, 34: 8432--8443

    2021

  5. [13]

    Cheng, C.-A.; Xie, T.; Jiang, N.; and Agarwal, A. 2022. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, 3852--3878. PMLR

    2022

  6. [14]

    Eysenbach, B.; Gupta, A.; Ibarz, J.; and Levine, S. 2019. Diversity is All You Need: Learning Skills without a Reward Function. In International Conference on Learning Representations

    2019

  7. [15]

    Eysenbach, B.; Salakhutdinov, R.; and Levine, S. 2021. The information geometry of unsupervised reinforcement learning. arXiv preprint arXiv:2110.02719

    2021 arXiv

  8. [16]

    Florensa, C.; Duan, Y.; and Abbeel, P. 2017. Stochastic neural networks for hierarchical reinforcement learning. arXiv preprint arXiv:1704.03012

    2017 arXiv

  9. [17]

    Fu, J.; Kumar, A.; Nachum, O.; Tucker, G.; and Levine, S. 2020. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219

    2020 arXiv

  10. [18]

    Fujimoto, S.; and Gu, S. S. 2021. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34: 20132--20145

    2021

  11. [19]

    Fujimoto, S.; Hoof, H.; and Meger, D. 2018. Addressing function approximation error in actor-critic methods. In International conference on machine learning, 1587--1596. PMLR

    2018

  12. [20]

    Fujimoto, S.; Meger, D.; and Precup, D. 2019. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, 2052--2062. PMLR

    2019

  13. [21]

    S.; Mankowitz, D

    Gulcehre, C.; Wang, Z.; Novikov, A.; Paine, T.; G \'o mez, S.; Zolna, K.; Agarwal, R.; Merel, J. S.; Mankowitz, D. J.; Paduraru, C.; et al. 2020. Rl unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 7248--7259

    2020

  14. [22]

    Haarnoja, T.; Zhou, A.; Abbeel, P.; and Levine, S. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, 1861--1870. PMLR

    2018

  15. [23]

    He, S.; Jiang, Y.; Zhang, H.; Shao, J.; and Ji, X. 2022. Wasserstein unsupervised reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 6884--6892

    2022

  16. [24]

    Hessel, M.; Modayil, J.; Van Hasselt, H.; Schaul, T.; Ostrovski, G.; Dabney, W.; Horgan, D.; Piot, B.; Azar, M.; and Silver, D. 2018. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32

    2018

  17. [25]

    Janner, M.; Fu, J.; Zhang, M.; and Levine, S. 2019. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32

    2019

  18. [26]

    Kidambi, R.; Rajeswaran, A.; Netrapalli, P.; and Joachims, T. 2020. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33: 21810--21823

    2020

  19. [27]

    Kim, J.; Park, S.; and Kim, G. 2021. Unsupervised skill discovery with bottleneck option learning. arXiv preprint arXiv:2106.14305

    2021 arXiv

  20. [28]

    R.; Sobh, I.; Talpaert, V.; Mannion, P.; Al Sallab, A

    Kiran, B. R.; Sobh, I.; Talpaert, V.; Mannion, P.; Al Sallab, A. A.; Yogamani, S.; and P \'e rez, P. 2021. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(6): 4909--4926

    2021

  21. [29]

    Kumar, A.; Fu, J.; Soh, M.; Tucker, G.; and Levine, S. 2019. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32

    2019

  22. [30]

    Kumar, A.; Zhou, A.; Tucker, G.; and Levine, S. 2020. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179--1191

    2020

  23. [31]

    Lambert, N.; Wulfmeier, M.; Whitney, W.; Byravan, A.; Bloesch, M.; Dasagi, V.; Hertweck, T.; and Riedmiller, M. 2022. The challenges of exploration for offline reinforcement learning. arXiv preprint arXiv:2201.11861

    2022 arXiv

  24. [32]

    Laskin, M.; Yarats, D.; Liu, H.; Lee, K.; Zhan, A.; Lu, K.; Cang, C.; Pinto, L.; and Abbeel, P. 2021. URLB: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:2110.15191

    2021 arXiv

  25. [33]

    Lee, L.; Eysenbach, B.; Parisotto, E.; Xing, E.; Levine, S.; and Salakhutdinov, R. 2019. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274

    2019 arXiv

  26. [34]

    Levine, S.; Kumar, A.; Tucker, G.; and Fu, J. 2020. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643

    2020 arXiv

  27. [35]

    Liu, H.; and Abbeel, P. 2021 a . Aps: Active pretraining with successor features. In International Conference on Machine Learning, 6736--6747. PMLR

    2021

  28. [36]

    Liu, H.; and Abbeel, P. 2021 b . Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34: 18459--18473

    2021

  29. [37]

    Matsushima, T.; Furuta, H.; Matsuo, Y.; Nachum, O.; and Gu, S. 2020. Deployment-efficient reinforcement learning via model-based offline optimization. arXiv preprint arXiv:2006.03647

    2020 arXiv

  30. [38]

    A.; Veness, J.; Bellemare, M

    Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015. Human-level control through deep reinforcement learning. nature, 518(7540): 529--533

    2015

  31. [39]

    M \"u ller, A. 1997. Integral probability metrics and their generating classes of functions. Advances in applied probability, 29(2): 429--443

    1997

  32. [40]

    A.; and Darrell, T

    Pathak, D.; Agrawal, P.; Efros, A. A.; and Darrell, T. 2017. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, 2778--2787. PMLR

    2017

  33. [41]

    Pathak, D.; Gandhi, D.; and Gupta, A. 2019. Self-supervised exploration via disagreement. In International conference on machine learning, 5062--5071. PMLR

    2019

  34. [42]

    B.; Kumar, A.; Zhang, G.; and Levine, S

    Peng, X. B.; Kumar, A.; Zhang, G.; and Levine, S. 2019. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177

    2019 arXiv

  35. [43]

    P.; Vinyals, O.; Hassabis, D.; Wierstra, D.; and Blundell, C

    Pritzel, A.; Uria, B.; Srinivasan, S.; Badia, A. P.; Vinyals, O.; Hassabis, D.; Wierstra, D.; and Blundell, C. 2017. Neural episodic control. In International conference on machine learning, 2827--2836. PMLR

    2017

  36. [44]

    Rezaeifar, S.; Dadashi, R.; Vieillard, N.; Hussenot, L.; Bachem, O.; Pietquin, O.; and Geist, M. 2022. Offline reinforcement learning as anti-exploration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 8106--8114

    2022

  37. [45]

    Schl \"o mer, T.; Heck, D.; and Deussen, O. 2011. Farthest-point optimized point sets with maximized minimum distance. In Proceedings of the ACM SIGGRAPH Symposium on High Performance Graphics, 135--142

    2011

  38. [46]

    Sharma, A.; Gu, S.; Levine, S.; Kumar, V.; and Hausman, K. 2020. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations (ICLR)

    2020

  39. [47]

    Y.; Springenberg, J

    Siegel, N. Y.; Springenberg, J. T.; Berkenkamp, F.; Abdolmaleki, A.; Neunert, M.; Lampe, T.; Hafner, R.; Heess, N.; and Riedmiller, M. 2020. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396

    2020 arXiv

  40. [48]

    J.; Guez, A.; Sifre, L.; Van Den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; et al

    Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.; Van Den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; et al. 2016. Mastering the game of Go with deep neural networks and tree search. nature, 529(7587): 484--489

    2016

  41. [49]

    Silver, D.; Schrittwieser, J.; Simonyan, K.; Antonoglou, I.; Huang, A.; Guez, A.; Hubert, T.; Baker, L.; Lai, M.; Bolton, A.; et al. 2017. Mastering the game of go without human knowledge. nature, 550(7676): 354--359

    2017

  42. [50]

    Sodhani, S.; Zhang, A.; and Pineau, J. 2021. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning, 9767--9779. PMLR

    2021

  43. [51]

    Strouse, D.; Baumli, K.; Warde-Farley, D.; Mnih, V.; and Hansen, S. 2021. Learning more skills through optimistic exploration. arXiv preprint arXiv:2107.14226

    2021 arXiv

  44. [52]

    Tassa, Y.; Doron, Y.; Muldal, A.; Erez, T.; Li, Y.; Casas, D. d. L.; Budden, D.; Abdolmaleki, A.; Merel, J.; Lefrancq, A.; et al. 2018. Deepmind control suite. arXiv preprint arXiv:1801.00690

    2018 arXiv

  45. [53]

    Todorov, E.; Erez, T.; and MuJoCo, Y. 2012. A physics engine for model-based control. In Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 5026--5033

    2012

  46. [54]

    D.; O'Rourke, J.; and Goodman, J

    Toth, C. D.; O'Rourke, J.; and Goodman, J. E. 2017. Handbook of discrete and computational geometry. CRC press

    2017

  47. [55]

    Wang, H.; Feng, D.; Ding, B.; and Li, W. 2022. Offline Imitation Learning Using Reward-free Exploratory Data. In Proceedings of the 2022 5th International Conference on Algorithms, Computing and Artificial Intelligence, 1--9

    2022

  48. [56]

    P.; and Kakade, S

    Wang, R.; Foster, D. P.; and Kakade, S. M. 2020. What are the statistical limits of offline RL with linear function approximation? arXiv preprint arXiv:2010.11895

    2020 arXiv

  49. [57]

    Wu, Y.; Tucker, G.; and Nachum, O. 2019. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361

    2019 arXiv

  50. [58]

    Yarats, D.; Brandfonbrener, D.; Liu, H.; Laskin, M.; Abbeel, P.; Lazaric, A.; and Pinto, L. 2022. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201.13425

    2022 arXiv

  51. [59]

    Yarats, D.; Fergus, R.; Lazaric, A.; and Pinto, L. 2021. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, 11920--11931. PMLR

    2021

  52. [60]

    Yu, T.; Kumar, S.; Gupta, A.; Levine, S.; Hausman, K.; and Finn, C. 2020 a . Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824--5836

    2020

  53. [61]

    Yu, T.; Quillen, D.; He, Z.; Julian, R.; Hausman, K.; Finn, C.; and Levine, S. 2020 b . Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, 1094--1100. PMLR

    2020

  54. [62]

    Y.; Levine, S.; Finn, C.; and Ma, T

    Yu, T.; Thomas, G.; Yu, L.; Ermon, S.; Zou, J. Y.; Levine, S.; Finn, C.; and Ma, T. 2020 c . Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33: 14129--14142

    2020

  55. [63]

    Zhang, H.; Shao, J.; Jiang, Y.; He, S.; Zhang, G.; and Ji, X. 2022. State Deviation Correction for Offline Reinforcement Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 9022--9030

    2022

  56. [64]

    P.; and Westerlund, T

    Zhao, W.; Queralta, J. P.; and Westerlund, T. 2020. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI), 737--744. IEEE

    2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.