pith. sign in

arxiv: 2201.13425 · v3 · pith:MWBD3BWKnew · submitted 2022-01-31 · 💻 cs.LG · cs.AI

Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning

Denis Yarats , David Brandfonbrener , Hao Liu , Michael Laskin , Pieter Abbeel , Alessandro Lazaric , Lerrel Pinto This is my paper
classification 💻 cs.LG cs.AI
keywords dataofflineexorlexploratorylearningalgorithmschangedownstream
0
0 comments X
read the original abstract

Recent progress in deep learning has relied on access to large and diverse datasets. Such data-driven progress has been less evident in offline reinforcement learning (RL), because offline RL data is usually collected to optimize specific target tasks limiting the data's diversity. In this work, we propose Exploratory data for Offline RL (ExORL), a data-centric approach to offline RL. ExORL first generates data with unsupervised reward-free exploration, then relabels this data with a downstream reward before training a policy with offline RL. We find that exploratory data allows vanilla off-policy RL algorithms, without any offline-specific modifications, to outperform or match state-of-the-art offline RL algorithms on downstream tasks. Our findings suggest that data generation is as important as algorithmic advances for offline RL and hence requires careful consideration from the community. Code and data can be found at https://github.com/denisyarats/exorl .

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Distributionally Robust Multi-Task Reinforcement Learning via Adaptive Task Sampling

    cs.LG 2026-05 unverdicted novelty 7.0

    DRATS derives a minimax objective from a feasibility formulation of MTRL to adaptively sample tasks with the largest return gaps, leading to better worst-task performance on MetaWorld benchmarks.

  2. Ring Attention with Blockwise Transformers for Near-Infinite Context

    cs.CL 2023-10 unverdicted novelty 7.0

    Ring Attention uses blockwise computation and ring communication to let Transformers process sequences up to device-count times longer than prior memory-efficient methods.

  3. Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximization

    cs.LG 2026-06 unverdicted novelty 6.0

    ROVER pretrains transferable exploration policies by maximizing occupancy coverage with a learned resolvent world model and virtual sink state, outperforming baselines on sparse navigation tasks.

  4. AdamO: A Collapse-Suppressed Optimizer for Offline RL

    cs.LG 2026-05 unverdicted novelty 6.0

    AdamO modifies Adam with an orthogonality correction to ensure the spectral radius of the TD update operator stays below one, providing a theoretical stability guarantee for offline RL.

  5. Saliency-Guided Representation with Consistency Policy Learning for Visual Unsupervised Reinforcement Learning

    cs.CV 2026-04 unverdicted novelty 6.0

    SRCP improves zero-shot generalization of successor representation methods in visual unsupervised reinforcement learning via saliency-guided representations and consistency policies.

  6. When Dynamics Shift, Robust Task Inference Wins: Offline Imitation Learning with Behavior Foundation Models Revisited

    cs.LG 2026-05 unverdicted novelty 5.0

    Robust minimax task inference in BFMs achieves dynamics-shift robustness from nominal offline data alone and outperforms standard baselines.

  7. WestWorld: A Knowledge-Encoded Scalable Trajectory World Model for Diverse Robotic Systems

    cs.LG 2026-03 unverdicted novelty 5.0

    WestWorld introduces a scalable trajectory world model with Sys-MoE routing via system embeddings and structural embeddings for physical knowledge, pretrained on 89 environments to improve zero-shot prediction and rea...