REVIEW 9 cited by
Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Using a high Update-To-Data (UTD) ratio, model-based methods have recently achieved much higher sample efficiency than previous model-free methods for continuous-action DRL benchmarks. In this paper, we introduce a simple model-free algorithm, Randomized Ensembled Double Q-Learning (REDQ), and show that its performance is just as good as, if not better than, a state-of-the-art model-based algorithm for the MuJoCo benchmark. Moreover, REDQ can achieve this performance using fewer parameters than the model-based method, and with less wall-clock run time. REDQ has three carefully integrated ingredients which allow it to achieve its high performance: (i) a UTD ratio >> 1; (ii) an ensemble of Q functions; (iii) in-target minimization across a random subset of Q functions from the ensemble. Through carefully designed experiments, we provide a detailed analysis of REDQ and related model-free algorithms. To our knowledge, REDQ is the first successful model-free DRL algorithm for continuous-action spaces using a UTD ratio >> 1.
Forward citations
Cited by 9 Pith papers
-
StaQ it! Growing neural networks for Policy Mirror Descent
StaQ, a finite-memory Policy Mirror Descent algorithm, converges to the optimal entropy-regularized policy with a sufficiently large window of past Q-functions and performs competitively with baselines.
-
The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
Introduces LEAST, an adaptive early-episode-stopping rule for off-policy deep RL that improves learning efficiency on MuJoCo and DeepMind Control benchmarks.
-
Safe Planning and Policy Optimization via World Model Learning
SPOWL is a model-based safe RL method that uses a value-equivalent world model, a Lagrangian-trained safe policy, and adaptive planning thresholds to achieve low-cost, high-reward control on SafetyGymnasium tasks.
-
Universal Value-Function Uncertainties
UVU measures value-function uncertainty by the TD-trained prediction error between an online network and a fixed random target, and in the infinite-width NTK limit this error exactly matches ensemble variance.
-
Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only
PORL fine-tunes a pre-trained policy online by first collecting epsilon-greedy interaction data and learning a fresh Q-function from scratch, removing the need for pre-trained critics or offline data.
-
Hadamax Encoding: Elevating Performance in Model-Free Atari
Hadamax, a Hadamard-product and max-pooling encoder, improves PQN's median human-normalized Atari-57 score by about 80% with no algorithmic changes.
-
What Matters for Simulation to Online Reinforcement Learning on Real Robots
Sim-to-online RL on three real robots is stabilized by retaining data, warm-starting the replay buffer, and using asymmetric actor-critic updates with a low actor learning rate.
-
Exploring the robustness of TractOracle methods in RL-based tractography
Oracle-guided RL tractography remains substantially better than non-oracle baselines across five dMRI datasets, and Iterative Reward Training raises the number of streamlines accepted by anatomical filters.
-
A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Forget and Grow (FoG) combines decaying replay weights for old experiences with progressive critic-network expansion to improve continuous-control reinforcement learning, beating BRO, SimBa, and TD-MPC2 on most of 41 ...
Discussion (0). Sign in to comment.