REVIEW 6 cited by
Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent advancements in off-policy Reinforcement Learning (RL) have significantly improved sample efficiency, primarily due to the incorporation of various forms of regularization that enable more gradient update steps than traditional agents. However, many of these techniques have been tested in limited settings, often on tasks from single simulation benchmarks and against well-known algorithms rather than a range of regularization approaches. This limits our understanding of the specific mechanisms driving RL improvements. To address this, we implemented over 60 different off-policy agents, each integrating established regularization techniques from recent state-of-the-art algorithms. We tested these agents across 14 diverse tasks from 2 simulation benchmarks, measuring training metrics related to overestimation, overfitting, and plasticity loss -- issues that motivate the examined regularization techniques. Our findings reveal that while the effectiveness of a specific regularization setup varies with the task, certain combinations consistently demonstrate robust and superior performance. Notably, a simple Soft Actor-Critic agent, appropriately regularized, reliably finds a better-performing policy within the training regime, which previously was achieved mainly through model-based approaches.
Forward citations
Cited by 6 Pith papers
-
Memory Merge DQN: Sensitivity Weighted Target Updates for Stable Value Learning
Replacing the hard target-copy in DQN with a Q-value-sensitivity-weighted merge of the last K network copies yields competitive Atari performance, but only marginally beats architecture-matched baselines.
-
Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning
A constrained rational activation with denominator degree one larger than numerator and no constant term stabilizes high-UTD continuous control, while trading off long-term plasticity.
-
On the Effect of Regularization in Policy Mirror Descent
A large empirical sweep shows that in Policy Mirror Descent, MDP and Drift regularizers are partly substitutable, yet their precise combination determines temperature robustness.
-
Ensemble Elastic DQN: A Step Dependent Ensemble Approach for Reducing Overestimation in Deep Value-Based Reinforcement Learning
EEDQN sets its bootstrap horizon using a Q-value difference threshold and then uses the ensemble mean for one-step targets and the ensemble minimum for multi-step targets, reporting the best final score on four of fiv...
-
Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning
Utility-scaled partial neuron resets prevent policy collapse in long-horizon continual RL while matching or beating binary-reset and uniform-decay baselines on several benchmarks.
-
A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Forget and Grow (FoG) combines decaying replay weights for old experiences with progressive critic-network expansion to improve continuous-control reinforcement learning, beating BRO, SimBa, and TD-MPC2 on most of 41 ...
Discussion (0). Sign in to comment.