REVIEW 6 cited by
Off-Policy Deep Reinforcement Learning without Exploration
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.
Forward citations
Cited by 6 Pith papers
-
Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.
-
Good Rankers, Bad Objectives: Bilinear Contrastive Critics under Expressive Policy Search
Bilinear contrastive critics remain good compatibility rankers but are unsafe to maximize for action selection; cosine bounding does not fix value decalibration, while Bellman TD-Q does.
-
Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?
On six robot-manipulation tasks, offline Q-pretraining does not accelerate online RL fine-tuning from a pretrained policy, while seeding the replay buffer with rollouts from an ensemble of policies (IPE) improves fina...
-
Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation
CQ2L uses Morse-network uncertainty to select in-distribution action queries and to scale CQL's regularization, reporting higher D4RL scores than the prior OAP method.
-
FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning
FlowQ uses energy-guided flow matching to learn an offline RL policy approximating π(a|s) ∝ πβ(a|s) exp(Q(s,a)) with guidance applied during training rather than at inference.
-
A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Forget and Grow (FoG) combines decaying replay weights for old experiences with progressive critic-network expansion to improve continuous-control reinforcement learning, beating BRO, SimBa, and TD-MPC2 on most of 41 ...
Discussion (0). Sign in to comment.