REVIEW 6 cited by
Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In offline reinforcement learning, weighted regression is a common method to ensure the learned policy stays close to the behavior policy and to prevent selecting out-of-sample actions. In this work, we show that due to the limited distributional expressivity of policy models, previous methods might still select unseen actions during training, which deviates from their initial motivation. To address this problem, we adopt a generative approach by decoupling the learned policy into two parts: an expressive generative behavior model and an action evaluation model. The key insight is that such decoupling avoids learning an explicitly parameterized policy model with a closed-form expression. Directly learning the behavior policy allows us to leverage existing advances in generative modeling, such as diffusion-based methods, to model diverse behaviors. As for action evaluation, we combine our method with an in-sample planning technique to further avoid selecting out-of-sample actions and increase computational efficiency. Experimental results on D4RL datasets show that our proposed method achieves competitive or superior performance compared with state-of-the-art offline RL methods, especially in complex tasks such as AntMaze. We also empirically demonstrate that our method can successfully learn from a heterogeneous dataset containing multiple distinctive but similarly successful strategies, whereas previous unimodal policies fail.
Forward citations
Cited by 6 Pith papers
-
From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning
Residual off-policy RL with selective BC regularization and value-guided sampling contracts a pretrained generative robot policy around successful actions, reaching high success on hard long-horizon tasks from pixels ...
-
Decision Flow Policy Optimization
Decision Flow frames the gradual action generation of flow-based policies as a flow MDP and updates the flow policy with flow-level value functions, reporting state-of-the-art results on several D4RL tasks.
-
Skill Expansion and Composition in Parameter Space
PSEC shows that weighting and summing LoRA skill modules inside a diffusion policy network outperforms composing the same skills in action or noise space across D4RL, DSRL, DMC, and Meta-World tasks.
-
A Single Diffusion-Policy Controller for Multi-Task Block Pushing with Zero-Shot Sim-to-Real Transfer
One diffusion policy trained via energy-guided RL solves multi-shape block pushing without demos and transfers zero-shot to real robots under varied conditions.
-
Value Flows
Value Flows fits the full return distribution in RL with a flow-matching critic and reweights its learning objective by estimated return variance; the central theoretical guarantee does not follow from the stated equations.
-
Offline Reinforcement Learning with Penalized Action Noise Injection
Injecting noise-perturbed actions into offline Q-learning with a distance penalty improves D4RL performance over IQL and TD3 baselines, formalized as Q-learning in a Noisy Action MDP.
Discussion (0). Continue with ORCID to comment.