The reviewed record of science sign in
Pith

arxiv: 2312.11752 · v5 · pith:TGTLWJTB · submitted 2023-12-18 · cs.LG · cs.AI

Learning a Diffusion Model Policy from Rewards via Q-Score Matching

Michael Psenka , Alejandro Escontrela , Pieter Abbeel , Yi Ma This is my paper

Reviewed by Pithpith:TGTLWJTBopen to challenge →

classification cs.LG cs.AI
keywords diffusionmodellearningmatchingpoliciespolicyq-scorestructure
0
0 comments X
read the original abstract

Diffusion models have become a popular choice for representing actor policies in behavior cloning and offline reinforcement learning. This is due to their natural ability to optimize an expressive class of distributions over a continuous space. However, previous works fail to exploit the score-based structure of diffusion models, and instead utilize a simple behavior cloning term to train the actor, limiting their ability in the actor-critic setting. In this paper, we present a theoretical framework linking the structure of diffusion model policies to a learned Q-function, by linking the structure between the score of the policy to the action gradient of the Q-function. We focus on off-policy reinforcement learning and propose a new policy update method from this theory, which we denote Q-score matching. Notably, this algorithm only needs to differentiate through the denoising model rather than the entire diffusion model evaluation, and converged policies through Q-score matching are implicitly multi-modal and explorative in continuous domains. We conduct experiments in simulated environments to demonstrate the viability of our proposed method and compare to popular baselines. Source code is available from the project website: https://michaelpsenka.io/qsm.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning

    cs.LG 2026-06 unverdicted novelty 7.0

    QGF performs test-time policy optimization for flow models in RL by guiding a behavior-cloned reference policy with value-function gradients, achieving strong results on high-dimensional offline RL benchmarks without ...

  2. Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance

    cs.RO 2026-05 unverdicted novelty 7.0

    CGPO integrates training-free critic guidance into diffusion denoising to produce high-Q actions as regression targets, yielding SOTA results on MuJoCo locomotion and successful Franka arm grasping.

  3. Reinforcement Learning via Value Gradient Flow

    cs.LG 2026-04 unverdicted novelty 7.0

    VGF solves behavior-regularized RL by transporting particles from a reference distribution to the value-induced optimal policy via discrete value-guided gradient flow.

  4. EXPO: Stable Reinforcement Learning with Expressive Policies

    cs.LG 2025-07 conditional novelty 7.0

    EXPO stabilizes online RL for expressive policies by training a base policy with imitation and using a lightweight Gaussian edit policy to select higher-value actions on the fly for sampling and TD backups.

  5. Steering Your Diffusion Policy with Latent Space Reinforcement Learning

    cs.RO 2025-06 unverdicted novelty 7.0

    DSRL steers pretrained diffusion policies for robotics by applying RL to their latent noise inputs, achieving sample-efficient real-world adaptation with only black-box access.

  6. GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios

    cs.LG 2026-06 unverdicted novelty 6.0

    GenPO++ achieves exact Jacobian-free likelihood ratio computation for generative flow policies by embedding history states as auxiliary memory in a high-order reversible ODE solver.

  7. Adversarial Dual On-Policy Distillation from Expressive Teacher

    cs.LG 2026-05 unverdicted novelty 6.0

    FA-OPD co-trains a flow-matching teacher and MLP student via adversarial dual on-policy distillation, improving robustness over baselines on six robot benchmarks with noisy or limited demonstrations.

  8. Global Convergence of Sampling-Based Nonconvex Optimization through Diffusion-Style Smoothing

    cs.LG 2026-05 unverdicted novelty 6.0

    Recasts sampling-based nonconvex optimization as smoothed gradient descent to obtain non-asymptotic convergence guarantees and introduces the DIDA annealed algorithm that converges to the global optimum.

  9. Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning

    cs.MA 2026-05 unverdicted novelty 6.0

    Decentralized diffusion policies trained with importance sampling score matching enhance exploration and performance in cooperative MARL over Gaussian policy baselines.

  10. How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?

    cs.LG 2026-02 unverdicted novelty 6.0

    ALGD augments the Lagrangian to locally convexify the energy landscape in diffusion models, stabilizing safe RL training and generation without changing optimal policies.

  11. Diffusion Policy Policy Optimization

    cs.RO 2024-09 unverdicted novelty 6.0

    DPPO fine-tunes diffusion policies via policy gradients and outperforms prior RL approaches for diffusion policies and PG-tuned alternatives on robot benchmarks while enabling stable training and hardware deployment.

  12. Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimization

    cs.LG 2026-05 unverdicted novelty 5.0

    MBDPO reformulates policy optimization as a diffusion process over searched trajectories in latent world models to reduce misalignment between search and value learning.