Pith. sign in

REVIEW 3 major objections 6 minor 70 references

Exploratory Diffusion Model for Unsupervised Reinforcement Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Diffusion-model denoising errors, read as state novelty, supply an intrinsic reward that improves unsupervised exploration and accelerates downstream fine-tuning.

desk verdict Strong empirical results with a shaky theoretical wrapper: the exploration reward's ELBO justification doesn't hold as stated, but the experiments alone justify a serious referee. read the letter →

arxiv 2502.07279 v2 pith:NXLXNXYC submitted 2025-02-11 cs.LG cs.AI

Chengyang Ying , Huayu Chen , Xinning Zhou , Zhongkai Hao , Hang Su , Jun Zhu This is my paper · ORCID
classification cs.LGcs.AI
keywords unsupervisedreinforcementlearningdiffusionmodelsintrinsicrewardsscore-basedexplorationstatecoveragefine-tuningcontinuouscontrolreplaybufferdensityestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Unsupervised reinforcement learning needs a way to tell the agent which states have not been explored, without any task reward. ExDM (Exploratory Diffusion Model) proposes to fit a diffusion model to the states already stored in the replay buffer and to use the model's denoising error as an intrinsic reward: states that the diffusion model fits poorly are treated as less explored, so the agent is pulled toward them. The paper argues that diffusion models have enough expressive power to represent the heterogeneous data collected during exploration, unlike the simple Gaussian or skill policies used by earlier methods, and it proves that maximal state entropy requires non-deterministic, expressive policies. A Gaussian behavior policy is kept for cheap action selection during pre-training, while the pre-trained diffusion policy is fine-tuned to downstream tasks with an alternating Q-learning and score-distillation procedure that carries a policy-improvement guarantee. If the claims hold, unsupervised pre-training becomes more effective in complicated environments, with higher state coverage and faster adaptation to downstream control tasks.

What carries the argument

Two mechanisms carry ExDM. The first is the state-diffusion-based intrinsic reward: the ELBO identity $-\log p_{\theta'}(s) \le \mathbb{E}[w_t\|\epsilon_{\theta'}(s_t|t)-\epsilon\|^2] + C$ lets the paper drop the constant and weights and use $R_{\text{score}}(s)=\mathbb{E}[\|\epsilon_{\theta'}(s_t|t)-\epsilon\|^2]$ as a novelty signal; high denoising error is interpreted as low visitation. The second is the alternating fine-tuning scheme: IQL updates the Q function via expectile regression, CEP parameterizes the energy guidance $f_{\phi_{n-1}}$ for sampling from $\pi_d e^{Q/\beta}$, and the fine-tuned diffusion policy is obtained by score distillation $\min_{\psi} \mathbb{E}\|\epsilon_{\psi}(a_t|s,t) - \epsilon_{\theta}(a_t|s,t) - f_{\phi_{n-1}}(s,a_t,t)\|^2$. A Gaussian behavior policy $\pi_g$ handles online interaction during pre-training, sidestepping the multi-step sampling cost of the diffusion policy, while Theorem 3.1 provides the theoretical motivation for needing an expressive policy in the first place.

What would settle it

Take a fixed replay buffer collected by a uniform random policy in a maze, bin its states by visitation count, and compute $R_{\text{score}}$ for each bin; the claim predicts that high $R_{\text{score}}$ occurs only in low-visitation bins. If high $R_{\text{score}}$ appears in frequently visited but structurally complicated regions, the intrinsic reward is tracking modeling difficulty rather than novelty, and the exploration mechanism fails.

Watch

Extended reading notes

Core claim

The central claim is that a diffusion model trained on the replay buffer gives both a better exploration signal and a better policy initialization than the simple models used by previous URL methods. Concretely, ExDM trains a conditional noise predictor $\epsilon_{\theta'}(s_t|t)$ on states and a diffusion policy $\epsilon_{\theta}(a_t|s,t)$ on state-action pairs. Since $-\log p_{\theta'}(s)$ is bounded above by the denoising loss plus a state-independent constant, the paper defines the score-based intrinsic reward $R_{\text{score}}(s)=\mathbb{E}_{\epsilon,t}[\|\epsilon_{\theta'}(s_t|t)-\epsilon\|^2]$ and trains a Gaussian behavior policy to maximize it, which directs the agent toward poorly fitted states. The authors also prove that the maximum-state-entropy policy is generically non-deterministic (Theorem 3.1), motivating this expressive policy class. For downstream tasks, they formulate fine-tuning as maximization of return with a KL penalty to the pre-trained policy, solve it by alternating between IQL-based Q updates and contrastive energy prediction followed by score distillation, and prove monotone policy improvement (Theorem 3.2). In experiments, ExDM reports state coverage of 0.71 in Square-large versus 0.47 for the best baseline, and better aggregate downstream scores in URLB (IQM 0.80 versus 0.71).

Load-bearing premise

The method stands on the assumption that the diffusion model's denoising error for a state is a faithful measure of how little that state has been explored, rather than a measure of how complex or poorly trained that region is.

Editorial extensions

If this is right

  • Unsupervised pre-training with ExDM reaches higher state coverage in maze environments than prior methods, including 0.71 in Square-large compared with 0.47 for the strongest baseline.
  • The pre-trained Gaussian behavior policy fine-tunes with standard DDPG to aggregate scores above prior URL methods, with IQM 0.80 versus 0.71 for the next-best method.
  • The pre-trained diffusion policy can be fine-tuned online in limited steps using alternating Q-learning and score distillation, with a policy-improvement guarantee.
  • Because the behavior policy is Gaussian, ExDM keeps data collection cheap even though the density model is a diffusion model.
  • The result that maximum state entropy requires non-deterministic policies motivates using expressive policy classes rather than simple Gaussian policies in unsupervised RL.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The score-based reward could be tested as a general density proxy: pairing it with a cheaper density estimator might preserve exploration gains at lower compute, though the paper does not test this.
  • The proof that maximum-state-entropy policies are generically non-deterministic suggests that environments with larger action spaces are exactly where expressive policy classes matter most, a prediction that could be checked in discrete-action settings.
  • The alternating fine-tuning analysis is not inherently limited to diffusion policies and could be adapted to any expressive policy class with sampling-based score distillation.
  • The diffusion model may serve as a reusable replay-buffer density model for other self-supervised objectives, such as prioritizing data by novelty during later training stages.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ExDM, an unsupervised reinforcement learning method that trains a diffusion model on the state distribution of the replay buffer during reward-free pre-training. It uses the denoising score-matching loss as a score-based intrinsic reward (Eq. 8) to guide a Gaussian behavior policy toward poorly fitted or unvisited states, and it pre-trains a diffusion policy that is later fine-tuned for downstream tasks via an alternating scheme based on IQL and contrastive energy prediction. The submission includes a theoretical claim about the complexity of maximum-state-entropy policies (Theorem 3.1), a convergence analysis for the fine-tuning objective (Theorem 3.2), and extensive experiments in Maze2d and URLB showing improved state coverage and downstream fine-tuning performance relative to existing URL baselines.

Significance. If the empirical results are reliable, ExDM is a meaningful step toward using expressive generative models in unsupervised RL: in Maze2d Square-large it reports coverage 0.71 versus 0.47 for CIC, and in URLB it reports IQM 0.80 versus 0.71 for CeSD. The experimental effort is substantial, with 7 mazes, 11 baselines, 10 seeds, four URLB domains with four tasks each, and structured ablations, and the supplementary material includes code. The central caveat is that the paper's theoretical support for the intrinsic reward and for the diffusion-policy fine-tuning update contains errors or unjustified assumptions, so the contribution currently rests mainly on the empirical evidence.

major comments (3)
  1. [Sec. 3.1, Eqs. (7)-(8)] The derivation of Rscore as a proxy for -log p_theta'(s) is not valid as stated. In Eq. (7), C is said to be independent of theta', but it is not independent of s: the DDPM ELBO includes L_T = KL(q(s_T|s) || p(s_T)), which contains a term proportional to alpha_T^2 ||s||^2. Since Rscore is used to compare different states, an additive state-dependent term changes the ranking of states and cannot be discarded. Furthermore, Eq. (8) drops the weights w_t and replaces the weighted denoising objective with an unweighted one. Unweighted denoising score matching measures local score magnitude rather than negative log-density; it can be large at sharp high-density boundaries and small in flat low-density regions. Hence a high Rscore may indicate a hard-to-fit visited region rather than a genuinely novel state. Because Rscore is the only intrinsic reward in Algorithm 1, the theoretical link between this reward and entropy-maximizing exploration is not established, although the experiments may support the heuristic.
  2. [Appendix A.1, Theorem 3.1] Theorem 3.1 is stated for arbitrary discrete MDPs, but the proof assumes that the state distributions of the M deterministic policies are i.i.d. and uniformly distributed over the simplex H. This is a distributional assumption about a random MDP, not a property of a fixed MDP; for a given transition kernel, the state distributions of deterministic policies are correlated and can be highly non-uniform. The theorem should either be restated under the explicit random-polytope assumption or qualified accordingly. In addition, the final step of Eq. (18) is not a valid bound: it replaces the binomial coefficient C_{M}^{S-1} by M, but for S>2 the coefficient can be much larger than M (e.g., S=3, M=100 gives C_2^100 = 4950). Thus the displayed proof does not establish the stated bound 1 - M S v(S)^M; a bound of the form 1 - M^{S-1} S v(S)^M would be more plausible, but the text as written is internally inconsistent.
  3. [Sec. 3.2, Eq. (14) and Appendix B.2] The distillation objective in Eq. (14) does not implement the guided-score formula that the text claims. In Eq. (13), f_phi is a positive scalar contrastive energy predictor, and in CEP the guided score is nabla_a log pi_d + nabla_a log f (or, in noise-prediction form, epsilon_psi should match epsilon_theta - sigma_t nabla_a log f). Eq. (14) instead adds f itself to the noise prediction epsilon_theta. As written, the optimal psi* cannot be the score of pi_n proportional to pi_d e^{Q/beta}, so the claim following Eq. (14) is unsupported and Algorithm 2 is not reproducible from the equations. This is load-bearing for the diffusion-policy fine-tuning contribution.
minor comments (6)
  1. [Fig. 1] Figure 1 contains the garbled placeholder "R?????" for the intrinsic reward; it should be R_score.
  2. [Table 2] Table 2 lists "Seed frames 4000" twice; one row should be removed. Also, "Number of fine-turning frames" should read "Number of fine-tuning frames."
  3. [Eq. (7)] The statement that C is "a constant independent of theta'" is technically true but misleading because C depends on s. The text should say so explicitly, especially since Eq. (8) is used to compare different states.
  4. [Theorem 3.1] The informal phrase "will fast converge to 1 with the increasing of A" should be replaced by a precise convergence-rate statement, since the proof only gives a lower bound under the uniform-vertex assumption.
  5. [Table 2, alpha_t definition] The entry "alpha_t = - (beta_1 - beta_0)/4 t^2 - beta_0/2 t" gives the exponent of alpha_t, not alpha_t itself; in the VP-SDE parameterization it should read alpha_t = exp(-beta_0 t/2 - (beta_1 - beta_0) t^2 / 4).
  6. [Sec. 5.1] The definition of state coverage ratio as the proportion of 0.01 x 0.01 bins visited should clarify whether the bins are computed from the discretized trajectories and how boundary bins are treated.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular reduction in the central exploration or fine-tuning claims; minor self-citations to CEP and PEAC are external prior work and do not make the derivation circular.

full rationale

The central exploration signal is Rscore(s)=E||eps_theta'(s_t|t)-eps||^2 (Eq. 8), justified as a tractable proxy for -log p_theta'(s) via the ELBO (Eq. 7). This is a closed-loop intrinsic reward: the diffusion model is trained on the agent's own replay buffer and its denoising loss rewards poorly fitted states. That is a standard novelty signal, not a logical circularity: neither the model nor the reward is fitted to the coverage or downstream-return targets that are later reported, and those targets are measured against external baselines. The ELBO constant and weight-dropping concern raised by the skeptical reader is real but is an approximation and robustness issue, not a self-referential reduction: Eq. (8) is not claimed to equal -log p_theta'(s), and the paper's theorems do not require that equality. The fine-tuning scheme relies on contrastive energy prediction (CEP, [39]) and PEAC ([65]), both involving overlapping authors; CEP is an externally published ICML paper with its own theorems, and the current paper does not fit any parameter to its own targets through those citations. Theorems 3.1 and 3.2 are proved from stated assumptions in Appendix A, and Section 5.6 openly states limitations such as discrete action spaces and computational cost. No equation or fitted parameter reduces the paper's claims to its inputs by construction.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central algorithm rests on standard RL machinery plus a heuristic identification of the diffusion denoising loss with state novelty. The only ad hoc element is the random-polytope model in the proof of Theorem 3.1, which is not stated as an assumption in the theorem. No new physical entities are introduced.

free parameters (5)
  • beta (KL penalty coefficient in Eq. 9)
    Trade-off between downstream reward and staying close to the pre-trained policy; value is needed in the fine-tuning objective and is not specified in the paper.
  • IQL expectile tau
    Hyperparameter for expectile regression in Eq. (31), inherited from IQL; affects Q-function training in fine-tuning.
  • Diffusion sampling steps = 15
    Number of DPM-Solver steps used for trajectory collection and evaluation; chosen following prior work, affects fine-tuning performance.
  • Gaussian exploration stddev = 0.2
    Exploration noise stddev of the behavior policy, from DDPG; influences the diversity of collected data.
  • CEP sample count K
    Number of actions sampled in Eq. (13) for contrastive energy prediction; not specified in the paper.
assumptions (6)
  • domain assumption MDP model with discount factor gamma and initial state distribution rho0
    Standard RL framework used throughout; not proven in the paper.
  • standard math The set of feasible discounted state distributions is a convex polytope whose vertices are deterministic-policy state distributions (from Eysenbach et al.)
    Used in the proof of Theorem 3.1; cited from [18].
  • ad hoc to paper In Theorem 3.1, the state distributions of the M deterministic policies are i.i.d. uniform over the simplex
    Unstated in the theorem statement but introduced in the proof (Appendix A.1); not a property of a general MDP.
  • standard math The diffusion model ELBO in Eq. (7) provides a valid upper bound on -log p_theta'(s)
    Standard result from Ho et al. [23]; used to define the score-based intrinsic reward.
  • standard math Soft policy iteration converges for the KL-regularized objective (Theorem 3.2)
    Classical soft RL result [19,20]; the paper gives a proof relying on standard regularity conditions.
  • domain assumption CEP's Theorems 3.1 and 3.2 correctly represent intermediate energy guidance
    The fine-tuning distillation uses CEP as an external method [39]; the paper does not re-prove it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploratory Diffusion Model for Unsupervised Reinforcement Learning." pith.science (2026). https://pith.science/paper/NXLXNXYC

@misc{pith2026250207279,
  author       = {Pith},
  title        = {Pith review of: Exploratory Diffusion Model for Unsupervised Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NXLXNXYC}},
  note         = {Machine review of arXiv:2502.07279}
}
read the original abstract

Unsupervised reinforcement learning (URL) aims to pre-train agents by exploring diverse states or skills in reward-free environments, facilitating efficient adaptation to downstream tasks. As the agent cannot access extrinsic rewards during unsupervised exploration, existing methods design intrinsic rewards to model the explored data and encourage further exploration. However, the explored data are always heterogeneous, posing the requirements of powerful representation abilities for both intrinsic reward models and pre-trained policies. In this work, we propose the Exploratory Diffusion Model (ExDM), which leverages the strong expressive ability of diffusion models to fit the explored data, simultaneously boosting exploration and providing an efficient initialization for downstream tasks. Specifically, ExDM can accurately estimate the distribution of collected data in the replay buffer with the diffusion model and introduces the score-based intrinsic reward, encouraging the agent to explore less-visited states. After obtaining the pre-trained policies, ExDM enables rapid adaptation to downstream tasks. In detail, we provide theoretical analyses and practical algorithms for fine-tuning diffusion policies, addressing key challenges such as training instability and computational complexity caused by multi-step sampling. Extensive experiments demonstrate that ExDM outperforms existing SOTA baselines in efficient unsupervised exploration and fast fine-tuning downstream tasks, especially in structurally complicated environments.

Figures

Figures reproduced from arXiv: 2502.07279 by the authors.

Figure 1
Figure 1. Overview of Exploratory Diffusion Model (ExDM). Different from standard RL, URL aims to explore in reward-free environments, requiring expressive policies and models to fit heterogeneous data (Theorem 3.1). During pre-training, ExDM employs the diffusion model to model the heterogeneous exploration data and calculate score-based intrinsic rewards to encourage exploration. Moreover, we adopt a Gaussian behavior polic… view at source ↗
Figure 2
Figure 2. Visualization of trajectories explored by different URL methods in the most compli￾cated mazes. Full results of all 11 algorithms in 7 mazes are in Appendix C.4. Domains Square-a Square-b Square-c Square-d Square-tree Square-bottleneck Square-large ICM 0.58 ± 0.04 0.53 ± 0.06 0.47 ± 0.07 0.49 ± 0.06 0.49 ± 0.05 0.32 ± 0.07 0.25 ± 0.04 RND 0.50 ± 0.14 0.39 ± 0.08 0.52 ± 0.16 0.32 ± 0.05 0.28 ± 0.06 0.33 ± 0.06 0.33 ±… view at source ↗
Figure 3
Figure 3. State coverage ratios of different algorithms in 7 mazes during pre-training. 0.2 0.4 0.6 0.8 ExDM CeSD BeCL CIC RND Disagreement LBS ICM SMM DIAYN LSD Median ( ) 0.2 0.4 0.6 0.8 IQM ( ) 0.2 0.4 0.6 0.8 Mean ( ) 0.4 0.6 0.8 Optimality Gap ( ) Expert Normalized Score [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Aggregate metrics [1] in URLB fine-tuned by DDPG. Each statistic for every algorithm has 160 runs (4 domains × 4 downstream tasks × 10 seeds). converges much faster with better exploration efficiency. In complicated mazes like Square-bottleneck and Square-large that ha…
Figure 5
Figure 5. Figure 5: Aggregate metrics [1] in URLB of different fine-tuning methods for diffusion policies. (a) (b) (c) [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Ablation studies on (a) pre-training timesteps; (b) Q learning choices (ExDM w/o IQL utilizes in-sample Q learning); (c) fine-tuned diffusion policy sampling steps for downstream tasks. 5.5 Ablation Studies Pre-training Steps. We first do ablation studies on pre-traine…
Figure 7
Figure 7. Figure 7: Illustration of domains with their downstream tasks in URLB [ [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Visualization of explored trajectories by URL methods in [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Visualizations of explored trajectories by URL methods in mazes [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Visualizations of explored trajectories by URL methods in mazes [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Ablation study of pre-training steps in URLB. C.7 Computing Resource In experiments, all agents are trained by GeForce RTX 2080 Ti with Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz. In maze2d / urlb, pre-training ExDM (each seed, domain) takes around 0.5 / 2 days, respe…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 29 canonical work pages

  1. [1]

    Deep reinforcement learning at the edge of the statistical precipice

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304–29320, 2021

    work page 2021

  2. [2]

    Tenenbaum, Tommi S

    Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2023

    2023

  3. [3]

    Diffusion for world modeling: Visual details matter in atari

    Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. arXiv preprint arXiv:2405.12399, 2024

    arXiv 2024

  4. [4]

    Random polytopes, convex bodies, and approximation

    Adrian Baddeley, Imre Bárány, and Rolf Schneider. Random polytopes, convex bodies, and approximation. Stochastic Geometry: Lectures given at the CIME Summer School held in Martina Franca, Italy, September 13–18, 2004, pages 77–118, 2007

    work page 2004

  5. [5]

    Constrained Ensemble Exploration for Unsupervised Skill Discovery

    Chenjia Bai, Rushuai Yang, Qiaosheng Zhang, Kang Xu, Yi Chen, Ting Xiao, and Xuelong Li. Constrained ensemble exploration for unsupervised skill discovery. arXiv preprint arXiv:2405.16030, 2024

    work page Pith review arXiv 2024

  6. [6]

    Exploration by random network distillation

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018

    2018

  7. [7]

    Explore, discover and learn: Unsupervised discovery of state-covering skills

    Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pages 1317–1327. PMLR, 2020

    work page 2020

  8. [8]

    Dime: Diffusion-based maximum entropy reinforcement learning

    Onur Celik, Zechu Li, Denis Blessing, Ge Li, Daniel Palanicek, Jan Peters, Georgia Chalvatzaki, and Gerhard Neumann. Dime: Diffusion-based maximum entropy reinforcement learning. arXiv preprint arXiv:2502.02316, 2025

    arXiv 2025

Show all 70 references
  1. [9]

    Diffusion forcing: Next-token prediction meets full-sequence diffusion

    Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. arXiv preprint arXiv:2407.01392, 2024

    2024 arXiv

  2. [10]

    Simple hierarchical planning with diffusion

    Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion. arXiv preprint arXiv:2401.02644, 2024

    2024 arXiv

  3. [11]

    Offline reinforcement learning via high-fidelity generative behavior modeling

    Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference on Learning Representations, 2023

    2023

  4. [12]

    Aligning diffusion behaviors with q-functions for efficient continuous control

    Huayu Chen, Kaiwen Zheng, Hang Su, and Jun Zhu. Aligning diffusion behaviors with q-functions for efficient continuous control. arXiv preprint arXiv:2407.09024, 2024

    2024 arXiv

  5. [13]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023

    2023

  6. [14]

    Diffusion posterior sampling for general noisy inverse problems

    Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.14687, 2022

    2022 arXiv

  7. [15]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021

    2021

  8. [16]

    Diffusion world model

    Zihan Ding, Amy Zhang, Yuandong Tian, and Qinqing Zheng. Diffusion world model. arXiv preprint arXiv:2402.03570, 2024

    2024 arXiv

  9. [17]

    Diversity is all you need: Learning skills without a reward function

    Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2018

    2018

  10. [18]

    The information geometry of unsupervised reinforcement learning

    Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The information geometry of unsupervised reinforcement learning. In International Conference on Learning Representations, 2021

    2021

  11. [19]

    Reinforcement learning with deep energy-based policies

    Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pages 1352–1361. PMLR, 2017. 10

    2017

  12. [20]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018

    2018

  13. [21]

    Idql: Implicit q-learning as an actor-critic method with diffusion policies

    Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023

    2023 arXiv

  14. [22]

    Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning

    Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. Advances in neural information processing systems, 36:64896–64917, 2023

    2023

  15. [23]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  16. [24]

    Langevin soft actor-critic: Efficient exploration through uncertainty-driven critic learning

    Haque Ishfaq, Guangyuan Wang, Sami Nur Islam, and Doina Precup. Langevin soft actor-critic: Efficient exploration through uncertainty-driven critic learning. arXiv preprint arXiv:2501.17827, 2025

    2025 arXiv

  17. [25]

    Planning with diffusion for flexible behavior synthesis

    Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pages 9902–9915. PMLR, 2022

    2022

  18. [26]

    Efficient diffusion policies for offline reinforcement learning

    Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  19. [27]

    Unsupervised skill discovery with bottleneck option learning

    Jaekyeom Kim, Seohong Park, and Gunhee Kim. Unsupervised skill discovery with bottleneck option learning. In International Conference on Machine Learning, pages 5572–5582. PMLR, 2021

    2021

  20. [28]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022

    2022

  21. [29]

    Unsuper- vised reinforcement learning with contrastive intrinsic control

    Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Unsuper- vised reinforcement learning with contrastive intrinsic control. Advances in Neural Information Processing Systems, 35:34478–34491, 2022

    2022

  22. [30]

    Urlb: Unsupervised reinforcement learning benchmark

    Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Rou...

    2021

  23. [31]

    Efficient exploration via state marginal matching

    Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274, 2019

    1906 arXiv

  24. [32]

    Hierarchical diffusion for offline decision making

    Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, pages 20035–20064. PMLR, 2023

    2023

  25. [33]

    Learning multimodal behaviors from scratch with diffusion policy gradient

    Zechu Li, Rickmer Krohn, Tao Chen, Anurag Ajay, Pulkit Agrawal, and Georgia Chalvatzaki. Learning multimodal behaviors from scratch with diffusion policy gradient. arXiv preprint arXiv:2406.00681, 2024

    2024 arXiv

  26. [34]

    Adaptdiffuser: Diffusion models as adaptive self-evolving planners

    Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. In International Conference on Machine Learning, pages 20725–20745. PMLR, 2023

    2023

  27. [35]

    Continuous control with deep reinforcement learning

    TP Lillicrap. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  28. [36]

    Aps: Active pretraining with successor features

    Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pages 6736–6747. PMLR, 2021

    2021

  29. [37]

    Behavior from the void: Unsupervised active pre-training

    Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34:18459–18473, 2021

    2021

  30. [38]

    Energy-guided diffusion sampling for offline-to-online reinforcement learning

    Xu-Hui Liu, Tian-Shuo Liu, Shengyi Jiang, Ruifeng Chen, Zhilong Zhang, Xinwei Chen, and Yang Yu. Energy-guided diffusion sampling for offline-to-online reinforcement learning. arXiv preprint arXiv:2407.12448, 2024

    2024 arXiv

  31. [39]

    Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning

    Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, pages 22825–22855. PMLR, 2023. 11

    2023

  32. [40]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775–5787, 2022

    2022

  33. [41]

    Synthetic experience replay.Advances in Neural Information Processing Systems, 36, 2024

    Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay.Advances in Neural Information Processing Systems, 36, 2024

    2024

  34. [42]

    Soft diffusion actor-critic: Efficient online reinforcement learning for diffusion policy

    Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Soft diffusion actor-critic: Efficient online reinforcement learning for diffusion policy. arXiv preprint arXiv:2502.00361, 2025

    2025 arXiv

  35. [43]

    Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone

    Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone. arXiv preprint arXiv:2412.06685, 2024

    2024 arXiv

  36. [44]

    Curiosity-driven exploration via latent bayesian surprise

    Pietro Mazzaglia, Ozan Catal, Tim Verbelen, and Bart Dhoedt. Curiosity-driven exploration via latent bayesian surprise. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7752–7760, 2022

    2022

  37. [45]

    Lipschitz-constrained unsupervised skill discovery

    Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz-constrained unsupervised skill discovery. In International Conference on Learning Representations, 2022

    2022

  38. [46]

    Metra: Scalable unsupervised rl with metric-aware abstraction

    Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metric-aware abstraction. arXiv preprint arXiv:2310.08887, 2023

    2023 arXiv

  39. [47]

    Curiosity-driven exploration by self-supervised prediction

    Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778–2787. PMLR, 2017

    2017

  40. [48]

    Self-supervised exploration via disagreement

    Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International conference on machine learning, pages 5062–5071. PMLR, 2019

    2019

  41. [49]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019

    1910 arXiv

  42. [50]

    Learning a diffusion model policy from rewards via q-score matching

    Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching. arXiv preprint arXiv:2312.11752, 2023

    2023 arXiv

  43. [51]

    Diffusion policy policy optimization

    Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint arXiv:2409.00588, 2024

    2024 arXiv

  44. [52]

    Photorealistic text-to- image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in neural informatio...

    2022

  45. [53]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  46. [54]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. PMLR, 2015

    2015

  47. [55]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021

    2021

  48. [56]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021

    2021

  49. [57]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  50. [58]

    Deepmind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  51. [59]

    Prioritized generative replay

    Renhao Wang, Kevin Frans, Pieter Abbeel, Sergey Levine, and Alexei A Efros. Prioritized generative replay. arXiv preprint arXiv:2410.18082, 2024. 12

    2024 arXiv

  52. [60]

    Diffusion policies as an expressive policy class for offline reinforcement learning

    Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023

    2023

  53. [61]

    A problem in geometric probability

    James G Wendel. A problem in geometric probability. Mathematica Scandinavica, 11(1):109–111, 1962

    1962

  54. [62]

    Leveraging skills from unlabeled prior data for efficient online exploration

    Max Wilcoxson, Qiyang Li, Kevin Frans, and Sergey Levine. Leveraging skills from unlabeled prior data for efficient online exploration. arXiv preprint arXiv:2410.18076, 2024

    2024 arXiv

  55. [63]

    Policy representation via diffusion probability model for reinforcement learning

    Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305.13122, 2023

    2023 arXiv

  56. [64]

    Behavior contrastive learning for unsupervised skill discovery

    Rushuai Yang, Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang, Peng Liu, and Xuelong Li. Behavior contrastive learning for unsupervised skill discovery. In International Conference on Machine Learning, pages 39183–39204. PMLR, 2023

    2023

  57. [65]

    Peac: Unsupervised pre-training for cross-embodiment reinforcement learning

    Chengyang Ying, Hao Zhongkai, Xinning Zhou, Xuezhou Xu, Hang Su, Xingxing Zhang, and Jun Zhu. Peac: Unsupervised pre-training for cross-embodiment reinforcement learning. Advances in Neural Information Processing Systems, 37:54632–54669, 2024

    2024

  58. [66]

    Towards safe reinforcement learning via constraining conditional value-at-risk

    Chengyang Ying, Xinning Zhou, Hang Su, Dong Yan, Ning Chen, and Jun Zhu. Towards safe reinforcement learning via constraining conditional value-at-risk. arXiv preprint arXiv:2206.04436, 2022

    2022 arXiv

  59. [67]

    Automatic intrinsic reward shaping for exploration in deep reinforcement learning

    Mingqi Yuan, Bo Li, Xin Jin, and Wenjun Zeng. Automatic intrinsic reward shaping for exploration in deep reinforcement learning. In International Conference on Machine Learning, pages 40531–40554. PMLR, 2023

    2023

  60. [68]

    Euclid: Towards efficient unsupervised reinforcement learning with multi-choice dynamics model

    Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Jinyi Liu, Yingfeng Chen, and Changjie Fan. Euclid: Towards efficient unsupervised reinforcement learning with multi-choice dynamics model. arXiv preprint arXiv:2210.00498, 2022

    arXiv

  61. [69]

    A mixture of surprises for unsupervised reinforcement learning

    Andrew Zhao, Matthieu Lin, Yangguang Li, Yong-Jin Liu, and Gao Huang. A mixture of surprises for unsupervised reinforcement learning. Advances in Neural Information Processing Systems, 35:26078– 26090, 2022

    2022

  62. [70]

    ∞X i=0 γi (R(si, ai) − βDKL(π(·|si)∥πd(·|si))) s0 = s, a0 = a # =Es∼ρ0,a∼π(·|s)

    Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Haoquan Guo, Tingting Chen, and Weinan Zhang. Diffusion models for reinforcement learning: A survey. arXiv preprint arXiv:2311.01223, 2023. 13 A Theoretical analyses A.1 The details and proof of Theorem 3.1 In t...

    2023 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.