Pith. sign in

REVIEW 3 major objections 6 minor 38 references

Adaptive Diffusion Policy Optimization for Robotic Manipulation

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Replacing the inner-loop optimizer with an adaptive-gradient update rule makes six diffusion-based reinforcement learning methods train faster and more stably on robotic control benchmarks.

desk verdict The ADAPG twist is a plausible minor idea, but the paper's central empirical claim is unsupported because Algorithm 1 is under-specified and the comparison is uncontrolled. read the letter →

arxiv 2505.08376 v1 pith:MTN35MQY submitted 2025-05-13 cs.RO cs.AIcs.LG

Huiyun Jiang , Zhuang Yang This is my paper · ORCID
classification cs.ROcs.AIcs.LG
keywords diffusionpolicyreinforcementlearningadaptivegradientmethodsroboticmanipulationcontinuouscontrolADAPGfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Diffusion models can represent robot policies as rich, multimodal action distributions, but optimizing those policies with reinforcement learning has been slow and unstable. This paper claims that part of the trouble is the optimizer rather than the objective, and proposes ADPO: a drop-in replacement of the inner-loop policy update with an adaptive-gradient rule called ADAPG. Applying ADPO to six existing diffusion-based RL methods (DPPO, DIPO, IDQL, DAWR, QSM, DQL) is reported to give faster convergence, more stable training, and better or equal final rewards or success rates across three locomotion and three manipulation tasks. A reader should care because this suggests many diffusion-based RL methods could be improved without redesigning their loss functions, merely by changing how each gradient step is computed.

What carries the argument

The machinery is the ADAPG update, an adaptive-gradient rule whose step at iteration $i$ uses two running averages, $m_i = \beta_1 m_{i-1} + (1-\beta_1) g_i$ and $v_i = \beta_2 v_{i-1} + (1-\beta_2) g_i^2$, then forms an intermediate point $h_i = \theta_{i-1} - \eta \frac{(1-\lambda)g_i + \lambda m_i}{\sqrt{v_i}+\varepsilon}$ and finally sets $\theta_i = \omega h_i + (1-\omega) h_{i-1}$. The discount $\lambda$ interpolates between the raw gradient and the momentum average, while $\omega$ blends successive intermediate points, which is intended to damp oscillation and error accumulation from noisy gradients. This update is inserted as the inner loop of Algorithm 1 in place of the AdamW weight-decay update, so the actor and critic loss formulas of the six baseline methods are unchanged; only the optimizer minimizing them changes.

What would settle it

Re-run ADPPO and DPPO on Hopper-v2 with identical inner-loop step counts and identical discount $\lambda$, changing only whether the inner update is Eq. (23) or Eq. (24); if the reward gap reported in Table I does not appear, the central claim is not supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that the update rule in Eq. (24), ADAPG, is a better inner-loop optimizer for diffusion-based reinforcement learning than the AdamW-style update in Eq. (23) that it replaces. In the authors' experiments, every one of the six wrapped methods matches or beats its baseline in the qualitative comparison of Table I; ADPPO, ADIPO, and ADQL are marked above baseline in most of the six tasks, while the remaining variants are on par in some tasks and above in others. The gains are reported as clearest in the harder manipulation tasks, where the ADPO variants preserve training stability and converge faster. The paper presents this as evidence that adaptive policy gradient methods, previously little used in RL, are a generally useful accelerator for diffusion policies.

Load-bearing premise

The reported gains depend on unstated details of the inner update loop: how many gradient steps are taken, what the discount factor is, and which objective the gradient is computed from, so the improvement could come from those choices rather than from the ADAPG rule itself.

Editorial extensions

If this is right

  • Swapping in ADAPG requires no change to the actor or critic loss of the wrapped method, so the framework should port directly to other diffusion-policy RL objectives.
  • The reported gains are largest on the long-horizon manipulation tasks, suggesting ADPO is most useful when training stability is hardest to maintain.
  • The stabilizer $\varepsilon = 10^{-11}$ is reported to work across all tested tasks without tuning, removing one hyperparameter from the pipeline.
  • The momentum weight $\omega$ is task- and method-dependent, with best values ranging from $0.6$ to $1.5$, so practical deployment still requires tuning it per setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the paper leaves implicit: if ADAPG's benefit comes from better per-gradient adaptation, the gap between ADPO and baseline should widen when gradient noise is high but the number of inner steps is held equal; this could be checked by measuring update variance on a single task.
  • Because Algorithm 1 does not state the inner-loop step count $T$ or the discount $\lambda$, part of the reported speedup could come from additional gradient steps per batch; a controlled replication holding $T$ and $\lambda$ fixed across optimizers would separate the rule's contribution from added computation.
  • Since Eq. (24) recovers different optimizer behaviors as $\lambda$ and $\omega$ vary, ADPO can be viewed as a family of adaptive steps; searching over $(\lambda,\omega)$ per task and comparing against tuned AdamW would test whether the benefit is the rule itself or simply better-tuned adaptive steps.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ADPO, an Adam-based adaptive gradient framework for fine-tuning diffusion-model policies in reinforcement learning. It replaces the optimizer step in six diffusion-based RL methods (DPPO, DIPO, IDQL, DAWR, QSM, DQL) with an ADAPG update (Eq. 24) that interpolates between adaptive gradient estimators using a Katyusha-style momentum term. Experiments on Hopper-v2, HalfCheetah-v2, Walker2d-v2, Lift, Can, and Square compare the six ADPO variants against the original methods, reporting a qualitative Y/-- comparison in Table I and a hyperparameter sensitivity study for epsilon and omega. The paper claims faster and more stable training with better or comparable final performance.

Significance. The problem is timely: diffusion policies are expressive but notoriously hard to optimize in RL, and a generic optimizer-level drop-in that accelerates several existing methods would be practically useful. The paper also deserves credit for attempting a broad benchmark across six methods and two benchmark suites, and for explicitly studying epsilon and omega sensitivity. However, the empirical evidence as presented does not substantiate the central claim. The algorithm is under-specified (T, rho, lambda, beta1, beta2), the comparison is uncontrolled with respect to inner-loop gradient steps and per-task hyperparameter tuning, and the results are reported without numbers or significance tests. If the missing details and controls are supplied in a revision, the contribution could be of moderate practical value; as written, the load-bearing evidence is missing.

major comments (3)
  1. [Algorithm 1, Eq. (24), Table III] Algorithm 1 is under-specified in a way that undermines the controlled comparison. The inner loop 'for i = 1 to T do' uses T, which Table III defines as the environment horizon (1000 for GYM, 300 for ROBOMIMIC). If ADPO takes T gradient steps per batch while the baselines perform one optimizer update, the gains in Table I could be due to extra gradient steps rather than to the ADAPG rule. The gradient is written as g_i = grad_theta V^{pi_theta}(rho), where rho is never defined and no derivation connects this objective to the actor losses in Eqs. (10), (13), (16), (17), (21), and (22). Furthermore, lambda in Eq. (24) is never assigned a value, and beta1 and beta2 are not specified for the experiments. Please specify T (or rename the horizon), define rho, state lambda, beta1, and beta2, and run a controlled comparison with the same number of inner optimizer steps for ADPO and baselines.
  2. [Section V.B, Table I, Figures 2 and 3] The headline empirical claim is supported only by a qualitative Y/-- table without numerical values, error bars, or significance tests. Table I assigns 'Y' or '--' but reports no quantities; Figures 2 and 3 show curves with overlapping shaded regions, and for several pairs (e.g., HalfCheetah ADPPO vs DPPO in Figure 2(c)) the curves are visually indistinguishable. The statement in Section V.B that 'ADPO can significantly improve the performance' is therefore not substantiated. Provide final mean +/- std tables, per-seed results, and a significance test (e.g., paired t-test or bootstrap) for each ADPO-baseline pair.
  3. [Table II and Section V.C] The comparison is not controlled because omega in Eq. (24) is tuned per task and per method (Table II), and the ablation in Section V.C shows omega is sensitive to the environment. If ADPO's advantage depends on per-task selection of omega while baselines use fixed default hyperparameters, the Y/-- scores could reflect hyperparameter tuning rather than the ADAPG update. Please report the search procedure, show that the same tuning budget is given to baselines, or provide results for a fixed omega.
minor comments (6)
  1. [Section V.B] The text refers to 'Figures 1.a and 1.b' but the actual figures are numbered 2 and 3; update the cross-references.
  2. [Section III.C] The text states that GYM tasks are discrete actions, but Hopper-v2, HalfCheetah-v2, and Walker2d-v2 are continuous control tasks.
  3. [Eq. (18)] Equation (18) writes Q_phi1, Q_phi1 instead of Q_phi1, Q_phi2 for the two Q-networks; fix the typo.
  4. [Algorithm 1] Algorithm 1 initializes Q_phi3 only for double Q learning; since the paper uses double Q-learning, the reference to Q_phi3 is confusing and should be removed or justified.
  5. [Section V.C] The sentence 'This indicates that epsilon is robust and doesn't require extensive tuning for different environments' is duplicated verbatim; remove the repetition.
  6. [Algorithm 1] The line 'update actor policy pi_theta using Eq. (10), (11), (13), (16), (17), (21), (22)' lists all method losses together, but each ADPO variant presumably uses only its own loss; clarify how the correct loss is selected for each method.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ADPO's central claim is an empirical comparison using a newly assembled adaptive optimizer, and no prediction is equivalent to an input by construction.

full rationale

The paper's central claim is empirical: replacing the policy optimizer of six diffusion-based RL methods with the ADAPG update in Eq. (24) yields better or comparable performance. Nothing in the derivation chain reduces this claim to its own inputs. ADAPG is presented as an explicit composition of known components (Adam-style first/second moments, a discount factor lambda, and a Katyusha-style momentum term); Eq. (24) is an update rule, not a theorem derived from the baseline losses, so there is no self-definitional step. The per-task values of omega in Table II are tuned hyperparameters, and reporting results after tuning is standard empirical practice; the outcome is not encoded in the ADAPG update by construction, so this is not a fitted-input-called-prediction circularity. The paper contains no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled in via citation. The most serious issues are the under-specification of Algorithm 1 (the objective rho for the gradient, the inner-loop count T which is also the environment horizon, and the free parameter lambda in Eq. (24)) and the possibility of uncontrolled extra gradient steps or tuned omega conferring an advantage. These are reproducibility, correctness, and experimental-control risks, not circularity: they do not make the reported result equivalent to the method's definition. Therefore the appropriate circularity score is 0, with the caveat that the empirical comparison should be re-examined for fairness and reproducibility rather than for circular reasoning.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities, but it relies on four free parameters, two of which are absent from the experimental description. The main load-bearing assumption is that the ADAPG update, which is a minor variant of Adam, provides stable and fast convergence for diffusion policy RL without any supporting analysis.

free parameters (4)
  • omega (Katyusha momentum mixing weight) = 0.6 to 1.5 depending on task and method (Table II)
    Tuned separately for each task and each method to maximize performance; no cross-task selection criterion is given.
  • lambda (interpolation discount in Eq. 24) = not reported
    Appears in the ADAPG update but its value is never specified in the experimental setup, so it is either tuned or left at an undisclosed default.
  • T (inner-loop optimizer steps in Algorithm 1) = not reported
    Algorithm 1 loops i=1..T and updates theta via Eq. (24), but T is never stated; the same symbol is also used for the environment horizon in Table III.
  • epsilon (denominator smoothing) = 1e-11
    Chosen by hand in Section V.B and reported as robust, but still a free choice that affects the update scale.
assumptions (4)
  • domain assumption Adaptive gradient methods such as Adam and RMSProp improve convergence of RL policy optimization
    Invoked in Section II.B to motivate ADAPG; no proof is given for the RL setting.
  • standard math Katyusha momentum reduces oscillation and avoids local optima
    Taken from reference [3] and applied heuristically to the Adam-style update without adapting Katyusha's convergence conditions or proving them for this setting.
  • standard math The policy gradient with advantage (Eq. 4) gives unbiased gradient estimates
    Assumed in Section III.A and used to motivate the actor losses for all baselines.
  • domain assumption A diffusion policy can embed an RL MDP by adding a denoising Markov chain (Eq. 8)
    Follows the DPPO framework [25]; the paper does not re-derive the validity of this two-level MDP construction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Diffusion Policy Optimization for Robotic Manipulation." pith.science (2026). https://pith.science/paper/MTN35MQY

@misc{pith2026250508376,
  author       = {Pith},
  title        = {Pith review of: Adaptive Diffusion Policy Optimization for Robotic Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MTN35MQY}},
  note         = {Machine review of arXiv:2505.08376}
}
read the original abstract

Recent studies have shown the great potential of diffusion models in improving reinforcement learning (RL) by modeling complex policies, expressing a high degree of multi-modality, and efficiently handling high-dimensional continuous control tasks. However, there is currently limited research on how to optimize diffusion-based polices (e.g., Diffusion Policy) fast and stably. In this paper, we propose an Adam-based Diffusion Policy Optimization (ADPO), a fast algorithmic framework containing best practices for fine-tuning diffusion-based polices in robotic control tasks using the adaptive gradient descent method in RL. Adaptive gradient method is less studied in training RL, let alone diffusion-based policies. We confirm that ADPO outperforms other diffusion-based RL methods in terms of overall effectiveness for fine-tuning on standard robotic tasks. Concretely, we conduct extensive experiments on standard robotic control tasks to test ADPO, where, particularly, six popular diffusion-based RL methods are provided as benchmark methods. Experimental results show that ADPO acquires better or comparable performance than the baseline methods. Finally, we systematically analyze the sensitivity of multiple hyperparameters in standard robotics tasks, providing guidance for subsequent practical applications. Our video demonstrations are released in https://github.com/Timeless-lab/ADPO.git.

Figures

Figures reproduced from arXiv: 2505.08376 by the authors.

Figure 1
Figure 1. The first line shows the case of GYM benchmark, and [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Comparison of ADAPG diffusion-based RL methods [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. A comparison of the performance of ADPPO with [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 13 canonical work pages

  1. [1]

    Diffusion policies for out-of-distribution generalization in offline reinforcement learning

    Suzan Ece Ada, Erhan Oztop, and Emre Ugur. Diffusion policies for out-of-distribution generalization in offline reinforcement learning. IEEE Robotics and Automation Letters , 9(4):3116–3123, 2024

    work page 2024

  2. [2]

    Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657 , 2022

    Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657 , 2022

    arXiv 2022

  3. [3]

    Katyusha: The first direct acceleration of stochastic gradient methods

    Zeyuan Allen-Zhu. Katyusha: The first direct acceleration of stochastic gradient methods. Journal of Machine Learning Research , 18(221):1– 51, 2018

    work page 2018

  4. [4]

    Pattern recognition and machine learning , volume 4

    Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning , volume 4. Springer, 2006

    2006

  5. [5]

    Openai gym

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016

    arXiv 2016

  6. [6]

    A survey on generative diffusion models

    Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-Ann Heng, and Stan Z Li. A survey on generative diffusion models. IEEE Transactions on Knowledge and Data Engineering , 2024

    2024

  7. [7]

    Offline reinforcement learning via high-fidelity generative behavior modeling

    Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209.14548 , 2022

    arXiv 2022

  8. [8]

    Diffusiondet: Diffusion model for object detection

    Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVF international conference on computer vision , pages 19830–19843, 2023

    work page 2023

Show all 38 references
  1. [9]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research , page 02783649241273668, 2023

    2023

  2. [10]

    A reinforcement learning based artificial bee colony algorithm with application in robot path planning

    Yibing Cui, Wei Hu, and Ahmed Rahmani. A reinforcement learning based artificial bee colony algorithm with application in robot path planning. Expert Systems with Applications , 203:117389, 2022

    2022

  3. [11]

    Diffusion-based reinforcement learning via q-weighted variational policy optimization

    Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-based reinforcement learning via q-weighted variational policy optimization. arXiv preprint arXiv:2405.16173, 2024

    2024 arXiv

  4. [12]

    Behavior-regularized diffusion policy optimiza- tion for offline reinforcement learning

    Chen-Xiao Gao, Chenyang Wu, Mingjun Cao, Chenjun Xiao, Yang Yu, and Zongzhang Zhang. Behavior-regularized diffusion policy optimiza- tion for offline reinforcement learning. arXiv preprint arXiv:2502.04778, 2025

    2025 arXiv

  5. [13]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861–1870. Pmlr, 2018

    2018

  6. [14]

    Idql: Implicit q-learning as an actor-critic method with diffusion policies

    Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023

    2023 arXiv

  7. [15]

    Denoising diffusion prob- abilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion prob- abilistic models. Advances in neural information processing systems , 33:6840–6851, 2020

    2020

  8. [16]

    Efficient diffusion policies for offline reinforcement learning

    Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems , 36:67195–67212, 2023

    2023

  9. [17]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014

    2014 arXiv

  10. [18]

    Hierarchical diffusion for offline decision making

    Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning , pages 20035–20064. PMLR, 2023

    2023

  11. [19]

    What matters in learning from of- fline human demonstrations for robot manipulation

    Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from of- fline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298, 2021

    2021 arXiv

  12. [20]

    Advantage-weighted regression: Simple and scalable off-policy rein- forcement learning

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy rein- forcement learning. arXiv preprint arXiv:1910.00177 , 2019

    1910 arXiv

  13. [21]

    Reinforcement learning by reward- weighted regression for operational space control

    Jan Peters and Stefan Schaal. Reinforcement learning by reward- weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning , pages 745–750, 2007

    2007

  14. [22]

    Deep reinforcement learning for autonomous driving in amazon web services deepracer

    Bohdan Petryshyn, Serhii Postupaiev, Soufiane Ben Bari, and Armantas Ostreika. Deep reinforcement learning for autonomous driving in amazon web services deepracer. Information, 15(2):113, 2024

    2024

  15. [23]

    Learning a diffusion model policy from rewards via q-score matching

    Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching. arXiv preprint arXiv:2312.11752 , 2023

    2023 arXiv

  16. [24]

    An adaptive reinforcement learning-based multimodal data fusion framework for human–robot confrontation gaming

    Wen Qi, Haoyu Fan, Hamid Reza Karimi, and Hang Su. An adaptive reinforcement learning-based multimodal data fusion framework for human–robot confrontation gaming. Neural Networks , 164:489–496, 2023

    2023

  17. [25]

    Diffusion policy policy optimization

    Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint arXiv:2409.00588, 2024

    2024 arXiv

  18. [26]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889–1897. PMLR, 2015

    2015

  19. [27]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  20. [28]

    Deep unsupervised learning using nonequilibrium thermody- namics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermody- namics. In International conference on machine learning , pages 2256–

  21. [29]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  22. [30]

    Policy gradient methods for reinforcement learning with function approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Man- sour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999

    1999

  23. [31]

    Reinforcement learning: An introduction

    Sebastian Thrun and Michael L Littman. Reinforcement learning: An introduction. AI Magazine , 21(1):103–103, 2000

    2000

  24. [32]

    Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude

    Tijmen Tieleman. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning , 4(2):26, 2012

    2012

  25. [33]

    Diffusion actor-critic with entropy regulator

    Yinuo Wang, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. Advances in Neural Information Processing Systems , 37:54183–54204, 2024

    2024

  26. [34]

    Diffusion policies as an expressive policy class for offline reinforcement learning

    Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193 , 2022

    2022 arXiv

  27. [35]

    Versatile diffusion: Text, images and variations all in one diffusion model

    Xingqian Xu, Zhangyang Wang, Gong Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7754–7765, 2023

    2023

  28. [36]

    Diffusion models: A comprehensive survey of methods and applications

    Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1–39, 2023

    2023

  29. [37]

    Policy representation via diffusion probability model for reinforcement learning

    Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305.13122 , 2023

    2023 arXiv

  30. [38]

    Madiff: Offline multi- agent learning with diffusion models

    Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, and Weinan Zhang. Madiff: Offline multi- agent learning with diffusion models. Advances in Neural Information Processing Systems, 37:4177–4206, 2024. APPENDIX SETTINGS OF THE EXPERIMENT Fo...

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.