Pith. sign in

REVIEW 3 major objections 3 minor 36 references

DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning

T0 review · 3 major / 3 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DADiff claims that in reinforcement learning, the performance gap between source and target domains is controlled by how well a target-domain diffusion model denoises source transitions, and uses that residual to modify rewards or filter da

desk verdict DADiff has a genuinely new idea—measuring source-target dynamics mismatch with diffusion latent trajectories—but the estimator in Eq. 10 is not the KL divergence from the paper's own Theorem 4.2, so the theoretical justification needs real repair. read the letter →

arxiv 2607.16090 v1 pith:HJG6FHTA submitted 2026-07-17 cs.LG cs.AI

Hanyang Chen , Anirudh Satheesh , Longchao Da , Hua Wei This is my paper
classification cs.LGcs.AI
keywords diffusionmodelsreinforcementlearningdynamicsmismatchdomainadaptationrewardmodificationdataselectiongenerativetrajectorydeviationsim-to-realtransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that generative trajectory deviation—the difference between source and target diffusion trajectories for the same next-state prediction—is the right quantity for online dynamics adaptation. It proves a theorem bounding the target-vs-source performance difference of any policy by the KL divergence between source and target latent-state transitions, then estimates that divergence with a diffusion model trained on limited target data. If correct, dynamics mismatch can be measured without a domain classifier or learned representation: just ask how well the target model denoises a source transition. The method works in two modes, penalizing rewards or selecting low-deviation data, and experiments across kinematic, morphology, friction, and gravity shifts show it matching or beating prior approaches.

What carries the argument

The central object is the generative trajectory in a diffusion model: the chain of latent states going from pure noise to the next state, conditioned on the current state and action. Theorem 4.2 decomposes the policy performance gap into KL divergences along this chain. The operational machinery is the estimated deviation formula, which computes a per-transition scalar by summing weighted squared residuals between the forward-process noise and the target-domain noise model at every diffusion step. This scalar is what carries the algorithm, since it can be plugged directly into reward modification or data selection.

What would settle it

Run the estimator on two identical domains—same transition dynamics but different random seeds—and compute the deviation for many transitions; any systematically nonzero value shows the residual reflects source-model noise rather than dynamics mismatch, falsifying the claim that the formula measures true generative trajectory deviation.

Watch

Extended reading notes

Core claim

The central discovery is Theorem 4.2: for any policy, the expected-return gap between the source and target domains is bounded by two terms—the deviation of initial latent states and the sum of KL divergences of latent-state transitions between the domains. Because both generative trajectories start from the same initial latent noise, the first term vanishes, so the entire performance gap is controlled by latent transition mismatch. DADiff instantiates this bound with a denoising diffusion probabilistic model: it trains a target-conditioned noise model on target transitions, then for each source transition measures a scalar deviation by the squared residual between the forward-process noise

Load-bearing premise

The proof assumes that for source transitions the reverse-process noise equals the forward-process sample, so the estimated deviation equals the theoretical trajectory deviation; if a trained source model predicts noise differently from the actual forward noise, the estimator bundles source-model error into the dynamics-mismatch measure and the bound no longer justifies the algorithm.

Editorial extensions

If this is right

  • If the bound is correct, policy performance in the target domain can be improved by penalizing source transitions that the target diffusion model denoises poorly.
  • Filtering source data to the lowest-deviation quantile is a valid alternative, and the paper finds it outperforms reward modification in certain tasks where low-reward transitions matter.
  • Measuring mismatch along the full latent trajectory is finer-grained than comparisons at the next-state level alone, which is why the paper argues it beats classifier-based and representation-based approaches.
  • The bound reduces to prior representation-mismatch bounds when the number of latent states is one, so the analysis generalizes that earlier theoretical guarantee.
  • The method needs only limited target-domain interactions—the target noise model is trained on the same small budget as the policy—so it fits the online adaptation setting without extra simulator access.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that the same residual estimator could be built with other conditional generative models, such as flow matching, since only the chain of latent states matters; the paper hints at this connection but does not pursue it.
  • The practical estimator likely mixes source-model irreducibility error into the dynamics-mismatch measure, because it replaces the source reverse-process noise with the forward-process sample; this is my inference, not a claim the paper makes.
  • The data-selection variant suggests a natural curriculum extension: tighten the deviation threshold as more target data accumulates, a schedule the paper does not explore.
  • The bound assumes the two domains share the same initial latent state; if the domains ever used different noise schedules or forward processes, term (a) would not vanish and the estimator could miss part of the gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes DADiff, a diffusion-based framework for online dynamics adaptation in RL. When a policy is trained in a source domain and only limited target-domain interactions are allowed, DADiff estimates the dynamics mismatch by comparing source and target generative trajectories of the next state in a DDPM. Two variants are introduced: DADiff-modify, which penalizes source rewards by the estimated deviation, and DADiff-select, which filters source data by low deviation. The paper presents a theoretical bound (Theorem 4.2) relating the policy performance gap between domains to a generative trajectory deviation, derives an estimator of that deviation (Eq. 10), and reports experiments on MuJoCo tasks with kinematic, morphology, friction, and gravity shifts.

Significance. The idea of using diffusion-model latent trajectories to measure dynamics mismatch is creative and potentially useful; the paper also provides code and a reasonably broad empirical study. If the theoretical bridge between the bound and the implemented estimator were sound, the paper would offer a new principled perspective on off-dynamics RL. However, the central estimator does not actually compute the quantity in the theorem: the substitution of the forward-process noise for the source reverse-process noise is unjustified in stochastic dynamics, and the generative-trajectory construction used in the theory is conflated with the forward-process latents used in the algorithm. The empirical results may still be valuable as a heuristic, but the claimed theoretical justification is not established.

major comments (3)
  1. [Section IV-B, Eqs. (8) and (10)] The step in Section IV-B claiming that for source transitions ε_src(s'_k,s,a,k)=ε is not valid. In DDPM, a trained noise model predicts the conditional expectation E[ε | s'_k,s,a,k], not the particular ε sample used in the forward process. Equation (8) requires ∥ε_src(s'_k,s,a,k) − ε_tar(s'_k,s,a,k)∥², while Eq. (10) uses ∥ε − ε_θ^tar(...)∥² with ε being the forward-process sample. For a well-trained source model, ε_src(s'_k,s,a,k) differs from ε almost surely, with the difference equal to the source model's irreducible denoising error. Thus d(s,a,s') in Eq. (10) is not the KL divergence in Eq. (8) and includes source sampling noise beyond any true dynamics mismatch. In deterministic dynamics with a perfect source model the substitution is approximately valid, but the paper explicitly claims robustness in stochastic environments (Table I, ς>0), where the error is non-negligible. To keep
  2. [Definition 4.1 and Remark after Theorem 4.2] Term (a) is claimed to vanish because the generative trajectories in both domains 'start from the same initial latent state s'_K'. Under the forward-process parameterization actually used in Eq. (10), s'_K = √ᾱ_K s'_0 + √(1−ᾱ_K)ε is a deterministic function of the next state s'_0 and a fresh noise ε. Since P_src(s'_0|s,a) and P_tar(s'_0|s,a) differ, the marginal distributions of s'_K also differ (unless ᾱ_K=0), so DKL(Psrc(s'_K|s,a)∥Ptar(s'_K|s,a)) does not vanish. If instead s'_K is intended to be a common noise prior independent of (s,a), then the s'_k obtained from forward noising of a source tuple in Eq. (10) are not the reverse-process latent states appearing in the theorem. The manuscript needs to state which construction is meant and prove that term (a) is zero under that construction; currently the two constructions are conflated.
  3. [Eq. (10) and Theorem 4.2] Even after correcting the ε_src substitution, Eq. (10) uses the trained target model ε_θ^tar in place of the true target transition kernel P_tar that appears in Theorem 4.2. No finite-sample or model-error bound is given, so the quantity actually computed is not formally shown to bound the performance gap. This is a standard approximation in practice, but the wording 'd(s,a,s') can be practically estimated as follows' obscures the additional approximation error. Please add a formal statement of this approximation or explicitly declare that the algorithm is heuristic without a guarantee.
minor comments (3)
  1. [Appendix VII-B, proof of Theorem 7.2] The first line of the proof writes η_Msrc(π)−η_Mtar(π) = γ/(1−γ) E_ρπsrc[ E_Psrc[r(s,a)] − E_Ptar[r(s,a)] ]. This should involve V_{M_tar}^{π}(s') rather than r(s,a), following Lemma 7.1; the subsequent lines correctly use the value function, so this appears to be a typo.
  2. [Throughout] The method name is written inconsistently as 'DADiff' and 'DADIFF' (e.g., abstract vs. Figure 2). Please unify.
  3. [Table I] The stochastic-dynamics experiment compares only DADiff-modify with PAR. Adding DARC or VGDF would make the claim of robustness under stochastic transitions more convincing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the bound is derived from standard lemmas, and the deviation estimate is not fitted to the target performance gap.

full rationale

The derivation chain is self-contained rather than circular. Theorem 4.2 follows from the SLBO telescoping lemma (Lemma 7.1) and Pinsker's inequality via a standard KL decomposition over the auxiliary latent states; the generative trajectory is defined so that its marginal equals the transition dynamics, so the bound is not a covert restatement of the algorithm's output. The deviation d(s,a,s') in Eq. 8 is exactly the latent-transition KL term appearing in the bound, and Eq. 10 is a practical estimator of that term, not a fitted surrogate for the empirical performance gap. No parameter is fitted to the reported η difference, and the empirical results are measured against external MuJoCo baselines rather than being forced by the method's normalization. The only self-citation ([2], a survey by Da and Wei) appears in the introduction and is not load-bearing for the theoretical or algorithmic claims. The reviewer's concern that Eq. 10 replaces the source reverse-process noise ε_src with the forward-sample ε is a validity gap in the estimator—arguably a correctness issue—but it does not make the derivation equivalent to its inputs by construction. The paper's central claim is therefore not circular, though the tightness of the connection between Eq. 10 and Eq. 8 may warrant further scrutiny.

Assumptions & free parameters 3 free parameters · 6 assumptions · 1 invented entities

The central claim rests on the construction of a K-step generative trajectory with a shared initial latent state, the Gaussian reverse-process model of a conditional DDPM, and the unproven identification of the source noise model with the forward noise sample. The free hyperparameters λ, ξ, and K are tuned per task in the parameter study, so the empirical results include fitted constants. No new physical entities are introduced; the 'generative trajectory' is a modeling construct without independent evidence.

free parameters (3)
  • λ (reward penalty coefficient) = not reported; parameter study uses 0.01–5.0
    Introduced in Eq. 11 to balance original reward and deviation penalty; Section V-C shows optimal value is task-dependent, so it is a hand-tuned/fitted constant for each environment.
  • ξ% (data selection quantile) = not reported; parameter study uses 0–100%
    Introduced in Eq. 13 to decide which fraction of source transitions to keep; Section V-C shows optimal value is task-dependent.
  • K (number of diffusion timesteps) = parameter study uses 10–200; K=100 recommended
    Number of latent states in the generative trajectory; chosen by hand, not derived from the theory.
assumptions (6)
  • standard math SLBO performance-difference lemma (Lemma 7.1) for MDPs with shared rewards
    Used as the starting point for the proof of Theorem 7.2 in Appendix VII-B.
  • standard math Pinsker's inequality and subadditivity of the square root function
    Used in steps (a) and (c) of the proof of Theorem 7.2 to pass from TV distance to KL divergence.
  • domain assumption Source and target MDPs share state/action spaces and reward function, with only transition dynamics differing; rewards are bounded by rmax
    Problem setting in Section III-A; required for the performance-difference bound.
  • ad hoc to paper Each domain's next-state distribution can be represented as a K-step latent Markov chain with a common initial latent state s'_K in both domains
    Definition 4.1 and the remark in Section IV-A; this construction makes term (a) of Theorem 4.2 vanish by assumption.
  • domain assumption The DDPM reverse process conditioned on (s,a) gives Gaussian latent-state transitions that faithfully model each domain's dynamics (Eq. 7)
    Section IV-B uses Eq. 6-7 to write the latent transition as a Gaussian and compute the KL in closed form.
  • ad hoc to paper For source transitions, the source reverse-process noise equals the forward-process sample ε
    Eq. 8 to Eq. 10 in Section IV-B; this is the load-bearing practical simplification, and it is not proven.
invented entities (1)
  • Generative trajectory of latent states {s'_k}_{k=1}^K
    purpose: Decomposes the next-state transition into a K-step Markov chain so that dynamics mismatch can be measured along the whole denoising path
    Introduced in Definition 4.1 as a modeling construct; it has no independent falsifiable handle outside the algorithm.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning." pith.science (2026). https://pith.science/paper/HJG6FHTA

@misc{pith2026260716090,
  author       = {Pith},
  title        = {Pith review of: DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HJG6FHTA}},
  note         = {Machine review of arXiv:2607.16090}
}
read the original abstract

Transferring policies across domains poses a vital challenge in reinforcement learning, due to the dynamics mismatch between the source and target domains. In this paper, we consider the setting of online dynamics adaptation, where policies are trained in the source domain with sufficient data, while only limited interactions with the target domain are allowed. There are a few existing works that address the dynamics mismatch by employing domain classifiers, value-guided data filtering, or representation learning. Instead, we study the domain adaptation problem from a generative modeling perspective. Specifically, we introduce DADiff, a diffusion-based framework that leverages the discrepancy between source and target domain generative trajectories in the generation process of the next state to estimate the dynamics mismatch. Both reward modification and data selection variants are developed to adapt the policy to the target domain. We also provide a theoretical analysis to show that the performance difference of a given policy between the two domains is bounded by the generative trajectory deviation. More discussions on the applicability of the variants and the connection between our theoretical analysis and the prior work are further provided. We conduct extensive experiments in environments with various shifts to validate the effectiveness of our method. The results demonstrate that our method provides superior performance compared to existing approaches, effectively addressing the dynamics mismatch. We provide the code of our method at https://github.com/hanyang-chen/DADiff-release

Figures

Figures reproduced from arXiv: 2607.16090 by the authors.

Figure 1
Figure 1. Illustration of DADIFF. This figure visualizes the generative trajecto￾ries in the source and target domains. The deviation d(s, a, s′ ) is measured by the discrepancy dk of each latent state s ′ k in the source and target domain generative trajectories. Remark. The Markov-chain definition enables the transition dynamics to be decomposed into multiple conditional probabilities, i.e., PM(s ′ 0 |s, a) = R PM(s ′ K|s, … view at source ↗
Figure 2
Figure 2. Adaptation performance under kinematic, morphology, friction, and gravity shifts (from top to bottom). The solid curves and the shaded regions [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. GPU memory and runtime comparisons on the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Parameter study. The solid curves and the shaded regions denote the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Reward distribution comparison between the source-domain rewards [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 11 linked inside Pith

  1. [1]

    Sim-to-real transfer in deep reinforcement learning for robotics: a survey,

    W. Zhao, J. P. Queralta, and T. Westerlund, “Sim-to-real transfer in deep reinforcement learning for robotics: a survey,” in2020 IEEE symposium series on computational intelligence (SSCI). IEEE, 2020, pp. 737–744

    2020

  2. [2]

    A survey of sim-to-real methods in rl: Progress, prospects and challenges with foundation models,

    L. Da, J. Turnau, T. P. Kutralingam, A. Velasquez, P. Shakarian, and H. Wei, “A survey of sim-to-real methods in rl: Progress, prospects and challenges with foundation models,”arXiv preprint arXiv:2502.13187, 2025

    arXiv 2025

  3. [3]

    Cross-domain policy adaptation via value-guided data filtering,

    K. Xu, C. Bai, X. Ma, D. Wang, B. Zhao, Z. Wang, X. Li, and W. Li, “Cross-domain policy adaptation via value-guided data filtering,”Ad- vances in Neural Information Processing Systems, vol. 36, pp. 73 395– 73 421, 2023

    2023

  4. [4]

    Cross-domain policy adaptation by capturing representation mismatch,

    J. Lyu, C. Bai, J. Yang, Z. Lu, and X. Li, “Cross-domain policy adaptation by capturing representation mismatch,”arXiv preprint arXiv:2405.15369, 2024

    arXiv 2024

  5. [5]

    Sim-to- real transfer of robotic control with dynamics randomization,

    X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to- real transfer of robotic control with dynamics randomization,” in2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018, pp. 3803–3810

    2018

  6. [6]

    Active domain randomization,

    B. Mehta, M. Diaz, F. Golemo, C. J. Pal, and L. Paull, “Active domain randomization,” inConference on Robot Learning. PMLR, 2020, pp. 1162–1176

    2020

  7. [7]

    Flow-based domain randomization for learning and sequencing robotic skills,

    A. Curtis, E. Li, M. Noseworthy, N. Gothoskar, S. Chitta, H. Li, L. P. Kaelbling, and N. E. Carey, “Flow-based domain randomization for learning and sequencing robotic skills,” inForty-second International Conference on Machine Learning, 2025

    2025

  8. [8]

    Closing the sim-to-real loop: Adapting simula- tion randomization with real world experience,

    Y . Chebotar, A. Handa, V . Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox, “Closing the sim-to-real loop: Adapting simula- tion randomization with real world experience,” in2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 8973–8979

    2019

Show all 36 references
  1. [9]

    Off-dynamics reinforcement learning: Training for transfer with domain classifiers,

    B. Eysenbach, S. Asawa, S. Chaudhari, S. Levine, and R. Salakhutdi- nov, “Off-dynamics reinforcement learning: Training for transfer with domain classifiers,”arXiv preprint arXiv:2006.13916, 2020

    2006 arXiv

  2. [10]

    Cross-domain policy adaptation by capturing representation mismatch,

    J. Lyu, C. Bai, J. Yang, Z. Lu, and X. Li, “Cross-domain policy adaptation by capturing representation mismatch,” inProceedings of the 41st International Conference on Machine Learning, 2024, pp. 33 638–33 663

    2024

  3. [11]

    Deep unsupervised learning using nonequilibrium thermodynamics,

    J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” inInternational conference on machine learning. pmlr, 2015, pp. 2256–2265

    2015

  4. [12]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,”Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020

    2020

  5. [13]

    State regularized policy optimization on data with dynamics shift,

    Z. Xue, Q. Cai, S. Liu, D. Zheng, P. Jiang, K. Gai, and B. An, “State regularized policy optimization on data with dynamics shift,”Advances in neural information processing systems, vol. 36, pp. 32 926–32 937, 2023

    2023

  6. [14]

    Policy adaptation from foundation model feedback,

    Y . Ge, A. Macaluso, L. E. Li, P. Luo, and X. Wang, “Policy adaptation from foundation model feedback,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 19 059–19 069

    2023

  7. [15]

    Cross- domain reinforcement learning under distinct state-action spaces via hybrid q functions

    K.-C. Pan, M. Chen, Y .-D. Huang, X. Liu, and P.-C. Hsieh, “Cross- domain reinforcement learning under distinct state-action spaces via hybrid q functions.”

  8. [16]

    Robust vi- sual domain randomization for reinforcement learning,

    R. B. Slaoui, W. R. Clements, J. N. Foerster, and S. Toth, “Robust vi- sual domain randomization for reinforcement learning,”arXiv preprint arXiv:1910.10537, 2019

    1910 arXiv

  9. [17]

    Variance reduced domain randomization for reinforcement learning with policy gradi- ent,

    Y . Jiang, C. Li, W. Dai, J. Zou, and H. Xiong, “Variance reduced domain randomization for reinforcement learning with policy gradi- ent,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 2, pp. 1031–1048, 2023

    2023

  10. [18]

    Learning to adapt in dynamic, real-world environments through meta-reinforcement learning,

    A. Nagabandi, I. Clavera, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn, “Learning to adapt in dynamic, real-world environments through meta-reinforcement learning,”arXiv preprint arXiv:1803.11347, 2018

    2018 arXiv

  11. [19]

    Zero-shot policy transfer with disentangled task representation of meta-reinforcement learning,

    Z. Wu, Y . Xie, W. Lian, C. Wang, Y . Guo, J. Chen, S. Schaal, and M. Tomizuka, “Zero-shot policy transfer with disentangled task representation of meta-reinforcement learning,”arXiv preprint arXiv:2210.00350, 2022

    2022 arXiv

  12. [20]

    Cross-domain imitation from observations,

    D. S. Raychaudhuri, S. Paul, J. Vanbaar, and A. K. Roy-Chowdhury, “Cross-domain imitation from observations,” inInternational confer- ence on machine learning. PMLR, 2021, pp. 8902–8912

    2021

  13. [21]

    Cross- domain imitation learning via optimal transport,

    A. Fickinger, S. Cohen, S. Russell, and B. Amos, “Cross- domain imitation learning via optimal transport,”arXiv preprint arXiv:2110.03684, 2021

    2021

  14. [22]

    Off-dynamics reinforce- ment learning via domain adaptation and reward augmented imitation,

    Y . Guo, Y . Wang, Y . Shi, P. Xu, and A. Liu, “Off-dynamics reinforce- ment learning via domain adaptation and reward augmented imitation,” Advances in Neural Information Processing Systems, vol. 37, pp. 136 326–136 360, 2024

    2024

  15. [23]

    Policy learning for off- dynamics rl with deficient support,

    L. L. P. Van, H. T. Tran, and S. Gupta, “Policy learning for off- dynamics rl with deficient support,”arXiv preprint arXiv:2402.10765, 2024

    2024 arXiv

  16. [24]

    Contrastive representation for data filtering in cross-domain offline reinforcement learning,

    X. Wen, C. Bai, K. Xu, X. Yu, Y . Zhang, X. Li, and Z. Wang, “Contrastive representation for data filtering in cross-domain offline reinforcement learning,”arXiv preprint arXiv:2405.06192, 2024

    2024 arXiv

  17. [25]

    Efficient diffusion policies for offline reinforcement learning,

    B. Kang, X. Ma, C. Du, T. Pang, and S. Yan, “Efficient diffusion policies for offline reinforcement learning,”Advances in Neural Infor- mation Processing Systems, vol. 36, pp. 67 195–67 212, 2023

    2023

  18. [26]

    Diffusion policy: Visuomotor policy learning via action diffusion,

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,”The International Journal of Robotics Research, vol. 44, no. 10-11, pp. 1684–1704, 2025

    2025

  19. [27]

    Synthetic experience replay,

    C. Lu, P. Ball, Y . W. Teh, and J. Parker-Holder, “Synthetic experience replay,”Advances in Neural Information Processing Systems, vol. 36, pp. 46 323–46 344, 2023

    2023

  20. [28]

    Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning,

    H. He, C. Bai, K. Xu, Z. Yang, W. Zhang, D. Wang, B. Zhao, and X. Li, “Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning,”Advances in neural information processing systems, vol. 36, pp. 64 896–64 917, 2023

    2023

  21. [29]

    Diffusion actor-critic with entropy reg- ulator,

    Y . Wang, L. Wang, Y . Jiang, W. Zou, T. Liu, X. Song, W. Wang, L. Xiao, J. Wu, J. Duan,et al., “Diffusion actor-critic with entropy reg- ulator,”Advances in Neural Information Processing Systems, vol. 37, pp. 54 183–54 204, 2024

    2024

  22. [30]

    Madiff: Offline multi-agent learning with diffusion models,

    Z. Zhu, M. Liu, L. Mao, B. Kang, M. Xu, Y . Yu, S. Ermon, and W. Zhang, “Madiff: Offline multi-agent learning with diffusion models,”Advances in Neural Information Processing Systems, vol. 37, pp. 4177–4206, 2024

    2024

  23. [31]

    Dmc: Nearest neighbor guidance diffusion model for offline cross- domain reinforcement learning,

    L. L. P. Van, M. H. Nguyen, D. Kieu, H. Le, H. T. Tran, and S. Gupta, “Dmc: Nearest neighbor guidance diffusion model for offline cross- domain reinforcement learning,”arXiv preprint arXiv:2507.20499, 2025

    2025

  24. [32]

    Soft actor-critic algorithms and applications,

    T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V . Kumar, H. Zhu, A. Gupta, P. Abbeel,et al., “Soft actor-critic algorithms and applications,”arXiv preprint arXiv:1812.05905, 2018

    2018 arXiv

  25. [33]

    Mujoco: A physics engine for model-based control,

    E. Todorov, T. Erez, and Y . Tassa, “Mujoco: A physics engine for model-based control,” in2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012, pp. 5026–5033

    2012

  26. [34]

    Openai gym,

    G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schul- man, J. Tang, and W. Zaremba, “Openai gym,”arXiv preprint arXiv:1606.01540, 2016

    2016 arXiv

  27. [35]

    Odrl: A benchmark for off-dynamics reinforcement learning,

    J. Lyu, K. Xu, J. Xu, J.-W. Yang, Z. Zhang, C. Bai, Z. Lu, X. Li, et al., “Odrl: A benchmark for off-dynamics reinforcement learning,” Advances in Neural Information Processing Systems, vol. 37, pp. 59 859–59 911, 2024

    2024

  28. [36]

    Algorithmic framework for model-based deep reinforcement learning with theoret- ical guarantees,

    Y . Luo, H. Xu, Y . Li, Y . Tian, T. Darrell, and T. Ma, “Algorithmic framework for model-based deep reinforcement learning with theoret- ical guarantees,”arXiv preprint arXiv:1807.03858, 2018

    2018 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.