Model-Based Diffusion Sampling for Predictive Control in Offline Decision Making

Haldun Balim; Na Li; Yilun Du

REVIEW 4 major objections 3 minor 1 cited by

MPDiffuser claims that alternating updates from a task-conditioned diffusion planner and a diffusion dynamics model during sampling produce trajectories that are both task-aligned and dynamically feasible, improving offline decision-making

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 17:42 UTC pith:O5G5XF4U

load-bearing objection A genuinely useful empirical method — interleaving planner and dynamics diffusion models in the reverse process — with an honest but informal theory section; claims need softening, but this deserves a serious referee. the 4 major comments →

arxiv 2512.08280 v3 pith:O5G5XF4U submitted 2025-12-09 cs.RO cs.AIcs.SYeess.SY

Model-Based Diffusion Sampling for Predictive Control in Offline Decision Making

Haldun Balim , Na Li , Yilun Du This is my paper

classification cs.RO cs.AIcs.SYeess.SY

keywords offline reinforcement learningdiffusion modelsmodel predictive controltrajectory planningdynamics feasibilityconstrained decision makingsafe controlquadruped locomotion

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

MPDiffuser claims that the trajectory feasibility gap in diffusion-based offline control can be closed by composing a task-conditioned diffusion planner with a separate diffusion dynamics model and interleaving their updates at every denoising step. The alternating sampler is presented as an approximation of the exponential-tilted distribution q*(τ) ∝ p_planner(τ) p_dynamics(τ)^λ, so generated rollouts preserve task intent while remaining consistent with the system's transition kernel. A lightweight ranker then selects the trajectory that best satisfies reward and cost objectives. The method reports consistent gains over prior diffusion-based planners on D4RL and DSRL benchmarks, exploits random and suboptimal data for dynamics training alone, adapts to changed dynamics by fine-tuning only the dynamics module, and runs onboard a real quadruped. If right, this gives a practical recipe for turning trajectory diffusion into closed-loop, constraint-aware control.

Core claim

On the paper's own terms, the central claim is that MPDiffuser's alternating planner–dynamics diffusion sampling yields trajectories that are both task-aligned and dynamically feasible, improving normalized returns and safety over prior diffusion-based offline decision-making methods and enabling real quadruped deployment. The authors summarize the mechanism as interleaving planner and dynamics updates during sampling, progressively correcting feasibility while preserving task intent. The dynamics model is a conditional diffusion model over states only, given the action sequence, which lets it be trained on diverse or suboptimal data without affecting planner quality; the ranker then filters

What carries the argument

The load-bearing mechanism is the alternating reverse-sampling loop (Algorithm 1): at each diffusion noise level k, a dynamics step denoises the state trajectory conditioned on the current action proposal, then a planner step jointly denoises states and actions with classifier-free guidance. The intended target is the product-of-experts distribution q*(τ|x0) ∝ p_pl(τ|x0) p_dyn(τ|x0)^λ — an exponential tilting of the planner distribution by the dynamics likelihood — whose exact score is intractable; the alternation is justified as a first-order operator-splitting approximation of the combined score flow, and Appendix N explicitly says this is a rationale, not a formal guarantee.

Load-bearing premise

The learned diffusion dynamics model supplies an accurate local score approximation to the true transition kernel at every diffusion noise level, and the alternating updates steer samples toward the product distribution q*; Appendix N states explicitly that no formal theoretical guarantee is provided, and Appendix L shows that under planner–dynamics distribution mismatch the dynamics model can over-correct and degrade performance.

What would settle it

Use a system with a known transition kernel (e.g., the paper's double-integrator), train planner and dynamics models, and compare alternating-sampler trajectories against exact samples from q* ∝ p_pl p_dyn^λ obtained via importance weighting; a large divergence in return or dynamics error would falsify the operator-splitting approximation. Simpler and directly reproducible: repeat the paper's medium-expert-planner/medium-replay-dynamics mismatch while sweeping the guidance scale and diffusion steps; if tuning eliminates or monotonically worsens the measured performance gap, the claim that dyna

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Trajectory diffusion can be made dynamically feasible without retraining the planner: letting the dynamics model train on additional random or suboptimal trajectories raises FetchPickAndPlace success from 0.75 to 0.86, where the planner sees only expert demonstrations.
Adaptation to changed system dynamics is cheap: after a simulated hardware defect, fine-tuning only the dynamics diffusion model on 100 play episodes recovers and exceeds the original performance, while a planner-only baseline forgets earlier behavior.
Safety in constrained offline decision making depends on feasible candidate rollouts: planning without dynamics consistency degrades as the number of ranking samples grows (because high-return 'hallucinated' trajectories get selected), whereas MPDiffuser's success rate stays high.
The compositional design scales to visual observations by running diffusion in a learned latent space with an auxiliary latent dynamics predictor, improving average return over prior diffusion baselines in a proof-of-concept.
Real-time on-robot control is feasible with single-sample DDIM inference, action chunking, asynchronous planning, and warm-starting; on the Unitree Go2, MPDiffuser tracks a 0.5 m/s velocity command more closely than the dataset policy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Editorial inference: Appendix L's mismatched-dataset failure suggests the alternating sampler can serve as a distribution-shift detector — when the planner's proposals leave the dynamics model's training support, the dynamics corrections degrade, turning degradation into a signal for retraining or fallback.
Editorial inference: because the target distribution is an exponential tilt q* ∝ p_pl p_dyn^λ, the ranker's reward/cost scaling and the dynamics exponent λ are likely coupled knobs; tuning λ jointly with guidance scale could provide a principled dial for the safety–reward tradeoff that the paper currently adjusts post-hoc.
Editorial inference: the 2x-compute ablation (planner with 200 steps vs 100 alternating steps) separates the mechanism from raw denoising budget, but a sharper test would hold total network evaluations fixed and vary only the interleaving frequency; if alternation, not budget, drives the gains, the operator-splitting account is supported.
Editorial inference: since the dynamics model is trained on states only, it is essentially a reusable world model that could be shared across planners trained on different tasks or reward functions in the same environment — a transfer-learning extension the paper does not pursue.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A genuinely useful empirical method — interleaving planner and dynamics diffusion models in the reverse process — with an honest but informal theory section; claims need softening, but this deserves a serious referee.

read the letter

Short version: the core idea is new, the empirical package is unusually broad, and the theory appendix is honest about what it does not prove. I would send this to review.

What is actually new: MPDiffuser does not use the dynamics model as a post-hoc filter (D-MPC) or as an optimization projection (Romer et al.). It alternates planner and dynamics denoising steps inside the reverse chain, so the dynamics model actively shapes every sample. That is a real algorithmic difference, and it shows in the single-sample regime where D-MPC’s dynamics model is inert. The paper also gets credit for broad evaluation: D4RL, DSRL, a clean Car U-Maze feasibility test, a visual pilot, and a real Unitree Go2 deployment with onboard compute. The ablations are genuinely informative — especially Appendix H (alternating vs. simply doubling planner steps), Appendix I (alternating beats combined score updates), and Appendix L (a controlled distribution-mismatch failure). Code is promised.

Soft spots, in order of size. First, the theory is motivation, not proof. Appendix N explicitly says there is no formal guarantee. Equation (15) approximates the score of the tilted distribution as a sum of individual scores, which is exact only at zero noise, and the Lie–Trotter analogy does not give finite-step convergence with learned approximate scores. That is fine if presented as intuition, but the paper should not lean so hard on it. Second, the "consistent improvements" wording is contradicted by Table 1: Decision Diffuser beats MPDiffuser on Hopper Med-Exp and HalfCheetah Medium, and some MPDiffuser cells show 0.0 standard deviation across 5 runs while baseline columns have no error bars at all. The empirical claim should be toned to "generally improves." Third, Appendix L is a real sensitivity: with the planner on medium-expert and dynamics on medium-replay, Hopper drops to 70.3 vs. 98.2 for matching data. The authors are honest about it, but this kind of failure mode belongs in the main text limitations, not only in an appendix. None of this kills the paper; the main experiments all use matching data, and the failure mode is plausibly manageable.

Who this is for: anyone working on diffusion-based planning, offline RL with dynamics awareness, or safe control from offline data. The method is practical and worth building on. It deserves a serious referee. I would recommend conditional acceptance with three asks: soften the consistency claim, provide baseline error bars and fix the too-clean std entries, and move the core limitation and the informal-theory status into the main text.

Referee Report

4 major / 3 minor

Summary. The paper introduces MPDiffuser, a compositional generative framework for offline decision making. It trains a conditional diffusion planner over state-action trajectories and a conditional diffusion dynamics model over states given actions, then interleaves their denoising updates during reverse-time sampling. A ranker selects the candidate trajectory that best satisfies task reward and cost criteria. The central claim is that alternating planner/dynamics updates preserve task alignment while improving dynamic feasibility, giving better normalized returns and safety on D4RL and DSRL benchmarks, and enabling real quadruped deployment. The appendix presents a theoretical rationale based on exponential tilting and operator splitting, along with extensive ablations and a real-robot experiment.

Significance. If the empirical results hold, the modular composition is a useful design pattern: dynamics consistency can be injected into diffusion planning without retraining the planner, and the dynamics model can exploit additional random or suboptimal data. Strengths include the broad evaluation across D4RL, DSRL, visual domains, and a real Unitree Go2 deployment; the released code; and systematic ablations of architecture choices, conditioning, and distribution mismatch. However, the theoretical support is explicitly informal, and the headline claim of 'consistent improvements' over prior diffusion methods is not fully supported by the reported tables. The contribution is therefore primarily empirical and heuristic, and the presentation needs to be aligned with that status.

major comments (4)

[Table 1 / Abstract] The abstract claims 'consistent improvements over prior diffusion-based methods,' but Table 1 shows Decision Diffuser outperforms MPDiffuser on Hopper Med-Exp (111.8 vs 110.4) and HalfCheetah Medium (49.1 vs 47.9), and MPDiffuser+Rank also loses to D-MPC or Decision Diffuser on several other entries. The word 'consistent' is therefore not supported by the paper's own results. Please revise the claim (e.g., 'generally competitive' or 'in the majority of tasks') or provide paired significance tests across seeds. This is load-bearing because the abstract and introduction repeat this phrase.
[Appendix N, Eq. (15)] The theoretical rationale is the only support for the alternating sampler, yet the text states 'we do not provide a formal theoretical guarantee.' Equation (15) approximates the score of the tilted target q* by a sum of individual scores, and as the text notes this is exact only at zero noise. The Lie–Trotter operator-splitting analogy does not constitute a convergence proof for the finite reverse process with K=100 and different variable updates for planner and dynamics. Since alternating sampling is the core contribution, this gap is load-bearing: either give a rigorous statement under idealized assumptions or explicitly label the method as a heuristic and adjust the abstract/intro framing accordingly.
[Tables 1 and 4 / experimental reporting] Baseline columns are reported as point estimates without standard deviations, while MPDiffuser columns include errors, several of which are exactly 0.0 across 5 runs. Without baseline error bars, claims of improvement are not statistically substantiated; zero variance on continuous D4RL tasks is also surprising and should be explained (e.g., deterministic evaluation, fixed seeds, or rounding). Please report error bars / multiple seeds for all methods or clearly state which numbers are taken from prior papers and which are re-run.
[Appendix L and Sec. 4.1] The 'Mixed' experiment shows that when the planner is trained on medium-expert data and the dynamics model on medium-replay, Hopper performance drops from 98.2 (matching data) to 70.3, worse than the planner-only baseline. This is a concrete failure of the 'preserving task intent' claim under distribution shift. The main text should mention this limitation and qualify the abstract's statement that MPDiffuser 'preserves task intent' while correcting feasibility; as written, the failure mode appears only in the appendix and the conclusion states unqualified consistency with system dynamics.

minor comments (3)

[Figure 1] The right-hand panel's labels (e.g., 'τK x,τK u', 'Planner Dyn. Model') are difficult to parse. Please use a cleaner diagram with explicit arrows for the alternating updates.
[Section 2.2] The forward process equations have formatting issues, e.g., 'qk|0 = N(τ k; √¯αk,(1−¯α k)I' is missing a closing parenthesis and has inconsistent spacing. Please proofread the display math.
[Table 16] The column headers 'Med-Rep' and 'Med-Exp' are ambiguous; 'Med-Rep' should be 'Med-Replay' and 'Med-Exp' should be 'Med-Expert' for consistency with the rest of the paper.

Circularity Check

0 steps flagged

No significant circularity: the paper's empirical claims are evaluated on external benchmarks and true-environment rollouts, while the theoretical rationale is explicitly non-formal and does not disguise fitted quantities as predictions.

full rationale

The derivation chain is not circular. The planner and dynamics diffusion models are trained with independent denoising-score losses on the offline dataset (Sec. 3.1), and their composition is evaluated by external D4RL/DSRL benchmarks, by open-loop simulation under the true environment (Fig. 4, Apps. C/E), and by real-robot deployment. Feasibility is therefore measured against the environment, not against the models' own outputs. The ranker uses learned reward/cost models, but the final scores are computed by the environment or by true simulation, so no fitted value is renamed as a prediction. The theoretical target q* in Sec. 3.4/App. N is explicitly a post-hoc rationale: the paper states 'we do not provide a formal theoretical guarantee' and 'we present this analysis as a theoretical motivation rather than a formal statement.' Equation (15) is an acknowledged approximation and the operator-splitting analogy is heuristic, not a derivation that reduces to its inputs. The highlighted failure mode in App. L (distribution mismatch causing a large performance drop) is an honest limitation and further confirms that the empirical claims are not forced by construction. Self-citations to Diffuser/Decision Diffuser are used as baselines or background, not as load-bearing justifications of the central mechanism.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The paper introduces no new physical or ontological entities; the dynamics diffusion model and ranker are algorithmic components. The main free parameters are task-tuned hyperparameters, and the theoretical target distribution is heuristic rather than a derived law.

free parameters (4)

guidance scale ω = tuned per task (typical 1.5–2.5; App. M)
Controls strength of task conditioning in classifier-free guidance; App. M shows sensitivity on Fetch.
return scale and cost scale = tuned per task (App. B)
Used to condition planner/dynamics and to score ranker candidates; changing them shifts safety–reward tradeoff.
number of ranking samples N = 64 (D4RL), 16 (DSRL), varied 1–32 in Fetch (App. M)
Ranker search width; performance saturates around 8–16 samples.
planning horizon H and diffusion steps K = H=16–64 per environment; K=10–100
Chosen per task; affects trajectory length, compute, and feasibility.

axioms (6)

standard math Diffusion models trained with denoising score matching approximate the true data score distribution.
Invoked in Sec. 2.2 when defining both planner and dynamics models.
domain assumption The offline dataset D is sampled from the true system and adequately covers transitions needed for the task.
Required for both planner and dynamics model to learn meaningful behavior; standard offline RL assumption.
domain assumption The dynamics model trained on the same dataset generalizes to action sequences proposed by the planner.
Appendix L shows this can fail when planner and dynamics are trained on different data distributions (Mixed settings degrade).
domain assumption Learned reward and cost models used by the ranker are accurate enough for trajectory selection.
Ranker in Sec. 4.1/4.2 selects trajectories using learned MLP reward/cost models; no error analysis of these models is provided.
ad hoc to paper Alternating planner and dynamics diffusion updates approximates sampling from q* ∝ p_pl p_dyn^λ.
Appendix N explicitly states 'we do not provide a formal theoretical guarantee' and motivates the approach only via operator-splitting analogy.
domain assumption The transition kernel can be relaxed to a Gaussian density for defining dynamics probability pdyn.
Appendix N introduces a Gaussian relaxation of the deterministic dynamics to avoid Dirac-measure brittleness.

pith-pipeline@v1.3.0-alltime-deepseek · 24055 in / 11986 out tokens · 122458 ms · 2026-08-03T17:42:34.454286+00:00 · methodology

0 comments

read the original abstract

Offline decision-making via diffusion models often produces trajectories that are misaligned with system dynamics, limiting their reliability for control. We propose Model Predictive Diffuser (MPDiffuser), a compositional diffusion framework that combines a diffusion planner with a dynamics diffusion model to generate task-aligned and dynamically plausible trajectories. MPDiffuser interleaves planner and dynamics updates during sampling, progressively correcting feasibility while preserving task intent. A lightweight ranking module then selects trajectories that best satisfy task objectives. The compositional design improves sample efficiency and adaptability by enabling the dynamics model to leverage diverse and previously unseen data independently of the planner. Empirically, we demonstrate consistent improvements over prior diffusion-based methods on unconstrained (D4RL) and constrained (DSRL) benchmarks, and validate practicality through deployment on a real quadrupedal robot.

Figures

Figures reproduced from arXiv: 2512.08280 by Haldun Balim, Na Li, Yilun Du.

**Figure 1.** Figure 1: Framework Overview. Left: MPDiffuser, which couples a diffusion planner with a diffusion dynamics model, complemented by a ranking module. Right: Comparison highlighting key differences between our method and prior trajectory-level diffusion methods. a theoretical interpretation as approximating a distribution combining planner priors with dynamics consistency. In contrast to prior diffusion methods that r… view at source ↗

**Figure 2.** Figure 2: Illustrative scenario: We compare sampled state trajectories with open-loop simulations obtained by executing the sampled actions on a simple car model (cf. App. C). Diffuser produces infeasible trajectories, and Decision Diffuser yields plausible states whose actions diverge when executed. In contrast, MPDiffuser generates trajectories that are faithful to system dynamics. 3. Method Below, we introduce… view at source ↗

**Figure 3.** Figure 3: Fetch PickandPlace Num. rand. traj. MPD D-MPC 0 0.75 0.60 2000 0.81 0.68 4000 0.79 0.56 6000 0.78 0.53 8000 0.82 0.55 10000 0.86 0.60 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** Figure 4: Dynamics consistency of sampled trajectories. Mean state error over the prediction horizon for block position and all state dimensions [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

**Figure 5.** Figure 5: Walker2D illustration, higlights defective joint Original Pre-FT Post-FT Diffuser 79.6 25.9 6.8 D-MPC 76.2 22.7 30.7 Planner 75.9 58.6 56.0 MPD 77.6 58.6 66.4 MPD+Rank 77.6 51.0 63.4 [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

**Figure 6.** Figure 6: Pendulum environment. Num. Samples 1 4 8 16 32 64 Planner 62 89 91 88 74 66 SafeDiffuser 49 62 47 42 46 45 MPDiffuser 69 84 93 93 92 91 [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

**Figure 8.** Figure 8: Real-world demo. Estimated velocity from the Unitree Go2 deployment. compact latent representation of these stacked frames with latent dimension 32. In addition to the standard reconstruction loss, we introduce a latent-space dynamics loss by training an auxiliary dynamics predictor that maps the current latent and action to the next latent. This encourages the learned representation to better reflect th… view at source ↗

**Figure 9.** Figure 9: Datasets for Deep Data-Driven Reinforcement Learning (D4RL) (Fu et al., 2021). (a) Hopper-v4 (b) HalfCheetah-v4 (c) Walker2d-v4 (d) SafetyGymPoint (e) SafetyGymCar [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗

**Figure 10.** Figure 10: Datasets for Safe Reinforcement Learning (DSRL) (Liu et al., 2024) In this section, we outline the key architectural and hyperparameter choices: • Both the planner noise model ϵ pl θ and the dynamics noise model ϵ dyn θ are implemented as temporal U-Nets as proposed by Janner et al. (2022). Each network consists of six repeated residual blocks, where each block contains two temporal 13 [PITH_FULL_IMAGE:f… view at source ↗

**Figure 11.** Figure 11: Custom datasets generated for this work. convolutions, followed by group normalization Wu & He (2018) and a Swish activation Ramachandran et al. (2017). Conditioning inputs y(τ ) and the initial state x0 are first processed with a two-layer MLP and then injected into the U-Net through FiLM layers Perez et al. (2018). • We optimize ϵθ and fϕ using Adam (Kingma, 2014) with a learning rate of 2 × 10−4 , a ba… view at source ↗

**Figure 12.** Figure 12: MPDiffuser more closely aligns with expert behavior. Average state error relative to the expert trajectory for p = 0.8. The top panel compares generated (predicted) states from each method to the states obtained by expert. The bottom panel compares states obtained by simulating the system with the generated actions. The proposed method achieves the lowest error in both cases, highlighting the benefit of d… view at source ↗

**Figure 13.** Figure 13: Performance vs. planning time: Trade-off between performance (normalized average return), and planning cost, measured in wall-clock time after warm-starting the reverse diffusion process. The results are obtained using a single NVIDIA RTX 4090 GPU G. Computation Budget, Replanning Experiment We analyze the runtime characteristics of our compositional diffusion procedure in the D4RL hopper-medium-expert-v2… view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Expressivity and Statistical Trade-offs in Diffusion Policy Learning
stat.ML 2026-07 accept novelty 7.0

Drift Lipschitz budget K yields 1/K value approximation for diffusion policies, with matching lower bound, and finite-sample rates Õ(n^{-2/(m+6)}) (generic) or Õ(n^{-2/(m+4)}) (dissipative).

Reference graph

Works this paper leans on

26 extracted references · 11 linked inside Pith · cited by 1 Pith paper

[1]

Naively, each environment step requires running a full reverse diffusion chain, which can be computationally expensive. To accelerate planning, we adopt a warm-start strategy: the generated trajectory from the previous step is partially diffused forward for a fixed number of steps, after which the same number of reverse diffusion steps are applied to obta...

2022
[2]

In Section E we consider a linear system with a stochastic expert, providing a controlled setting where feasibility can be studied in detail

In Section D we compare two schemes for incorporating the initial state—inpainting versus FiLM-based conditioning—through an ablation on D4RL Hopper. In Section E we consider a linear system with a stochastic expert, providing a controlled setting where feasibility can be studied in detail. In Section F, we evaluate the performance of MPDiffuser under mod...

2023
[10]

A., Du, Y ., and Xu, D

Luo, Y ., Mishra, U. A., Du, Y ., and Xu, D. Genera- tive trajectory stitching through diffusion composition. InICRA 2025 Workshop on Foundation Models and Neuro-Symbolic AI for Robotics,

2025
[11]

Pearce, T., Rashid, T., Kanervisto, A., Bignell, D., Sun, M., Georgescu, R., Macua, S

doi: 10.1109/LRA.2023.3270034. Pearce, T., Rashid, T., Kanervisto, A., Bignell, D., Sun, M., Georgescu, R., Macua, S. V ., Tan, S. Z., Momen- nejad, I., Hofmann, K., and Devlin, S. Imitating hu- man behaviour with diffusion models. InThe Eleventh International Conference on Learning Representations,

arXiv 2023
[13]

Goal- conditioned imitation learning using score-based diffu- sion policies.arXiv preprint arXiv:2304.02532,

Reuss, M., Li, M., Jia, X., and Lioutikov, R. Goal- conditioned imitation learning using score-based diffu- sion policies.arXiv preprint arXiv:2304.02532,

Pith/arXiv arXiv
[14]

Dynamics model benefits from conditioning.Normalized return on D4RL medium-replay tasks. K. Should Trajectory Denoisers Be Causal? A natural question is whether the denoiser should mirror the forward-time causality of the underlying dynamics or whether such a restriction limits its modeling capacity. Motivated by this, and following observations in Chen e...

2024
[15]

Latent dif- fusion planning for imitation learning.arXiv preprint arXiv:2504.16925,

Xie, A., Rybkin, O., Sadigh, D., and Finn, C. Latent dif- fusion planning for imitation learning.arXiv preprint arXiv:2504.16925,

Pith/arXiv arXiv
[16]

URL https://arxiv.org/abs/2506. 12544. Zhao, H., Han, X., Zhu, Z., Liu, M., Yu, Y ., and Zhang, W. Long-horizon rollout via dynamics diffusion for offline reinforcement learning.arXiv preprint arXiv:2405.19189,

Pith/arXiv arXiv
[18]

Several recent works attempt to alleviate this issue through inverse dynamics models (IDMs)

and corroborated by our results (Figure 2), this often yields trajectories that deviate from true transition structures—demonstrating that producing perfectly dynamically consistent sequences with diffusion models is inherently difficult. Several recent works attempt to alleviate this issue through inverse dynamics models (IDMs). For instance, Ajay et al....

2023
[19]

In this section, we outline the key architectural and hyperparameter choices: • Both the planner noise model ϵpl θ and the dynamics noise model ϵdyn θ are implemented as temporal U-Nets as proposed by Janner et al. (2022). Each network consists of six repeated residual blocks, where each block contains two temporal 13 Model-Based Diffusion Sampling for Pr...

2022
[20]

We track an exponential moving average of the weights with decay 0.005, which is employed for evaluation

with a learning rate of 2×10 −4, a batch size of 64, and 1×10 6 training steps. We track an exponential moving average of the weights with decay 0.005, which is employed for evaluation. • The conditioning vector is randomly dropped during training with probabilityp= 0.25. • We use K= 100 diffusion steps for D4RL and DSRL benchmarks, K= 10 for Unitree Go2 ...

2022
[23]

As reported in Table 15, causal architectures lead to a slight drop in performance

and evaluate them on D4RL medium-replay tasks. As reported in Table 15, causal architectures lead to a slight drop in performance. Although system dynamics are inherently causal, the optimal denoiser in a diffusion model need not be: score estimation at each timestep is a smoothing operation that benefits from future context (Wiener, 1964), and similar ob...

1964
[24]

Intuitively, ppl concentrates on trajectories that resemble the dataset and thus capture task structure, and preferences present in demonstrations

denote theinducedtrajectory distribution obtained by running a fixed (e.g., DDIM) sampling procedure from the learned score/denoiser, conditioned on the initial state x0. Intuitively, ppl concentrates on trajectories that resemble the dataset and thus capture task structure, and preferences present in demonstrations. Projection toward dynamics feasibility...

1999
[25]

(11) Equivalently, we can write: q∗(τ|x 0)∝p pl(τ|x 0)p dyn(τ|x 0)λ.(12) Thus the optimal target distribution q∗ is a combination of the planner distribution and the dynamics distribution, with the exponentλcontrolling their relative influence. Sampling from q∗.Directly characterizing q∗ is difficult in practice: we do not have an explicit form for the pl...

2021
[26]

This expectation is intractable as it requires marginalizing over all possible clean trajectories consistent with τ k

i ,(14) where the expectation is over the posterior distribution of clean trajectories given the noisy observation. This expectation is intractable as it requires marginalizing over all possible clean trajectories consistent with τ k. Following common practice in score-based diffusion models, we approximate this with a sum of individual scores: sq∗ (τ k, ...

2006
[2006]

G., and Levine, S

Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit q-learning as an actor- critic method with diffusion policies.arXiv preprint arXiv:2304.10573,

Pith/arXiv arXiv
[2007]

T., Baierl, M., Koert, D., and Peters, J

Carvalho, J., Le, A. T., Baierl, M., Koert, D., and Peters, J. Motion planning diffusion: Learning and planning of robot motions with diffusion models. In2023 IEEE/RSJ International Conference on Intelligent Robots and Sys- tems (IROS), pp. 1916–1923. IEEE,

1916
[2014]

Diffwave: A versatile diffusion model for audio synthesis

Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761,

Pith/arXiv arXiv 2009
[2015]

and Schmidhuber, J

Ha, D. and Schmidhuber, J. World models.arXiv preprint arXiv:1803.10122, 2(3),

Pith/arXiv arXiv
[2017]

org/abs/1710.05941

URL https://arxiv. org/abs/1710.05941. Rawlings, J. B., Mayne, D. Q., and Diehl, M.Model Pre- dictive Control: Theory, Computation, and Design. Nob Hill Publishing,

Pith/arXiv arXiv
[2018]

and Salimans, T

Ho, J. and Salimans, T. Classifier-free diffusion guidance. InNeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications,

2021
[2021]

Fast pol- icy learning through imitation and reinforcement.arXiv preprint arXiv:1805.10413,

Cheng, C.-A., Yan, X., Wagener, N., and Boots, B. Fast pol- icy learning through imitation and reinforcement.arXiv preprint arXiv:1805.10413,

Pith/arXiv arXiv
[2022]

Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

Pith/arXiv arXiv
[2023]

Concrete problems in ai safety

Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schul- man, J., and Man ´e, D. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565,

Pith/arXiv arXiv
[2024]

doi: 10.1109/tnnls.2023.3250269

ISSN 2162-2388. doi: 10.1109/tnnls.2023.3250269. URL http://dx.doi. org/10.1109/TNNLS.2023.3250269. Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4 {rl}: Datasets for deep data-driven reinforcement learning,

arXiv 2023
[2025]

Proximal policy optimization algorithms

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

Pith/arXiv arXiv

[1] [1]

Naively, each environment step requires running a full reverse diffusion chain, which can be computationally expensive. To accelerate planning, we adopt a warm-start strategy: the generated trajectory from the previous step is partially diffused forward for a fixed number of steps, after which the same number of reverse diffusion steps are applied to obta...

2022

[2] [2]

In Section E we consider a linear system with a stochastic expert, providing a controlled setting where feasibility can be studied in detail

In Section D we compare two schemes for incorporating the initial state—inpainting versus FiLM-based conditioning—through an ablation on D4RL Hopper. In Section E we consider a linear system with a stochastic expert, providing a controlled setting where feasibility can be studied in detail. In Section F, we evaluate the performance of MPDiffuser under mod...

2023

[3] [10]

A., Du, Y ., and Xu, D

Luo, Y ., Mishra, U. A., Du, Y ., and Xu, D. Genera- tive trajectory stitching through diffusion composition. InICRA 2025 Workshop on Foundation Models and Neuro-Symbolic AI for Robotics,

2025

[4] [11]

Pearce, T., Rashid, T., Kanervisto, A., Bignell, D., Sun, M., Georgescu, R., Macua, S

doi: 10.1109/LRA.2023.3270034. Pearce, T., Rashid, T., Kanervisto, A., Bignell, D., Sun, M., Georgescu, R., Macua, S. V ., Tan, S. Z., Momen- nejad, I., Hofmann, K., and Devlin, S. Imitating hu- man behaviour with diffusion models. InThe Eleventh International Conference on Learning Representations,

arXiv 2023

[5] [13]

Goal- conditioned imitation learning using score-based diffu- sion policies.arXiv preprint arXiv:2304.02532,

Reuss, M., Li, M., Jia, X., and Lioutikov, R. Goal- conditioned imitation learning using score-based diffu- sion policies.arXiv preprint arXiv:2304.02532,

Pith/arXiv arXiv

[6] [14]

Dynamics model benefits from conditioning.Normalized return on D4RL medium-replay tasks. K. Should Trajectory Denoisers Be Causal? A natural question is whether the denoiser should mirror the forward-time causality of the underlying dynamics or whether such a restriction limits its modeling capacity. Motivated by this, and following observations in Chen e...

2024

[7] [15]

Latent dif- fusion planning for imitation learning.arXiv preprint arXiv:2504.16925,

Xie, A., Rybkin, O., Sadigh, D., and Finn, C. Latent dif- fusion planning for imitation learning.arXiv preprint arXiv:2504.16925,

Pith/arXiv arXiv

[8] [16]

URL https://arxiv.org/abs/2506. 12544. Zhao, H., Han, X., Zhu, Z., Liu, M., Yu, Y ., and Zhang, W. Long-horizon rollout via dynamics diffusion for offline reinforcement learning.arXiv preprint arXiv:2405.19189,

Pith/arXiv arXiv

[9] [18]

Several recent works attempt to alleviate this issue through inverse dynamics models (IDMs)

and corroborated by our results (Figure 2), this often yields trajectories that deviate from true transition structures—demonstrating that producing perfectly dynamically consistent sequences with diffusion models is inherently difficult. Several recent works attempt to alleviate this issue through inverse dynamics models (IDMs). For instance, Ajay et al....

2023

[10] [19]

In this section, we outline the key architectural and hyperparameter choices: • Both the planner noise model ϵpl θ and the dynamics noise model ϵdyn θ are implemented as temporal U-Nets as proposed by Janner et al. (2022). Each network consists of six repeated residual blocks, where each block contains two temporal 13 Model-Based Diffusion Sampling for Pr...

2022

[11] [20]

We track an exponential moving average of the weights with decay 0.005, which is employed for evaluation

with a learning rate of 2×10 −4, a batch size of 64, and 1×10 6 training steps. We track an exponential moving average of the weights with decay 0.005, which is employed for evaluation. • The conditioning vector is randomly dropped during training with probabilityp= 0.25. • We use K= 100 diffusion steps for D4RL and DSRL benchmarks, K= 10 for Unitree Go2 ...

2022

[12] [23]

As reported in Table 15, causal architectures lead to a slight drop in performance

and evaluate them on D4RL medium-replay tasks. As reported in Table 15, causal architectures lead to a slight drop in performance. Although system dynamics are inherently causal, the optimal denoiser in a diffusion model need not be: score estimation at each timestep is a smoothing operation that benefits from future context (Wiener, 1964), and similar ob...

1964

[13] [24]

Intuitively, ppl concentrates on trajectories that resemble the dataset and thus capture task structure, and preferences present in demonstrations

denote theinducedtrajectory distribution obtained by running a fixed (e.g., DDIM) sampling procedure from the learned score/denoiser, conditioned on the initial state x0. Intuitively, ppl concentrates on trajectories that resemble the dataset and thus capture task structure, and preferences present in demonstrations. Projection toward dynamics feasibility...

1999

[14] [25]

(11) Equivalently, we can write: q∗(τ|x 0)∝p pl(τ|x 0)p dyn(τ|x 0)λ.(12) Thus the optimal target distribution q∗ is a combination of the planner distribution and the dynamics distribution, with the exponentλcontrolling their relative influence. Sampling from q∗.Directly characterizing q∗ is difficult in practice: we do not have an explicit form for the pl...

2021

[15] [26]

This expectation is intractable as it requires marginalizing over all possible clean trajectories consistent with τ k

i ,(14) where the expectation is over the posterior distribution of clean trajectories given the noisy observation. This expectation is intractable as it requires marginalizing over all possible clean trajectories consistent with τ k. Following common practice in score-based diffusion models, we approximate this with a sum of individual scores: sq∗ (τ k, ...

2006

[16] [2006]

G., and Levine, S

Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit q-learning as an actor- critic method with diffusion policies.arXiv preprint arXiv:2304.10573,

Pith/arXiv arXiv

[17] [2007]

T., Baierl, M., Koert, D., and Peters, J

Carvalho, J., Le, A. T., Baierl, M., Koert, D., and Peters, J. Motion planning diffusion: Learning and planning of robot motions with diffusion models. In2023 IEEE/RSJ International Conference on Intelligent Robots and Sys- tems (IROS), pp. 1916–1923. IEEE,

1916

[18] [2014]

Diffwave: A versatile diffusion model for audio synthesis

Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761,

Pith/arXiv arXiv 2009

[19] [2015]

and Schmidhuber, J

Ha, D. and Schmidhuber, J. World models.arXiv preprint arXiv:1803.10122, 2(3),

Pith/arXiv arXiv

[20] [2017]

org/abs/1710.05941

URL https://arxiv. org/abs/1710.05941. Rawlings, J. B., Mayne, D. Q., and Diehl, M.Model Pre- dictive Control: Theory, Computation, and Design. Nob Hill Publishing,

Pith/arXiv arXiv

[21] [2018]

and Salimans, T

Ho, J. and Salimans, T. Classifier-free diffusion guidance. InNeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications,

2021

[22] [2021]

Fast pol- icy learning through imitation and reinforcement.arXiv preprint arXiv:1805.10413,

Cheng, C.-A., Yan, X., Wagener, N., and Boots, B. Fast pol- icy learning through imitation and reinforcement.arXiv preprint arXiv:1805.10413,

Pith/arXiv arXiv

[23] [2022]

Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

Pith/arXiv arXiv

[24] [2023]

Concrete problems in ai safety

Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schul- man, J., and Man ´e, D. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565,

Pith/arXiv arXiv

[25] [2024]

doi: 10.1109/tnnls.2023.3250269

ISSN 2162-2388. doi: 10.1109/tnnls.2023.3250269. URL http://dx.doi. org/10.1109/TNNLS.2023.3250269. Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4 {rl}: Datasets for deep data-driven reinforcement learning,

arXiv 2023

[26] [2025]

Proximal policy optimization algorithms

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

Pith/arXiv arXiv