Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Behavior-regularized offline RL extends to diffusion policies when the KL penalty is summed stepwise along the denoising path, and the paper proves both objectives share the same optimal policy.

desk verdict BDPO has a clean pathwise-KL idea and strong D4RL results, but Theorem 4.2's equivalence relies on an idealized behavior process that finite-N diffusion and neural approximation break. read the letter →

arxiv 2502.04778 v2 pith:HXMXM3GM submitted 2025-02-07 cs.LG cs.AI

Chen-Xiao Gao , Chenyang Wu , Mingjun Cao , Chenjun Xiao , Yang Yu , Zongzhang Zhang This is my paper · ORCID
classification cs.LGcs.AI
keywords behavior-regularizedofflinereinforcementlearningdiffusionpoliciespathwiseKLdivergencetwo-time-scaleactor-criticsoftpolicyiterationD4RLbenchmarkGirsanovtheoremBoltzmann
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Offline reinforcement learning needs a brake: the learned policy should stay close to the policy that collected the data, or value estimates for unseen actions run away. That behavior regularization is straightforward for Gaussian or deterministic policies, but the KL divergence it requires is intractable for diffusion policies, whose action log-likelihoods are hard to compute. BDPO measures the KL along the diffusion generation path instead of on the final actions: because every reverse step is Gaussian, the divergence decomposes into a sum of per-step terms that are computable in closed form. The paper's central theorem states that this substitution changes nothing about the answer — the optimal policy of the pathwise problem is the optimal policy of the original KL-regularized objective. If the claim holds, expressive diffusion policies can receive the same pessimism-based safety guarantees as simple parametric policies, at the cost of an efficient two-time-scale actor-critic that never differentiates through the whole generation path.

What carries the argument

The load-bearing object is the pathwise KL, the divergence between two diffusion generation processes accumulated step by step rather than at their final outputs. Its defining identity is the decomposition $D_{\mathrm{KL}}[p^{\pi,s}_{0:N}\,\|\,p^{\nu,s}_{0:N}] = \sum_{n=1}^{N}\mathbb{E}_{a_n}[\ell^{\pi,s}_n(a_n)]$, which is valid because both processes share the same Gaussian prior and each transition is conditionally Gaussian; each term reduces to the mean-squared error between the two denoising predictions divided by $2\sigma_n^2$. Theorem 4.2 is the argument's hinge: its proof derives the optimal reverse process explicitly and shows it is the forward process initialized at the Boltzmann policy $\pi_Q(a|s) \propto \nu(a|s)\exp(Q(s,a)/\eta)$, which marginalizes exactly to the optimum of the standard KL-regularized RL objective. Around this identity the paper builds a two-time-scale value architecture in which $Q^\pi$ backs up rewards minus accumulated path penalties across environment timesteps while $V^{\pi,s}_n$ backs up the remaining penalty sum across diffusion steps, so the actor is trained by a single-step Gaussian KL minimization at every denoising step.

What would settle it

On a synthetic energy dataset with a known Boltzmann target, train a deliberately weak behavior diffusion model (fewer denoising steps or truncated training) and run BDPO, then compare the final action distribution against $p(a|s) \propto \nu(a|s)\exp(Q(s,a)/\eta)$ using that same $\nu$; if the mismatch grows with the behavior model's error, the equivalence of Theorem 4.2 depends on the idealization $p^{\nu} = q^{\nu}$. A complementary check is numerical: with a density estimator, compute the marginal KL $D_{\mathrm{KL}}[\pi(\cdot|s)\,\|\,\nu(\cdot|s)]$ and the accumulated per-step penalties on the same states, and see how far apart they become as diffusion steps shrink.

Watch

Extended reading notes

Core claim

Behavior-regularized offline RL augments returns with a KL penalty $D_{\mathrm{KL}}[\pi(\cdot|s)\,\|\,\nu(\cdot|s)]$ that holds the policy near the behavior policy that collected the dataset; for diffusion policies this divergence between action marginals is intractable. BDPO instead regularizes the whole generation path, exploiting the Markov property to decompose the divergence into a sum of per-step KLs between reverse transition kernels, $D_{\mathrm{KL}}[p^{\pi,s}_{0:N}\,\|\,p^{\nu,s}_{0:N}] = \sum_{n=1}^{N}\mathbb{E}[\ell^{\pi,s}_n(a_n)]$ with $\ell^{\pi,s}_n(a_n) = D_{\mathrm{KL}}[p^{\pi,s,a_n}_{n-1|n}\,\|\,p^{\nu,s,a_n}_{n-1|n}]$. Because each reverse transition is an isotropic Gaussian, every term becomes a closed-form squared distance between the denoising means, weighted by the noise schedule. Theorem 4.2 establishes that solving this pathwise problem yields the same optimal policy as the standard objective: the optimal reverse process is the forward noising process started from the Boltzmann policy $\pi_Q(a|s) \propto \nu(a|s)\exp(Q(s,a)/\eta)$, so its marginal at the clean step coincides with the KL-regularized optimum. The paper concretizes this in a two-time-scale actor-critic where an environment-level critic accumulates the path penalties into its TD target and diffusion value functions $V^{\pi,s}_n$ estimate the remaining penalty cost from each intermediate denoising state, so each policy improvement step touches only one reverse transition.

Load-bearing premise

The argument assumes the trained behavior diffusion policy is effectively the exact reverse process of the dataset's forward noising process, so that the per-step KL penalties equal the true divergence between the actor and behavior generation paths; the paper gives no quantitative bound on what fails as the behavior model's approximation error grows.

Editorial extensions

If this is right

  • Diffusion policies obtain the same behavior-regularized objective as Gaussian and deterministic policies, inheriting its protection against out-of-distribution actions while keeping their expressive multi-modal coverage.
  • Policy improvement costs a constant amount per environment step regardless of the number of denoising steps, since only single reverse transitions are optimized; the paper reports actor runtime of roughly 17–30 minutes for $N = 5, 20, 50$, versus growth from 57 to 308 minutes for full-path backpropagation.
  • As the number of diffusion steps grows, the pathwise penalty converges to the continuous-time KL between the two reverse SDEs obtained from Girsanov's theorem, so the discrete penalty is a consistent discretization of a well-defined continuous quantity.
  • On synthetic 2D energy datasets the final samples match the Boltzmann target distribution $p \propto \nu \exp(E/\eta)$, and on D4RL the method reports the highest normalized-score aggregates among the compared baselines for both locomotion and antmaze tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Nothing in the paper bounds how the pathwise optimum degrades when the trained behavior diffusion deviates from the true reverse process, which the equivalence proof assumes; a quantitative sensitivity analysis of the penalty to behavior-model approximation error would decide how robust the framework is in practice.
  • The same decomposition should transfer to other generative policy parameterizations with tractable per-step transitions, such as flow matching or consistency models, where the per-step divergence would again be a cheap mean discrepancy between vector fields.
  • Because the algorithm samples intermediate actions from the dataset's forward noising process rather than from the actor's own path, it is off-policy in the diffusion MDP; an on-policy version would match the theory more tightly at the sampling cost the paper mentions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces BDPO, a behavior-regularized offline RL algorithm for diffusion policies. Instead of regularizing the intractable KL divergence between action marginals, it uses a pathwise KL divergence computed as a sum of KL divergences between per-step reverse-transition kernels (Eq. (11)). The authors prove, under idealized assumptions, that optimizing this pathwise objective yields the same optimal policy as the marginal-KL objective (Theorem 4.2), and they develop an actor-critic method with two value scales: a standard environment-level Q-function and per-diffusion-step value functions V_n that amortize policy improvement. Experiments on synthetic 2D tasks and D4RL continuous control benchmarks show strong empirical performance, and Appendix D connects the pathwise KL to a Girsanov-based continuous-time formula.

Significance. If the equivalence in Theorem 4.2 holds in the setting where the method is applied, the paper makes a useful contribution: it gives a tractable way to impose behavior regularization on multimodal diffusion policies, an algorithmic scheme that avoids backpropagating through the entire diffusion path, and a clean two-timescale actor-critic formulation. The paper also ships a substantial appendix with proofs, a Girsanov consistency check, runtime and inference-time measurements, and ablations on the regularization strength, LCB coefficient, and number of diffusion steps. These are genuine strengths. The empirical results on D4RL are strong and broadly supportive, and the synthetic experiments directly verify that the learned sampler approximates the target Boltzmann distribution. The main caveat is that the central equivalence theorem requires the learned behavior diffusion process to be exactly the reverse process of the forward process initialized at the data policy; the paper does not quantify how the equivalence degrades under finite-N discretization and neural approximation error.

major comments (3)
  1. [Appendix C.2, Eq. (35)–(36); Theorem 4.2] Theorem 4.2/Appendix C.2, Eqs. (35)–(36): the proof assumes pν is exactly the reverse process qν of the forward process initialized at ν. For finite N (N=5 in Table 3), pν_N = N(0,I) differs from qν_N, and neural approximation adds further error. The data-processing inequality then implies the pathwise penalty is an upper bound on the intended marginal KL, so the optimizer of Eq. (12) need not equal π*(a|s) ∝ ν(a|s) exp(Q*(s,a)/η). No quantitative degradation bound is given, so the central equivalence claim is not established in the regime actually used in the experiments.
  2. [Section 4.3 and Algorithm 1] The convergence propositions (Lemma C.3, Proposition 4.4, Proposition 4.5) are proved for the idealized procedure in which policy evaluation and improvement use on-policy diffusion trajectories pπ. In contrast, Algorithm 1 trains V_ϕ and the actor on triplets (s, a, n, a_n) sampled with a_n ~ q_{n|0}(·|a) from the dataset, as explicitly stated in Section 4.3 ('Selection of States, Diffusion Steps and Actions'). This is an off-policy surrogate, and the paper provides no importance-weighting or bias/variance argument connecting it to the on-policy expectations in Eqs. (15) and (16). Since the actual algorithm is what achieves the reported results, the convergence guarantees do not directly apply to the implemented method; this should be addressed either by a proof that the off-policy samples form a valid surrogate or by a separate convergence analysis for the practical objective.
  3. [Section 4.3, Eq. (17)] The analytical KL formula in Eq. (17) uses the Gaussian approximation of each reverse transition. The paper correctly notes in Section 4.3 that this is exact in the continuous-time limit and Appendix D verifies consistency via Girsanov's theorem. However, with N=5 the per-step reverse transitions are not exactly Gaussian with the specified σ_n, and the approximation error of Eq. (17) is not discussed. In the synthetic experiments N=50 is used, but D4RL uses N=5, so the discretization error is different between the two settings. The authors should state whether the empirical conclusions on D4RL depend on the Gaussian approximation being accurate at N=5.
minor comments (6)
  1. [Algorithm 1 caption] The algorithm title contains a typo: 'Diffuion' should be 'Diffusion'.
  2. [Section 5.3, Table 2 discussion] In the sentence after Table 2, 'pertaining phase' should be 'pretraining phase'.
  3. [Algorithm 1, line 8] The notation 'a_n^t' is used in the pseudocode and text, but the same symbol was introduced with 't' as an environment timestep; consider writing 'a_n' or 'a_i,n' to avoid confusion with the environment timestep index.
  4. [Appendix E.6, Figure 18] The ablation reports N=2,5,10, but the synthetic experiments use N=50; adding an N=50 point on the D4RL tasks would make it easier to see how close the discrete pathwise penalty is to the continuous-time limit.
  5. [Figure 16 caption] The caption says 'D4RL Locomotion datasets' but the figure shows antmaze navigation results; the caption should say 'Navigation datasets'.
  6. [Section 2] The related work section for diffusion policies in offline RL would benefit from a brief comparison with 'Diffusion-QL' and 'DAC' on the key algorithmic difference: BDPO replaces full-path gradient backpropagation with per-step value functions. This is mentioned, but the distinction between 'V_n' and the Q-function guidance of DAC could be made more explicit earlier.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the pathwise-KL decomposition is algebraic, the optimality theorem is derived from external soft-policy-iteration results, and the key KL reduction is a shared-kernel identity; only an idealized-assumption gap and non-load-bearing self-citations are noted.

full rationale

Walked the claimed derivation chain. Eq. (11) is a purely algebraic decomposition of the pathwise KL given the common Gaussian prior p^pi_N = p^nu_N = N(0,I); no fitted quantity enters. Theorem C.1 instantiates the diffusion path as an MDP and applies the standard KL-regularized soft-policy-iteration recursion (citing Wu et al. 2019 and Haarnoja et al. 2018, both external), yielding p* as the reverse process of the forward process with initial pi_Q proportional to nu exp(Q/eta); this is derived, not assumed. Theorem C.2's key reduction, DKL[q^{pi_Q}_{0:N} || q^nu_{0:N}] = DKL[pi_Q || nu], uses only the fact that the two forward processes share the same transition kernels; it is a mathematical identity, not an imported premise. The equivalence is genuinely conditional on the idealization that the learned p^nu equals the exact reverse of q^nu, and on a prior N(0,I) that is not exactly q^nu_N for finite N; the finite-N mismatch (N=5 in experiments) is an omitted caveat in the proof, and the neural approximation error is unquantified, but this is a robustness/approximation gap rather than a circular definition of the target result. Self-citations (ACT, Shribak et al., Ran et al.) appear only in related-work or contextual remarks and never carry a load-bearing premise. No fitted hyperparameter is relabeled as a prediction: eta and rho are tuned per task, and the theoretical statements are parameter-free with respect to those tuned values. Verdict: no significant circularity; the low score reflects only non-load-bearing self-citations and the idealized-assumption gap.

Assumptions & free parameters 4 free parameters · 6 assumptions · 1 invented entities

The central theoretical result relies on standard diffusion assumptions plus the exactness of the learned behavior reverse process; the algorithm also relies on a heuristic off-policy sampling scheme. The tuned hyperparameters η, ρ, N, and the ensemble size are fitted per dataset, but they do not enter the equivalence theorem as fitted inputs.

free parameters (4)
  • η (regularization strength) = 0.05 to 1.0 per D4RL dataset (Table 4); 0.06 for synthetic 2D tasks
    Controls how strongly the policy is constrained toward the behavior policy; tuned per dataset in the D4RL experiments.
  • ρ (LCB coefficient) = 0.5 to 2.0 per D4RL task (Table 4); 0.0 for synthetic 2D tasks
    Controls the level of pessimism in the ensemble value target; tuned per dataset and shown to affect performance in Figure 7.
  • Diffusion steps N = 5 for D4RL, 50 for synthetic 2D, ablation over {2,5,10}
    Number of reverse diffusion steps; a design choice balancing generation quality and runtime, with degradation observed at N=2 (Figure 18).
  • Value ensemble size = 10, except 20 for hopper-medium-v2 exclusively
    The ensemble size for Q and V networks is increased for one task without a stated reason beyond stabilizing that task (Appendix B.1).
assumptions (6)
  • domain assumption The learned behavior diffusion pν is identical to the true reverse transition qν of the forward process with initial distribution equal to the dataset policy
    Used in the proof of Theorem C.1 (Eq. 28) and in Theorem 4.2 to identify pν with qν; holds only for a perfectly trained diffusion model, with no error bound given.
  • domain assumption Reverse transitions of actor and behavior policies are isotropic Gaussians with identical diagonal variance σ_n I
    Eq. (8) and Eq. (17): the KL penalty is computed as ||µπ - µν||² / (2σ_n²), which requires shared fixed variance; if the actor uses a different variance or non-Gaussian kernel, the analytic KL is invalid.
  • ad hoc to paper Assumption 4.3: sup over actions of π(a|s)/ν(a|s) < ∞
    Assumed to guarantee boundedness and convergence of soft policy evaluation (Propositions 4.4 and 4.5); not always satisfied for diffusion policies whose support may be larger than the behavior's.
  • ad hoc to paper Off-policy training samples (s, a, n, a_n ~ q_{n|0}) form a sufficient surrogate for on-policy diffusion trajectories
    Section 4.3: the algorithm uses this heuristic and the paper states it 'works sufficiently well' without proof; the convergence propositions assume the on-policy distribution.
  • standard math Standard MDP assumptions: bounded rewards, discount γ in (0,1), Markovian dynamics
    Used throughout the RL formalization in Section 3; standard for reinforcement learning theory.
  • standard math Forward diffusion follows Gaussian transitions with known variance schedule, and the prior is standard Gaussian
    Eqs. (5)-(8), standard DDPM assumption; needed for the KL decomposition and the Girsanov analysis in Appendix D.
invented entities (1)
  • Diffusion value function V_n(s, a_n)
    purpose: Estimates the expected cumulative KL penalties from diffusion step n to 0 for a given state and intermediate action, enabling single-step policy improvement without backpropagating through all diffusion steps.
    Introduced in Eq. (15) and Figure 3. It is an algorithmic construct trained by TD across diffusion steps; there is no external falsifiable prediction attached to it beyond the policy's benchmark performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/HXMXM3GM

@misc{pith2026250204778,
  author       = {Pith},
  title        = {Pith review of: Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HXMXM3GM}},
  note         = {Machine review of arXiv:2502.04778}
}
read the original abstract

Behavior regularization, which constrains the policy to stay close to some behavior policy, is widely used in offline reinforcement learning (RL) to manage the risk of hazardous exploitation of unseen actions. Nevertheless, existing literature on behavior-regularized RL primarily focuses on explicit policy parameterizations, such as Gaussian policies. Consequently, it remains unclear how to extend this framework to more advanced policy parameterizations, such as diffusion models. In this paper, we introduce BDPO, a principled behavior-regularized RL framework tailored for diffusion-based policies, thereby combining the expressive power of diffusion policies and the robustness provided by regularization. The key ingredient of our method is to calculate the Kullback-Leibler (KL) regularization analytically as the accumulated discrepancies in reverse-time transition kernels along the diffusion trajectory. By integrating the regularization, we develop an efficient two-time-scale actor-critic RL algorithm that produces the optimal policy while respecting the behavior constraint. Comprehensive evaluations conducted on synthetic 2D tasks and continuous control tasks from the D4RL benchmark validate its effectiveness and superior performance.

Figures

Figures reproduced from arXiv: 2502.04778 by the authors.

Figure 1
Figure 1. Illustration of the behavior-regularized RL framework with different policy parameterizations. Unimodal policies, such as deterministic policies (left), compute the behavior as the center of mass and therefore lead to misleading regularizations; while our method (right) harnesses the flexibility of diffusion models, and the regularization is calculated as the accumulated discrepancies in diffusion directions of the … view at source ↗
Figure 2
Figure 2. Semantic illustration of the interplay between diffusion policies and the environment. We use orange to denote the tran￾sition p π,st,an n−1|n and the penalty ℓ π,st n (see Section 4.1) associated with the diffusion generation process, whereas blue signifies the transition T(·|st, a0 t ) and the reward rt from the original environ￾ment MDP. In this paper, we will use p π and p π,s to denote the dif￾fusion policy and… view at source ↗
Figure 3
Figure 3. Semantic illustration of the TD backup for the Q-value function Q π (blue) and diffusion value function V π,s n (orange). The update of Q π (Eq. (13)) requires reward, penalties along the diffusion trajectory, and the Q-values at the next state. The update of V π,s n (Eq. (15)) involves the single-step penalty and the diffusion value at the next diffusion step n − 1. At first glance, one might treat the pathwise KL … view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Generation paths of BDPO on the 8gaussian (top), 2spirals (middle), and moons (down) datasets. The regularization strength is set to η = 0.06, which is identical to [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Illustration of the 8gaussians, 2spirals, and moons datasets. The top row depicts the original data distribution pdata, while the second row depicts the target distribution ptarget at η = 0.06 by re-sampling data points according to their energies. Remark (Actor-critic…
Figure 6
Figure 6. Figure 6: Sensitivity analysis of the regularization strength η. For each configuration, we report the mean and the standard deviation of the performances aggregated from 5 independent seeds and 10 evaluation episodes for each seed. 0 0.5M 1M 1.5M 2M Training Steps 40 50 60 70 N…
Figure 7
Figure 7. Figure 7: Sensitivity analysis of the lower confidence bound coefficient ρ. For each configuration, we report the mean and the standard deviation of the performances aggregated from 5 independent seeds and 10 evaluation episodes for each seed. dataset and visualizes the iterativ…
Figure 8
Figure 8. Figure 8: Comparison of different policy parameterizations. Re￾sults are taken from 10 evaluation episodes and 5 seeds. datasets, where variations may occur due to the sparse re￾ward nature of these datasets. 5.3. Analysis of BDPO Regularization Strength η. The hyperparameter η …
Figure 9
Figure 9. Figure 9: Illustration of synthetic 2D datasets and sampling results with various temperature η. We also use Gym MuJoCo (Brockman et al., 2016), which provides a series of continuous control tasks to assess the performance of BDPO as well as the baseline methods. Locomotion Task…
Figure 10
Figure 10. Figure 10: Illustration of the Locomotion and Navigation Tasks [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Algorithm Runtime of BDPO, DAC and Diffusion-QL. x n = x( nT N ), and sn(x n) = s(x( nT N ), nT N ). Therefore, lim N→∞ DKL [p π 0:N ∥p ν 0:N ] = lim N→∞ Ep π 0:N "X N n=1 βn(1 − α¯n) 2(1 − α¯n−1)(1 − βn) ∥s π n (x n ) − s ν n (x n )∥ 2 # = lim N→∞ Ep π 0:N "X N n=1 β…
Figure 12
Figure 12. Figure 12: Inference Time Comparison. E.2. Discussion about Inference Time Apart from training time, we also evaluate inference time since the latency is critical in real-world applications like robotic control. To benchmark the inference time accurately, we sampled 10,000 state…
Figure 13
Figure 13. Figure 13: Illustration of the diffusion policy and the diffusion value function from BDPO on synthetic 2D datasets. The regularization strength is set to η = 0.06. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Illustration of the diffusion policy and the Q-value function from DAC on synthetic 2D datasets. The regularization strength is set to η = 0.06. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Evaluation scores of BDPO on D4RL Locomotion datasets. Results are aggregated using 5 independent seeds and 10 evaluation episodes for each seed. 0 0.5M 1M 1.5M 2M Training Steps 25 50 75 100 Normalized Score antmaze-umaze-v0 0 0.5M 1M 1.5M 2M Training Steps 60 80 100…
Figure 16
Figure 16. Figure 16: Evaluation scores of BDPO on D4RL Locomotion datasets. Results are aggregated using 5 independent seeds and 100 evaluation episodes for each seed. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]
Figure 17
Figure 17. Figure 17: Ablation study with the policy parameterizations. Results are aggregated using 5 independent seeds and 10 evaluation episodes for each seed. 0 0.5M 1M 1.5M 2M Training Steps 0 20 40 60 80 100 Normalized Score hopper-medium-v2 step=2 step=5 step=10 0 0.5M 1M 1.5M 2M Tr…
Figure 18
Figure 18. Figure 18: Ablation study with the number of diffusion steps N. Results are aggregated using 5 independent seeds and 10 evaluation episodes for each seed. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_18.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Decision Flow Policy Optimization

    cs.LG 2025-05 reject novelty 6.0 of 10

    Decision Flow frames the gradual action generation of flow-based policies as a flow MDP and updates the flow policy with flow-level value functions, reporting state-of-the-art results on several D4RL tasks.

  2. ReBRAC-v2: The Return of the King

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A fixed-recipe offline RL method combining normalizing-flow actors, categorical critics, staged training, and test-time refinement beats recent flow-based baselines by 22.5 points averaged over ten OGBench categories.

Reference graph

Works this paper leans on

56 extracted references · 38 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    B., Jaakkola, T

    Ajay, A., Du, Y., Gupta, A., Tenenbaum, J. B., Jaakkola, T. S., and Agrawal, P. Is conditional generative modeling all you need for decision making? In International Conference on Learning Representations (ICLR), 2023

    work page 2023

  3. [3]

    J., Mittal, S., Lemos, P., Liu, C., Sendera, M., Ravanbakhsh, S., Gidel, G., Bengio, Y., Malkin, N., and Tong, A

    Akhound - Sadegh, T., Rector - Brooks, J., Bose, A. J., Mittal, S., Lemos, P., Liu, C., Sendera, M., Ravanbakhsh, S., Gidel, G., Bengio, Y., Malkin, N., and Tong, A. Iterated denoising energy matching for sampling from boltzmann densities. In International Conference on Machine Learning (ICML), 2024

    work page 2024

  4. [4]

    Diffusion for world modeling: Visual details matter in atari

    Alonso, E., Jelley, A., Micheli, V., Kanervisto, A., Storkey, A., Pearce, T., and Fleuret, F. Diffusion for world modeling: Visual details matter in atari. arXiv preprint arXiv:2405.12399, 2024

    arXiv 2024

  5. [5]

    An, G., Moon, S., Kim, J., and Song, H. O. Uncertainty-based offline reinforcement learning with diversified Q -ensemble. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    work page 2021

  6. [6]

    Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning

    Bai, C., Wang, L., Yang, Z., Deng, Z., Garg, A., Liu, P., and Wang, Z. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2022

    work page 2022

  7. [7]

    Open AI gym

    Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Open AI gym. arXiv preprint arXiv:1606.01540, 2016

    arXiv 2016

  8. [8]

    Offline reinforcement learning via high-fidelity generative behavior modeling

    Chen, H., Lu, C., Ying, C., Su, H., and Zhu, J. Offline reinforcement learning via high-fidelity generative behavior modeling. In International Conference on Learning Representations (ICLR), 2023

    work page 2023

Show all 56 references
  1. [9]

    Score regularized policy optimization through diffusion behavior

    Chen, H., Lu, C., Wang, Z., Su, H., and Zhu, J. Score regularized policy optimization through diffusion behavior. In International Conference on Learning Representations (ICLR), 2024 a

    2024

  2. [10]

    Decision transformer: Reinforcement learning via sequence modeling

    Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  3. [11]

    Diffusion policies creating a trust region for offline reinforcement learning

    Chen, T., Wang, Z., and Zhou, M. Diffusion policies creating a trust region for offline reinforcement learning. arXiv preprint arXiv:2405.19690, 2024 b

    2024 arXiv

  4. [12]

    Soft actor-critic for discrete action settings

    Christodoulou, P. Soft actor-critic for discrete action settings. arXiv preprint arXiv:1910.07207, 2019

    1910 arXiv

  5. [13]

    Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning

    Fang, L., Liu, R., Zhang, J., Wang, W., and Jing, B.-Y. Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. arXiv preprint arXiv:2405.20555, 2024

    2024 arXiv

  6. [14]

    MINDE : Mutual information neural diffusion estimation

    Franzese, G., BOUNOUA, M., and Michiardi, P. MINDE : Mutual information neural diffusion estimation. In International Conference on Learning Representations (ICLR), 2024

    2024

  7. [15]

    D4RL: D atasets for deep data-driven reinforcement learning

    Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4RL: D atasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    2004 arXiv

  8. [16]

    and Gu, S

    Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  9. [17]

    Off-policy deep reinforcement learning without exploration

    Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning (ICML), 2019

    2019

  10. [18]

    ACT: E mpowering decision transformer with dynamic programming via advantage conditioning

    Gao, C.-X., Wu, C., Cao, M., Kong, R., Zhang, Z., and Yu, Y. ACT: E mpowering decision transformer with dynamic programming via advantage conditioning. In AAAI Conference on Artificial Intelligence (AAAI) , 2024

    2024

  11. [19]

    Extreme Q -learning: Maxent RL without entropy

    Garg, D., Hejna, J., Geist, M., and Ermon, S. Extreme Q -learning: Maxent RL without entropy. In International Conference on Learning Representations (ICLR), 2023

    2023

  12. [20]

    Reinforcement learning with deep energy-based policies

    Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning (ICML), 2017

    2017

  13. [21]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML), 2018

    2018

  14. [22]

    G., and Levine, S

    Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit Q -learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023

    2023 arXiv

  15. [23]

    Denoising diffusion probabilistic models

    Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  16. [24]

    B., and Levine, S

    Janner, M., Du, Y., Tenenbaum, J. B., and Levine, S. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning (ICML), 2022

    2022

  17. [25]

    Policy rehearsing: Training generalizable policies for reinforcement learning

    Jia, C., Gao, C., Yin, H., Zhang, F., Chen, X.-H., Xu, T., Yuan, L., Zhang, Z., Zhou, Z.-H., and Yu, Y. Policy rehearsing: Training generalizable policies for reinforcement learning. In International Conference on Learning Representations (ICLR), 2024

    2024

  18. [26]

    Efficient diffusion policies for offline reinforcement learning

    Kang, B., Ma, X., Du, C., Pang, T., and Yan, S. Efficient diffusion policies for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  19. [27]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015

    2015

  20. [28]

    Offline reinforcement learning with implicit Q -learning

    Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit Q -learning. In International Conference on Learning Representations (ICLR), 2022

    2022

  21. [29]

    Stabilizing off-policy Q -learning via bootstrapping error reduction

    Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabilizing off-policy Q -learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems (NearIPS), 2019

    2019

  22. [30]

    Discor: Corrective feedback in reinforcement learning via distribution correction

    Kumar, A., Gupta, A., and Levine, S. Discor: Corrective feedback in reinforcement learning via distribution correction. In Advances in Neural Information Processing Systems (NeurIPS), 2020 a

    2020

  23. [31]

    Conservative Q -learning for offline reinforcement learning

    Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative Q -learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020 b

    2020

  24. [32]

    A workflow for offline model-free robotic reinforcement learning

    Kumar, A., Singh, A., Tian, S., Finn, C., and Levine, S. A workflow for offline model-free robotic reinforcement learning. In Conference on Robot Learning (CORL), 2021

    2021

  25. [33]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  26. [34]

    Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning

    Lu, C., Chen, H., Chen, J., Su, H., Li, C., and Zhu, J. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning (ICML), 2023

    2023

  27. [35]

    Reward-consistent dynamics models are strongly generalizable for offline reinforcement learning

    Luo, F., Xu, T., Cao, X., and Yu, Y. Reward-consistent dynamics models are strongly generalizable for offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2024

    2024

  28. [36]

    Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning

    Mao, L., Xu, H., Zhan, X., Zhang, W., and Zhang, A. Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  29. [37]

    Transformers are sample-efficient world models

    Micheli, V., Alonso, E., and Fleuret, F. Transformers are sample-efficient world models. In International Conference on Learning Representations (ICLR), 2023

    2023

  30. [38]

    Awac: Accelerating online reinforcement learning with offline datasets

    Nair, A., Gupta, A., Dalal, M., and Levine, S. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  31. [39]

    Stochastic differential equations: A n introduction with applications

    Oksendal, B. Stochastic differential equations: A n introduction with applications . Springer Science & Business Media, 2013

    2013

  32. [40]

    L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models t...

    2022

  33. [41]

    Policy regularization with dataset constraint for offline reinforcement learning

    Ran, Y., Li, Y., Zhang, F., Zhang, Z., and Yu, Y. Policy regularization with dataset constraint for offline reinforcement learning. In International Conference on Machine Learning (ICML), 2023

    2023

  34. [42]

    Z., Lidard, J., Ankile, L

    Ren, A. Z., Lidard, J., Ankile, L. L., Simeonov, A., Agrawal, P., Majumdar, A., Burchfiel, B., Dai, H., and Simchowitz, M. Diffusion policy policy optimization. arXiv preprint arXiv:2409.00588, 2024

    2024 arXiv

  35. [43]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  36. [44]

    Diffusion spectral representation for reinforcement learning

    Shribak, D., Gao, C.-X., Li, Y., Xiao, C., and Dai, B. Diffusion spectral representation for reinforcement learning. arXiv preprint arXiv:2406.16121, 2024

    2024 arXiv

  37. [45]

    Denoising diffusion implicit models

    Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  38. [46]

    P., Kumar, A., Ermon, S., and Poole, B

    Song, Y., Sohl - Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021

    2021

  39. [47]

    Model-bellman inconsistency for model-based offline reinforcement learning

    Sun, Y., Zhang, J., Jia, C., Lin, H., Ye, J., and Yu, Y. Model-bellman inconsistency for model-based offline reinforcement learning. In International Conference on Machine Learning (ICML), 2023

    2023

  40. [48]

    Revisiting the minimalist approach to offline reinforcement learning

    Tarasov, D., Kurenkov, V., Nikulin, A., and Kolesnikov, S. Revisiting the minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  41. [49]

    u l c ehre, C ., Wang, Z., Pfaff, T., Wu, Y., Ring, R., Yogatama, D., W \

    Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang, A., Sifre, L., Cai, T., Agapiou, J. P., Jaderberg, M., Vezhnevets, A. S., Leblond, R., Po...

    2019

  42. [50]

    Diffusion actor-critic with entropy regulator

    Wang, Y., Wang, L., Jiang, Y., Zou, W., Liu, T., Song, X., Wang, W., Xiao, L., Wu, J., Duan, J., et al. Diffusion actor-critic with entropy regulator. arXiv preprint arXiv:2405.15177, 2024

    2024 arXiv

  43. [51]

    J., and Zhou, M

    Wang, Z., Hunt, J. J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2023

    2023

  44. [52]

    Behavior regularized offline reinforcement learning

    Wu, Y., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019

    1911 arXiv

  45. [53]

    Xu, H., Jiang, L., Li, J., Yang, Z., Wang, Z., Chan, W. K. V., and Zhan, X. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations (ICLR), 2023

    2023

  46. [54]

    Y., Levine, S., Finn, C., and Ma, T

    Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. MOPO: M odel-based offline policy optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  47. [55]

    o lund, J., Sch \

    Zhang, R., Luo, Z., Sj \"o lund, J., Sch \"o n, T. B., and Mattsson, P. Entropy-regularized diffusion policy with q-ensembles for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024 a

    2024

  48. [56]

    o lund, J., Sch \

    Zhang, R., Luo, Z., Sj \"o lund, J., Sch \"o n, T. B., and Mattsson, P. Entropy-regularized diffusion policy with Q -ensembles for offline reinforcement learning. arXiv preprint arXiv:2402.04080, 2024 b

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.