Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ADG recovers corrupted offline RL datasets with a two-model diffusion pipeline, letting standard algorithms beat robust baselines.

desk verdict ADG's empirical data-recovery pipeline is a useful and novel contribution, but the theoretical guarantees in Proposition 4.4 and Theorem 4.3 do not hold as stated. read the letter →

arxiv 2505.23871 v2 pith:FYAOVL35 submitted 2025-05-29 cs.LG cs.AI

Zeyuan Liu , Zhihe Yang , Jiawei Xu , Rui Yang , Jiafei Lyu , Baoxiang Wang , Yunjian Xu , Xiu Li This is my paper · ORCID
classification cs.LGcs.AI
keywords offlinereinforcementlearningdatacorruptiondiffusionmodelsambientDDPMdatasetrecoverydetectionstateandactionsequencemodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that corrupted offline RL datasets can be repaired rather than robustly learned from, by adding a three-stage diffusion preprocessing layer. It introduces ADG, which trains an ambient DDPM on partially corrupted trajectory slices, uses the noise-prediction norm to flag corrupted transitions, trains a standard DDPM on the flagged-clean subset, and then denoises the corrupted transitions. The central claim is that any downstream offline RL algorithm—pessimistic Q-learning, policy-constraint methods, or sequence models—improves on the recovered dataset, and often outperforms its dedicated robust variant, under both random and adversarial state, action, and reward corruption. The theoretical support is that an ambient DDPM trained on approximated distributions can learn the clean data distribution once a sufficiently large diffusion timestep is chosen, and that the detection signal is strongest at that timestep.

What carries the argument

The paper's central mechanism is the ambient DDPM detector paired with a selectively trained standard DDPM denoiser. The detector uses an ambient-diffusion objective with an ambient timestep $k_a$; Corollary 4.1 shows that minimizing a reweighted prediction loss over forward-noised samples yields the conditional noise predictor $\mathbb{E}[\epsilon \mid x_k]$ for all $k \ge k_a$. Theorem 4.3 then shows that, for bounded corruption scale, the clean forward distribution and the partially corrupted forward distribution can be made arbitrarily close in KL divergence by choosing $k_a$ large enough, while Proposition 4.4 shows that the detector's signal-to-noise ratio is maximized at $k_a$. The detector flags samples by thresholding $e_\theta(\check{z}_t) = \|\epsilon_\theta(\check{\tau}_t, k_a)_{H+1}\|_F^2$ with threshold $\zeta$; the denoiser is a standard DDPM trained only on the clean subset, using a mask to ignore corrupted positions within trajectory slices, and it refines the flagged corrupted transitions.

What would settle it

Take an offline dataset with known corruption labels, train the ambient DDPM at the paper's $k_a = 30$, and measure the false-positive and false-negative rates of the $e_\theta(\check{z}_t) > \zeta$ rule against ground truth at corruption scales $\alpha \in \{0.5, 1.0, 2.0\}$; if the false-negative rate is not small enough that the clean subset remains effectively noise-free, or if the same $\zeta$ fails to transfer across corruption scales, then the recovery stage's premise fails.

Watch

Extended reading notes

Core claim

The paper's core claim is that a data-only repair pipeline can make standard offline RL algorithms more robust than algorithms explicitly designed for corrupted data. ADG works in three stages: first, an ambient DDPM is trained on trajectory slices from the partially corrupted dataset, using a loss that provably recovers the noise predictor of the clean distribution even though some training samples are corrupted; second, the squared Frobenius norm of that predictor at a chosen ambient timestep distinguishes corrupted from clean transitions; third, a standard DDPM is trained only on the detected-clean subset and used to renoise and denoise the corrupted transitions, restoring them before RL training. The paper reports consistent improvements across CQL, IQL, RIQL, DT, and RDT on MuJoCo, Kitchen, and Adroit benchmarks, under random, adversarial, and Gaussian corruption, and for missing-data settings, with the recovered datasets often outperforming both filtered datasets and the noised baselines.

Load-bearing premise

The whole pipeline depends on the detector's noise-prediction norm, computed at timestep $k_a$ and thresholded by $\zeta$, cleanly separating corrupted from clean samples; if it does not, the supposedly clean subset used to train the denoiser is itself contaminated and the recovery stage has no stated guarantee.

Editorial extensions

If this is right

  • ADG can be wrapped around any offline RL algorithm without modifying the algorithm's loss, since it only replaces the dataset that the algorithm consumes.
  • Standard non-robust algorithms trained on ADG-recovered data outperform their explicitly robust variants on most tested corruption scenarios, suggesting that dataset repair can substitute for robust algorithmic design.
  • Recovery outperforms simple filtering: filtered datasets lose sequential information and degrade performance, while ADG's recovered datasets restore trajectory structure that MDP-based and sequence-modeling methods both use.
  • ADG improves performance under state-only, full-element, Gaussian, adversarial, and missing-data corruption, and remains effective across dataset sizes from 1% to 100% and corruption rates up to 0.5.
  • Separating the detector and denoiser into two independent diffusion models is important; sharing one model creates loss interference and reduces downstream performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the noise-prediction norm cleanly separates clean and corrupted transitions as reported, the same detector could serve as a general data-quality audit for offline datasets, flagging suspicious transitions before any RL training begins.
  • The two-model recipe may transfer to other partially corrupted time-series domains beyond RL, such as robotics demonstrations or medical sensor logs, wherever clean samples form a large uncontaminated majority and corruption is sparse and additive.
  • The paper's theory is stated for scaled Gaussian noise, so an immediate testable extension is whether the detection guarantee survives structured or adversarial dropouts, which the missing-data experiments probe only empirically.
  • Because ADG is data-only, it can be stacked with robust offline RL algorithms rather than competing with them; the paper's ablations suggest this combination should be explored at higher corruption rates than the current $\eta = 0.3$ setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ADG, a three-stage data-only preprocessing method for offline reinforcement learning under random and adversarial corruption. Stage 1 trains an 'Ambient DDPM' on the partially corrupted dataset; Stage 2 uses the squared Frobenius norm of the noise prediction at a chosen timestep k_a to classify transitions as clean or corrupted; Stage 3 trains a standard DDPM on the predicted clean subset and uses it to refine the predicted corrupted samples before running any offline RL algorithm. Experiments on MuJoCo, Kitchen, and Adroit benchmarks, with state-only and full-element corruption, show consistent improvements for CQL, IQL, RIQL, DT, RDT, and two recent methods, A2PR and NUNO; ablations study the choice of k_a, the threshold zeta, the slice size H, and single- versus dual-model variants.

Significance. If the empirical claims hold, ADG would provide a practical dataset-level layer that can be composed with essentially any offline RL algorithm and consistently outperform dedicated robust variants on corrupted datasets. The paper's strengths are its broad experimental coverage (multiple tasks, corruption types, dataset scales, and algorithms), four-seed averages, detailed ablations, reported compute overhead, and visualizations of detection and recovery. The advertised theoretical guarantees, however, are not established as stated: Proposition 4.4's SNR proof drops the forward-process diffusion noise, Theorem 4.3 rests on an assumption close to the result it claims to prove, and the training objective in Eq. (6) is not clearly the objective implemented in Algorithm 1. These issues are load-bearing because the clean/corrupted partition produced by the detector is what makes the subsequent denoiser training feasible. The empirical program is strong enough to warrant a major revision rather than rejection, but the theoretical section needs substantial rework and the algorithm specification needs to be made internally consistent.

major comments (4)
  1. [Section 4.3, Appendix A.3, Eq. (30)] The derivation of the detection SNR omits the forward-process diffusion noise. In DDPM, the noised sample is x_k = sqrt(alpha_bar_k) x + sqrt(1-alpha_bar_k) xi, so the prediction target in Eq. (28) equals xi + [sqrt(alpha_bar_k)/sqrt(1-alpha_bar_k)] * iota * I_noise * epsilon, not the expression in Eq. (30). Consequently, the 'noise-free' case in Eq. (31) does not have a zero prediction target; the standard DDPM objective requires the model to predict the nonzero diffusion noise xi even for clean inputs. The denominator of the SNR formula in Eq. (5) therefore misses the unit-variance xi contribution, and the proof does not establish the claimed maximum at k = k_a. A corrected calculation is needed; if the xi term is retained and the network error variance is assumed constant, the ratio becomes iota^2 * alpha_bar_k / ((1 - alpha_bar_k)(1 + sigma_k^2)), which is still monotone in k under that assumption, but this is not the formula or proof given in the paper.
  2. [Section 4.2, Assumption 4.2 and Theorem 4.3] Assumption 4.2 asserts the very learnability property that Theorem 4.3 is supposed to provide. The theorem only shows that DKL[q(x_k|x_0) || q(check_x_k|check_x_0)] can be made smaller than any prescribed c for k >= k_a, and then invokes Assumption 4.2; the quantification over c in the theorem is not tied to the existential c in the assumption, so the statement is close to a restatement of the assumption. No verifiable condition on the corruption rate, noise scale, or network class is given. In addition, Theorem 4.3 requires k_a large enough for the KL divergence to be small, while Proposition 4.4 requires k_a small for maximal detection SNR; the paper acknowledges a trade-off in Section 4.2 but provides no quantitative reconciliation. Since the detector's output selects the clean subset used to train the denoiser, this gap is load-bearing for the overall method.
  3. [Section 4.4, Eq. (6) versus Algorithm 1] The training loss actually implemented is not specified consistently. Equation (6) defines check_tau^k_t as a conditional noising of the observed check_tau^{k_a}_t, namely check_tau^k_t = sqrt(alpha_bar_k/alpha_bar_{k_a}) check_tau^{k_a}_t + sqrt((alpha_bar_{k_a} - alpha_bar_k)/alpha_bar_{k_a}) epsilon, which matches Corollary 4.1. Algorithm 1, however, produces noised elements as tilde_z^k_t = sqrt(alpha_bar_k) check_z_t + sqrt(1 - alpha_bar_k) epsilon, which is the standard forward noising from x0. These are different training objectives unless check_z_t is identified with x0, but the algorithm text says check_z_t is the observed component that 'may or may not contain noise'. The paper needs to state which objective was actually used in the experiments and reconcile it with the ambient theory.
  4. [Section 4.4, Stage 3] No theoretical guarantee is stated for the recovery step. The theoretical results concern only the ambient DDPM used for detection; there is no bound on the reconstruction error of the naive DDPM applied to the detected corrupted samples, nor any connection between the recovered dataset and downstream policy performance. Given the abstract's claim of 'theoretical guarantees,' this should be scoped explicitly, or the recovery stage needs its own analysis.
minor comments (5)
  1. [Tables 1 and 2] Several table cells contain rendering artifacts, e.g., '34.0±13.466.3±15.9', '26.±5.0', and '0.7±0.0 20.3±7.3'; these should be cleaned up so each entry has a single mean and standard deviation.
  2. [Section 5.2] The aggregate improvement percentages quoted for MDP-based algorithms (69.1%) and non-MDP algorithms (17.4%) do not obviously match the per-algorithm averages in Tables 1 and 2; please state the aggregation rule or correct the numbers.
  3. [Algorithm 1, Step 3] The recovery formula uses an unspecified timestep k; the text in Section 4.4 refers to a reverse DDPM process p_phi( tau^{0:k_a}_t ), but Algorithm 1 appears to perform a single denoising step. Please specify the exact reverse procedure and the starting timestep.
  4. [Throughout] There are several typos and proofreading issues, including 'revese', 'Appendix a A.3', 'corrputed', 'offine', and 'Datastet'.
  5. [Table 3] The hyperparameter table lists 'Learning Rate (lr) 5' and then a separate 'Learning Rate 1e-4'; please clarify which learning rate applies to which network component.

Circularity Check

2 steps flagged · score 6.0 of 10

ADG's empirical pipeline is self-contained, but its advertised theoretical guarantees are partly circular: Theorem 4.3 restates Assumption 4.2, and Proposition 4.4's SNR derivation assumes the clean/corrupted separability it is meant to prove.

  1. self definitional [Section 4.2, Assumption 4.2 and Theorem 4.3 (proof in Appendix A.2)]
    "Assumption 4.2. There exists a positive constant c such that, for any k ≥ ka, if the Kullback-Leibler (KL) divergence satisfies DKL[q(xk|x0)∥ϱ(xk|x0)] < c, then the ambient DDPM with k ≥ ka, as introduced in Corollary 4.1, can be effectively learned from samples drawn from the approximated distribution ϱ(xk|x0). ... Theorem 4.3. Let Assumption 4.2 hold. ... ambient DDPM with k ≥ ka ... can instead be effectively learned from samples drawn from q(ˇxk|ˇx0)."

    The theorem's conclusion is exactly Assumption 4.2 instantiated with ϱ = q(ˇxk|ˇx0). Lemmas A.1 and A.2 only prove the KL antecedent: they show that the Gaussian forward distributions q(xk|x0) and q(ˇxk|ˇx0) become close for large k. The nontrivial content, that closeness of KL implies effective learnability of the ambient DDPM, is imported verbatim from the assumption. Thus the advertised 'theoretical guarantee' for learning on partially corrupted datasets is not derived; it is the input of the theorem.

  2. other [Section 4.3, Proposition 4.4 and Appendix A.3, Eqs. (28)-(32)]
    "Substituting Eq.(29) into Eq.(28), we obtain: ϵpred = (√¯αk · ι · Inoise / √(1 − ¯αk)) · ε ... Case 1: Noise-free data (Inoise = 0) ... ϵθ( ˇxk, k) = ϵpred + δkθ = 0 + δkθ."

    The proof replaces the actual forward-process input xk = √¯αk ˇx + √(1−¯αk)ξ with the scaled original hk(ˇx)=√¯αk ˇx, dropping the forward diffusion noise ξ. For clean data this makes the 'perfect prediction' ϵpred zero, so the detector's clean-case output is assumed to be pure error. But the ambient DDPM trained via Corollary 4.1 satisfies ϵθ*(xk,k)=E[ϵ|xk], which is nonzero for clean data because the model is trained to predict the forward diffusion noise. The SNR formula (5) therefore builds in the clean/corrupted separation it is supposed to establish; with the retained forward noise, the denominator is not mnσ² and the claimed maximum at k=ka is unsupported.

full rationale

ADG's empirical contribution is not circular: the detector is trained with the ambient loss on the corrupted dataset, the denoiser is trained on the predicted-clean subset, and the recovered dataset is evaluated by downstream RL scores and by MSE against ground truth; no downstream score is fed back into the training loss, and comparisons against CQL, IQL, RIQL, DT, and RDT under random, adversarial, Gaussian, and missing-data corruption provide independent evidence that the pipeline helps. I find no load-bearing self-citation: reference [34] (RDT) shares authors but is used only for the trajectory storage scheme and as a baseline, not as a uniqueness or correctness argument. The circularity is confined to the theory section. First, Theorem 4.3's conclusion, that ambient DDPM can be effectively learned from q(ˇxk|ˇx0), is exactly Assumption 4.2 with ϱ = q(ˇxk|ˇx0); Lemmas A.1 and A.2 only establish the KL antecedent, so the learnability guarantee is assumed rather than proved. Second, Proposition 4.4's SNR proof sets the clean-case prediction to zero (Eq. 31) after replacing the forward-noised input by the scaled original hk(ˇx)=√¯αk ˇx (Eqs. 29-30), dropping the forward diffusion noise that Corollary 4.1's minimizer is trained to predict; the clean/corrupted separability that the detector is supposed to establish is thus built into the proof. These two issues undermine the 'theoretical guarantees' claim but do not invalidate the empirical evaluation. There is also minor threshold tuning: ζ = 0.20 is selected on walker2d via downstream D4RL scores and then used in the main tables, which slightly inflates that task's reported result but does not affect the consistent cross-task improvement pattern. Overall, the central empirical method is self-contained, while the supporting theory is partially circular; this is a correctness risk at the detection threshold k=ka, but the empirical findings stand on their own.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on one ad hoc learnability assumption, several unvalidated distributional assumptions about noise and generalization, and three hand-tuned hyperparameters. There are no newly invented physical or conceptual entities. The numbers that matter most are ka, ζ, and H, each selected using downstream task performance.

free parameters (3)
  • ka (ambient noise timestep) = 30
    Selected in Section 5.3 (Figure 3) to minimize false negatives and maximize downstream D4RL scores; appears in Corollary 4.1 and Eq. (6).
  • zeta (corruption threshold) = 0.20
    Chosen from ablations in Figure 4 and Appendix D.9; used both to classify corrupted samples and to mask the denoiser training loss in Eq. (7).
  • H (temporal slice size) = 5
    Selected from the ablation in Appendix D.6; detector and denoiser performance improve up to H=3 and then plateau, and H=5 is used in the main experiments.
assumptions (4)
  • ad hoc to paper Assumption 4.2: There exists a positive constant c such that if DKL[q(xk|x0) || ϱ(xk|x0)] < c, then the ambient DDPM with k ≥ ka can be effectively learned from the approximated distribution.
    This is not proved and it is exactly the learning guarantee that Theorem 4.3 is supposed to deliver, so the theoretical bridge is conditional on an unstated learnability property.
  • domain assumption The noise prediction error follows δkθ ~ N(0, σk²I), with σk equal across diffusion timesteps.
    Used in Proposition 4.4 to derive the closed-form SNR formula; no empirical or theoretical justification is given for Gaussian prediction errors with uniform variance.
  • domain assumption The additive Gaussian noise model in Proposition 4.4 transfers to the uniform random attacks and PGD-based adversarial attacks used in the experiments.
    Appendix C.1 defines corruption with uniform noise and projected gradient descent, not additive Gaussian noise; the separation guarantees are only derived for Gaussian ι·ε.
  • domain assumption A standard DDPM trained on the detected-clean subset can denoise corrupted samples not present in that subset.
    Recovery depends on the denoiser generalizing from predicted-clean trajectories to corrupted ones; no distributional closeness or error bound is provided for this step.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/FYAOVL35

@misc{pith2026250523871,
  author       = {Pith},
  title        = {Pith review of: ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FYAOVL35}},
  note         = {Machine review of arXiv:2505.23871}
}
read the original abstract

Real-world datasets collected from sensors or human inputs are prone to noise and errors, posing significant challenges for applying offline reinforcement learning (RL). While existing methods have made progress in addressing corrupted actions and rewards, they remain insufficient for handling corruption in high-dimensional state spaces and for cases where multiple elements in the dataset are corrupted simultaneously. Diffusion models, known for their strong denoising capabilities, offer a promising direction for this problem-but their tendency to overfit noisy samples limits their direct applicability. To overcome this, we propose Ambient Diffusion-Guided Dataset Recovery (ADG), a novel approach that pioneers the use of diffusion models to tackle data corruption in offline RL. First, we introduce Ambient Denoising Diffusion Probabilistic Models (DDPM) from approximated distributions, which enable learning on partially corrupted datasets with theoretical guarantees. Second, we use the noise-prediction property of Ambient DDPM to distinguish between clean and corrupted data, and then use the clean subset to train a standard DDPM. Third, we employ the trained standard DDPM to refine the previously identified corrupted data, enhancing data quality for subsequent offline RL training. A notable strength of ADG is its versatility-it can be seamlessly integrated with any offline RL algorithm. Experiments on a range of benchmarks, including MuJoCo, Kitchen, and Adroit, demonstrate that ADG effectively mitigates the impact of corrupted data and improves the robustness of offline RL under various noise settings, achieving state-of-the-art results.

Figures

Figures reproduced from arXiv: 2505.23871 by the authors.

Figure 1
Figure 1. Overview of the training processes for the detector and denoiser (left) and the dataset [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The performance of baseline algorithms (CQL, IQL, RIQL, DT, and RDT) is evaluated [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The FN rate during detector training (left), and the performance of IQL and DT using [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Results on “walker2d-medium-replay￾v2” with Random State Attack (Appendix C.1): (a) MSE vs. ground truth, and (b) DT perfor￾mance across ζ. Best results at ζ = 0.20. Impact of Selective Training. To evaluate the impact of selective training on denoiser, we vary ζ withi…
Figure 5
Figure 5. Figure 5: Comparison results among Naive, ADG with a single diffusion model, and ADG with separate diffusion models for IQL (upper) and DT (lower). ADG (w/ SD) denotes ADG using a single diffusion model. Impact of Using Two Separate Diffusion Mod￾els for Detection and Denoising.…
Figure 6
Figure 6. Figure 6: Neural network structure of ADG. C.3 Hyperparameters We present the hyperparameters and other details of ADG in [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Visualization of the distribution of e(xˇ) for samples in the datasets, including halfcheetah, hopper, walker2d, kitchen-complete, kitchen-mixed, kitchen-partial, door, hammer, and relocate, denoted as “ha”, “ho”, “wa”, “kit-c”, “kit-m”, “kit-p”, “do”, “hm”, and “re”, …
Figure 8
Figure 8. Figure 8: Visualization of the denoising effect of ADG on the “hopper-medium-replay-v2” dataset. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: The performance of IQL (left) and RIQL (right) on the noised, filtered, purified, and [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: The performance of the ADG detector and denoiser under varying temporal slice sizes [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Performance of ADG under random corruption across different dataset scales. [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Results under various corruption rates (a) and scales (b) on the “walker2d-medium [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Results of (a) the detection performance in the dataset recovery process and (b) D4RL [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Comparison results among Naive, ADG with a single diffusion model, and ADG with [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Ambient Diffusion Omni: Training Good Models with Bad Data

    cs.GR 2025-06 conditional novelty 6.0 of 10

    Ambient Diffusion Omni trains diffusion models on mixed-quality data by learning when corrupted images can be treated as clean, improving generation quality and diversity.

Reference graph

Works this paper leans on

43 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Ambient diffusion posterior sampling: Solving inverse problems with diffusion models trained on corrupted data

    Asad Aali, Giannis Daras, Brett Levac, Sidharth Kumar, Alexandros G Dimakis, and Jonathan I Tamir. Ambient diffusion posterior sampling: Solving inverse problems with diffusion models trained on corrupted data. arXiv preprint arXiv:2403.08728, 2024

    arXiv 2024

  2. [2]

    Offline Reinforcement Learning from Datasets with Structured Non-Stationarity

    Johannes Ackermann, Takayuki Osa, and Masashi Sugiyama. Offline reinforcement learning from datasets with structured non-stationarity. arXiv preprint arXiv:2405.14114, 2024

    work page Pith review arXiv 2024

  3. [3]

    Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2023

    Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2023

    work page 2023

  4. [4]

    Uncertainty-based offline reinforcement learning with diversified q-ensemble

    Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information process- ing systems, 34:7436–7447, 2021

    work page 2021

  5. [5]

    Pessimistic bootstrapping for uncertainty-driven offline reinforcement learn- ing

    Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learn- ing. arXiv preprint arXiv:2202.11566, 2022

    arXiv 2022

  6. [6]

    Decision transformer: Reinforcement learning via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021

    2021

  7. [7]

    Exact policy recovery in offline rl with both heavy-tailed rewards and data corruption

    Yiding Chen, Xuezhou Zhang, Qiaomin Xie, and Xiaojin Zhu. Exact policy recovery in offline rl with both heavy-tailed rewards and data corruption. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11416–11424, 2024

    work page 2024

  8. [8]

    Consistent diffusion meets tweedie: Training exact ambient diffusion models with noisy data

    Giannis Daras, Alex Dimakis, and Constantinos Costis Daskalakis. Consistent diffusion meets tweedie: Training exact ambient diffusion models with noisy data. In Forty-first International Conference on Machine Learning, 2024

    work page 2024

Show all 43 references
  1. [9]

    D4rl: Datasets for deep data-driven reinforcement learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    2004 arXiv

  2. [10]

    A minimalist approach to offline reinforcement learn- ing

    Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learn- ing. Advances in neural information processing systems, 34:20132–20145, 2021

    2021

  3. [11]

    Off-policy deep reinforcement learning with- out exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning with- out exploration. In International conference on machine learning, pages 2052–2062. PMLR, 2019

    2019

  4. [12]

    Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters

    Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. Advances in Neural Information Processing Systems, 35:18267–18281, 2022. 10

    2022

  5. [13]

    IDQL: Implicit Q-learning as an actor-critic method with diffusion policies

    Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023

    2023 arXiv

  6. [14]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Ad- vances in Neural Information Processing Systems, 33:6840–6851, 2020

    2020

  7. [15]

    Planning with diffusion for flexible behavior synthesis

    Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning , pages 9902–

  8. [16]

    Offline reinforcement learning as one big sequence modeling problem

    Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems , 34:1273– 1286, 2021

    2021

  9. [17]

    Neural stochastic differential equations for uncertainty-aware offline rl

    Cevahir Koprulu, Franck Djeumou, et al. Neural stochastic differential equations for uncertainty-aware offline rl. In The Thirteenth International Conference on Learning Rep- resentations

  10. [18]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021

    2021 arXiv

  11. [19]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179– 1191, 2020

    2020

  12. [20]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020

    2020

  13. [21]

    Survival instinct in offline reinforcement learning

    Anqi Li, Dipendra Misra, Andrey Kolobov, and Ching-An Cheng. Survival instinct in offline reinforcement learning. Advances in neural information processing systems, 36, 2024

    2024

  14. [22]

    Robust preference optimization with provable noise tolerance for llms

    Xize Liang, Chao Chen, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. Robust preference optimization with provable noise tolerance for llms. arXiv preprint arXiv:2404.04102, 2024

    2024 arXiv

  15. [23]

    Adapt- diffuser: Diffusion models as adaptive self-evolving planners

    Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adapt- diffuser: Diffusion models as adaptive self-evolving planners. In International Conference on Machine Learning, pages 20725–20745. PMLR, 2023

    2023

  16. [24]

    Adaptive advantage-guided policy regularization for offline reinforcement learning

    Tenglong Liu, Yang Li, Yixing Lan, Hao Gao, Wei Pan, and Xin Xu. Adaptive advantage-guided policy regularization for offline reinforcement learning. arXiv preprint arXiv:2405.19909, 2024

    2024 arXiv

  17. [25]

    Towards deep learning models resistant to adversarial attacks

    Aleksander Madry. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017

    2017 arXiv

  18. [26]

    Robust re- inforcement learning using offline data

    Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust re- inforcement learning using offline data. Advances in neural information processing systems , 35:32211–32224, 2022

    2022

  19. [27]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  20. [28]

    Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity

    Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. Journal of Machine Learning Research, 25(200):1–91, 2024

    2024

  21. [29]

    Unleashing the power of pre- trained language models for offline reinforcement learning

    Ruizhe Shi, Yuyao Liu, Yanjie Ze, Simon S Du, and Huazhe Xu. Unleashing the power of pre- trained language models for offline reinforcement learning. arXiv preprint arXiv:2310.20587, 2023. 11

    2023 arXiv

  22. [30]

    Deep un- supervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep un- supervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015

    2015

  23. [31]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020

    2020

  24. [32]

    Diffusion policies as an expressive policy class for offline reinforcement learning

    Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022

    2022

  25. [33]

    Copa: Certifying robust policies for offline reinforcement learning against poisoning attacks

    Fan Wu, Linyi Li, Chejian Xu, Huan Zhang, Bhavya Kailkhura, Krishnaram Kenthapadi, Ding Zhao, and Bo Li. Copa: Certifying robust policies for offline reinforcement learning against poisoning attacks. arXiv preprint arXiv:2203.08398, 2022

    2022 arXiv

  26. [34]

    Tackling data corruption in offline reinforcement learning via sequence modeling

    Jiawei Xu, Rui Yang, Shuang Qiu, Feng Luo, Meng Fang, Baoxiang Wang, and Lei Han. Tackling data corruption in offline reinforcement learning via sequence modeling. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  27. [35]

    Rorl: Robust offline reinforcement learning via conservative smoothing

    Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. Advances in neural infor- mation processing systems, 35:23851–23866, 2022

    2022

  28. [36]

    Regularizing hidden states enables learning generalizable reward model for llms.arXiv preprint arXiv:2406.10216, 2024

    Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for llms.arXiv preprint arXiv:2406.10216, 2024

    2024 arXiv

  29. [37]

    Towards robust offline reinforcement learning under diverse data corruption

    Rui Yang, Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, and Tong Zhang. Towards robust offline reinforcement learning under diverse data corruption. In The Twelfth International Conference on Learning Representations, 2024

    2024

  30. [38]

    Dmbp: Diffusion model-based predictor for robust offline rein- forcement learning against state observation perturbations

    Zhihe Yang and Yunjian Xu. Dmbp: Diffusion model-based predictor for robust offline rein- forcement learning against state observation perturbations. In The Twelfth International Con- ference on Learning Representations, 2024

    2024

  31. [39]

    Towards robust model-based reinforce- ment learning against adversarial corruption

    Chenlu Ye, Jiafan He, Quanquan Gu, and Tong Zhang. Towards robust model-based reinforce- ment learning against adversarial corruption. arXiv preprint arXiv:2402.08991, 2024

    2024 arXiv

  32. [40]

    Corruption-robust offline reinforcement learning with general function approximation

    Chenlu Ye, Rui Yang, Quanquan Gu, and Tong Zhang. Corruption-robust offline reinforcement learning with general function approximation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  33. [41]

    Robust reinforcement learning on state observations with learned optimal adversary

    Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint arXiv:2101.08452, 2021

    2021 arXiv

  34. [42]

    Robust deep reinforcement learning against adversarial perturbations on state observa- tions

    Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observa- tions. Advances in Neural Information Processing Systems, 33:21024–21037, 2020

    2020

  35. [43]

    medium-replay- v2

    Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Corruption-robust offline rein- forcement learning. In International Conference on Artificial Intelligence and Statistics, pages 5757–5773. PMLR, 2022. 12 A Theoretical Interpretations A.1 Proof for Corollary 4.1 Firstly, w...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.