Pith. sign in

REVIEW 3 major objections 6 minor 41 references

Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modeling

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that a transformer generating backward-from-goal trajectories, truncated when cumulative drift exceeds a threshold, can augment offline RL safely, with the policy gap bounded by the threshold times a penalty.

desk verdict Plausible offline RL augmentation heuristic with a broken theoretical guarantee: the theorem is vacuous and the pessimistic MDP is ill-defined. read the letter →

arxiv 2502.06491 v2 pith:MADJDFNL submitted 2025-02-10 cs.LG cs.AI

Shenghong He This is my paper · ORCID
classification cs.LGcs.AI
keywords model-basedofflinereinforcementlearningreliability-guaranteedsequencemodelingtrajectorytruncationpessimisticMDPtransformermodeldataaugmentationD4RLbenchmarkhigh-returngeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a model-based offline reinforcement learning algorithm can create useful extra training data by respecting the fact that real transitions depend on the whole prior trajectory, not just the current state and action. Its method, the Reliability-guaranteed Transformer (RT), trains a transformer on offline trajectories, measures the cumulative drift of each generated step from the real data, and stops generating as soon as that drift crosses a threshold; the same drift value is used to penalize rewards inside a pessimistic MDP. The paper claims a theoretical guarantee — a policy learned from this pessimistic MDP differs from the true optimal policy by at most a constant times the threshold times a penalty — and reports that RT improves the returns of off-the-shelf offline RL algorithms on D4RL continuous-control and maze benchmarks, outperforming both model-free and model-based baselines. If the central claim is right, an automatic, reliability-based stopping rule is the missing piece that makes model-generated data trustworthy enough to train policies from static datasets.

What carries the argument

The load-bearing object is the cumulative reliability $\Gamma(s_t, \tau_{<t})$ of Definition 1 — an attention-weighted running total of the total variation distance between the true and learned transition distributions along the generated prefix. It does double duty: it is the trigger for the truncation metric $U_t$ (generation stops when $\Gamma$ passes the threshold $\alpha$), and it is the penalty term subtracted from the reward in Eq. (6) inside the $\alpha$-pessimistic MDP whose policy gap Theorem 1 bounds. Since the true distribution is unavailable, the mechanism is realized by a variational autoencoder whose reconstruction error on $(s,a,s')$ stands in for the total variation distance, with $\alpha$ calibrated as the maximum reconstruction error in the offline dataset.

What would settle it

A concrete check: in a small environment whose true transition kernel is known — the paper's own BoxBall grid suffices — compute the real distance between the learned dynamics and the environment step by step, and compare the points at which RT's variational-autoencoder proxy crosses its threshold with the points where the true drift becomes large. If the two orderings disagree, or if perturbing the threshold alpha by +/-50% barely changes which trajectories survive, then the reliability metric is not the mechanism carrying the argument, and the reported bound is not tracking the quantity it claims to track.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that sequence-level reliability, not per-step uncertainty, is the right quantity for deciding how much of a generated trajectory to trust. RT defines the cumulative reliability $\Gamma(s_t, \tau_{<t})$ as a softmax-weighted running sum of total variation distances between the true transition distribution $P(\cdot|s,a)$ and the estimated one $\hat{P}(\cdot|s,a)$; when $\Gamma$ exceeds the threshold $\alpha$, the truncation metric $U_t$ marks the trajectory unreliable, the pessimistic MDP zeroes the transition and penalizes the reward by $\beta\,\Gamma/\alpha$, and generation halts. Theorem 1 asserts that the policy learned from this $\alpha$-pessimistic MDP satisfies $|V^P(\pi^*) - V^P(\pi)| \le C\alpha\beta$ in the true MDP, so small $\alpha$ and $\beta$ give a strict control on suboptimality. Because the true dynamics are unknown, the paper replaces the total variation distance with the reconstruction error of a variational autoencoder and sets $\alpha$ to the worst reconstruction error observed on the offline dataset. The experiments then claim that RT generates out-of-distribution but high-value states, adaptively beats fixed-length truncation, and that backward generation from goal states, combined with reliability truncation, is what yields the reported gains.

Load-bearing premise

The method assumes that a neural network's reconstruction error on state-action-next-state triples behaves the same as the true distance between the real environment's dynamics and the learned model's dynamics, so the cutoff for trusting a generated trajectory is set by how badly the network reconstructs the existing data; if that proxy fails in high-dimensional settings, the truncation rule and the penalty both measure the wrong quantity.

Editorial extensions

If this is right

  • RT can be added to any model-free offline RL method (BCQ, CQL, or IQL) and lift its normalized return on D4RL tasks, with the largest gains on maze and ant-maze domains where model-based baselines such as MOReL and MOPO collect near-zero reward.
  • Reliability-based truncation removes the need to hand-tune generation length: fixed-length variants RT-F3 and RT-F5 underperform automatic truncation on long-horizon sparse tasks.
  • Backward generation from goal states guarantees that generated trajectories include a high-reward endpoint, which the paper shows is the decisive advantage over forward-only generation on AntMaze-large.
  • Theorem 1's bound makes alpha and beta the two dials that trade guaranteed suboptimality against how much high-return data can be generated: smaller values shrink the worst-case gap but force shorter trajectories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the variational autoencoder proxy is a poor stand-in for total variation distance in high-dimensional continuous control, the truncation rule degrades into a heuristic early-stopping rule; a natural hardening would replace the proxy with a quantity that provably bounds the true distance, such as ensemble disagreement or a Lipschitz-regularized model error.
  • The paper's own forward-versus-backward comparison suggests a design principle the authors leave implicit: stitching backward from known high-reward states is cheaper and safer than predicting forward into the unknown, a principle that could transfer to diffusion-style trajectory generators, which currently fare worse when high-reward data is scarce.
  • A testable extension of Theorem 1 is to measure the empirical gap $|V^P(\pi^*) - V^P(\pi)|$ across the benchmark tasks and check whether it scales roughly linearly in the product alpha times beta; if the observed gap is flat in either knob, the bound is not the operative mechanism and generation quality is what matters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Reliability-guaranteed Transformer (RT), an offline-RL data augmentation method that trains a transformer to generate backward trajectories conditioned on high rewards, estimates per-step reliability with a variational autoencoder, truncates generated trajectories when the cumulative reliability exceeds a threshold, and augments offline data for a model-free offline RL algorithm. The manuscript claims a theoretical performance guarantee for the resulting policy (Theorem 1), presents D4RL benchmark comparisons against augmentation and offline-RL baselines, and includes ablations on generation direction, reliability estimation, and a BoxBall diagnostic environment.

Significance. If the theory and algorithm were sound, the paper would make a useful contribution to offline model-based RL: it proposes a concrete mechanism for using historical context, a dynamic truncation rule, and high-return-conditioned backward generation, and it evaluates these components across multiple D4RL domains and base algorithms. The empirical study is fairly broad, and the BoxBall visualization is a helpful qualitative check of unreliable trajectory generation. However, the central advertised result — a reliability-guaranteed, provably bounded policy — is not established. The main theorem is stated without proof and is vacuous as written, the "α-pessimistic MDP" is ill-defined because its transition function is not a valid kernel, and the VAE-based reliability estimate is not connected to the total-variation quantity used in the theorem. These are load-bearing issues, not presentation defects.

major comments (3)
  1. [Section 3.1, Theorem 1] The theorem asserts the existence of constants C and β but supplies no derivation. For any fixed policy pair and any αβ > 0, the inequality can be satisfied trivially by taking C = |V^P(π*) − V^P(π)|/(αβ); moreover, β already denotes the penalty hyperparameter in Eq. (6), so the theorem cannot choose it after π is fixed. No explicit dependence on horizon, discount, reward range, or model-estimation error is given, and no proof appears in the text or appendix. The abstract's claim that the paper theoretically proves performance guarantees is therefore unsupported.
  2. [Section 3.1, Eq. (5)] The transition function is defined to be identically zero for all next states when U_t = 1, so \hat P does not integrate to 1 and the α-pessimistic MDP is not an MDP. Consequently, "the policy learned from the α-pessimistic MDP" is not well-defined. If truncation is meant to terminate the episode, this needs an explicit absorbing state or terminal transition, and the theorem's value functions need to account for it. The notation is also inconsistent: the left side conditions on s′, while the right side uses s_t.
  3. [Section 3.1, Eq. (3) and following paragraph] Definition 1 defines Γ using total variation distance DIST(P, \hat P), which requires the true dynamics P, but Section 3.1 then replaces DIST with VAE reconstruction error and sets α to the maximum reconstruction error on the offline dataset. No bound or calibration connects VAE reconstruction error to total variation distance in the continuous-control environments used, so Theorem 1, even if it were valid, would not apply to the implemented algorithm. Since the same fitted VAE is used both to fix α and to score generated trajectories, the reliability classification is conditional on the quality of this proxy, which is not established. In addition, the first equality in Eq. (3) conditions on trajectory history while the summand uses P(·|s_i,a_i), so the definition needs to be stated consistently for history-dependent transition models.
minor comments (6)
  1. [Section 3.1, Eq. (6)] Eq. (6) has an unmatched parenthesis and should clarify the intended range of the penalty; as written, \hat R(s_t,a_t) = r(s_t,a_t) − β Γ(s_t,τ_{<t})/α) is malformed.
  2. [Algorithm 1] Line 7 contains the duplicated word "for for," and line 11 is an incomplete sentence ("Use V AE to calculate..."); both should be corrected.
  3. [Figures and text] Figure 1 uses "TUAU" where the text refers to TATU, and Section 4.3 contains "donoted" instead of "denoted"; these typos should be fixed.
  4. [Section 3.1] The phrase "strict performance lower bound" is confusing because Theorem 1 is an upper bound on the absolute value difference between the two value functions.
  5. [Abstract and Section 3.1] The abstract's "weighted variational distance" is not the same as the total variation distance in Eq. (3) or the VAE reconstruction loss; the terminology should be harmonized so that the reliability metric is precisely defined.
  6. [Table 1] Table 1 provides error bars only for RT+BCQ and not for the baseline methods, which makes it difficult to assess whether the reported improvements are significant.

Circularity Check

2 steps flagged · score 6.0 of 10

The O(alpha beta) performance bound is existentially vacuous, and the reliability truncation is a fitted VAE reconstruction threshold, so the paper's two central guarantees reduce to tautological or self-referential quantities.

  1. other [Section 3.1, Theorem 1]
    "Theorem 1. Let π be the policy learned from the α-pessimistic MDP , and π∗ be the optimal policy in the true MDP M. Then there exist constants C and β such that the performance difference between the two policies in the true MDP satisfies: |V P (π∗) − V P (π)| ≤ C α β, where V P (π) denotes the value function of the policy π in the real M."

    The theorem asserts existence of C and β without specifying their dependence on horizon, discount, reward range, or the policy pair. For any fixed αβ > 0 and bounded value functions, the inequality is satisfied by taking C = |V^P(π*) − V^P(π)| / (αβ). Hence the statement holds for every policy by construction and imposes no constraint; it cannot support the paper's claim that setting α and β controls the performance gap at O(αβ). Moreover, β already appears as the penalty hyperparameter in Eq. (6), so treating it as an existentially chosen constant in the theorem severs the bound from the algorithm it supposedly analyzes.

  2. fitted input called prediction [Section 3.1, after Eq. (7); Algorithm 1 lines 11-16]
    "By learning the V AE, we identify the maximum reconstruction error in the offline data and set it as the threshold α. Next, the generated states and actions use the V AE to calculate the DIST , which is then used as a reliable value for the generated data to compute the cumulative reliability of the trajectory."

    The truncation threshold α is the maximum VAE reconstruction error over the offline dataset, and the same VAE supplies the per-step 'DIST' score for generated (s,a,s') tuples. Thus Eq. (4)'s label Ut=0 ('reliable') is, by construction, equivalent to 'the VAE reconstruction error of the generated tuple is no larger than the largest reconstruction error seen in training.' This is an in-distribution test of the fitted autoencoder, not the total variation distance to the true P required by Definition 1. The paper asserts the substitution without proving that VAE reconstruction error tracks DIST(P, \hat P), so the reliability 'prediction' is a comparison against the fitted VAE's own boundary rather than an externally verified quantity.

full rationale

The paper's derivation chain contains two load-bearing reductions that are empty by construction. First, Theorem 1 is existential in C and β, and the inequality is satisfied for every policy choice by choosing C large enough; the advertised O(αβ) performance guarantee is therefore a tautology rather than a derived bound. Second, the reliability mechanism replaces the unknown DIST(P, \hat P) in Definition 1 with VAE reconstruction error, sets α to the maximum reconstruction error on the offline dataset, and then scores generated trajectories with the same VAE. The truncation rule labels a generated trajectory 'reliable' exactly when its VAE reconstruction error falls inside the training envelope; this is a self-consistency check of the fitted autoencoder, not a measurement of divergence from the true transition distribution, and no calibration links the VAE error to the total-variation α used in the theorem. The empirical D4RL comparisons are external and give the paper independent content, so I do not charge full circularity; however, the two central 'guarantees' — the policy bound and the reliability truncation — reduce respectively to a vacuous inequality and a fitted threshold. In addition, Eq. (5) defines the pessimistic transition as 0 when Ut=1, which is not a valid probability kernel, so the 'α-pessimistic MDP' referenced by Theorem 1 is ill-defined; this further severs the theory from the implemented algorithm. No load-bearing self-citation is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central method rests on a set of fitted thresholds and unproven modeling assumptions: the VAE proxy for total variation distance, the deterministic and fully observable environment assumptions, and the hand-chosen penalty beta. No independent evidence is supplied for the proxy.

free parameters (3)
  • alpha (cumulative reliability threshold) = not reported
    Set to the maximum VAE reconstruction error over the offline dataset, so the truncation criterion is calibrated to the same data it filters.
  • beta (reliability penalty) = not reported
    Hand-chosen penalty in the pessimistic reward Eq (6); the claimed bound is linear in alpha*beta but no value or sensitivity analysis is given.
  • high-reward classifier threshold for H_t = not reported
    The paper defines a binary high-reward event H_t but does not specify the reward threshold or classifier training objective; this determines which candidate actions are sampled.
assumptions (4)
  • domain assumption The transition function of the environment is deterministic.
    Stated at the end of Section 2: 'we assume that the transition function of the environment is determined and propose RT to augment the offline data.'
  • domain assumption The state space is fully observable.
    The conclusion notes: 'RT presupposes a fully observable state space in its empirical evaluations.'
  • ad hoc to paper VAE reconstruction error is a valid proxy for the total variation distance between true and estimated transition distributions.
    Section 3.1 replaces DIST(P, hatP) with VAE reconstruction error without validation; the reliability metric and truncation rule depend on this identification.
  • ad hoc to paper The attention-weighted cumulative VAE error is a valid measure of accumulated trajectory generation error.
    Equation (3) defines cumulative reliability as an attention-weighted sum of per-step distances, but no argument shows this weighted sum is the right accumulation rule for truncation decisions.
invented entities (2)
  • alpha-pessimistic MDP (hatM_p)
    purpose: Formal object used to derive a performance bound by truncating unreliable transitions and penalizing rewards.
    A mathematical construction defined in Section 3.1; it has no observable counterpart and cannot be falsified outside the paper.
  • High-reward indicator H_t
    purpose: Latent binary variable used to condition generation on actions that produce high rewards.
    The paper invokes a binary classifier P(H_t|...) but gives no training objective, reward threshold, or external validation, so H_t has no independent falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modeling." pith.science (2026). https://pith.science/paper/MADJDFNL

@misc{pith2026250206491,
  author       = {Pith},
  title        = {Pith review of: Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MADJDFNL}},
  note         = {Machine review of arXiv:2502.06491}
}
abstract

Model-based offline reinforcement learning (MORL) aims to learn a policy by exploiting a dynamics model derived from an existing dataset. Applying conservative quantification to the dynamics model, most existing works on MORL generate trajectories that approximate the real data distribution to facilitate policy learning by using current information (e.g., the state and action at time step $t$). However, these works neglect the impact of historical information on environmental dynamics, leading to the generation of unreliable trajectories that may not align with the real data distribution. In this paper, we propose a new MORL algorithm \textbf{R}eliability-guaranteed \textbf{T}ransformer (RT), which can eliminate unreliable trajectories by calculating the cumulative reliability of the generated trajectory (i.e., using a weighted variational distance away from the real data). Moreover, by sampling candidate actions with high rewards, RT can efficiently generate high-return trajectories from the existing offline data. We theoretically prove the performance guarantees of RT in policy learning, and empirically demonstrate its effectiveness against state-of-the-art model-based methods on several benchmark tasks.

Figures

Figures reproduced from arXiv: 2502.06491 by the authors.

Figure 1
Figure 1. Performance of RT, ROMI, CABI, TATU and DStitch combined with different model-free algorithms (averaged over 10 random [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. Visualization of generated data. (a) represents the distribu [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. The impact of different truncation mechanisms on the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 37 canonical work pages

  1. [1]

    [Asadi et al., 2018] Kavosh Asadi, Dipendra Misra, and Michael L. Littman. Lipschitz continuity in model-based reinforcement learning. In ICML, volume 80 of Pro- ceedings of Machine Learning Research , pages 264–273. PMLR,

    work page 2018

  2. [5]

    De- cision transformer: Reinforcement learning via sequence modeling

    [Chen et al., 2021] Lili Chen, Kevin Lu, Aravind Ra- jeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. De- cision transformer: Reinforcement learning via sequence modeling. In NeurIPS, pages 15084–15097,

    work page 2021

  3. [7]

    UMBRELLA: uncertainty-aware model-based offline re- inforcement learning leveraging planning

    [Diehl et al., 2021] Christopher Diehl, Timo Sievernich, Martin Kr ¨uger, Frank Hoffmann, and Torsten Bertram. UMBRELLA: uncertainty-aware model-based offline re- inforcement learning leveraging planning. CoRR, abs/2111.11097,

    arXiv 2021

  4. [8]

    D4RL: datasets for deep data-driven reinforcement learning

    [Fu et al., 2020] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219,

    arXiv 2020

  5. [11]

    Bridging the data gap between training and inference for unsupervised neural machine translation

    [He et al., 2022] Zhiwei He, Xing Wang, Rui Wang, Shum- ing Shi, and Zhaopeng Tu. Bridging the data gap between training and inference for unsupervised neural machine translation. In ACL, pages 6611–6623. Association for Computational Linguistics,

    work page 2022

  6. [14]

    Offline reinforcement learning as one big se- quence modeling problem

    [Janner et al., 2021] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big se- quence modeling problem. In NeurIPS, pages 1273–1286,

    work page 2021

  7. [15]

    Varshney, Caiming Xiong, and Richard Socher

    [Keskar et al., 2019] Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. CTRL: A conditional transformer language model for con- trollable generation. CoRR, abs/1909.05858,

    arXiv 2019

  8. [16]

    Morel: Model-based offline reinforcement learning

    [Kidambi et al., 2020] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In NeurIPS,

    work page 2020

Show all 41 references
  1. [17]

    Offline reinforcement learning with im- plicit q-learning

    [Kostrikov et al., 2022] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with im- plicit q-learning. In ICLR. OpenReview.net,

    2022

  2. [18]

    Conservative q-learning for offline reinforcement learning

    [Kumar et al., 2020] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In NeurIPS,

    2020

  3. [19]

    Multi-game decision transformers

    [Lee et al., 2022] Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, and Igor Mordatch. Multi-game decision transformers. In NeurIPS,

    2022

  4. [20]

    Distribution- conditioned adversarial variational autoencoder for valid instrumental variable generation

    [Li and Yao, 2024] Xinshu Li and Lina Yao. Distribution- conditioned adversarial variational autoencoder for valid instrumental variable generation. In AAAI, pages 13664– 13672. AAAI Press,

    2024

  5. [21]

    Diffstitch: Boosting of- fline reinforcement learning with diffusion-based trajec- tory stitching

    [Li et al., 2024] Guanghe Li, Yixiang Shan, Zhengbang Zhu, Ting Long, and Weinan Zhang. Diffstitch: Boosting of- fline reinforcement learning with diffusion-based trajec- tory stitching. In ICML. OpenReview.net,

    2024

  6. [22]

    Ball, Yee Whye Teh, and Jack Parker-Holder

    [Lu et al., 2023] Cong Lu, Philip J. Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay. In NeurIPS,

    2023

  7. [23]

    Luis, Alessandro G

    [Luis et al., 2023] Carlos E. Luis, Alessandro G. Bottero, Julia Vinogradska, Felix Berkenkamp, and Jan Peters. Model-based uncertainty in value functions. In AAAI, vol- ume 206 of Proceedings of Machine Learning Research , pages 8029–8052. PMLR,

    2023

  8. [24]

    Double check your state before trusting it: Confidence- aware bidirectional offline model-based imagination

    [Lyu et al., 2022] Jiafei Lyu, Xiu Li, and Zongqing Lu. Double check your state before trusting it: Confidence- aware bidirectional offline model-based imagination. In NeurIPS,

    2022

  9. [25]

    Reining generalization in offline re- inforcement learning via representation distinction

    [Ma et al., 2023] Yi Ma, Hongyao Tang, Dong Li, and Zhaopeng Meng. Reining generalization in offline re- inforcement learning via representation distinction. In NeurIPS,

    2023

  10. [26]

    Offline imitation learn- ing with model-based reverse augmentation

    [Shao et al., 2024] Jie-Jing Shao, Hao-Sen Shi, Lan-Zhe Guo, and Yu-Feng Li. Offline imitation learn- ing with model-based reverse augmentation. CoRR, abs/2406.12550,

    2024 arXiv

  11. [27]

    Model-bellman in- consistency for model-based offline reinforcement learn- ing

    [Sun et al., 2023] Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-bellman in- consistency for model-based offline reinforcement learn- ing. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 33177–33194. PMLR,

    2023

  12. [28]

    [Swazinna et al., 2021] Phillip Swazinna, Steffen Udluft, and Thomas A. Runkler. Overcoming model bias for ro- bust offline deep reinforcement learning. Eng. Appl. Artif. Intell., 104:104366,

    2021

  13. [29]

    Self-correcting models for model-based reinforcement learning

    [Talvitie, 2017] Erik Talvitie. Self-correcting models for model-based reinforcement learning. In Satinder Singh and Shaul Markovitch, editors, AAAI, pages 2597–2603. AAAI Press,

    2017

  14. [31]

    Visualizing data using t-sne.Journal of machine learning research, 9(11),

    [Van der Maaten and Hinton, 2008] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine learning research, 9(11),

    2008

  15. [33]

    Offline reinforcement learning with reverse model-based imagination

    [Wang et al., 2021] Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline reinforcement learning with reverse model-based imagination. In NeurIPS, pages 29420–29432,

    2021

  16. [34]

    Q-learning decision transformer: Leveraging dynamic programming for conditional se- quence modelling in offline RL

    [Yamagata et al., 2023] Taku Yamagata, Ahmed Khalil, and Ra´ul Santos-Rodr´ıguez. Q-learning decision transformer: Leveraging dynamic programming for conditional se- quence modelling in offline RL. In ICML, volume 202 of Proceedings of Machine Learning Research , pages 38989–3...

    2023

  17. [35]

    Pareto policy pool for model-based offline reinforcement learning

    [Yang et al., 2022] Yijun Yang, Jing Jiang, Tianyi Zhou, Jie Ma, and Yuhui Shi. Pareto policy pool for model-based offline reinforcement learning. In ICLR. OpenReview.net,

    2022

  18. [36]

    Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma

    [Yu et al., 2020] Tianhe Yu, Garrett Thomas, Lantao Yu, Ste- fano Ermon, James Y . Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy opti- mization. In NeurIPS,

    2020

  19. [37]

    Combo: Conservative offline model-based policy opti- mization

    [Yu et al., 2021] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy opti- mization. Advances in neural information processing sys- tems, 34:28954–28967,

    2021

  20. [38]

    Model-based offline planning with trajectory prun- ing

    [Zhan et al., 2022] Xianyuan Zhan, Xiangyu Zhu, and Hao- ran Xu. Model-based offline planning with trajectory prun- ing. In Luc De Raedt, editor, IJCAI, pages 3716–3722. ijcai.org,

    2022

  21. [39]

    Uncertainty-driven trajectory truncation for data augmen- tation in offline reinforcement learning

    [Zhang et al., 2023] Junjie Zhang, Jiafei Lyu, Xiaoteng Ma, Jiangpeng Yan, Jun Yang, Le Wan, and Xiu Li. Uncertainty-driven trajectory truncation for data augmen- tation in offline reinforcement learning. In ECAI, volume 372 of Frontiers in Artificial Intelligence and Applicat...

    2023

  22. [40]

    Conditional vari- ational autoencoder for sign language translation with cross-modal alignment

    [Zhao et al., 2024] Rui Zhao, Liang Zhang, Biao Fu, Cong Hu, Jinsong Su, and Yidong Chen. Conditional vari- ational autoencoder for sign language translation with cross-modal alignment. In AAAI, pages 19643–19651. AAAI Press,

    2024

  23. [41]

    Is model ensemble necessary? model- based RL via a single model with lipschitz regularized value function

    [Zheng et al., 2023] Ruijie Zheng, Xiyao Wang, Huazhe Xu, and Furong Huang. Is model ensemble necessary? model- based RL via a single model with lipschitz regularized value function. In ICLR. OpenReview.net, 2023

    2023

  24. [2008]

    Jamieson

    [Wagenmaker et al., 2023] Andrew Wagenmaker, Guanya Shi, and Kevin G. Jamieson. Optimal exploration for model-based RL in nonlinear systems. In NeurIPS,

    2023

  25. [2010]

    When to trust your model: Model-based policy optimization

    [Janner et al., 2019] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In NeurIPS, pages 12498–12509,

    2019

  26. [2017]

    Behavioral cloning from observation

    [Torabi et al., 2018] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In IJCAI, pages 4950–4957. ijcai.org,

    2018

  27. [2018]

    Waypoint trans- former: Reinforcement learning via supervised learning with intermediate targets

    [Badrinath et al., 2023] Anirudhan Badrinath, Yannis Flet- Berliac, Allen Nie, and Emma Brunskill. Waypoint trans- former: Reinforcement learning via supervised learning with intermediate targets. In NeurIPS,

    2023

  28. [2019]

    ACT: empowering decision transformer with dynamic program- ming via advantage conditioning

    [Gao et al., 2024] Chenxiao Gao, Chenyang Wu, Mingjun Cao, Rui Kong, Zongzhang Zhang, and Yang Yu. ACT: empowering decision transformer with dynamic program- ming via advantage conditioning. In AAAI, pages 12127– 12135. AAAI Press,

    2024

  29. [2020]

    Off-policy deep reinforcement learning without exploration

    [Fujimoto et al., 2019] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICML, volume 97 of Proceedings of Machine Learning Research, pages 2052–2062. PMLR,

    2019

  30. [2021]

    Lapo: Latent-variable advantage-weighted policy optimization for offline rein- forcement learning

    [Chen et al., 2022] Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang, Alex Yuan Gao, Wenzhe Li, Liang Bin, Chelsea Finn, and Chongjie Zhang. Lapo: Latent-variable advantage-weighted policy optimization for offline rein- forcement learning. Advances in Neural Information Pro- ...

    2022

  31. [2022]

    [Ho and Yeung, 2010] Siu-Wai Ho and Raymond W. Yeung. The interplay between entropy and variational distance. IEEE Trans. Inf. Theory, 56(12):5906–5929,

    2010

  32. [2023]

    Data-efficient task generalization via probabilistic model-based meta reinforcement learn- ing

    [Bhardwaj et al., 2024] Arjun Bhardwaj, Jonas Rothfuss, Bhavya Sukhija, Yarden As, Marco Hutter, Stelian Coros, and Andreas Krause. Data-efficient task generalization via probabilistic model-based meta reinforcement learn- ing. IEEE Robotics Autom. Lett., 9(4):3918–3925,

    2024

  33. [2024]

    Blanchet, Miao Lu, Tong Zhang, and Han Zhong

    [Blanchet et al., 2023] Jose H. Blanchet, Miao Lu, Tong Zhang, and Han Zhong. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. In NeurIPS,

    2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.