REVIEW 5 major objections 6 minor 16 references
TransDreamerV3: Implanting Transformer In DreamerV3
T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read By replacing DreamerV3's GRU core with a transformer encoder, TransDreamerV3 claims to beat the original on Freeway, Pong, and Crafter.
desk verdict A clearly-written student paper swapping GRU for a naive transformer in DreamerV3, but single-run results at mismatched training steps don't support the headline claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the naive transformer state-space model (TSSM) used as the deterministic state model in DreamerV3. It replaces the gated recurrent unit with a transformer encoder that takes the sequence of past stochastic representation states and actions as input, independent of prior deterministic states. The representation model's belief omits the deterministic state, imagined trajectories are capped at three per training sample with replay prioritized by reward, and the transformer parameters are frozen while the policy trains. This manually constructed attention mechanism omits positional encoding and dropout, applies layer normalization after attention and feed-forward steps, and is what carries the claimed memory advantage.
What would settle it
Train DreamerV3 and TransDreamerV3 on Freeway, Pong, and Crafter with identical step counts and at least three seeds each, then compare final returns; if DreamerV3 matches or beats TransDreamerV3 when training time is equalized, the paper's central claim is refuted.
Extended reading notes
Core claim
The paper claims that swapping the GRU in DreamerV3's RSSM for a transformer encoder, the TSSM-style deterministic state model used by TransDreamer, yields a world-model agent that surpasses DreamerV3 in every environment tested except boxing. Freeway is the strongest result: DreamerV3 needs about 12 million steps to get non-zero reward, while TransDreamerV3 achieves it in about 3 million steps. In Crafter, TransDreamerV3 clearly outperforms DreamerV3, and in Pong it overtakes DreamerV3 after 2.3 million steps. The authors attribute the improvement to the attention mechanism's direct access to past representation states and actions, as opposed to the GRU's compressed hidden state.
Load-bearing premise
The load-bearing premise is that comparing models at different training-step counts still counts as a fair comparison; if the performance differences come from training budget or seed noise rather than the transformer architecture, the central claim collapses.
Editorial extensions
If this is right
- If the result holds, attention-based world models can match or beat recurrent ones on Atari and Crafter even with a deliberately minimal transformer implementation.
- Freeway's earlier non-zero reward suggests the transformer backbone needs fewer environment interactions to discover reward, which would matter for sample efficiency.
- Crafter's large gap indicates the benefit may grow in tasks that require remembering and composing skills over longer horizons.
- The fact that TransDreamer still beats TransDreamerV3 suggests that full use of past context, not just the presence of a transformer, is what unlocks the larger gains.
Reading between the lines
- A natural testable extension is ablating the naive transformer's missing pieces (positional encoding, dropout, attention to all past states) one at a time to isolate which omission costs the most.
- Because the comparison is at different training-step counts, the Freeway result would be stronger if re-run with matched step counts and multiple seeds; that is our inference, not the paper's.
- The frozen-transformer policy training hints at a possible decoupling: the world model can act as a fixed feature extractor while the actor-critic learns, which could simplify future architectures.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes TransDreamerV3, an RL agent that replaces the GRU in DreamerV3's RSSM with a naive transformer encoder. The authors report experiments on Atari-Boxing, Atari-Freeway, Atari-Pong, and Crafter, and claim that TransDreamerV3 outperforms DreamerV3 in all tested tasks except Boxing. They also report that Minecraft experiments failed due to engineering issues. The paper provides a GitHub repository with code and a model checkpoint. The central claim is that the attention mechanism provides advantages over the GRU's gating in the world model.
Significance. If the empirical claim were established, the paper would provide a useful data point on replacing recurrent state-space models with transformer state-space models inside DreamerV3. The authors are honest about limitations: they state that Minecraft was not completed and that comparisons were made at different training steps. The release of code and a checkpoint is a positive contribution. However, the significance is limited by the lack of any theoretical or rigorous empirical support; there are no seeds, error bars, ablations, or matched training budgets. The architectural contribution is a direct adaptation of existing TSSM ideas from TransDreamer, and the modifications are not isolated. Consequently, the paper's contribution is mainly an engineering report rather than a validated research result.
major comments (5)
- [Section 4.1] The paper's central claim that TransDreamerV3 surpasses DreamerV3 is not supported by the reported experiments. The text explicitly says 'Due to time and resource constraints, we compare performances at different training steps.' For Freeway, DreamerV3 achieves nonzero reward at 12 million steps while TransDreamerV3 achieves it at 3 million steps; comparing these endpoints cannot establish superiority because DreamerV3 may overtake at a later step. For Pong, the paper states that the TransDreamer baseline was not trained up to 2.3 million steps, so the comparison is incomplete. No seeds or error bars are reported for any task. The statement that 'within each task, the comparison is fair as all models are trained under the same settings' does not address the step-count mismatch; same settings do not imply the same number of environment steps or the same seeds. A matched-step, multi-seed comparison with error bars is required to support the performance claim.
- [Section 3.2 and Section 3.3] The transformer implementation omits positional encoding while the model is defined as h_t = f_transformer(z_{1:t-1}, a_{1:t-1}) in Section 3.2. Without positional encoding, the attention operation is permutation-invariant, so the model cannot distinguish the order of past states and actions. This contradicts the paper's motivation that the transformer provides long-term memory and improved temporal reasoning. The implementation section says positional encoding is omitted 'due to only considering a the prior state,' which is inconsistent with processing a sequence of past states. The authors must either add positional encoding or clarify that the model only uses the immediately previous state; in the latter case, the claim that the attention mechanism improves over the GRU's gated mechanism is unfounded.
- [Section 4.1, final paragraph] The sentence 'In every tested environment except boxing, our model, employing a naive transformer, surpasses DreamerV3 in performance' is not supported by the data described in the same section. For Boxing, the paper reports that TransDreamer (not TransDreamerV3) exhibited superior performance, and no TransDreamerV3 result for Boxing is provided. For Pong, the comparison is made at 2.3 million steps even though the TransDreamer baseline was not trained to that point, and for Freeway the models reach nonzero reward at different step counts. The conclusion overreaches the experimental results presented.
- [Section 4.2] The Minecraft experiment, which is a central motivation in the introduction and related work via the diamond challenge, failed and produced no results. The paper states that evaluation code issues prevented comparison with DreamerV3. The claim that TransDreamerV3 would perform the Minecraft Diamond task faster and more efficiently is therefore untested. The paper should either present Minecraft results or remove this claim from the introduction and conclusion.
- [Section 3.2] The paper attributes the observed performance differences to the transformer replacing the GRU, but multiple changes are made simultaneously. In addition to the transformer, the authors limit imagined trajectories to three per training sample and change replay buffer prioritization to favor higher-reward trajectories (Section 3.2). These changes are not ablated, so even if the matched-step multi-seed comparison showed an improvement, it could not be attributed specifically to the attention mechanism. An ablation of each modification is needed to support the paper's architectural claims.
minor comments (6)
- [Throughout] The term 'naive transformer' is used in the abstract and introduction but is only defined in Section 3.3; define it at first use.
- [Section 3.2] There is a typo: 'repay buffer' should be 'replay buffer.'
- [Section 3.3] The phrase 'due to only considering a the prior state' contains a typo and is ambiguous; clarify what input the transformer actually receives.
- [Figure 3] Figure 3 is referenced in Section 4 but does not appear in the provided text; ensure it is included and that axes are labeled with environment steps and mean reward.
- [Section 2.1] In Section 2.1, reference [7] is cited for DreamerV3, but in the bibliography [7] is the DreamerV2 paper and [8] is DreamerV3; correct the citation.
- [Section 6] The contribution section stating 'We have completed 100% of this project without any outside help or guidance' is not appropriate for a journal article and should be removed or rewritten.
Circularity Check
No circularity: the empirical claim rests on external baselines and prior architectural components, not on fitted targets or self-citations.
full rationale
The paper's central claim is an empirical comparison of TransDreamerV3 against DreamerV3 and TransDreamer on Atari and Crafter tasks. No parameter is fitted to a target outcome, no equation defines the reported performance in terms of the architecture being tested, and no uniqueness theorem or load-bearing self-citation is invoked. The TSSM backbone is adopted from TransDreamer [2] as prior work, and the comparison baseline DreamerV3 [8] is external; the paper's own contribution is an implementation and a benchmark comparison rather than a derivation. The paper explicitly acknowledges a limitation: 'Due to time and resource constraints, we compare performances at different training steps.' This is a genuine validity concern because unmatched step counts or seed noise could confound the comparison, but it is not a circularity mechanism: the outcome is not built into the inputs by construction. The reader's circularity score of 0 is therefore appropriate; the derivation chain is self-contained in the sense that the empirical result is not equivalent to any fitted parameter or self-citation.
Assumptions & free parameters
free parameters (1)
- imagination rollout count =
3
assumptions (4)
- domain assumption The representation model q(z_t|x_t), without the deterministic state h_t, performs similarly to the Dreamer representation model as shown in TransDreamer [2].
- ad hoc to paper A transformer without positional encoding can represent temporal order when it processes only the prior state sequence.
- ad hoc to paper Freezing transformer parameters during policy learning does not reduce world model quality.
- domain assumption DreamerV3 hyperparameters remain appropriate when RSSM is replaced by TSSM.
Cite this review
Pith. "Pith review of TransDreamerV3: Implanting Transformer In DreamerV3." pith.science (2026). https://pith.science/paper/IK3VTBG7
@misc{pith2026250617103,
author = {Pith},
title = {Pith review of: TransDreamerV3: Implanting Transformer In DreamerV3},
year = {2026},
howpublished = {\url{https://pith.science/paper/IK3VTBG7}},
note = {Machine review of arXiv:2506.17103}
}
read the original abstract
This paper introduces TransDreamerV3, a reinforcement learning model that enhances the DreamerV3 architecture by integrating a transformer encoder. The model is designed to improve memory and decision-making capabilities in complex environments. We conducted experiments on Atari-Boxing, Atari-Freeway, Atari-Pong, and Crafter tasks, where TransDreamerV3 demonstrated improved performance over DreamerV3, particularly in the Atari-Freeway and Crafter tasks. While issues in the Minecraft task and limited training across all tasks were noted, TransDreamerV3 displays advancement in world model-based reinforcement learning, leveraging transformer architectures.
Figures
Reference graph
Works this paper leans on
-
[1]
Andrea Banino, Adrià Puigdomènech Badia, Raphael Köster, Martin J. Chadwick, Vinicius Zambaldi, Demis Hassabis, Caswell Barry, Matthew Botvinick, Dharshan Kumaran, and Charles Blundell. Memo: A deep network for flexible combination of episodic memories, 2020
work page 2020
-
[2]
Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Transdreamer: Reinforcement learning with transformer world models.arXiv preprint arXiv:2202.09481, 2022
arXiv 2022
-
[3]
Decision transformer: Reinforcement learning via sequence modeling.Advances in neural information processing systems, 34:15084–15097, 2021
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling.Advances in neural information processing systems, 34:15084–15097, 2021
2021
-
[4]
World models.ArXiv, abs/1803.10122, 2018
David R Ha and Jürgen Schmidhuber. World models.ArXiv, abs/1803.10122, 2018
arXiv 2018
-
[5]
Lillicrap, Jimmy Ba, and Mohammad Norouzi
Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination.ArXiv, abs/1912.01603, 2019
arXiv 1912
-
[6]
Danijar Hafner, Timothy P. Lillicrap, Ian S. Fischer, Ruben Villegas, David R Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels.ArXiv, abs/1811.04551, 2018
arXiv 2018
-
[7]
Lillicrap, Mohammad Norouzi, and Jimmy Ba
Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models.ArXiv, abs/2010.02193, 2020
arXiv 2010
-
[8]
Mastering diverse domains through world models.arXiv preprint arXiv:2301.04104, 2023
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models.arXiv preprint arXiv:2301.04104, 2023
arXiv 2023
Show all 16 references
-
[9]
Going beyond linear transformers with recurrent fast weight programmers.Advances in Neural Information Processing Systems, 34:7703– 7717, 2021
Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond linear transformers with recurrent fast weight programmers.Advances in Neural Information Processing Systems, 34:7703– 7717, 2021
2021
-
[10]
Offline reinforcement learning as one big sequence modeling problem.Advances in neural information processing systems, 34:1273–1286, 2021
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem.Advances in neural information processing systems, 34:1273–1286, 2021
2021
-
[11]
Transformers are sample efficient world models
Vincent Micheli, Eloi Alonso, and Franccois Fleuret. Transformers are sample efficient world models. ArXiv, abs/2209.00588, 2022
2022 arXiv
-
[12]
Stabilizing transformers for reinforce- ment learning
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforce- ment learning. InInternational conference on machine learning, pages 7487...
2020
-
[13]
Rapid task-solving in novel environments, 2021
Sam Ritter, Ryan Faulkner, Laurent Sartran, Adam Santoro, Matt Botvinick, and David Raposo. Rapid task-solving in novel environments, 2021
2021
-
[14]
Pearson, 4 edition, 2020
Peter Norvig Stuart Russell.Artificial Intelligence: A Modern Approach. Pearson, 4 edition, 2020
2020
-
[15]
Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeural Information Processing Systems, 2017
2017
-
[16]
Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, and P. Abbeel. Daydreamer: World models for physical robot learning. InConference on Robot Learning, 2022. 7
2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.