Pith. sign in

REVIEW 3 major objections 5 minor 61 references

Improving Transformer World Models for Data-Efficient RL

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A transformer world-model agent reaches 69.66% reward on Craftax-classic after 1M steps, topping the 65% human expert mark, via three targeted changes to how the model is tokenized, trained, and used.

desk verdict Solid empirical ladder for transformer world models, but the 'superhuman' claim rests on comparing a health-free success-rate metric to prior rewards that include health. read the letter →

arxiv 2502.01591 v3 pith:D55O2SY2 submitted 2025-02-03 cs.LG cs.AI

Antoine Dedieu , Joseph Ortiz , Xinghua Lou , Carter Wendelken , Wolfgang Lehrach , J Swaroop Guntupalli , Miguel Lazaro-Gredilla , Kevin Patrick Murphy This is my paper · ORCID
classification cs.LGcs.AI
keywords model-basedreinforcementlearningtransformerworldmodelsdata-efficientRLCraftax-classicnearest-neighbortokenizationblockteacherforcingDynawithwarmupsampleefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Training a transformer world model to generate imaginary experience for a reinforcement learning policy usually suffers from two problems: the tokenizer's vocabulary keeps changing during learning, and the model must predict long token sequences one by one, which compounds errors. The paper claims both problems have targeted fixes: a static nearest-neighbor tokenizer over image patches keeps the code vocabulary fixed so the world model learns a stationary mapping; block teacher forcing decodes all tokens of the next timestep in parallel using a block-causal attention mask; and a Dyna-with-warmup schedule trains the policy on real experience first, adding imagined rollouts only after the world model has been trained long enough to be useful. On Craftax-classic, these improvements compound, raising reward from 31.93% for the base transformer-world-model agent to 69.66% after 1M environment steps, past the 65% human expert score that the paper says no prior agent had beaten. The same ladder transfers, with modest adaptations, to Craftax-full, MinAtar, and two-player OpenSpiel games, which the paper offers as evidence that the recipe is general rather than benchmark-specific.

What carries the argument

The load-bearing pieces are three named mechanisms. Nearest-neighbor tokenizer (NNT): each non-overlapping image patch is mapped to its closest code in a codebook of previously seen patches whenever the Euclidean distance is below a threshold $\tau=0.75$; otherwise the patch itself is appended as a new code. Because codes are never updated once added, the token targets presented to the transformer world model are stationary, whereas a VQ-VAE codebook keeps shifting during training. Block teacher forcing (BTF): the world model is trained with a block-causal attention mask and supervision that predicts all $L$ tokens of the next timestep in parallel, conditioned only on the history and the action, so the model reasons jointly about an entire future frame before sampling and avoids autoregressive drift; it also roughly halves rollout time. Dyna with warmup: the Proximal Policy Optimization (PPO) policy is updated immediately on real trajectories, and imaginary rollouts from the world model are mixed in only after a warmup period of $T_{\mathrm{BP}}=200{,}000$ environment steps, so an immature world model cannot corrupt policy learning; starting imagination at step 0 instead drops reward from 67.42% to 33.54%.

What would settle it

Run the same human-expert protocol used for Crafter (a small panel of expert players completing 100 episodes) directly on Craftax-classic and average their rewards; if that average exceeds 69.66%, then the claim of being the first agent to surpass human expert reward on this benchmark is falsified, even though the agent-to-agent state of the art could still stand.

Watch

Extended reading notes

Core claim

The central claim is that a transformer world model can be turned into a highly data-efficient reinforcement learning engine if three complementary design choices are made on top of a solid model-free baseline: train the policy on both real and imagined trajectories but delay imagination until the model is trained (Dyna with warmup); tokenize image patches with a static nearest-neighbor codebook instead of a continuously updated VQ-VAE (vector-quantized variational autoencoder); and train the world model with block teacher forcing, predicting all tokens of the next frame in parallel. Each rung of the resulting ladder contributes a clear jump in reward on Craftax-classic — from 31.93% (baseline) to 43.36% (Dyna), 58.92% (patch tokenization), 64.96% (nearest-neighbor codebook), and finally 67.42% with block teacher forcing, or 69.66% with additional world-model training. The paper reports this best agent as a new state of the art, surpassing the prior best 53.2% (DreamerV3) and exceeding the 65.0% human expert reward (a statistic carried from the near-replica Crafter benchmark) for the first time, with a score of 31.77% that still trails the human expert score of 50.5%.

Load-bearing premise

The 'first time above human expert' headline assumes that the 65% human reward measured on the original Crafter benchmark applies to the near-replica Craftax-classic environment, an assumption the paper itself flags with an asterisk stating the number 'may not exactly match Craftax-classic.'

Editorial extensions

If this is right

  • Removing any single rung of the ladder hurts: dropping Dyna lowers reward to 55.02%, dropping NNT to 60.66%, dropping both NNT and patches to 45.86%, and dropping BTF to 64.96%.
  • Starting imagination at step 0 instead of after the warmup collapses reward from 67.42% to 33.54%, making the warmup load-bearing rather than cosmetic.
  • Block teacher forcing is roughly twice as fast as autoregressive rollout while being more accurate, improving both the reward and the compute budget of imagination.
  • The improved model-free baseline (a CNN+GRU policy trained with PPO) alone reaches 55.49% on Craftax-classic, above DreamerV3's reported 53.2%, so the MBRL gains stack on top of a competitive model-free agent.
  • The same ladder of improvements also improves results on MinAtar, Craftax-full, and three two-player OpenSpiel games, supporting generality beyond Craftax-classic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension is to measure human expert performance on Craftax-classic using the original Crafter protocol. The headline comparison leans on a 65% human number carried over from the near-replica Crafter benchmark, a caveat the paper itself asterisks; if the true human value differs materially, the 'first time above human' framing may not survive even though the agent-to-agent state
  • The paper's own rerun of DreamerV3 gives 47.18% rather than the cited 53.2%, which the paper attributes to hyperparameter discrepancies, so the claimed margin over the prior state of the art depends on which number one trusts; an independent rerun of DreamerV3 under the paper's exact protocol would settle the comparison.
  • The stationary-codebook idea should transfer beyond grid games: any world model whose observation tokenizer keeps shifting forces the dynamics model to chase a moving target. Applying the NNT-style static codebook to continuous-control or Atari environments would separate the benefit of patch alignment, which the paper shows matters (5×5 and 9×9 patches are worse than 7×7), from the benefit of a f
  • Block teacher forcing's factor-of-two speedup suggests that parallel frame decoding could make imagined rollouts cheap enough for decision-time planners such as tree search, not just background planners, which is an extension the paper does not pursue.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes three modifications to transformer-based world models for model-based RL: (a) Dyna with warmup, training the policy on real and imagined data only after the world model has been trained for a number of steps; (b) a patch-based nearest-neighbor tokenizer (NNT) with a static, growing codebook; and (c) block teacher forcing (BTF), a parallel next-timestep token prediction objective with block-causal attention. The authors evaluate these contributions on Craftax-classic and report a reward of 69.66% and a score of 31.77% after 1M environment steps, claiming to surpass DreamerV3 (53.2%) and human expert performance (65.0%) for the first time. They also include ablations, an external evaluation of TWM rollout quality via a symbol extractor, and additional experiments on Craftax-full, MinAtar, and three OpenSpiel games.

Significance. The three proposed components are clearly motivated and the paper is generally well designed: the ladder of improvements is evaluated with 10 seeds and standard errors, each rung is ablated, and the TWM rollouts are assessed with a separately trained symbol classifier on held-out trajectories rather than only by proxy image error. If the reported results are reproducible, the methodological contributions are likely useful to the MBRL community. However, the headline claims of state-of-the-art and superhuman performance rest on a reward metric that excludes health rewards and on human data collected in a different (though near-identical) environment, so those claims are not yet demonstrated on a common basis.

major comments (3)
  1. [§1, footnote 1; Table 1; §A.2.2] The paper defines its reward as R = (1/N) Σ_i s_i, the arithmetic mean of per-achievement success rates, explicitly "ignoring minor contributions to the reward based on the health of the agent" (footnote 1), and Appendix A.2.2 states that the world model uses a binary reward target that ignores damage and healing. However, the comparison numbers in Table 1 and the abstract (DreamerV3 53.2%, human expert 65.0%, IRIS 25%, Δ-IRIS 35%) are taken from prior work that reports the full environment reward, which includes a -0.1 per damage and +0.1 per healing. These are not the same quantity, and the paper never reports the health component or recalibrates the prior results to a common metric; therefore the statements "new SOTA reward" and "exceeding human performance of 65.0% for the first time" are not supported by the numbers as presented. The authors should report both the achievement-mean and the full environment reward for all methods, or otherwise adjust the comparison to a single consistent metric.
  2. [§4.2; Table 1] The human expert reference values (reward 65.0±10.5, score 50.5±6.8) are measured on Crafter, not on Craftax-classic; the asterisk in Table 1 acknowledges this, but the abstract and Section 4.2 state "for the first time" surpassing human performance without qualification. Because human performance may differ between Crafter and Craftax-classic, the paper should either measure human play on Craftax-classic or explicitly hedge the claim in the abstract and conclusion. At minimum, the unhedged "for the first time" language should be revised to indicate that the comparison is to Crafter-based human data.
  3. [Table 1, 'Our DreamerV3' row] The in-house DreamerV3 run is reported at 47.18±3.88, about six points below the published 53.2% figure, and the paper does not specify whether this run was scored with the paper's achievement-mean reward (excluding health) or with the raw environment reward. Without this information, and without a breakdown of the health component for this run, the comparison to DreamerV3 is ambiguous even on the same benchmark. Please clarify the evaluation protocol for this row.
minor comments (5)
  1. [§3.6, Eq. (3)] Equation (3) defines LBTF as a log-likelihood, but the text does not explicitly state that this is the training objective; please clarify the relationship between this objective and the block-causal attention mask shown in Figure 2.
  2. [Appendix A.2.2] In the list of TWM losses, the item beginning "and (3 The third loss" has a missing opening parenthesis; please fix the typo.
  3. [Appendix E, Table 6] Table 6 lists the Craftax-classic buffer size as 48,000, while Table 5 and Section A.3.4 specify a buffer size of 128,000; please reconcile these values, and check whether the Classic/Full columns are swapped.
  4. [§4.1 bullet list] The bullet list under Section 4.1 repeats "M5: M4 + BTF. Our best MBRL (slow)" twice; the second entry should be reformatted to avoid duplication.
  5. [Figure 6, right panel] The caption refers to "Rollouts v2" without defining the label; please clarify what v2 indicates.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the agent and world model are trained and evaluated against external Craftax benchmarks, and no headline number is fitted and re-predicted.

full rationale

The paper's central claims are empirical results on Craftax-classic, Craftax-full, MinAtar, and OpenSpiel, obtained by training a tokenizer, transformer world model, and PPO policy with explicit objectives (Eqs. 2, 3, 5) and then evaluating in the environment. No step fits a parameter to the headline reward or score and then presents that same fitted quantity as a prediction. The nearest-neighbor tokenizer is static by construction, and the paper explicitly acknowledges that its low reconstruction error follows by construction, so that is an admitted property rather than a circular prediction. The TWM rollout evaluation uses a CNN symbol classifier trained on 100k observation-symbol pairs with 99% validation accuracy, which is an external, independently trained metric. The ablation ladder (M1 through M5) compares genuinely different models, and the Dyna-with-warmup, NNT, and BTF contributions are each evaluated against environment reward. The only self-citation is Murphy (2024), a general RL survey used for background, which is not load-bearing. The human-expert and prior-method numbers carry an asterisk noting they may not exactly match Craftax-classic, and the reward definition ignores health, but this is a metric-comparability and correctness concern, not circularity: no derivation in the paper is equivalent to its inputs by construction. The paper is self-contained against external benchmarks, so the honest finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The load-bearing assumptions are mostly domain-specific: the human baseline comes from a different benchmark, the tokenizer exploits known patch structure, and the fidelity metric relies on a learned symbol classifier. No new physical or conceptual entities are introduced; the 'chance player' used for OpenSpiel is a standard game-tree device.

free parameters (6)
  • NNT codebook size K = 4096
    Section 4.1: 'we found it critical to use a larger codebook size of K = 4096 and a threshold of tau = 0.75' for NNT.
  • NNT distance threshold tau = 0.75
    Section 4.1: threshold for adding new patches to the codebook; described as critical for performance.
  • Background planning warmup T_BP = 200k environment steps
    Section 4.3 ablates 0, 100k, 200k, 400k, 600k; 200k is used in all main runs.
  • Patch size = 7x7
    Section 3.5 and Figure 5: uses the oracle-provided ground truth patch size; performance drops with 5x5 or 9x9 patches.
  • TWM update count for the headline result (slow variant) = 4000
    Section 4.1: the best MBRL (slow) uses N_iters_WM = 4k to reach 69.66%, while the fast variant uses 500 and reaches 67.42%.
  • MinAtar reward/done loss weight = 10
    Appendix F Table 8: assigns a weight of 10 to reward and done cross-entropy losses to handle high cost of bad actions.
assumptions (3)
  • domain assumption Human expert reward from Crafter transfers to Craftax-classic
    The abstract's claim of exceeding human performance compares against a human score measured on the original Crafter benchmark (Table 1, asterisk), not on Craftax-classic itself.
  • domain assumption Oracle patch grid structure
    Section 3.5 splits observations into 9x9 patches of 7x7, using knowledge of the environment's grid; Section 4.3 shows sensitivity to patch size.
  • domain assumption Symbol extractor as rollout fidelity proxy
    Section 4.4 uses a CNN trained to 99% validation accuracy on ground-truth symbols to score TWM rollouts; this assumes the proxy reflects dynamics fidelity relevant for policy learning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Transformer World Models for Data-Efficient RL." pith.science (2026). https://pith.science/paper/D55O2SY2

@misc{pith2026250201591,
  author       = {Pith},
  title        = {Pith review of: Improving Transformer World Models for Data-Efficient RL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D55O2SY2}},
  note         = {Machine review of arXiv:2502.01591}
}
read the original abstract

We present three improvements to the standard model-based RL paradigm based on transformers: (a) "Dyna with warmup", which trains the policy on real and imaginary data, but only starts using imaginary data after the world model has been sufficiently trained; (b) "nearest neighbor tokenizer" for image patches, which improves upon previous tokenization schemes, which are needed when using a transformer world model (TWM), by ensuring the code words are static after creation, thus providing a constant target for TWM learning; and (c) "block teacher forcing", which allows the TWM to reason jointly about the future tokens of the next timestep, instead of generating them sequentially. We then show that our method significantly improves upon prior methods in various environments. We mostly focus on the challenging Craftax-classic benchmark, where our method achieves a reward of 69.66% after only 1M environment steps, significantly outperforming DreamerV3, which achieves 53.2%, and exceeding human performance of 65.0% for the first time. We also show preliminary results on Craftax-full, MinAtar, and three different two-player games, to illustrate the generality of the approach.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 21 canonical work pages

  1. [1]

    Agarwal, S

    P. Agarwal, S. Andrews, and S. E. Kahou. Learning to play atari in a world of tokens. ICML, 2024

    work page 2024

  2. [2]

    Alonso, A

    E. Alonso, A. Jelley, V. Micheli, A. Kanervisto, A. Storkey, T. Pearce, and F. Fleuret. Diffusion for world modeling: Visual details matter in atari. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=NadTwTODgC

    work page 2024

  3. [3]

    Alver and D

    S. Alver and D. Precup. A look at value-based decision-time vs. background planning methods across different settings. In Seventeenth European Workshop on Reinforcement Learning, Oct. 2024. URL https://openreview.net/pdf?id=Vx2ETvHId8

    work page 2024

  4. [4]

    J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  5. [5]

    Bradbury, R

    J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. Vander P las, S. Wanderman- M ilne, and Q. Zhang. JAX : composable transformations of P ython+ N um P y programs, 2018. URL http://github.com/jax-ml/jax

    2018

  6. [6]

    Chen, Y.-F

    C. Chen, Y.-F. Wu, J. Yoon, and S. Ahn. Transdreamer: Reinforcement learning with transformer world models. URL http://arxiv. org/abs/2202, 9481, 2022

    work page 2022

  7. [7]

    Cohen, K

    L. Cohen, K. Wang, B. Kang, and S. Mannor. Improving token-based world models with parallel observation prediction. arXiv preprint arXiv:2402.05643, 2024

    arXiv 2024

  8. [8]

    Cohen, K

    L. Cohen, K. Wang, B. Kang, U. Gadot, and S. Mannor. M3: A modular world model over streams of tokens. arXiv preprint arXiv:2502.11537, 2025

    arXiv 2025

Show all 61 references
  1. [9]

    Espeholt, H

    L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. IMPALA : Scalable distributed deep- RL with importance weighted actor-learner architectures. In ICML, pages 1407--1416. PMLR, July 2018 a . ...

    2018

  2. [10]

    Espeholt, H

    L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pages 1407--1416. PMLR, 2018 b

    2018

  3. [11]

    Farebrother, J

    J. Farebrother, J. Orbay, Q. Vuong, A. A. Taiga, Y. Chebotar, T. Xiao, A. Irpan, S. Levine, P. S. Castro, A. Faust, A. Kumar, and R. Agarwal. Stop regressing: Training value functions via classification for scalable deep RL . In Forty-first International Conference on Machine ...

    2024

  4. [12]

    J. Guan, S. Verch, C. Voelcker, E. Jackson, N. Papernot, and W. Cunningham. Temporal-difference learning using distributed error signals. Advances in Neural Information Processing Systems, 37: 0 108710--108734, 2024

    2024

  5. [13]

    Ha and J

    D. Ha and J. Schmidhuber. World models. In NIPS, 2018 a . URL http://arxiv.org/abs/1803.10122

    2018 arXiv

  6. [14]

    Ha and J

    D. Ha and J. Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018 b

    2018

  7. [15]

    D. Hafner. Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.06780, 2021

    2021 arXiv

  8. [16]

    Hafner, T

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2020 a . URL https://openreview.net/forum?id=S1lOTC4tDS

    2020

  9. [17]

    Hafner, T

    D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020 b

    2010 arXiv

  10. [18]

    Hafner, J

    D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023

    2023 arXiv

  11. [19]

    Hansen, H

    N. Hansen, H. Su, and X. Wang. TD - MPC2 : Scalable, robust world models for continuous control. 2024. URL http://arxiv.org/abs/2310.16828

    2024 arXiv

  12. [20]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770--778, 2016

    2016

  13. [21]

    Hessel, J

    M. Hessel, J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018. URL http://arxiv.org/abs/1710.02298

    2018 arXiv

  14. [22]

    G. Z. Holland, E. J. Talvitie, and M. Bowling. The effect of planning shape on dyna-style planning in high-dimensional state spaces. arXiv [cs.AI], June 2018. URL http://arxiv.org/abs/1806.01825

    2018 arXiv

  15. [23]

    Ioffe and C

    S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448--456. pmlr, 2015

    2015

  16. [24]

    Kaiser, M

    L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, A. Mohiuddin, R. Sepassi, G. Tucker, and H. Michalewski. Model-based reinforcement learning for atari. arXiv [cs.LG], Mar. 2019. URL http://arxiv.org/ab...

    2019 arXiv

  17. [25]

    Kapturowski, G

    S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018

    2018

  18. [26]

    Kauvar, C

    I. Kauvar, C. Doyle, L. Zhou, and N. Haber. Curious replay for model-based adaptation. In ICML, June 2023. URL https://arxiv.org/abs/2306.15934

    2023 arXiv

  19. [27]

    D. P. Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  20. [28]

    Lambert, K

    N. Lambert, K. Pister, and R. Calandra. Investigating compounding prediction errors in learned dynamics models. arXiv [cs.LG], Mar. 2022. URL http://arxiv.org/abs/2203.09637

    2022 arXiv

  21. [29]

    Lanctot, E

    M. Lanctot, E. Lockhart, J.-B. Lespiau, V. Zambaldi, S. Upadhyay, J. P\' e rolat, S. Srinivasan, F. Timbers, K. Tuyls, S. Omidshafiei, D. Hennes, D. Morrill, P. Muller, T. Ewalds, R. Faulkner, J. Kram\' a r, B. D. Vylder, B. Saeta, J. Bradbury, D. Ding, S. Borgeaud, M. Lai, J....

    1908 arXiv

  22. [30]

    Lei Ba, J

    J. Lei Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. ArXiv e-prints, pages arXiv--1607, 2016

    2016

  23. [31]

    Lewis, D

    M. Lewis, D. Yarats, Y. N. Dauphin, D. Parikh, and D. Batra. Deal or no deal? end-to-end learning for negotiation dialogues. arXiv preprint arXiv:1706.05125, 2017

    2017 arXiv

  24. [32]

    C. Lu, J. Kuba, A. Letcher, L. Metz, C. Schroeder de Witt, and J. Foerster. Discovered policy optimisation. Advances in Neural Information Processing Systems, 35: 0 16455--16468, 2022

    2022

  25. [33]

    Matthews, M

    M. Matthews, M. Beukman, B. Ellis, M. Samvelyan, M. Jackson, S. Coward, and J. Foerster. Craftax: A lightning-fast benchmark for open-ended reinforcement learning. arXiv preprint arXiv:2402.16801, 2024

    2024 arXiv

  26. [34]

    Micheli, E

    V. Micheli, E. Alonso, and F. Fleuret. Transformers are sample-efficient world models. arXiv preprint arXiv:2209.00588, 2022

    2022 arXiv

  27. [35]

    Micheli, E

    V. Micheli, E. Alonso, and F. Fleuret. Efficient world models with context-aware tokenization. arXiv preprint arXiv:2406.19320, 2024

    2024 arXiv

  28. [36]

    Mirzasoleiman, J

    B. Mirzasoleiman, J. Bilmes, and J. Leskovec. Coresets for data-efficient training of machine learning models. In ICML, 2020. URL http://proceedings.mlr.press/v119/mirzasoleiman20a/mirzasoleiman20a.pdf

    2020

  29. [37]

    T. M. Moerland, J. Broekens, A. Plaat, and C. M. Jonker. Model-based reinforcement learning: A survey. Foundations and Trends in Machine Learning, 16 0 (1): 0 1--118, 2023. URL https://arxiv.org/abs/2006.16712

    2023 arXiv

  30. [38]

    S. Moon, J. Yeom, B. Park, and H. O. Song. Discovering hierarchical achievements in reinforcement learning via contrastive learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  31. [39]

    K. Murphy. Reinforcement learning: An overview. arXiv preprint arXiv:2412.05265, 2024

    2024

  32. [40]

    T. Ni, B. Eysenbach, E. Seyedsalehi, M. Ma, C. Gehring, A. Mahajan, and P.-L. Bacon. Bridging state and history representations: Understanding self-predictive RL . In ICLR, Jan. 2024. URL http://arxiv.org/abs/2401.08898

    2024 arXiv

  33. [41]

    Awesome Model-Based Reinforcement Learning

    OpenDILab. Awesome Model-Based Reinforcement Learning . https://github.com/opendilab/awesome-model-based-RL

  34. [42]

    Oquab, T

    M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. J...

    2024

  35. [43]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  36. [44]

    Ramachandran, B

    P. Ramachandran, B. Zoph, and Q. V. Le. Swish: a self-gated activation function. arXiv preprint arXiv:1710.05941, 7 0 (1): 0 5, 2017

    2017 arXiv

  37. [45]

    N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R \"a dle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  38. [46]

    Robine, M

    J. Robine, M. H \"o ftmann, T. Uelwer, and S. Harmeling. Transformer-based world models are happy with 100k interactions. arXiv preprint arXiv:2303.07109, 2023

    2023 arXiv

  39. [47]

    Schrittwieser, I

    J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588 0 (7839): 0 604--609, 2020

    2020

  40. [48]

    Schulman, P

    J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015

    2015 arXiv

  41. [49]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  42. [50]

    Schwarzer, J

    M. Schwarzer, J. Obando-Ceron, A. Courville, M. Bellemare, R. Agarwal, and P. S. Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In ICML, May 2023. URL http://arxiv.org/abs/2305.19452

    2023 arXiv

  43. [51]

    Southey, M

    F. Southey, M. P. Bowling, B. Larson, C. Piccione, N. Burch, D. Billings, and C. Rayner. Bayes' bluff: Opponent modelling in poker. arXiv preprint arXiv:1207.1411, 2012

    2012 arXiv

  44. [52]

    J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  45. [53]

    Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023

    2023 arXiv

  46. [54]

    R. S. Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, pages 216--224. Elsevier, 1990

    1990

  47. [55]

    R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  48. [56]

    Toledo, L

    E. Toledo, L. Midgley, D. Byrne, C. R. Tilbury, M. Macfarlane, C. Courtot, and A. Laterre. Flashbax: Streamlining experience replay buffers for reinforcement learning with jax, 2023. URL https://github.com/instadeepai/flashbax/

    2023

  49. [57]

    Van Den Oord, O

    A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017

    2017

  50. [58]

    H. P. Van Hasselt, M. Hessel, and J. Aslanides. When to use parametric models in reinforcement learning? Advances in Neural Information Processing Systems, 32, 2019

    2019

  51. [59]

    W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y. Gao. Mastering atari games with limited data. In NIPS, Nov. 2021. URL https://openreview.net/pdf?id=OKrNPg3xR3T

    2021

  52. [60]

    Young and T

    K. Young and T. Tian . Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.03176, 2019

    1903 arXiv

  53. [61]

    Zhang, G

    W. Zhang, G. Wang, J. Sun, Y. Yuan, and G. Huang. Storm: Efficient stochastic transformer based world models for reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.