Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Hadamax Encoding: Elevating Performance in Model-Free Atari

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that swapping in a Hadamax encoder—max-pooling the Hadamard product of two parallel GELU-activated convolutional branches—raises PQN's median human-normalized Atari-57 score by 80% and surpasses Rainbow-DQN with no…

desk verdict A solid architectural contribution with a robust internal result; the 'surpasses Rainbow' claim is weaker than advertised and needs a controlled comparison. read the letter →

arxiv 2505.15345 v2 pith:UNJTQ6LU submitted 2025-05-21 cs.LG cs.AI

Jacob E. Kooi , Zhao Yang , Vincent François-Lavet This is my paper
classification cs.LGcs.AI
keywords Hadamardproductmax-poolingencoderarchitecturemodel-freereinforcementlearningAtari-57PQNGELUeffectiverank
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that encoder architecture alone can be a major lever in model-free reinforcement learning, not a source of small or unstable gains. The authors add a Hadamax encoder—three convolutional layers that compute the Hadamard (elementwise) product of two parallel GELU-activated branches and downsample with max-pooling—to the recent PQN algorithm, leaving every algorithmic setting untouched. On the full Atari-57 suite at 200M frames, Hadamax-PQN reaches a median human-normalized score about 80% higher than vanilla PQN and surpasses the official Rainbow-DQN score after roughly 90M frames, while remaining more than an order of magnitude faster. If the claim holds, it means a cheap, drop-in architectural change can rival algorithmic innovations and should become the default encoder baseline for pixel-based RL.

What carries the argument

The Hadamax encoder is a shallow three-convolutional-layer encoder whose hidden representation at depth $j$ is $z_j = \mathrm{MP}\big( f(\mathrm{LN}(z_{j-1}A_1^{(j)})) \odot f(\mathrm{LN}(z_{j-1}A_2^{(j)})) \big)$, where $\mathrm{LN}$ is layer normalization, $f$ is GELU, $\odot$ is the Hadamard (elementwise) product, and $\mathrm{MP}$ is max-pooling. Downsampling is done by max-pooling windows rather than by convolutional strides, and the final features are max-pooled before flattening instead of average-pooled. The mechanism works by letting two parallel branches interact multiplicatively, which enlarges effective representation capacity without adding depth or width: the measured effect is higher effective rank in the deeper convolutional layers and less rank decay over training, while PQN's LayerNorm stabilizes the zero-saturating GELU activations enough that elementwise multiplication does not produce dead neurons.

What would settle it

Run Rainbow-DQN and PQN under the exact 5-seed, 200M-frame evaluation protocol used for Hadamax-PQN and recompute the median human-normalized score; if the matched-protocol Rainbow score is not below Hadamax-PQN's, the paper's headline comparison fails.

Watch

Extended reading notes

Core claim

The central claim is that replacing the conventional Nature CNN encoder with the Hadamax encoder yields a substantial, algorithm-independent performance gain in model-free Atari play. The authors report that Hadamax-PQN, with no algorithmic hyperparameter modifications, achieves an 80% performance gain over the PQN baseline on Atari-57, significantly surpasses Rainbow-DQN's official 200M-frame scores, and reaches Rainbow-level performance in about 90M frames instead of 200M. The same encoder applied to C51 improves Atari-10 scores by roughly 70%, and smaller gains appear for DQN and Rainbow on Atari-3. Diagnostic measurements show Hadamax keeps deeper convolutional layers at higher effective rank with less rank decay during training and slightly fewer dead neurons in the final layer, which the authors take as evidence of greater representation capacity without network scaling.

Load-bearing premise

The headline comparison to Rainbow-DQN assumes that the officially published Rainbow scores, which the authors say run higher than typical reimplementations, are directly comparable to their own five-seed Hadamax-PQN runs; a mismatch in seeds or evaluation protocol could change that result, whereas the 80% gain over PQN is an internal comparison.

Editorial extensions

If this is right

  • Hadamax-PQN reaches Rainbow-DQN's reported 200M-frame median score at about 90M frames, so the same performance can be obtained in under half the training budget.
  • The Hadamax encoder transfers to other value-based agents: it boosts C51 by about 70% on Atari-10 and improves DQN and Rainbow on Atari-3, suggesting it is a general drop-in encoder rather than a PQN-specific fix.
  • Ablations show max-pooling is the largest single contributor, followed by Hadamard products, then GELU; the full combination matters more than any one modification.
  • Because no algorithmic or hyperparameter changes are involved, Hadamax-PQN can serve as a stronger and faster base on which future algorithmic improvements, including exploration techniques, can be tested.
  • Deeper versions of the Hadamax encoder (5 and 7 layers) do not improve over the 3-layer version at 40M frames, indicating simple depth scaling is not the route to further gains within this design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same encoder logic could be tested on offline RL benchmarks and pixel-based continuous control, where representation capacity is also a bottleneck; the paper only demonstrates Atari value-based agents.
  • The finding suggests a general design principle—replace strided convolutions with max-pooling and enrich features via multiplicative interactions—that may transfer to supervised or self-supervised vision encoders, though the paper makes no such claim.
  • Because effective rank and dead-neuron metrics were measured on a subset of five high-improvement games, an extension would be to track these quantities on low-improvement and negative-improvement games to see whether rank maintenance predicts per-game gains.
  • If the 80% gain is reproducible in other JAX-based RL codebases, Hadamax could become a standard encoder for pixel-based model-free research, shifting attention from algorithm tweaks to encoder synthesis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces the Hadamax encoder, which replaces convolutional downsampling in the "Nature" DQN encoder with max-pooling, computes Hadamard products of two parallel layer-normalized branches per block, and uses GELU activations. The encoder is plugged into PQN without changing algorithmic hyperparameters, evaluated on Atari-57 for 200M frames with 5 seeds, and compared with PQN, PQN-ResNet-15, and published DQN/C51/Rainbow scores. The authors report an 80% median human-normalized performance gain over PQN, claim that Hadamax-PQN surpasses Rainbow-DQN while being more than an order of magnitude faster, show that the encoder also improves C51, DQN, and Rainbow on subsets, and analyze effective rank and dead neurons.

Significance. If the internal comparison is taken at face value, the contribution is significant: a simple, purely architectural change yields a large and reproducible gain over a modern baseline, and the released code and hyperparameter table make the result easy to verify. The 5-seed per-game table and the positive transfer to C51, DQN, and Rainbow strengthen the claim that the encoder, not task-specific tuning, drives the improvement. However, the paper's headline external claim that Hadamax-PQN "significantly surpasses Rainbow-DQN" and achieves "state-of-the-art model-free performance" is not yet supported, because it rests on comparing different evaluation protocols and on a baseline set that excludes other modern model-free agents. The core contribution can likely be salvaged by adding a matched control and narrowing the claims, so the present version warrants a major revision rather than rejection.

major comments (3)
  1. [§5.1, abstract and Fig. 5] The comparison that underlies the statement that Hadamax-PQN "significantly surpasses Rainbow-DQN" is not a controlled one: the DQN, C51, and Rainbow scores are taken from the original publications, and the paragraph in §5.1 explicitly concedes that those official scores are "generally higher than their practical implementations on various GitHub repositories." Since the Hadamax-PQN runs use the EnvPool/JAX pipeline with 5 seeds, any differences in sticky-action settings, no-op starts, frame-skip, evaluation length, or seed handling can account for part of the gap. This is load-bearing for the abstract and introduction, so the authors should either run Rainbow, C51, and DDQN in the same evaluation harness (the cleanrl implementations already used elsewhere in the paper make this feasible) or replace the phrase "significantly surpasses" with a clearly delimited claim that does not rest on cross-paper protocol comparability.
  2. [§5.1, Fig. 5, Table 2] No confidence intervals, standard deviations, or significance tests are reported for the headline 80% gain or for the comparison to Rainbow. With only 5 seeds, "significantly" is a statistical claim that the paper does not support. The large effect size makes the result plausible, but the authors should report per-game per-seed distributions, the median with a bootstrap interval, and preferably a paired test across the 57 games when comparing PQN and Hadamax-PQN, as well as a matched-protocol test for the Rainbow comparison.
  3. [Abstract, §1, §2, §5.1] The phrase "state-of-the-art model-free performance" is not supported by the chosen baselines: the comparison set is DQN, C51, Rainbow, PQN, and PQN-ResNet-15, while R2D2 (mentioned in §2) and other recent model-free agents such as Agent57 are not included. The claim should be narrowed to "outperforms the compared baselines under the same protocol" unless a broader comparison is added. This is a framing issue rather than an internal inconsistency, but it is load-bearing for the paper's stated contribution.
minor comments (5)
  1. [§5.4 and Fig. 10] The title "Which Design Choice is most Important?" should be qualified to the 40M-frame setting, because the ablation study is performed only at 40M frames while the headline results are at 200M frames.
  2. [Appendix C.3] The description of baselines is ambiguous: it says Rainbow, C51, and DQN use cleanrl implementations with default hyperparameters, but the main 200M-frame comparison uses scores taken from the original papers. Please clarify which protocol applies to each figure and table.
  3. [§2] The sentence "PQN [17] algorithmically simplifies DQN and uses LayerNorm [4] to provably stabilize optimization" uses the word "provably" without a proof or theorem reference in this paper; if the proof is in [17], say so explicitly.
  4. [Fig. 6] The summary statistic "over 17 games having more than 100% improvement, compared to only one single game having more than a 50% decrease" uses asymmetric thresholds and would be more informative if the full distribution of per-game changes were shown with confidence intervals.
  5. [Table 2] The per-game score table reports only mean final scores without standard deviations or per-seed values, which makes it impossible to assess the stability of the reported 80% gain across games.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circular derivation; the only self-citation is motivational and independently re-tested, so the main caveats are benchmark-comparability concerns rather than circularity.

full rationale

The paper's central claims are empirical measurements, not consequences of its definitions. Equation (5) defines the Hadamax encoder as max-pooling of a Hadamard product of LayerNormed, GELU-activated parallel convolutions; this is a construction, not a derivation of the reported gains. The 80% improvement over PQN and the surpassing of Rainbow-DQN are read from measured training curves in Figure 5, and no parameter of Hadamax-PQN is fitted to those baseline scores. The one overlapping-author citation, [30] ('Hadamard representations: Augmenting hyperbolic tangents in RL'), motivates the Hadamard product, but the contribution is established inside the paper: Figure 10 ablates the Hadamard component by subtraction ('No Hadamard') and by addition ('PQN + Hadamard'), and Section 5.2 re-tests the encoder on C51. The prior work is therefore not load-bearing. The authors' note that official DDQN/C51/Rainbow scores are 'generally higher than their practical implementations on various GitHub repositories' raises a legitimate protocol-comparability question for the external baseline claim, but that is a correctness or benchmarking risk, not a circular reduction: those external scores are not inputs from which the Hadamax gain is derived. Stated limitations (limited testing on BBF/Dreamer and extra computational overhead) are scoping statements, not circularity. No equation in the paper makes the predicted gain equivalent to a fitted parameter or to a self-cited uniqueness theorem.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The central claim rests on standard RL assumptions plus several empirical design hypotheses, two of which are inherited from the authors' prior work. The paper contains no new physical or mathematical entities; its contribution is architectural and empirical.

free parameters (1)
  • Max-pooling window sizes and strides (4x4/4, 2x2/2, 3x3/1)
    Hand-chosen architectural parameters in the Hadamax encoder (Appendix B) that control downsampling and the final pooling before flattening. They are not tuned to the data and are kept fixed across all games, but the reported performance depends on them.
assumptions (6)
  • domain assumption Hadamard products between parallel hidden layers increase effective rank and downstream performance without increasing dimensionality.
    Used in Section 4.2 to justify the Hadamard operation; based on the authors' prior paper [30] and supported post-hoc by the effective-rank measurements in Section 5.3, but not proven in this paper.
  • domain assumption PQN's LayerNorm-based stability enables Hadamard products with zero-saturating activations (ReLU/GELU) without dead-neuron collapse.
    Invoked in Section 4.2 and discussed in Section 5.3 (dead-neuron fraction, Figure 9). The paper provides empirical evidence but no proof; if false, the encoder might not transfer to other base algorithms.
  • domain assumption Max-pooling preserves the strongest features for value-based RL better than strided convolutions, and average-pooling before the linear layer would smooth away feature importance.
    Stated in Section 4.1 as the design hypothesis behind the max-pooling choice; supported by the ablation in Section 5.4 but not proven.
  • domain assumption The median human-normalized score over 57 games is a valid aggregate metric for comparing Atari agents.
    Used throughout Section 5 and Appendix D.1; standard in the field but relies on the chosen human and random scores and on averaging seeds.
  • domain assumption Atari-10 and Atari-3 subsets with their published coefficients are representative proxies for the full Atari-57 median.
    Used in Section 5.2 and Appendix D.3 for the C51/DQN/Rainbow generalization experiments; the coefficients come from [2] and are taken as given.
  • standard math Standard MDP formulation, Q-learning, and TD/lambda-return updates (Section 3.1) are correct and applicable.
    Background theory; not in question for this empirical study.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hadamax Encoding: Elevating Performance in Model-Free Atari." pith.science (2026). https://pith.science/paper/UNJTQ6LU

@misc{pith2026250515345,
  author       = {Pith},
  title        = {Pith review of: Hadamax Encoding: Elevating Performance in Model-Free Atari},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UNJTQ6LU}},
  note         = {Machine review of arXiv:2505.15345}
}
read the original abstract

Neural network architectures have a large impact in machine learning. In reinforcement learning, network architectures have remained notably simple, as changes often lead to small gains in performance. This work introduces a novel encoder architecture for pixel-based model-free reinforcement learning. The Hadamax (\textbf{Hada}mard \textbf{max}-pooling) encoder achieves state-of-the-art performance by max-pooling Hadamard products between GELU-activated parallel hidden layers. Based on the recent PQN algorithm, the Hadamax encoder achieves state-of-the-art model-free performance in the Atari-57 benchmark. Specifically, without applying any algorithmic hyperparameter modifications, Hadamax-PQN achieves an 80\% performance gain over vanilla PQN and significantly surpasses Rainbow-DQN. For reproducibility, the full code is available on \href{https://github.com/Jacobkooi/Hadamax}{GitHub}.

Figures

Figures reproduced from arXiv: 2505.15345 by the authors.

Figure 1
Figure 1. Performance versus GPU hours in the full Atari [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Encoder architectures of DQN, PQN , the proposed [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. ReLU and GELU. where Φ(x) is the cumulative distribution function of the standard normal distribution. Equivalently, it can be expressed using the error function as: GELU(x) = 0.5x  1 + erf  x √ 2  In contrast to the ReLU, which converts negative inputs to zero, GELU permits small negative values to pass through in a softened form (See [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The Atari-57 domain. Environments: The full 57-game Atari domain [6] is used as a standardized benchmark for eval￾uating our algorithm’s performance. In line with best practices in the field, we focus on the me￾dian human-normalized score over all 57 games [37, 27, 21,…
Figure 5
Figure 5. Figure 5: Median Human-Normalized performance training PQN, PQN (Resnet-15) and Hadamax [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Per-game improvement of Hadamax-PQN over PQN (Log Scale). [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: C51 with and without a Hadamax encoder on Atari-10. The Hadamax encoder not only enhances the performance of PQN, but also works effectively with other reinforcement learn￾ing agents. To showcase this, the C51 algorithm is evaluated on the Atari-10 benchmark for 40M en…
Figure 8
Figure 8. Figure 8: Effective rank [31] of the 4 hidden layers for both the baseline PQN and the Hadamax￾PQN setting. Although there is no visible difference between the first and final layer, the deeper convolutional layers show a lower effective rank in the baseline setting, as well as …
Figure 9
Figure 9. Figure 9: Fraction of dead neurons over 200M frames. Further investigation into the penultimate layer’s fraction of dead neurons shows a small decrease from the baseline (see [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Ablations of Hadamax-PQN, each represented as a subtraction from the full Hadamax [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 11
Figure 11. Figure 11: Hadamax encoder depth Ablations. Simply using deeper convolutional Hadamax encoders does not seem to improve performance. Although there are more promising ways to scale the Hadamax encoder both in depth and width, the computational cost was the limiting factor in pur…
Figure 12
Figure 12. Figure 12: Performance gains of DQN, C51 and Rainbow with Hadamax encoders on a subset of [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents

    cs.LG 2025-10 conditional novelty 5.0 of 10

    Simplicial embeddings — group-wise softmax feature layers — improve sample efficiency and final performance of FastTD3, FastSAC, and PPO across continuous- and discrete-control benchmarks at no meaningful runtime cost.

Reference graph

Works this paper leans on

62 extracted references · 36 canonical work pages · cited by 1 Pith paper

  1. [1]

    Learning to Play Atari in a World of Tokens

    P. Agarwal, S. Andrews, and S. E. Kahou. Learning to play atari in a world of tokens. arXiv preprint arXiv:2406.01361, 2024

    work page Pith review arXiv 2024

  2. [2]

    Aitchison, P

    M. Aitchison, P. Sweetser, and M. Hutter. Atari-5: Distilling the arcade learning environment down to five games. In International Conference on Machine Learning, pages 421–438. PMLR, 2023

    work page 2023

  3. [3]

    Alonso, A

    E. Alonso, A. Jelley, V . Micheli, A. Kanervisto, A. J. Storkey, T. Pearce, and F. Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757–58791, 2024

    work page 2024

  4. [4]

    J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  5. [5]

    M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In International conference on machine learning, pages 449–458. PMLR, 2017

    work page 2017

  6. [6]

    M. G. Bellemare, Y . Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of artificial intelligence research, 47:253–279, 2013

    work page 2013

  7. [7]

    Bhatt, D

    A. Bhatt, D. Palenicek, B. Belousov, M. Argus, A. Amiranashvili, T. Brox, and J. Peters. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. arXiv preprint arXiv:1902.05605, 2019

    arXiv 1902

  8. [8]

    Bonnet, D

    C. Bonnet, D. Luo, D. Byrne, S. Surana, S. Abramowitz, P. Duckworth, V . Coyette, L. I. Midgley, E. Tegegn, T. Kalloniatis, O. Mahjoub, M. Macfarlane, A. P. Smit, N. Grinsztajn, R. Boige, C. N. Waters, M. A. Mimouni, U. A. M. Sob, R. de Kock, S. Singh, D. Furelos-Blanco, V . Le, A. Pretorius, and A. Laterre. Jumanji: a diverse suite of scalable reinforcem...

    work page 2024

Show all 62 references
  1. [9]

    Bradbury, R

    J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018

    2018

  2. [10]

    X. Chen, C. Wang, Z. Zhou, and K. Ross. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021

    2021 arXiv

  3. [11]

    G. G. Chrysos, Y . Wu, R. Pascanu, P. Torr, and V . Cevher. Hadamard product in deep learn- ing: Introduction, advances and challenges. arXiv preprint arXiv:2504.13112, April 2025. arXiv:2504.13112v1 [cs.LG]

    2025 arXiv

  4. [12]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo- gi...

    2019

  5. [13]

    Espeholt, H

    L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V . Mnih, T. Ward, Y . Doron, V . Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning , pages 1407–1416. P...

    2018

  6. [14]

    Fortunato, M

    M. Fortunato, M. G. Azar, B. Piot, J. Menick, I. Osband, A. Graves, V . Mnih, R. Munos, D. Has- sabis, O. Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv:1706.10295, 2017

    2017 arXiv

  7. [15]

    François-Lavet, P

    V . François-Lavet, P. Henderson, R. Islam, M. G. Bellemare, J. Pineau, et al. An introduction to deep reinforcement learning. Foundations and Trends® in Machine Learning, 11(3-4):219–354, 2018

    2018

  8. [16]

    C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax - a differentiable physics engine for large scale rigid body simulation, 2021

    2021

  9. [17]

    Gallici, M

    M. Gallici, M. Fellows, B. Ellis, B. Pou, I. Masmitja, J. N. Foerster, and M. Martin. Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.04811, 2024

    2024 arXiv

  10. [18]

    Gulcehre, S

    C. Gulcehre, S. Srinivasan, J. Sygnowski, G. Ostrovski, M. Farajtabar, M. Hoffman, R. Pascanu, and A. Doucet. An empirical study of implicit regularization in deep offline RL. Transactions on Machine Learning Research, 2022. 10

    2022

  11. [19]

    Haarnoja, A

    T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V . Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018

    2018 arXiv

  12. [20]

    Hafner, T

    D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555–2565. PMLR, 2019

    2019

  13. [21]

    Hafner, T

    D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020

    2010 arXiv

  14. [22]

    Hafner, T

    D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering Atari with Discrete World Models. 10 2020

    2020

  15. [23]

    Hafner, J

    D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023

    2023 arXiv

  16. [24]

    N. A. Hansen, H. Su, and X. Wang. Temporal difference learning for model predictive control. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors,Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of ...

    2022

  17. [25]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR ’16, pages 770–778. IEEE, June 2016

    2016

  18. [26]

    Hendrycks and K

    D. Hendrycks and K. Gimpel. Gaussian Error Linear Units (GELUs). 2016

    2016

  19. [27]

    Hessel, J

    M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  20. [28]

    Huang, R

    S. Huang, R. F. J. Dossa, C. Ye, J. Braga, D. Chakraborty, K. Mehta, and J. G. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1–18, 2022

    2022

  21. [29]

    Kapturowski, G

    S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018

    2018

  22. [30]

    J. E. Kooi, M. Hoogendoorn, and V . François-Lavet. Hadamard representations: Augmenting hyperbolic tangents in rl, 2024

    2024

  23. [31]

    Kumar, R

    A. Kumar, R. Agarwal, D. Ghosh, and S. Levine. Implicit under-parameterization inhibits data- efficient deep reinforcement learning. In International Conference on Learning Representations, 2021

    2021

  24. [32]

    R. T. Lange. gymnax: A JAX-based reinforcement learning environment library, 2022

    2022

  25. [33]

    Laskin, A

    M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pages 5639–5650. PMLR, 2020

    2020

  26. [34]

    H. Lee, D. Hwang, D. Kim, H. Kim, J. J. Tai, K. Subramanian, P. R. Wurman, J. Choo, P. Stone, and T. Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint arXiv:2410.09754, 2024

    arXiv

  27. [35]

    C. Lu, J. Kuba, A. Letcher, L. Metz, C. Schroeder de Witt, and J. Foerster. Discovered policy optimisation. Advances in Neural Information Processing Systems, 35:16455–16468, 2022

    2022

  28. [36]

    V . Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013

    2013 arXiv

  29. [37]

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep ...

    2015

  30. [38]

    Nauman, M

    M. Nauman, M. Ostaszewski, K. Jankowski, P. Miło ´s, and M. Cygan. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. arXiv preprint arXiv:2405.16158, 2024. 11

    2024 arXiv

  31. [39]

    T. Ni, B. Eysenbach, E. Seyedsalehi, M. Ma, C. Gehring, A. Mahajan, and P.-L. Bacon. Bridging state and history representations: Understanding self-predictive rl. arXiv preprint arXiv:2401.08898, 2024

    2024 arXiv

  32. [40]

    Obando-Ceron, A

    J. Obando-Ceron, A. Courville, and P. S. Castro. In deep reinforcement learning, a pruned network is a good network. arXiv preprint arXiv:2402.12479, 2024

    2024 arXiv

  33. [41]

    Obando-Ceron, G

    J. Obando-Ceron, G. Sokar, T. Willi, C. Lyle, J. Farebrother, J. Foerster, G. K. Dziugaite, D. Precup, and P. S. Castro. Mixtures of experts unlock parameter scaling for deep rl. arXiv preprint arXiv:2402.08609, 2024

    2024 arXiv

  34. [42]

    Radford, K

    A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. OpenAI, 2018

    2018

  35. [43]

    Rutherford, B

    A. Rutherford, B. Ellis, M. Gallici, J. Cook, A. Lupu, G. Ingvarsson, T. Willi, A. Khan, C. S. de Witt, A. Souly, S. Bandyopadhyay, M. Samvelyan, M. Jiang, R. T. Lange, S. Whiteson, B. Lacerda, N. Hawes, T. Rocktaschel, C. Lu, and J. N. Foerster. Jaxmarl: Multi-agent rl enviro...

    2023 arXiv

  36. [44]

    Schrittwieser, I

    J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020

    2020

  37. [45]

    Schrittwieser, I

    J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver. Mastering atari, go, chess and shogi by planning with a learned model, 2019. cite arxiv:1911.08265

    2019 arXiv

  38. [46]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  39. [47]

    Schwarzer, A

    M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929, 2020

    2007 arXiv

  40. [48]

    Schwarzer, J

    M. Schwarzer, J. S. O. Ceron, A. Courville, M. G. Bellemare, R. Agarwal, and P. S. Castro. Big- ger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pages 30365–30380. PMLR, 2023

    2023

  41. [49]

    Sokar, R

    G. Sokar, R. Agarwal, P. S. Castro, and U. Evci. The dormant neuron phenomenon in deep reinforcement learning. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume ...

    2023

  42. [50]

    R. K. Srivastava, K. Greff, and J. Schmidhuber. Training very deep networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015

    2015

  43. [51]

    R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018

    2018

  44. [52]

    Van Hasselt, A

    H. Van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016

    2016

  45. [53]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Processing...

    2017

  46. [54]

    S. Wang, S. Liu, W. Ye, J. You, and Y . Gao. Efficientzero v2: Mastering discrete and continuous control with limited data. arXiv preprint arXiv:2403.00564, 2024

    2024 arXiv

  47. [55]

    Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995–2003. PMLR, 2016

    1995

  48. [56]

    J. Weng, M. Lin, S. Huang, B. Liu, D. Makoviichuk, V . Makoviychuk, Z. Liu, Y . Song, T. Luo, Y . Jiang, Z. Xu, and S. Yan. EnvPool: A highly parallel reinforcement learning environment execution engine. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, edi...

    2022

  49. [57]

    W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y . Gao. Mastering atari games with limited data. Advances in neural information processing systems, 34:25476–25488, 2021

    2021

  50. [58]

    SAME " ) 15 # Second block 16 x1 = nn . Conv (64 , k e r n e l _ s i z e =(4 , 4) , strides =(1 , 1) , padding =

    W. Zhang, G. Wang, J. Sun, Y . Yuan, and G. Huang. Storm: Efficient stochastic transformer based world models for reinforcement learning. Advances in Neural Information Processing Systems, 36:27147–27166, 2023. 13 Appendix Table of Contents A Impact Statement 15 B Hadamax Enco...

    2023

  51. [59]

    For each game in the subset, compute the normalized score Zi as: Zi = 100 × xi − ri hi − ri where xi is the raw score, and ri and hi are the random and human scores for game i, respectively (see Table 2 for values)

  52. [60]

    Apply the log transform: ϕ(Zi) = log10(1 + max(0, Zi))

  53. [61]

    Compute the weighted sum f = P i∈I ciϕ(Zi), where I is the subset of games and ci are the subset-specific coefficients

  54. [62]

    Obtain the predicted median score as: ˆt = 10f − 1 For Atari-3, the subset comprises Battle Zone, Name This Game, and Phoenix, with coefficients ci = [0.3706, 0.5133, 0.1015]. For Atari-10, the subset includes Amidar, Bowling, Frostbite, Kung Fu Master, River Raid, Battle Zone...

    1942

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.