Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding weight normalization to CrossQ's batch-normalized critic stabilizes training and lets the method scale to higher update-to-data ratios, matching or beating larger baselines on 25 continuous-control tasks.

desk verdict Weight normalization gives CrossQ a real scaling boost, and the empirical work is solid enough to deserve a serious look despite reproducibility gaps. read the letter →

arxiv 2502.07523 v2 pith:DCGAVBLV submitted 2025-02-11 cs.LG cs.AI

Daniel Palenicek , Florian Vogt , Joe Watson , Jan Peters This is my paper · ORCID
classification cs.LGcs.AI
keywords reinforcementlearningsampleefficiencyupdate-to-dataratiobatchnormalizationweightlossofplasticityeffectiveratecontinuouscontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that CrossQ, a sample-efficient off-policy reinforcement learning algorithm that uses batch normalization inside its actor and critic networks, can be made to scale to higher update-to-data (UTD) ratios by adding weight normalization. The authors find that vanilla CrossQ becomes brittle on harder continuous-control tasks: critic parameter norms grow, the effective learning rate drops, and dead neurons increase, and this is worse at higher UTD ratios. CrossQ plus weight normalization holds the first two linear layers of the critic at unit norm, counteracting the norm growth and keeping the effective learning rate stable, so that performance improves or stays flat as the UTD ratio rises. The result matches or beats recent high-UTD baselines across 25 tasks from two continuous-control benchmark suites, DMC and MyoSuite, without network resets and with about 90% fewer network parameters.

What carries the argument

The central mechanism is the pairing of batch normalization with weight normalization in the critic. Weight normalization reparameterizes each weight vector as $\tilde{w} = c\, w / \lVert w \rVert_2$, and because batch normalization makes the network scale-invariant, holding $c$ fixed does not alter the network's outputs while ensuring gradients scale inversely with the parameter scale, so the effective learning rate stays stable. A second piece is the reinstated target network: the critic and the target critic are both evaluated in training mode on the same joined state-action batch, so the batch-normalization layers compute shared statistics and the target values remain in-distribution.

What would settle it

Run CrossQ with weight normalization at UTD 5 on a hard task such as the DMC humanoid while evaluating the target critic in inference mode with its own separately computed batch statistics; if training remains stable and matches the joined-forward-pass version, the shared-statistics target design is not load-bearing, whereas if it diverges or the target values drift, the paper's account of the mechanism is confirmed.

Watch

Extended reading notes

Core claim

The central claim is that the instability of CrossQ at higher UTD ratios is driven by unbounded growth in the critic's weight norms, which lowers the effective learning rate and accelerates loss of plasticity, and that weight normalization is a simple and sufficient remedy. Because batch normalization makes the critic scale-invariant, constraining the weights to fixed norm does not change the function the network represents, yet it reverses the gradient-scaling effect: gradients scale inversely with the parameter scale, so holding the norm constant keeps the effective learning rate fixed. With weight normalization applied to the first two linear layers and weight decay on the final layer, CrossQ with weight normalization scales its UTD ratio reliably, matches or outperforms BRO and SIMBA on the hardest dog and humanoid tasks at lower UTD ratios, and removes the need for periodic network resets. The authors also re-introduce target networks by evaluating both the critic and the target critic in training mode on a joined state-action batch, so that the batch-normalization statistics are shared and target values stay within the training distribution.

Load-bearing premise

The load-bearing premise is that the reinstated target network works correctly when both the critic and the target critic are evaluated in training mode on a joined state-action batch, because this makes the batch-normalization statistics shared and keeps the target values in-distribution; if this mechanism fails at high UTD ratios, target values would be biased and the claimed stable scaling would break.

Editorial extensions

If this is right

  • Higher UTD ratios translate into sample-efficiency gains for CrossQ with weight normalization: performance improves or stays flat as the UTD ratio is raised, rather than collapsing.
  • CrossQ plus weight normalization at UTD 5 is competitive with BRO and SIMBA across 25 DMC and MyoSuite tasks, including the dog and humanoid environments, and does so without network resets or an extra exploration policy.
  • The method uses roughly 600,000 network parameters compared with about 5 million for BRO, about a 90% reduction.
  • Ablations show that weight normalization is the main stabilizer: removing it causes the largest performance drop, and the hard unit-norm constraint outperforms a soft L2 penalty across regularization scales.
  • Target networks contribute substantially on DMC; removing them degrades performance almost as much as removing weight normalization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the effective-learning-rate mechanism is as generic as the paper's reasoning suggests, the same weight-normalization fix could stabilize other normalization-based actor-critic algorithms beyond CrossQ, but the paper does not test this.
  • The joined-forward-pass target-network design may be a general recipe for adding target networks to any batch-normalized critic, since it resolves the train/evaluation distribution mismatch, though the paper only demonstrates it inside CrossQ with weight normalization.
  • A natural stress test, which the paper itself notes is out of scope, is pushing the UTD ratio beyond 20 or moving to vision-based inputs; the paper only evaluates continuous state-action tasks.
  • Keeping the critic on a fixed-norm manifold might reduce the need for other plasticity interventions such as parameter resets in continual-learning settings, but any such benefit beyond the benchmark tasks studied here remains to be shown.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies the scaling behavior of CrossQ, a batch-normalized off-policy actor-critic method, as the update-to-data (UTD) ratio increases. The authors show that vanilla CrossQ suffers from growing critic parameter norms, a decreasing effective learning rate, and increasing dead-neuron fractions, particularly on high-dimensional DMC and MyoSuite tasks. To address this, they propose CrossQ+WN, which combines weight normalization on the first two linear layers of the critic with a re-introduced target network that is evaluated in training mode on a joined batch of current and next state-action pairs, so that batch-normalization statistics are shared. They report that CrossQ+WN scales reliably with UTD ratios and achieves competitive or superior performance to BRO, SIMBA, and SR-SAC across 25 continuous control tasks with 10 seeds each. The paper also includes ablations showing the importance of weight normalization and target networks, and argues that the method removes the need for network resets.

Significance. If the empirical claims hold, CrossQ+WN is a simple and practical enhancement that improves the sample-efficiency/compute trade-off of CrossQ while avoiding network resets. The connection to effective learning rate is well-motivated by prior work on normalization and loss of plasticity, and the benchmark evaluation is broad (25 tasks, 10 seeds each). The paper reports IQM with 90% bootstrap confidence intervals, which is a methodological strength, and the ablations (removing WN, target networks, comparing against soft L2) are informative. The main weaknesses are the reliance on imported baseline data rather than re-running baselines under the paper's protocol, the unvalidated target-network statistics-sharing mechanism, and the absence of a statistical test for the central 'reliable scaling' claim. These are load-bearing issues for the paper's primary contribution.

major comments (3)
  1. [Section 4, 'Target networks'] The target-network design described here is load-bearing for the claim of stable high-UTD training, but it is not validated in isolation. The paper states that the joined forward pass through both the critic and the target critic, with both evaluated in training mode, satisfies the distribution-mismatch requirement 'exactly as proposed by Bhatt et al.' However, CrossQ removed the target network entirely; in the new design, the target network has different parameters from the critic, and computing its batch-normalization statistics over the current batch makes the target value depend on the composition of the current mini-batch, not only on the next state-action pair. The ablation in Section 5.4 removes the target network altogether, showing that target networks matter, but it does not test alternative ways to compute the target's normalization statistics (e.g., using running averages, or statistics from the next-state subset). Consequently, the specific mechanism--shared training-mode statistics across two distinct networks--is asserted without a dedicated experiment or theory, despite the paper's own limitation statement in Section 7 that the theory does not connect to convergence or sample efficiency.
  2. [Section 5.1, baseline protocol] The central comparison with BRO and SIMBA is based on official evaluation data rather than re-running these baselines under the paper's protocol. The paper says the BRO codebase 'followed the same evaluation protocol,' but Table 1 records substantial algorithmic differences (BRO uses discount 0.99 on MyoSuite while CrossQ+WN uses 0.95; batch size 128 vs. 256; target entropy |A| vs. |A|/2; action repeat 1 vs. 2). The paper does not verify that evaluation frequency, evaluation length, environment version, and reward scaling are identical across all sources. This concern is acute for the wall-clock comparison in Figure 2, whose caption claims a single RTX 4090 workstation while Section 5.1 reports all runs on RTX 3090 and A5000 GPUs; it is unclear where the baseline wall-clock numbers come from. As a result, the claim that CrossQ+WN 'nearly always beats' BRO and SIMBA at the same UTD ratio is not established to the same standard as the method's own within-paper results.
  3. [Section 5.3, Figure 4] The 'reliable scaling' claim is supported by IQM curves and 90% CIs, but no significance test is reported. Visual inspection shows overlapping confidence intervals for adjacent UTD ratios in several environments (e.g., DMC UTD=10 and UTD=20 in the aggregated curves), and the per-environment curves in Figure 7 exhibit high variance. The statement that 'with increasing compute, the performance increases or stays constant' is a deterministic claim about a stochastic comparison. A paired bootstrap test across seeds/environments, or a per-environment sign test, would be needed to substantiate 'reliably scales.' Without such a test, the paper's primary title claim is under-supported.
minor comments (5)
  1. [Section 4, Eq. (2)] The norm constraint c is introduced but never ablated; the paper fixes the projected weight norm to unit norm (c=1). Since c is a hyperparameter of the method, a sensitivity analysis would be informative.
  2. [Section 6] The sentence 'WN has been shown to keep ELRs stable and prevent loss of plasticity [29], when combined with LN...' appears twice near the end of the second and fourth paragraphs; this duplicate should be removed.
  3. [Section 5.4, Figure 5 caption] The caption contains the typo 'penality' and the text does not specify the range of regularization scales swept for the soft L2 baseline; providing this detail would improve reproducibility.
  4. [Section 5.1] The paper does not report total compute or memory usage, despite the checklist claiming sufficient compute-resource disclosure; the paper only mentions GPU types.
  5. [Section 4, Implementation details] The phrase 'we use AdamW with a decay of 0 (which falls back to vanilla Adam) for the normalized intermediate dense layers and 1e-2 otherwise' is ambiguous: 'otherwise' should specify which parameters receive weight decay 0.01 (presumably the final critic layer and the actor). Table 1 confirms this, but the text could be clearer.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central empirical claim is benchmarked against external baselines, the WN motivation is adopted from external theorems, and the self-citation of CrossQ is a base-algorithm reference rather than a load-bearing justification.

full rationale

The paper's main contribution is empirical: CrossQ+WN is evaluated against BRO, SIMBA, TD-MPC2, and SR-SAC on 25 external continuous-control tasks, with results reported as IQM and stratified bootstrap confidence intervals over 10 seeds. The proposed modification—weight normalization on the first two linear layers plus target networks evaluated in training mode on the joined state-action batch—is not derived by fitting a parameter to the target quantity and then renaming the fit a prediction. The theoretical statements (Theorems 1 and 2) are explicitly attributed to Van Laarhoven [43] and are used only to motivate why a constant weight norm may stabilize the effective learning rate; they are not used to derive the scaling result. CrossQ [4] is prior work by overlapping authors, but it serves as the base algorithm and as a baseline, not as the justification for the new empirical claim; the new claim remains externally falsifiable through the provided benchmark comparisons and official baseline data. The target-network design is empirically ablated (Figure 5) and acknowledged as important, but the statistics-sharing mechanism is not isolated in a dedicated experiment. That is a validation gap, not a circular reduction: no equation in the paper makes a predicted quantity equal to its input by construction. The limitations section also explicitly concedes that the theory does not directly connect to convergence rates, which further confirms that the scaling claim is empirical rather than derived from the cited theorems.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim is empirical and rests on a small set of hyperparameters (WN norm, weight decay, number of normalized layers, target momentum) and on design assumptions about BN statistics with target networks and the comparability of official baseline data. No new theoretical entities are introduced.

free parameters (4)
  • WN norm constraint c = 1 (unit norm)
    The L2 norm of weights in the first two layers is constrained to unit norm after each gradient step. This constant is chosen by hand; it controls the effective learning rate but is not tuned per task.
  • Weight decay coefficient = 1e-2
    Applied via AdamW to the unbounded final dense layer (and to actor/temperature for CrossQ+WN per Table 1). Chosen by hand without per-task tuning.
  • Number of WN-normalized layers = 2
    WN is applied to the first two linear layers only; the final layer is unconstrained except for weight decay. This choice is motivated by Lyle et al. but is an ad hoc design decision.
  • Target network momentum tau = 0.005
    Standard Polyak averaging coefficient; re-introduction of target networks is part of the method and this value is a hyperparameter.
assumptions (5)
  • standard math Scale invariance of BN-normalized networks
    Known result from Van Laarhoven, proved in Appendix A. Used to justify that WN can constrain weight norms without changing the critic's outputs.
  • standard math Gradient scaling inversely proportional to parameter norm under scale invariance
    Known result from Van Laarhoven, proved in Appendix B. Used to argue that constant norm maintains a constant effective learning rate.
  • domain assumption Effective learning rate (ELR) framework governs training dynamics
    The paper's mechanism relies on the ELR framework from Van Hasselt et al. and Lyle et al. If ELR does not govern plasticity in this setting, the explanation fails, though the method might still work.
  • ad hoc to paper Joined forward pass with shared training-mode BN statistics handles distribution mismatch for target values
    Section 4 states that both critic and target critic are evaluated in training mode on the joined batch. This is a specific design assumption not proven theoretically; the paper notes it requires careful consideration following Bhatt et al.
  • domain assumption Official BRO and SIMBA evaluation data are directly comparable to the authors' runs
    Section 5.1 states that for computational reasons they take official evaluation data from BRO and SIMBA. This assumes identical evaluation protocol, seeds, and environment versions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization." pith.science (2026). https://pith.science/paper/DCGAVBLV

@misc{pith2026250207523,
  author       = {Pith},
  title        = {Pith review of: Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DCGAVBLV}},
  note         = {Machine review of arXiv:2502.07523}
}
read the original abstract

Reinforcement learning has achieved significant milestones, but sample efficiency remains a bottleneck for real-world applications. Recently, CrossQ has demonstrated state-of-the-art sample efficiency with a low update-to-data (UTD) ratio of 1. In this work, we explore CrossQ's scaling behavior with higher UTD ratios. We identify challenges in the training dynamics, which are emphasized by higher UTD ratios. To address these, we integrate weight normalization into the CrossQ framework, a solution that stabilizes training, has been shown to prevent potential loss of plasticity and keeps the effective learning rate constant. Our proposed approach reliably scales with increasing UTD ratios, achieving competitive performance across 25 challenging continuous control tasks on the DeepMind Control Suite and Myosuite benchmarks, notably the complex dog and humanoid environments. This work eliminates the need for drastic interventions, such as network resets, and offers a simple yet robust pathway for improving sample efficiency and scalability in model-free reinforcement learning.

Figures

Figures reproduced from arXiv: 2502.07523 by the authors.

Figure 1
Figure 1. CrossQ + WN UTD=2 outperforms SIMBA UTD=2 and BRO UTD=2. In comparison, our proposed CrossQ + WN is a simple algorithm that, unlike BRO, does not require extra exploration policies or full parameter resets. We present results for 25 complex continuous control tasks from the DMC and MyoSuite benchmarking suites. 1.0 marks the maximum score achievable on the respective benchmarks (DMC return up to 1000 / MyoSuite up t… view at source ↗
Figure 2
Figure 2. Comparing performance against wall clock time, measured in environment steps per second [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Growing parameter norms hinder learning. The performance benefits of CrossQ fail to scale to more complex, higher dimensional tasks such as humanoid locomotion and muscular manipulation. Investigating this, we find that the critic parameter norms increase significantly with increasing UTD ratios. As a result, the effective learning rate (ELR) drops and the number of dead neurons increases. Regularizing the critic pa… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: CrossQ WN UTD scaling behavior. We plot the IQM return and 90% confidence intervals for different UTD ratios. Results are aggregated over 15 DMC environments and 10 random seeds each according to Agarwal et al. [1]. The sample efficiency scales reliably with increasing…
Figure 5
Figure 5. Figure 5: An ablation study comparing CrossQ + WN against a soft L2 penality on the weights, as well as other design decisions such as target networks. The results show that the hard constraint outperforms the soft approach across a range of regularization scales and tasks. Unce…
Figure 6
Figure 6. Figure 6: CrossQ WN UTD scaling behavior. We plot the IQM return and 90% stratified bootstrapped confidence intervals for different UTD ratios. The results are aggregated over 15 DMC environments and 10 random seeds each according to Agarwal et al. [1]. The sample efficiency sca…
Figure 7
Figure 7. Figure 7: CrossQ WN + UTD=5 against baselines. We compare our proposed CrossQ + WN UTD=5 against two baselines, BRO [31] and SR-SAC UTD=32. Results are reported on all 15 DMC and 10 MyoSuite tasks. We plot the IQM and 90% stratified bootstrapped confidence intervals over 10 rand…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies

    cs.LG 2025-08 conditional novelty 4.0 of 10

    A survey that categorizes deep reinforcement learning scaling strategies into data, network, and training budget dimensions and outlines challenges for scaling DRL systems.

Reference graph

Works this paper leans on

62 extracted references · 44 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deep reinforcement learning at the edge of the statistical precipice

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 2021

    work page 2021

  2. [2]

    Ash and Ryan P

    Jordan T. Ash and Ryan P. Adams. On warm-starting neural network training, 2020. URL https://arxiv.org/abs/1910.08475

    arXiv 2020

  3. [3]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  4. [4]

    CrossQ: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity

    Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. CrossQ: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In International conference on learning representations, 2024

    work page 2024

  5. [5]

    Gomes, and Kilian Q

    Johan Bjorck, Carla P. Gomes, and Kilian Q. Weinberger. Towards deeper deep reinforcement learning with spectral normalization, 2022. URL https://arxiv.org/abs/2106.01151

    arXiv 2022

  6. [6]

    Gait in eight: Efficient on-robot learning for omnidirectional quadruped locomotion

    Nico Bohlinger, Jonathan Kinzel, Daniel Palenicek, Lukasz Antczak, and Jan Peters. Gait in eight: Efficient on-robot learning for omnidirectional quadruped locomotion. arXiv preprint arXiv:2503.08375, 2025

    arXiv 2025

  7. [7]

    Myosuite–a contact-rich simulation suite for musculoskeletal motor control

    Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. Myosuite–a contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205.13600, 2022

    arXiv 2022

  8. [8]

    Randomized ensembled double Q- learning: Learning fast without a model

    Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double Q- learning: Learning fast without a model. InInternational conference on learning representations, 2021

    work page 2021

Show all 62 references
  1. [9]

    Sample-efficient reinforcement learning by breaking the replay ratio barrier

    Pierluca D’Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In International conference on learning representations, 2022

    2022

  2. [10]

    Weight clipping for deep continual and reinforcement learning

    Mohamed Elsayed, Qingfeng Lan, Clare Lyle, and A Rupam Mahmood. Weight clipping for deep continual and reinforcement learning. In Reinforcement Learning Conference, 2024

    2024

  3. [11]

    Jordan, Joseph E

    Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E. Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. In International Conference on Machine Learning, 2018

    2018

  4. [12]

    Sharpness-aware min- imization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware min- imization for efficiently improving generalization. In International Conference on Learning Representations (ICLR), 2021

    2021

  5. [13]

    Soft actor-critic algorithms and applications

    Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018

    2018 arXiv

  6. [14]

    Dream to control: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representa- tions, 2020. 10

    2020

  7. [15]

    Td-mpc2: Scalable, robust world models for continuous control, 2024

    Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control, 2024. URL https://arxiv.org/abs/2310.16828

    2024 arXiv

  8. [16]

    Learning continuous control policies by stochastic value gradients

    Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in neural information processing systems, 2015

    2015

  9. [17]

    Dropout q-functions for doubly efficient reinforcement learning

    Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsu- ruoka. Dropout q-functions for doubly efficient reinforcement learning. In International conference on learning representations, 2021

    2021

  10. [18]

    Normalization techniques in training dnns: Methodology, analysis and application

    Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Normalization techniques in training dnns: Methodology, analysis and application. IEEE transactions on pattern analysis and machine intelligence, 2023

    2023

  11. [19]

    Dissecting deep rl with high update ratios: Combatting value overestimation and divergence

    Marcel Hussing, Claas V oelcker, Igor Gilitschenski, Amir-massoud Farahmand, and Eric Eaton. Dissecting deep rl with high update ratios: Combatting value overestimation and divergence. arXiv preprint arXiv:2403.05996, 2024

    2024 arXiv

  12. [20]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015

    2015 arXiv

  13. [21]

    When to trust your model: Model-based policy optimization

    Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in neural information processing systems, 2019

    2019

  14. [22]

    Deepmellow: Remov- ing the need for a target network in deep q-learning

    Seungchan Kim, Kavosh Asadi, Michael Littman, and George Konidaris. Deepmellow: Remov- ing the need for a target network in deep q-learning. In Proceedings of the Twenty-Eighth Inter- national Joint Conference on Artificial Intelligence, IJCAI-19, pages 2733–2739. Internationa...

    2019 doi

  15. [23]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  16. [24]

    JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 2021

    Ilya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 2021. URL https://github.com/ikostrikov/jaxrl

    2021

  17. [25]

    Plastic: Improving input and label plasticity for sample efficient reinforcement learning, 2023

    Hojoon Lee, Hanseul Cho, Hyunseung Kim, Daehoon Gwak, Joonkee Kim, Jaegul Choo, Se-Young Yun, and Chulhee Yun. Plastic: Improving input and label plasticity for sample efficient reinforcement learning, 2023. URL https://arxiv.org/abs/2306.10711

    2023 arXiv

  18. [26]

    Simba: Simplicity bias for scaling up parameters in deep reinforcement learning

    Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subrama- nian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. International Conference on Learning Represent...

    2025

  19. [27]

    Efficient deep reinforcement learning requires regulating overfitting

    Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine. Efficient deep reinforcement learning requires regulating overfitting. In International conference on learning representations, 2023

    2023

  20. [28]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  21. [29]

    Normalization and effective learning rates in reinforcement learning

    Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, Hado van Hasselt, Razvan Pascanu, and s Will Dabney. Normalization and effective learning rates in reinforcement learning. In Neural information processing systems, 2024

    2024

  22. [30]

    Grokking deep reinforcement learning

    Miguel Morales. Grokking deep reinforcement learning. 2020

    2020

  23. [31]

    Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control

    Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miło´s, and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. In Advances in neural information processing systems, 2024

    2024

  24. [32]

    The primacy bias in deep reinforcement learning

    Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, 2022. 11

    2022

  25. [33]

    Multi-goal reinforcement learning: Challenging robotics environments and request for research, 2018

    Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for resear...

    2018 arXiv

  26. [34]

    Markov decision processes: discrete stochastic dynamic programming

    Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  27. [35]

    Weight normalization: A simple reparameterization to accelerate training of deep neural networks.Advances in Neural Information Processing Systems (NeurIPS), 2016

    Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks.Advances in Neural Information Processing Systems (NeurIPS), 2016

    2016

  28. [36]

    Bigger, better, faster: Human-level atari with human-level efficiency,

    Max Schwarzer, Johan Obando-Ceron, Aaron Courville, Marc Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency,

  29. [37]

    Understanding and improving convolutional neural networks via concatenated rectified linear units, 2016

    Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units, 2016. URL https: //arxiv.org/abs/1603.05201

    2016 arXiv

  30. [38]

    Integrated architectures for learning, planning, and reacting based on approximating dynamic programming

    Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Machine learning, 1990

    1990

  31. [39]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2018

    2018

  32. [40]

    Deepmind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  33. [41]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International conference on intelligent robots and systems, 2012

    2012

  34. [42]

    When to use parametric models in reinforcement learning? In Advances in Neural Information Processing Systems, 2019

    Hado P Van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? In Advances in Neural Information Processing Systems, 2019

    2019

  35. [43]

    L2 regularization versus batch and weight normalization

    Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350, 2017

    2017 arXiv

  36. [44]

    Mad-td: Model-augmented data stabilizes high update ratio rl, 2025

    Claas A V oelcker, Marcel Hussing, Eric Eaton, Amir massoud Farahmand, and Igor Gilitschen- ski. Mad-td: Model-augmented data stabilizes high update ratio rl, 2025. URL https: //arxiv.org/abs/2410.08896

    2025 arXiv

  37. [45]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems (NeurIPS), 2019. A Proof Scale Invariance Proof of Theorem 1. f (X; cw, cb, γ, β) = g(cXw + cb) − µ(g(cXw + cb)) σ(g(cXw + cb)) γ + β (5) = cg(Xw + b) − cµ(g(X...

    2019

  38. [47]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We have made sure that our claims reflect our contributions. Guidelines: • The answer NA means that the abstract and i...

  39. [48]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We have a dedicated conclusion and limitations discussion at the end of the paper. Guidelines: • The answer NA means that the paper has no limitation...

  40. [49]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] 17 Justification: Our theory is adopted from prior work and the proofs are provided in the appendix. Guidel...

  41. [50]

    To aid reproducibility, we plan to release the code together with the camera-ready version of the paper

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  42. [51]

    We plan to release the code together with the publication of the paper

    Open access to data and code 18 Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: At the current time we do...

  43. [52]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We have parts explaining impl...

  44. [53]

    Results are aggregated over multiple environments and 10 seeds each

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer:[Yes] Justification: We provide IQM and 90$ stratified bootstrap confide...

  45. [54]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We reference the type o...

  46. [55]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: As this paper did not involve human participants or real-world datasets or expe...

  47. [56]

    As actor-critic methods already enjoy a long history, there is no additional societal impact with this research contribution

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: The paper examines performance improvements of existing actor-critic meth- ods. As actor-critic methods al...

  48. [57]

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] , Justification: [NA] Guidelin...

  49. [58]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite and link...

  50. [59]

    Guidelines: • The answer NA means that the paper does not release new assets

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: We do not provide any new assets. Guidelines: • The answer NA means that the paper does not release new assets. • Res...

  51. [60]

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  52. [61]

    • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  53. [62]

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

  54. [2023]

    URL https://arxiv.org/abs/2305.19452

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.