Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For PPO, more parallel environments beat longer rollouts at the same data budget.

desk verdict Useful empirical study of Nenvs vs NRO in PPO, but the headline comparison is confounded by GAE horizon changes and the abstract oversells the generality. read the letter →

arxiv 2506.03404 v1 pith:OQUQLVBZ submitted 2025-06-03 cs.LG cs.AI

Walter Mayor , Johan Obando-Ceron , Aaron Courville , Pablo Samuel Castro This is my paper
classification cs.LGcs.AI
keywords on-policyreinforcementlearningPPOparallelenvironmentsrolloutlengthdatadiversitynetworkplasticityoptimizationstabilitysampleefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how an on-policy reinforcement learning agent should spend its data collection budget: split it across more parallel environments or stretch it into longer rollouts. Using PPO on ten Atari games, plus procedurally generated and continuous-control environments, it reports that at a fixed batch size $B = N_{\text{envs}} \times N_{\text{RO}}$, increasing the environment count and shortening the rollout consistently beats the reverse. The paper ties these performance gains to concrete signs of network health: higher feature rank, fewer dormant neurons, lower weight norm and gradient kurtosis, lower policy variance, and higher effective sample size. It also finds that larger batches can undo the performance damage of training more epochs over the same data. If the result holds, the practical guidance is that parallelism is the right scaling lever for on-policy RL, not rollout length.

What carries the argument

The central object is the decomposition of the training batch into environment count $N_{\text{envs}}$ and rollout length $N_{\text{RO}}$. The argument turns on two competing mechanisms: more environments widen the support of starting states and environment stochasticity, improving state-action coverage, while longer rollouts reduce bias in return estimates but add variance and temporal correlation. The paper connects these data-collection choices to optimization stability through a family of diagnostic metrics, including feature rank, dormant-neuron fraction, weight norm, gradient kurtosis, policy variance, and effective sample size, and uses those metrics to explain why more environments make training both more effective and more stable.

What would settle it

Run the fixed-budget comparison on the same games with the return-estimation window held fixed across both configurations (for example, computing advantage estimates from a fixed number of future steps separately from the rollout length). If final performance becomes roughly equal, the effect is driven by the changed rollout horizon, not by the number of parallel environments.

Watch

Extended reading notes

Core claim

The central claim is that the batch decomposition $B = N_{\text{envs}} \times N_{\text{RO}}$ is not a neutral description of a data budget: how the product is split between parallel environment count and rollout length matters. At a fixed budget, moving from $N_{\text{envs}}=8, N_{\text{RO}}=128$ to $N_{\text{envs}}=128, N_{\text{RO}}=8$ improves final scores on an Atari benchmark, and when the budget is increased, doubling $N_{\text{envs}}$ produces larger gains than doubling $N_{\text{RO}}$. The paper further claims that these gains come with measurable changes in network health: higher feature rank, fewer dormant neurons, lower weight norm, lower gradient kurtosis, lower policy variance, and higher effective sample size, which it reads as evidence that more parallel environments increase state-action coverage and stabilize optimization rather than merely adding data. A value-based counterpart, PQN, shows milder gains, which the paper attributes qualitatively to the difference between policy-based and value-based losses.

Load-bearing premise

The fixed-budget comparisons hold the product of environments and rollout length constant, but they do not hold the number of future steps used in return estimation constant, so the benefit credited to more environments could partly come from a changed bias-variance trade-off in return estimates rather than from data diversity alone.

Editorial extensions

If this is right

  • With a fixed data budget, reallocating budget from rollout length to environment count should raise final performance without spending more environment interactions.
  • Doubling the number of parallel environments can offset the performance drop caused by doubling the number of training epochs on the same batch.
  • Agents trained with more environments should show higher feature rank, fewer dormant neurons, lower weight norm and gradient kurtosis, and lower policy variance, indicating healthier optimization dynamics.
  • The scaling guidance should transfer beyond Atari to procedurally generated games and high-throughput continuous-control tasks.
  • Separate actor and critic encoders should benefit from larger parallel data collections, but only when the data budget is scaled as well.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit practical takeaway is that when hardware is the bottleneck, parallel simulation capacity is a better investment than longer trajectories, because the results imply more environments buy more performance per sample.
  • Because rollout length is entangled with the advantage-estimation horizon at fixed $N_{\text{envs}} \times N_{\text{RO}}$, a cleaner test would hold the advantage window constant while sweeping the two factors to see whether coverage alone drives the effect.
  • The milder PQN gains suggest the coverage benefit may interact with the loss function, so sweeping environment count on other value-based algorithms would test whether the guidance is universal or specific to policy-gradient methods.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper empirically studies the effect of parallelized data collection on deep RL agents, specifically the trade-off between the number of parallel environments (Nenvs) and the rollout length (NRO) in PPO and PQN. The authors compare configurations with a fixed data budget (B = Nenvs * NRO) and with scaled data budgets, using CleanRL implementations on Atari-10, Procgen, and Isaac Gym. They report that increasing Nenvs is generally more effective than increasing NRO, yielding higher returns and improved network plasticity metrics (feature rank, dormant neurons, weight norm, gradient kurtosis). They also investigate the effect of increasing the number of epochs and of architectural choices. The central claim is that scaling Nenvs over NRO is more effective for fixed or scaling data budgets.

Significance. The study addresses a practically important question for RL practitioners using GPU-vectorized environments: how to allocate a data collection budget between parallel environments and rollout length. The evaluation is reasonably thorough, with 5 seeds, IQM aggregation with bootstrap CIs, multiple Atari games, and additional benchmarks. The plasticity metrics are clearly defined in the appendix, and the code is based on the widely used CleanRL framework, which favors reproducibility. If the central claim were cleanly established, the paper would provide actionable guidance. However, due to the confound described below, the causal interpretation of the central result is not yet secure, and the generalization to PQN is overstated given the reported results.

major comments (3)
  1. [Sec. 4.2, Fig. 1(a)] The fixed-budget comparison varies NRO from 128 to 8 while holding B constant, which changes the GAE truncation horizon used in PPO. With the reported γ=0.99 and λ=0.95, the 8-step horizon yields substantially more biased advantage estimates than the 128-step horizon. The paper acknowledges this bias-variance trade-off in Sec. 3.2 but does not control for it. The observed advantage of (Nenvs=128, NRO=8) over (Nenvs=8, NRO=128) could therefore be driven by the shorter advantage-estimation horizon rather than by increased data diversity. Please add a control experiment that isolates the data-diversity effect, for example by using a fixed effective GAE horizon across configurations, or by evaluating the same configurations with full Monte Carlo returns. Alternatively, temper the central claim to acknowledge this confound.
  2. [Table 1 / Fig. 12] The default discount factor is inconsistent: Table 1 lists γ=0.99 for PPO and PQN, while the Fig. 12 caption states “0.95 (Default)”. This inconsistency affects the interpretation of the robustness analysis in Sec. 5.3, where the authors claim that the gains from NRO×2 are unaffected by varying γ. Please correct the error and ensure that all reported hyperparameters match the code and experiments.
  3. [Abstract and Sec. 6.1] The abstract and introduction claim that larger datasets and scaling Nenvs improve performance “across a variety of settings”, but Sec. 6.1 reports that PQN shows no significant performance improvements with larger Nenvs (Fig. 14). While the text acknowledges this in Sec. 6.1 and the Discussion, the high-level claims are overstated. Please revise the abstract and discussion to specify that the benefits are primarily demonstrated for PPO, with PQN showing only minor improvements in learning dynamics and no significant returns gains.
minor comments (4)
  1. [Sec. 4.1] The paper says it uses default hyperparameters, but in Sec. 4.2 and 5.3 it changes epochs, learning rate, and gamma; please clarify which defaults are used for which figure, especially for the epoch scaling experiments.
  2. [App. B.7] The coverage metric (Cvg) relies on a UMAP projection of the batch data; the grid size G and UMAP hyperparameters (n_neighbors, min_dist, etc.) are not reported. Please provide them for reproducibility.
  3. [Fig. 5] The text states that decoupled architectures benefit more from scaling parallel data, but Fig. 5 shows only four games; please specify whether these are representative and whether aggregated IQM results are available.
  4. [Sec. 3.2] The notation B is defined as the batch size but later used for the product Nenvs×NRO; consider clarifying the distinction between per-update batch size and total environment steps.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical measurements with metrics defined independently of the performance results.

full rationale

The paper is an empirical study, not a derivation. Its central conclusion—that for a fixed data budget it is more effective to scale Nenvs over NRO—comes from direct measurements reported in Sec. 4.2, comparing (Nenvs=8, NRO=128) with (Nenvs=128, NRO=8) at fixed B=Nenvs*NRO and reporting human-normalized IQM scores. There is no fitted parameter or statistical model whose output is then called a prediction. The explanatory metrics in Sec. 5 (feature rank, dormant neurons, weight norm, gradient kurtosis, ESS, policy variance) are defined in Appendix B by explicit formulas that do not take the performance results as inputs; they are used as independent correlates of the observed outcomes. The PPO objective is taken from the standard external reference (Schulman et al., 2017), and no uniqueness theorem or ansatz is imported from the authors' own prior work to force the conclusion. Although the paper cites its own prior work for metric definitions and related phenomena (e.g., Sokar et al., 2023; Ceron et al., 2024), these citations are not load-bearing for the reported comparisons. The GAE-horizon confound noted by a skeptical reader—that varying NRO changes the advantage estimation horizon—is a legitimate correctness or identification concern, but it is not circularity: the measured outcome is not defined in terms of the explanatory variables, and the paper's conclusion does not reduce to its inputs by construction. The paper is self-contained against external benchmarks and reports measured outcomes, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted; the analysis uses CleanRL default hyperparameters as fixed experimental settings. No new theoretical entities are introduced. The central claim rests on standard RL background and on the validity of plasticity and coverage metrics as measurement tools.

assumptions (4)
  • domain assumption The metrics feature rank, dormant neuron percentage, weight norm, and gradient kurtosis are valid and sufficient proxies for network plasticity and optimization stability.
    The paper's explanation of why Nenvs scaling helps relies on these metrics, but their validity as plasticity proxies is taken from prior work (Sokar et al., Moalla et al.) without independent verification here.
  • domain assumption The Atari-10 game subset is representative of the full Atari benchmark.
    The paper uses Atari-10, citing Aitchison et al., to draw general conclusions about Atari performance; if the subset is not representative, the claim's scope shrinks.
  • domain assumption UMAP projections preserve enough structure for the coverage metric (Cvg) to meaningfully compare state-space coverage across configurations.
    Cvg is computed on a 2D UMAP embedding; the reliability of this coverage comparison is not independently validated.
  • domain assumption The CleanRL PPO implementation and its default hyperparameters are an appropriate and standard baseline.
    All experiments use CleanRL defaults; conclusions about scaling may not transfer to other PPO implementations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks." pith.science (2026). https://pith.science/paper/OQUQLVBZ

@misc{pith2026250603404,
  author       = {Pith},
  title        = {Pith review of: The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OQUQLVBZ}},
  note         = {Machine review of arXiv:2506.03404}
}
read the original abstract

The use of parallel actors for data collection has been an effective technique used in reinforcement learning (RL) algorithms. The manner in which data is collected in these algorithms, controlled via the number of parallel environments and the rollout length, induces a form of bias-variance trade-off; the number of training passes over the collected data, on the other hand, must strike a balance between sample efficiency and overfitting. We conduct an empirical analysis of these trade-offs on PPO, one of the most popular RL algorithms that uses parallel actors, and establish connections to network plasticity and, more generally, optimization stability. We examine its impact on network architectures, as well as the hyper-parameter sensitivity when scaling data. Our analyses indicate that larger dataset sizes can increase final performance across a variety of settings, and that scaling parallel environments is more effective than increasing rollout lengths. These findings highlight the critical role of data collection strategies in improving agent performance.

Figures

Figures reproduced from arXiv: 2506.03404 by the authors.

Figure 1
Figure 1. (a) Collecting the data with a large Nenvs is more effective; both experiments contain the same amount of data. (b) Scaling parallel data collection improve sample efficiency and final performance. Scaling data by increasing the Nenvs is more effective than NRO. (c) Scaling parallel data collection diminishes performance degradation as the number of epochs increases. Performance collapse is better mitigated by incre… view at source ↗
Figure 2
Figure 2. shows how varying the number of parallel environ￾ments (Nenvs) and rollouts per environment (NRO) affects the coverage of the learned state distribution. Increasing Nenvs enhances spatial diversity by exposing the policy to a wider range of initial conditions and environment stochas￾ticity, while increasing NRO improves temporal depth but may induce stronger trajectory correlations. Configurations with higher Nenvs … view at source ↗
Figure 3
Figure 3. Empirical analyses for four representative games with different amount of parallel data for PPO (Schulman et al., 2017). From left to right: training returns, feature rank, dormant neurons percentage (Sokar et al., 2023), weight norm and gradient kurtosis. All results averaged over 5 seeds, shaded areas represent 95% confidence intervals. See Sec. 4 for training details. Dormant Neurons We measure the fraction of ne… view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Increasing Nenvs for parallel data collection leads to lower policy variance and higher ESS, which results in a higher average episodic return. All results averaged over 5 seeds, shaded areas represent 95% confidence intervals. See Sec. 4 for training details. Key obse…
Figure 5
Figure 5. Figure 5: When using separate networks to represent the policy and value function (Cobbe et al., 2021), scaling parallel data collection improves final performance. Using decoupled architectures with default settings collapses performance; this collapse is mitigated by scaling N…
Figure 6
Figure 6. Figure 6: Scaling parallel data collection on Parallelised Q-network (PQN) (Gallici et al., 2024) marginally mitigates representation deterioration and slightly improves final performance. From left to right: training returns, feature rank, dormant neurons percentage (Sokar et a…
Figure 7
Figure 7. Figure 7: Proximal Policy Optimization (PPO) (Schulman et al., 2017; Huang et al., 2022) on Isaac Gym environments (Makoviychuk et al.). (Left) Scaling data by increasing the Nenvs is more effective than NRO; both experiments contains the same amount of data. (Right) Increasing …
Figure 8
Figure 8. Figure 8: we observe the same general tendency: increasing B improves performance, but it is more effective to do so by scaling Nenvs. 6.3. Parallel environments and exploration In sparse-reward or hard-exploration settings, increasing the number of parallel environments enhance…
Figure 9
Figure 9. Figure 9: State-space coverage under varying levels of parallel data collection. Increasing the Nenvs leads to improved spatial coverage across Atari games. This suggests that scaling Nenvs promotes broader exploration and diverse training data under a fixed sample budget. The c…
Figure 10
Figure 10. Figure 10: Empirical analyses for ALE games with different amount of parallel data for PPO (Schulman et al., 2017). From left to right: training returns, feature rank, dormant neurons percentage (Sokar et al., 2023), weight norm and gradient kurtosis. All results averaged over 5…
Figure 11
Figure 11. Figure 11: Varying the learning rate by a factor proportional to the increase to Nenvs. In this figure Nenvs was scaled by a factor of 2, as Hilton et al. (2022) suggest increasing the learning accordingly. In addition to that increase, we explore decreasing the learning by the …
Figure 12
Figure 12. Figure 12: Performance gains from NRO × 2 are unaffected by varying γ, where the default value isγ = 0.95. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Empirical analyses for ALE games with different amount of parallel data for PQN (Gallici et al., 2024). From left to right: training returns, feature rank, dormant neurons percentage (Sokar et al., 2023), weight norm and gradient kurtosis. All results averaged over 5 …
Figure 14
Figure 14. Figure 14: Evaluating the impact of scaling parallel data collection for PQN agent (Gallici et al., 2024) on Atari-10 benchmark (Aitchison et al., 2023). See Section 4.1 for training details. 3.0 3.1 3.2 3.3 3.4 Freeway ×10 1 Average Episodic Return 0.0 0.5 1.0 1.5 2.0 ×10 2 Fea…
Figure 15
Figure 15. Figure 15: Evaluating the impact of increasing Nenvs on a separate set of ALE games. These are the so-called “hard exploration games” from Taiga et al. (2020). We can observe that increased batch size results in equivalent or improved performance, and overall improvement on the …
Figure 16
Figure 16. Figure 16: Evaluating the impact of increasing Nenvs on a separate set of ALE games. These are the so-called “hard exploration games” from Taiga et al. (2020). Scaling data by increasing the Nenvs is more effective than NRO. All results averaged over 5 seeds, shaded areas repres…
Figure 17
Figure 17. Figure 17: Empirical analyses for four representative games with different amount of parallel data for PPO (Schulman et al., 2017). From left to right: training returns, feature rank, dormant neurons percentage (Sokar et al., 2023), weight norm and gradient kurtosis. Increasing …
Figure 18
Figure 18. Figure 18: Proximal Policy Optimization (PPO) (Schulman et al., 2017; Huang et al., 2022) on Isaac Gym environments (Makoviychuk et al.) when increasing Nenvs. Increasing Nenvs for parallel data collection improves final performance. We report returns over 5 runs for each experi…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. X-NavDP: Generalizing Navigation Diffusion Policy to Novel Behavior and Embodiments with Group Q-score Reweighted Matching

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Group Q-score reweighted matching plus self-bootstrapped trajectory perturbation lifts cross-embodiment diffusion navigation from 61% to 84% success in sim and ~10% to 65% in hard real cases.

  2. Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies

    cs.LG 2025-08 conditional novelty 4.0 of 10

    A survey that categorizes deep reinforcement learning scaling strategies into data, network, and training budget dimensions and outlines challenges for scaling DRL systems.

Reference graph

Works this paper leans on

70 extracted references · 51 canonical work pages · cited by 2 Pith papers

  1. [1]

    Legged locomotion in challenging terrains using egocentric vision

    Agarwal, A., Kumar, A., Malik, J., and Pathak, D. Legged locomotion in challenging terrains using egocentric vision. In 6th Annual Conference on Robot Learning, 2022. URL https://openreview.net/forum?id=Re3NjSwf0WF

    work page 2022

  2. [2]

    S., Courville, A

    Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., and Bellemare, M. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34: 0 29304--29320, 2021

    work page 2021

  3. [3]

    Atari-5: Distilling the arcade learning environment down to five games

    Aitchison, M., Sweetser, P., and Hutter, M. Atari-5: Distilling the arcade learning environment down to five games. In International Conference on Machine Learning, pp.\ 421--438. PMLR, 2023

    work page 2023

  4. [4]

    What matters for on-policy deep actor-critic methods? a large-scale study

    Andrychowicz, M., Raichuk, A., Sta \'n czyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., et al. What matters for on-policy deep actor-critic methods? a large-scale study. In International conference on learning representations

    work page

  5. [5]

    For valid generalization the size of the weights is more important than the size of the network

    Bartlett, P. For valid generalization the size of the weights is more important than the size of the network. Advances in neural information processing systems, 9, 1996

    work page 1996

  6. [6]

    G., Naddaf, Y., Veness, J., and Bowling, M

    Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 0 253--279, jun 2013. doi:10.1613/jair.3912

    doi:10.1613/jair.3912 2013

  7. [7]

    G., Candido, S., Castro, P

    Bellemare, M. G., Candido, S., Castro, P. S., Gong, J., Machado, M. C., Moitra, S., Ponda, S. S., and Wang, Z. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588: 0 77 -- 82, 2020

    work page 2020

  8. [8]

    A study on the plasticity of neural networks

    Berariu, T., Czarnecki, W., De, S., Bornschein, J., Smith, S., Pascanu, R., and Clopath, C. A study on the plasticity of neural networks. arXiv preprint arXiv:2106.00042, 2021

    arXiv 2021

Show all 70 references
  1. [9]

    Exploration by random network distillation

    Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Exploration by random network distillation. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=H1lJJnR5Ym

    2019

  2. [10]

    Ceron, J. S. O., Ara \'u jo, J. G. M., Courville, A., and Castro, P. S. On the consistency of hyper-parameter selection in value-based deep reinforcement learning. In Reinforcement Learning Conference, 2024 a

    2024

  3. [11]

    Ceron, J. S. O., Courville, A., and Castro, P. S. In value-based deep reinforcement learning, a pruned network is a good network. In International Conference on Machine Learning, pp.\ 38495--38519. PMLR, 2024 b

    2024

  4. [12]

    Ceron, J. S. O., Sokar, G., Willi, T., Lyle, C., Farebrother, J., Foerster, J. N., Dziugaite, G. K., Precup, D., and Castro, P. S. Mixtures of experts unlock parameter scaling for deep rl. In International Conference on Machine Learning, pp.\ 38520--38540. PMLR, 2024 c

    2024

  5. [13]

    Leveraging procedural generation to benchmark reinforcement learning

    Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. Leveraging procedural generation to benchmark reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020

    2020

  6. [14]

    W., Hilton, J., Klimov, O., and Schulman, J

    Cobbe, K. W., Hilton, J., Klimov, O., and Schulman, J. Phasic policy gradient. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ 2020--2027. PMLR, 18--24 Jul 202...

    2020

  7. [15]

    G., and Courville, A

    D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L., Bellemare, M. G., and Courville, A. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?...

    2023

  8. [16]

    Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures

    Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pp.\ 1407--1...

    2018

  9. [17]

    J., Schrittwieser, J., Swirszcz, G., et al

    Fawzi, A., Balog, M., Huang, A., Hubert, T., Romera-Paredes, B., Barekatain, M., Novikov, A., R Ruiz, F. J., Schrittwieser, J., Swirszcz, G., et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610 0 (7930): 0 47--53, 2022

    2022

  10. [18]

    N., and Martin, M

    Gallici, M., Fellows, M., Ellis, B., Pou, B., Masmitja, I., Foerster, J. N., and Martin, M. Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.04811, 2024

    2024 arXiv

  11. [19]

    On proximal policy optimization’s heavy-tailed gradients

    Garg, S., Zhanson, J., Parisotto, E., Prasad, A., Kolter, Z., Lipton, Z., Balakrishnan, S., Salakhutdinov, R., and Ravikumar, P. On proximal policy optimization’s heavy-tailed gradients. In International Conference on Machine Learning, pp.\ 3610--3619. PMLR, 2021

    2021

  12. [20]

    Dextreme: Transfer of agile in-hand manipulation from simulation to reality

    Handa, A., Allshire, A., Makoviychuk, V., Petrenko, A., Singh, R., Liu, J., Makoviichuk, D., Van Wyk, K., Zhurkevich, A., Sundaralingam, B., et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEEE International Conference on Robotics an...

    2023

  13. [21]

    R., Millman, K

    Harris, C. R., Millman, K. J., Van Der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. Array programming with numpy. Nature, 585 0 (7825): 0 357--362, 2020

    2020

  14. [22]

    Batch size-invariance for policy optimization

    Hilton, J., Cobbe, K., and Schulman, J. Batch size-invariance for policy optimization. Advances in Neural Information Processing Systems, 35: 0 17086--17098, 2022

    2022

  15. [23]

    E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R

    Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. R. Improving neural networks by preventing co-adaptation of feature detectors. arxiv 2012. arXiv preprint arXiv:1207.0580, 2012

    2012 arXiv

  16. [24]

    Distributed prioritized experience replay

    Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., van Hasselt, H., and Silver, D. Distributed prioritized experience replay. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=H1Dy---0Z

    2018

  17. [25]

    Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and Ara \ A s jo, J. G. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23 0 (274): 0 1--18, 2022

    2022

  18. [26]

    Hunter, J. D. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9 0 (03): 0 90--95, 2007

    2007

  19. [27]

    and Ash, J

    Juliani, A. and Ash, J. T. A study of plasticity loss in on-policy deep reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=MsUf8kpKTF

    2024

  20. [28]

    Towards continual reinforcement learning: A review and perspectives

    Khetarpal, K., Riemer, M., Rish, I., and Precup, D. Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research, 75: 0 1401--1476, 2022

    2022

  21. [29]

    Jupyter Notebooks a publishing format for reproducible computational workflows

    Kluyver , T., Ragan-Kelley , B., P \'e rez , F., Granger , B., Bussonnier , M., Frederic , J., Kelley , K., Hamrick , J., Grout , J., Corlay , S., Ivanov , P., Avila , D., Abdalla , S., Willing , C., and Jupyter Development Team . Jupyter Notebooks a publishing format for repr...

    2016 doi

  22. [30]

    Pgx: Hardware-accelerated parallel game simulators for reinforcement learning

    Koyamada, S., Okano, S., Nishimori, S., Murata, Y., Habara, K., Kita, H., and Ishii, S. Pgx: Hardware-accelerated parallel game simulators for reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  23. [31]

    and Hertz, J

    Krogh, A. and Hertz, J. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991

    1991

  24. [32]

    Offline q-learning on diverse multi-task data both scales and generalizes

    Kumar, A., Agarwal, R., Geng, X., Tucker, G., and Levine, S. Offline q-learning on diverse multi-task data both scales and generalizes. In The Eleventh International Conference on Learning Representations, 2022

    2022

  25. [33]

    Parallel q-learning: Scaling off-policy reinforcement learning under massively parallel simulation

    Li, Z., Chen, T., Hong, Z.-W., Ajay, A., and Agrawal, P. Parallel q-learning: Scaling off-policy reinforcement learning under massively parallel simulation. In International Conference on Machine Learning, pp.\ 19440--19459. PMLR, 2023

    2023

  26. [34]

    Acceleration for deep reinforcement learning using parallel and distributed computing: A survey

    Liu, Z., Xu, X., Qiao, P., and Li, D. Acceleration for deep reinforcement learning using parallel and distributed computing: A survey. ACM Computing Surveys, 57 0 (4): 0 1--35, 2024

    2024

  27. [35]

    A., Pascanu, R., and Dabney, W

    Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., and Dabney, W. Understanding plasticity in neural networks. In International Conference on Machine Learning, pp.\ 23190--23211. PMLR, 2023

    2023

  28. [36]

    Normalization and effective learning rates in reinforcement learning

    Lyle, C., Zheng, Z., Khetarpal, K., Martens, J., van Hasselt, H., Pascanu, R., and Dabney, W. Normalization and effective learning rates in reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net...

    2024

  29. [37]

    Isaac gym: High performance gpu based physics simulation for robot learning

    Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A., et al. Isaac gym: High performance gpu based physics simulation for robot learning. In Thirty-fifth Conference on Neural Information Processing Systems D...

  30. [38]

    Miles 430 macklin, david hoeller, nikita rudin, arthur allshire, ankur handa, and gavriel state

    Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., and Storey, K. Miles 430 macklin, david hoeller, nikita rudin, arthur allshire, ankur handa, and gavriel state. isaac 431 gym: High performance gpu based physics simulation for robot learning. Proceedings of the 432 Neural Info...

    2021

  31. [39]

    Effective sample size for importance sampling based on discrepancy measures

    Martino, L., Elvira, V., and Louzada, F. Effective sample size for importance sampling based on discrepancy measures. Signal Processing, 131: 0 386--401, 2017

    2017

  32. [40]

    Umap: Uniform manifold approximation and projection for dimension reduction

    McInnes, L., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018

    2018 arXiv

  33. [41]

    McKenzie, M. C. and McDonnell, M. D. Modern value based reinforcement learning: A chronological review. IEEE Access, 10: 0 134704--134725, 2022. doi:10.1109/ACCESS.2022.3228647

    2022

  34. [42]

    Python for Data Analysis: Data Wrangling with Pandas, NumPy , and IPython

    McKinney, W. Python for Data Analysis: Data Wrangling with Pandas, NumPy , and IPython . O'Reilly Media, 1 edition, February 2013. ISBN 9789351100065. URL http://www.amazon.com/exec/obidos/redirect?tag=citeulike07-20&path=ASIN/1449319793

    2013

  35. [43]

    A., Veness, J., Bellemare, M

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. Human-level cont...

    2015

  36. [44]

    P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K

    Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In Balcan, M. F. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Conference on Machine Learning, ...

    1928

  37. [45]

    No representation, no trust: Connecting representation, collapse, and trust issues in PPO

    Moalla, S., Miele, A., Pyatko, D., Pascanu, R., and Gulcehre, C. No representation, no trust: Connecting representation, collapse, and trust issues in PPO . In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?...

    2024

  38. [46]

    Norm-based capacity control in neural networks

    Neyshabur, B., Tomioka, R., and Srebro, N. Norm-based capacity control in neural networks. In Conference on learning theory, pp.\ 1376--1401. PMLR, 2015

    2015

  39. [47]

    The primacy bias in deep reinforcement learning

    Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learnin...

    2022

  40. [48]

    Obando Ceron, J., Bellemare, M., and Castro, P. S. Small batch deep reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  41. [49]

    Oliphant, T. E. Python for scientific computing. Computing in Science & Engineering, 9 0 (3): 0 10--20, 2007. doi:10.1109/MCSE.2007.58

    2007 doi

  42. [50]

    Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning

    Petrenko, A., Huang, Z., Kumar, T., Sukhatme, G., and Koltun, V. Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning. In International Conference on Machine Learning, pp.\ 7652--7662. PMLR, 2020

    2020

  43. [51]

    Puterman, M. L. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994. ISBN 0471619779

    1994

  44. [52]

    Learning to walk in minutes using massively parallel deep reinforcement learning

    Rudin, N., Hoeller, D., Reist, P., and Hutter, M. Learning to walk in minutes using massively parallel deep reinforcement learning. In 5th Annual Conference on Robot Learning, 2021. URL https://openreview.net/forum?id=wK2fDDJ5VcF

    2021

  45. [53]

    Learning to walk in minutes using massively parallel deep reinforcement learning

    Rudin, N., Hoeller, D., Reist, P., and Hutter, M. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pp.\ 91--100. PMLR, 2022

    2022

  46. [54]

    The phenomenon of policy churn

    Schaul, T., Barreto, A., Quan, J., and Ostrovski, G. The phenomenon of policy churn. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 2537--2549. Curran Associates, Inc., 2022

    2022

  47. [55]

    High-dimensional continuous control using generalized advantage estimation

    Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015

    2015 arXiv

  48. [56]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  49. [57]

    S., Courville, A., Bellemare, M

    Schwarzer, M., Obando Ceron, J. S., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S. Bigger, better, faster: Human-level A tari with human-level efficiency. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings o...

    2023

  50. [58]

    J., Lee, J., Antognini, J., Sohl-Dickstein, J., Frostig, R., and Dahl, G

    Shallue, C. J., Lee, J., Antognini, J., Sohl-Dickstein, J., Frostig, R., and Dahl, G. E. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research, 20 0 (112): 0 1--49, 2019

    2019

  51. [59]

    Sapg: split and aggregate policy gradients

    Singla, J., Agarwal, A., and Pathak, D. Sapg: split and aggregate policy gradients. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  52. [60]

    S., and Evci, U

    Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, pp.\ 32145--32168. PMLR, 2023

    2023

  53. [61]

    and Abbeel, P

    Stooke, A. and Abbeel, P. Accelerated methods for deep reinforcement learning. arXiv preprint arXiv:1803.02811, 2018

    2018 arXiv

  54. [62]

    Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249

    2018

  55. [63]

    S., McAllester, D., Singh, S., and Mansour, Y

    Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 13th International Conference on Neural Information Processing Systems, NIPS'99, pp.\ 1057–1063, Cambridge, MA, USA, ...

    1999

  56. [64]

    A., Fedus, W., Machado, M

    Taiga, A. A., Fedus, W., Machado, M. C., Courville, A., and Bellemare, M. G. On bonus based exploration methods in the arcade learning environment. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=BJewlyStDr

    2020

  57. [65]

    A., Agarwal, R., Farebrother, J., Courville, A., and Bellemare, M

    Taiga, A. A., Agarwal, R., Farebrother, J., Courville, A., and Bellemare, M. G. Investigating multi-task pretraining and generalization in reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=sS...

    2023

  58. [66]

    and Drake Jr, F

    Van Rossum, G. and Drake Jr, F. L. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995

    1995

  59. [67]

    M., Mathieu, M., Dudzik, A., Chung, J., Choi, D

    Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575 0 (7782): 0 350--354, 2019

    2019

  60. [68]

    Envpool: A highly parallel reinforcement learning environment execution engine

    Weng, J., Lin, M., Huang, S., Liu, B., Makoviichuk, D., Makoviychuk, V., Liu, Z., Song, Y., Luo, T., Jiang, Y., et al. Envpool: A highly parallel reinforcement learning environment execution engine. Advances in Neural Information Processing Systems, 35: 0 22409--22421, 2022

    2022

  61. [69]

    Harnessing structures for value-based planning and reinforcement learning

    Yang, Y., Zhang, G., Xu, Z., and Katabi, D. Harnessing structures for value-based planning and reinforcement learning. arXiv preprint arXiv:1909.12255, 2019

    1909 arXiv

  62. [70]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.