Pith. sign in

REVIEW 3 major objections 8 minor 54 references

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control

T0 review · 3 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FoG, a continuous-control agent that forgets old replay data and grows its critic, reaches normalized score 0.92 and beats BRO, SimBa, and TD-MPC2 across 41 tasks.

desk verdict FoG is a solid method-level empirical paper with broad evaluation and honest limitations; the central theory is overstated relative to the actual sampler, and the MuJoCo TD-MPC2 margin is shaky, but the core empirical claim is defensible and worth serious review. read the letter →

arxiv 2507.02712 v1 pith:FRHUN3OF submitted 2025-07-03 cs.LG

Zilin Kang , Chenyuan Hu , Yu Luo , Zhecheng Yuan , Ruijie Zheng , Huazhe Xu This is my paper
classification cs.LG
keywords FoGprimacybiasexperiencereplaydecaynetworkexpansioncontinuouscontroldeepreinforcementlearningmodelscalingratio
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that primacy bias—the tendency of deep RL agents to overfit the earliest transitions in their replay buffer—can be overcome by two simple mechanisms inspired by infantile amnesia: Experience Replay Decay, which gradually lowers the sampling weight of old transitions, and Network Expansion, which adds fresh critic parameters early in training. It argues that resets alone cannot fix the problem, because older transitions are still sampled many more times than newer ones. The method, FoG, combines these mechanisms with the OBAC actor-critic backbone, a scaled critic, and a higher replay ratio. The empirical claim is that FoG reaches a normalized score of 0.92 on 41 tasks across four benchmarks, ahead of BRO at 0.76, TD-MPC2 at 0.70, and SimBa at 0.69, and that network expansion unlocks favorable scaling up to about 23M parameters.

What carries the argument

The two load-bearing mechanisms are ER Decay and Network Expansion. ER Decay assigns transition $i$ at time $t$ the sampling weight $w_{t,i} = \max(\tau, (1-\epsilon)^{t-i})$, so older samples are replayed less often without vanishing entirely; this is the 'forgetting' side. Network Expansion adds blocks of two dense layers with layer normalization and residual connections to the critic at fixed early iterations, reinitializing the optimizer and scaling the learning rate by the parameter count; this is the 'growing' side. Together, on the OBAC offline-boosted actor-critic backbone with a replay ratio of 10, they reduce primacy bias, keep critic loss low on recent transitions, and lower the dormant-neuron ratio.

What would settle it

Run FoG with only ER Decay and only Network Expansion across the full 41-task suite, and also ablate the 250k OBAC wait; if either ablation changes the normalized score by less than the reported gaps, the claim that the two named mechanisms drive the gains is falsified. Separately, under the implemented weight floor $\tau$, compute the expected sample count of the oldest transition; if it grows linearly with training steps rather than staying bounded by a constant, the bounded-sample-count theorem does not hold for the actual sampler.

Watch

Extended reading notes

Core claim

The central claim, in the authors' terms, is that very simple algorithmic changes—ER decay, network expansion, and simple model-structure modifications—are enough to overcome primacy bias and make an off-policy continuous-control agent reach state-of-the-art performance. The paper reports that FoG beats BRO, SimBa, and TD-MPC2 across 41 locomotion and manipulation tasks, with the largest margins in the hardest DMC tasks and on MuJoCo, and that it does so with the same hyperparameters across benchmarks. It also reports that network expansion prevents the loss explosions and dormant-neuron buildup that fixed large critic networks suffer, and that FoG outperforms larger fixed-size baselines under comparable compute.

Load-bearing premise

The load-bearing premise is that the performance gains come from ER Decay and Network Expansion as generalizable mechanisms, not from the OBAC backbone, the hand-tuned reset lists, the 250k OBAC wait, or the higher replay ratio used in the experiments.

Editorial extensions

If this is right

  • If the paper is right, primacy bias can be mitigated by sampling-weight decay and capacity growth, so resets are not the only or best tool.
  • Fixed-size critic networks saturate around 5M parameters in earlier work; FoG's expansion mechanism extends favorable scaling to about 23M parameters.
  • A single hyperparameter set, apart from a reset list for four simple HumanoidBench tasks, transfers across locomotion, manipulation, and sparse-reward benchmarks.
  • FoG's lower dormant-neuron ratio and more structured t-SNE features suggest the gains come with improved plasticity and representation quality, not merely more compute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension the paper leaves open is whether the two mechanisms transfer to other backbones; the paper's own appendix reports that FoG-BRO does not beat BRO, so a fair test is to apply ER Decay and Network Expansion to a model-based method or to TD-MPC2.
  • Because the implemented sampler uses a minimum weight floor $\tau$, the constant-sample-count theorem proven for pure exponential decay may not cover the actual sampler; measuring the oldest transition's expected sample count over long training would show how much of the bound survives.
  • The amnesia analogy suggests decay should be tied to how fast the data distribution shifts; a version with adaptive $\epsilon$ keyed to buffer loss or policy change could be tested against the fixed $\epsilon=10^{-5}$ used here.
  • Network expansion could be triggered by plasticity signals, such as the dormant-neuron ratio crossing a threshold, instead of at the fixed 50k and 200k iterations, potentially allowing the method to scale to longer horizons.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. FoG is a continuous-control deep RL algorithm built on OBAC, scaled critic networks (up to 23M parameters), and a replay ratio of 10. It adds two mechanisms motivated by infantile amnesia: ER Decay, which down-weights older transitions in the replay buffer, and Network Expansion, which grows the critic with additional residual blocks at 50k and 200k updates. The paper reports experiments on 41 tasks from MuJoCo, DMControl, Meta-World, and HumanoidBench, achieving a normalized total score of 0.92 ± 0.01, ahead of BRO (0.76), SimBa (0.69), and TD-MPC2 (0.70). Ablations indicate that both ER Decay and Network Expansion contribute to the gains, and the paper provides theorems intended to characterize the sampling behavior of replay buffers with and without decay.

Significance. The empirical core is substantial and likely reproducible: the evaluation spans four suites with official baselines and includes ablations, learning curves, and a released implementation. The normalized-score improvements are consistent across suites, though the margin over BRO is largest on DMC-Hard and smallest on HumanoidBench. The two mechanisms are simple and clearly described, making the method easy to adopt. If the results hold, FoG provides a competitive alternative to model-free and model-based baselines and demonstrates a practical way to combine forgetting and network growth to mitigate primacy bias. The main weaknesses are the mismatch between Theorem 3.2 and the implemented sampler, the incomplete isolation of the two mechanisms from other FoG-specific components, and the limited generality across backbones. These weaknesses do not invalidate the empirical finding but require revision before the theoretical and conceptual claims can be taken as stated.

major comments (3)
  1. [Section 3.2 / Appendix A.2] Theorem 3.2 is stated for the implemented ER Decay sampler, but the proof in Appendix A.2 analyzes a floorless exponential decay, while Section 3.2 defines the sampling weight as w_{t,i} = max(τ, (1-ε)^(t-i)). With the reported hyperparameters ε=1e-5 and τ=0.1, the floor is active for any transition older than roughly 230k steps, and for a fixed old transition the denominator S_t grows linearly, so the expected sample count grows as β ln T rather than being bounded by a constant C. Therefore the theorem as stated does not hold for the actual algorithm and cannot be used to justify the claim that ER Decay 'balances memory' through a constant bound. The authors should either restrict the theorem to the floorless case (where the constant bound is valid) or re-analyze the floored sampler and revise the interpretation accordingly.
  2. [Section 4.2 / Figure 2] The central claim that 'the forget-and-grow approach is crucial for FoG's superior performance' is only partially supported because the comparison to 'OBAC+scaling' in Figure 2 is not defined in the main text, and FoG includes additional modifications beyond ER Decay and Network Expansion, specifically the OBAC wait of 250k iterations, task-specific reset lists, and a pessimism coefficient that is set to 0 in Meta-World. The ablations in Figures 8 and 9 toggle ER Decay and Network Expansion within FoG, but they do not remove these other components, so it is not established that the two named mechanisms alone account for the gap over OBAC+scaling. Please clarify what OBAC+scaling includes and provide an ablation that isolates ER Decay and Network Expansion from the OBAC wait and reset list changes.
  3. [Appendix B.4] The paper's own cross-backbone results show that FoG-BRO performs no better than BRO and FoG-SAC does not outperform BRO, which limits the generality of ER Decay and Network Expansion as mechanisms that mitigate primacy bias beyond the OBAC backbone. The manuscript explicitly acknowledges this in Appendix B.4, but the title and introduction frame the contribution as a general strategy for deep reinforcement learning scaling. The authors should temper the generalizability claim to the OBAC/SAC family or provide at least one successful combination with a strong, independently developed backbone to support the broader framing.
minor comments (8)
  1. [Appendix A.1 / Theorem 3.1] Theorem 3.1's lower bound appears misprinted: the derivation yields ln(N/(t-1)) + 1/N - 1, not ln(N/(t-1)) + 1/(N-1).
  2. [Section 4.1 / Appendix B.1] Section 4.1 states FoG 'with identical hyperparameters' achieves high performance across all benchmarks, but Appendix B.1 reports a different pessimism coefficient for Meta-World and a different reset list for four HumanoidBench tasks; please clarify.
  3. [Appendix B.2] The description 'decay the learning rate by the number of parameters' is imprecise; the formula uses the ratio of dense-layer counts, not parameter counts, and should be stated as such.
  4. [Figure 4 / Section 3.2] Figure 4's simulation uses ε=1e-4 and τ=0.01, while the default hyperparameters in Table 1 are ε=1e-5 and τ=0.1; either use the default values in the figure or note that the figure is illustrative.
  5. [Figure 11 / Section 4.2] The dormant-ratio threshold is not specified; please state how dormant neurons are defined (e.g., maximum absolute activation or gradient below a threshold).
  6. [Abstract] The phrase 'dual processes' should be 'two processes' or 'dual process' for grammatical correctness.
  7. [Figure 2 caption] The caption of Figure 2 refers to 'OBAC+scaling' without defining it; define it in the caption or main text.
  8. [Section 3.3 / Appendix B.1] The reset list is mentioned only in the list of key ideas in Section 3.3, while the actual reset schedules appear only in Appendix B.1, making it hard to reproduce the algorithm from the main text; consider moving the default reset list to the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FoG's central comparisons are external and its ablations isolate the claimed mechanisms; the Theorem 3.2 proof-scope mismatch is a correctness gap, not a circular reduction.

full rationale

The claimed derivation chain is empirical rather than deductive: FoG's headline result is a measured normalized score across 41 tasks against external baselines using their official implementations (Appendix B.5), and the ablations compare FoG with fixed-size variants, PER, and CUER to isolate the two mechanisms. The only same-group dependency is the OBAC backbone (Luo et al., 2024), but the paper explicitly reports that plain OBAC+scaling performs at the level of SimBa/BRO while FoG improves on it, so the cited OBAC result is not doing the work of the FoG claim. The theorem in Section 3.2 is a genuine analytic statement about the floorless exponential sampler; the implemented floored sampler is a different object (Section 3.2: w_{t,i}=max(τ,(1−ϵ)^{t−i}) versus Appendix A.2 proof using (1−ϵ)^t without a floor). That mismatch is a correctness gap in the theoretical motivation, not a circularity: the final empirical scores are not constructed from the theorem, and no fitted parameter is relabeled as a prediction. The paper's own limitations (Section 5 and Appendix B.4) concede that the mechanisms rest on empirical and intuitive support and do not transfer to BRO, which supports treating the headline as an external empirical finding rather than an assumption restated.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central empirical claim depends primarily on hand-chosen hyperparameters (ε, τ, expansion schedule, OBAC wait, reset lists) and on the OBAC backbone from the authors' own prior work. The theory section contributes only heuristic bounds under a simplified sampling model that does not match the implemented algorithm.

free parameters (7)
  • ER Decay rate ε = 1e-5 (main); 1e-4 (motivating example)
    Controls how quickly older transitions lose sampling weight; no sensitivity analysis reported; different values used in the motivating example vs the final algorithm.
  • Minimum buffer weight τ = 0.1 (main); 1e-1 (motivating example)
    Floor on transition sampling weight that prevents old transitions from vanishing; interacts with Theorem 3.2, which ignores it.
  • Critic expansion schedule = 50k and 200k iterations, 2 layers at a time; reset to depth 2 on reset
    Chosen by hand; expansion iters and amount are not swept.
  • OBAC wait = 250k network iterations
    Protection period after reset during which OBAC is disabled; authors report 'found to yield fine results when set to 250k in all cases' without sensitivity analysis.
  • Reset lists = 15k/50k/250k/500k/750k for 4 simple HumanoidBench tasks; 15k/50k/100k/200k/400k/600k/800k otherwise
    Two hand-chosen reset schedules; per-benchmark variation is itself a tuning choice.
  • Replay ratio and batch size = 10 and 256
    High replay ratio setting chosen to stress primacy bias; not swept in FoG.
  • Pessimism coefficient = 0 for MetaWorld, 1 otherwise
    Uses Moskovitz et al. (2022) pessimistic Q-setting; chosen per benchmark without ablation.
assumptions (5)
  • standard math Harmonic number bounds ln n + 1/n < H_n < ln n + 1
    Used in the proof of Theorem 3.1 (Appendix A.1) to sandwich E[nt]/β.
  • standard math Geometric series sum Σ_{t≥0} (1-ε)^t = 1/ε
    Used in the proof of Theorem 3.2 (Appendix A.2) to bound E[n1] < β/ε.
  • domain assumption Replay sampling model: one transition added per step and β samples drawn per step, uniform over the current buffer
    Theorems 3.1 and 3.2 assume this idealized online buffer; real buffers with resets, variable batch processing, and the OBAC offline component deviate from it (Section 3.1).
  • domain assumption Primacy bias is the dominant mechanism limiting performance in high-replay-ratio continuous control
    The paper's motivating hypothesis; supported by ablations and heatmaps, but not proven against alternative explanations such as recency-weighted optimization or increased network capacity (Sections 3.1, 3.2).
  • ad hoc to paper Theorem 3.2 analyzes pure exponential decay without the max(τ, (1-ε)^(t-i)) floor used in the implemented ER Decay
    The proof in Appendix A.2 omits the floor; with the floor, transitions at the τ level are sampled with near-constant probability and the constant-bound conclusion does not carry over to the actual algorithm (Section 3.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control." pith.science (2026). https://pith.science/paper/FRHUN3OF

@misc{pith2026250702712,
  author       = {Pith},
  title        = {Pith review of: A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FRHUN3OF}},
  note         = {Machine review of arXiv:2507.02712}
}
read the original abstract

Deep reinforcement learning for continuous control has recently achieved impressive progress. However, existing methods often suffer from primacy bias, a tendency to overfit early experiences stored in the replay buffer, which limits an RL agent's sample efficiency and generalizability. In contrast, humans are less susceptible to such bias, partly due to infantile amnesia, where the formation of new neurons disrupts early memory traces, leading to the forgetting of initial experiences. Inspired by this dual processes of forgetting and growing in neuroscience, in this paper, we propose Forget and Grow (FoG), a new deep RL algorithm with two mechanisms introduced. First, Experience Replay Decay (ER Decay) "forgetting early experience", which balances memory by gradually reducing the influence of early experiences. Second, Network Expansion, "growing neural capacity", which enhances agents' capability to exploit the patterns of existing data by dynamically adding new parameters during training. Empirical results on four major continuous control benchmarks with more than 40 tasks demonstrate the superior performance of FoG against SoTA existing deep RL algorithms, including BRO, SimBa, and TD-MPC2.

Figures

Figures reproduced from arXiv: 2507.02712 by the authors.

Figure 1
Figure 1. Overview. Top: we illustrate two key components of our strategy: ER Decay and Network Expansion. Bottom: comparison of normalized score. FoG outperforms popular model-based and model-free methods including TD-MPC2, SimBa and BRO. 1 arXiv:2507.02712v1 [cs.LG] 3 Jul 2025 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Normalized scores of algorithms on DMC-Hard tasks (5 hardest Dog & Humanoid tasks). The performance of the OBAC+scaling method is comparable to that of SimBa and BRO, but when combined with FoG, it achieves superior results. However, in the field of deep reinforcement learning, agents typically do not have a natural mechanism to forget their early training experiences. Instead, they often overfit to initial data, wh… view at source ↗
Figure 3
Figure 3. Learning Curves and Heat Maps of SAC Variants. Top Left: Return curves of various SAC variants in humanoid-walk. Top Center: Critic loss heatmap of Normal SAC in humanoid-walk. Top Right: Critic loss heatmap of SAC with reset in the humanoid-walk environment. Bottom Left: Return curves of various SAC variants in the HalfCheetah-v4. Bottom Center: Critic loss heatmap of Balanced SAC in humanoid-walk. Bottom Right: Cr… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Sample times of transitions in a normal buffer and a decayed buffer with ϵ = 1e − 4,τ = 0.01 over 100k steps [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Changes in critic loss over time for PER(left) and ER decay(right) in humanoid-walk. The darker the color near the diagonal, the less influenced by primacy bias; conversely, the darker the color towards the top-right corner indicates greater influence from primacy bias…
Figure 6
Figure 6. Figure 6: Network expansion illustration. We initialize networks with fewer parameters and progressively add a new block (in the frame) to residual connections at each expansion step. performance in the early stages of training, it tends to en￾counter early convergence issues in…
Figure 7
Figure 7. Figure 7: Main results. We provide performance comparisons for 16 of the 41 tasks, four for each task suite. Please refer to Appendix C for the comprehensive results. The solid lines are the average return/success rate, while the shades indicate 95% confidence intervals. All alg…
Figure 8
Figure 8. Figure 8: Choice of experience replay methods. We adopt 4 tasks from Mujoco and DMControl, two for each task suite, to compare different experience replay methods. Mean of 3 runs; shaded areas are 95% confidence intervals [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Ablation on network expansion. We adopt 4 tasks from Mujoco and DMControl, two for each task suite, to showcase the necessity of network expansion. Mean of 3 runs; shaded areas are 95% confidence intervals [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 11
Figure 11. Figure 11: Dormant ratio during training. We measure the ra￾tio of dormant neurons during training on HalfCheetah-v4 and humanoid-walk. Red lines indicate time steps where network expansion happens. signals, causing the loss to explode. However, network ex￾pansion helps to suppr…
Figure 12
Figure 12. Figure 12: T-SNE visualization of representations. We visual￾ize the representations via t-SNE after training 0.2M steps on HalfCheetah-v4. From left to right are the t-SNE results of 2 blocks, 4 blocks and expansion from 2 to 4 blocks. FoG effectively reduces the number of dorm…
Figure 13
Figure 13. Figure 13: The results of SAC and SAC with ER decay 4 tasks in Mujoco. We also evaluated the data efficiency of the FoG-enhanced SAC (FoG-SAC) algorithm in both DMC-Hard and MuJoCo environments. We found that FoG-SAC achieved data efficiency close to that of BRO, showing signifi…
Figure 14
Figure 14. Figure 14: The results of FoG-SAC 1. BRO: We use the official implementation from https://github.com/naumix/BiggerRegularizedOptimistic. 2. SimBa: We use SimBa-SAC from official implementation and results from https://github.com/SonyResearch/simba. 3. TD-MPC2: We use the officia…
Figure 15
Figure 15. Figure 15: The results of 4 tasks in Mujoco. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: The results of 4 tasks in DM Control Easy [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: The results of 4 tasks in DM Control Medium [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: The results of 5 tasks in DM Control Hard. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_18.png]
Figure 19
Figure 19. Figure 19: The results of 5 tasks in Meta-World. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 20
Figure 20. Figure 20: The results of 14 tasks in HumanoidBench. D. Performance comparison under similar computation cost To demonstrate that the performance gains of FoG are not merely due to increased computation, we provide a performance comparison under similar computational costs. We e…
Figure 21
Figure 21. Figure 21: The results under similar computation cost E. Comparison to similar methods We conducted a comparative analysis between FoG and the Neuroplastic Expansion (NE) algorithm(Liu et al., 2024), utilizing the official NE implementation to ensure fairness. The evaluation was…
Figure 22
Figure 22. Figure 22: Comparison to Neuroplastic Expansion method 22 [PITH_FULL_IMAGE:figures/full_fig_p022_22.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 20 canonical work pages

  1. [1]

    G., Martinez-Canabal, A., Restivo, L., Yiu, A

    Akers, K. G., Martinez-Canabal, A., Restivo, L., Yiu, A. P., De Cristofaro, A., Hsiang, H.-L., Wheeler, A. L., Guskjolen, A., Niibori, Y., Shoji, H., et al. Hippocampal neurogenesis regulates forgetting during adulthood and infancy. Science, 344 0 (6184): 0 598--602, 2014

    work page 2014

  2. [2]

    Alberini, C. M. and Travaglia, A. Infantile amnesia: a critical period of learning to learn and remember. Journal of Neuroscience, 37 0 (24): 0 5783--5795, 2017

    work page 2017

  3. [3]

    Hindsight experience replay, 2018

    Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay, 2018. URL https://arxiv.org/abs/1707.01495

    arXiv 2018

  4. [4]

    Towards Deeper Deep Reinforcement Learning with Spectral Normalization

    Bjorck, J., Gomes, C. P., and Weinberger, K. Q. Towards deeper deep reinforcement learning with spectral normalization, 2022. URL https://arxiv.org/abs/2106.01151

    work page Pith review arXiv 2022

  5. [5]

    Randomized ensembled double q-learning: Learning fast without a model, 2021

    Chen, X., Wang, C., Zhou, Z., and Ross, K. Randomized ensembled double q-learning: Learning fast without a model, 2021. URL https://arxiv.org/abs/2101.05982

    arXiv 2021

  6. [6]

    Reinforcement Learning with Combinatorial Actions: An Application to Vehicle Routing

    Delarue, A., Anderson, R., and Tjandraatmadja, C. Reinforcement learning with combinatorial actions: An application to vehicle routing, 2020. URL https://arxiv.org/abs/2010.12001

    work page Pith review arXiv 2020

  7. [7]

    G., and Courville, A

    D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L., Bellemare, M. G., and Courville, A. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022

    work page 2022

  8. [8]

    Revisiting fundamentals of experience replay

    Fedus, W., Ramachandran, P., Agarwal, R., Bengio, Y., Larochelle, H., Rowland, M., and Dabney, W. Revisiting fundamentals of experience replay. In International conference on machine learning, pp.\ 3061--3071. PMLR, 2020

    work page 2020

Show all 54 references
  1. [9]

    Addressing function approximation error in actor-critic methods

    Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp.\ 1587--1596. PMLR, 2018

    2018

  2. [10]

    Off-policy deep reinforcement learning without exploration, 2019

    Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration, 2019. URL https://arxiv.org/abs/1812.02900

    2019 arXiv

  3. [11]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018. URL https://arxiv.org/abs/1801.01290

    2018 arXiv

  4. [12]

    Mastering diverse domains through world models, 2024

    Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models, 2024. URL https://arxiv.org/abs/2301.04104

    2024 arXiv

  5. [13]

    B., Friesen, A

    Hamrick, J. B., Friesen, A. L., Behbahani, F., Guez, A., Viola, F., Witherspoon, S., Anthony, T., Buesing, L., Veličković, P., and Weber, T. On the role of planning in model-based deep reinforcement learning, 2021. URL https://arxiv.org/abs/2011.04021

    2021 arXiv

  6. [14]

    Td-mpc2: Scalable, robust world models for continuous control, 2024

    Hansen, N., Su, H., and Wang, X. Td-mpc2: Scalable, robust world models for continuous control, 2024. URL https://arxiv.org/abs/2310.16828

    2024 arXiv

  7. [15]

    Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically, 2017. URL https://arxiv.org/abs/1712.00409

    2017 arXiv

  8. [16]

    Josselyn, S. A. and Frankland, P. W. Infantile amnesia: a neurogenic hypothesis. Learning & Memory, 19 0 (9): 0 423--433, 2012

    2012

  9. [17]

    Offline q-learning on diverse multi-task data both scales and generalizes, 2023 a

    Kumar, A., Agarwal, R., Geng, X., Tucker, G., and Levine, S. Offline q-learning on diverse multi-task data both scales and generalizes, 2023 a . URL https://arxiv.org/abs/2211.15144

    2023 arXiv

  10. [18]

    Kumar, S., Marklund, H., and Roy, B. V. Maintaining plasticity in continual learning via regenerative regularization, 2023 b . URL https://arxiv.org/abs/2308.11958

    2023 arXiv

  11. [19]

    Maxmin q-learning: Controlling the estimation bias of q-learning, 2021

    Lan, Q., Pan, Y., Fyshe, A., and White, M. Maxmin q-learning: Controlling the estimation bias of q-learning, 2021. URL https://arxiv.org/abs/2002.06487

    2021 arXiv

  12. [20]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.\ 1207--1216, Stanford, CA, 2000. Morgan Kaufmann

    2000

  13. [21]

    J., Subramanian, K., Wurman, P

    Lee, H., Hwang, D., Kim, D., Kim, H., Tai, J. J., Subramanian, K., Wurman, P. R., Choo, J., Stone, P., and Seno, T. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint arXiv:2410.09754, 2024

    arXiv

  14. [22]

    Multi-game decision transformers, 2022

    Lee, K.-H., Nachum, O., Yang, M., Lee, L., Freeman, D., Xu, W., Guadarrama, S., Fischer, I., Jang, E., Michalewski, H., and Mordatch, I. Multi-game decision transformers, 2022. URL https://arxiv.org/abs/2205.15241

    2022 arXiv

  15. [23]

    Efficient deep reinforcement learning requires regulating overfitting, 2023

    Li, Q., Kumar, A., Kostrikov, I., and Levine, S. Efficient deep reinforcement learning requires regulating overfitting, 2023. URL https://arxiv.org/abs/2304.10466

    2023 arXiv

  16. [24]

    Self-improving reactive agents based on reinforcement learning, planning and teaching

    Lin, L.-J. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8: 0 293--321, 1992

    1992

  17. [25]

    Neuroplastic expansion in deep reinforcement learning

    Liu, J., Obando-Ceron, J., Courville, A., and Pan, L. Neuroplastic expansion in deep reinforcement learning. arXiv preprint arXiv:2410.07994, 2024

    2024 arXiv

  18. [26]

    Offline-boosted actor-critic: Adaptively blending optimal historical behaviors in deep off-policy rl

    Luo, Y., Ji, T., Sun, F., Zhang, J., Xu, H., and Zhan, X. Offline-boosted actor-critic: Adaptively blending optimal historical behaviors in deep off-policy rl. arXiv preprint arXiv:2405.18520, 2024

    2024 arXiv

  19. [27]

    Off-policy rl algorithms can be sample-efficient for continuous control via sample multiple reuse, 2023

    Lyu, J., Wan, L., Lu, Z., and Li, X. Off-policy rl algorithms can be sample-efficient for continuous control via sample multiple reuse, 2023. URL https://arxiv.org/abs/2305.18443

    2023 arXiv

  20. [28]

    Revisiting plasticity in visual reinforcement learning: Data

    Ma, G., Li, L., Zhang, S., Liu, Z., Wang, Z., Chen, Y., Shen, L., Wang, X., and Tao, D. Revisiting plasticity in visual reinforcement learning: Data. Modules and Training Stages, 2023

    2023

  21. [29]

    Learning better with less: effective augmentation for sample-efficient visual reinforcement learning

    Ma, G., Zhang, L., Wang, H., Li, L., Wang, Z., Wang, Z., Shen, L., Wang, X., and Tao, D. Learning better with less: effective augmentation for sample-efficient visual reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  22. [30]

    A., Veness, J., Bellemare, M

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518 0 (7540): 0 529--533, 2015

    2015

  23. [31]

    Moskovitz, T., Parker-Holder, J., Pacchiano, A., Arbel, M., and Jordan, M. I. Tactical optimism and pessimism for deep reinforcement learning, 2022. URL https://arxiv.org/abs/2102.03765

    2022 arXiv

  24. [32]

    Safe and efficient off-policy reinforcement learning

    Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. Safe and efficient off-policy reinforcement learning. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2...

    2016

  25. [33]

    Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning, 2024 a

    Nauman, M., Bortkiewicz, M., Miłoś, P., Trzciński, T., Ostaszewski, M., and Cygan, M. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning, 2024 a . URL https://arxiv.org/abs/2403.00514

    2024 arXiv

  26. [34]

    Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control

    Nauman, M., Ostaszewski, M., Jankowski, K., Mi o \'s , P., and Cygan, M. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. arXiv preprint arXiv:2405.16158, 2024 b

    2024 arXiv

  27. [35]

    The primacy bias in deep reinforcement learning

    Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learnin...

    2022

  28. [36]

    The primacy bias in deep reinforcement learning

    Nikishin, E., Schwarzer, M., D’Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp.\ 16828--16847. PMLR, 2022 b

    2022

  29. [37]

    Deep reinforcement learning with plasticity injection, 2023

    Nikishin, E., Oh, J., Ostrovski, G., Lyle, C., Pascanu, R., Dabney, W., and Barreto, A. Deep reinforcement learning with plasticity injection, 2023. URL https://arxiv.org/abs/2305.15555

    2023 arXiv

  30. [38]

    F., Maximo, M

    Prudencio, R. F., Maximo, M. R., and Colombini, E. L. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  31. [39]

    The primacy bias in model-based rl

    Qiao, Z., Lyu, J., and Li, X. The primacy bias in model-based rl. arXiv preprint arXiv:2310.15017, 2023

    2023 arXiv

  32. [40]

    Prioritized experience replay, 2016

    Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay, 2016. URL https://arxiv.org/abs/1511.05952

    2016 arXiv

  33. [41]

    Schwarzer, M., Obando-Ceron, J., Courville, A., Bellemare, M., Agarwal, R., and Castro, P. S. Bigger, better, faster: Human-level atari with human-level efficiency, 2023. URL https://arxiv.org/abs/2305.19452

    2023 arXiv

  34. [42]

    Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation, 2024

    Sferrazza, C., Huang, D.-M., Lin, X., Lee, Y., and Abbeel, P. Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation, 2024. URL https://arxiv.org/abs/2403.10506

    2024 arXiv

  35. [43]

    D2rl: Deep dense architectures in reinforcement learning

    Sinha, S., Bharadhwaj, H., Srinivas, A., and Garg, A. D2rl: Deep dense architectures in reinforcement learning. arXiv preprint arXiv:2010.09163, 2020

    2010 arXiv

  36. [44]

    S., and Evci, U

    Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, pp.\ 32145--32168. PMLR, 2023

    2023

  37. [45]

    Model-based off-policy deep reinforcement learning with model-embedding

    Tan, X., Qu, C., Xiong, J., Zhang, J., Qiu, X., and Jin, Y. Model-based off-policy deep reinforcement learning with model-embedding. IEEE Transactions on Emerging Topics in Computational Intelligence, 8 0 (4): 0 2974--2986, 2024. doi:10.1109/TETCI.2024.3369636

    2024

  38. [46]

    Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  39. [47]

    Mujoco: A physics engine for model-based control

    Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp.\ 5026--5033. IEEE, 2012

    2012

  40. [48]

    Deep reinforcement learning and the deadly triad, 2018

    van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. Deep reinforcement learning and the deadly triad, 2018. URL https://arxiv.org/abs/1812.02648

    2018 arXiv

  41. [49]

    Drm: Mastering visual reinforcement learning through dormant ratio minimization

    Xu, G., Zheng, R., Liang, Y., Wang, X., Yuan, Z., Ji, T., Luo, Y., Liu, X., Yuan, J., Hua, P., et al. Drm: Mastering visual reinforcement learning through dormant ratio minimization. arXiv preprint arXiv:2310.19668, 2023

    2023 arXiv

  42. [50]

    Yang, Y., Ding, Z., Wang, R., Modares, H., and Wunsch, D. C. Data-driven human-robot interaction without velocity measurement using off-policy reinforcement learning. IEEE/CAA Journal of Automatica Sinica, 9 0 (1): 0 47--63, 2022. doi:10.1109/JAS.2021.1004258

    2022

  43. [51]

    S., Mutlu, F

    Yenicesu, A. S., Mutlu, F. B., Kozat, S. S., and Oguz, O. S. Cuer: Corrected uniform experience replay for off-policy continuous deep reinforcement learning algorithms, 2024. URL https://arxiv.org/abs/2406.09030

    2024 arXiv

  44. [52]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp.\ 1094--1100. PMLR, 2020

    2020

  45. [53]

    and Sutton, R

    Zhang, S. and Sutton, R. S. A deeper look at experience replay, 2018. URL https://arxiv.org/abs/1712.01275

    2018 arXiv

  46. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.