Pith. sign in

REVIEW 3 major objections 4 minor 84 references

Parseval Regularization for Continual Reinforcement Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding a Parseval orthogonality regularizer to policy and value networks substantially improves continual RL agents' ability to keep learning across task sequences.

desk verdict Useful empirical contribution with a real statistics problem in the significance claims—worth reviewing, but the error bars need fixing. read the letter →

arxiv 2412.07224 v1 pith:MOL3M43N submitted 2024-12-10 cs.LG cs.AI

Wesley Chung , Lynn Cherif , David Meger , Doina Precup This is my paper · ORCID
classification cs.LGcs.AI
keywords ParsevalregularizationcontinualreinforcementlearningplasticitylossorthogonalweightmatricesdynamicalisometrypolicygradientmethodsMetaWorld20-10benchmarknonstationaryenvironments
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that continual RL agents lose the ability to learn new tasks because training moves their weights away from the well-conditioned region where orthogonal initialization places them. To counter that drift, it adds Parseval regularization—a term penalizing $\|WW^\top - sI\|_F^2$ on every hidden dense layer of the policy and value networks—so the weight matrices stay close to orthogonal throughout training. On a 15-by-15 gridworld, two CARL environments, and a new 20-sequence, 10-task MetaWorld benchmark, the regularized agent outperforms the base RPO/PPO agent and also layer norm, shrink-and-perturb, and regenerative regularization. The paper finds that both parts of the regularizer matter, that the angle term helps more than the norm term, and that adding a few diagonal-layer parameters restores the expressiveness that orthogonality removes. If the claim holds, a simple, low-overhead regularizer is a broadly effective remedy for plasticity loss in nonstationary RL.

What carries the argument

The central object is the Parseval regularization loss $L_{\text{Parseval}}(W)=\lambda\|WW^\top - sI\|_F^2$, added to every hidden dense layer of the actor and critic. It encourages the rows of $W$ to be mutually orthogonal and to have squared $\ell^2$-norm $s$, pushing all singular values toward $\sqrt{s}$, which mimics the dynamical isometry of orthogonal initialization during training. Because exact orthogonality forces the network to be Lipschitz and limits capacity, the paper pairs the regularizer with a small fix: a learnable diagonal layer (or a learnable input scale) after each regularized layer, which relaxes the Lipschitz constraint at linear parameter cost. The regularizer carries the argument by keeping the loss landscape well conditioned across task changes, and the paper tests the mechanism by ablating its norm and angle components, varying activation functions and widths, and tracking stable rank, neuron cosine similarity, and Jacobian entry spread.

What would settle it

Rerun the MetaWorld20-10 comparison using sequence-level statistics—one summary number per seed per sequence—and resample over the 20 sequences with a bootstrap; if the Parseval advantage over the base agent and layer norm evaporates under this resampling, the central claim of significant improvement is refuted. As a second check, run a Parseval-regularized agent on a sequence where the base agent already learns every task; if it shows no speedup there, the regularizer's value is confined to plasticity-loss regimes rather than training in general.

Watch

Extended reading notes

Core claim

The central claim is that keeping weight matrices near-orthogonal during training—not just at initialization—preserves the optimization geometry that makes deep networks easy to update, and that this preservation translates directly into faster, better learning on new RL tasks. The paper establishes this empirically: Parseval regularization greatly improves the base agent on all four task suites, and its performance profiles dominate the layer norm, shrink-and-perturb, and regenerative-regularization baselines. Ablations show that regularizing only the angles between weight vectors already helps, regularizing only their norms helps less, and the full regularizer works best. Network measurements show that Parseval keeps the stable rank of the weight matrices near full and the cosine similarity between neuron weight vectors near zero, while baseline agents' ranks collapse and their neurons become correlated; it also tightens the spread of input-output Jacobian entries. The paper concludes that these properties are worthwhile targets for continual RL and that the optimization benefits of orthogonality outweigh the loss of network expressiveness.

Load-bearing premise

The statistical comparison treats each task in a sequence as an independent datapoint when building DKW confidence bands, but tasks within a sequence share the same agent and seed, so those datapoints are correlated; if that correlation invalidates the bands, the reported significance of Parseval's improvement is weaker than it appears.

Editorial extensions

If this is right

  • Practitioners can add Parseval regularization to an existing PPO/RPO agent with little code and modest runtime overhead—measured at 1.8% to 11.4%—and expect improved learning after task changes.
  • The norm and angle components of the regularizer act separately, so future methods can target weight-scale drift or direction-collapse independently depending on which bottleneck dominates.
  • Stable rank, neuron cosine similarity, and the spread of input-output Jacobian entries can serve as practical training-time diagnostics for plasticity loss.
  • The benefit persists across tanh, ReLU, Mish, CReLU, and MaxMin activations and across network widths, indicating the mechanism is architectural rather than activation-specific.
  • Because Parseval regularization sets no reset schedule, it offers a timescale-free alternative to weight-reset and reinitialization methods for nonstationary environments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not claim a full causal theory of plasticity loss; an immediate extension would be to test whether the same regularizer delays plasticity loss in supervised continual learning benchmarks, where task changes are more drastic.
  • The contrast between subgroup-Parseval results (more groups hurt) and the initialization-rank experiments (lower initial rank sometimes helps) suggests that orthogonality matters mainly when maintained throughout training, which is testable by intervening at different times.
  • A selective variant that applies Parseval only to layers whose stable rank drops fastest, guided by the paper's diagnostics, could deliver most of the benefit at lower compute.
  • The near-zero neuron correlation maintained by Parseval resembles feature-diversity mechanisms, so one could test whether the regularizer also improves adaptation when task changes are signaled rather than hidden.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes applying Parseval regularization, which penalizes W W^T - sI in Frobenius norm, to the policy and value networks of an RPO/PPO agent in order to maintain near-orthogonal weights during continual reinforcement learning and thereby mitigate plasticity loss. The authors evaluate their method against the base agent and three plasticity-oriented baselines on a gridworld navigation task, two CARL environments (LunarLander and DMCQuadruped), and a new MetaWorld benchmark consisting of 20 fixed sequences of 10 tasks with 3 seeds each. They report performance profiles with DKW-based simultaneous confidence bands and claim that Parseval regularization significantly improves the base agent. The paper also contains ablations separating the angle and norm components of the regularizer, studies of diagonal layers and input scaling, experiments with different activations and widths, and diagnostic analyses of stable rank, neuron weight correlation, the input-output Jacobian, policy entropy, and initialization properties.

Significance. If the empirical claim holds, this is a simple, low-cost, and broadly applicable intervention for continual RL: it adds a few lines to the loss, has modest computational overhead (reported as 1.8-11.4% runtime), and appears to help across several task suites. The paper's strengths are its breadth of evaluation, the careful decomposition of the regularizer into angle and norm effects, and the connection of performance to interpretable network diagnostics such as stable rank and cosine similarity. The regularizer is defined independently of the measured outcomes, so there is no circularity in the central claim. The main weakness is statistical: the 'significant benefits' claim rests on confidence bands computed under an iid assumption that the task-level datapoints do not satisfy, and the benchmark-specific hyperparameter selection is not accounted for. With a run-level reanalysis the qualitative findings may survive, but the current evidence as presented overstates the precision of the reported improvements.

major comments (3)
  1. [Section 4 (Performance profiles); Appendix C.2] The DKW simultaneous confidence bands require an iid sample, but the datapoints used to form the MetaWorld performance profiles are not independent. Each 10-task sequence is produced by a single agent/seed with shared weights and optimizer state, so the 10 task summaries within a sequence are correlated; the same issue affects the CARL and gridworld profiles. The effective number of independent replicates for MetaWorld is at most 60 (20 sequences x 3 seeds), not 600, which widens the band by roughly a factor of sqrt(10) (from about 0.05 to about 0.158 for a 90% band). The abstract's 'significant benefits' claim is therefore not established by the reported statistics. Please recompute the simultaneous bands with run-level summaries (e.g., per-run average success), a clustered bootstrap that resamples whole sequences, or a mixed-effects model with task nested in run, and report whether the Parseval versus base comparisons remain significant under that analysis.
  2. [Section 4 / Appendix C.5] The regularization strength for Parseval and the hyperparameters of the comparison algorithms were selected by small sweeps on the same benchmark used for evaluation ('small sweeps were conducted on relevant hyperparameters and the best setting was chosen'). This selection on the evaluation distribution can inflate the reported gains, especially when the paper makes a general claim about broad effectiveness. Please report the exact selected hyperparameters per algorithm and environment (including the chosen Parseval regularization strength, which is not given in Table 2), describe the selection protocol, and provide a sensitivity analysis over the swept lambda values, or validate on a held-out set of task sequences. If the qualitative conclusions are robust across the swept range, that would materially strengthen the paper.
  3. [Section 3.1, Figs. 1-2, Table 2] The architecture used in the headline MetaWorld comparison should be stated unambiguously. Table 2 lists 'Additional parameters: Diag Layer' for MetaWorld, and the text says diagonal layers will be used in all later MetaWorld experiments 'for consistency,' which I read as applying to all algorithms; if that is correct, the comparison is matched. However, Fig. 1's legend simply distinguishes 'Parseval' from 'Base,' and Fig. 2 shows that adding the diagonal layer hurts the no-Parseval base while helping the Parseval agent. Because the auxiliary capacity interacts with the regularizer, the paper should explicitly state in Section 4.1 that the main result is for the diagonal-layer architecture and should point to the no-extra-parameter comparison ('With Parseval, No Extra Parameters' versus 'No Parseval, No Extra Parameters') as evidence that the regularizer alone, without the additional capacity, also helps.
minor comments (4)
  1. [Section 3] The displayed final objective appears to be missing a summation sign before the Parseval term: 'L(θ) = Lp(θ) + λv Lv(θ) + λ P(# layers)-1 k=1 L_Parseval(Wk)' should read λ ∑_{k=1}^{#layers−1} L_Parseval(W_k). In addition, the definition of L_Parseval(W) already includes λ, so the final objective appears to multiply λ twice unless one of the two occurrences is a typographical convention.
  2. [Section 4.4] The displayed stable-rank formula, srank(A) = ∑ σ_i^2 / max_i σ_i, is dimensionally inconsistent with the stated property that it equals n when all singular values are equal; the denominator should be max_i σ_i^2 (or the expression should be normalized accordingly). Please confirm that the formula matches what was actually computed.
  3. [Appendix C.3] The code listings for the CARL sequence generation contain truncated or corrupted lines (e.g., 'nonl ocal i', incomplete 'if i %' blocks), which prevents the reader from reproducing the exact context sequences from the text. Please provide complete, runnable code in the supplement.
  4. [Figs. 1 and 4] The figure legends and captions should state that the MetaWorld runs use a diagonal layer for all algorithms and that the DMCQuadruped runs use an input scale, per Table 2. This would avoid the impression of an architecture mismatch between Parseval and the baselines.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central empirical comparison is self-contained, and the network-analysis metrics are confirmatory sanity checks rather than fitted predictions.

full rationale

No circular step was found. The paper's central claim is empirical: adding Parseval regularization L_Parseval(W)=lambda||W W^T - s I||_F^2 to the policy and value losses improves continual RL performance on gridworld, CARL, and MetaWorld. The regularizer is defined independently of the measured outcomes, and the performance comparisons are against the base RPO/PPO agent and standard baselines (layer norm, shrink-and-perturb, regenerative regularization). The analysis metrics (stable rank, neuron cosine similarity, input-output Jacobian spread) are explicitly linked to the regularizer's intended effect: orthogonal rows imply equal singular values, hence near-full stable rank and near-zero weight-vector correlation. The paper presents these as confirmatory measurements of the mechanism, not as the evidence for the performance claim; the performance claim rests on performance profiles and learning curves. The skeptical concern about DKW confidence bands being applied to non-iid task-level datapoints is a statistical-support issue (tasks within a sequence share an agent and seed), not an instance of a result being defined into existence or a fitted value being renamed as a prediction. Hyperparameters and the MetaWorld task subset were selected on the evaluation benchmark, which can inflate measured gains, but that is benchmark-selection bias rather than circular reasoning. No load-bearing self-citation or imported uniqueness theorem appears: citations to Parseval networks, RPO, CleanRL, and the baseline methods are external sources, and the paper's own contribution is not justified by those citations. The derivation chain is self-contained in the sense that the regularization term and the evaluation protocol are specified ex ante, and the reported benefits are empirical findings rather than consequences of the definitions.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper rests on standard RL assumptions and on the representativeness of the chosen benchmarks; no new entities are introduced. The main unstated statistical premise is the validity of the DKW confidence bands for dependent samples.

free parameters (3)
  • Parseval regularization strength lambda = swept over {1e-2, 1e-3, 1e-4, 1e-5}; best per benchmark chosen
    The central results depend on this hyperparameter; selecting the best value on the same benchmark is a form of fitting (Section 4, Base agent; Appendix C.5).
  • Scaling factor s = sqrt(2) (default)
    Controls the target row norm; set by default, not tuned, but still a free choice affecting the regularizer's behavior (Section 3).
  • Entropy regularization coefficient = 0.0 for MetaWorld, 0.01 for Gridworld, 0.0001 for CARL
    These coefficients are part of the base agent hyperparameters and differ per environment; they affect policy entropy and results (Appendix C.5, Table 2).
assumptions (4)
  • standard math The DKW inequality provides valid simultaneous confidence bands for the performance profile.
    Applied in Appendix C.2, but the datapoints are not iid because tasks within a sequence share the same agent trajectory, violating the DKW assumption.
  • domain assumption Tasks in the benchmark are sampled and the same fixed sequences are shown to all agents.
    The paper assumes that the fixed set of 20 sequences and the selected environments are representative of continual RL; different sequences might change conclusions (Section 4, Metaworld20-10).
  • domain assumption Success rate averaged over the learning curve is a faithful summary of task performance.
    The paper uses the mean success rate over the learning curve per task; different aggregations (e.g., final performance, area under curve) might change profiles (Section 4).
  • domain assumption RPO/PPO from CleanRL is a suitable base agent and the hyperparameter adjustments do not disadvantage baselines.
    The base agent and tuned hyperparameters are used for all algorithms; if the base is weaker than typical, regularizer benefits may be inflated (Section 4, Base agent).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Parseval Regularization for Continual Reinforcement Learning." pith.science (2026). https://pith.science/paper/MOL3M43N

@misc{pith2026241207224,
  author       = {Pith},
  title        = {Pith review of: Parseval Regularization for Continual Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MOL3M43N}},
  note         = {Machine review of arXiv:2412.07224}
}
read the original abstract

Loss of plasticity, trainability loss, and primacy bias have been identified as issues arising when training deep neural networks on sequences of tasks -- all referring to the increased difficulty in training on new tasks. We propose to use Parseval regularization, which maintains orthogonality of weight matrices, to preserve useful optimization properties and improve training in a continual reinforcement learning setting. We show that it provides significant benefits to RL agents on a suite of gridworld, CARL and MetaWorld tasks. We conduct comprehensive ablations to identify the source of its benefits and investigate the effect of certain metrics associated to network trainability including weight matrix rank, weight norms and policy entropy.

Figures

Figures reproduced from arXiv: 2412.07224 by the authors.

Figure 1
Figure 1. Performance of algorithms on Metaworld tasks. The tasks change every 1 million steps, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparing performance profiles of di￾agonal layers and learnable input scales on Meta￾world sequences. Either addition helps with Parse￾val regularization. By restricting the weights to be orthogonal, the network may be overly constrained. In partic￾ular, if the weight matrices of all the layers are orthogonal and the activation function is Lip￾schitz, then the function given by the neural network is also Lipschitz.… view at source ↗
Figure 3
Figure 3. The left plot shows performance profiles of Parseval regularization on Metaworld sequences [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Performance of algorithms on gridworld and CARL environments. Parseval regularization [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Performance profiles for different architecture choices. (Left and center) Varying activation [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: These plots present learning curves on a single task. On the left, we compare different [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Performance profile for two settings of the initialization scale. With Parseval regularization, [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: The first plot shows learning curves for different values of entropy regularization. The [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: The distribution of the squared entries of the input-output Jacobian along training for the [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: The cosine similarity of row vectors of the weight matrices for the first two layers. Parseval [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: The cosine stable rank of the weight matrices for the first two layers. Parseval regularization [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Parameter norms when using subgroups for Parseval regularization. Subgroups:64 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Performance of linear neural networks compared to using activation functions. The linear [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Gridworld layout. There are nine rooms with doorways indicated by the orange lines. [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

84 extracted references · 47 canonical work pages

  1. [1]

    Abbas, R

    Z. Abbas, R. Zhao, J. Modayil, A. White, and M. C. Machado. Loss of plasticity in continual deep reinforcement learning. arXiv preprint arXiv:2303.07507, 2023

    arXiv 2023

  2. [2]

    Achille, M

    A. Achille, M. Rovere, and S. Soatto. Critical learning periods in deep neural networks. arXiv preprint arXiv:1711.08856, 2017

    arXiv 2017

  3. [3]

    Agarwal, M

    R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304–29320, 2021

    2021

  4. [4]

    C. Anil, J. Lucas, and R. Grosse. Sorting out lipschitz function approximation. In International Conference on Machine Learning, pages 291–301. PMLR, 2019

    2019

  5. [5]

    Ash and R

    J. Ash and R. P. Adams. On warm-starting neural network training. Advances in neural information processing systems, 33:3884–3894, 2020

    2020

  6. [6]

    J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  7. [7]

    Bachlechner, B

    T. Bachlechner, B. P. Majumder, H. Mao, G. Cottrell, and J. McAuley. Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, pages 1352–1361. PMLR, 2021

    2021

  8. [8]

    Balduzzi, M

    D. Balduzzi, M. Frean, L. Leary, J. Lewis, K. W.-D. Ma, and B. McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In International Conference on Machine Learning, pages 342–350. PMLR, 2017

    work page 2017

Show all 84 references
  1. [9]

    Benjamins, T

    C. Benjamins, T. Eimer, F. Schubert, A. Biedenkapp, B. Rosenhahn, F. Hutter, and M. Lin- dauer. Carl: A benchmark for contextual and adaptive reinforcement learning. arXiv preprint arXiv:2110.02102, 2021

    2021 arXiv

  2. [10]

    Blumenfeld, D

    Y . Blumenfeld, D. Gilboa, and D. Soudry. Beyond signal propagation: is feature diversity necessary in deep neural network initialization? In International Conference on Machine Learning, pages 960–969. PMLR, 2020

    2020

  3. [11]

    Cisse, P

    M. Cisse, P. Bojanowski, E. Grave, Y . Dauphin, and N. Usunier. Parseval networks: Improving robustness to adversarial examples. In International conference on machine learning, pages 854–863. PMLR, 2017

    2017

  4. [12]

    Dohare, R

    S. Dohare, R. S. Sutton, and A. R. Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325, 2021

    2021 arXiv

  5. [13]

    D’Oro, M

    P. D’Oro, M. Schwarzer, E. Nikishin, P.-L. Bacon, M. G. Bellemare, and A. Courville. Sample- efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022. 10

    2022

  6. [14]

    Elfwing, E

    S. Elfwing, E. Uchibe, and K. Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. arxiv e-prints (2017). arXiv preprint arXiv:1702.03118, 1702, 2017

    2017 arXiv

  7. [15]

    R. M. French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135, 1999

    1999

  8. [16]

    Glorot and Y

    X. Glorot and Y . Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010

    2010

  9. [17]

    I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y . Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013

    2013 arXiv

  10. [18]

    Gulrajani, F

    I. Gulrajani, F. Ahmed, M. Arjovsky, V . Dumoulin, and A. C. Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017

    2017

  11. [19]

    B. Hanin. Which neural net architectures give rise to exploding and vanishing gradients? Advances in neural information processing systems, 31, 2018

    2018

  12. [20]

    Hanin and D

    B. Hanin and D. Rolnick. How to start training: The effect of initialization and architecture. Advances in Neural Information Processing Systems, 31, 2018

    2018

  13. [21]

    Hansen, H

    N. Hansen, H. Su, and X. Wang. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828, 2023

    2023 arXiv

  14. [22]

    K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015

    2015

  15. [23]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778, 2016

    2016

  16. [24]

    Hendrycks and K

    D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  17. [25]

    Hochreiter, Y

    S. Hochreiter, Y . Bengio, P. Frasconi, J. Schmidhuber, et al. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001

    2001

  18. [26]

    W. Hu, L. Xiao, and J. Pennington. Provable benefit of orthogonal initialization in optimizing deep linear networks. arXiv preprint arXiv:2001.05992, 2020

    2001 arXiv

  19. [27]

    Huang, R

    S. Huang, R. F. J. Dossa, A. Raffin, A. Kanervisto, and W. Wang. The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023, 2022

    2023

  20. [28]

    Huang, R

    S. Huang, R. F. J. Dossa, C. Ye, J. Braga, D. Chakraborty, K. Mehta, and J. G. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1–18, 2022. URL http://jmlr.org/papers/v23/ 21-1342.html

    2022

  21. [29]

    Jastrzebski, D

    S. Jastrzebski, D. Arpit, O. Astrand, G. B. Kerg, H. Wang, C. Xiong, R. Socher, K. Cho, and K. J. Geras. Catastrophic fisher explosion: Early phase fisher matrix impacts generalization. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Mach...

    2021

  22. [30]

    Khetarpal, M

    K. Khetarpal, M. Riemer, I. Rish, and D. Precup. Towards continual reinforcement learning: A review and perspectives. arxiv. arXiv preprint arXiv:2012.13490, 2020

    2012 arXiv

  23. [31]

    Kirkpatrick, R

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017. 11

    2017

  24. [32]

    Kumar, R

    A. Kumar, R. Agarwal, D. Ghosh, and S. Levine. Implicit under-parameterization inhibits data-efficient deep reinforcement learning. arXiv preprint arXiv:2010.14498, 2020

    2010 arXiv

  25. [33]

    Kumar, H

    S. Kumar, H. Marklund, and B. Van Roy. Maintaining plasticity via regenerative regularization. arXiv preprint arXiv:2308.11958, 2023

    2023 arXiv

  26. [34]

    Lewandowski, H

    A. Lewandowski, H. Tanaka, D. Schuurmans, and M. C. Machado. Curvature explains loss of plasticity. arXiv preprint arXiv:2312.00246, 2023

    2023 arXiv

  27. [35]

    Lopez-Paz and M

    D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017

    2017

  28. [36]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  29. [37]

    C. Lyle, M. Rowland, and W. Dabney. Understanding and preventing capacity loss in reinforce- ment learning. arXiv preprint arXiv:2204.09560, 2022

    2022 arXiv

  30. [38]

    C. Lyle, Z. Zheng, E. Nikishin, B. A. Pires, R. Pascanu, and W. Dabney. Understanding plasticity in neural networks. arXiv preprint arXiv:2303.01486, 2023

    2023 arXiv

  31. [39]

    Masarczyk, M

    W. Masarczyk, M. Ostaszewski, E. Imani, R. Pascanu, P. Miło´s, and T. Trzcinski. The tunnel effect: Building data representations in deep neural networks. Advances in Neural Information Processing Systems, 36, 2024

    2024

  32. [40]

    D. Misra. Mish: A self regularized non-monotonic activation function. arXiv preprint arXiv:1908.08681, 2019

    1908 arXiv

  33. [41]

    Nair and G

    V . Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807– 814, 2010

    2010

  34. [42]

    Neyshabur, H

    B. Neyshabur, H. Sedghi, and C. Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512–523, 2020

    2020

  35. [43]

    Nikishin, M

    E. Nikishin, M. Schwarzer, P. D’Oro, P.-L. Bacon, and A. Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, pages 16828–16847. PMLR, 2022

    2022

  36. [44]

    Nikishin, J

    E. Nikishin, J. Oh, G. Ostrovski, C. Lyle, R. Pascanu, W. Dabney, and A. Barreto. Deep reinforcement learning with plasticity injection. arXiv preprint arXiv:2305.15555, 2023

    2023 arXiv

  37. [45]

    Papyan, X

    V . Papyan, X. Han, and D. L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652–24663, 2020

    2020

  38. [46]

    Pascanu, T

    R. Pascanu, T. Mikolov, and Y . Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310–1318. Pmlr, 2013

    2013

  39. [47]

    Pennington, S

    J. Pennington, S. Schoenholz, and S. Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. Advances in neural information processing systems, 30, 2017

    2017

  40. [48]

    Pennington, S

    J. Pennington, S. Schoenholz, and S. Ganguli. The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics, pages 1924–1932. PMLR, 2018

    1924

  41. [49]

    Prach and C

    B. Prach and C. H. Lampert. Almost-orthogonal layers for efficient general-purpose lipschitz networks. In European Conference on Computer Vision, pages 350–365. Springer, 2022

    2022

  42. [50]

    M. M. Rahman and Y . Xue. Robust policy optimization in deep reinforcement learning.arXiv preprint arXiv:2212.07536, 2022

    2022 arXiv

  43. [51]

    Ramachandran, B

    P. Ramachandran, B. Zoph, and Q. V . Le. Swish: a self-gated activation function.arXiv preprint arXiv:1710.05941, 7(1):5, 2017. 12

    2017 arXiv

  44. [52]

    A. Saxe, S. Sodhani, and S. J. Lewallen. The neural race reduction: Dynamics of abstraction in gated networks. In International Conference on Machine Learning, pages 19287–19309. PMLR, 2022

    2022

  45. [53]

    A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013

    2013 arXiv

  46. [54]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  47. [55]

    Shang, K

    W. Shang, K. Sohn, D. Almeida, and H. Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In international conference on machine learning, pages 2217–2225. PMLR, 2016

    2016

  48. [56]

    Singla and S

    S. Singla and S. Feizi. Skew orthogonal convolutions. In International Conference on Machine Learning, pages 9756–9766. PMLR, 2021

    2021

  49. [57]

    Sokar, R

    G. Sokar, R. Agarwal, P. S. Castro, and U. Evci. The dormant neuron phenomenon in deep reinforcement learning. arXiv preprint arXiv:2302.12902, 2023

    2023 arXiv

  50. [58]

    A. A. Team, J. Bauer, K. Baumli, S. Baveja, F. Behbahani, A. Bhoopchand, N. Bradley-Schmieg, M. Chang, N. Clay, A. Collister, et al. Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608, 2023

    2023 arXiv

  51. [59]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  52. [60]

    Verwimp, S

    E. Verwimp, S. Ben-David, M. Bethge, A. Cossu, A. Gepperth, T. L. Hayes, E. Hüllermeier, C. Kanan, D. Kudithipudi, C. H. Lampert, et al. Continual learning: Applications and the road forward. arXiv preprint arXiv:2311.11908, 2023

    2023 arXiv

  53. [61]

    L. Xiao, Y . Bahri, J. Sohl-Dickstein, S. Schoenholz, and J. Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In International Conference on Machine Learning, pages 5393–5402. PMLR, 2018

    2018

  54. [62]

    T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094–1100. PMLR, 2020

    2020

  55. [63]

    Zaidi, T

    S. Zaidi, T. Berariu, H. Kim, J. Bornschein, C. Clopath, Y . W. Teh, and R. Pascanu. When does re-initialization work? In Proceedings on, pages 12–26. PMLR, 2023

    2023

  56. [64]

    Identity

    H. Zhang, Y . N. Dauphin, and T. Ma. Fixup initialization: Residual learning without normaliza- tion. arXiv preprint arXiv:1901.09321, 2019. 13 Appendix A Additional experiments We investigate other aspects of plasticity including the role of entropy and experiments with pertu...

    1901 arXiv

  57. [65]

    handle-pull-side, peg-unplug-side, coffee-push, soccer, drawer-close, reach-wall, plate-slide- back, window-open, plate-slide-side, plate-slide-back-side

  58. [66]

    window-close, window-open, hand-insert, door-lock, reach, button-press, sweep-into, coffee- button, door-close, push

  59. [67]

    window-close, reach-wall, sweep-into, reach, soccer, coffee-push, plate-slide-side, drawer- close, hand-insert, door-close

  60. [68]

    plate-slide-back, reach-wall, door-lock, peg-unplug-side, push, button-press, plate-slide- back-side, coffee-push, coffee-button, handle-pull-side

  61. [69]

    push, coffee-button, sweep-into, door-close, drawer-close, soccer, peg-unplug-side, hand- insert, door-lock, reach

  62. [70]

    button-press, plate-slide-back-side, window-close, plate-slide-side, peg-unplug-side, plate- slide-back, coffee-button, window-open, handle-pull-side, door-close

  63. [71]

    push, button-press, plate-slide-back, drawer-close, soccer, plate-slide-side, reach-wall, coffee-push, window-close, door-lock

  64. [72]

    plate-slide-side, hand-insert, handle-pull-side, plate-slide-back-side, window-open, sweep- into, reach-wall, reach, soccer, peg-unplug-side

  65. [73]

    hand-insert, reach, window-close, drawer-close, window-open, coffee-button, plate-slide- back, coffee-push, push, plate-slide-back-side

  66. [74]

    sweep-into, peg-unplug-side, window-close, door-lock, hand-insert, handle-pull-side, window-open, door-close, button-press, reach-wall

  67. [75]

    reach, door-lock, sweep-into, push, button-press, coffee-push, handle-pull-side, plate-slide- side, door-close, drawer-close

  68. [76]

    plate-slide-back-side, soccer, sweep-into, handle-pull-side, plate-slide-side, peg-unplug-side, door-lock, reach, plate-slide-back, coffee-button

  69. [77]

    reach-wall, plate-slide-back, drawer-close, hand-insert, coffee-push, coffee-button, window- close, plate-slide-back-side, door-close, button-press

  70. [78]

    soccer, drawer-close, push, sweep-into, window-open, reach-wall, door-lock, window-close, reach, hand-insert

  71. [79]

    plate-slide-back, plate-slide-side, door-close, push, peg-unplug-side, plate-slide-back-side, coffee-push, coffee-button, button-press, soccer

  72. [80]

    hand-insert, coffee-button, soccer, window-open, push, reach, drawer-close, handle-pull-side, door-lock, plate-slide-back-side

  73. [81]

    coffee-push, door-close, handle-pull-side, window-close, plate-slide-back, reach-wall, sweep-into, window-open, plate-slide-side, peg-unplug-side

  74. [82]

    coffee-push, button-press, reach, peg-unplug-side, reach-wall, door-close, window-open, handle-pull-side, plate-slide-back-side, soccer

  75. [83]

    sweep-into, plate-slide-side, button-press, drawer-close, push, coffee-button, door-lock, hand-insert, plate-slide-back, window-close

  76. [84]

    So due to the seed, the same environment (e.g

    reach, button-press, plate-slide-side, door-close, plate-slide-back-side, plate-slide-back, coffee-button, sweep-into, reach-wall, drawer-close Note that each of these tasks additionally have a goal location that can be varied. So due to the seed, the same environment (e.g. do...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.