Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn

Aaron Courville; Glen Berseth; Hongyao Tang; Johan Obando-Ceron; Pablo Samuel Castro

REVIEW 3 major objections 5 minor 1 cited by

Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reducing churn — the uncontrolled change in a network's outputs for data outside the training batch — restores plasticity in continual RL, and a simple regularizer beats six prior methods across 24 benchmark settings.

desk verdict A broad, believable empirical case that churn reduction helps continual RL, burdened by a theory section whose NTK mechanism doesn't line up with the PPO experiments. read the letter →

arxiv 2506.00592 v1 pith:DMP34KVZ submitted 2025-05-31 cs.LG cs.AI

Hongyao Tang , Johan Obando-Ceron , Pablo Samuel Castro , Aaron Courville , Glen Berseth This is my paper

classification cs.LGcs.AI

keywords lossofplasticitycontinualreinforcementlearningchurnneuraltangentkernelC-CHAINgradientdecorrelationnon-stationary

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that loss of plasticity in continual reinforcement learning is driven by churn: the way a mini-batch update silently rewrites the network's outputs for data not in that batch. Using the Neural Tangent Kernel (NTK) matrix as a bridge, the authors argue that as training progresses the NTK loses rank, gradients become correlated, churn worsens, and the error dynamics of Equation 7 destabilize in a vicious cycle that is the phenomenon known as plasticity loss. The proposed remedy, C-CHAIN, continually minimizes churn for out-of-batch reference data alongside regular training, and the paper shows this preserves NTK rank, decorrelates gradients, and adaptively rescales step sizes. If correct, churn is not a side effect but a controllable causal lever, and a cheap regularizer can replace resets, clipping, and other interventions. Empirically, C-CHAIN achieves the best aggregate scores on continual Gym Control and ProcGen and beats six related methods across 24 continual RL settings.

What carries the argument

The machinery is the empirical Neural Tangent Kernel matrix $N_\theta(i,j) = \nabla_\theta f_\theta(x_i)^\top \nabla_\theta f_\theta(x_j)$, whose rank serves as the paper's formal indicator of plasticity, together with the first-order churn identity $C_f(\bar{x}, \theta, \Delta\theta) \approx \nabla_\theta f_\theta(\bar{x})^\top \Delta\theta$. Combining these yields the vector form $C_f(\theta, \Delta\theta) \approx -\eta N_\theta S G_L$, showing that the NTK determines churn independently of the loss; substituting this into the error recursion gives the update matrix $I - \eta N_{\theta_t} S_i$ that carries the stability argument. C-CHAIN is the algorithm that minimizes churn on a reference batch disjoint from the training batch, and its gradient decomposition into a kernel-suppression term and a projection term is what supports the two-efficacy claim.

What would settle it

Train a continual RL agent while counterfactually forcing the empirical NTK to keep high rank without reducing churn (for example, injecting orthogonal gradient directions that stabilize rank but leave out-of-batch outputs free to move); if plasticity still degrades, churn is not the operative cause, and if plasticity is preserved, churn is incidental to rank. A cheaper test: compute the $②$ term for the PPO clipped objective and check whether it equals $-\bar{g}^\top g \cdot g$; if not, the step-size-adjustment interpretation is miscalibrated for the main experiments.

Watch

Extended reading notes

Core claim

The central claim is that churn and plasticity loss are two views of the same NTK collapse. The error dynamics $E_i(\theta_{t+1}) \approx (I - \eta N_{\theta_t} S_i) E_i(\theta_t)$ show that when all pairs of data points develop correlated gradients, the NTK matrix $N_\theta$ loses rank, out-of-batch outputs swing uncontrollably with every update, and learning becomes unstable exactly as tasks keep changing. Churn reduction breaks the cycle: C-CHAIN's loss $L_f^{\mathrm{cr}}(\theta) = \tfrac{1}{2}\mathbb{E}_{\bar{x} \in B_{\mathrm{ref}}}[C_f(\bar{x}, \theta, \Delta\theta)^2]$ pushes against churn on reference data, and its gradient splits into two effects — a decorrelation term that suppresses off-diagonal NTK entries and a projection term that rescales the regular training gradient by the kernel between reference and training data. The paper demonstrates the rank-preserving effect empirically and shows that the two components add up to the full benefit, with the decorrelation (orthogonal) component carrying most of the weight.

Load-bearing premise

The two-effect analysis is derived for a TD/Q-learning loss, but the main experiments train with PPO's clipped policy objective, for which the key gradient identity that produces the step-size-adjustment term does not hold exactly; if that interpretation does not transfer, half the explanatory story is unsupported in the settings where the method is actually evaluated.

Editorial extensions

If this is right

If churn is a causal lever, NTK rank collapse is preventable by regularization alone, without resets, weight clipping, or parameter-space constraints.
Because C-CHAIN only needs a reference batch from current data and never needs to know when a task switched, it can be layered onto existing RL algorithms (the paper demonstrates PPO and DoubleDQN) with a single relative-scale hyperparameter.
The decomposition predicts that the decorrelation effect dominates, so cheaper approximations to the Hessian or kernel term could yield most of the benefit.
Better-than-oracle results on continual control suggest churn reduction preserves useful transfer across related tasks rather than merely blocking interference.
The same recipe transfers to continuous control and to a value-based agent, indicating the mechanism is not specific to one objective or action space.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The theory's two-effect derivation uses a TD/Q-learning loss, while the main experiments run PPO's clipped objective; if the projection interpretation fails to transfer, the step-size-adjustment half of the story weakens while the decorrelation half may still hold — an asymmetry worth testing directly.
A natural stress test: if churn is the operative cause, then a counterfactual that keeps the NTK full-rank while allowing churn (or reduces churn while keeping rank low) should separate the two mechanisms; the paper's own MNIST results, where C-CHAIN helps less than in RL, hint that churn's role is amplified when the data distribution is shaped by the agent's own policy.
Because churn reduction and parameter-near-initialization methods act on different objects (kernel off-diagonals versus parameter distance), combining C-CHAIN with weight clipping or L2-init-style regularization might compound gains, a hypothesis the paper does not test.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful part of this paper is the empirical study, and it is genuinely good. The authors take the churn-reduction loss from Tang & Berseth 2024, adapt it to continual RL (C-CHAIN), and test it across 24 environments: Gym Control, all of ProcGen, DMC, MinAtar, plus two continual SL baselines. They compare against six methods, report 6–12 seeds, use reliable metrics with confidence intervals, and release code. On aggregate, C-CHAIN beats vanilla and the other baselines on Gym Control and ProcGen, with non-overlapping CIs on the reliable metrics. The ablation separating the projective and orthogonal components is also informative. This is a solid, honest empirical contribution.

The theory is the soft spot, and the stress-test note lands. The NTK bridging argument is developed for a scalar output under a fixed-target squared loss, where the update is Δθ = -η G^T ∇_f L. But the experiments use PPO with a clipped policy objective and a vector-valued policy; the two-effect decomposition in Eqs. 10–12 does not carry over as written. The empirical NTK in Appendix B is also computed from gradients of the PPO loss, not from gradients of the network output f_θ as defined in Eq. 2. So the measured rank collapse is not the same object the theory explains. The authors do use the NTK measurements predictively rather than to fit constants, so the circularity burden is low. But the causal chain from churn reduction to rank preservation to plasticity is asserted more than demonstrated. The "vicious cycle" in Eq. 7 is also heuristic, not derived. These are real gaps, but they don't undermine the empirical claim that C-CHAIN works; they undermine only the stated mechanism.

The paper is also honest about its limits. C-CHAIN underperforms L2 Init and Weight Clipping on the two MNIST continual settings, and the authors say so. The MountainCar result is weaker and they discuss why. Self-citation is to a prior churn paper that is the direct basis for the loss, which is appropriate.

Who is this for? Researchers working on plasticity loss, continual RL, or RL optimization generally. The empirical finding that a simple regularizer helps across many environments is worth engaging with, even if the theoretical packaging overreaches. I'd want to see the NTK rank measured on network-output gradients for PPO, or a PPO-specific derivation, before fully buying the mechanism. But the paper deserves serious peer review; it is not a desk reject. I'd accept it with a request for a substantial revision of the theory section, and I'd cite it for the empirical result.

Referee Report

3 major / 5 minor

Summary. The paper studies loss of plasticity in continual reinforcement learning through the lens of churn, defined as network output variability on out-of-batch data induced by mini-batch training. The authors use the empirical NTK to argue that rank collapse and churn exacerbate each other, propose a regularizer C-CHAIN that minimizes churn on a reference batch, and show that it improves average performance over vanilla PPO and six prior methods on 24 continual RL environments spanning Gym Control, ProcGen, DMC, and MinAtar. They also report reliable-metric confidence intervals and an NTK analysis intended to support the causal mechanism.

Significance. If the causal story holds, the paper makes a valuable contribution: it identifies churn as a controllable and measurable factor in plasticity loss, proposes a simple add-on regularizer that works across multiple RL algorithms and environments, and provides reproducible code. The empirical results are strong: C-CHAIN achieves the best aggregate scores in both Gym Control and ProcGen, with non-overlapping confidence intervals versus the best baseline on median, IQM, and optimality gap. The two-effect decomposition in Section 4.3 is a useful conceptual framework even if the derivation is heuristic. However, the theory-experiment gap discussed below means the causal mechanism is not yet established.

major comments (3)

[Section 4.3, Eq. 12] The two-effect decomposition is derived for a scalar-output network trained on a TD or squared loss, with the specific identity ∇_{Q_θ} L_Q = -(Q_θ(x) - target) leading to Eq. 12. The main experiments use PPO's clipped surrogate objective for a vector-valued stochastic policy; for that loss, ∇_θ(∇_f L) is not -g, and the parameter update is not of the assumed form Δθ = -η E_x[∇_θ f_θ(x) ∇_f L]. Therefore the claimed gradient-decorrelation and step-size adjustment effects are not shown to apply to the actual experimental setting. The paper should either extend the derivation to the PPO objective, present the analysis as a heuristic motivating the method, or test the two-effect decomposition on the TD/DQN setting where it applies.
[Appendix B and Eq. 2] The empirical NTK in Figures 6 and 7 is computed from gradients of the PPO objective (g_i = ∇_θ L_PPO), not from gradients of the network output f_θ(x) as defined in Eq. 2. Rank collapse of this loss-gradient Gram matrix is a different object from the theoretical NTK, so the measurements do not directly support the claim that churn reduction prevents rank collapse of the NTK in Eq. 2. The authors should recompute the empirical NTK using output gradients, or explicitly justify and clearly label the loss-gradient Gram matrix as a proxy, adjusting the wording in Section 5.2 accordingly.
[Section 4.2, Eq. 7] The vicious cycle between rank decrease and churn exacerbation is asserted rather than derived. Equation 7 is a linear recurrence for the error with a fixed N_θ; it does not by itself imply that churn lowers the rank of N_θ or that lower rank increases churn. The text should either provide a formal argument or explicitly state this as a hypothesis supported by the empirical observations.

minor comments (5)

[Figure 1] The caption and surrounding text do not clearly explain the visual notation (e.g., which shaded regions correspond to training versus reference data); consider adding a legend or a more explicit textual description.
[Eq. 5] The notation 'G_L is the gradient matrix of ∇_{f_θ} L(θ)' is confusing, and the matrix dimensions in the product N_θ S G_L are not made precise; please clarify the dimensions and the definition of G_L.
[Section 4.3, after Eq. 12] The sentence 'This term could either dampen or accelerate the regular gradient g ∇_f L_f which depends on the sign of the kernel...' has an ambiguous antecedent for 'which depends'; rephrase to indicate clearly what depends on the sign.
[Section A.1] For MountainCar, the observation noise scale σ = 0.02 is chosen manually after finding σ = 2.0 too large; this per-environment design choice appears only in the appendix and should be mentioned in the main text when describing the setup.
[Abstract and Conclusion] Given the theory-experiment gap noted in the major comments, the abstract and conclusion use 'demonstrate' for the causal claims; consider softening to 'suggest' or 'provide evidence for' until the mismatch is resolved.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: C-CHAIN's churn objective is definitional, but the plasticity and NTK-rank claims are independently derived and empirically tested.

full rationale

The derivation chain is self-contained: churn is defined in Eq. 3, expressed through the NTK in Eqs. 4-5, used to derive the continual-learning error dynamics in Eq. 7, and then minimized by the C-CHAIN loss in Eq. 8. The paper's causal claims—that reducing churn suppresses NTK off-diagonal entries, maintains approximate rank, and improves continual RL performance—are not obtained by fitting the target variables; they are checked against held-out measurements of rank and off-diagonal magnitude (Figs. 6-7) and against aggregate returns on 24 continual RL settings. The only self-citation with author overlap, Tang and Berseth 2024, supplies the churn definition and the original single-task churn-reduction idea; it does not by itself justify the continual-RL NTK-rank mechanism, which is derived in Eqs. 10-12 and tested independently. The reader's flagged concerns—that Eq. 12 is derived for TD/Q-learning while the main experiments use PPO, and that Appendix B's empirical NTK is a loss-gradient Gram matrix rather than the output-gradient NTK of Eq. 2—are correctness/validity risks, not circularity: they do not make any prediction equal to an input by construction. Score 1 reflects the minor non-load-bearing self-citation; no circular step was found.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central mechanism rests on a first-order approximation of churn and on the assumption that PPO's clipped objective behaves like a TD target in the two-effect decomposition. The empirical NTK is measured on a different object than the theory defines, which is the main fragility.

free parameters (2)

β (target relative loss for auto λ) = 1e3 to 1e5 (Gym/ProcGen), 0.5/0.05 (DMC), 0.01 (MinAtar)
Controls the strength of churn reduction relative to the base RL loss. Searched per benchmark; reported results use the best setting.
σ noise scale for MountainCar = 0.02 instead of 2.0
Chosen so the task remains learnable; changes the difficulty of the MountainCar continual sequence relative to other environments.

assumptions (4)

domain assumption First-order Taylor expansion of churn (Eq 3), dropping O(||Δθ||²)
The entire churn-NTK derivation assumes the linear term dominates; in practice RL updates can be large.
domain assumption Low-dimensional empirical objective and simplicity bias cause NTK rank decrease (Section 4.2)
This is a hypothesis about implicit bias, not a proven mechanism.
ad hoc to paper The two-effects decomposition (①②) derived for TD loss extends to PPO's clipped objective
Eq 12 uses TD; experiments use PPO; the projection interpretation is assumed, not derived, for PPO.
ad hoc to paper The empirical NTK computed from PPO loss gradients (Appendix B) represents the NTK of the network output in Eq 2
Appendix B uses gradients of the PPO objective, not of f_θ(x); the rank behavior of this loss-gradient kernel is assumed to reflect the output NTK.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn." pith.science (2026). https://pith.science/paper/DMP34KVZ

@misc{pith2026250600592,
  author       = {Pith},
  title        = {Pith review of: Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DMP34KVZ}},
  note         = {Machine review of arXiv:2506.00592}
}

read the original abstract

Plasticity, or the ability of an agent to adapt to new tasks, environments, or distributions, is crucial for continual learning. In this paper, we study the loss of plasticity in deep continual RL from the lens of churn: network output variability for out-of-batch data induced by mini-batch training. We demonstrate that (1) the loss of plasticity is accompanied by the exacerbation of churn due to the gradual rank decrease of the Neural Tangent Kernel (NTK) matrix; (2) reducing churn helps prevent rank collapse and adjusts the step size of regular RL gradients adaptively. Moreover, we introduce Continual Churn Approximated Reduction (C-CHAIN) and demonstrate it improves learning performance and outperforms baselines in a diverse range of continual learning environments on OpenAI Gym Control, ProcGen, DeepMind Control Suite, and MinAtar benchmarks.

Figures

Figures reproduced from arXiv: 2506.00592 by the authors.

**Figure 1.** An illustration of the matrix NθS, where the datapoints are arranged with a separation between the training data and the remaining ones (i.e., the reference data). on its diagonal that corresponds to the sampling results of Btrain. The vector form in Equation 5 indicates that the NTK matrix Nθ plays an important role in determining the churn, independent of the loss function (or objective function) and the sampling … view at source ↗

**Figure 2.** The continual learning settings for Gym Control and ProcGen. The task switches for continual CartPole (above) and Starpilot (below) are illustrated for demonstration. (i.e., Starpilot, Fruitbot, Chaser, Dodgeball) were adopted in (Muppidi et al., 2024). For each environment, the task sequence consists of 5 instances procedurally generated by sampling a unique game level. The budget is 2M steps per task [PITH_FULL_I… view at source ↗

**Figure 4.** Learning curves of different methods in four (of sixteen) continual ProcGen environments. The curves and the shades are means and standard errors over six seeds. learning within each task, while C-CHAIN also mitigates it effectively. This also indicates that naive network resetting is not adequate to address plasticity loss in continual RL. Among the methods in comparison, TRAC is a competitive baseline in most env… view at source ↗

Figures from the paper (9 more)

**Figure 5.** Figure 5: Performance comparison with Reliable metrics (Agarwal et al., 2021). For each method in the comparison, the results are aggregated over sixteen continual ProcGen environments with six random seeds for each. 0 100 200 300 400 500 Timestep (2e4) 10 20 30 40 50 60 70 80 A…

**Figure 7.** Figure 7: Visualization of the empirical NTK matrix (100 by 100) for Vanilla (left) and C-CHAIN (right) at 8M timesteps (10M in total) in continual ProcGen Starpilot. matrix (corresponding to Section 4.2) and (2) the ablation of the two effects of reducing churn (i.e., ①, ② in S…

**Figure 8.** Figure 8: Ablation study for the projective and orthogonal gradient components of C-CHAIN in ProcGen Starpilot and Chaser. approximate the ② term and use the orthogonal component for the ① term. By applying either component only, we have two variants of C-CHAIN, denoted by Proj…

**Figure 9.** Figure 9: Learning curves for PPO and C-CHAIN PPO in continual DMC. The curves and the shades are means and standard errors over twelve seeds. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

**Figure 10.** Figure 10: Learning curves for DoubleDQN and C-CHAIN in continual MinAtar. The curves and the shades are means and standard errors over twelve seeds. A.4. Continual Supervised Learning We follow the settings in L2 Init (Kumar et al., 2023b) and adopt RandomLabel-MNIST and Permut…

**Figure 11.** Figure 11: Analysis on NTK matrix in terms of approximate rank (above), the sum of absolute off-diagonal values (middle), and the sum of absolute diagonal values(below) in continual ProcGen Starpilot. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

**Figure 12.** Figure 12: Learning curves of different methods in four (of sixteen) continual ProcGen environments: fruitbot, chaser, bigfish, caveflyer. The curves and the shades are means and standard errors over six seeds. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]

**Figure 13.** Figure 13: Learning curves of different methods in four (of sixteen) continual ProcGen environments: climber, ninja, jumper, heist. The curves and the shades are means and standard errors over six seeds. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

**Figure 14.** Figure 14: Learning curves of different methods in four (of sixteen) continual ProcGen environments: leaper, maze, plunder, bossfight. The curves and the shades are means and standard errors over six seeds. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning
cs.LG 2026-07 conditional novelty 5.5 of 10

Utility-scaled partial neuron resets prevent policy collapse in long-horizon continual RL while matching or beating binary-reset and uniform-decay baselines on several benchmarks.

Reference graph

Works this paper leans on

26 extracted references · 10 canonical work pages · cited by 1 Pith paper

[1]

in terms of accuracy with mean and standard error over three seeds. Algorithm RandomLabel-MNIST Permuted-MNIST Vanilla 0.1501±0.0030 0.6430±0.0029 C-CHAIN 0.2482±0.0141 0.6797±0.0042 L2 Init 0.8607±0.0021 0.8039±0.0027 Weight Clipping 0.4304±0.0052 0.8281±0.0001 is then computed by following the definition in Equation 2: Nθ(i, j) =g⊤ i gj for i, j∈ {1,2, ...

work page 2020
[5]

F., Rahman, P., Sutton, R

Dohare, S., Hernandez-Garcia, J. F., Rahman, P., Sutton, R. S., and Mahmood, A. R. Maintaining plasticity in deep continual learning.arXiv preprint, arXiv:2306.13812,

arXiv
[6]

T., Lupu, A., Goldie, A

Ellis, B., Jackson, M. T., Lupu, A., Goldie, A. D., Fel- lows, M., Whiteson, S., and Foerster, J. Adam on local time: Addressing nonstationarity in rl with relative adam timesteps.arXiv preprint, arXiv:2412.17113,

arXiv
[7]

and Mahmood, A

Elsayed, M. and Mahmood, A. R. Addressing loss of plas- ticity and catastrophic forgetting in continual learning. arXiv preprint, arXiv:2404.00781,

arXiv
[8]

Elsayed, M., Farrahi, H., Dangel, F., and Mahmood, A. R. Revisiting scalable hessian diagonal approximations for applications in reinforcement learning. InICML, 2024a. Elsayed, M., Lan, Q., Lyle, C., and Mahmood, A. R. Weight clipping for deep continual and reinforcement learning. RLJ, 5:2198–2217, 2024b. 10 Mitigating Plasticity Loss in Continual RL by R...

work page 2024
[10]

Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T

1109/MCSE.2007.58. Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T. P., and Wayne, G. Experience replay for continual learning. In NeurIPS, pp. 348–358,

work page 2007
[11]

V ., Pas- canu, R., Martens, J., and Dabney, W

Lyle, C., Zheng, Z., Khetarpal, K., Hasselt, H. V ., Pas- canu, R., Martens, J., and Dabney, W. Disentangling the causes of plasticity loss in neural networks.arXiv preprint, arXiv:2402.18762,

arXiv
[12]

Revisiting plasticity in visual reinforcement learning: Data, modules and training stages.arXiv preprint, arXiv:2310.07418,

Ma, G., Li, L., Zhang, S., Liu, Z., Wang, Z., Chen, Y ., Shen, L., Wang, X., and Tao, D. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages.arXiv preprint, arXiv:2310.07418,

arXiv

Show all 26 references

[16]

Proximal policy optimization algorithms

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint, arXiv:1707.06347,

arXiv
[17]

P., and Riedmiller, M

Tassa, Y ., Doron, Y ., Muldal, A., Erez, T., Li, Y ., de Las Casas, D., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., Lillicrap, T. P., and Riedmiller, M. A. Deep- mind control suite.arXiv preprint, arXiv:1801.00690,

arXiv
[18]

Deep reinforcement learning and the deadly triad.arXiv preprint, arXiv:1812.02648,

van Hasselt, H., Doron, Y ., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. Deep reinforcement learning and the deadly triad.arXiv preprint, arXiv:1812.02648,

arXiv
[19]

Continual learning for large language models: A survey

Wu, T., Luo, L., Li, Y .-F., Pan, S., Vu, T.-T., and Haffari, G. Continual learning for large language models: A survey. arXiv preprint, arXiv:2402.01364,

arXiv
[20]

and Tian, T

Young, K. and Tian, T. Minatar: An atari-inspired testbed for more efficient reinforcement learning experiments. arXiv preprint, arXiv:1903.03176,

1903 arXiv
[21]

Experimental Details A.1

12 Mitigating Plasticity Loss in Continual RL by Reducing Churn A. Experimental Details A.1. Continual RL Environment SetupsWe follow the setups in (Muppidi et al., 2024). For Gym Control, we use four environments: CartPole-v1, Acrobot-v1, LunarLander-v2 and MountainCar-v0. Fo...

2024
[22]

Therefore, we useσ= 0.02for MountainCar-v0

and we found σ= 2.0 is too large for this environment to be learnable. Therefore, we useσ= 0.02for MountainCar-v0. For ProcGen, we use all sixteen environments in the suite, while only four (i.e., Starpilot, Fruitbot, Chaser, Dodgeball) were adopted in (Muppidi et al., 2024). ...

2024
[23]

The values of conventional hyperparameters are taken from the recommended values inCleanRL

Hyperparameters of PPO and C-CHAIN used in continual DMC environments. The values of conventional hyperparameters are taken from the recommended values inCleanRL. PPO Hyperparameters Learning Rate 3e−4 Training Interval 2048 steps Discount Factor (γ) 0.99 GAE Parameter (λ) 0.9...
[24]

The values of conventional hyperpa- rameters are taken from the recommended values in (Young & Tian, 2019)

Hyperparameters of DoubleDQN and C-CHAIN used in continual MinAtar environments. The values of conventional hyperpa- rameters are taken from the recommended values in (Young & Tian, 2019). DoubleDQN Hyperparameters Learning Rate 3e−4 Training Interval 1 step Discount Factor (γ...

2019
[25]

We can observe that C-CHAIN improves the vanilla agent but does not perform on par with L2 Init and Weight Clipping. This shows that the efficacy of C-CHAIN is relatively limited in the two continual supervised learning environments, especially in contrast to its superiority i...

2024
[2013]

URL http://www.amazon.com/exec/obidos/ redirect?tag=citeulike07-20&path= ASIN/1449319793

ISBN 9789351100065. URL http://www.amazon.com/exec/obidos/ redirect?tag=citeulike07-20&path= ASIN/1449319793. Mnih, V ., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. Asyn- chronous methods for deep reinforcement learning. I...

1928
[2016]

Kumar, A., Agarwal, R., Ghosh, D., and Levine, S

doi: 10.3233/978-1-61499-649-1-87. Kumar, A., Agarwal, R., Ghosh, D., and Levine, S. Im- plicit under-parameterization inhibits data-efficient deep reinforcement learning. InICLR,

doi
[2019]

The phenomenon of policy churn.arXiv preprint, arXiv:2206.00730,

Schaul, T., Barreto, A., Quan, J., and Ostrovski, G. The phenomenon of policy churn.arXiv preprint, arXiv:2206.00730,

arXiv
[2020]

L., Pascanu, R., and Clopath, C

Berariu, T., Czarnecki, W., De, S., Bornschein, J., Smith, S. L., Pascanu, R., and Clopath, C. A study on the plastic- ity of neural networks.arXiv preprint, arXiv:2106.00042,

arXiv
[2021]

Openai gym

Brockman, G., Cheung, V ., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv preprint, arXiv:1606.01540,

arXiv
[2022]

J., Liu, Y ., and Van Roy, B

Kumar, S., Marklund, H., Rao, A., Zhu, Y ., Jeon, H. J., Liu, Y ., and Van Roy, B. Continual learning as computation- ally constrained reinforcement learning.arXiv preprint, arXiv:2307.04345, 2023a. Kumar, S., Marklund, H., and Roy, B. V . Maintaining plas- ticity in continual...

arXiv
[2023]

Towards charac- terizing divergence in deep q-learning.arXiv preprint, arXiv:1903.08894,

Achiam, J., Knight, E., and Abbeel, P. Towards charac- terizing divergence in deep q-learning.arXiv preprint, arXiv:1903.08894,

1903 arXiv
[2024]

Lever- aging procedural generation to benchmark reinforcement learning

Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. Lever- aging procedural generation to benchmark reinforcement learning. InICML, volume 119, pp. 2048–2056,

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

in terms of accuracy with mean and standard error over three seeds. Algorithm RandomLabel-MNIST Permuted-MNIST Vanilla 0.1501±0.0030 0.6430±0.0029 C-CHAIN 0.2482±0.0141 0.6797±0.0042 L2 Init 0.8607±0.0021 0.8039±0.0027 Weight Clipping 0.4304±0.0052 0.8281±0.0001 is then computed by following the definition in Equation 2: Nθ(i, j) =g⊤ i gj for i, j∈ {1,2, ...

work page 2020

[2] [5]

F., Rahman, P., Sutton, R

Dohare, S., Hernandez-Garcia, J. F., Rahman, P., Sutton, R. S., and Mahmood, A. R. Maintaining plasticity in deep continual learning.arXiv preprint, arXiv:2306.13812,

arXiv

[3] [6]

T., Lupu, A., Goldie, A

Ellis, B., Jackson, M. T., Lupu, A., Goldie, A. D., Fel- lows, M., Whiteson, S., and Foerster, J. Adam on local time: Addressing nonstationarity in rl with relative adam timesteps.arXiv preprint, arXiv:2412.17113,

arXiv

[4] [7]

and Mahmood, A

Elsayed, M. and Mahmood, A. R. Addressing loss of plas- ticity and catastrophic forgetting in continual learning. arXiv preprint, arXiv:2404.00781,

arXiv

[5] [8]

Elsayed, M., Farrahi, H., Dangel, F., and Mahmood, A. R. Revisiting scalable hessian diagonal approximations for applications in reinforcement learning. InICML, 2024a. Elsayed, M., Lan, Q., Lyle, C., and Mahmood, A. R. Weight clipping for deep continual and reinforcement learning. RLJ, 5:2198–2217, 2024b. 10 Mitigating Plasticity Loss in Continual RL by R...

work page 2024

[6] [10]

Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T

1109/MCSE.2007.58. Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T. P., and Wayne, G. Experience replay for continual learning. In NeurIPS, pp. 348–358,

work page 2007

[7] [11]

V ., Pas- canu, R., Martens, J., and Dabney, W

Lyle, C., Zheng, Z., Khetarpal, K., Hasselt, H. V ., Pas- canu, R., Martens, J., and Dabney, W. Disentangling the causes of plasticity loss in neural networks.arXiv preprint, arXiv:2402.18762,

arXiv

[8] [12]

Revisiting plasticity in visual reinforcement learning: Data, modules and training stages.arXiv preprint, arXiv:2310.07418,

Ma, G., Li, L., Zhang, S., Liu, Z., Wang, Z., Chen, Y ., Shen, L., Wang, X., and Tao, D. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages.arXiv preprint, arXiv:2310.07418,

arXiv