REVIEW 7 cited by
Disentangling the Causes of Plasticity Loss in Neural Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Underpinning the past decades of work on the design, initialization, and optimization of neural networks is a seemingly innocuous assumption: that the network is trained on a \textit{stationary} data distribution. In settings where this assumption is violated, e.g.\ deep reinforcement learning, learning algorithms become unstable and brittle with respect to hyperparameters and even random seeds. One factor driving this instability is the loss of plasticity, meaning that updating the network's predictions in response to new information becomes more difficult as training progresses. While many recent works provide analyses and partial solutions to this phenomenon, a fundamental question remains unanswered: to what extent do known mechanisms of plasticity loss overlap, and how can mitigation strategies be combined to best maintain the trainability of a network? This paper addresses these questions, showing that loss of plasticity can be decomposed into multiple independent mechanisms and that, while intervening on any single mechanism is insufficient to avoid the loss of plasticity in all cases, intervening on multiple mechanisms in conjunction results in highly robust learning algorithms. We show that a combination of layer normalization and weight decay is highly effective at maintaining plasticity in a variety of synthetic nonstationary learning tasks, and further demonstrate its effectiveness on naturally arising nonstationarities, including reinforcement learning in the Arcade Learning Environment.
Forward citations
Cited by 7 Pith papers
-
Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning
TeLAPA preserves behaviorally diverse policy neighborhoods in a shared latent space, improving MiniGrid continual RL transfer, revisit recovery, and retention over single-model preservation.
-
Continual Hyperbolic Learning of Instances and Classes
HyperCLIC embeds the instance-class hierarchy in hyperbolic space and uses hyperbolic classification and distillation losses to continuously learn both fine-grained instances and coarse-grained classes on EgoObjects, ...
-
Optimistic critics can empower small actors
Optimistic critic estimates (mean or max over two Q functions) counteract value underestimation and recover most of the performance lost by shrinking the actor in SAC and DrQ.
-
Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn
Reducing churn in continual RL via C-CHAIN prevents NTK rank collapse and substantially improves learning across four benchmark suites.
-
Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning
Utility-scaled partial neuron resets prevent policy collapse in long-horizon continual RL while matching or beating binary-reset and uniform-decay baselines on several benchmarks.
-
An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon?
Emergent misalignment and realignment are brittle surface effects driven by dataset artifacts like response length rather than stable representational changes.
-
Relative Value Learning
A critic that learns antisymmetric value differences ∆(s_i,s_j)=V(s_i)−V(s_j) has a provably contracting Bellman operator and an unbiased advantage estimator, and PPO with this critic matches standard PPO on Atari.
Discussion (0). Sign in to comment.