REVIEW 5 cited by
Normalization and effective learning rates in reinforcement learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Normalization layers have recently experienced a renaissance in the deep reinforcement learning and continual learning literature, with several works highlighting diverse benefits such as improving loss landscape conditioning and combatting overestimation bias. However, normalization brings with it a subtle but important side effect: an equivalence between growth in the norm of the network parameters and decay in the effective learning rate. This becomes problematic in continual learning settings, where the resulting effective learning rate schedule may decay to near zero too quickly relative to the timescale of the learning problem. We propose to make the learning rate schedule explicit with a simple re-parameterization which we call Normalize-and-Project (NaP), which couples the insertion of normalization layers with weight projection, ensuring that the effective learning rate remains constant throughout training. This technique reveals itself as a powerful analytical tool to better understand learning rate schedules in deep reinforcement learning, and as a means of improving robustness to nonstationarity in synthetic plasticity loss benchmarks along with both the single-task and sequential variants of the Arcade Learning Environment. We also show that our approach can be easily applied to popular architectures such as ResNets and transformers while recovering and in some cases even slightly improving the performance of the base model in common stationary benchmarks.
Forward citations
Cited by 5 Pith papers
-
Relative Value Learning
A critic that learns antisymmetric value differences ∆(s_i,s_j)=V(s_i)−V(s_j) has a provably contracting Bellman operator and an unbiased advantage estimator, and PPO with this critic matches standard PPO on Atari.
-
Reinitializing weights vs units for maintaining plasticity in neural networks
Selective weight reinitialization, which resets the least useful weights, maintains plasticity in small and layer-normalized networks where unit-level reinitialization methods fail.
-
Is Exploration or Optimization the Problem for Deep Reinforcement Learning?
Deep RL agents' best experienced trajectories are 2-3 times better than their learned policy's average return, suggesting exploitation and optimization issues dominate exploration challenges.
-
Optimizers Qualitatively Alter Solutions And We Should Leverage This
Deep learning optimizers should be designed to induce desired solution properties, not just convergence speed; different optimizers demonstrably land in qualitatively different minima.
-
Recovering Plasticity of Neural Networks via Soft Weight Rescaling
Soft Weight Rescaling shrinks each layer's weights toward their initialization at every step, bounding weight norms and improving plasticity and test accuracy in continual, warm-start, and single-task learning.
Discussion (0). Continue with ORCID to comment.