Pith. sign in

REVIEW 5 major objections 6 minor 45 references

Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss

T0 review · 5 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A stochastic activation that drops positive and negative preactivations at different rates keeps neural networks trainable by pushing them toward linear behavior.

desk verdict Simple, broadly effective activation for plasticity loss, but the linearization theory in the abstract does not survive contact with the paper's own bound. read the letter →

arxiv 2502.01342 v2 pith:MLZOBHML submitted 2025-02-03 cs.LG cs.AI

Sangyeon Park , Isaac Han , Seungwon Oh , Kyung-Joong Kim This is my paper · ORCID
classification cs.LGcs.AI
keywords plasticitylosscontinuallearningdropoutactivationfunctionreinforcementregularizationdeeplinearnetworksgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that plasticity loss—a neural network's gradual loss of ability to learn from new or shifted data—can be prevented by a small change to the activation function. The proposed AID (Activation by Interval-wise Dropout) applies different dropout probabilities to positive versus negative preactivation values: with probability $p$ a unit passes its positive part, and with probability $1-p$ it passes its negative part, so at test time the layer behaves as the deterministic modified leaky ReLU $r_p(x)=\tfrac{1}{2}x+(p-\tfrac{1}{2})|x|$. The authors prove that this stochastic activation regularizes a two-layer network toward the behavior of a deep linear network, and linear networks are known to retain plasticity. They report that AID keeps networks trainable across permuted-MNIST and random-label continual tasks, preserves generalization in continual learning on CIFAR10, CIFAR100, and TinyImageNet, improves DQN sample efficiency on Atari under a high replay ratio, and reduces the generalization gap in standard supervised learning. If correct, AID is a one-line replacement for ReLU that preserves learning capacity without resets, re-initialization, or added penalty terms.

What carries the argument

The central object is the simplified AID$_p$ activation: for each preactivation value, a Bernoulli mask with probability $p$ selects ReLU and probability $1-p$ selects the negative ReLU $\min(x,0)$, which is equivalent to applying dropout with rate $1-p$ to positive values and rate $p$ to negative values. At test time AID$_p$ acts as the deterministic modified leaky ReLU $r_p(x)=\tfrac{1}{2}x+(p-\tfrac{1}{2})|x|$. The load-bearing device is Theorem 4.1, which decomposes the expected AID training loss into a deterministic loss plus a regularization term that vanishes only when the network behaves linearly; the proof adapts the Drop-activation argument of Liang et al. (2021) with an inequality correction that weakens the bound. This decomposition is what links AID to the plasticity-preserving behavior of deep linear networks.

What would settle it

Run the permuted-MNIST trainability experiment twice, once with AID and once with the deterministic test-time activation $r_p$ alone (no stochastic masks). If the deterministic version retains trainability as well as AID does, then the interval-wise dropout mechanism is not what preserves plasticity.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a dropout rule which treats positive and negative preactivations asymmetrically converts the activation itself into a regularizer that pushes the network toward linearity. For the simplified AID$_p$, Theorem 4.1 shows that the expected training loss is at least the loss under the deterministic modified leaky ReLU $r_p$ plus a nonnegative penalty equal to $\frac{4p(1-p)}{n(2p-1)^2}\|W_2(\tfrac{1}{2}W_1x)-W_2r_p(W_1x)\|_2^2$, which vanishes only when the layer behaves like the linear map $W_2(\tfrac{1}{2}W_1x)$. Because prior work has established that deep linear networks do not suffer from plasticity loss, the paper concludes that this linearization effect is what keeps networks plastic. The paper also shows empirically that AID maintains dormant neuron ratios, effective rank, and average sign entropy better than vanilla ReLU or standard Dropout, and that standard Dropout fails because each of its sampled subnetworks independently loses plasticity.

Load-bearing premise

The argument depends on a theory whose regularization coefficient is tiny for the wide layers and dropout probabilities used in the main experiments, so the claimed push toward linear behavior may be too weak to actually steer a deep network's training.

Editorial extensions

If this is right

  • Replacing ReLU with AID in continual learning or reinforcement learning agents should reduce plasticity loss without neuron re-initialization, weight clipping, or explicit regularizers.
  • AID's linearization regularizer explains why standard Dropout does not help plasticity: Dropout's subnetworks each suffer the same plasticity loss, whereas AID's interval-dependent rates actively push units toward linear behavior.
  • Because AID is compatible with He initialization, it can be swapped into existing architectures such as CNNs, ResNets, and VGGs without changing initialization or optimizer settings.
  • The single hyperparameter $p$ controls the strength of linearization (p close to 0.5 is nearly linear, p = 1 is exactly ReLU), so practitioners can tune one number to trade expressive nonlinearity against plasticity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the linearization mechanism is the true driver, then a deterministic $r_p$ activation with an explicit penalty matching Theorem 4.1's coefficient should reproduce AID's plasticity benefits; testing this would separate the effect of the stochastic masks from the regularizer they induce.
  • The theory implies an optimal $p$ that balances plasticity against representational power and likely varies with task difficulty and network width; the paper's sweeps (p=0.99 for trainability, p=0.7-0.9 for generalizability, p=0.99-0.999 for reinforcement learning) can be read as empirically tracing that trade-off.
  • The paper's own preactivation analysis suggests a simpler plasticity diagnostic than its three metrics: monitoring how much the spread of preactivation values drifts across tasks, since AID keeps that spread bounded while Dropout lets it grow large.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes AID (Activation by Interval-wise Dropout), a stochastic activation that applies different dropout probabilities to positive and negative preactivations; the simplified version AID_p applies ReLU with probability p and negative ReLU with probability 1-p, and uses modified leaky ReLU r_p at test time. The authors claim that AID regularizes networks toward linear behavior, based on Theorem 4.1, and that it is compatible with He initialization. They evaluate AID on continual learning benchmarks (permuted MNIST, random-label MNIST, CIFAR10/100, TinyImageNet under continual full, continual limited, and class-incremental settings), on Atari reinforcement learning with replay ratio 1, and on standard supervised learning, comparing against many baselines. The paper reports that AID consistently maintains trainability and improves generalization, and includes additional analyses of dormant neuron ratio, effective rank, and sign entropy.

Significance. If the empirical results hold, AID is a simple, drop-in activation that mitigates plasticity loss across non-stationary and stationary settings, and the breadth of the evaluation—continual learning, reinforcement learning, and standard supervised learning, with many baselines—is a clear strength. The paper also provides a PyTorch implementation, pseudo-code, and extended analyses of plasticity metrics. However, the central theoretical explanation is not established: the proof of Theorem 4.1 yields only a loose lower bound, and the claimed linearization mechanism is not actually shown to operate in the experimental regime. The empirical contribution is valuable, but the paper currently overstates the theoretical support.

major comments (5)
  1. [Section 4.4 / Appendix B, Eq. (6)-(7)] The proof of Theorem 4.1 does not establish that AID's loss penalizes the linearity term R_p. The exact difference between the AID loss and the deterministic modified-leaky-ReLU loss is computed in Appendix B as p(1-p)||W2(I-2D)diag(W1x)||_F^2, which is a per-unit column-shrinkage/variance penalty. The term R_p appears only after the norm inequality in Eq. (6) divides by n, so the theorem's lower bound LAID >= L_p + [4p(1-p)/(n(2p-1)^2)] R_p does not imply that minimizing LAID exerts gradient pressure on R_p. The abstract's claim that AID 'regularizes the network, promoting behavior analogous to that of deep linear networks' is therefore not supported by the proof as written.
  2. [Section 5.2 / Tables 4, 5, 9] Even if one accepted the lower-bound interpretation in Theorem 4.1, the regularization coefficient is numerically negligible in the experimental regime: for p=0.99 and n=2000, 4p(1-p)/(n(2p-1)^2) is about 2e-5, while the exact gap coefficient p(1-p) is about 0.01. The trainability and reinforcement-learning experiments select p in [0.9, 0.999], the regime farthest from p=0.5 where the claimed linearization is strongest. The paper should either show direct evidence that R_p actually decreases during AID training in the architectures and p values used, or substantially weaken the linearization claim.
  3. [Section 4.4 / Appendix B] Theorem 4.1 is proved only for a two-layer network with square weight matrices and no bias terms, while all experiments use deep networks, biases, convolutional architectures, and residual connections. No argument is given that the proof extends to these settings, yet the abstract and conclusion make claims about 'the network' broadly. The scope mismatch between the theorem and the empirical claims should be explicitly addressed.
  4. [Appendix F.6 / Tables 4-10] The hyperparameter search space and the tables of 'optimal hyperparameters' are reported, but the paper does not describe a validation split or selection protocol. If hyperparameters, including the AID coefficient p, were selected by final test accuracy, the comparisons may be biased in favor of the proposed method. The authors should clarify how model selection was performed, or re-run the comparisons with hyperparameters chosen on a held-out validation set.
  5. [Figures 3, 4, 5, 7] Several key figures omit error bars or confidence intervals. Figure 3's caption states 'For visibility, we did not show the region using standard deviation,' but no error information is provided elsewhere for those plots, and Figures 4, 5, and 7 similarly lack error bars despite the text reporting results over 5 seeds (3 for VGG-16). Without a measure of variability, it is difficult to assess whether the reported advantages of AID are statistically meaningful.
minor comments (6)
  1. [Section 4.3] The phrase 'should retain its nonlinearity at test time at test time' contains a duplicated phrase; please fix.
  2. [Algorithms 1 and 2] The probability notation is inconsistent between the general AID in Algorithm 1, where p_j is a dropout probability used as Bernoulli(1-p_j), and the simplified AID in Algorithm 2, where p is a keep probability for positive values. This should be clarified to avoid confusion.
  3. [Appendix F.6] The word 'optimal' in 'optimal hyperparameters' is used without stating the optimization criterion; please specify that the reported values are the best according to the chosen selection metric and protocol.
  4. [References] The reference 'Kingma, D. P. Adam: A method for stochastic optimization' is incomplete; the standard citation is Kingma and Ba (2015).
  5. [Appendix G.2.2 / Figure 11] In the provided version of the manuscript, the caption and surrounding text of Figure 11 contain unreadable glyph sequences (e.g., '/uni00000013/uni00000011/uni00000013/uni00000013/uni00000014/uni0000004f/uni00000055'); please verify that the compiled PDF renders this section correctly.
  6. [Appendix D] The argument for Property 2 is informal: it states that 'the same condition' holds for AID, but the zero-centered symmetric preactivation assumption should be stated explicitly as a condition of the property. The variance and derivative computations themselves appear correct.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Theorem 4.1 is a self-contained mathematical inequality, and the empirical claims are externally benchmarked rather than derived from fitted constants.

full rationale

The central theoretical claim is Theorem 4.1, whose proof in Appendix B is a self-contained derivation under stated assumptions (2-layer network, no biases, square weight matrices, given input/target vectors). The regularizer Rp is explicitly defined as the squared distance between the linear-map output W2(1/2 W1x) and the modified-leaky-ReLU output W2 r_p(W1x), and the lower bound L_AID >= L_p + [4p(1-p)/(n(2p-1)^2)] Rp is obtained by algebraic expansion of the exact excess term p(1-p)||W2(I-2D)diag(W1x)||_F^2, not by assuming the conclusion. The derivation uses only the definition of AID (equivalently, applying ReLU with probability p and negative ReLU with probability 1-p, as shown in Property 1). No fitted parameter is renamed as a prediction: the hyperparameter p is a method parameter selected by hyperparameter sweep, and the reported results are direct benchmark comparisons against external baselines. The citation of linear networks not suffering from plasticity loss (Dohare et al., Lewandowski et al.) is external evidence used for interpretation, not a load-bearing self-citation, and the paper contains no self-citations. The footnote correcting Liang et al. (2021) weakens the bound but does not make the argument circular; at most it raises a quantitative-strength or interpretation concern about whether the lower-bound coefficient is large enough at p values such as 0.99, which is a correctness risk rather than a reduction of the result to its inputs. The generalizability discussion is explicitly left open as future work, and the empirical results are not claimed to follow tautologically from Theorem 4.1. Therefore no step in the derivation chain reduces by construction or by self-citation to its own inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on one fitted hyperparameter, p, and on several external results about linear networks and plasticity, plus an unproven extrapolation from a 2-layer theorem to deep architectures. No new physical or ontological entities are introduced.

free parameters (1)
  • AID coefficient p = 0.6 to 0.999 per experiment (e.g., 0.7, 0.8, 0.9, 0.99, 0.999 in Tables 4-10)
    The single hyperparameter controlling the dropout rates for positive and negative preactivations. It is grid-searched per dataset, model, and RL game, and the central results use the best p found. This is a fitted method parameter rather than a derived constant.
assumptions (4)
  • domain assumption Deep linear networks do not suffer from plasticity loss
    Invoked in Section 4.4 to connect the regularization-to-linear term to plasticity retention. The paper cites Dohare et al. (2024) and Lewandowski et al. (2024) but does not derive this result.
  • ad hoc to paper The 2-layer no-bias analysis in Theorem 4.1 extends to deep networks and to the architectures used in experiments
    The theorem is proven only for W1, W2 in R^{n x n} without biases, yet the paper applies the conclusion to ResNet-18, VGG-16, and DQN. No deep-network version of the bound is given.
  • standard math Preactivation values in He initialization follow a zero-centered symmetric distribution
    Used in Appendix D to show E[AID_p(y)^2] = 1/2 Var(y) and the derivative is 1 with probability 1/2, matching ReLU. This is the standard He et al. (2015) assumption.
  • domain assumption Preactivation distribution shift is a primary cause of plasticity loss
    Used in Appendix G.1.3 to interpret AID's bounded distribution shift as the reason it avoids plasticity loss. The paper cites Lyle et al. (2024) for this causal claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss." pith.science (2026). https://pith.science/paper/MLZOBHML

@misc{pith2026250201342,
  author       = {Pith},
  title        = {Pith review of: Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MLZOBHML}},
  note         = {Machine review of arXiv:2502.01342}
}
read the original abstract

Plasticity loss, a critical challenge in neural network training, limits a model's ability to adapt to new tasks or shifts in data distribution. This paper introduces AID (Activation by Interval-wise Dropout), a novel method inspired by Dropout, designed to address plasticity loss. Unlike Dropout, AID generates subnetworks by applying Dropout with different probabilities on each preactivation interval. Theoretical analysis reveals that AID regularizes the network, promoting behavior analogous to that of deep linear networks, which do not suffer from plasticity loss. We validate the effectiveness of AID in maintaining plasticity across various benchmarks, including continual learning tasks on standard image classification datasets such as CIFAR10, CIFAR100, and TinyImageNet. Furthermore, we show that AID enhances reinforcement learning performance in the Arcade Learning Environment benchmark.

Figures

Figures reproduced from arXiv: 2502.01342 by the authors.

Figure 1
Figure 1. AID Architecture for Simplified Version. We apply Dropout at a rate of 1 − p for positive values and p for negative values, where mpos ∼ Ber(p) and mneg ∼ Ber(1 − p). Unlike ReLU activation, AID allows utilizing negative preactivation and regularizing toward linear network, effectively mitigating plasticity loss. the reinforcement learning domain, caused by its inherent non-stationarity (Sokar et al., 2023; Kumar et… view at source ↗
Figure 2
Figure 2. Left. Random label MNIST experiment using an 8-layer MLP. Higher dropout probabilities result in significant trainability loss. Middle. Accuracy of the subnetworks trained on random target. Each subnetworks are sampled from original network after each epoch. Subnetworks of the Dropout also experience trainability loss. Right. Warm-start scenario of Resnet-18 model with CIFAR100 dataset. Dropout improves generalizati… view at source ↗
Figure 3
Figure 3. Results for Trainability Experiments. We plot the train accuracy comparisons across different optimizers on Permuted MNIST and Random Label MNIST for each method, with the x-axis representing tasks. For visibility, we did not show the region using standard deviation. Notably, AID consistently achieves high accuracy across all conditions, demonstrating its robustness in maintaining trainability. • Re-initialization: … view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Results on Continual Full & Limited Settings. The dataset was divided into 10 chunks, with models trained over 10 stages of 100 epochs each. The settings are distinguished by whether access to previously seen data is allowed (top row, continual full) or not (bottom row…
Figure 5
Figure 5. Figure 5: Results on Class-Incremental Setting. The figure shows the relative accuracy compared to a model trained from scratch. AID consistently outperforms the full reset approach, maintaining its advantage throughout the entire training process. a model trained with full rese…
Figure 7
Figure 7. Figure 7: Additional Results for Trainability on Various Conditions. We plot the train accuracy comparisons across different optimizers and learning rates on Permuted MNIST and Random Label MNIST for each method. We train SGD with learning rate 3e − 2, 3e − 3 and Adam with learn…
Figure 8
Figure 8. Figure 8: Metrics for Measuring Plasticity Loss. This figure presents the Dormant Neuron Ratio, Average Sign Entropy, and Effective Rank across tasks, comparing Vanilla, Dropout, and AID on random label MNIST (Top) and permuted MNIST (Bottom). Notably, AID maintains key metrics …
Figure 9
Figure 9. Figure 9: Left & Middle. Visualization of the preactivation distribution shift for Dropout and AID at the first and tenth tasks. Q1, Q2, and Q3 represent the first, second (median), and third quartiles, respectively. Right. Comparison of the preactivation distributions of AID an…
Figure 10
Figure 10. Figure 10: Generalizability Comparison. Test accuracies of cold-start and warm-start variants for vanilla, Dropout, and AID (top row). To facilitate direct comparison, we report the generalizability gap, computed as the accuracy difference between cold-start and warm-start (bott…
Figure 11
Figure 11. Figure 11: Hyperparameter Sensitivity for AID. Heatmaps show the final epoch test accuracy and standard deviation for different values of p and learning rate on continual full setting. The red lines in the colorbars indicate the baseline accuracy of full reset. Values above this…
Figure 12
Figure 12. Figure 12: Test Accuracy for Learned Classes in Class-Incremental Setting. As new classes are introduced, the task complexity increases, leading to a gradual decline in accuracy. 0 250 500 750 1000 1250 1500 1750 2000 Epoch 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Test Accuracy CIFAR100 (Res…
Figure 13
Figure 13. Figure 13: Test Accuracy for Total Classes in Class-Incremental Setting. In the main section of the paper, we primarily presented results for the difference from full reset in class-incremental experiment. To provide a more detailed analysis, we include additional plots in Figur…
Figure 14
Figure 14. Figure 14: Average RAW Scores across 17 Atari Games. We train DQN model over 10 million frames with replay ratio 1. The comparison includes vanilla DQN, Dropout and AID. Shaded regions indicate the standard deviation across 5 runs. In the reinforcement learning experiments, we s…
Figure 15
Figure 15. Figure 15: Effective Rank of 3 Games. The AID method maintains a higher effective rank compared to the vanilla and Dropout model throughout training. Following the approach in Section G.1.2, we analyze the feature rank for three games—Asterix, BeamRider, and DemonAt￾tack—where A…
Figure 16
Figure 16. Figure 16: Learning Curves on Standard Supervised Learning. The figure presents training and test accuracy curves for CIFAR10 (CNN), CIFAR100 (Resnet-18), and TinyImageNet (VGG-16). We divide learning rate by 10 at 100th and 150th epochs. AID not only effectively mitigates plast…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 13 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Abbas, Z., Zhao, R., Modayil, J., White, A., and Machado, M. C. Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents, pp.\ 620--636. PMLR, 2023

    2023

  3. [3]

    Resetting the optimizer in deep rl: An empirical study

    Asadi, K., Fakoor, R., and Sabach, S. Resetting the optimizer in deep rl: An empirical study. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  4. [4]

    and Adams, R

    Ash, J. and Adams, R. P. On warm-starting neural network training. Advances in neural information processing systems, 33: 0 3884--3894, 2020

    work page 2020

  5. [5]

    L., Kiros, J

    Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  6. [6]

    G., Naddaf, Y., Veness, J., and Bowling, M

    Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 0 253--279, 2013

    work page 2013

  7. [7]

    A study on the plasticity of neural networks

    Berariu, T., Czarnecki, W., De, S., Bornschein, J., Smith, S., Pascanu, R., and Clopath, C. A study on the plasticity of neural networks. arXiv preprint arXiv:2106.00042, 2021

    arXiv 2021

  8. [8]

    Adaptive rational activations to boost deep reinforcement learning

    Delfosse, Q., Schramowski, P., Mundt, M., Molina, A., and Kersting, K. Adaptive rational activations to boost deep reinforcement learning. In ICLR, 2024

    work page 2024

Show all 45 references
  1. [9]

    S., and Mahmood, A

    Dohare, S., Sutton, R. S., and Mahmood, A. R. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325, 2021

    2021 arXiv

  2. [10]

    F., Rahman, P., Mahmood, A

    Dohare, S., Hernandez-Garcia, J. F., Rahman, P., Mahmood, A. R., and Sutton, R. S. Maintaining plasticity in deep continual learning. arXiv preprint arXiv:2306.13812, 2023

    2023 arXiv

  3. [11]

    F., Lan, Q., Rahman, P., Mahmood, A

    Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plasticity in deep continual learning. Nature, 632 0 (8026): 0 768--774, 2024

    2024

  4. [12]

    and Mahmood, A

    Elsayed, M. and Mahmood, A. R. Addressing loss of plasticity and catastrophic forgetting in continual learning. arXiv preprint arXiv:2404.00781, 2024

    2024 arXiv

  5. [13]

    Elsayed, M., Lan, Q., Lyle, C., and Mahmood, A. R. Weight clipping for deep continual and reinforcement learning. arXiv preprint arXiv:2407.01704, 2024

    2024 arXiv

  6. [14]

    C., Clopath, C., Busoniu, L., and Pascanu, R

    Gogianu, F., Berariu, T., Rosca, M. C., Clopath, C., Busoniu, L., and Pascanu, R. Spectral normalisation for deep reinforcement learning: an optimisation perspective. In International Conference on Machine Learning, pp.\ 3734--3744. PMLR, 2021

    2021

  7. [15]

    and Wagener, N

    Hausknecht, M. and Wagener, N. Consistent dropout for policy gradient reinforcement learning. arXiv preprint arXiv:2202.11818, 2022

    2022 arXiv

  8. [16]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.\ 1026--1034, 2015

    2015

  9. [17]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  10. [18]

    Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and Ara \ A s jo, J. G. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23 0 (274): 0 1--18, 2022

    2022

  11. [19]

    Huber, P. J. Robust estimation of a location parameter. In Breakthroughs in statistics: Methodology and distribution, pp.\ 492--518. Springer, 1992

    1992

  12. [20]

    Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  13. [21]

    A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al

    Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 0 (13): 0 3521--3526, 2017

    2017

  14. [22]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009

    2009

  15. [23]

    and Hertz, J

    Krogh, A. and Hertz, J. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991

    1991

  16. [24]

    Implicit under-parameterization inhibits data-efficient deep reinforcement learning

    Kumar, A., Agarwal, R., Ghosh, D., and Levine, S. Implicit under-parameterization inhibits data-efficient deep reinforcement learning. arXiv preprint arXiv:2010.14498, 2020

    2010 arXiv

  17. [25]

    Maintaining plasticity via regenerative regularization

    Kumar, S., Marklund, H., and Van Roy, B. Maintaining plasticity via regenerative regularization. arXiv preprint arXiv:2308.11958, 2023

    2023 arXiv

  18. [26]

    and Yang, X

    Le, Y. and Yang, X. Tiny imagenet visual recognition challenge. CS 231N, 7 0 (7): 0 3, 2015

    2015

  19. [27]

    Plastic: Improving input and label plasticity for sample efficient reinforcement learning

    Lee, H., Cho, H., Kim, H., Gwak, D., Kim, J., Choo, J., Yun, S.-Y., and Yun, C. Plastic: Improving input and label plasticity for sample efficient reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024 a

    2024

  20. [28]

    Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks

    Lee, H., Cho, H., Kim, H., Kim, D., Min, D., Choo, J., and Lyle, C. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. In International Conference on Machine Learning, pp.\ 26416--26438. PMLR, 2024 b

    2024

  21. [29]

    Lewandowski, A., Tanaka, H., Schuurmans, D., and Machado, M. C. Curvature explains loss of plasticity. 2023

    2023

  22. [30]

    Lewandowski, A., Schuurmans, D., and Machado, M. C. Plastic learning with deep fourier features. arXiv preprint arXiv:2410.20634, 2024

    2024 arXiv

  23. [31]

    Drop-activation: implicit parameter reduction and harmonious regularization

    Liang, S., Khoo, Y., and Yang, H. Drop-activation: implicit parameter reduction and harmonious regularization. Communications on Applied Mathematics and Computation, 3: 0 293--311, 2021

    2021

  24. [32]

    Understanding and preventing capacity loss in reinforcement learning

    Lyle, C., Rowland, M., and Dabney, W. Understanding and preventing capacity loss in reinforcement learning. arXiv preprint arXiv:2204.09560, 2022

    2022 arXiv

  25. [33]

    A., Pascanu, R., and Dabney, W

    Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., and Dabney, W. Understanding plasticity in neural networks. In International Conference on Machine Learning, pp.\ 23190--23211. PMLR, 2023

    2023

  26. [34]

    Disentangling the causes of plasticity loss in neural networks

    Lyle, C., Zheng, Z., Khetarpal, K., van Hasselt, H., Pascanu, R., Martens, J., and Dabney, W. Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18762, 2024

    2024 arXiv

  27. [35]

    Revisiting plasticity in visual reinforcement learning: Data, modules and training stages

    Ma, G., Li, L., Zhang, S., Liu, Z., Wang, Z., Chen, Y., Shen, L., Wang, X., and Tao, D. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages. arXiv preprint arXiv:2310.07418, 2023

    2023 arXiv

  28. [36]

    I., Farajtabar, M., and Ghasemzadeh, H

    Mirzadeh, S. I., Farajtabar, M., and Ghasemzadeh, H. Dropout as an implicit gating mechanism for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp.\ 232--233, 2020

    2020

  29. [37]

    A., Veness, J., Bellemare, M

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518 0 (7540): 0 529--533, 2015

    2015

  30. [38]

    The primacy bias in deep reinforcement learning

    Nikishin, E., Schwarzer, M., D’Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp.\ 16828--16847. PMLR, 2022

    2022

  31. [39]

    Deep reinforcement learning with plasticity injection

    Nikishin, E., Oh, J., Ostrovski, G., Lyle, C., Pascanu, R., Dabney, W., and Barreto, A. Deep reinforcement learning with plasticity injection. Advances in Neural Information Processing Systems, 36, 2024

    2024

  32. [40]

    Shen, M., Yin, H., Molchanov, P., Mao, L., and Alvarez, J. M. Step out and seek around: On warm-start training with incremental data. arXiv preprint arXiv:2406.04484, 2024

    2024 arXiv

  33. [41]

    Dash: Warm-starting neural network training in stationary settings without loss of plasticity

    Shin, B., Oh, J., Cho, H., and Yun, C. Dash: Warm-starting neural network training in stationary settings without loss of plasticity. Advances in Neural Information Processing Systems, 37: 0 43300--43340, 2024

    2024

  34. [42]

    and Zisserman, A

    Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  35. [43]

    S., and Evci, U

    Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, pp.\ 32145--32168. PMLR, 2023

    2023

  36. [44]

    Dropout: a simple way to prevent neural networks from overfitting

    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15 0 (1): 0 1929--1958, 2014

    1929

  37. [45]

    Empirical evaluation of rectified activations in convolutional network

    Xu, B. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853, 2015

    2015 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.