Pith. sign in

REVIEW 4 major objections 5 minor 43 references

Catastrophic Forgetting Mitigation Through Plateau Phase Activity Profiling

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new regularization method, PPAP, claims that tracking a network's parameter jitter during the final training plateau identifies which weights can be safely adapted in later tasks, reducing catastrophic forgetting without sacrificing…

desk verdict A plateau-windowed importance measure whose own equation ranks steep directions high, not flat ones; the empirical claims outrun the data. read the letter →

arxiv 2507.08736 v1 pith:UN5BBYDR submitted 2025-07-11 cs.LG cs.AI

Idan Mashiach , Oren Glickman , Tom Tirer This is my paper
classification cs.LGcs.AI
keywords catastrophicforgettingcontinuallearningregularizationparameterimportanceplateauphaselosslandscapeSynapticIntelligenceEWC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that during the final plateau phase of training, when the loss has stopped decreasing, parameters that keep moving and varying reveal flat directions in the loss landscape, so updating them on the next task should not harm old-task performance. It introduces the Plateau Phase Activity Profile (PPAP), a per-parameter flexibility score computed only from this plateau dynamics, and uses it to modulate optimizer updates on task t+1. The claim is that plateau-phase activity ranks weights better than the full-trajectory importance used by Synaptic Intelligence or the post-hoc importance used by EWC, yielding a better balance between retaining old tasks and learning new ones. If correct, continual learning can be improved with negligible memory overhead, since the profile is computed incrementally during normal training and stored as a single vector.

What carries the argument

The central object is the Plateau Phase Activity Profile (PPAP) score, a per-weight flexibility measure built from plateau-phase dynamics. For each weight w, at each step i, the method computes A_i^w = \$\Delta$\theta_i^w \cdot \frac{\partial L}{\partial \theta_w} \cdot f(\$\Delta$ L_i), with the Gaussian factor f(\$\Delta$ L_i) = e^{-k(\$\Delta$ L_i)^2} suppressing steps where the loss changes substantially, so only near-stable plateau steps contribute. It accumulates the sum of absolute values S_w and the running standard deviation \sigma_w of these products, min-max normalizes both, combines them multiplicatively, and normalizes again. The profile is integrated by an interpolated update rule \$\Delta$\theta_{\text{modified}} = r\cdot\$\Delta$\$\theta$ + (1-r)\cdot\$\Delta$\$\theta$ \odot P, which directly scales the raw optimizer update by the profile rather than adding a loss penalty. The method relies on optimizers with momentum, such as SGD with momentum or Adam, because their smoothing of the trajectory is what makes plateau jitter measurable.

What would settle it

Train a model on task A, record the PPAP scores, then measure the increase in task A's loss when each weight is perturbed by a fixed small step; if high-PPAP weights do not show systematically smaller loss increases than low-PPAP weights, the flatness interpretation is wrong. Alternatively, compare PPAP against a profile built from random weights with the same update rule: if both perform equally, the plateau-activity signal is not the operative factor.

Watch

Extended reading notes

Core claim

The paper's central claim is that a parameter's movement and variability during the final loss plateau, rather than its behavior during the entire training run, indicate directions in the loss landscape that are relatively flat, making them safe to update when learning a new task. For each weight w, the method computes, at each training step i, the quantity A_i^w = \$\Delta$\theta_i^w \cdot \frac{\partial L}{\partial \theta_w} \cdot f(\$\Delta$ L_i), where f(\$\Delta$ L_i) = e^{-k(\$\Delta$ L_i)^2} down-weights steps with large loss changes. It accumulates the sum of absolute values S_w and the standard deviation \sigma_w of these A_i^w over the plateau, normalizes each with min-max scaling, multiplies them, and normalizes again to obtain the PPAP score P_w. During the next task, the optimizer's update is modified to \$\Delta$\theta_{\text{modified}} = r\cdot\$\Delta$\$\theta$ + (1-r)\cdot\$\Delta$\$\theta$ \odot P, so weights with high plateau activity receive stronger updates while low-activity weights are conserved. The paper reports that PPAP consistently achieves equal or better accuracy than SI on the CIFAR10-CIFAR100 benchmark and consistently achieves higher Euclidean distances, balancing retention and adaptation, than both SI and EWC across most LOCO CIFAR100 configurations.

Load-bearing premise

The method assumes that a weight's random-looking movement during the final plateau is a sign of flatness of the loss surface along that weight, so updating that weight in the next task will not hurt old tasks; the paper does not directly measure curvature to verify this.

Editorial extensions

If this is right

  • Regularization for catastrophic forgetting can be built from end-of-training dynamics alone, avoiding the need to accumulate importance signals over the entire training trajectory.
  • Weights with high plateau activity should be updated more, not less, when learning a new task, inverting the usual intuition that important weights are those that moved a lot during learning.
  • The approach is architecturally agnostic: it requires only an update-hook modification and a stored per-weight vector, so it can be layered onto existing momentum-based optimizers.
  • On the CIFAR10-CIFAR100 and LOCO CIFAR100 benchmarks, PPAP achieves a better trade-off between old-task retention and new-task performance than SI and EWC in most configurations tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's experiments, a direct curvature test would strengthen the flatness claim: measure the loss change on the old task when perturbing high- versus low-PPAP weights by a fixed step, and check whether high-score weights are indeed flatter.
  • Because the method uses Adam, part of the measured plateau jitter could be an artifact of Adam's per-coordinate normalization rather than true landscape flatness; ablating with plain SGD or comparing against a random-profile baseline would clarify the mechanism.
  • A natural extension is layer-wise normalization of the profile, which the paper lists as a future direction and which could change which layers are allowed to adapt.
  • The same profile idea could be tested in transfer learning or pretraining-finetuning settings, where the plateau of the pretraining run might predict which weights are safe to fine-tune on a downstream task.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Plateau Phase Activity Profile (PPAP), a continual-learning method that computes a per-parameter flexibility score from parameter movement and variability during the final training plateau of a task. The score is then used to modulate the optimizer updates on subsequent tasks via Eq. (10), with the stated rationale that plateau-active parameters lie in relatively flat directions of the loss landscape and can therefore be updated more freely without harming old-task performance. The authors compare PPAP against SI on a CIFAR10-CIFAR100 sequential benchmark and against SI and EWC in a Leave-One-Class-Out setting on CIFAR100, reporting that PPAP often achieves a better trade-off between retention and adaptation.

Significance. If the proposed flatness interpretation were valid, PPAP would be an attractive low-overhead continual-learning method: it uses an online, constant-memory computation, and the paper provides a concrete algorithm and detailed experimental configurations. The empirical setup covers two benchmark scenarios and sweeps regularization strengths, which is a useful attempt at a fair comparison. However, the central mechanism is not established, and the reported evidence is too thin to support the paper's main claims. The significance of the contribution therefore depends on whether the authors can substantiate the link between plateau activity and flatness and provide statistically reliable comparisons.

major comments (4)
  1. [Sec. 4.2, Eq. (4)] The activity measure defined in Eq. (4), A_i^w = Δθ_i^w · (∂L_i/∂θ_w) · f(ΔL_i), cannot identify flat directions. For a gradient-descent-type update, Δθ_i^w is approximately −η ∂L/∂θ_w, and with Adam's per-coordinate normalization it is approximately −η sign(∂L/∂θ_w), so |A_i^w| is proportional to the gradient magnitude (or its square), not to local flatness. A genuinely flat direction has near-zero gradient and therefore near-zero |A|, so after min-max normalization it receives P_w ≈ 0, and Eq. (10) then suppresses updates along exactly the directions the paper claims should be updated more freely. This directly contradicts the Sec. 4.1 assertion that higher plateau activity reveals relatively flat directions and undermines the mechanism used to justify the update rule.
  2. [Sec. 4.1, Sec. 4.2] The flatness interpretation is asserted without direct measurement. The paper does not compute Hessian-vector products, per-parameter loss changes under perturbation, or any other curvature estimate, and references [17,18] address flat minima and training regimes generally, not the claim that momentum-optimizer plateau jitter identifies flat directions. A mechanism-only experiment, for example perturbing individual parameters during the plateau and measuring the induced loss change, is necessary to support the core claim that high plateau activity corresponds to adaptability without affecting previous-task loss.
  3. [Table 2, Sec. 5.2.2] The LOCO results in Table 2 are presented without error bars, standard deviations, or significance tests, and the claimed Euclidean-distance advantage over SI and EWC is often only about 0.005–0.015, which is small relative to typical run-to-run variability. The statement that PPAP 'consistently achieves higher Euclidean distances than both SI and EWC approaches across most configurations' is therefore not established by the reported data. Additionally, the hyperparameters compared (r for PPAP vs. K for SI and EWC) have different meanings, so the frontier comparison needs at least per-configuration variance reporting or a Pareto analysis before such a conclusion can be drawn.
  4. [Table 1, Sec. 5.1.2] The claim that PPAP 'consistently achieves equal or better accuracy than SI over all tasks' is contradicted by the paper's own Table 1 on Task 4, where SI reports 0.786 and PPAP reports 0.751±0.025. Even accounting for the reported standard deviation, the point estimate is lower, so the claim should be qualified, and SI's variance should be reported for a fair comparison.
minor comments (5)
  1. [Sec. 4.1 and Figure 2 caption] The sentence 'SI assigns a high importance score to θ2 and a low score to θ1 because θ1 exhibited greater gradients' is internally inconsistent and conflicts with the caption of Figure 2, which states that SI assigns a high score to θ1; please clarify the intended assignment.
  2. [Sec. 5.2.2] There is a typo in the sentence 'none of the methods has has perfect performance'; it should read 'has'.
  3. [Appendix C.2] The symbol K is used both for the Gaussian width (k=25) and for the SI/EWC regularization strengths; please rename one of these to avoid confusion.
  4. [Appendix D, Table 2] The paper does not state the number of random seeds used for the LOCO results or whether the reported values are averaged over the 20 superclass choices only; a full variance decomposition would allow the reader to judge the significance of the differences.
  5. [Figure 3 caption] The caption says that each approach except 'From scratch' is evaluated across six tasks with accuracy measured on each task's validation data, while the text states that 'Fine tuning' shows accuracy after Task 6; please clarify the evaluation point.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: PPAP's flatness premise is an empirical assertion rather than a definitional identity, and the empirical comparisons are external.

full rationale

I examined the derivation chain. Eq. (4) defines per-parameter activity as A_i^w = Δθ_i^w · (∂L_i/∂θ_w) · f(ΔL_i); Eqs. (5)–(9) accumulate, normalize, and merge this into the PPAP score P; Eq. (10) modulates task-t+1 updates by P. The paper's central claim that high plateau activity reveals flat loss-landscape directions (Sec. 4.1, 'parameters that exhibit higher activity ... reveal directions in the loss landscape that are relatively flat') is an empirical premise, not a definitional reduction: 'activity' is defined through movement-gradient products, while 'flat direction' is an asserted geometric property of the loss landscape. The two are not defined in terms of each other, so the claim could be false without being circular. The empirical evaluation compares PPAP against SI and EWC with regularization-strength sweeps on both sides, and Table 2 reports all frontier points, so no fitted parameter is being relabeled as a prediction. The only author self-citation, [43], appears in a future-work sentence ('our PPAP approach may benefit from optimizers that aim at reaching flatter minima [42, 43]') and is not load-bearing for any derivation or empirical claim. The reviewer's concern that Eq. (4) may actually measure gradient magnitude rather than flatness is a correctness/validity critique, not a circularity: the method could be internally consistent yet scientifically wrong. I therefore find no circular step and score 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

Three domain assumptions carry the method: plateau activity signals flat, task-safe directions; momentum optimizers make that activity informative; and the per-batch loss difference reliably gates the plateau window. Two hand-set hyperparameters (k and r) plus the baseline strength sweeps shape the reported frontiers. No invented entities are introduced.

free parameters (3)
  • k (Gaussian gate width) = 25 (both setups)
    Controls which training steps count as plateau via f(Delta L) = exp(-k Delta L^2). Chosen by hand; no sensitivity analysis or ablation is reported, and the very definition of the plateau window depends on it.
  • r (PPAP update-mix coefficient) = 0.03 for CIFAR10-CIFAR100; 0.05, 0.1, 0.2, 0.3 for LOCO
    Eq. 10 blends the raw optimizer update with the PPAP-scaled update. The reported frontier selects the best of the swept values, so the headline comparison depends on a favorable choice of r.
  • Baseline regularization strengths = SI K in 0.05, 0.005, 0.0005, 0.00005; EWC K in 10, 100, 500, 1000
    Swept for both baselines to draw Pareto frontiers; the comparison is only as fair as these sweeps are thorough.
assumptions (3)
  • domain assumption Parameters that move more during the final training plateau lie in flat directions of the loss landscape, and this flatness survives the switch to the next task.
    Sec 4.1 states this as the core motivation. Refs [17,18] support flat-minima and training-regime effects in general, not a per-parameter link between plateau jitter and curvature. No direct validation is provided. If false, PPAP is an arbitrary gate on SI's importance measure.
  • domain assumption Momentum-based optimizers (SGD with momentum, Adam) smooth the trajectory so that measured plateau activity is an informative signal rather than pure optimizer noise.
    Sec 4 states 'This smoothness is essential for our method.' No experiment isolates this dependence, and Adam's per-coordinate normalization can produce jitter unrelated to curvature.
  • domain assumption The per-batch loss difference Delta L_i = L(theta_{i+1}; B_i) - L(theta_i; B_i) reliably indicates the training phase for gating the plateau window.
    Eq. 2 and 3 in Sec 4.2. A single-minibatch before and after loss difference is noisy; the Gaussian gate assumes it separates fast-learning steps from plateau steps reliably.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Catastrophic Forgetting Mitigation Through Plateau Phase Activity Profiling." pith.science (2026). https://pith.science/paper/UN5BBYDR

@misc{pith2026250708736,
  author       = {Pith},
  title        = {Pith review of: Catastrophic Forgetting Mitigation Through Plateau Phase Activity Profiling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UN5BBYDR}},
  note         = {Machine review of arXiv:2507.08736}
}
read the original abstract

Catastrophic forgetting in deep neural networks occurs when learning new tasks degrades performance on previously learned tasks due to knowledge overwriting. Among the approaches to mitigate this issue, regularization techniques aim to identify and constrain "important" parameters to preserve previous knowledge. In the highly nonconvex optimization landscape of deep learning, we propose a novel perspective: tracking parameters during the final training plateau is more effective than monitoring them throughout the entire training process. We argue that parameters that exhibit higher activity (movement and variability) during this plateau reveal directions in the loss landscape that are relatively flat, making them suitable for adaptation to new tasks while preserving knowledge from previous ones. Our comprehensive experiments demonstrate that this approach achieves superior performance in balancing catastrophic forgetting mitigation with strong performance on newly learned tasks.

Figures

Figures reproduced from arXiv: 2507.08736 by the authors.

Figure 1
Figure 1. Schematic illustration of parameter space trajectories in two tasks (taken from [ [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Schematic illustration of a complex parameter space trajectories demonstrating the limita [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of different training approaches on CIFAR10-CIFAR100 tasks. Gray bars [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: LOCO CIFAR100 Results: Comparison of different methods using Cartesian coordinates, [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 30 canonical work pages

  1. [1]

    Imagenet classification with deep convolutional neural networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012

    2012

  2. [2]

    Deep residual learning for im- age recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016

    2016

  3. [3]

    Deep Learning

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, Cambridge, MA, 2016

    2016

  4. [4]

    Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In The Psychology of Learning and Motivation, volume 24, pages 109–165. Academic Press, 1989

    work page 1989

  5. [5]

    Connectionist models of recognition memory: constraints imposed by learning and forgetting functions

    Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990

    1990

  6. [6]

    Lifelong learning algorithms

    Sebastian Thrun and Tom M Mitchell. Lifelong learning algorithms. Learning to learn, pages 181–209, 1995

    work page 1995

  7. [7]

    Continual learning in neural networks

    German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual learning in neural networks. Philosophical Transactions of the Royal Society B , 374(1772):20180077, 2019

    work page 2019

  8. [8]

    A continual learning survey: Defying forgetting in classification tasks

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. arXiv preprint arXiv:1909.08383, 2019

    arXiv 1909

Show all 43 references
  1. [9]

    Three scenarios for continual learning

    Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734, 2022

    1904 arXiv

  2. [10]

    Hayes, and Christopher Kanan

    Ronald Kemker, Marc McClure, Angelina Abitino, Tyler L. Hayes, and Christopher Kanan. Measuring catastrophic forgetting in neural networks. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI), 2018

    2018

  3. [11]

    Catastrophic forgetting in deep learning: A comprehensive taxonomy

    Vinay V Ramasesh, Eva L Dyer, and Maithra Raghu. Catastrophic forgetting in deep learning: A comprehensive taxonomy. arXiv preprint arXiv:2007.04258, 2021

    2007 arXiv

  4. [12]

    Continual learning through synaptic intelligence

    Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987–3995. PMLR, 2017

    2017

  5. [13]

    The loss surfaces of multilayer networks

    Anna Choromanska, Mikael Henaff, Michael Mathieu, Gerard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics (AISTATS), pages 192–204, 2015

    2015

  6. [14]

    Deep learning without poor local minima

    Kenji Kawaguchi. Deep learning without poor local minima. Advances in Neural Information Processing Systems (NeurIPS), 29, 2016

    2016

  7. [15]

    Visualizing the loss landscape of neural nets

    Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. NeurIPS, 31, 2018

    2018

  8. [16]

    Qualitatively characterizing neural network optimization problems

    Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv:1412.6544, 2014. 10

    2014 arXiv

  9. [17]

    Under- standing the role of training regimes in continual learning

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the role of training regimes in continual learning. In Advances in Neural Information Processing Systems, volume 33, pages 9220–9231, 2020

    2020

  10. [18]

    On large-batch training for deep learning: Generalization gap and sharp minima

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. ICLR, 2017

    2017

  11. [19]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  12. [20]

    Packnet: Adding multiple tasks to a single network by iterative pruning

    Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. CVPR, 2018

    2018

  13. [21]

    Overcoming catastrophic forgetting with hard attention to the task

    Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. ICML, 2018

    2018

  14. [22]

    Progressive neural networks

    Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. In arXiv preprint arXiv:1606.04671, 2016

    2016 arXiv

  15. [23]

    Dynamically expandable networks

    Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Dynamically expandable networks. In International Conference on Machine Learning, pages 3982–3990. PMLR, 2017

    2017

  16. [24]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. EMNLP, 2021

    2021

  17. [25]

    Parameter-efficient model adaptation for vision transformers

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Parameter-efficient model adaptation for vision transformers. ACL, 2021

    2021

  18. [26]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  19. [27]

    Gradient based sample selection for online continual learning

    Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. NeurIPS, 32, 2019

    2019

  20. [28]

    icarl: Incremental classifier and representation learning

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. CVPR, 2017

    2017

  21. [29]

    Gdumb: A simple approach that questions our progress in continual learning

    Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. ECCV, 2020

    2020

  22. [30]

    Efficient lifelong learning with a-gem

    Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In International Conference on Learning Representations, 2018

    2018

  23. [31]

    Continual learning with deep generative replay

    Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pages 2990–2999, 2017

    2017

  24. [32]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catas- trophic forgetting...

    2017

  25. [33]

    Learning without forgetting

    Zhizhong Li and Derek Hoiem. Learning without forgetting. In European Conference on Computer Vision, pages 614–629. Springer, 2017

    2017

  26. [34]

    Memory aware synapses: Learning what (not) to forget

    Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. ECCV, pages 139–154, 2018

    2018

  27. [35]

    Gradient episodic memory for continual learning

    David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pages 6467–6476, 2017. 11

    2017

  28. [36]

    Orthogonal gradient descent for continual learning

    Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pages 3762–3773. PMLR, 2020

    2020

  29. [37]

    Riemannian walk for incremental learning: Understanding forgetting and intransigence

    Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, pages 532–547, 2018

    2018

  30. [38]

    Variational continual learning

    Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. ICLR, 2018

    2018

  31. [39]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  32. [40]

    Last layer re-training is sufficient for robustness to spurious correlations

    Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. In The Eleventh International Conference on Learning Representations, 2023

    2023

  33. [41]

    Simple and fast group robustness by automatic feature reweighting

    Shikai Qiu, Andres Potapczynski, Pavel Izmailov, and Andrew Gordon Wilson. Simple and fast group robustness by automatic feature reweighting. In International Conference on Machine Learning, pages 28448–28467. PMLR, 2023

    2023

  34. [42]

    Sharpness-aware min- imization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2021

    2021

  35. [43]

    Lorenza: Enhancing generalization in low-rank gradient llm training via efficient zeroth-order adaptive sam

    Yehonathan Refael, Iftach Arbel, Ofir Lindenbaum, and Tom Tirer. Lorenza: Enhancing generalization in low-rank gradient llm training via efficient zeroth-order adaptive sam. arXiv preprint arXiv:2502.19571, 2025. 12 A Online Standard Deviation Computation The standard deviatio...

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.