Pith. sign in

REVIEW 2 major objections 5 minor 16 references

How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networks

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Repeatedly resampling the last layer's weights during pretraining makes a network recover faster after transfer to new classes, and switching from SGD to Adam changes whether previously learned tasks keep improving.

desk verdict Solid Omni-image transfer results and a genuinely useful per-task loss diagnostic, but the zap-divergence mechanism rests on an unvalidated weight-similarity proxy and a few too many single-run plots. read the letter →

arxiv 2507.01559 v1 pith:67ZSGDGC submitted 2025-07-02 cs.LG cs.CV

Lapo Frati , Neil Traft , Jeff Clune , Nick Cheney This is my paper
classification cs.LGcs.CV
keywords continuallearningcatastrophicforgettingweightresamplingtransferfew-shotoptimizerdynamicsbackwardAdam
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Zapping—repeatedly replacing the weights of the final fully connected layer—is known to improve continual and transfer learning, and this paper tries to identify why. Using a small convolutional network on Omniglot and the harder natural-image Omni-image dataset, it shows that zapped models re-align faster after a simulated transfer shock, that their lower layers are less disturbed by the shock, and that zapping during pretraining turns a cheap baseline into a near match for a meta-learning pretraining method. It then tracks the loss of 100 tasks individually during sequential few-shot transfer and finds that the Adam optimizer produces continued improvement on previously seen tasks, while SGD shows the usual pattern of forgetting and recovery. The paper's conclusion is that both the resampling schedule and the optimizer's gradient statistics shape whether tasks interfere, synergize, or are forgotten.

What carries the argument

Zapping is the central intervention: at set intervals during pretraining the weights of the final fully connected layer are thrown away and re-randomized, either across the whole layer (zapped-IID) or for one class's output unit (ASB). To observe the consequences, the paper introduces zap-divergence, in which a trained model is copied, one copy's last layer is resampled, both copies train on identical minibatches, and the cosine similarity of each layer's flattened weight vector is recorded after every step. The second instrument is per-task loss tracking, which plots the loss of all 100 sequentially learned classes separately so that interference and backward transfer become visible as rising and falling lines on classes not currently being trained. The proposed explanation for the optimizer effect is Adam's exponential moving average of squared gradients, which the paper likens to an empirical Fisher information preconditioner, giving updates a memory that plain momentum lacks.

What would settle it

A direct check is to compute, side by side, the weight cosine similarity and a functional similarity (for example, agreement of predictions or logits on held-out transfer examples) between the zapped and control models after each training step. If the two measures diverge—if predictions become similar while weights remain far apart, or vice versa—then the zap-divergence plots do not establish faster functional recovery. A second decisive experiment would compare transfer accuracy of zapped and non-zapped models after matching the total number of post-zap training steps and functional similarity of the starting features.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that pretraining with zapping changes how a network responds to the standard transfer operation of replacing the last layer. In the zap-divergence protocol, after one copy of a zapped model has its last layer resampled, its fully connected layer returns toward the control's weight trajectory more quickly than in a non-zapped model, and the convolutional layers drift less from the control during subsequent training. In sequential transfer, per-task loss curves show that Adam, especially at low learning rates, allows loss on a finished task to keep decreasing (backward transfer), whereas SGD yields the familiar drop-then-rise pattern; in full-model tuning, Adam starts slower but ends higher (85.8% versus 76.7% accuracy over three epochs). Zapping also reduces cross-task interference when the whole model is tuned. These observations are offered as evidence that zapping promotes transferable features and that optimizer choice, not only the learning algorithm, drives continual-learning dynamics.

Load-bearing premise

The load-bearing premise is that cosine similarity between entire flattened weight vectors measures meaningful re-alignment of what the network does, so faster weight re-alignment after a zap counts as faster functional recovery.

Editorial extensions

If this is right

  • A model pretrained with zapping can end up with higher accuracy on new classes than it had on its own pretraining classes, because the transfer operation is no longer as disruptive.
  • The inexpensive zapped-IID pretraining approaches the transfer accuracy of Meta-ASB on Omni-image (27.3% versus 30.5%), so zapping is a low-cost substitute for higher-order meta-gradients.
  • In continual transfer, comparisons between optimizers that stop after one epoch can favor SGD; Adam's advantage materializes after later epochs because of its continued improvement of earlier tasks.
  • The learning-rate trade-off is structured by layer: lower rates protect lower-layer features, while higher rates speed recovery of the resampled layer, which explains the narrow range of workable learning rates in few-shot transfer.
  • Per-task loss plots, rather than average accuracy alone, reveal concurrent forgetting, backward transfer, and epoch-boundary interference that a single scalar metric hides.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if whole-weight cosine similarity tracks functional similarity, zap-divergence is a cheap diagnostic for transfer robustness; if it does not, the re-alignment numbers in the paper measure parameter drift, not behavioral recovery.
  • Editorial extension: Adam's apparent backward transfer suggests adaptive optimizers may already implement a form of gradient preconditioning similar to continual-learning penalties, which implies that adding explicit Fisher-based penalties on top of Adam could be redundant; this is testable in the same 100-task setup.
  • Editorial extension: because replacing the last layer is standard practice when adapting pretrained models, zapping during pretraining may generalize to larger-scale vision and language models; a direct test would add periodic last-layer resampling to pretraining and measure downstream few-shot adaptivity.
  • Editorial extension: the per-task curves are sensitive to task order, so the same visualization could be used to study curriculum effects, for example by reshuffling the 100 classes and checking whether the interference pattern in the paper's Figure 5 changes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper studies the effects of repeatedly resampling last-layer weights ("zapping") during pretraining and of optimizer choice on transfer learning and continual learning. Using Omni-image and Omniglot, the authors report that zapped pretraining improves few-shot transfer accuracy across IID, ASB, and Meta-ASB pretraining modes (Table 1, Fig. 14). They introduce a "zap-divergence" protocol (§2.2) that measures cosine similarity between treatment and control models after a last-layer resampling, and they interpret the resulting curves in Fig. 2b as evidence that zapped models re-align faster in the FC layer and are less affected in convolutional layers. In §3.3 the authors track per-task losses during sequential transfer and report that Adam, especially at low learning rates, produces continued improvement and backward transfer, while SGD exhibits more forgetting; in full-model training, Adam starts slower but overtakes SGD after several epochs (Table 3). The paper concludes with a speculative discussion connecting Adam's second-moment statistics to the empirical Fisher information and natural gradient descent.

Significance. If the results hold, the paper offers a cheap and simple pretraining intervention—zapping—that improves few-shot transfer, and it provides a detailed per-task characterization of optimizer-induced learning and forgetting dynamics. The paper has concrete strengths: accuracy tables in Table 1 and Tables 2–3 include standard deviations and at least one significance test, the main transfer result is consistent across three pretraining methods, and the learning-rate sweeps in Figs. 3, 8, 10, 11, and 13 are informative. The per-task loss visualization methodology is a useful tool for studying continual learning. However, the central mechanistic claim about transfer-shock recovery rests on an unvalidated weight-similarity proxy, and the per-task loss dynamics are presented almost entirely as qualitative readings of single trajectories without error bars. These points are load-bearing for the paper's explanatory contributions, although the direct accuracy improvements are more robust.

major comments (2)
  1. [§2.2, Fig. 2b] The zap-divergence analysis measures cosine similarity between entire flattened weight vectors, explicitly deviating from Jin et al. (2020, Def. 3.1), but the paper never validates that this whole-vector weight similarity tracks functional similarity. For a network with InstanceNorm and ReLU, two parameter settings can have very different weight-vector cosine similarity while producing nearly identical outputs (e.g., through large common-mode or scale changes), and conversely high weight similarity does not guarantee similar behavior. Because the paper's mechanistic conclusion—that zapped models "more quickly recover" and that convolutional layers are "less affected"—is read directly from these cosine curves, the proxy needs calibration. Concretely, the authors should report functional alignment (e.g., output-logit or feature-map agreement) for the same treatment-control pairs, or repeat the analysis with the neuron-wise metric of Jin et al. and show the conclusions are unchanged. In addition, Fig. 2b appears to show a single trajectory without error bars; multiple seeds and shaded intervals are needed to distinguish the dashed/solid gap from run-to-run variability.
  2. [§3.3, Figs. 4–6 and App. Figs. 10–13] The central observation of continued learning and backward transfer under Adam is based on qualitative inspection of per-task loss curves that appear to be single trajectories without error bars or replicate overlays. Claims such as "loss continues decreasing for many steps even after training on that task has finished" (Fig. 4b) and the epoch-boundary interference and reversion dynamics in Fig. 5 are read from the shape of individual curves. The accuracy tables (Tables 2–3) do support the final-accuracy comparison between optimizers, but the per-task dynamics are load-bearing for the paper's mechanistic narrative that Adam's second-moment information drives sustained improvement on prior tasks. The authors should provide at least several replicates (or shaded error bands) and a quantitative criterion for "continued improvement" (e.g., a slope or signed change of held-out task loss after its training interval) so that the claims are not purely visual.
minor comments (5)
  1. [§3.1, near Table 1] The parenthetical comparison "20.9% without zapping during ABS pretraining, compared to 24.8% with zapping" appears to misstate the conditions: 20.9% is the pre-train accuracy of ASB+zap and 24.8% is the transfer accuracy of ASB+zap; the non-zapped ASB row has pre-train accuracy 18.6% and transfer accuracy 19.1%. Please correct the wording to avoid confusing the reader.
  2. [§2.2, Fig. 2b caption] The caption's explanation of solid versus dashed lines is internally inconsistent: it first says dashed/solid lines show whether a model has undergone zapping during pre-training, then says solid lines indicate models that "just received their first zap" while dashed lines correspond to models that "had received several zaps already." Since both treatment conditions receive one last-layer resampling at the start of the measurement, the distinction is the pretraining history, not the number of zaps. Please rewrite the caption to state unambiguously that both treatment and control are copies of one pretrained model and that the line style denotes the pretraining condition.
  3. [§3.1] The paper reports p = 0.0017 for the IID versus ASB comparison, but it does not describe the statistical test, the number of replicates, or whether multiple comparisons across the six pretraining conditions were handled; please provide this information.
  4. [§4.3] The connection between Adam and the empirical Fisher information is presented as a hypothesis, but the discussion occasionally reads as if it were an established mechanism; please label it explicitly as a hypothesis and note that no direct evidence links the second-moment statistics to the observed backward transfer.
  5. [Conclusion] There is a duplicated word in the conclusion ("the the squared-gradient information"); also, the notation "SGD.µ = 0.9 =Adam.β1" in the Fig. 4 caption would benefit from cleaner formatting.

Circularity Check

0 steps flagged · score 2.0 of 10

Empirical study with no circular reduction; central claims rest on fresh transfer and per-task-loss measurements, with only minor non-load-bearing self-citations for setup choices.

full rationale

The paper's central claims are empirical: zapped models recover faster from transfer shock and Adam enables continued learning / minimal forgetting in sequential transfer. These are supported by new measurements—transfer-test accuracies on Omni-image (Table 1, Fig. 14), zap-divergence cosine-similarity curves for zapped vs. non-zapped pretraining (Fig. 2b), and per-task loss traces for SGD vs. Adam (Figs. 4-6, Tables 2-3). None of these outcomes is derived from an equation that contains the target conclusion by construction. The zap-divergence protocol independently compares a resampled treatment model with an unperturbed control and contrasts models that did vs. did not receive zapping during pretraining; the faster re-alignment of the FC layer and the smaller conv-layer drift are read off the measured curves, not imposed by the definition. The Adam backward-transfer observation is a direct loss trace, not a fitted parameter renamed as a prediction. The main self-reliance is on Frati et al. (2024) for the zapping schedule and ASB/Meta-ASB procedures and on Frati et al. (2023) for the Omni-image dataset; these are setup choices and do not force the measured outcomes. The concern that whole-weight-vector cosine similarity may not track functional re-alignment is a proxy-validity issue, not circularity, since no equation identifies the conclusion with its input. Overall, no significant circularity is present; the score reflects only the minor, non-load-bearing self-citations for experimental setup.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper does not derive a closed-form result, so free parameter accounting is minimal. The central empirical claims depend on a fixed zapping schedule inherited from the authors' prior work, on weight-space cosine similarity as a proxy for functional behavior, and on the representativeness of per-task loss curves. No new theoretical entities are introduced.

free parameters (1)
  • Zap schedule = zapped-IID: whole FC resampled at end of every epoch; ASB: single-class neuron resampled per batch
    Inherited from Frati et al. 2024, justified by prior empirical tuning and not re-swept in this paper; all reported zapping benefits are conditional on this schedule.
assumptions (3)
  • domain assumption Cosine similarity between whole flattened weight vectors is a meaningful proxy for functional similarity and recovery.
    Used in §2.2; the authors explicitly choose whole-vector cosine similarity instead of per-neuron averaging but do not validate it against function-space similarity.
  • domain assumption Per-task loss curves faithfully expose task-level learning, forgetting, synergy, and interference.
    Assumed in §2.3 and §3.3 when interpreting the 100-line loss plots; class order is randomized but the qualitative conclusions come from single trajectories without error bars.
  • domain assumption Results on Omniglot and Omni-image with this small convnet generalize to continual learning and transfer in general.
    The significance claims in §1 and §4 rest on two datasets and one architecture; no larger-scale evidence is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networks." pith.science (2026). https://pith.science/paper/67ZSGDGC

@misc{pith2026250701559,
  author       = {Pith},
  title        = {Pith review of: How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/67ZSGDGC}},
  note         = {Machine review of arXiv:2507.01559}
}
read the original abstract

Recent work in continual learning has highlighted the beneficial effect of resampling weights in the last layer of a neural network (``zapping"). Although empirical results demonstrate the effectiveness of this approach, the underlying mechanisms that drive these improvements remain unclear. In this work, we investigate in detail the pattern of learning and forgetting that take place inside a convolutional neural network when trained in challenging settings such as continual learning and few-shot transfer learning, with handwritten characters and natural images. Our experiments show that models that have undergone zapping during training more quickly recover from the shock of transferring to a new domain. Furthermore, to better observe the effect of continual learning in a multi-task setting we measure how each individual task is affected. This shows that, not only zapping, but the choice of optimizer can also deeply affect the dynamics of learning and forgetting, causing complex patterns of synergy/interference between tasks to emerge when the model learns sequentially at transfer time.

Figures

Figures reproduced from arXiv: 2507.01559 by the authors.

Figure 1
Figure 1. Test accuracy on classes during standard fine [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Measuring the effect of zapping the last layer of a model during i.i.d. training with the Adam optimizer. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Cosine similarity of layers after re-sampling the FC layer, using the Adam optimizer across 10 learning rates [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Comparison of learning dynamics in linear probing (colors indicate order of training): Both optimizers [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Full-model sequential training using Adam ( [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Full-model sequential training using SGD ( [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Convnet structure with normalization layers [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Cosine similarities for all layers, for multiple learning rates. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Per-task loss plot in an IID setting (vs sequential in other plots) at transfer time: Since batches contain a [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Different training dynamics for various learning rates [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Different training dynamics for various learning rates [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Comparing the effect of zapping on per-task losses: i) in the linear probing case learning is faster and losses [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Different training dynamics for various learning rates [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Training (left) and test (right) accuracy on classes during standard fine-tuning on the omni-image subset of [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 4 canonical work pages

  1. [4]

    Lapo Frati, Neil Traft, and Nick Cheney

    URL https://arxiv.org/abs/2002.10365v1. Lapo Frati, Neil Traft, and Nick Cheney. Omnimage: Evolving 1k image cliques for few-shot learning. InProceedings of the Genetic and Evolutionary Computation Conference , pp. 476–484,

    arXiv 2002

  2. [6]

    Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio

    URL https://arxiv.org/abs/1802.10026v4. Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research , 18 (187):1–30,

    arXiv

  3. [9]

    org/abs/1412.6980v9

    URL https://arxiv. org/abs/1412.6980v9. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521–3526,

    arXiv

  4. [10]

    Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang

    URL https://arxiv.org/abs/1806.08342v6. Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution,

    arXiv

  5. [12]

    Demystifying a Dark Art: Understanding Real-World Machine Learning Model Development

    URL https://arxiv.org/abs/2005.01520v1. Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes, 2018a. URL https://arxiv.org/abs/1804.08838v1. Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information p...

    work page Pith review arXiv 2005

  6. [14]

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu

    URL https://arxiv.org/abs/1607.08022v3. Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5362–5383,

    arXiv

  7. [15]

    Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson

    doi: 10.1109/TPAMI.2024.3367329. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems , 27,

    arXiv 2024

  8. [16]

    A N ETWORK STRUCTURE Following Frati et al

    12 Preprint. A N ETWORK STRUCTURE Following Frati et al. (2024) we employ a compact convolutional neural network consisting of three blocks, each con- taining convolution, InstanceNorm Ulyanov et al. (2017), ReLU activation, and max pooling layers (except for the final block, which omits pooling). All convolutional layers maintain 256 output channels. The...

    work page 2024

Show all 16 references
  1. [1964]

    icarl: Incremental classifier and representation learning

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pp. 2001–2010,

    2001

  2. [2017]

    Roy, and Surya Ganguli

    Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel, 2020a. URL https://arxiv.or...

    2010 arXiv

  3. [2018]

    Shun-Ichi Amari and Scott C Douglas

    URL https://arxiv.org/abs/1610.01644v4. Shun-Ichi Amari and Scott C Douglas. Why natural gradient? In Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP’98 (Cat. No. 98CH36181) , volume 2, pp. 1213–

    1998 arXiv

  4. [2019]

    Gaojie Jin, Xinping Yi, Liang Zhang, Lijun Zhang, Sven Schewe, and Xiaowei Huang

    URL https:// arxiv.org/abs/1905.12588v2. Gaojie Jin, Xinping Yi, Liang Zhang, Lijun Zhang, Sven Schewe, and Xiaowei Huang. How does weight correlation affect generalisation ability of deep neural networks? Advances in Neural Information Processing Systems , 33: 21346–21356,

    1905 arXiv

  5. [2020]

    Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Adelani, Pontus Lars Erik Saito Stenetorp, Sebastian Riedel, and Mikel Artetxe

    URL https://arxiv.org/abs/2002.09571v2. Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Adelani, Pontus Lars Erik Saito Stenetorp, Sebastian Riedel, and Mikel Artetxe. Improving language plasticity via pretraining with active forgetting. Advances in Neural Infor- mation ...

    2002 arXiv

  6. [2022]

    Dharshan Kumaran, Demis Hassabis, and James L McClelland

    URL https://arxiv.org/abs/2202.10054v1. Dharshan Kumaran, Demis Hassabis, and James L McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in cognitive sciences, 20(7):512–534,

    arXiv

  7. [2023]

    Reset it and forget it: Relearning last-layer weights improves continual and transfer learning

    Lapo Frati, Neil Traft, Jeff Clune, and Nick Cheney. Reset it and forget it: Relearning last-layer weights improves continual and transfer learning. In ECAI 2024, pp. 2998–3005. IOS Press,

    2024

  8. [2024]

    Khurram Javed and Martha White

    URL https://arxiv.org/abs/2405.12807v11. Khurram Javed and Martha White. Meta-learning representations for continual learning,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.