Pith. sign in

REVIEW 3 major objections 6 minor 89 references

Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Models

T0 review · 3 major / 6 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Continual self-supervised learning still needs large-scale pre-training paradigms; SSL's relative robustness to forgetting is not enough by itself.

desk verdict Solid first dedicated CSSL survey: clean taxonomy, honest protocol critique, and a scaling argument that matches the literature; the SSL-robustness story is synthesis, not new causal proof. read the letter →

arxiv 2607.09785 v1 pith:37DG3PRV submitted 2026-07-08 cs.CV cs.AI

Sergi Masip , Alicja Dobrzeniecka , Jonathan Swinnen , Joachim Collin , Bart{l}omiej Twardowski , Szymon {L}ukasik , Tinne Tuytelaars This is my paper
classification cs.CVcs.AI
keywords continualself-supervisedlearningcatastrophicforgettingrepresentationvisionmodelsvision-languagepre-trainingstability-plasticityevaluationprotocols
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Real systems that must keep learning from unlabeled visual streams cannot wait for labels or full retraining from scratch. This survey argues that continual self-supervised learning (CSSL) is the natural setting for that problem, yet the field lacks consistent evaluation and still treats itself largely as supervised continual learning without labels. The authors show why self-supervised objectives forget less readily than supervised ones: they learn task-agnostic features and sit in smoother, flatter loss landscapes. They then organize existing methods into a single taxonomy of forgetting-mitigation strategies and conclude that progress now requires leaving small benchmarks behind for continual pre-training of large models. A reader who cares about lifelong robots or foundation models that keep adapting without labels will find both a map of the current toolkit and a clear statement of what is still missing.

What carries the argument

A unified taxonomy of forgetting-mitigation strategies (distillation, especially projected distillation; replay; weight regularization; architectural isolation; model merging; objective-level adaptation) that organizes the literature and exposes the stability-plasticity trade-off under self-supervised losses.

What would settle it

A controlled multi-objective experiment that holds architecture, data volume, and training length fixed while swapping only the loss (supervised versus several SSL families) and measures both representation rank/CKA stability and forgetting; if the robustness gap disappears or is explained by other factors, the paper's explanatory claim fails.

Watch

Extended reading notes

Core claim

Self-supervised objectives produce more stable, task-agnostic representations and flatter loss landscapes than supervised objectives, so they are intrinsically less prone to catastrophic forgetting; yet this advantage does not solve continual adaptation. Existing CSSL methods still suffer gradual forgetting and plasticity loss, evaluation protocols are inconsistent, and advancing the field requires moving from small-scale benchmarks to continual pre-training paradigms for large-scale systems.

Load-bearing premise

The claim that SSL's reduced forgetting is mainly caused by task-agnostic features and flatter loss landscapes rests on a limited set of cited empirical studies rather than a controlled analysis that isolates those factors from training length, data volume, or other confounds.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This survey reviews continual self-supervised learning (CSSL) for vision, with extensions to vision–language settings. It (i) catalogs training protocols, metrics, and benchmarks and argues that inconsistent evaluation practices impede fair comparison; (ii) synthesizes prior evidence that SSL objectives are relatively robust to catastrophic forgetting, attributing this mainly to task-agnostic representations and smoother/flatter loss landscapes; (iii) organizes existing methods into a six-family taxonomy (distillation, weight regularization, replay, architectural methods, model merging, objective-level adaptation), with additional discussion of SSL as a component in hybrid supervised CL pipelines; and (iv) identifies open challenges—evaluation standardization, the stability–plasticity trade-off, cross-objective generalization, and scaling—and argues that the field should move from small-scale class-incremental benchmarks toward continual pre-training of large-scale systems.

Significance. The paper fills a clear gap: CSSL for vision (and its multimodal extensions) has lacked a dedicated, systematic survey, while related reviews treat it only peripherally. The taxonomy is coherent and usable, the multimodal and hybrid sections are timely, and the scaling argument is well motivated by documented gaps in evaluation, compute, and long-horizon plasticity. Strengths include explicit takeaways per method family, a challenges map (Fig. 2), and a programmatic agenda that is falsifiable in the sense that it points to concrete next experiments (unified protocols, ViT-native objectives under streams, continual foundation-model pre-training). As a literature synthesis rather than a new causal study, its value is organizational and directional; if the field adopts the recommended evaluation and scaling practices, the survey would serve as a useful reference point.

major comments (3)
  1. [Section II / Table I] §II and Table I: The abstract and introduction list “analysis of existing evaluation protocols and highlight[ing of] inconsistencies that hinder fair comparison” as a primary contribution, yet §II-D and Table I only name a handful of datasets with coarse protocol/modality/scale labels. There is no systematic documentation of the concrete variations that actually break comparability (e.g., number of tasks/splits of CIFAR-100 or ImageNet-100, epochs vs. one-pass, linear vs. k-NN vs. fine-tuning probes, presence/absence of task boundaries, reporting of AA vs. F vs. BWT vs. CKA). Without such a comparison, the inconsistency claim is asserted more than demonstrated. A compact protocol-variation table (or expanded Table I) is needed to make this contribution load-bearing.
  2. [Section IV] §IV: The six-family taxonomy is clear in narrative form and Fig. 3, but the survey lacks a consolidated methods table (method, year, family, SSL objective(s), offline vs. online/task-free, replay/distillation/etc. components, main benchmarks). For a taxonomy survey this is a standard reference artifact; its absence reduces the paper’s utility as a lookup resource and makes it harder to verify coverage and family boundaries. Adding such a table would substantially strengthen the central organizational claim without changing the taxonomy itself.
  3. [Section III] §III: The robustness discussion is framed as SSL “stems from two core properties” (task-agnostic features; smoother/flatter landscapes), citing a relatively small set of CKA and landscape studies. As a survey synthesis this is acceptable, but the causal language should be tempered: confounds such as training length, data volume, and batch size are only lightly acknowledged, and no multi-objective controlled isolation is claimed in the cited works. Softening to “working hypotheses supported by prior empirical studies” and explicitly listing confounds would align the epistemic status with the evidence and avoid over-reading §III as a primary mechanistic result. This does not undermine the taxonomy or scaling agenda, which do not depend on that mechanism being uniquely true.
minor comments (6)
  1. [Section II-C] §II-C: Multimodal metrics (AKA, AZS) are introduced briefly; a one-line formal definition or pointer to the exact formulas in [22], [23] would help readers who do not have those papers open.
  2. [Figure 2] Fig. 2 is useful but dense; ensuring that every acronym (PFR, CaSSLe, CLA, POCON, LUMP, etc.) is expanded at first mention in the caption or adjacent text would improve standalone readability.
  3. [Section IV-A] §IV-A takeaway notes the extra forward pass cost of projected distillation; a short quantitative remark (e.g., relative wall-clock or FLOPs where reported in the cited works) would make the efficiency caveat more concrete.
  4. [Section VI-C] §VI-C: The observation that findings may not transfer across contrastive / non-contrastive / MIM families is important; citing the specific CaSSLe/Branch-Tuning/CLA results in a small summary bullet or table would make the “family-specific behaviors” claim easier to reuse.
  5. Minor polish: ensure consistent hyphenation of “self-supervised” / “task-agnostic” / “vision-language” and consistent capitalization of method names (e.g., SwAV vs. SwA V) throughout.
  6. [References] References: a few arXiv-only entries that now have venue versions (if any) could be updated for archival stability; not blocking.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: literature survey with external grounding, no self-referential derivations or fitted-as-prediction steps.

full rationale

This paper is a systematic survey of continual self-supervised learning. Its contributions are organizational (evaluation-protocol critique, method taxonomy by forgetting-mitigation strategy, open-challenge framing) and programmatic (call for continual pre-training at foundation-model scale). It does not claim a first-principles derivation, uniqueness theorem, or quantitative prediction obtained by fitting parameters to data. Section III’s account of SSL robustness (task-agnostic geometry, flatter loss landscapes) is explicitly a synthesis of prior empirical CKA and landscape studies, not a closed-form result forced by the authors’ own definitions. Occasional citations of co-authored related work (e.g., PFR, POCON) appear as ordinary pointers within a broad external literature and do not load-bear any uniqueness claim or redefine the target quantities. There is therefore no self-definitional loop, no fitted input renamed as prediction, and no ansatz smuggled in via self-citation. Circularity score is zero.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

As a survey the paper inherits the empirical and theoretical claims of the cited CSSL literature rather than introducing free parameters or new physical entities. The main load-bearing background assumptions are standard continual-learning and SSL premises (catastrophic forgetting exists, SSL pretext tasks yield useful representations, CKA measures representational similarity). No numerical free parameters are fitted; the taxonomy categories are organizational rather than postulated mechanisms.

assumptions (3)
  • domain assumption Self-supervised objectives produce more task-agnostic, higher-rank representations and flatter loss landscapes than supervised cross-entropy, thereby reducing catastrophic forgetting.
    Invoked throughout Section III as the explanatory core; rests on cited empirical studies (CKA analyses, landscape visualizations) rather than a derivation internal to the survey.
  • domain assumption Existing CSSL evaluation protocols differ substantially in task construction, offline/online setting, and probe type, rendering direct numerical comparison unreliable.
    Stated in Section II and used to motivate the call for unified benchmarks; treated as an observed fact of the literature.
  • standard math Joint training on the full data stream remains a valid upper-bound reference for measuring forgetting and plasticity gaps.
    Standard continual-learning convention adopted without re-derivation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Models." pith.science (2026). https://pith.science/paper/37DG3PRV

@misc{pith2026260709785,
  author       = {Pith},
  title        = {Pith review of: Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/37DG3PRV}},
  note         = {Machine review of arXiv:2607.09785}
}
read the original abstract

Traditionally, continual learning has assumed access to labeled data, yet many real-world applications -- such as lifelong robotics -- require models to adapt continuously from unlabeled streams. This has led to the development of continual self-supervised learning (CSSL), a rapidly growing area that lacks a dedicated, systematic review. In this work, we present a comprehensive survey of CSSL for vision, with connections to emerging vision-language settings. First, we analyze existing evaluation protocols and highlight inconsistencies that hinder fair comparison. We then examine why self-supervised objectives exhibit improved robustness to catastrophic forgetting, relating this to task-agnostic representations and smoother loss landscapes. Next, we organize existing methods into a unified taxonomy based on their forgetting-mitigation strategies, including distillation, replay, regularization, architectural approaches, model merging, and objective-level adaptation. Finally, we identify open challenges such as scalability and the need for fast adaptability. We argue that advancing CSSL requires moving beyond small-scale benchmarks towards continual pre-training paradigms for large-scale systems.

Figures

Figures reproduced from arXiv: 2607.09785 by the authors.

Figure 1
Figure 1. A conceptual visualization of CSSL. The bottom part [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Challenges in CSSL and representative solutions. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

89 extracted references · 11 linked inside Pith

  1. [1]

    Geode: a geographically diverse evaluation dataset for object recognition,

    V . V . Ramaswamyet al., “Geode: a geographically diverse evaluation dataset for object recognition,”Advances in Neural Information Process- ing Systems, vol. 36, pp. 66 127–66 137, 2023

    2023

  2. [2]

    Catastrophic interference in connec- tionist networks: The sequential learning problem,

    M. McCloskey and N. J. Cohen, “Catastrophic interference in connec- tionist networks: The sequential learning problem,” ser. Psychology of Learning and Motivation, G. H. Bower, Ed. Academic Press, 1989, vol. 24, pp. 109–165

    1989

  3. [3]

    Three continual learning scenarios,

    G. M. Van de Ven and A. S. Tolias, “Three continual learning scenarios,” inNeurIPS Continual Learning Workshop, vol. 1, no. 9, 2018, p. 4

    2018

  4. [4]

    Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey ,

    L. Jing and Y . Tian, “ Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey ,”IEEE TPAMI, vol. 43, no. 11, pp. 4037–4058, Nov. 2021

    2021

  5. [5]

    The future of continual learning in the era of foundation models: Three key directions,

    J. Bellet al., “The future of continual learning in the era of foundation models: Three key directions,” 2025. [Online]. Available: https://arxiv.org/abs/2506.03320

    arXiv 2025

  6. [6]

    Continual learning of large language models: A compre- hensive survey,

    H. Shiet al., “Continual learning of large language models: A compre- hensive survey,”ACM Comput. Surv., vol. 58, no. 5, Nov. 2025

    2025

  7. [7]

    Continual learning for vlms: A survey and taxonomy beyond forgetting,

    Y . Liuet al., “Continual learning for vlms: A survey and taxonomy beyond forgetting,” 2025. [Online]. Available: https: //arxiv.org/abs/2508.04227

    arXiv 2025

  8. [8]

    Beyond supervised continual learning: a review,

    B. Bagus, A. Gepperth, and T. Lesort, “Beyond supervised continual learning: a review,” 2022. [Online]. Available: https: //arxiv.org/abs/2208.14307

    arXiv 2022

Show all 89 references
  1. [9]

    Towards label-efficient incremental learning: A survey,

    M. Kilickaya, J. van de Weijer, and Y . M. Asano, “Towards label-efficient incremental learning: A survey,” 2023. [Online]. Available: https://arxiv.org/abs/2302.00353

    2023 arXiv

  2. [10]

    A survey of the self supervised learning mechanisms for vision transformers,

    A. Khanet al., “A survey of the self supervised learning mechanisms for vision transformers,” 2025

    2025

  3. [11]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” inInt. Conf. on Machine Learning, 2020, pp. 1597–1607

    2020

  4. [12]

    Momentum contrast for unsupervised visual representation learning,

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2020, pp. 9729–9738

    2020

  5. [13]

    Improved baselines with momentum contrastive learning,

    X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” 2020. [Online]. Available: https://arxiv.org/abs/2003.04297

    2020 arXiv

  6. [14]

    Representation learning with contrastive predictive coding,

    A. van den Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” 2019. [Online]. Available: https://arxiv.org/abs/1807.03748

    2019 arXiv

  7. [15]

    Bootstrap your own latent-a new approach to self- supervised learning,

    J.-B. Grillet al., “Bootstrap your own latent-a new approach to self- supervised learning,” inAdvances in Neural Information Processing Systems, vol. 33, 2020, pp. 21 271–21 284

    2020

  8. [16]

    Exploring simple siamese representation learning,

    X. Chen and K. He, “Exploring simple siamese representation learning,” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2021, pp. 15 750–15 758

    2021

  9. [17]

    Barlow twins: Self-supervised learning via redundancy reduction,

    J. Zbontar, L. Jing, I. Misra, Y . LeCun, and S. Deny, “Barlow twins: Self-supervised learning via redundancy reduction,” inInt. Conf. on Machine Learning. PMLR, 2021, pp. 12 310–12 320

    2021

  10. [18]

    Vicreg: Variance-invariance- covariance regularization for self-supervised learning,

    A. Bardes, J. Ponce, and Y . LeCun, “Vicreg: Variance-invariance- covariance regularization for self-supervised learning,” inInt. Conf. on Learning Representations, 2022

    2022

  11. [19]

    Unsupervised learning of visual features by contrasting cluster assignments,

    M. Caronet al., “Unsupervised learning of visual features by contrasting cluster assignments,” vol. 33, 2020, pp. 9912–9924

    2020

  12. [20]

    Masked autoencoders are scalable vision learners,

    K. Heet al., “Masked autoencoders are scalable vision learners,” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2022, pp. 16 000–16 009

    2022

  13. [21]

    Cla: Latent alignment for online continual self-supervised learning,

    G. Cignoni, A. Cossu, A. Gomez-Villa, J. van de Weijer, and A. Carta, “Cla: Latent alignment for online continual self-supervised learning,”

  14. [22]

    Available: https://arxiv.org/abs/2507.10434

    [Online]. Available: https://arxiv.org/abs/2507.10434

    arXiv

  15. [23]

    A practitioner’s guide to continual multimodal pretraining,

    V . Udandaraoet al., “A practitioner’s guide to continual multimodal pretraining,” inAdvances in Neural Information Processing Systems, 2024

    2024

  16. [24]

    Tic-clip: Continual training of clip models,

    S. Garget al., “Tic-clip: Continual training of clip models,” inInt. Conf. on Learning Representations, 2024

    2024

  17. [25]

    Branch-tuning: balancing stability and plasticity for continual self-supervised learning,

    W. Liu, F. Zhu, and C.-L. Liu, “Branch-tuning: balancing stability and plasticity for continual self-supervised learning,”IEEE Transactions on Neural Networks and Learning Systems, 2025

    2025

  18. [26]

    Similarity of neural network representations revisited,

    S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, “Similarity of neural network representations revisited,” inInt. Conf. on Machine Learning. PMlR, 2019, pp. 3519–3529

    2019

  19. [27]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, “Learning multiple layers of features from tiny images,” pp. 32–33, 2009. [Online]. Available: https://www.cs.toronto.edu/~kriz/ learning-features-2009-TR.pdf

    2009

  20. [28]

    Imagenet: A large-scale hierarchical image database,

    J. Denget al., “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE Conf. on Computer Vision and Pattern Recognition. Ieee, 2009, pp. 248–255

    2009

  21. [29]

    Representational continuity for unsupervised continual learning,

    D. Madaan, J. Yoon, Y . Li, Y . Liu, and S. J. Hwang, “Representational continuity for unsupervised continual learning,” inInt. Conf. on Learning Representations, 2022

    2022

  22. [30]

    Self-supervised class incremental learning,

    Z. Ni, S. Tang, and Y . Zhuang, “Self-supervised class incremental learning,” 2021. [Online]. Available: https://arxiv.org/abs/2111.11208

    2021 arXiv

  23. [31]

    Self-supervised models are continual learners,

    E. Finiet al., “Self-supervised models are continual learners,” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2022, pp. 9621–9630

    2022

  24. [32]

    Prevalence of neural collapse during the terminal phase of deep learning training,

    V . Papyan, X. Han, and D. L. Donoho, “Prevalence of neural collapse during the terminal phase of deep learning training,”Proc. of the National Academy of Sciences, vol. 117, no. 40, pp. 24 652–24 663, 2020

    2020

  25. [33]

    Perturbation analysis of neural collapse,

    T. Tirer, H. Huang, and J. Niles-Weed, “Perturbation analysis of neural collapse,” inInt. Conf. on Machine Learning. PMLR, 2023, pp. 34 301– 34 329

    2023

  26. [34]

    Why do better loss functions lead to less transferable features?

    S. Kornblith, T. Chen, H. Lee, and M. Norouzi, “Why do better loss functions lead to less transferable features?”Advances in Neural Information Processing Systems, vol. 34, pp. 28 648–28 662, 2021

    2021

  27. [35]

    The tunnel effect: Building data representations in deep neural networks,

    W. Masarczyket al., “The tunnel effect: Building data representations in deep neural networks,”Advances in Neural Information Processing Systems, vol. 36, pp. 76 772–76 805, 2023

    2023

  28. [36]

    What variables affect out-of-distribution generalization in pretrained models?

    Y . Harun, K. Lee, J. Gallardo, G. Krishnan, and C. Kanan, “What variables affect out-of-distribution generalization in pretrained models?” Advances in Neural Information Processing Systems, vol. 37, pp. 56 479– 56 525, 2024

    2024

  29. [37]

    Know your self-supervised learning: A survey on image-based generative and discriminative training,

    U. Ozbulak,et al., “Know your self-supervised learning: A survey on image-based generative and discriminative training,”Transactions on Machine Learning Research, 2024

    2024

  30. [38]

    Reverse engineering self-supervised learning,

    I. Ben-Shaul, R. Shwartz-Ziv, T. Galanti, S. Dekel, and Y . LeCun, “Reverse engineering self-supervised learning,”Advances in Neural Information Processing Systems, vol. 36, pp. 58 324–58 345, 2023

    2023

  31. [39]

    On the discriminability of self-supervised representation learning,

    Z. Song, W. Qiang, C. Zheng, F. Sun, and H. Xiong, “On the discriminability of self-supervised representation learning,”Information Sciences, p. 122556, 2025

    2025

  32. [40]

    Emerging properties in self-supervised vision trans- formers,

    M. Caronet al., “Emerging properties in self-supervised vision trans- formers,” inProc. of the IEEE/CVF Int. Conf. on Computer Vision, 2021, pp. 9650–9660

    2021

  33. [41]

    Self-supervised learning from images with a joint- embedding predictive architecture,

    M. Assranet al., “Self-supervised learning from images with a joint- embedding predictive architecture,” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2023, pp. 15 619–15 629

    2023

  34. [42]

    Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank,

    Q. Garrido, R. Balestriero, L. Najman, and Y . Lecun, “Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank,” inInt. Conf. on Machine Learning. PMLR, 2023, pp. 10 929–10 974

    2023

  35. [43]

    How well does self-supervised pre-training perform with streaming data?

    D. Huet al., “How well does self-supervised pre-training perform with streaming data?” inInt. Conf. on Learning Representations, 2022

    2022

  36. [44]

    Task agnostic representation consolidation: a self-supervised based continual learning approach,

    P. S. Bhat, B. Zonooz, and E. Arani, “Task agnostic representation consolidation: a self-supervised based continual learning approach,” in Conf. on Lifelong Learning Agents. PMLR, 2022, pp. 390–405

    2022

  37. [45]

    Towards efficient and effective self-supervised learning of visual representations,

    S. Addepalli, K. Bhogale, P. Dey, and R. V . Babu, “Towards efficient and effective self-supervised learning of visual representations,” inEuropean Conf. on Computer Vision. Springer, 2022, pp. 523–538

    2022

  38. [46]

    Accelerating self- supervised learning via efficient training strategies,

    M. T. Koçyi ˘git, T. M. Hospedales, and H. Bilen, “Accelerating self- supervised learning via efficient training strategies,” inProc. of the IEEE/CVF Winter Conf. on Applications of Computer Vision, 2023, pp. 5654–5664

    2023

  39. [47]

    Faster convergence and uncorrelated gradients in self-supervised online continual learning,

    K. Imai, N. Hayashi, T. Hirakawa, T. Yamashita, and H. Fujiyoshi, “Faster convergence and uncorrelated gradients in self-supervised online continual learning,” inProc. of the Asian Conf. on Computer Vision, 2024, pp. 436–453

    2024

  40. [48]

    Continually learning self-supervised representations with pro- jected functional regularization,

    A. Gomez-Villa, B. Twardowski, L. Yu, A. D. Bagdanov, and J. Van de Weijer, “Continually learning self-supervised representations with pro- jected functional regularization,” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2022, pp. 3867–3877

    2022

  41. [49]

    Plasticity-optimized complementary networks for unsupervised continual learning,

    A. Gomez-Villa, B. Twardowski, K. Wang, and J. Van de Weijer, “Plasticity-optimized complementary networks for unsupervised continual learning,” inProc. of the IEEE/CVF Winter Conf. on Applications of Computer Vision, 2024, pp. 1690–1700

    2024

  42. [50]

    Continual pre-training mitigates forgetting in language and vision,

    A. Cossuet al., “Continual pre-training mitigates forgetting in language and vision,”Neural Networks, vol. 179, p. 106492, 2024

    2024

  43. [51]

    Adaptive self-supervised continual learning,

    L. Wu, Z. Wang, and J. Liu, “Adaptive self-supervised continual learning,” inECAI 2023. IOS Press, 2023, pp. 2680–2687

    2023

  44. [52]

    How to merge your multimodal models over time?

    S. Dziadzioet al., “How to merge your multimodal models over time?” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2025, pp. 20 479–20 491

    2025

  45. [53]

    Cromo-mixup: Augmenting cross-model represen- tations for continual self-supervised learning,

    E. Mushtaqet al., “Cromo-mixup: Augmenting cross-model represen- tations for continual self-supervised learning,” inEuropean Conf. on Computer Vision. Springer, 2024, pp. 311–328

    2024

  46. [54]

    Continual self-supervised learning: Towards universal multi- modal medical data representation learning,

    Y . Yeet al., “Continual self-supervised learning: Towards universal multi- modal medical data representation learning,” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2024, pp. 11 114– 11 124

    2024

  47. [55]

    Continual self-supervised learning with masked autoencoders in remote sensing,

    L. Möllenbrok, B. Rasti, and B. Demir, “Continual self-supervised learning with masked autoencoders in remote sensing,”IEEE Geoscience and Remote Sensing Letters, 2025

    2025

  48. [56]

    Generative negative text replay for continual vision- language pretraining,

    S. Yanet al., “Generative negative text replay for continual vision- language pretraining,” inEuropean Conf. on Computer Vision. Springer, 2022, pp. 22–38

    2022

  49. [57]

    Ctp: Towards vision- language continual pretraining via compatible momentum contrast and topology preservation,

    H. Zhu, Y . Wei, X. Liang, C. Zhang, and Y . Zhao, “Ctp: Towards vision- language continual pretraining via compatible momentum contrast and topology preservation,” inProc. of the IEEE/CVF Int. Conf. on Computer Vision, 2023, pp. 22 257–22 267

    2023

  50. [58]

    Continual vision- language representation learning with off-diagonal information,

    Z. Ni, L. Wei, S. Tang, Y . Zhuang, and Q. Tian, “Continual vision- language representation learning with off-diagonal information,” inProc. of the 40th Int. Conf. on Machine Learning. PMLR, 2023, pp. 26 129– 26 149

    2023

  51. [59]

    Continual contrastive learning for image classification,

    Z. Lin, Y . Wang, and H. Lin, “Continual contrastive learning for image classification,” in2022 IEEE International conference on multimedia and expo (ICME). IEEE, 2022, pp. 1–6

    2022

  52. [60]

    Revisiting supervision for continual representation learning,

    D. Marczak, S. Cygert, T. Trzci ´nski, and B. Twardowski, “Revisiting supervision for continual representation learning,” inEuropean Conf. on Computer Vision. Springer, 2024, pp. 181–197

    2024

  53. [61]

    Regularizing with pseudo-negatives for continual self-supervised learning,

    S. Cha, K. Cho, and T. Moon, “Regularizing with pseudo-negatives for continual self-supervised learning,” inInt. Conf. on Machine Learning. PMLR, 2024, pp. 6048–6065

    2024

  54. [62]

    C-clip: Multimodal continual learn- ing for vision-language model,

    W. Liu, F. Zhu, L. Wei, and Q. Tian, “C-clip: Multimodal continual learn- ing for vision-language model,” inInt. Conf. on Learning Representations, 2025

    2025

  55. [63]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatricket al., “Overcoming catastrophic forgetting in neural networks,”Proc. of the national academy of sciences, vol. 114, no. 13, pp. 3521–3526, 2017

    2017

  56. [64]

    Continual learning through synaptic intelligence,

    F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” inInt. Conf. on Machine Learning. Pmlr, 2017, pp. 3987–3995

    2017

  57. [65]

    Memory aware synapses: Learning what (not) to forget,

    R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, “Memory aware synapses: Learning what (not) to forget,” inProc. of the European Conf on Computer Vision, 2018, pp. 139–154

    2018

  58. [66]

    Are labels needed for incremental instance learning?

    M. Kilickaya and J. Vanschoren, “Are labels needed for incremental instance learning?” inProc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2023, pp. 2401–2409

    2023

  59. [67]

    Continual barlow twins: continual self- supervised learning for remote sensing semantic segmentation,

    V . Marsocci and S. Scardapane, “Continual barlow twins: continual self- supervised learning for remote sensing semantic segmentation,”IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 16, pp. 5049–5060, 2023

    2023

  60. [68]

    Continual retinal vision-language pre-training upon incremental imaging modalities,

    Y . Yao, R. Wu, Y . Zhou, and T. Zhou, “Continual retinal vision-language pre-training upon incremental imaging modalities,” inMedical Image Computing and Computer Assisted Intervention. Springer, 2025, pp. 111–121

    2025

  61. [69]

    Beyond cosine decay: On the effectiveness of infinite learning rate schedule for continual pre-training,

    V . Singhet al., “Beyond cosine decay: On the effectiveness of infinite learning rate schedule for continual pre-training,”Fourth Conf. on Lifelong Learning Agents, 2025

    2025

  62. [70]

    Experience replay for continual learning,

    D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne, “Experience replay for continual learning,”Advances in Neural Information Processing Systems, vol. 32, 2019

    2019

  63. [71]

    mixup: Beyond empirical risk minimization,

    H. Zhang, M. Cisse, Y . N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,”Int. Conf. on Learning Representations, 2018

    2018

  64. [72]

    Contrastive continuity on augmentation stability re- hearsal for continual self-supervised learning,

    H. Chenget al., “Contrastive continuity on augmentation stability re- hearsal for continual self-supervised learning,” inProc. of the IEEE/CVF Int. Conf. on Computer Vision, 2023, pp. 5707–5717

    2023

  65. [73]

    Cucl: Codebook for unsupervised continual learning,

    C. Chenget al., “Cucl: Codebook for unsupervised continual learning,” inProc. of the 31st ACM Int. Conf. on Multimedia, 2023, pp. 1729–1737

    2023

  66. [74]

    Effective data selection and replay for unsupervised continual learning,

    L. Hanmoet al., “Effective data selection and replay for unsupervised continual learning,” inIEEE 40th Int. Conf. on Data Engineering. IEEE, 2024, pp. 1449–1463

    2024

  67. [75]

    Memory storyboard: Leveraging temporal segmentation for streaming self-supervised learning from egocentric videos,

    Y . Yang and M. Ren, “Memory storyboard: Leveraging temporal segmentation for streaming self-supervised learning from egocentric videos,”Fourth Conference on Lifelong Learning Agents, 2025

    2025

  68. [76]

    Unsupervised continual learning via self-adaptive deep clustering approach,

    M. Pratama, A. Ashfahani, and E. Lughofer, “Unsupervised continual learning via self-adaptive deep clustering approach,” inInt. Workshop on Continual Semi-Supervised Learning. Springer, 2021, pp. 48–61

    2021

  69. [77]

    Efficient self-supervised continual learning with progressive task-correlated layer freezing,

    L. Yang, S. Lin, F. Zhang, J. Zhang, and D. Fan, “Efficient self-supervised continual learning with progressive task-correlated layer freezing,” in 26th Int. Symposium on Quality Electronic Design. IEEE, 2025, pp. 1–8

    2025

  70. [78]

    Lora: Low-rank adaptation of large language models,

    E. Huet al., “Lora: Low-rank adaptation of large language models,” in Int. Conf. on Learning Representations, 2022

    2022

  71. [79]

    Learning transferable visual models from natural language supervision,

    A. Radfordet al., “Learning transferable visual models from natural language supervision,”CoRR, vol. 139, pp. 8748–8763, 2021. [Online]. Available: https://arxiv.org/abs/2103.00020

    2021 arXiv

  72. [80]

    Replay-free online continual learning with self-supervised multipatches,

    G. Cignoni, A. Cossu, A. Gomez-Villa, J. van de Weijer, and A. Carta, “Replay-free online continual learning with self-supervised multipatches,”

  73. [81]

    Available: https://arxiv.org/abs/2502.09140

    [Online]. Available: https://arxiv.org/abs/2502.09140

    arXiv

  74. [82]

    Infoucl: Learning informative representations for unsupervised continual learning,

    L. Zhang, J. Zhao, Q. Wu, L. Pan, and H. Li, “Infoucl: Learning informative representations for unsupervised continual learning,”IEEE Transactions on Multimedia, vol. 26, pp. 10 779–10 791, 2024

    2024

  75. [83]

    Self-supervised learning aided class-incremental lifelong learning,

    S. Zhang, G. Shen, J. Huang, and Z.-H. Deng, “Self-supervised learning aided class-incremental lifelong learning,” 2020. [Online]. Available: https://arxiv.org/abs/2006.05882

    2020 arXiv

  76. [84]

    Dualnet: Continual learning, fast and slow,

    Q. Pham, C. Liu, and S. Hoi, “Dualnet: Continual learning, fast and slow,”Advances in Neural Information Processing Systems, vol. 34, pp. 16 131–16 144, 2021

    2021

  77. [85]

    Self-supervised continual learning for sar-atr: A local feature adaptation framework,

    S. Danget al., “Self-supervised continual learning for sar-atr: A local feature adaptation framework,”IEEE Transactions on Aerospace and Electronic Systems, vol. 62, pp. 1378–1393, 2025

    2025

  78. [86]

    Kaizen: Practical self-supervised continual learning with continual fine-tuning,

    C. I. Tanget al., “Kaizen: Practical self-supervised continual learning with continual fine-tuning,” inProc. of the IEEE/CVF Winter Conf. on Applications of Computer Vision, 2024, pp. 2841–2850

    2024

  79. [87]

    Masked autoencoders are efficient class incremental learners,

    J.-T. Zhai, X. Liu, A. D. Bagdanov, K. Li, and M.-M. Cheng, “Masked autoencoders are efficient class incremental learners,” inProc. of the IEEE/CVF Int. Conf. on Computer Vision, 2023, pp. 19 104–19 113

    2023

  80. [88]

    Masked autoencoders are parameter-efficient federated continual learners,

    Y . He and X. Wang, “Masked autoencoders are parameter-efficient federated continual learners,” in2024 IEEE Int. Conf. on Big Data (BigData). IEEE, 2024, pp. 3682–3691

    2024

  81. [89]

    Understanding contrastive representation learning through alignment and uniformity on the hypersphere,

    T. Wang and P. Isola, “Understanding contrastive representation learning through alignment and uniformity on the hypersphere,” inInt. Conf. on Machine Learning. PMLR, 2020, pp. 9929–9939

    2020

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.