Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

Mind the Gap: Preserving and Compensating for the Modality Gap in CLIP-Based Continual Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The separation between CLIP's image and text features is a living gauge of forgotten knowledge, and this paper turns that gauge into a training rule.

desk verdict MG-CLIP is a solid empirical contribution with a real confound between its modality-gap narrative and plain early stopping; worth reviewing, but the causal claim needs a matched-epoch control. read the letter →

arxiv 2507.09118 v1 pith:VSIXFBR6 submitted 2025-07-12 cs.CV cs.LG

Linlan Huang , Xusheng Cao , Haori Lu , Yifan Meng , Fei Yang , Xialei Liu This is my paper · ORCID
classification cs.CVcs.LG
keywords modalitygapcontinuallearningclass-incrementalCLIPvision-languagemodelsearlystoppingzero-shotgeneralizationlow-rankadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the modality gap of CLIP — the characteristic separation between image features and text features in the shared embedding space — acts as a gauge of how much pre-trained knowledge survives fine-tuning. Its central claim is that forgetting in continual learning is driven by the widening of this gap, and that keeping the gap relatively stable should therefore prevent forgetting. The proposed method, MG-CLIP, does two things: it stops fine-tuning each task early, as soon as the mean image-to-text similarity of non-matching pairs drifts beyond a threshold, and it adds a classifier trained purely in the visual space to compensate for the fact that the preserved gap restricts text classifiers. On five class-incremental benchmarks the paper reports that this recipe beats replay-based and prompt-based methods while keeping CLIP's zero-shot ability closer to its original level. The reason a reader should care is that it turns a structural curiosity of vision-language models into a training signal that needs no replay data, no memory banks, and almost no extra parameters.

What carries the argument

The carrying object is the modality gap itself, quantified by the mean negative image-text cosine similarity of Eq. (2): the average cosine distance from an image feature to the text features of all classes it does not belong to. The control mechanism is the relative-change criterion $\Delta$ of Eq. (3), which measures how far the negative similarity has drifted from its zero-shot value during the first task and, once it exceeds a 10% threshold, fixes the number of fine-tuning epochs $e$ that is then applied to all later tasks. The compensating mechanism is an intra-modal visual-space classifier: new-class weights are initialized from class prototypes and trained on frozen image features, and its softmax output is blended with the text classifier's output through the combination $p(x) = f_{\mathrm{clip}}(x, t) + \beta \cdot \mathrm{softmax}(W_v^\top x)$. A rank argument (Eq. (4)) supplies the theoretical motivation for compensation: any optimal classifier can be assumed to live in the span of the image features, and a text-feature classifier, sitting in a separate cone, can only reach a lower-rank subspace of that span, with an error bounded below by the singular values that lie outside the text-feature subspace.

What would settle it

Keep every component of MG-CLIP but replace the modality-gap criterion with a fixed small epoch count (say two epochs) or with validation-based early stopping; if ImageNet-R and VTAB accuracies are statistically unchanged, the gap signal is not doing the causal work. A second check: on a dataset where the negative cosine drops past the 10% threshold within the first epoch or two — for instance under a higher initial learning rate — the budget shrinks to one epoch for all tasks, and if performance stays high despite the gap widening beyond the claimed safe regime, the threshold is not actually detecting knowledge damage.

Watch

Extended reading notes

Core claim

The paper's discovery, stated on its own terms, is that in CLIP-based class-incremental learning the mean cosine similarity between an image and the text embeddings of non-matching classes is a reliable reflection of how much pre-trained knowledge remains: as tasks accumulate, naive cross-entropy fine-tuning pushes this negative similarity downward, the modality gap widens, and accuracy on earlier tasks falls in step. Fine-tuning with a direct alignment loss collapses the gap instead, and that also degrades performance, so the paper concludes the healthy regime is a stable middle: preserve the gap, do not shrink or expand it. Concretely, it proposes computing the negative-similarity statistic on the first task, fine-tuning with LoRA until the relative change of that statistic crosses a threshold, and reusing the resulting epoch budget for every later task; then, because the preserved gap leaves text classifiers confined to a lower-rank subspace of the image feature space, it trains a cosine classifier on image features alone and adds its softmax output to the text classifier's logits at inference. If the central claim is right, the modality gap is not a nuisance to be removed but a preserved quantity that trades off stability and plasticity for CLIP.

Load-bearing premise

Everything rests on the assumption that the drop in the average similarity between an image and the wrong-class text embeddings, measured on the first task, is a reliable early-warning signal for damage to CLIP's pre-trained knowledge, and that the single epoch budget it produces keeps holding for every later task and every dataset.

Editorial extensions

If this is right

  • Class-incremental learning with CLIP can beat replay-based and prompt-based methods on CIFAR-100, ImageNet-R, ImageNet-100, ImageNet-1K, and VTAB using no replay data and about 0.54M extra parameters.
  • Zero-shot generalization on Food101, Oxford Pets, and ImageNet-1K stays near or above the original CLIP's level after all tasks, whereas replay-based baselines degrade substantially.
  • The method transfers across backbones (ViT-B/16 and ViT-L/14), task counts (5, 10, and 20 tasks), and data regimes, with its largest edge in the harder 20-task and few-shot settings.
  • The approach is compatible with replay if a user wants it: adding 2000 rehearsed samples improves its results further, which the paper reports as a separate experiment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the negative image-text similarity is a general indicator of pre-trained knowledge, the same $\Delta$ criterion could serve as an early-stopping or checkpoint-selection signal for any CLIP fine-tuning, not just continual learning — for instance in domain adaptation or instruction tuning where forgetting is also a concern.
  • The transfer of the epoch budget from the first task to all later tasks is the fragile link; a per-task adaptive budget that re-measures $\Delta$ at each boundary could strengthen the method where drift dynamics differ between tasks.
  • The rank-based bound suggests a cheap diagnostic: the ratio between the effective rank of text features and that of image features on a dataset should predict how much a visual-space classifier helps, which could be tested without any training.
  • Training the visual-space classifier with distillation toward the text classifier's zero-shot scores might further protect pre-trained knowledge, since currently the compensated classifier is trained purely on new-class prototypes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MG-CLIP, a replay-free method for class-incremental learning with CLIP. The central claim is that the modality gap, measured as the mean cosine similarity between image and text features, reflects how much pre-trained knowledge is preserved; keeping this gap relatively stable during fine-tuning mitigates catastrophic forgetting. To this end, the paper introduces an adaptive epoch budget: the number of fine-tuning epochs is chosen on the first task by monitoring relative change in the mean negative image-text similarity (Eq. 3) and then reused for all later tasks. In addition, the method trains an intra-modal classifier in visual feature space and fuses its output with the text classifier at inference (Eq. 5). Experiments on CIFAR-100, ImageNet-R, ImageNet-100, ImageNet-1K, and VTAB, together with zero-shot and ablation studies, are reported as state-of-the-art without replay.

Significance. If validated, the paper would make a useful contribution to CLIP-based continual learning: it proposes a simple, replay-free method that improves average and last accuracy while preserving zero-shot ability, and it provides a linear-algebra argument in the supplement explaining why a text classifier is restricted by the modality gap. The authors release code and include several ablations and backbone variants. The strength of the contribution, however, rests on the causal interpretation of the modality-gap signal. As presented, the method's only control variable is a global epoch budget estimated once on the first task, so the reported gains are equally consistent with the well-known effect that shorter fine-tuning causes less forgetting. The paper does not currently disentangle the proposed mechanism from generic early stopping, which makes the central claim load-bearing and in need of additional controls.

major comments (3)
  1. [Sec. 4.2, Algorithm 1, Eq. (3)] The proposed modality-gap preservation is not actively enforced during training. Algorithm 1 measures neg on the first task, selects the largest epoch e for which the relative change in Eq. (3) stays below alpha, and then fine-tunes every later task for exactly e epochs. No loss term, regularizer, or online re-check constrains the gap during tasks 2..T. Thus the actual treatment is 'fine-tune each task for e epochs', and any early-stopping rule that yields the same e would produce the same gap evolution. The paper does not compare against a matched-epoch control (e.g., e chosen by validation accuracy, a fixed epoch budget, or a frozen backbone), so Tables 1, 2, and 5 cannot distinguish the effect of 'modality gap preservation' from the effect of training duration. This is the central novelty claim, and it needs a direct control experiment to be supported.
  2. [Table 1, Table 3, Sec. 5.2] The state-of-the-art claim is not supported with statistical evidence. Table 1 reports that MG-CLIP is averaged over three class orders but gives no standard deviation, and the baseline numbers are listed without variance or repeated runs. Several margins are small, for example ImageNet-1K Last accuracy is 73.68 for MG-CLIP versus 73.19 for CLAP, and ImageNet-100 Last accuracy is 78.38 versus 79.16 for CLAP. The ablation in Table 3 likewise reports single runs. Without error bars or significance tests, the claimed improvements over replay-based methods are not established.
  3. [Sec. 5.1, Supplementary Sec. 4.3, Table 14] The adaptiveness of the method is weaker than claimed. The threshold alpha and fusion weight beta are tuned on a single dataset in the supplementary material and then fixed across all datasets, and the per-task epoch estimation experiment in Table 14 changes Last accuracy on ImageNet-R by only 0.15 percentage points over the fixed-budget version. This suggests that the first-task gap signal mainly selects one global integer e rather than providing per-task adaptation. The authors should report the epoch budgets selected for each dataset, study sensitivity of alpha and beta across datasets, and justify why the first-task signal transfers to later tasks and other datasets.
minor comments (5)
  1. [Sec. 4.1] The section title contains a typo: 'Efffect' should be 'Effect'.
  2. [Table 5] The 'Distill' baseline is not described in the main text; please specify which distillation method is used, since it plays a role in interpreting the cosine-similarity comparison.
  3. [Algorithm 1] The pseudocode uses nege+1 in the while condition before it has been assigned when e is updated; the indexing should be clarified so that the stopping epoch is unambiguous.
  4. [Supplementary Sec. 4.3] The hyperparameter-selection subsection does not name the dataset on which alpha and beta were chosen; please state this explicitly, as it affects the generality of the fixed settings.
  5. [Sec. 5.1] The sentence 'The performance of ours is the average over three different class orders' should be reworded for grammatical clarity, and the number of seeds or class orders for baselines should be stated.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; MG-CLIP's central claims are supported by external benchmarks and independent derivations, with only minor hyperparameter tuning on a single dataset.

full rationale

The claimed derivation chain is not circular. The modality-gap measures (Eqs. 1-2) and the preservation criterion (Eq. 3) are defined directly from CLIP features, not from the method's own outputs, and the epoch budget e is estimated once on the first task and then applied to later tasks; there is no equation in which a predicted quantity is set equal to a fitted value. The proof that an optimal classifier exists in the image feature span (supplementary Sec. 2.1) is a self-contained linear-algebra argument independent of the fitted hyperparameters. The visual-space compensation classifier is trained on frozen features and its combination weight β is a standard hyperparameter. The paper explicitly states in its Limitations that 'our current method does not incorporate specially designed loss functions or parameter constraint to mitigate forgetting,' confirming that the proposed preservation is an early-stopping rule based on the first-task gap signal rather than an enforced constraint; this is a potential confound with training duration, not circularity. Hyperparameters α and β are selected on one dataset (supplementary Sec. 4.3) and held fixed elsewhere, which is ordinary validation, not a fitted input renamed as a prediction. No load-bearing self-citation or imported uniqueness theorem appears; RAPF [12] is a prior-work baseline, not a premise of MG-CLIP's derivation. Overall, the method is adjudicated against external benchmarks and its components are independently meaningful, so the circularity score is low.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central method rests on two data-dependent hyperparameters (alpha, beta), a transfer assumption for the stopping criterion, and the standard linear algebra claim about classifier subspaces; no new physical entities are introduced.

free parameters (5)
  • Modality gap threshold alpha = 10%
    Selected on ImageNet-100 (Supp. Table 10) to set the stopping epoch; used for all datasets.
  • Inference fusion weight beta = 4
    Selected on ImageNet-100 (Supp. Table 9) to combine text and visual classifier scores.
  • LoRA rank = 8
    Chosen after a rank sweep on ImageNet-100 (Supp. Table 8); method is relatively insensitive.
  • Visual classifier training epochs = 3
    Fixed for all datasets; no sensitivity analysis reported.
  • Learning rates = 1e-3 backbone, 5e-4 classifier
    Standard Adam settings; no sweep reported.
assumptions (4)
  • domain assumption CLIP's pre-trained feature geometry, including the modality gap, encodes knowledge worth preserving.
    Used throughout Sec. 4.1; supported only by correlational plots (Fig. 2), not by causal proof.
  • domain assumption Cross-entropy fine-tuning on classification tasks systematically expands the modality gap due to asymmetric positive/negative pair structure.
    Sec. 4.1 text; illustrated in Fig. 3 but not proven.
  • ad hoc to paper Relative change in mean negative cosine (Eq. 3) is a valid early-warning signal for pre-trained knowledge loss and transfers across tasks.
    This is the core design premise of MG-CLIP's epoch selection (Algorithm 1); no independent evidence is given for transfer across tasks.
  • standard math A classifier that is optimal for classification can be restricted to the image feature span; text classifiers are limited to a low-rank subspace of that span.
    Proved by SVD and Frobenius projection in Supp. Sec. 2; this is a standard linear algebra result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mind the Gap: Preserving and Compensating for the Modality Gap in CLIP-Based Continual Learning." pith.science (2026). https://pith.science/paper/VSIXFBR6

@misc{pith2026250709118,
  author       = {Pith},
  title        = {Pith review of: Mind the Gap: Preserving and Compensating for the Modality Gap in CLIP-Based Continual Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VSIXFBR6}},
  note         = {Machine review of arXiv:2507.09118}
}
read the original abstract

Continual learning aims to enable models to learn sequentially from continuously incoming data while retaining performance on previously learned tasks. With the Contrastive Language-Image Pre-trained model (CLIP) exhibiting strong capabilities across various downstream tasks, there has been growing interest in leveraging CLIP for continual learning in such scenarios. Most existing works overlook the inherent modality gap in CLIP, a key factor in its generalization and adaptability. In this paper, we analyze the variations in the modality gap during the fine-tuning of vision-language pre-trained models. Our observations reveal that the modality gap effectively reflects the extent to which pre-trained knowledge is preserved. Based on these insights, we propose a simple yet effective method, MG-CLIP, that improves CLIP's performance in class-incremental learning. Our approach leverages modality gap preservation to mitigate forgetting and modality gap compensation to enhance the capacity for new data, introducing a novel modality-gap-based perspective for continual learning. Extensive experiments on multiple benchmarks demonstrate that our method outperforms existing approaches without requiring additional replay data. Our code is available at https://github.com/linlany/MindtheGap.

Figures

Figures reproduced from arXiv: 2507.09118 by the authors.

Figure 1
Figure 1. Left: In continual learning, we preserve the modality [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. The variation in the mean of positive and negative cosine [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Our method tackles CLIP-based continual learning through dual mechanisms: (1) [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Comparison of accuracy and learnable parameters of [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The features of the image and its corresponding caption [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting

    cs.CV 2025-08 unverdicted novelty 7.0 of 10

    The paper offers a comprehensive survey and proposes a new taxonomy for continual learning strategies in VLMs and MLLMs to combat catastrophic forgetting beyond traditional methods.

  2. Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Aligning images to multi-view caption cores while suppressing orthogonal residual text and disagreement-aware temperature improves robust zero-shot recognition and LVLM transfer.

  3. The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Model

    cs.LG 2026-07 conditional novelty 5.0 of 10

    CLIP embeddings are modeled as a mixture of von Mises-Fisher distributions on the unit sphere, improving out-of-distribution detection and semantic decomposition over single-Gaussian baselines.

Reference graph

Works this paper leans on

57 extracted references · 41 canonical work pages · cited by 3 Pith papers

  1. [1]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part VI 13, pages 446–461. Springer, 2014. 7

    2014

  2. [2]

    Dual- teacher class-incremental learning with data-free generative replay

    Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Dual- teacher class-incremental learning with data-free generative replay. In CVPR, pages 3543–3552, 2021. 2

    work page 2021

  3. [3]

    A continual learning survey: Defying for- getting in classification tasks

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ale ˇs Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying for- getting in classification tasks. IEEE TPAMI, 44(7):3366– 3385, 2021. 2

    work page 2021

  4. [4]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255. Ieee, 2009. 6

    2009

  5. [5]

    Dytox: Transformers for continual learn- ing with dynamic token expansion

    Arthur Douillard, Alexandre Ram ´e, Guillaume Couairon, and Matthieu Cord. Dytox: Transformers for continual learn- ing with dynamic token expansion. In CVPR, pages 9285– 9295, 2022. 2

    work page 2022

  6. [6]

    Mitigate the gap: In- vestigating approaches for improving cross-modal alignment in clip

    Sedigheh Eslami and Gerard de Melo. Mitigate the gap: In- vestigating approaches for improving cross-modal alignment in clip. arXiv preprint arXiv:2406.17639, 2024. 2

    arXiv 2024

  7. [7]

    It’s not a modality gap: Characterizing and addressing the contrastive gap

    Abrar Fahim, Alex Murphy, and Alona Fyshe. It’s not a modality gap: Characterizing and addressing the contrastive gap. arXiv preprint arXiv:2405.18570, 2024. 3, 4

    arXiv 2024

  8. [8]

    Benchmarking neu- ral network robustness to common corruptions and perturba- tions

    Dan Hendrycks and Thomas Dietterich. Benchmarking neu- ral network robustness to common corruptions and perturba- tions. Proceedings of the International Conference on Learn- ing Representations, 2019. 2

    work page 2019

Show all 57 references
  1. [9]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICC...

    2021

  2. [10]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. 6

    2022

  3. [11]

    Reclip: Refine contrastive language image pre-training with source free domain adaptation

    Xuefeng Hu, Ke Zhang, Lu Xia, Albert Chen, Jiajia Luo, Yuyin Sun, Ken Wang, Nan Qiao, Xiao Zeng, Min Sun, et al. Reclip: Refine contrastive language image pre-training with source free domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Compu...

    2024

  4. [12]

    Class-incremental learning with clip: Adaptive representa- tion adjustment and parameter fusion

    Linlan Huang, Xusheng Cao, Haori Lu, and Xialei Liu. Class-incremental learning with clip: Adaptive representa- tion adjustment and parameter fusion. In European Confer- ence on Computer Vision, pages 214–231. Springer, 2024. 1, 2, 6

    2024

  5. [13]

    Clap4clip: Contin- ual learning with probabilistic finetuning for vision-language models

    Saurav Jha, Dong Gong, and Lina Yao. Clap4clip: Contin- ual learning with probabilistic finetuning for vision-language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 1, 2, 6

    2024

  6. [14]

    Overcoming catastrophic forgetting in neu- ral networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neu- ral networks. Proceedings of the national academy of sc...

    2017

  7. [15]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 5

    2009

  8. [16]

    Do pre-trained models benefit equally in continual learning? In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pages 6485–6493, 2023

    Kuan-Ying Lee, Yuanyi Zhong, and Yu-Xiong Wang. Do pre-trained models benefit equally in continual learning? In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pages 6485–6493, 2023. 1

    2023

  9. [17]

    Unbiased region- language alignment for open-vocabulary dense prediction

    Yunheng Li, Yuxuan Li, Quansheng Zeng, Wenhai Wang, Qibin Hou, and Ming-Ming Cheng. Unbiased region- language alignment for open-vocabulary dense prediction. arXiv preprint arXiv:2412.06244, 2024. 1

    2024

  10. [18]

    Cascade-CLIP: Cascaded vision- language embeddings alignment for zero-shot semantic seg- mentation

    Yunheng Li, Zhong-Yu Li, Quan-Sheng Zeng, Qibin Hou, and Ming-Ming Cheng. Cascade-CLIP: Cascaded vision- language embeddings alignment for zero-shot semantic seg- mentation. In Proceedings of the 41st International Con- ference on Machine Learning, pages 28243–28258. PMLR,

  11. [19]

    Learning without forgetting

    Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE TPAMI, 40(12):2935–2947, 2017. 1, 2

    2017

  12. [20]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Ser- ena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Sys- tems, 35:17612–17625, 2022. 1, 3

    2022

  13. [21]

    Gradient episodic memory for continual learning

    David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neu- ral information processing systems, 30, 2017. 2

    2017

  14. [22]

    Magmax: Leveraging model merging for seamless continual learning

    Daniel Marczak, Bartłomiej Twardowski, Tomasz Trzci ´nski, and Sebastian Cygert. Magmax: Leveraging model merging for seamless continual learning. In European Conference on Computer Vision, pages 379–395. Springer, 2024. 1, 2, 6, 7

    2024

  15. [23]

    Class-incremental learning: survey and performance evalua- tion on image classification.IEEE TPAMI, 45(5):5513–5533,

    Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evalua- tion on image classification.IEEE TPAMI, 45(5):5513–5533,

  16. [24]

    Umap: Uniform manifold approximation and projection for dimen- sion reduction

    Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimen- sion reduction. arXiv preprint arXiv:1802.03426, 2018. 1

    2018 arXiv

  17. [25]

    An empirical investigation of the role of pre-training in lifelong learning

    Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning. arXiv preprint arXiv:2112.09153, 2021. 1

    2021 arXiv

  18. [26]

    Cross the gap: Exposing the intra-modal misalignment in clip via modality inversion

    Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Andrew D Bagdanov. Cross the gap: Exposing the intra-modal misalignment in clip via modality inversion. arXiv preprint arXiv:2502.04263, 2025. 2, 3

    2025 arXiv

  19. [27]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012. 7

    2012

  20. [28]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  21. [29]

    icarl: Incremental classifier and representation learning

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE con- ference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017. 2

    2001

  22. [30]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 1

    2021 arXiv

  23. [31]

    Coda-prompt: Contin- ual decomposed attention-based prompting for rehearsal-free continual learning

    James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Contin- ual decomposed attention-based prompting for rehearsal-free continual learning. In CVPR, pages 11909–11...

    2023

  24. [32]

    Regularizing second-order influences for continual learning

    Zhicheng Sun, Yadong Mu, and Gang Hua. Regularizing second-order influences for continual learning. In CVPR, pages 20166–20175, 2023. 2

    2023

  25. [33]

    Clip model is an efficient continual learner

    Vishal Thengane, Salman Khan, Munawar Hayat, and Fahad Khan. Clip model is an efficient continual learner. arXiv preprint arXiv:2210.03114, 2022. 1, 2, 6

    2022 arXiv

  26. [34]

    Three types of incremental learning

    Gido M Van de Ven, Tinne Tuytelaars, and Andreas S To- lias. Three types of incremental learning. Nature Machine Intelligence, 4(12):1185–1197, 2022. 1

    2022

  27. [35]

    Foster: Feature boosting and compression for class- incremental learning

    Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class- incremental learning. In ECCV, pages 398–414. Springer,

  28. [36]

    Memory replay with data com- pression for continual learning

    Liyuan Wang, Xingxing Zhang, Kuo Yang, Longhui Yu, Chongxuan Li, HONG Lanqing, Shifeng Zhang, Zhenguo Li, Yi Zhong, and Jun Zhu. Memory replay with data com- pression for continual learning. In International Conference on Learning Representations, 2021. 2

    2021

  29. [37]

    Dualprompt: Complementary prompting for rehearsal-free continual learning

    Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, pages 631–648. Springer, 2022. 2, 6

    2022

  30. [38]

    Learning to prompt for continual learning

    Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jen- nifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, pages 139–149, 2022. 2, 6

    2022

  31. [39]

    Class-incremental learning with strong pre-trained models

    Tz-Ying Wu, Gurumurthy Swaminathan, Zhizhong Li, Avinash Ravichandran, Nuno Vasconcelos, Rahul Bhotika, and Stefano Soatto. Class-incremental learning with strong pre-trained models. In CVPR, pages 9601–9610, 2022. 1

    2022

  32. [40]

    Class-incremental learning with strong pre-trained models

    Tz-Ying Wu, Gurumurthy Swaminathan, Zhizhong Li, Avinash Ravichandran, Nuno Vasconcelos, Rahul Bhotika, and Stefano Soatto. Class-incremental learning with strong pre-trained models. In CVPR, pages 9601–9610, 2022. 2

    2022

  33. [41]

    Der: Dynam- ically expandable representation for class incremental learn- ing

    Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynam- ically expandable representation for class incremental learn- ing. In CVPR, pages 3014–3023, 2021. 2

    2021

  34. [42]

    Lever- aging cross-modal neighbor representation for improved clip classification

    Chao Yi, Lu Ren, De-Chuan Zhan, and Han-Jia Ye. Lever- aging cross-modal neighbor representation for improved clip classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27402– 27411, 2024. 3

    2024

  35. [43]

    Boosting continual learning of vision-language models via mixture-of-experts adapters

    Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 23219–23230, 2024...

    2024

  36. [44]

    Masked autoencoders are efficient class incremental learners

    Jiang-Tian Zhai, Xialei Liu, Andrew D Bagdanov, Ke Li, and Ming-Ming Cheng. Masked autoencoders are efficient class incremental learners. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19104– 19113, 2023. 2

    2023

  37. [45]

    A large-scale study of representation learning with the visual task adaptation benchmark

    Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djo- longa, Andre Susano Pinto, Maxim Neumann, Alexey Doso- vitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv ...

    1910 arXiv

  38. [46]

    Slca: Slow learner with classifier align- ment for continual learning on a pre-trained model

    Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slca: Slow learner with classifier align- ment for continual learning on a pre-trained model. arXiv preprint arXiv:2303.05118, 2023. 2, 6

    2023 arXiv

  39. [47]

    Class-incremental learning via deep model consolidation

    Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafet- tin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremental learning via deep model consolidation. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pages 1131–1140, 2020. 2

    2020

  40. [48]

    Continual learning of image classes with language guidance from a vision-language model

    Wentao Zhang, Yujun Huang, Weizhuo Zhang, Tong Zhang, Qicheng Lao, Yue Yu, Wei-Shi Zheng, and Ruixuan Wang. Continual learning of image classes with language guidance from a vision-language model. IEEE Transactions on Cir- cuits and Systems for Video Technology, 2024. 2, 3

    2024

  41. [49]

    Preventing zero-shot transfer degradation in continual learning of vision-language models

    Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xi- angyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In ICCV, pages 19125–19136, 2023. 2

    2023

  42. [50]

    Co- transport for class-incremental learning

    Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Co- transport for class-incremental learning. In ACM MM, pages 1645–1654, 2021. 2

    2021

  43. [51]

    Revisiting class-incremental learning with pre- trained models: Generalizability and adaptivity are all you need

    Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental learning with pre- trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision , pages 1– 21, 2024. 2, 6

    2024

  44. [52]

    Learning with- out forgetting for vision-language models

    Da-Wei Zhou, Yuanhan Zhang, Yan Wang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learning with- out forgetting for vision-language models. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2025. 1, 2, 6 Mind the Gap: Preserving and Compensating for ...

    2025

  45. [53]

    6, we randomly sampled 512 image- caption pairs from LAION-400M [30], extracted features using CLIP, and applied UMAP [24] for dimensionality re- duction

    Visualization of Modality Gap As shown in the Fig. 6, we randomly sampled 512 image- caption pairs from LAION-400M [30], extracted features using CLIP, and applied UMAP [24] for dimensionality re- duction. The results reveal a clear clustering of features within the same modal...

  46. [54]

    Existence of an Optimal Classifier within Image Feature Space Let the CLIP image feature matrix be X ∈ Rd×n

    Proof of Image-Space Classifier and Modal- ity Gap Constraints 2.1. Existence of an Optimal Classifier within Image Feature Space Let the CLIP image feature matrix be X ∈ Rd×n. Con- sider a classifier W∗ ∈ Rd×C that achieves minimal cross- entropy loss under ideal conditions. ...

  47. [55]

    Given the large number of CIFAR100 ImageNet100 Avg Last Avg Last ours w/o replay 86.79 80.40 87.31 78.38 ours w/ replay 88.48 82.58 88.50 80.74 Table 6

    Implementation Details of Image Space and Classifier Space Analyzing To analyze these relationships, we first apply SVD to the image feature matrix and extract its corresponding basis vectors, denoted as Bi ∈ Rd×r. Given the large number of CIFAR100 ImageNet100 Avg Last Avg La...

  48. [56]

    Compatibility with replay methods

    More experiments 4.1. Compatibility with replay methods. Our method does not require rehearsal samples; however, it is still compatible with them. We tested our method with simple random sampling of rehearsal data, keeping the total number at 2000. The experimental results, sh...

  49. [57]

    Algorithm Pseudocode The overall pipeline of training is shown in the pseudocode: Algorithm 1 Training algorithm 1: Input: D = {X1, X2, . . . ,Xt} ▷ Training data in all tasks 2: Input: f 0 clip(·) ▷ Original CLIP 3: require: f T clip(·) ▷ CLIP after fine-tuning on T tasks 4: ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.