REVIEW 7 cited by
Don't Stop Learning: Towards Continual Learning for the CLIP Model
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The Contrastive Language-Image Pre-training (CLIP) Model is a recently proposed large-scale pre-train model which attracts increasing attention in the computer vision community. Benefiting from its gigantic image-text training set, the CLIP model has learned outstanding capabilities in zero-shot learning and image-text matching. To boost the recognition performance of CLIP on some target visual concepts, it is often desirable to further update the CLIP model by fine-tuning some classes-of-interest on extra training data. This operation, however, raises an important concern: will the update hurt the zero-shot learning or image-text matching capability of the CLIP, i.e., the catastrophic forgetting issue? If yes, could existing continual learning algorithms be adapted to alleviate the risk of catastrophic forgetting? To answer these questions, this work conducts a systemic study on the continual learning issue of the CLIP model. We construct evaluation protocols to measure the impact of fine-tuning updates and explore different ways to upgrade existing continual learning methods to mitigate the forgetting issue of the CLIP model. Our study reveals the particular challenges of CLIP continual learning problem and lays a foundation for further researches. Moreover, we propose a new algorithm, dubbed Learning without Forgetting via Replayed Vocabulary (VR-LwF), which shows exact effectiveness for alleviating the forgetting issue of the CLIP model.
Forward citations
Cited by 7 Pith papers
-
Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
The paper offers a comprehensive survey and proposes a new taxonomy for continual learning strategies in VLMs and MLLMs to combat catastrophic forgetting beyond traditional methods.
-
Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuning
Expert routing in multimodal continual instruction tuning is reformulated as soft task-as-class class-incremental learning, and a new 34-task fingerprint-reduced benchmark shows that plugging CIL classifiers into rout...
-
iDPA: Instance Decoupled Prompt Attention for Incremental Medical Object Detection
iDPA improves incremental medical object detection by generating instance-level prompts from bounding-box regions and decoupling prompt attention in a frozen GLIP model.
-
Satellites Reveal Mobility: A Commuting Origin-destination Flow Generator for Global Cities
Satellite imagery plus population is enough to generate commuting origin-destination flows that closely match models using detailed sociodemographic and point-of-interest data.
-
Leveraging Pre-Trained Models for Multimodal Class-Incremental Learning under Adaptive Fusion
A multimodal class-incremental learning method based on AudioCLIP with MoE adapters, adaptive audio-visual fusion, and a contrastive loss reports gains on three vision-audio-text datasets.
-
Textual Inversion for Efficient Adaptation of Open-Vocabulary Object Detectors Without Forgetting
Textual Inversion is applied to open-vocabulary object detectors to learn a few new tokens while keeping the VLM frozen and preserving zero-shot abilities.
-
ChordPrompt: Orchestrating Cross-Modal Prompt Synergy for Multi-Domain Incremental Learning in CLIP
Cross-modal prompt sharing with domain-adaptive retrieval improves CLIP's continual learning performance on multi-domain image classification benchmarks.
Discussion (0). Continue with ORCID to comment.