REVIEW 6 cited by
Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfer
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Multimodal Continual Instruction Tuning (MCIT) enables Multimodal Large Language Models (MLLMs) to meet continuously emerging requirements without expensive retraining. MCIT faces two major obstacles: catastrophic forgetting (where old knowledge is forgotten) and negative forward transfer (where the performance of future tasks is degraded). Although existing methods have greatly alleviated catastrophic forgetting, they still suffer from negative forward transfer. We discover a large discrepancy in different input embeddings by performing singular value decomposition (SVD) on input embeddings. This discrepancy results in the model learning irrelevant information for old and pre-trained tasks, leading to catastrophic forgetting and negative forward transfer. To address these issues, we propose Prompt Tuning with Positive Forward Transfer (Fwd-Prompt), a prompt-based method that projects the prompt gradient to the residual space to minimize interference between tasks and to the pre-trained subspace for reusing pre-trained knowledge. Our experiments demonstrate that Fwd-Prompt achieves state-of-the-art performance while updating fewer parameters and requiring no old samples. Our research illuminates the potential of continuously adapting MLLMs to new tasks under the instruction tuning paradigm and encourages future studies to explore MCIT.
Forward citations
Cited by 6 Pith papers
-
Domain-Aware Scaling Laws Uncover Data Synergy
Domain-aware scaling laws with fitted γ and σ synergy terms recover stable code-math interactions from observational LLM mixtures and correctly predict mixture rankings in controlled small-scale trainings.
-
Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs
HPA is a post-training parameter-selection method that keeps safety-aligned multimodal LLMs safe and reduces forgetting during continual visual instruction tuning.
-
Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
D-MoLE dynamically allocates LoRA experts to the most gradient-sensitive layers of a vision-language model for each new task, and routes inputs via per-task autoencoders, reporting near-zero forgetting on a nine-task ...
-
Continual Gradient Low-Rank Projection Fine-Tuning for LLMs
GORP jointly trains LoRA and full-rank parameters inside a low-rank gradient subspace built from Adam first moments, reporting higher average accuracy and lower forgetting than O-LoRA and N-LoRA on LLM continual learn...
-
Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
BranchLoRA reduces catastrophic forgetting in multimodal continual instruction tuning by using a shared LoRA matrix, task-specific branches, frozen experts, and learned task keys.
-
Continual Learning for Generative AI: From LLMs to MLLMs and Beyond
A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.
Discussion (0). Continue with ORCID to comment.