Pith. sign in

REVIEW 10 cited by

Investigating the Catastrophic Forgetting in Multimodal Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.10313 v4 pith:BM2JDM3F submitted 2023-09-19 cs.CL cs.AIcs.LG

Yuexiang Zhai , Shengbang Tong , Xiao Li , Mu Cai , Qing Qu , Yong Jae Lee , Yi Ma This is my paper
classification cs.CLcs.AIcs.LG
keywords fine-tuningimagemllmmllmsperformancecatastrophicforgettingllms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Following the success of GPT4, there has been a surge in interest in multimodal large language model (MLLM) research. This line of research focuses on developing general-purpose LLMs through fine-tuning pre-trained LLMs and vision models. However, catastrophic forgetting, a notorious phenomenon where the fine-tuned model fails to retain similar performance compared to the pre-trained model, still remains an inherent problem in multimodal LLMs (MLLM). In this paper, we introduce EMT: Evaluating MulTimodality for evaluating the catastrophic forgetting in MLLMs, by treating each MLLM as an image classifier. We first apply EMT to evaluate several open-source fine-tuned MLLMs and we discover that almost all evaluated MLLMs fail to retain the same performance levels as their vision encoders on standard image classification tasks. Moreover, we continue fine-tuning LLaVA, an MLLM and utilize EMT to assess performance throughout the fine-tuning. Interestingly, our results suggest that early-stage fine-tuning on an image dataset improves performance across other image datasets, by enhancing the alignment of text and visual features. However, as fine-tuning proceeds, the MLLMs begin to hallucinate, resulting in a significant loss of generalizability, even when the image encoder remains frozen. Our results suggest that MLLMs have yet to demonstrate performance on par with their vision models on standard image classification tasks and the current MLLM fine-tuning procedure still has room for improvement.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond the Need for Speed: Energy-Aware Code Generation via Simulation-Guided Reinforcement Learning

    cs.LG 2026-07 conditional novelty 6.5 of 10

    Simulation-in-the-loop GRPO on energy-labeled C++ pairs yields 12.63% CARET and beats human energy efficiency on 58.4% of valid held-out outputs, while IPC misranks energy on 67.8% of problems.

  2. Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning

    cs.LG 2025-06 reject novelty 6.0 of 10

    From Open LLM Leaderboard data grouped by base model, the authors recover a three-factor ordering of LLM capabilities and claim instruction-following causally supports math reasoning.

  3. Aligning LLM with human travel choices: a persona-based embedding learning approach

    cs.AI 2025-05 conditional novelty 6.0 of 10

    A persona-based embedding learning framework aligns LLM predictions with human travel mode choices, outperforming MNL and few-shot LLM baselines on the Swissmetro dataset.

  4. VistaVLA: Geometry- and Semantic-Aware 3D Gaussian-Grounded VLA for Robotic Manipulation

    cs.RO 2026-07 conditional novelty 5.5 of 10

    VistaVLA lifts multi-view vision-language features into 3D Gaussians, compresses them 99% via Merge-then-Query, and improves real-robot manipulation success by ~23% over baselines.

  5. Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Models

    cs.CV 2025-07 conditional novelty 5.5 of 10

    A monolithic multimodal LLM that cuts pre-training data by 58% and first-token latency by up to 69% while matching or beating its predecessor on 15 benchmarks.

  6. MoE-TTS: Enhancing Out-of-Domain Text Understanding for Description-based TTS via Mixture-of-Experts

    eess.AS 2025-08 conditional novelty 5.0 of 10

    MoE-TTS adds frozen text-expert MoE modules to a Qwen3-based TTS system and reports better out-of-domain description alignment than ElevenLabs and MiniMax on a small hand-built test set.

  7. Enhancing Multimodal Continual Instruction Tuning with BranchLoRA

    cs.CL 2025-05 conditional novelty 5.0 of 10

    BranchLoRA reduces catastrophic forgetting in multimodal continual instruction tuning by using a shared LoRA matrix, task-specific branches, frozen experts, and learned task keys.

  8. ReqBrain: Task-Specific Instruction Tuning of LLMs for AI-Assisted Requirements Generation

    cs.SE 2025-05 conditional novelty 5.0 of 10

    ReqBrain, a LoRA-fine-tuned Zephyr-7b-beta model, produces software requirements that human evaluators could not reliably tell apart from human-authored ones, with automatic metrics favoring it over untuned ChatGPT-4o.

  9. Architecting Clinical Collaboration: Multi-Agent Reasoning Systems for Multimodal Medical VQA

    cs.AI 2025-07 reject novelty 4.0 of 10

    A systematic study on dermatology VQA finds that multi-agent reasoning and retrieval architectures outperform fine-tuned open-source vision-language models, maintaining 70% accuracy under distribution shift.

  10. A Survey on Video Temporal Grounding with Multimodal Large Language Model

    cs.CV 2025-08 unverdicted novelty 3.0 of 10

    A taxonomized review of video temporal grounding with multimodal large language models, covering model roles, training paradigms, feature processing, benchmarks, and open problems.

Pith tools