Pith. sign in

REVIEW 16 cited by

Continual Learning of Large Language Models: A Comprehensive Survey

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.16789 v3 pith:VVRMG6YT submitted 2024-04-25 cs.LG cs.AIcs.CL

Haizhou Shi , Zihao Xu , Hengyi Wang , Weiyi Qin , Wenyuan Wang , Yibin Wang , Zifeng Wang , Sayna Ebrahimi
show 1 more author
Hao Wang
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords continualllmslearningsectionsurveycontinuityoverviewadaptation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The recent success of large language models (LLMs) trained on static, pre-collected, general datasets has sparked numerous research directions and applications. One such direction addresses the non-trivial challenge of integrating pre-trained LLMs into dynamic data distributions, task structures, and user preferences. Pre-trained LLMs, when tailored for specific needs, often experience significant performance degradation in previous knowledge domains -- a phenomenon known as "catastrophic forgetting". While extensively studied in the continual learning (CL) community, it presents new manifestations in the realm of LLMs. In this survey, we provide a comprehensive overview of the current research progress on LLMs within the context of CL. This survey is structured into four main sections: we first describe an overview of continually learning LLMs, consisting of two directions of continuity: vertical continuity (or vertical continual learning), i.e., continual adaptation from general to specific capabilities, and horizontal continuity (or horizontal continual learning), i.e., continual adaptation across time and domains (Section 3). We then summarize three stages of learning LLMs in the context of modern CL: Continual Pre-Training (CPT), Domain-Adaptive Pre-training (DAP), and Continual Fine-Tuning (CFT) (Section 4). Then we provide an overview of evaluation protocols for continual learning with LLMs, along with the current available data sources (Section 5). Finally, we discuss intriguing questions pertaining to continual learning for LLMs (Section 6). The full list of papers examined in this survey is available at https://github.com/Wang-ML-Lab/llm-continual-learning-survey.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A fine-tuning paradigm that prompts MLLMs to self-generate OCR text before translating document images improves DIMT quality and reduces catastrophic forgetting of OCR.

  2. Bridging the Gap: From Ad-hoc to Proactive Search in Conversations

    cs.IR 2025-06 conditional novelty 6.0 of 10

    Conv2Query fine-tunes an LLM to convert conversational context into ad-hoc queries, enabling off-the-shelf retrievers to work effectively on proactive search in conversations.

  3. Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation

    cs.LG 2025-05 conditional novelty 6.0 of 10

    MFT, a corrective self-distillation objective, reports 2 to 10 times better specialization-to-degeneralization ratios than standard finetuning across many models and three specialized domains.

  4. Towards Objective Fine-tuning: How LLMs' Prior Knowledge Causes Potential Poor Calibration?

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Fine-tuning on data aligned with an LLM's prior knowledge induces overconfidence, and CogCalib mitigates this by gating a calibration loss to known data.

  5. T2I-ConBench: Text-to-Image Benchmark for Continual Post-training

    cs.CV 2025-05 conditional novelty 6.0 of 10

    T2I-ConBench provides a unified multi-metric benchmark for continual post-training of text-to-image models and shows that all tested methods have notable weaknesses.

  6. Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Franken-Adapter swaps in newly trained multilingual word embeddings into an instruction-tuned LLM to enable zero-shot cross-lingual transfer, improving performance by up to 20% across 96 languages.

  7. Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Frozen-embedding GMMs route compact SVD-subspace LoRA adapters for task-agnostic continual learning with SOTA average performance and near-zero forgetting.

  8. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  9. Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting

    cs.LG 2025-10 conditional novelty 5.0 of 10

    RL post-training forgets less than SFT because it trains on on-policy data; refreshing SFT data each epoch also reduces forgetting.

  10. SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Adding a KL penalty between fine-tuned and original model logits on input tokens during RAG fine-tuning reduces catastrophic forgetting while preserving downstream performance.

  11. GeRe: Towards Efficient Anti-Forgetting in Continual Learning of LLM via General Samples Replay

    cs.CL 2025-08 conditional novelty 5.0 of 10

    A fixed 1,000-sample replay set from pretraining text, trained with a threshold-based margin loss on last-layer hidden states, reduces forgetting across 15 sequential finetuning tasks in Llama-3.1-8B.

  12. Continual Gradient Low-Rank Projection Fine-Tuning for LLMs

    cs.LG 2025-07 conditional novelty 5.0 of 10

    GORP jointly trains LoRA and full-rank parameters inside a low-rank gradient subspace built from Adam first moments, reporting higher average accuracy and lower forgetting than O-LoRA and N-LoRA on LLM continual learn...

  13. From Knowledge to Noise: CTIM-Rover and the Pitfalls of Episodic Memory in Software Engineering Agents

    cs.SE 2025-05 conditional novelty 5.0 of 10

    CTIM-Rover, an episodic-memory extension of AutoCodeRover, performs worse than the baseline on a 45-sample SWE-bench Verified subset, supporting the view that noisy memory items can degrade software engineering agents.

  14. Enhancing Memory Recall in LLMs with Gauss-Tin: A Hybrid Instructional and Gaussian Replay Approach

    cs.LG 2025-08 reject novelty 4.0 of 10

    Gauss-Tin, a replay method using a Gaussian mixture model with prompt-guided exemplar selection, reports positive backward transfer on the Natural Instructions benchmark versus sequential fine-tuning.

  15. Continual Learning for Generative AI: From LLMs to MLLMs and Beyond

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.

  16. The Future of Continual Learning in the Era of Foundation Models: Three Key Directions

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Continual learning should pivot from weight-update-based methods to continual compositionality and orchestration of foundation models and agents.

Pith tools