Pith. sign in

REVIEW 6 cited by

Spurious Forgetting in Continual Learning of Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2501.13453 v1 pith:JG65UMVE submitted 2025-01-23 cs.LG

Junhao Zheng , Xidi Cai , Shengjie Qiu , Qianli Ma This is my paper · ORCID
classification cs.LG
keywords continuallearningtaskalignmentknowledgemodelmodelsperformance
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent advancements in large language models (LLMs) reveal a perplexing phenomenon in continual learning: despite extensive training, models experience significant performance declines, raising questions about task alignment and underlying knowledge retention. This study first explores the concept of "spurious forgetting", proposing that such performance drops often reflect a decline in task alignment rather than true knowledge loss. Through controlled experiments with a synthesized dataset, we investigate the dynamics of model performance during the initial training phases of new tasks, discovering that early optimization steps can disrupt previously established task alignments. Our theoretical analysis connects these shifts to orthogonal updates in model weights, providing a robust framework for understanding this behavior. Ultimately, we introduce a Freezing strategy that fix the bottom layers of the model, leading to substantial improvements in four continual learning scenarios. Our findings underscore the critical distinction between task alignment and knowledge retention, paving the way for more effective strategies in continual learning.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking Transfer in Continual Learning: A Replay-Based Realisation

    cs.LG 2026-07 conditional novelty 7.0 of 10

    In continual learning, forward transfer requires target headroom, a persistent carrier, and a compatible source; routing replay by gradient signatures improves accuracy and stability over uniform replay.

  2. UItron: Foundational GUI Agent with Advanced Perception and Planning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    UItron, trained with multi-stage SFT and curriculum RL on a new 1M-step Chinese app dataset, beats prior GUI agents on perception, grounding, planning, and especially Chinese mobile app scenarios.

  3. Continual Learning in Transition

    cs.LG 2026-08 accept novelty 5.0 of 10

    A tri-axial framework of When, Where, and How organizes the ongoing transition of continual learning from parameter-centric updates to system-level capability evolution.

  4. Attribution-Guided Continual Learning for Large Language Models

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    LRP-derived element-wise parameter importance scores gate gradients so parameters critical to earlier tasks receive smaller updates during continual LLM fine-tuning.

  5. Mixture-of-Translators: Translating KV Caches Across Heterogeneous Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    MoT translates KV caches between LLMs using token-level translator mixtures and a context-correction loss; QA results are strong for same-family pairs but do not demonstrate true cross-architecture reuse.

  6. Continual Learning for Generative AI: From LLMs to MLLMs and Beyond

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.

Pith tools