Pith. sign in

REVIEW 4 cited by

SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.08295 v3 pith:TSVKSYRE submitted 2024-01-16 cs.CL

Weixiang Zhao , Shilong Wang , Yulin Hu , Yanyan Zhao , Bing Qin , Xuanyu Zhang , Qing Yang , Dongliang Xu
show 1 more author
Wanxiang Che
This is my paper · ORCID
classification cs.CL
keywords learningsaptknowledgemoduleselectionsharedaddressattention
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The continual learning (CL) ability is vital for deploying large language models (LLMs) in the dynamic world. Existing methods devise the learning module to acquire task-specific knowledge with parameter-efficient tuning (PET) block and the selection module to pick out the corresponding one for the testing input, aiming at handling the challenges of catastrophic forgetting and knowledge transfer in CL. However, these methods tend to address only one of the challenges, ignoring the potential of aligning the two modules to effectively address catastrophic forgetting and knowledge transfer simultaneously. To this end, we propose a novel Shared Attention Framework (SAPT), to align the PET learning and selection via the Shared Attentive Learning \& Selection module. Extensive Experiments on two CL benchmarks demonstrate the superiority of SAPT. Moreover, SAPT consistently demonstrates its superiority when we scale it to different model sizes (from 770M to 13B), different model architectures (T5 and LLaMA-2) and unseen tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Filling Memory Gaps: Enhancing Continual Semantic Parsing via SQL Syntax Variance-Guided LLMs without Real Data Replay

    cs.CL 2024-12 conditional novelty 6.0 of 10

    LECSP uses LLM-generated pseudo-samples guided by SQL syntax differences between tasks, plus dual-teacher distillation, to train a continual semantic parser without replaying real historical data.

  2. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  3. Continual Learning for Generative AI: From LLMs to MLLMs and Beyond

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.

  4. Unleashing the Power of Continual Learning on Non-Centralized Devices: A Survey

    cs.LG 2024-12 conditional novelty 4.0 of 10

    A review of non-centralized continual learning that taxonomizes data-, model-, and device-level methods and benchmarks twelve federated continual learning methods on six datasets.

Pith tools