REVIEW 9 cited by
Orthogonal Subspace Learning for Language Model Continual Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Benefiting from massive corpora and advanced hardware, large language models (LLMs) exhibit remarkable capabilities in language understanding and generation. However, their performance degrades in scenarios where multiple tasks are encountered sequentially, also known as catastrophic forgetting. In this paper, we propose orthogonal low-rank adaptation (O-LoRA), a simple and efficient approach for continual learning in language models, effectively mitigating catastrophic forgetting while learning new tasks. Specifically, O-LoRA learns tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Our method induces only marginal additional parameter costs and requires no user data storage for replay. Experimental results on continual learning benchmarks show that our method outperforms state-of-the-art methods. Furthermore, compared to previous approaches, our method excels in preserving the generalization ability of LLMs on unseen tasks.
Forward citations
Cited by 9 Pith papers
-
Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
The paper offers a comprehensive survey and proposes a new taxonomy for continual learning strategies in VLMs and MLLMs to combat catastrophic forgetting beyond traditional methods.
-
One Rank at a Time: Cascading Error Dynamics in Sequential Learning
Errors from each rank-1 step in sequential low-rank learning compound through factors that grow when singular values are close, so early steps deserve more compute.
-
Orthogonal Knowledge Refreshing for Domain-Incremental Object Detection
OKR beats exemplar-free domain-incremental detectors by 5.6–6.5 mAP using per-domain LoRA branches plus gradient orthogonality and prototype alignment.
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment
Adding a KL penalty between fine-tuned and original model logits on input tokens during RAG fine-tuning reduces catastrophic forgetting while preserving downstream performance.
-
Reliable Unlearning Harmful Information in LLMs with Metamorphosis Representation Projection
An LLM unlearning method that projects hidden states so harmful information is irreversibly removed while useful knowledge is preserved.
-
GeRe: Towards Efficient Anti-Forgetting in Continual Learning of LLM via General Samples Replay
A fixed 1,000-sample replay set from pretraining text, trained with a threshold-based margin loss on last-layer hidden states, reduces forgetting across 15 sequential finetuning tasks in Llama-3.1-8B.
-
SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting
SplitLoRA picks the LoRA update subspace size from previous-task gradient singular values using a hyperparameter alpha, and freezes the projection to keep updates in that subspace.
-
RegCL: Continual Adaptation of Segment Anything Model via Model Merging
RegCL merges LoRA adapters for SAM continually via an incremental form of RegMean, achieving higher retention than sequential fine-tuning across five segmentation datasets.
Discussion (0). Continue with ORCID to comment.