REVIEW 14 cited by
Gradient Projection Memory for Continual Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The ability to learn continually without forgetting the past tasks is a desired attribute for artificial learning systems. Existing approaches to enable such learning in artificial neural networks usually rely on network growth, importance based weight update or replay of old data from the memory. In contrast, we propose a novel approach where a neural network learns new tasks by taking gradient steps in the orthogonal direction to the gradient subspaces deemed important for the past tasks. We find the bases of these subspaces by analyzing network representations (activations) after learning each task with Singular Value Decomposition (SVD) in a single shot manner and store them in the memory as Gradient Projection Memory (GPM). With qualitative and quantitative analyses, we show that such orthogonal gradient descent induces minimum to no interference with the past tasks, thereby mitigates forgetting. We evaluate our algorithm on diverse image classification datasets with short and long sequences of tasks and report better or on-par performance compared to the state-of-the-art approaches.
Forward citations
Cited by 14 Pith papers
-
TriShield: Zero-Utility-Loss Defense Against Privacy Backdoors in Federated Language Model Fine-Tuning via Orthogonal Gradient Projection and Optimizer State Entanglement
TriShield combines artifact detection, Adam momentum pre-entanglement, and SVD task-subspace projection to drive NeuroImprint reconstruction to 0% with claimed near-zero utility loss.
-
SUM: Unified Geometric Surgery on Spatio-Temporal Adaptation Vectors for Federated Class Incremental Learning
SUM projects client and task adaptation vectors to remove directional interference during server aggregation, improving federated class-incremental learning accuracy without client-side changes.
-
PLATE: Plasticity-Tunable Efficient Adapters for Geometry-Aware Continual Learning
PLATE constructs frozen weight-derived bases B and Q and trains only a small core A, reducing catastrophic forgetting during data-free continual fine-tuning.
-
RainbowPrompt: Diversity-Enhanced Prompt-Evolving for Continual Learning
RainbowPrompt evolves past and new prompts via attention and alignment before averaging them, beating several prompt-CL baselines on image and video class-incremental benchmarks.
-
Zero-Shot Machine Unlearning with Proxy Adversarial Data Generation
ZS-PAG generates adversarial proxy samples from the forget set, projects unlearning updates into the orthogonal complement of remaining-class subspaces, and optimizes pseudo-labels with influence functions to enable z...
-
SHIELD: Secure Hypernetworks for Incremental Expansion Learning Defense
SHIELD uses a hypernetwork with IBP training and a new Interval MixUp technique to achieve certified robustness in continual learning, reporting state-of-the-art adversarial accuracy on MNIST, CIFAR-100, and miniImage...
-
Learning without Isolation: Pathway Protection for Continual Learning
LwI fuses old and new models with graph matching, matching similar channels in shallow layers and dissimilar channels in deep layers, to reduce catastrophic forgetting without storing old data.
-
Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
Frozen-embedding GMMs route compact SVD-subspace LoRA adapters for task-agnostic continual learning with SOTA average performance and near-zero forgetting.
-
Orthogonal Knowledge Refreshing for Domain-Incremental Object Detection
OKR beats exemplar-free domain-incremental detectors by 5.6–6.5 mAP using per-domain LoRA branches plus gradient orthogonality and prototype alignment.
-
SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment
Adding a KL penalty between fine-tuned and original model logits on input tokens during RAG fine-tuning reduces catastrophic forgetting while preserving downstream performance.
-
Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulation
Fine-tuning video generators on driving data can improve visual fidelity while degrading how accurately the model predicts the movement of cars and pedestrians.
-
Frugal Incremental Generative Modeling using Variational Autoencoders
A single replay-free conditional VAE with fixed-point-separated Gaussian priors and null-space gradient projection achieves competitive continual classification with drastically reduced memory.
-
Continual Learning Beyond Experience Rehearsal and Full Model Surrogates
SPARC achieves strong continual learning accuracy with a fraction of the parameters of surrogate-based methods by combining task-specific depthwise filters with shared pointwise filters updated by exponential averaging.
-
Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data
FedProj combines client-side gradient projection onto a global-knowledge loss with server-side ensemble distillation and outperforms existing federated learning methods on non-IID image and NLP benchmarks.
Discussion (0). Sign in to comment.