Pith. sign in

REVIEW 2 major objections 2 minor 1 cited by

CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning

T0 review · 2 major / 2 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A dual-adapter LoRA design lets a frozen ViT keep old classes while learning new ones, using only 0.3% trainable parameters and no stored exemplars.

desk verdict Clever dual-LoRA design, but the inference protocol mismatches training for old tasks and the reported accuracy depends on that mismatch. read the letter →

arxiv 2505.24816 v1 pith:NPOOBUC2 submitted 2025-05-30 cs.CV

Jiangpeng He , Zhihao Duan , Fengqing Zhu This is my paper
classification cs.CV
keywords class-incrementallearningrehearsal-freecontinuallow-rankadaptationLoRAtask-sharedadapterstask-specificknowledgedistillationorthogonalprojection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a pre-trained vision transformer can keep learning new classes, one task at a time, without storing any examples from earlier tasks. Its answer is CL-LoRA, a dual-adapter scheme that splits the transformer into early blocks that share one continually updated low-rank adapter and later blocks where each task gets its own adapter. The shared adapter fixes a random orthogonal down-projection matrix and only trains the up-projection, so knowledge from all tasks accumulates in one stable low-dimensional subspace; the task-specific adapters add learnable per-block scaling with an orthogonality penalty to keep tasks from interfering. On four standard benchmarks the result is state-of-the-art or competitive accuracy with only 0.3% trainable parameters, and inference cost drops from $O(NT)$ to $O(l + (N-l)T)$ because the shared blocks are run once. If the design holds, it shows that cross-task knowledge sharing and parameter efficiency can be combined in rehearsal-free continual learning.

What carries the argument

The machinery is a dual-adapter transformer: blocks 1 through $l$ host a task-shared LoRA branch ($A_s B_s$, with $B_s$ a fixed random orthogonal matrix from an SVD of a Gaussian matrix and $A_s$ zero-initialized and continuously updated), and blocks $l+1$ through $N$ host task-specific LoRA branches ($A_t B_t$) multiplied by learnable block-wise scaling factors $\mu_t$ and constrained by an orthogonality loss between tasks. Around this sit two stabilizers: early-exit knowledge distillation at block $l$, whose gradient is reassigned proportionally to the $\ell^2$ norms of the previous shared up-projection's weight vectors, and the orthogonality regularization on the block-weight vectors. The fixed $B_s$ plays the load-bearing role: it projects inputs into a low-dimensional space whose geometry is preserved because $B_s B_s^\top = I$, giving the shared up-projection $A_s$ a stable coordinate system in which to accumulate knowledge.

What would settle it

Run CL-LoRA on a benchmark where each new task changes the input distribution drastically (e.g., VTAB's cross-domain splits); if the fixed shared subspace cannot accommodate the shift, accuracy on earlier tasks should decay despite the distillation loss, which would show the method only handles moderate shifts. More directly, replace $B_s$ with a fixed random non-orthogonal matrix on ImageNet-R with $T=20$: the paper's Table 4 predicts a collapse from about 79% to under 3% final accuracy, so a failure to collapse would falsify the orthogonality claim.

Watch

Extended reading notes

Core claim

In class-incremental learning without rehearsal, the paper claims that a single shared low-rank adapter can carry cross-task knowledge if its down-projection is a fixed random orthogonal matrix $B_s$ (so $B_s B_s^\top = I$) and only the up-projection $A_s$ is trained, provided the gradient of a knowledge-distillation loss at the early-exit point is reassigned by the $\ell^2$ norms of the previous task's $A_s$ vectors. The shared adapter sits in the first $l$ transformer blocks, which learn general features, while task-specific adapters with learnable block-wise scaling factors (regularized to be nearly orthogonal between tasks) occupy the remaining blocks. With this split, CL-LoRA achieves the best reported accuracy on ImageNet-R, ImageNet-A, and VTAB, and competitive accuracy on CIFAR-100, at 0.3% trainable parameters; the shared blocks make inference cheaper because each test sample passes through them only once, reducing forward passes from $O(NT)$ to $O(l + (N-l)T)$.

Load-bearing premise

The method assumes that a fixed random orthogonal down-projection matrix $B_s$ creates a stable low-dimensional subspace in which shared knowledge can accumulate without forgetting; if that orthogonality or fixedness is removed, the shared adapter stops working, since random $B_s$ collapses accuracy to roughly 9% on CIFAR-100.

Editorial extensions

If this is right

  • Rehearsal-free CIL can work with a frozen backbone plus a tiny trainable fraction: 0.3% of backbone parameters suffice for strong accuracy on four benchmarks.
  • Inference on a stream of $T$ tasks costs $O(l + (N-l)T)$ adapter forward passes per test sample instead of $O(NT)$, so the method scales to longer task sequences.
  • A fixed random orthogonal down-projection outperforms a trainable down-projection at every tested split position $l$, in line with the asymmetry analysis of LoRA (reference [61]).
  • Very low ranks suffice: $r=1$ already gives strong results on CIFAR-100, and adding more projection matrices ($W_q$, $W_k$, $W_v$) does not reliably help.
  • The early-exit knowledge distillation plus gradient reassignment is what keeps the shared adapter from drifting to the most recent task; removing it degrades accuracy, especially on longer sequences.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same fixed-orthogonal down-projection idea could be tested as a drop-in replacement for the shared branches in other PEFT continual learners (e.g., prompts or adapters), where the shared component is currently trained without a fixed subspace.
  • The early-exit position $l$ is currently fixed at 6 for all datasets; the paper suggests it could be tuned from the first few tasks, which would make the method adaptive to datasets with different degrees of distribution shift.
  • Because the shared subspace is built into the architecture rather than enforced by replay, the approach might extend to online or blurry-boundary CIL where task identity is unknown during training; the paper notes this possibility but does not test it.
  • One testable prediction: if the orthogonality of $B_s$ matters mainly for geometry preservation, replacing the SVD construction with a QR-based orthogonalization should give identical accuracy; a small ablation could confirm this cheaply.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper proposes CL-LoRA, a dual-adapter architecture for rehearsal-free class-incremental learning (CIL) with pre-trained vision transformers. A task-shared LoRA branch with a fixed random orthogonal down-projection and a continuously updated up-projection is placed in the first l transformer blocks to accumulate cross-task knowledge, while task-specific LoRA branches with learnable block-wise scaling factors and an orthogonality penalty are used in the remaining blocks. The method is evaluated on CIFAR-100, ImageNet-R, ImageNet-A, and VTAB, reporting state-of-the-art or competitive accuracy with 0.3% trainable parameters and reduced inference complexity O(l+(N-l)T).

Significance. The paper is well-motivated and addresses a real limitation of adapter-based CIL, namely parameter redundancy and the lack of cross-task knowledge sharing. The design of a stable shared low-rank subspace via a fixed random orthogonal down-projection is interesting and is supported by a clear ablation (Table 4) showing the orthogonality property is essential. The experimental protocol is solid in many respects: ten seeds, standard benchmarks, and ablations for each proposed component. If the two load-bearing issues identified below are resolved, the method could be a useful contribution. However, the current evaluation protocol has an unresolved train/test mismatch for old-task prototypes, and the reported parameter efficiency appears to undercount the total stored parameters.

major comments (2)
  1. [Section 4.4, Eq. (14), Algorithm 1] There is a train/test inconsistency for old-task prototypes. During training of task i, prototypes p_i are computed with the shared up-projection A_s^i at that time, but at inference Eq. (14) passes every test sample through the final shared adapter A_s^T, because Algorithm 1 stores only the task-specific (A_i,B_i) and U_i, not per-task snapshots of A_s. Since A_s is continuously updated across tasks and gradient reassignment explicitly allows some dimensions to change, the features fed to the old task-specific branches are produced by a different shared adapter than the one used to compute the stored prototypes. The sentence in Section 4.4 that the prototypes are 'computed using the same adapter combination during training' is therefore not correct for the shared branch. This issue is unmeasured. If the correct procedure requires per-task A_s^i snapshots, the inference complexity becomes O(lT+(N-l)T)=O(NT), which invalidates the O(l+(N-l)T) scalability claim in Section 5.4. The authors should either store per-task snapshots of the shared adapter and revise the complexity claim, or provide direct evidence that A_s drift is negligible, for example by measuring prototype-feature distance or comparing final-adapter vs snapshot-adapter accuracy.
  2. [Table 1, Section 5.1, Sections 5.2 and 5.4] The reported parameter efficiency appears to count only the per-task incremental parameters, not the total number of additional parameters used at inference. With the stated setup (r=10, LoRA on Wq and Wv, l=6, N=12, ViT-B/16), the shared branch has 6 blocks x 2 projections x 768x10 = 92,160 trainable A_s parameters, and each new task adds 6 blocks x 2 projections x (768x10 + 10x768) = 184,320 trainable parameters. For T=20, the total is about 3.78 million parameters, which is roughly 4.4% of the ViT-B/16 backbone, not 0.3%. The comparison methods in Table 1 report total stored parameters, so this undercount makes the parameter-efficiency advantage in the abstract, Section 5.2, and Section 5.4 misleading. The table should report the total number of stored parameters at the end of the task sequence, or clearly state and consistently use a per-task metric.
minor comments (2)
  1. [Eq. (12)] U_t is defined as a concatenation of scalar block-scaling factors, so U_t is a vector and U_t^T U_i is a scalar; the double sum over j,k is ill-defined. The loss should be written as a single Frobenius norm, or the definition of U_t should be expanded to a matrix if the block weights are intended to be per-head or per-dimension.
  2. [Eq. (9) and Algorithm 1, line 22] The knowledge distillation loss is computed by passing both the current and previous shared-adapter features through the current local classifier h_t^phi. This is closer to a feature-consistency regularizer than to standard distillation from a previous task's classifier. The definition of s_{t-1} and the role of the current classifier should be clarified, since the notation suggests a sum over current classes i in C_t for both distributions.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: the central design is empirically validated against external benchmarks, and the only self-citation is a non-load-bearing motivation for gradient reassignment.

full rationale

The claimed derivation chain does not reduce to its own inputs. The shared-adapter design (fixed random orthogonal down-projection B_s with trainable up-projection A_s) is motivated by the external theoretical analysis in [61] and validated by direct ablations (Table 4 and Figure 3), not assumed. The gradient-reassignment mechanism in Eq. (10) is fully specified in this paper and its contribution is shown by the ablation in Table 2; the citation to the first author's prior work [13] is used only as motivation and is not load-bearing. The task-specific block-weight orthogonality in Eq. (12) is defined and optimized directly. Accuracy, parameter-efficiency, and inference-efficiency claims are empirical comparisons against external baselines (Table 1) or follow algebraically from Eqs. (8), (11), and Algorithm 1, so no fitted constant is renamed as a prediction. One non-circular concern is noted: Eq. (14) evaluates all tasks with the final shared up-projection A_s, whereas prototypes were computed during each task's training with the then-current A_s, and Algorithm 1 stores only (A_t, B_t) and U_t, not per-task A_s snapshots. This is a train/test feature-drift and scalability caveat, but it is not a circularity because no claimed result is defined in terms of itself.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on five hand-chosen hyperparameters and on three heuristic assumptions about ViT layer semantics, orthogonal random projections, and the meaning of L2 norms in LoRA up-projections. No new physical entities are introduced; the block-wise scaling factors are learnable parameters in an existing architecture.

free parameters (5)
  • lambda_1 (KD weight) = 5
    Global weighting of the knowledge distillation loss in Eq. (13); fixed across all benchmarks without a described tuning procedure.
  • lambda_2 (orthogonality weight) = 0.0001
    Global weighting of the block-wise orthogonality loss in Eq. (13); fixed across all benchmarks.
  • l (shared adapter position) = 6
    Number of early transformer blocks that receive the shared adapter, chosen as half of the 12 blocks; ablations show sensitivity to this value.
  • LoRA rank r = 10
    Rank of the low-rank adapters; Table 3 shows small ranks work well, so r=10 is a hand-chosen budget.
  • temperature tau for KD = 2
    Softening temperature in Eq. (9); fixed without sensitivity analysis.
assumptions (5)
  • domain assumption Early transformer blocks (first l) encode generalizable, cross-task visual features while later blocks encode task-specific features.
    The dual-adapter placement in Section 4.1 and Eq. (8) relies on this hierarchy; the flip experiment in Figure 5 supports it only on ImageNet-R.
  • ad hoc to paper A fixed random orthogonal down-projection B_s with B_s B_s^T = I preserves distances and angles in the low-rank space, providing a stable subspace for shared knowledge accumulation.
    Central to the task-shared adapter (Eqs. 6-7); no proof is given, and Table 4 shows a random non-orthogonal B_s collapses, so the orthogonality is critical.
  • ad hoc to paper L2 norms of up-projection weight vectors in A_s indicate which dimensions are important for prior knowledge, so reassigning KD gradients by these norms preserves that knowledge.
    Gradient reassignment in Eq. (10); motivated by ref. [13] but asserted without independent validation.
  • domain assumption Prototype-based classification with cosine similarity over frozen and adapter-augmented features is sufficient for CIL inference.
    Adopted from prior rehearsal-free PTM-based methods [56,58], used in Eq. (14); not re-derived.
  • domain assumption A pre-trained ViT-B/16 on ImageNet-21K provides a sufficiently general backbone for all four benchmarks.
    The method freezes this backbone; performance on VTAB and ImageNet-A depends on transferable pre-trained features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning." pith.science (2026). https://pith.science/paper/NPOOBUC2

@misc{pith2026250524816,
  author       = {Pith},
  title        = {Pith review of: CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NPOOBUC2}},
  note         = {Machine review of arXiv:2505.24816}
}
read the original abstract

Class-Incremental Learning (CIL) aims to learn new classes sequentially while retaining the knowledge of previously learned classes. Recently, pre-trained models (PTMs) combined with parameter-efficient fine-tuning (PEFT) have shown remarkable performance in rehearsal-free CIL without requiring exemplars from previous tasks. However, existing adapter-based methods, which incorporate lightweight learnable modules into PTMs for CIL, create new adapters for each new task, leading to both parameter redundancy and failure to leverage shared knowledge across tasks. In this work, we propose ContinuaL Low-Rank Adaptation (CL-LoRA), which introduces a novel dual-adapter architecture combining \textbf{task-shared adapters} to learn cross-task knowledge and \textbf{task-specific adapters} to capture unique features of each new task. Specifically, the shared adapters utilize random orthogonal matrices and leverage knowledge distillation with gradient reassignment to preserve essential shared knowledge. In addition, we introduce learnable block-wise weights for task-specific adapters, which mitigate inter-task interference while maintaining the model's plasticity. We demonstrate CL-LoRA consistently achieves promising performance under multiple benchmarks with reduced training and inference computation, establishing a more efficient and scalable paradigm for continual learning with pre-trained models.

Figures

Figures reproduced from arXiv: 2505.24816 by the authors.

Figure 1
Figure 1. Overview of our dual-adapter architecture. The task [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of CL-LoRA for class-incremental learning. We insert shared adapters [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Final step accuracy AT (%) on CIFAR-100 (T = 10) and ImageNet-R (T = 20) by varying the position l ∈ {0, 2, 4, 6, 8, 10, 12} to split task-shared and specific adapters. Shaded regions indicate ± standard deviation around the mean. position after which we switch from task-shared to task￾specific adapters. For example, l = 0 means all adapters are task-specific (without knowledge distillation), while l = 12 means all … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Inference scalability comparison with varied position [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison between ours design (task [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Continual Knowledge Consolidation LORA for Domain Incremental Learning

    cs.LG 2025-10 conditional novelty 5.0 of 10

    CONEC-LoRA reports state-of-the-art accuracy on four domain-incremental benchmarks by combining task-shared and task-specific LoRAs with a stochastic classifier and a learned domain-ID selector.

Reference graph

Works this paper leans on

63 extracted references · 53 canonical work pages · cited by 1 Pith paper

  1. [1]

    Il2m: Class incremen- tal learning with dual memory

    Eden Belouadah and Adrian Popescu. Il2m: Class incremen- tal learning with dual memory. Proceedings of the IEEE In- ternational Conference on Computer Vision, pages 583–592,

    work page

  2. [2]

    Class-incremental continual learning into the extended der-verse

    Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class-incremental continual learning into the extended der-verse. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. 6, 2

    work page 2022

  3. [3]

    Dark experience for gen- eral continual learning: a strong, simple baseline

    Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for gen- eral continual learning: a strong, simple baseline. Advances in neural information processing systems, 33:15920–15930,

    work page

  4. [4]

    Castro, Manuel J

    Francisco M. Castro, Manuel J. Marin-Jimenez, Nicolas Guil, Cordelia Schmid, and Karteek Alahari. End-to-end in- cremental learning. Proceedings of the European Conference on Computer Vision, 2018. 1, 5

    work page 2018

  5. [5]

    Efficient lifelong learning with a- gem

    Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a- gem. arXiv preprint arXiv:1812.00420, 2018. 2

    arXiv 2018

  6. [6]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pat- tern recognition, pages 248–255, 2009. 1, 6

    work page 2009

  7. [7]

    Loss of plasticity in deep continual learning

    Shibhansh Dohare, J Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026):768–774, 2024. 5

    work page 2024

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representa- tions, 2021. 3, 6

    work page 2021

Show all 63 references
  1. [9]

    Podnet: Pooled outputs distilla- tion for small-tasks incremental learning

    Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distilla- tion for small-tasks incremental learning. Proceedings of the European Conference on Computer Vision , pages 86–102,

  2. [10]

    A unified continual learning framework with general parameter-efficient tuning

    Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learning framework with general parameter-efficient tuning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11483–11493, 2023. 1, 2, 6

    2023

  3. [11]

    Beyond prompt learning: Continual adapter for efficient rehearsal-free continual learning

    Xinyuan Gao, Songlin Dong, Yuhang He, Qiang Wang, and Yihong Gong. Beyond prompt learning: Continual adapter for efficient rehearsal-free continual learning. Proceedings of European Conference on Computer Vision, 2024. 2

    2024

  4. [12]

    Pilora: Prototype guided incremental lora for federated class-incremental learning

    Haiyang Guo, Fei Zhu, Wenzhuo Liu, Xu-Yao Zhang, and Cheng-Lin Liu. Pilora: Prototype guided incremental lora for federated class-incremental learning. Proceedings of the European Conference on Computer Vision, 2024. 8

    2024

  5. [13]

    Gradient reweighting: Towards imbalanced class-incremental learning

    Jiangpeng He. Gradient reweighting: Towards imbalanced class-incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16668–16677, 2024. 5

    2024

  6. [14]

    Exemplar-free online con- tinual learning

    Jiangpeng He and Fengqing Zhu. Exemplar-free online con- tinual learning. arXiv preprint arXiv:2202.05491, 2022. 2, 3

    2022 arXiv

  7. [15]

    Incremental learning in online scenario

    Jiangpeng He, Runyu Mao, Zeman Shao, and Fengqing Zhu. Incremental learning in online scenario. Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 13926–13935, 2020. 8

    2020

  8. [16]

    The many faces of robust- ness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robust- ness: A critical analysis of out-of-distribution generalization. Proceedings of the IEEE/CVF international ...

    2021

  9. [17]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. Natural adversarial examples. Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15262–15271, 2021. 6

    2021

  10. [18]

    Learning a unified classifier incrementally via rebalancing

    Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pages 831–839, 2019. 1, 2

    2019

  11. [19]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. International conference on machine learn- ing, pages 2790–2799, 2019. 1, 2

    2019

  12. [20]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 1, 2, 3, 8

    2021 arXiv

  13. [21]

    Generating instance-level prompts for rehearsal-free continual learning

    Dahuin Jung, Dongyoon Han, Jihwan Bang, and Hwanjun Song. Generating instance-level prompts for rehearsal-free continual learning. Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 11847–11857,

  14. [22]

    Overcoming catastrophic forgetting in neu- ral networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neu- ral networks. The National Academy of Sciences , 114(13...

    2017

  15. [23]

    The singular value decompo- sition: Its computation and some applications

    Virginia Klema and Alan Laub. The singular value decompo- sition: Its computation and some applications. IEEE Trans- actions on automatic control, 25(2):164–176, 1980. 4

    1980

  16. [24]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical Report, 2009. 6

    2009

  17. [25]

    Learning without forgetting

    Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 40(12):2935–2947, 2017. 1, 2, 5

    2017

  18. [26]

    Inflora: Interference-free low-rank adaptation for continual learning

    Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23638–23647, 2024. 1, 2, 3, 5, 6, 8

    2024

  19. [27]

    Rotate your networks: Better weight consolidation and less catastrophic forgetting

    Xialei Liu, Marc Masana, Luis Herranz, Joost Van de Weijer, Antonio M Lopez, and Andrew D Bagdanov. Rotate your networks: Better weight consolidation and less catastrophic forgetting. 2018 24th International Conference on Pattern Recognition (ICPR), pages 2262–2268, 2018. 2

    2018

  20. [28]

    Generative feature replay for class- incremental learning

    Xialei Liu, Chenshen Wu, Mikel Menta, Luis Herranz, Bog- dan Raducanu, Andrew D Bagdanov, Shangling Jui, and Joost van de Weijer. Generative feature replay for class- incremental learning. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition Work...

    2020

  21. [29]

    Mnemonics training: Multi-class incremental learning without forgetting

    Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi-class incremental learning without forgetting. Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 12245–12254, 2020. 2

    2020

  22. [30]

    Adaptive aggre- gation networks for class-incremental learning

    Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggre- gation networks for class-incremental learning. Proceedings of the IEEE/CVF conference on Computer Vision and Pat- tern Recognition, pages 2544–2553, 2021. 2

    2021

  23. [31]

    Gradient episodic memory for continual learning

    David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neu- ral information processing systems, pages 6467–6476, 2017. 1, 2

    2017

  24. [32]

    Class-incremental exemplar compression for class- incremental learning

    Zilin Luo, Yaoyao Liu, Bernt Schiele, and Qianru Sun. Class-incremental exemplar compression for class- incremental learning. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 11371–11380, 2023. 2

    2023

  25. [33]

    Catastrophic inter- ference in connectionist networks: The sequential learning problem

    Michael McCloskey and Neal J Cohen. Catastrophic inter- ference in connectionist networks: The sequential learning problem. pages 109–165. Elsevier, 1989. 1

    1989

  26. [34]

    Ranpac: Ran- dom projections and pre-trained models for continual learn- ing

    Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Ran- dom projections and pre-trained models for continual learn- ing. Advances in Neural Information Processing Systems , 36, 2024. 1, 2, 6

    2024

  27. [35]

    Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning

    Jun-Yeong Moon, Keon-Hee Park, Jung Uk Kim, and Gyeong-Moon Park. Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning. Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 11731–11741, 2023. 8

    2023

  28. [36]

    Adaptive explicit knowledge transfer for knowledge distillation.arXiv preprint arXiv:2409.01679, 2024

    Hyungkeun Park and Jong-seok Lee. Adaptive explicit knowledge transfer for knowledge distillation.arXiv preprint arXiv:2409.01679, 2024. 5

    2024 arXiv

  29. [37]

    How do vision transformers work? International Conference on Learning Representa- tions, 2022

    Namuk Park and Songkuk Kim. How do vision transformers work? International Conference on Learning Representa- tions, 2022. 4, 2

    2022

  30. [38]

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. iCaRL: Incremental clas- sifier and representation learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ,

  31. [39]

    Convolutional prompting meets language models for continual learning

    Anurag Roy, Riddhiman Moulick, Vinay K Verma, Sap- tarshi Ghosh, and Abir Das. Convolutional prompting meets language models for continual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23616–23626, 2024. 2

    2024

  32. [40]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Chal- lenge. International Journal of Computer Vision ...

    2015

  33. [41]

    Coda-prompt: Con- tinual decomposed attention-based prompting for rehearsal- free continual learning

    James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Con- tinual decomposed attention-based prompting for rehearsal- free continual learning. Proceedings of the IEE...

    2023

  34. [42]

    Continual dif- fusion: Continual customization of text-to-image diffusion with c-loRA

    James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual dif- fusion: Continual customization of text-to-image diffusion with c-loRA. Transactions on Machine Learning Research,

  35. [43]

    Pilot: A pre-trained model-based continual learning toolbox

    Hai-Long Sun, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Pilot: A pre-trained model-based continual learning toolbox. arXiv preprint arXiv:2309.07117, 2023. 6, 2

    2023 arXiv

  36. [44]

    Dylora: Parameter efficient tuning of pre- trained models using dynamic search-free low-rank adapta- tion

    Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre- trained models using dynamic search-free low-rank adapta- tion. arXiv preprint arXiv:2210.07558, 2022. 3

    2022 arXiv

  37. [45]

    Foster: Feature boosting and compression for class- incremental learning

    Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class- incremental learning. Proceedings of the European Confer- ence on Computer Vision, pages 398–414, 2022. 2

    2022

  38. [46]

    A comprehensive survey of continual learning: theory, method and application

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 1

    2024

  39. [47]

    Or- thogonal subspace learning for language model continual learning

    Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Or- thogonal subspace learning for language model continual learning. 2023. 1, 2, 3, 5, 6, 8

    2023

  40. [48]

    Dualprompt: Complementary prompting for rehearsal-free continual learning

    Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. European Conference on Computer Vision, pages 631–648, 2022. 1, 2, 6, 7

    2022

  41. [49]

    Learning to prompt for continual learning

    Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jen- nifer Dy, and Tomas Pfister. Learning to prompt for continual learning. Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 139–149...

    2022

  42. [50]

    Online-lora: Task-free online continual learning via low rank adaptation

    Xiwen Wei, Guihong Li, and Radu Marculescu. Online-lora: Task-free online continual learning via low rank adaptation. NeurIPS 2024 Workshop on Scalable Continual Learning for Lifelong Foundation Models, 2024. 1, 2

    2024

  43. [51]

    Continual learning with low rank adap- tation

    Martin Wistuba, Prabhu Teja Sivaprasad, Lukas Balles, and Giovanni Zappella. Continual learning with low rank adap- tation. arXiv preprint arXiv:2311.17601, 2023. 2

    2023 arXiv

  44. [52]

    Large scale in- cremental learning

    Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale in- cremental learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019. 1, 2, 5

    2019

  45. [53]

    A large-scale study of representation learning with the visual task adaptation benchmark

    Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djo- longa, Andre Susano Pinto, Maxim Neumann, Alexey Doso- vitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv ...

    1910 arXiv

  46. [54]

    Adalora: Adaptive budget alloca- tion for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget alloca- tion for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023. 3

    2023 arXiv

  47. [55]

    Maintaining discrimination and fairness in class incremental learning

    Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu- Tao Xia. Maintaining discrimination and fairness in class incremental learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 13208– 13217, 2020. 5

    2020

  48. [56]

    Revisiting class-incremental learning with pre- trained models: Generalizability and adaptivity are all you need

    Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental learning with pre- trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision , pages 1– 21, 2024. 1, 2, 3, 6

    2024

  49. [57]

    Continual learning with pre-trained mod- els: A survey

    Da-Wei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre-trained mod- els: A survey. arXiv preprint arXiv:2401.16386, 2024. 1, 6, 2

    2024 arXiv

  50. [58]

    Expandable subspace ensemble for pre-trained model-based class-incremental learning

    Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class-incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23554–23564, 2024. 1, 2, 3, 6, 8

    2024

  51. [59]

    Class-incremental learning: A survey

    Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De- Chuan Zhan, and Ziwei Liu. Class-incremental learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9851–9873, 2024. 6, 2

    2024

  52. [60]

    Prototype augmentation and self-supervision for incremental learning

    Fei Zhu, Xu-Yao Zhang, Chuang Wang, Fei Yin, and Cheng- Lin Liu. Prototype augmentation and self-supervision for incremental learning. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 5871–5880, 2021. 2, 3

    2021

  53. [61]

    Asymmetry in low-rank adapters of foundation models

    Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz S´aez de Oc ´ariz Borde, Rickard Br ¨uel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models. arXiv preprint arXiv:2402.16842, 2024. 4,...

    2024 arXiv

  54. [62]

    Overview of Training Algorithm Algorithm 1 presents the training process of our CL-LoRA for class-incremental learning

    Extended Illustration of Methodology 7.1. Overview of Training Algorithm Algorithm 1 presents the training process of our CL-LoRA for class-incremental learning. Given a sequence of tasks {Tt}T t=1, our method learns each task sequentially while leveraging both task-shared and...

  55. [63]

    Results show last step accuracy AT and average accuracy A (%)

    and ImageNet-R ( T = 20). Results show last step accuracy AT and average accuracy A (%). In this work, as described in Section 4.1, we use SVD decomposition to generate the random orthogonal matrix Bs by first generating a random matrix M ∼ N(0, 1) fol- lowed by M = UΣV⊤ and s...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.