Pith. sign in

REVIEW 4 major objections 6 minor 54 references

Continual Gradient Low-Rank Projection Fine-Tuning for LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training full-rank and LoRA parameters in one shared low-rank gradient subspace lets an LLM keep learning new tasks without forgetting old ones.

desk verdict Useful incremental method for continual fine-tuning of LLMs, but the forgetting-protection mechanism rests on an unvalidated proxy and the empirical case lacks error bars. read the letter →

arxiv 2507.02503 v1 pith:5LQVLBY2 submitted 2025-07-03 cs.LG cs.AIcs.CE

Chenxu Wang , Yilin Lyu , Zicheng Sun , Liping Jing This is my paper · ORCID
classification cs.LGcs.AIcs.CE
keywords continuallearninglargelanguagemodelslow-rankadaptationgradientprojectioncatastrophicforgettingparameter-efficientfine-tuningstability-plasticitytrade-offAdamfirst-ordermoment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Continual fine-tuning of a large language model usually forces a choice: keep training cheap with low-rank adapters (LoRA) and suffer limited expressiveness, or train full-rank and pay in memory and in forgetting of earlier tasks. This paper argues that the choice is false: both kinds of parameters can be trained together inside a single low-rank gradient subspace, built not from sampled hidden features but from the first-order moment that Adam already maintains. New tasks' gradients are projected orthogonal to that shared subspace, so old knowledge is protected implicitly rather than by explicit parameter constraints. If this holds, continual learners can approach multi-task upper-bound accuracy while cutting forgetting several-fold on T5, LLaMA2, and the long-sequence TRACE benchmark. The result matters because it points to a cheap, general recipe for adapting one model to a stream of tasks without storing old data.

What carries the argument

Two coupled mechanisms carry the argument. The first, Gradient Shared Space Construction, treats Adam's first-order moment $M_t$ from each trained layer as a proxy for that task's gradient space: after a task finishes, GORP takes the SVD of each layer's moment, keeps only the top-$k$ singular vectors that satisfy a Frobenius-norm threshold, and concatenates them into a shared space $S$; for later tasks the moment is first projected orthogonal to $S$ so that only genuinely new directions are added, and small singular values are truncated to keep the space bounded. The second, Low-Rank Projection Optimization, applies the actual gradient constraints: for full-rank parameters, the gradient $G_t$ is compressed to $G'_t = U_{l,k}^\top G_t$, then projected to $P_t = G'_t - S(S^\top G'_t)$ before the Adam update, and the result is scaled back by $\alpha U_{l,k}P'_t V_{l,k}^\top$; the low-rank decomposition is refreshed only every $T$ steps to control cost. LoRA's $A$ matrix receives the same orthogonal projection directly. The work done by this machinery is to confine every task's updates to directions that do not re-use the gradient components that old tasks already claimed, while still allowing the model to move in any direction within the shared space.

What would settle it

Train two tasks in sequence with GORP but replace the gradient shared space with a random low-rank subspace of the same rank, or with moments computed from tasks with shuffled labels; then measure task-1 accuracy after task-2 training. If forgetting stays as low as in GORP, the moment-based subspace is not what provides stability; if forgetting rises, the proxy is doing the work. A complementary check is to build the shared space from terminal-step gradients only instead of Adam moments and compare backward transfer on the same task order.

Watch

Extended reading notes

Core claim

On its own terms, the paper's claim is that GORP—Gradient LOw Rank Projection—resolves the stability-plasticity dilemma in continual LLM fine-tuning by expanding the optimization space and then constraining it in the right place. It augments LoRA with learnable full-rank parameters for the current task, exploits the observation that gradients of large models are effectively low-rank along training, and compresses each full-rank gradient update into the top-k singular directions before submitting it to Adam. At the same time, the Adam first-order moment from each finished task is decomposed and appended to a shared, fixed-capacity gradient space; updates for all later tasks are projected to the orthogonal complement of that space. In experiments, this combination reaches 79.8% average accuracy on the T5 standard continual benchmark versus 78.8% for N-LoRA and 75.8% for O-LoRA, approaches the 80.0% multi-task upper bound, and drops backward transfer to -0.8% from -4.9% for N-LoRA; similar gains appear on LLaMA2-7B and TRACE. The authors' interpretation is that full-rank parameters supply plasticity while the gradient-space projection supplies stability, and that explicit parameter orthogonality, as used by O-LoRA, is an unnecessary constraint.

Load-bearing premise

The load-bearing assumption is that the Adam first-order moment of a task's gradients is a faithful summary of the gradient directions that task needs, so that forcing future tasks to be orthogonal to those moment directions truly protects the old task; if the moment is a poor proxy, the whole stability mechanism gives little protection.

Editorial extensions

If this is right

  • Continual fine-tuning can approach multi-task upper-bound accuracy (79.8 vs 80.0 for MTL on the T5 standard benchmark) without storing or replaying old data.
  • Longer task sequences become viable: on the 15-task benchmark GORP scores 76.0 average accuracy versus 72.4 for N-LoRA, narrowing the gap to per-task fine-tuning.
  • Forgetting on old tasks drops several-fold, with BWT of -0.8% on the T5 standard benchmark compared to -4.9% for N-LoRA and -7.8% for O-LoRA.
  • Generative performance on unseen tasks improves by 7.0% over N-LoRA and 26.2% over O-LoRA on average across task orders, suggesting better transfer.
  • Per-task wall-clock time stays close to O-LoRA (128.1 vs 128.5 seconds) while reported FLOPs drop to 1.8e-3 of O-LoRA's, so the added full-rank parameters do not hurt efficiency in practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the shared subspace is built from optimizer moments with no stored data, the same recipe could serve other multi-task settings where task identity is available only through training trajectories, such as task arithmetic, model merging, or continual instruction tuning.
  • The interval-based refresh of the SVD basis (every T steps) turns the low-rank projection into a stale-but-cheap approximation; a natural extension is to adapt T or update the basis incrementally, and measure whether the accuracy-forgetting trade-off shifts.
  • GORP's orthogonality is enforced in gradient space, not parameter space; this predicts that old-task performance should be correlated with gradient conflict rather than with parameter distance, a claim that could be tested directly during training by logging cosine similarity between proposed updates and old-task gradients.
  • If the moment-as-subspace idea generalizes, it suggests exposing Adam's internal moments as a diagnostic of what a model has learned per task would be a low-cost way to detect interference before it becomes forgetting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes GORP, a continual fine-tuning strategy for LLMs that jointly trains LoRA and full-rank parameters while constraining updates to a low-rank gradient subspace. The subspace is built from the Adam first-order moment of gradients after each task, replacing the hidden-feature-based subspace used by GPM-style methods. The method is evaluated on T5-Large and LLaMA2-7B across the standard CL benchmark, a 15-task benchmark, and the TRACE benchmark, reporting higher average accuracy and lower backward transfer than O-LoRA and N-LoRA, alongside a claim of reduced computational cost. The central contribution is the moment-based shared gradient space together with low-rank projected full-rank updates.

Significance. If the empirical claims hold, GORP would provide a practical middle ground between LoRA efficiency and full fine-tuning expressiveness for continual learning, and the moment-based subspace could reduce the cost of subspace construction compared with sampling hidden features. The paper has notable strengths: it ships code, covers several benchmarks and two model families, includes ablations of the main components, reports generalization to unseen tasks, and gives a complexity comparison. However, the forgetting-prevention mechanism is not validated, and the headline results are based on single runs with hyperparameters selected on the test benchmark, so the significance is conditional on substantial additional evidence.

major comments (4)
  1. [Section 3.1, Eqs. (5)-(7); Algorithm 1 lines 17-21] The stability mechanism is not supported. The subspace S is constructed from the first-order moment M_t of gradients that have already been projected orthogonal to S_{t-1} and, for full-rank parameters, further rank-k compressed by Eq. (6). Therefore M_t is a biased, recency-weighted average of the optimizer's movement in the allowed subspace, not an estimate of task t's raw gradient space or of the directions in which old-task losses are sensitive. Standard GPM-style protection relies on a null-space argument tied to features or gradients of old tasks; no analogous argument is given here, and no experiment measures alignment between M_t and raw task gradients or ablates the subspace construction. Since this projection is the only forgetting-prevention mechanism, the claimed BWT improvements (Table 4) could stem from the added full-rank capacity or implicit regularization rather than from the moment-based subspace. Please add a formal or empirical justification that orthogonalizing future gradients to span(M_t) protects old tasks.
  2. [Tables 2-4, Figure 3; Appendix D, Table 10] The central empirical claim rests on single-run comparisons. The average-accuracy margins over N-LoRA are 1.0 point on T5 standard (79.8 vs 78.8), 1.0 point on LLaMA2 (78.6 vs 77.6), and 3.6 points on the 15-task benchmark (76.0 vs 72.4); without multiple seeds or error bars these could be within run-to-run variance. In addition, the rank k=8 is selected using Order 1 results (Table 10) and scale factors differ per order (Appendix D), so the protocol should clarify how hyperparameters were chosen and whether the comparisons are fair. Please report means and standard deviations over at least three seeds, and state the selection procedure.
  3. [Section 4.5, Table 5] The efficiency claim is not supported by the reported numbers. GORP's FLOPs are reported as 0.125e12, three orders of magnitude lower than O-LoRA and N-LoRA, yet wall-clock time per task is essentially unchanged (128.1 vs 128.5 seconds). This suggests the FLOPs figure counts only the incremental low-rank operations rather than total training cost, making the comparison apples-to-oranges. Please define the scope of the FLOPs measurement and report total training FLOPs and memory usage.
  4. [Algorithm 1 and Eqs. (6)-(12)] The update rules are not reproducible as written. Eq. (6) defines G'_{t,l} = U^T_{l,k} G_{t,l} V_{l,k}, but Algorithm 1 line 9 applies only G'_t <- U^T_r G_t, and the text says the projection process is simplified to G'_{t,l} = U^T_{l,k} G_{t,l}. The reconstruction in Eq. (11) uses alpha U_{l,k} P' V^T_{l,k}, which is dimensionally inconsistent with a one-sided projection unless V is stored and reused. Also Algorithm 1 line 11, 'G'_t <- G'_{t-1}', appears to be a placeholder; it should specify that the previous low-rank factors are reused between intervals. Please make the algorithm self-contained.
minor comments (6)
  1. [Section 3.1, Eq. (1)] The approximation threshold epsilon_t is never defined or set; Appendix D gives lambda but not epsilon. Please specify how epsilon_t is chosen.
  2. [Algorithm 1] The same symbol M_t denotes the Adam first-order moment updated inside the loop (line 18) and the quantity used to update S after the loop (line 23); clarify that the final moment is used and how it relates to Eq. (2).
  3. [Figure 4] The bar labels B, G-S, G-L, G and the caption are confusing; the caption says L refers to low-rank projection for full-rank parameters and S refers to projection for LoRA, but the text describes adding low-rank projections to LoRA as improving over baseline. Please clarify which configuration each bar represents.
  4. [Appendix D] 'supplementary meterial' should be 'supplementary material', and the code link in the abstract should also appear in the main text.
  5. [Table 6] The header 'Domians' should be 'Domains'.
  6. [Section 4.2] The phrases 'GORP improves performance by 4%' and 'significant improvements' are not backed by significance tests; please rephrase as raw point differences or add statistical testing.

Circularity Check

2 steps flagged · score 3.0 of 10

Gradient-orthogonality 'evidence' is the algorithm's own constraint and Order-1 k is tuned on its own reported result; core accuracy claims remain independent.

  1. self definitional [Section 4.4 (Model Forgetting), Figure 2, Appendix C, Eq. 5]
    "Gradient space plays a crucial role in mitigating forgetting. While O-LoRA explicitly enforces orthogonality constraints on LoRA weights, GORP applies implicit constraints to regulate gradients. ... GORP demonstrates highly stable gradient orthogonality in Figure 2, enabling better gradient direction control while allowing parameters to update within a larger space, thereby increasing their degrees of freedom."

    The Gradient Orthogonality metric (Appendix C) is defined as ||G_{A,i}^T G_{A,j}||_2. But Eq. 5 defines the LoRA gradient used in training as G'_{A,l} = G_{A,l} - S_{t-1}(S_{t-1})^T G_{A,l}, i.e., the orthogonal projection onto the null space of the accumulated subspace S, with S built from earlier tasks' moment matrices (Eqs. 2-4). Hence any later-task LoRA gradient is orthogonal to earlier-task subspace components by construction; Figure 2's 'highly stable gradient orthogonality' is the enforcement mechanism itself, not an independent empirical benefit. Presenting it as evidence of forgetting control is circular.

  2. fitted input called prediction [Appendix E.1, Table 10; Table 2 Order-1]
    "Table 10 shows the impact of varying k values on the final results for the order 1. From the data, we observe that the rank of k = 8 yields superior performance compared to other values. ... we select k = 8 as the optimal rank for full-rank parameters."

    The rank k is chosen as the value that maximizes the final average accuracy on Order-1 of the Standard CL Benchmark, and Table 2 then reports GORP's Order-1 accuracy (79.7) as one of the main results. Thus the reported Order-1 number is a fitted quantity rather than an independent prediction. The other orders and TRACE results are less affected, so this is a localized evaluation-selection issue, not the central derivation.

full rationale

The paper is essentially an empirical training algorithm with no claimed first-principles derivation, so the main circularity patterns (self-citation chains, uniqueness theorems, ansatz smuggling, renaming) do not apply. The gradient shared space S is built online from the model's own Adam first-order moments and then used to project the same model's later gradients; this is self-referential in the standard way for gradient-projection continual learning, and it does not reproduce any fitted target value. The core accuracy claims are benchmarked against external baselines on multiple task orders and the TRACE suite, so they have independent content. Two supporting validations are weaker: (i) the reported 'highly stable gradient orthogonality' (Fig. 2) is a direct algebraic consequence of the projection in Eq. 5, so it is a tautology rather than evidence of anti-forgetting; and (ii) the full-rank rank k is selected in Appendix E.1 by maximizing Order-1 final accuracy, and the same Order-1 number is then reported in Table 2, which is tuning on the reported split. Neither flaw infects the central multi-order/TRACE accuracy comparisons, so the overall circularity score is low but not zero.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; the 'gradient shared space' is an algorithmic construct from existing gradients and does not require independent evidence. The main burden is the unproved moment-as-gradient-space assumption and the hand-selected hyperparameters.

free parameters (6)
  • Rank k for full-rank low-rank projection = 8
    Selected after tuning on Order 1 of the Standard CL Benchmark (Table 10), which makes the subsequent comparison to baselines partly dependent on this choice.
  • Scale factor alpha = 1 (Orders 1-3), 0.25 (Orders 4-6, LLaMA2)
    Set differently by order group and model; not justified by a general rule.
  • Full-rank learning rate = 1e-05 (T5 Orders 1-3), 1e-04 (T5 Orders 4-6), 1e-06 (LLaMA2)
    Differs across benchmark splits, suggesting per-benchmark tuning.
  • Low-rank update interval = 10 (T5), 20 (LLaMA2)
    Hand-chosen to reduce computational overhead; no sensitivity analysis for these values.
  • Subspace approximation threshold epsilon_t = unspecified
    Used in Eq. 1 and Eq. 3 to decide how many singular vectors to keep, but no value or selection procedure is reported.
  • lambda = 0.05 (T5), 0 (LLaMA2)
    Mentioned in Appendix D but never defined in the method, so its role and effect are unclear.
assumptions (4)
  • domain assumption Parameter gradients of transformer layers tend to be low-rank during training, so the top-k SVD projection retains most of the update information.
    Borrowed from Zhao et al. 2024 (GaLore) and used without additional validation in Section 3.2.
  • ad hoc to paper The Adam first-order moment of the gradient approximates the true gradient direction of a task and represents the overall gradient space better than sampled hidden features.
    Asserted in Section 3.1 with no proof or citation; it is the core justification for building S from M_t.
  • domain assumption Projecting new-task gradients orthogonal to the span of previous-task gradients prevents catastrophic forgetting.
    Standard assumption in gradient projection memory (Saha et al., 2021; O-LoRA) carried over without new proof.
  • standard math Evaluation metrics (average accuracy, BWT) computed on the final model reflect continual learning quality.
    Standard practice; not a point of concern.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continual Gradient Low-Rank Projection Fine-Tuning for LLMs." pith.science (2026). https://pith.science/paper/5LQVLBY2

@misc{pith2026250702503,
  author       = {Pith},
  title        = {Pith review of: Continual Gradient Low-Rank Projection Fine-Tuning for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5LQVLBY2}},
  note         = {Machine review of arXiv:2507.02503}
}
read the original abstract

Continual fine-tuning of Large Language Models (LLMs) is hampered by the trade-off between efficiency and expressiveness. Low-Rank Adaptation (LoRA) offers efficiency but constrains the model's ability to learn new tasks and transfer knowledge due to its low-rank nature and reliance on explicit parameter constraints. We propose GORP (Gradient LOw Rank Projection) for Continual Learning, a novel training strategy that overcomes these limitations by synergistically combining full and low-rank parameters and jointly updating within a unified low-rank gradient subspace. GORP expands the optimization space while preserving efficiency and mitigating catastrophic forgetting. Extensive experiments on continual learning benchmarks demonstrate GORP's superior performance compared to existing state-of-the-art approaches. Code is available at https://github.com/Wcxwcxw/GORP.

Figures

Figures reproduced from arXiv: 2507.02503 by the authors.

Figure 1
Figure 1. The framework of our Gradient Low Rank Projection (GORP) method. During [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The visualization comparison of gradient orthogonality between Baseline and our method using the T5 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison of the T5 model’s [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: The visualization comparison of parameter [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 4
Figure 4. Figure 4: Ablation study of our method. B refers to [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 25 canonical work pages

  1. [1]

    Kartikeya Badola, Shachi Dave, and Partha Talukdar. 2023. https://doi.org/10.18653/v1/2023.findings-acl.619 Parameter-efficient finetuning for robust continual multilingual learning . In Findings of the Association for Computational Linguistics: ACL 2023, pages 9763--9780, Toronto, Canada. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.findings-acl.619 2023

  2. [2]

    Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. https://doi.org/10.18653/v1/2022.acl-short.1 B it F it: Simple parameter-efficient fine-tuning for transformer-based masked language-models . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1--9, Dublin, Ireland. Association...

    doi:10.18653/v1/2022.acl-short.1 2022

  3. [3]

    Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. 2024. https://openreview.net/forum?id=aloEru2qCG Lo RA learns less and forgets less . Transactions on Machine Learning Research. Featured Certification

    2024

  4. [4]

    Haolin Chen and Philip N. Garner. 2024. https://arxiv.org/abs/2402.12220 Bayesian parameter-efficient fine-tuning for overcoming catastrophic forgetting . Preprint, arXiv:2402.12220

    work page Pith review arXiv 2024

  5. [5]

    Rajas Chitale, Ankit Vaidya, Aditya Kane, and Archana Santosh Ghotkar. 2023. https://openreview.net/forum?id=4CLNFKi12w Task arithmetic with lo RA for continual learning . In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@NeurIPS 2023)

    work page 2023

  6. [6]

    Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. Curran Associates Inc., Red Hook, NY, USA

    work page 2019

  7. [7]

    Wenyu Du, Shuang Cheng, Tongxu Luo, Zihan Qiu, Zeyu Huang, Ka Chun Cheung, Reynold Cheng, and Jie Fu. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.379 Unlocking continual learning abilities in language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6503--6522, Miami, Florida, USA. Association for Computat...

    work page doi:10.18653/v1/2024.findings-emnlp.379 2024

  8. [8]

    Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. 2020. https://proceedings.mlr.press/v108/farajtabar20a.html Orthogonal gradient descent for continual learning . In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 3762--3773. PMLR

    work page 2020

Show all 54 references
  1. [9]

    Yongchang Hao, Yanshuai Cao, and Lili Mou. 2024. https://arxiv.org/abs/2402.03293 Flora: Low-rank adapters are secretly gradient compressors . In Forty-first International Conference on Machine Learning

    2024 arXiv

  2. [10]

    Roee Hendel, Mor Geva, and Amir Globerson. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.624 In-context learning creates task vectors . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318--9333, Singapore. Association for Computational ...

    2023 doi

  3. [11]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. https://proceedings.mlr.press/v97/houlsby19a.html Parameter-efficient transfer learning for NLP . In Proceedings of the 36th In...

    2019

  4. [12]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  5. [13]

    Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. 2024. https://doi.org/10.18653/v1/2024.acl-long.77 Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal . In Proceedings of the ...

    2024 doi

  6. [14]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. https://www.pnas.org/doi/abs/1...

    2017 doi

  7. [15]

    Yajing Kong, Liu Liu, Zhen Wang, and Dacheng Tao. 2022. https://doi.org/10.1007/978-3-031-19809-0_13 Balancing stability and plasticity through advanced null space in continual learning . In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27...

    2022 doi

  8. [16]

    Zhizhong Li and Derek Hoiem. 2018. https://doi.org/10.1109/TPAMI.2017.2773081 Learning without forgetting . IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935--2947

    2018

  9. [17]

    Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. 2023. https://arxiv.org/abs/2307.05695 Relora: High-rank training through low-rank updates . Preprint, arXiv:2307.05695

    2023 arXiv

  10. [18]

    Yan-Shuo Liang and Wu-Jun Li. 2023. https://doi.org/10.1109/CVPR52729.2023.00755 Adaptive plasticity improvement for continual learning . In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7816--7825

    2023

  11. [19]

    Guoliang Lin, Hanlu Chu, and Hanjiang Lai. 2022 a . https://arxiv.org/abs/2110.07905 Towards better plasticity-stability trade-off in incremental learning: A simple linear connector . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), ...

    2022 arXiv

  12. [20]

    Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. 2022 b . https://proceedings.neurips.cc/paper_files/paper/2022/file/6728fcf94660c59c938319a6833a6073-Paper-Conference.pdf Beyond not-forgetting: Continual learning with backward knowledge transfer . In Advances in Neural Inform...

    2022

  13. [21]

    Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. 2022 c . https://openreview.net/forum?id=iEvAf8i6JjO TRGP: trust region gradient projection for continual learning . In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022...

    2022

  14. [22]

    Jialin Liu, Jianhua Wu, Jie Liu, and Yutai Duan. 2024 a . https://arxiv.org/abs/2409.19611 Learning attentional mixture of loras for language model continual learning . Preprint, arXiv:2409.19611

    2024 arXiv

  15. [23]

    Sheng Liu, Haotian Ye, Lei Xing, and James Y. Zou. 2024 b . https://proceedings.mlr.press/v235/liu24bx.html In-context vectors: Making in context learning more effective and controllable through latent space steering . In Proceedings of the 41st International Conference on Mac...

    2024

  16. [24]

    Yuheng Lu, Bingshuo Qian, Caixia Yuan, Huixing Jiang, and Xiaojie Wang. 2024. https://arxiv.org/abs/2410.16801 Controlled low-rank adaptation with subspace regularization for continued training on large language models . Preprint, arXiv:2410.16801

    2024 arXiv

  17. [25]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. https://aclanthology.org/P11-1015/ Learning word vectors for sentiment analysis . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: H...

    2011

  18. [26]

    Navyansh Mahla, Kshitij Sharad Jadhav, and Ganesh Ramakrishnan. 2025. https://arxiv.org/abs/2410.23111 Exploring gradient subspaces: Addressing and overcoming lora's limitations in federated fine-tuning of large language models . Preprint, arXiv:2410.23111

    2025 arXiv

  19. [27]

    Jingyang Qiao, Zhizhong Zhang, Xin Tan, Yanyun Qu, Wensheng Zhang, Zhi Han, and Yuan Xie. 2024. https://arxiv.org/abs/2405.13383 Gradient projection for continual parameter-efficient tuning . Preprint, arXiv:2405.13383

    2024 arXiv

  20. [28]

    Chengwei Qin and Shafiq R. Joty. 2022. https://openreview.net/forum?id=HCRVf71PMF LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of T5 . In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, Apri...

    2022

  21. [29]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1)

    2020

  22. [30]

    Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. 2023. https://openreview.net/forum?id=UJTgQBc91\_ Progressive prompts: Continual learning for language models . In The Eleventh International Conference on Learning Representations, I...

    2023

  23. [31]

    Gobinda Saha, Isha Garg, and Kaushik Roy. 2021. https://openreview.net/forum?id=3AOj0RCNC2 Gradient projection memory for continual learning . In International Conference on Learning Representations

    2021

  24. [32]

    Gobinda Saha and Kaushik Roy. 2023. https://doi.org/10.1609/aaai.v37i8.26157 Continual learning with scaled gradient projection . Proceedings of the AAAI Conference on Artificial Intelligence, 37(8):9677--9685

    2023 doi

  25. [33]

    Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. 2024. https://arxiv.org/abs/2404.16789 Continual learning of large language models: A comprehensive survey . Preprint, arXiv:2404.16789

    2024 arXiv

  26. [34]

    James Seale Smith, Yen - Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. 2024. https://openreview.net/forum?id=TZdEgwZ6f3 Continual diffusion: Continual customization of text-to-image diffusion with c-lora . Trans. Mach. Learn. Res., 2024

    2024

  27. [35]

    Chenyang Song, Xu Han, Zheni Zeng, Kuai Li, Chen Chen, Zhiyuan Liu, Maosong Sun, and Tao Yang. 2023. https://arxiv.org/abs/2309.14763 Conpet: Continual parameter-efficient tuning for large language models . Preprint, arXiv:2309.14763

    2023 arXiv

  28. [36]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama:...

    2023 arXiv

  29. [37]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. https://proceedings.neurips.cc/paper_files/paper/2019/file/4496bf24afe7fab6f046bf4923da8de6-Paper.pdf Superglue: A stickier benchmark for general-purp...

    2019

  30. [38]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5446 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : A...

    2018 doi

  31. [39]

    Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu. 2021. Training networks in null space of feature covariance for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 184--193

    2021

  32. [40]

    Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. 2023 a . https://doi.org/10.18653/v1/2023.findings-emnlp.715 Orthogonal subspace learning for language model continual learning . In Findings of the Association for Computat...

    2023 doi

  33. [41]

    Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang, Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023 b . https://arxiv.org/abs/2310.06762 Trace: A comprehensive benchmark for continual learning in large language models ...

    2023 arXiv

  34. [42]

    Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. 2022. https://doi.org/10.1109/CVPR52688.2022.00024 Learning to prompt for continual learning . In 2022 IEEE/CVF Conference on Computer Vision an...

    2022

  35. [43]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022 a . https://openreview.net/forum?id=yzkSU...

    2022

  36. [44]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022 b . http://papers.nips.cc/paper\_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html Chain-of-thought prompting elicits reason...

    2022

  37. [45]

    Martin Wistuba, Prabhu Teja S, Lukas Balles, and Giovanni Zappella. 2024. https://openreview.net/forum?id=DbsCOyoPRl Continual learning with low rank adaptation . In NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models

    2024

  38. [46]

    Wenhan Xia, Chengwei Qin, and Elad Hazan. 2024. https://arxiv.org/abs/2401.04151 Chain of lora: Efficient fine-tuning of language models via residual learning . Preprint, arXiv:2401.04151

    2024 arXiv

  39. [47]

    Shuo Yang, Kun-Peng Ning, Yu-Yang Liu, Jia-Yu Yao, Yong-Hong Tian, Yi-Bing Song, and Li Yuan. 2025. https://aclanthology.org/2025.coling-main.286/ Is parameter collision hindering continual learning in LLM s? In Proceedings of the 31st International Conference on Computational...

    2025

  40. [48]

    Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. https://proceedings.neurips.cc/paper_files/paper/2015/file/250cf8b51c773f3f8dc8b4be867a9a02-Paper.pdf Character-level convolutional networks for text classification . In Advances in Neural Information Processing Systems, volume 28...

    2015

  41. [49]

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024. https://openreview.net/forum?id=hYHsrKDiX7 Galore: Memory-efficient LLM training by gradient low-rank projection . In Forty-first International Conference on Machine Learning, ICM...

    2024

  42. [50]

    Yingxiu Zhao, Yinhe Zheng, Zhiliang Tian, Chang Gao, Jian Sun, and Nevin L. Zhang. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.766 Prompt conditioned VAE : Enhancing generative replay for lifelong learning in task-oriented dialogue . In Proceedings of the 2022 Conference...

    2022 doi

  43. [51]

    Junhao Zheng, Qianli Ma, Zhen Liu, Binquan Wu, and Huawen Feng. 2024 a . https://arxiv.org/abs/2401.09181 Beyond anti-forgetting: Multimodal continual instruction tuning with positive forward transfer . Preprint, arXiv:2401.09181

    2024 arXiv

  44. [52]

    Junhao Zheng, Shengjie Qiu, Chengming Shi, and Qianli Ma. 2024 b . https://arxiv.org/abs/2406.06391 Towards lifelong learning of large language models: A survey . Preprint, arXiv:2406.06391

    2024 arXiv

  45. [53]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  46. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.