Pith. sign in

REVIEW 4 major objections 4 minor 66 references

Soft-TransFormers for Continual Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Soft-TF claims that a frozen pre-trained transformer can keep learning new classes through per-task real-valued masks, with a proof that mask fine-tuning stays near the pre-trained solution.

desk verdict Real method, broken theory: the mask-based continual learner is worth a look, but the paper's proof and its own numbers don't back the strong claims. read the letter →

arxiv 2411.16073 v4 pith:XKG5HDJH submitted 2024-11-25 cs.LG cs.AIcs.CV

Haeyong Kang , Chang D. Yoo This is my paper · ORCID
classification cs.LGcs.AIcs.CV MSC 68T0568T07
keywords continuallearningclassincrementalsoftmasksfrozenpre-trainedtransformercatastrophicforgettingparameter-efficientfine-tuningwell-initializedlotterytickethypothesisprompt-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Soft-TransFormers (Soft-TF) claims that a frozen pre-trained transformer can learn a stream of new tasks by attaching one real-valued multiplicative mask per task to the query, key, value, and output projections of selected self-attention layers, while the underlying weights stay untouched. Because every mask is initialized at one, training begins exactly at the pre-trained solution, and the paper proves that under convex-Lipschitz assumptions, mask-space gradient descent has convergence rate and parameter drift controlled by the distance from the pre-trained weights to the task-optimal configuration. Two properties are claimed to follow: masks are never overwritten, so forgetting is structurally eliminated; and every masked model stays near the shared backbone, so selecting the wrong task's mask still evaluates a near-generalist function and task-inference errors are largely harmless. The paper reports leading accuracy with very low forgetting on vision and language class-incremental benchmarks at trainable-parameter budgets comparable to prompt, adapter, and LoRA methods, positioning the frozen-backbone-plus-soft-mask recipe as a replacement for those approaches.

What carries the argument

The load-bearing mechanism is the soft mask: a real-valued multiplier $m$, initialized to exactly 1 on the selected projection weights $w_Q$, $w_K$, $w_V$, and $w_O$, so the effective weight is $w \odot m$ and optimization starts at the pre-trained point. Alongside the shared G-prompt and task-specific E-prompt, each task owns a copy of these masks, and task selection picks the right mask at inference by task-key matching or by a one-shot gradient/entropy criterion. The analysis centers on the attention expression $m_Q \cdot (\text{prompt and patch outer products}) \cdot m_K^T$, reduced to convex-Lipschitz online learning, and the proof's conclusion is the ordering of drift bounds $B_m < B_p < B$ for masked, prompted, and unmodified configurations. This ordering is what converts the masks' initialization-at-one into near-zero forgetting and wrong-mask harmlessness: the architecture is built so that every decision the model makes happens through small multiplicative perturbations of a shared well-initialized backbone.

What would settle it

The decisive check is to measure accuracy on task t when the selector is forced to pick a different task's mask and compare with the correct mask and with the unmasked frozen backbone; the paper's claim predicts little drop, so a large drop refutes it. Also, on any nontrivial instance of the update $w(t+1) = w(t) - \eta v_t$, checking whether $||w(1)-w^*||^2$ equals $||w(T+1)-w^*||^2$ would show whether the lemma's key equality is a theorem.

Watch

Extended reading notes

Core claim

The central claim is that mask-only fine-tuning of a well-pre-trained transformer is not just another parameter-efficient trick: it is a way to keep the model's trajectory inside a small neighborhood of the pre-trained solution. For a frozen weight $w$ and a learnable real-valued mask $m$, the effective parameter is $w \odot m$, and the analysis in the appendix treats the attention bilinear form built from $m_Q z_t z_t^T m_K$ under standard convex-Lipschitz assumptions, deriving an upper bound on the average regret that is proportional to $B_m \rho / \sqrt{T}$, where $B_m$ is the distance from the masked configuration to the task optimum. The bound ranks mask drift below prompt drift ($B_m < B_p < B$), so the masked network stays closest to the shared pre-trained weights. That bounded drift is the load-bearing reason for both advertised properties: no parameter is ever overwritten, so forgetting is structurally impossible, and each task subnetwork is so close to the generalist backbone that even a wrong task mask yields a near-generalist function, decoupling accuracy from task-inference reliability. On the reported CIFAR-100 splits this yields high-90s accuracy with forgetting below one point for the gradient-based task selector, and on the BERT language benchmark it averages 79.4 across four task orders.

Load-bearing premise

The proof's load-bearing premise is that mask-space updates satisfy $||w(1)-w^*||^2 = ||w(T+1)-w^*||^2$ and that the mask drift bound sits strictly below the prompt drift bound; the equality does not follow from the update rule, so the bounded-drift conclusion is being assumed rather than derived.

Editorial extensions

If this is right

  • Forgetting becomes a structural guarantee rather than a managed risk: the frozen backbone and each task's mask are never overwritten, so no replay buffer or experience memory is needed to protect old tasks.
  • Task-inference errors stop being catastrophic: a wrong mask still leaves the model close to the pre-trained generalist, so class-incremental accuracy should remain high even when task identity is misidentified.
  • The method is a plug-in for existing prompt frameworks: soft masks can be trained alongside L2P, DualPrompt, PGP, and, per the abstract, HiDe-Prompt and NoRGa, with no change to inference cost beyond the unmodified backbone.
  • Initialization quality becomes a first-order factor: the proved bound ties convergence and drift to the distance between the pre-trained weights and the task-optimal solution, so better pre-training should translate directly into better continual learning.
  • At matched parameter budgets, mask fine-tuning outperforms the reported Adapter, LoRA, and prompt-only baselines on the tested vision and language CIL benchmarks, reaching high-90s accuracy with near-zero forgetting on CIFAR-100 splits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that wrong-mask harmlessness can be tested directly: random or deliberately wrong task assignment should degrade accuracy only mildly, and that experiment would separate the near-generalist claim from the quality of the task selector.
  • The distance-to-optimum bound suggests that pre-training progress, not the continual-learning update, dominates the reported gains; comparing the same architecture at successive pre-training checkpoints would show how quickly the mask advantage grows as initialization improves.
  • Because each task is an independent multiplier on the same backbone, averaging or interpolating masks across tasks is a natural next step for forward transfer, letting new tasks start from a blend of old masks rather than from the identity mask.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Soft-TransFormers (Soft-TF), a continual learning method that keeps a pretrained transformer frozen and learns per-task real-valued multiplicative masks over selected self-attention layers, optionally combined with prompts. The authors claim that mask-only fine-tuning enjoys bounded parameter drift, that forgetting is structurally eliminated, and that wrong-task inference is largely harmless; they support these claims with a convex-Lipschitz convergence analysis in Appendix A.1. The empirical section reports strong results on Split-CIFAR100, Split-ImageNet-R, CUB-200, TinyImageNet, and a BERT language benchmark against prompt-, adapter-, and LoRA-based baselines.

Significance. If the claims were established, the paper would make a substantial contribution: a rehearsal-free, parameter-efficient CIL method with a frozen backbone, no task-ID oracle, and state-of-the-art accuracy would be practically valuable. The experimental program is broad, includes ablations over mask location and initialization, comparisons with WSN, and provides code links. However, the paper's distinctive theoretical contribution is the advertised proof of bounded drift and harmless task-inference errors, and that proof is not valid as written. Moreover, the paper's own tables contradict the decoupling claim and the 'Upper-Bound' label. The empirical results may still be of interest, but they do not, by themselves, establish the paper's central narrative.

major comments (4)
  1. [Appendix A.1, Lemma 1, Eqs. (14)-(16)] The completed-square identity is algebraically incorrect. With the update w(t+1)=w(t)-eta*v_t, the correct identity is <w(t)-w*, v_t> = (1/(2*eta))[-||w(t+1)-w*||^2 + ||w(t)-w*||^2] + (eta/2)||v_t||^2. The paper instead writes -||w(t+1)-eta*v_t||^2, which equals -||w(t)-2*eta*v_t||^2 and is not the squared distance to w*. Consequently Eq. (16) is not a telescoping identity: the sum telescopes to ||w(1)-w*||^2 - ||w(T+1)-w*||^2, which equals zero only if the lemma's conclusion is assumed. The lemma also states the condition ||w(1)-w*||^2 = ||w(T+1)-w*||^2 as an assumption and later, in Eq. (17), invokes 'w(1)=0', contradicting the premise w(1) != 0 used in the Soft-TransFormer setting. Lemma 1 and Theorem 2 therefore do not establish bounded drift.
  2. [Appendix A.1, Eqs. (13), (20), (21)] The strict ordering B_m < B_p < B is asserted, not derived. The text supports it only with 'since all m are learnable parameters', but no argument shows that the distance from the mask-adapted final iterate to w* is smaller than the corresponding prompt-only distance, nor that both are smaller than the full-parameter distance. Because this ordering is exactly what is used to conclude that wrong-task masks are harmless and that accuracy is decoupled from task-inference reliability, the main theoretical conclusion is unsupported.
  3. [Tables 3, 6, and 8] The reported 'Upper-Bound of Soft-TF' is not an upper bound. On 10-Split-CIFAR100, Tables 3 and 6 report an upper bound of 93.90, while the same tables report Soft-TF with Gradient ID at 97.87, and Table 8 reports 98.05 for the same configuration. An upper bound cannot be lower than the method's own accuracy. The paper does not define how the upper-bound row is computed; this internal inconsistency undermines the claim that Soft-TF reaches an upper bound.
  4. [Abstract and Tables 1, 3, 8] The claim that 'task-inference errors are largely harmless' and that accuracy is 'decoupled from task-inference reliability' is contradicted by the paper's own Prompt-ID versus Gradient-ID results. For example, Table 1 and Table 8 show 92.35 with Prompt ID versus 98.05 with Gradient ID on 10-Split-CIFAR100, and 76.62 versus 83.70 on 10-Split-ImageNet-R. A 5-7 point drop is a substantial dependence on task-inference quality, not a demonstration of harmlessness.
minor comments (4)
  1. [Appendix A.1, Theorem 2] The statement 'if for all t we have that ||vt <= rho||' contains a typo; it should read ||v_t|| <= rho.
  2. [Table 3] The 'Upper-Bound of Soft-TF' row has missing entries for the pretrained-dataset and task-ID columns, and the same value 93.90 is repeated for both 10-Split-CIFAR100 and 5-Split-CUB200 in a way that is not explained.
  3. [Algorithm 1] Lines 14-16 of Algorithm 1 are incomplete: 'Calculate the prompted feature by' is followed by an empty line, and the per-sample loss line has no expression.
  4. [Throughout] There are numerous typographical issues, including 'Lipshitz' for 'Lipschitz', 'exsistances' for 'existence', and inconsistent spacing in 'T ransF ormers' and 'W ell-initialized'; these should be corrected.

Circularity Check

2 steps flagged · score 7.0 of 10

Appendix A.1's convergence proof assumes the bounded-drift conclusion: Lemma 1 postulates ||w(1)-w*||^2 = ||w(T+1)-w*||^2, and Theorem 2 assumes Bm < Bp < B, so the advertised drift bound and harmless-wrong-task claims are unproven.

  1. other [Appendix A.1, Lemma 1 and Eq. (16)]
    "Lemma 1. Let v1, · · ·, vT be an arbitrary sequence of vectors. Any algorithm with an well initialization (pre-trained model) w(1) ̸= 0 and an update rule of the form w(t+1) = w(t) − ηvt (11) satisfies with ||w(1) − w∗||2 = ||w(T +1) − w∗||2 ... The first sum on the right-hand side is a telescopic sum that collapses to ||w(1) − w∗||2 = ||w(T +1) − w∗||2 (16)"

    The lemma is supposed to prove that mask-only fine-tuning has bounded drift, but it assumes exactly that: the equality ||w(1)-w*||^2 = ||w(T+1)-w*||^2 is stated as a condition of Lemma 1 and then reused in Eq. (16) as if it were the result of telescoping. With the stated update rule w(t+1)=w(t)-ηv_t, the telescoping sum is ||w(1)-w*||^2 - ||w(T+1)-w*||^2, not the assumed equality. Moreover, Eq. (14) writes the completed-square term as -||w(t+1)-ηv_t||^2 instead of -||w(t+1)-w*||^2, so the subsequent collapse to ||w(1)-w*||^2 is algebraically invalid. The bounded-drift conclusion that supports 'forgetting is structurally eliminated' and 'wrong-task masks are largely harmless' therefore rests on the very property being proved.

  2. other [Appendix A.1, Theorem 2 and Eq. (21)]
    "Theorem 2. For every Bm < Bp < B, ρ >0 where Bm = ||w(T +1) m − w∗|| and Bp = ||w(T +1) p − w∗||, if for all t we have that ||vt ≤ ρ|| and if we set η = q B2 ρ2T , then for every w∗ with ||w(T +1) − w∗|| ≤B we have 1 T PT t=1 D w(t) − w∗, vt E ≤ Bmρ√ T < Bpρ√ T < Bρ√ T ."

    Theorem 2 assumes the strict ordering Bm < Bp < B as its premise and then 'proves' the same ordering as its conclusion (Eq. 21). The paper gives no derivation of Bm < Bp; it only asserts that '||w(1) m − w∗||2 < ||w(1) p − w∗||2 since all m are learnable parameters.' Thus the claim that mask drift is smaller than prompt drift, which is the basis for saying task-inference errors are largely harmless, is imported as an assumption rather than established. The theorem's conclusion is therefore equivalent to its premise.

full rationale

Score 7. The empirical evaluation is independent and self-contained: Soft-TF's benchmark numbers against prompt-, adapter-, LoRA-, and WSN-style baselines are direct experimental comparisons, and the self-citations to Kang et al. are used as baselines rather than as load-bearing proofs. However, the paper's advertised theory is circular. In Appendix A.1, Lemma 1 postulates ||w(1)-w*||^2 = ||w(T+1)-w*||^2, which is exactly the bounded-drift property the lemma is meant to establish; the telescoping-sum step labeled Eq. (16) is not a consequence of the update rule, and Eq. (14) also misstates the squared term as -||w(t+1)-ηv_t||^2 instead of -||w(t+1)-w*||^2. Theorem 2 then assumes Bm < Bp < B and returns the same inequality as its conclusion, so the strict ordering of mask drift below prompt drift is assumed, not proven. Consequently, the abstract's claims that convergence and parameter drift are controlled by the distance to a task-optimal configuration, and that wrong-task masks are largely harmless, reduce to the lemma's and theorem's own assumptions. This is a partial circularity in the theoretical narrative; the experimental results do not depend on that narrative. A separate, non-circularity correctness concern: Table 3 lists 'Upper-Bound of Soft-TF' as 93.90 on 10-Split-CIFAR100, which is lower than the reported Gradient-ID accuracy of 97.87 in the same table, a numerical impossibility if the upper bound assumes perfect task identity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method's reported gains rest on tuned choices (layer location, initialization, gradient-ID batch size) and on assumptions that the pretrained backbone is frozen and that test-time task inference is available. The theoretical analysis adds an unproved hypothesis and a flawed proof rather than independent support.

free parameters (4)
  • Mask layer location L[10,11,12] = L[10,11,12] for DualPrompt; L[3,4,5] for L2P
    Selected by layer-wise ablation on 10-Split-CIFAR100 (Figure 5, Section 5.3), then applied to all other benchmarks; selection on the evaluation benchmark.
  • Mask initialization = Uniform(1.0, 1.0)
    Table 4 compares Xavier, Kaiming, Normal, Uniform and selects Uniform on 10-Split-CIFAR100; this initialization choice affects the reported results.
  • Gradient ID shot count / batch = m-batch (full test mini-batch)
    Tables 9-10 show accuracy and test-time scale with the number of shots; the headline numbers use the full mini-batch, a tunable test-time parameter.
  • Number of masked layers = 3 layers (L[10,11,12])
    Ablation in Figure 5 and Tables 7-8 shows that 1, 2, or 3 layers can be used; 3 layers is the chosen budget for the main comparison.
assumptions (4)
  • domain assumption Convex-Lipschitz loss landscape
    Appendix A.1 restricts the analysis to convex-Lipschitz functions; the actual transformer cross-entropy loss is non-convex, so the theorem does not apply to the real setting.
  • ad hoc to paper Equality ||w(1)-w*||^2 = ||w(T+1)-w*||^2
    Lemma 1 in Appendix A.1 imposes this equality as a property of the update rule; it is not guaranteed and is used to derive the bounded-drift conclusion.
  • ad hoc to paper Well-initialized Lottery Ticket Hypothesis (WLTH)
    Section 1 postulates that a well-initialized dense network contains global solutions that retain old knowledge while learning new tasks; this is asserted, not proven.
  • domain assumption Test-time task inference via Gradient ID
    Algorithm 2 and Section 5.1 require a batch of test examples and gradient computation to select the task; the strongest reported numbers depend on this protocol.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Soft-TransFormers for Continual Learning." pith.science (2026). https://pith.science/paper/XKG5HDJH

@misc{pith2026241116073,
  author       = {Pith},
  title        = {Pith review of: Soft-TransFormers for Continual Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XKG5HDJH}},
  note         = {Machine review of arXiv:2411.16073}
}
read the original abstract

Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers. The masks are initialized at one, so optimization starts exactly at the pre-trained solution, and mask-space gradient descent is intrinsically biased toward modulating the backbone's dominant pathways; we prove that, under standard convex-Lipschitz assumptions, both the convergence rate and the parameter drift of mask-only fine-tuning are controlled by the distance from the pre-trained weights to a task-optimal configuration. This bounded drift yields two properties. Since the backbone and per-task masks are never overwritten, forgetting is structurally eliminated. And since every task subnetwork stays near the shared pre-trained solution, a wrong mask still evaluates a near-generalist function, so task-inference errors are largely harmless and class-incremental accuracy is decoupled from task-inference reliability. As a plug-in, Soft-TF couples with L2P, DualPrompt, HiDe-Prompt, and NoRGa, selecting masks by task-key matching, an entropy-gradient criterion, or a learned task-identity classifier. Across class-incremental learning benchmarks -- Split-CIFAR100, Split-ImageNet-R, CUB-200, and 5-Datasets -- Soft-TF consistently outperforms prompt-based, adapter-based, and LoRA-style baselines at comparable trainable-parameter budgets, while keeping inference cost identical to the unmodified backbone.

Figures

Figures reproduced from arXiv: 2411.16073 by the authors.

Figure 1
Figure 1. Soft-TransFormers (Soft-TF): the objective is to design a fully fine-tuned model that works well across multiple continual learning settings with incurring task-wise soft network training of attention and feedforward networks, leveraged by WLTH. Developing neural network models that leverage large-scaled pre-trained models. i.e., Vision Transformer (ViT) (Dosovitskiy et al., 2020) and Contrastive Language-Image Pre￾… view at source ↗
Figure 2
Figure 2. Comparisions of LLM-based fine-tuning methods (Adapters, LoRA) with Soft-TransFormer (Soft-TF): well-initialized Soft-TF θ ⊙mt is trained at le-th or only few attention layers while LLM fine-tuning methods such as Adapter (Houlsby et al., 2019) and LoRA (Hu et al., 2021) are trained at all attention layers. Note that all pre-trained parameters θ are fixed in fine-tuning t-th task parameter mt in Continual Learning (… view at source ↗
Figure 3
Figure 3. Attention Layer-(L[12]) Histogram Density Estimates of Pre-trained weight (θ) and Soft-TF (m) on 10-Split-CIFAR100. where f prompt θ defines the approach for attaching the prompt to the hidden embeddings. E-prompt & Soft-networks. e = {et} T t=1 is a set of task-dependent parameters, where et ∈ R Le×D has as sequence length of Le and the same embedding dimension D as the G-prompt, and T is the total number of tasks.… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Soft-TransFormers (Soft-TF): At training time, the E-Prompt and the Soft-network are selected according to task identity, and the selected G-Prompt, E-Prompt, and the Soft-networks (Soft-Attention and Feed Forwards) are trained together with a classifier. At test time,…
Figure 5
Figure 5. Figure 5: Layer-wise(L[∗]) Performances of Soft-TF on 10-Split-CIFAR100. Note that L[9,10,11] denotes Soft-TransFormer of 9, 10, 11 Layers. 5.2 Performances Performances of Soft-TF on Vision CIL. We compare our Soft-TransFormers (Soft￾TF) with state-of-the-art CIL baselines, as …
Figure 6
Figure 6. Figure 6: Attention map comparisons of DualPrompt and Soft-TF: at Layer-(L[12]) on a single category 7 sample x t i of 10-Split-CIFAR100. Attention Layer. We compare attention maps from the 12th attention layer (L[12]) of two different models, DualPrompt and Soft-TF, applied to …
Figure 7
Figure 7. Figure 7: Radar Chart of Comparisons in terms of average accuracy and forgetting between baselines and our SOTA method (Soft-TF). DualPrompt-PGP (Qiao et al., 2024) and DualPrompt-WSN (Kang et al., 2022) (c% sparsity) are baselines for prompt tuning and subnetworks. ACC refers t…
Figure 8
Figure 8. Figure 8: Comparisions of Soft-TransFormers with Subnetworks on 10-Split-CIFAR100. Note that L[10,11,12] denotes the fine-tuning layers of 10, 11, and 12. Peudo Codes. The overall process of the Soft-TransFormers (Soft-TF) during training and testing is described as Algorithm 1 …
Figure 9
Figure 9. Figure 9: Layer-(L[12]) Histogram Density Estimates of DualPrompt-Soft-TF’s Parame￾ters on 10-Split-CIFAR100. 0.20 0.00 0.20 Parameters 4 6 Density QKV_L[12]_Pre-trained Weight QKV_L[12]_Task2 0.20 0.00 0.20 Parameters 4 6 Density PROJ_L[12]_Pre-trained Weight PROJ_L[12]_Task2 (…
Figure 10
Figure 10. Figure 10: Layer-(L[12]) Histogram Density Estimates of Pre-trained Weight and DualPrompt-Soft-TF’s Parameters on 10-Split-CIFAR100. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_10.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 50 canonical work pages

  1. [1]

    Abai and N

    Z. Abai and N. Rajmalwar. Densenet models for tiny imagenet classification. arXiv preprint arXiv:1904.10429, 2019

    arXiv 1904

  2. [2]

    Aljundi, E

    R. Aljundi, E. Belilovsky, T. Tuytelaars, L. Charlin, M. Caccia, M. Lin, and L. Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems (NeurIPS), 2019

    work page 2019

  3. [3]

    Bohao, Z

    P. Bohao, Z. Tian, S. Liu, M.-C. Yang, and J. Jia. Scalable language model with generalized continual learning. In The Twelfth International Conference on Learning Representations, 2024

    work page 2024

  4. [4]

    Buzzega, M

    P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33: 0 15920--15930, 2020

    work page 2020

  5. [5]

    Caccia, R

    L. Caccia, R. Aljundi, N. Asadi, T. Tuytelaars, J. Pineau, and E. Belilovsky. New insights on reducing abrupt representation change in online continual learning. arXiv preprint arXiv:2104.05025, 2021

    arXiv 2021

  6. [6]

    Caron, H

    M. Caron, H. Touvron, I. Misra, H. J \'e gou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650--9660, 2021

    work page 2021

  7. [7]

    Chaudhry, M

    A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny. Efficient lifelong learning with a-gem. In Proceedings of the International Conference on Learning Representations (ICLR), 2019 a

    work page 2019

  8. [9]

    Chaudhry, M

    A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019 c

    arXiv 1902

Show all 66 references
  1. [10]

    Chaudhry, N

    A. Chaudhry, N. Khan, P. K. Dokania, and P. H. Torr. Continual learning in low-rank orthogonal subspaces. In Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  2. [11]

    J. Chen, Z. Yang, and D. Yang. Mixtext: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2147--2157, 2020

    2020

  3. [12]

    Chen, C.-J

    X. Chen, C.-J. Hsieh, and B. Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. arXiv preprint arXiv:2106.01548, 2021

    2021 arXiv

  4. [13]

    de Masson D'Autume, S

    C. de Masson D'Autume, S. Ruder, L. Kong, and D. Yogatama. Episodic memory in lifelong language learning. Advances in Neural Information Processing Systems, 32, 2019

    2019

  5. [14]

    D. Deng, G. Chen, J. Hao, Q. Wang, and P.-A. Heng. Flattening sharpness for dynamic gradient projection memory benefits continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  6. [15]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  7. [16]

    Douillard, A

    A. Douillard, A. Ram \'e , G. Couairon, and M. Cord. Dytox: Transformers for continual learning with dynamic token expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9285--9295, 2022

    2022

  8. [17]

    Hassabis, D

    D. Hassabis, D. Kumaran, C. Summerfield, and M. Botvinick. Neuroscience-inspired artificial intelligence. Neuron, 95 0 (2): 0 245--258, 2017

    2017

  9. [18]

    Hendrycks, S

    D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, ...

    2021

  10. [19]

    Houlsby, A

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790--2799. PMLR, 2019

    2019

  11. [20]

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  12. [21]

    Huang, Y

    Y. Huang, Y. Zhang, J. Chen, X. Wang, and D. Yang. Continual learning for text classification with information disentanglement based regularization. arXiv preprint arXiv:2104.05489, 2021

    2021 arXiv

  13. [22]

    S. Jung, H. Ahn, S. Cha, and T. Moon. Continual learning with node-importance based adaptive group sparse regularization. In Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  14. [23]

    H. Kang, R. J. L. Mina, S. R. H. Madjid, J. Yoon, M. Hasegawa-Johnson, S. J. Hwang, and C. D. Yoo. Forget-free continual learning with winning subnetworks. In International Conference on Machine Learning, pages 10734--10750. PMLR, 2022

    2022

  15. [24]

    H. Kang, J. Yoon, S. R. H. Madjid, S. J. Hwang, and C. D. Yoo. On the soft-subnetwork for few-shot class incremental learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=z57WK5lGeHd

    2023

  16. [25]

    H. Kang, J. Yoon, S. J. Hwang, and C. D. Yoo. Continual learning: Forget-free winning subnetworks for video representations, 2024 a . URL https://arxiv.org/abs/2312.11973

    2024 arXiv

  17. [26]

    H. Kang, J. Yoon, D. Kim, S. J. Hwang, and C. D. Yoo. Progressive fourier neural representation for sequential video compilation. In The Twelfth International Conference on Learning Representations, 2024 b . URL https://openreview.net/forum?id=rGFrRMBbOq

    2024

  18. [27]

    M. G. Z. A. Khan, M. F. Naeem, L. Van Gool, D. Stricker, F. Tombari, and M. Z. Afzal. Introducing language guidance in prompt-based continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11463--11473, 2023

    2023

  19. [28]

    S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54 0 (10s): 0 1--41, 2022

    2022

  20. [29]

    Kirkpatrick, R

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 0 (13): 0 3521--3526, 2017

    2017

  21. [30]

    Krizhevsky, G

    A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  22. [31]

    Kumar and H

    A. Kumar and H. Daume III. Learning task grouping and overlap in multi-task learning. In Proceedings of the International Conference on Machine Learning (ICML), 2012

    2012

  23. [32]

    X. Li, Y. Zhou, T. Wu, R. Socher, and C. Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In Proceedings of the International Conference on Machine Learning (ICML), 2019

    2019

  24. [33]

    X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  25. [34]

    Li and D

    Z. Li and D. Hoiem. Learning without forgetting. In Proceedings of the European Conference on Computer Vision (ECCV), 2016

    2016

  26. [35]

    Liang and W.-J

    Y.-S. Liang and W.-J. Li. Loss decoupling for task-agnostic continual learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  27. [36]

    H. Lin, B. Zhang, S. Feng, X. Li, and Y. Ye. Pcr: Proxy-based contrastive replay for online class-incremental continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24246--24255, 2023

    2023

  28. [37]

    Z. Mai, R. Li, H. Kim, and S. Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3589--3599, 2021

    2021

  29. [38]

    Mallya, D

    A. Mallya, D. Davis, and S. Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), 2018

    2018

  30. [39]

    McCloskey and N

    M. McCloskey and N. J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109--165. Elsevier, 1989

    1989

  31. [40]

    S. I. Mirzadeh, M. Farajtabar, D. Gorur, R. Pascanu, and H. Ghasemzadeh. Linear mode connectivity in multitask and continual learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  32. [41]

    Y. Pei, Z. Qing, S. Zhang, X. Wang, Y. Zhang, D. Zhao, and X. Qian. Space-time prompting for video class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11932--11942, October 2023

    2023

  33. [42]

    Qiao, zhizhong zhang, X

    J. Qiao, zhizhong zhang, X. Tan, C. Chen, Y. Qu, Y. Peng, and Y. Xie. Prompt gradient projection for continual learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=EH2O3h7sBI

    2024

  34. [43]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748--8763. PMLR, 2021

    2021

  35. [44]

    Razdaibiedina, Y

    A. Razdaibiedina, Y. Mao, R. Hou, M. Khabsa, M. Lewis, and A. Almahairi. Progressive prompts: Continual learning for language models. arXiv preprint arXiv:2301.12314, 2023

    2023 arXiv

  36. [45]

    Rebuffi, A

    S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001--2010, 2017

    2001

  37. [46]

    Riemer, I

    M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y. Tu, and G. Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910, 2018

    2018 arXiv

  38. [47]

    A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016

    2016 arXiv

  39. [48]

    G. Saha, I. Garg, and K. Roy. Gradient projection memory for continual learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  40. [49]

    Sarfraz, E

    F. Sarfraz, E. Arani, and B. Zonooz. Error sensitivity modulation based experience replay: Mitigating abrupt representation drift in continual learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=zlbci7019Z3

    2023

  41. [50]

    Serrà, D

    J. Serrà, D. Suris, M. Miron, and A. Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Proceedings of the International Conference on Machine Learning (ICML), 2018

    2018

  42. [51]

    Shalev-Shwartz and S

    S. Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014

    2014

  43. [52]

    H. Shin, J. K. Lee, J. Kim, and J. Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  44. [53]

    Singh, V

    P. Singh, V. K. Verma, P. Mazumder, L. Carin, and P. Rai. Calibrating cnns for lifelong learning. Advances in Neural Information Processing Systems, 33: 0 15579--15590, 2020

    2020

  45. [54]

    J. S. Smith, P. Cascante-Bonilla, A. Arbelle, D. Kim, R. Panda, D. Cox, D. Yang, Z. Kira, R. Feris, and L. Karlinsky. Construct-vl: Data-free continual structured vl concepts learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),...

    2023

  46. [55]

    J. S. Smith, L. Karlinsky, V. Gutta, P. Cascante-Bonilla, D. Kim, A. Arbelle, R. Panda, R. Feris, and Z. Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2023

  47. [56]

    W. Sun, Q. Li, J. Zhang, W. Wang, and Y.-a. Geng. Decoupling learning and remembering: A bilevel memory framework with knowledge projection for task-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20186--20195, 2023

    2023

  48. [57]

    S. Thrun. A Lifelong Learning Perspective for Mobile Robot Control. Elsevier, 1995

    1995

  49. [58]

    M. K. Titsias, J. Schwarz, A. G. d. G. Matthews, R. Pascanu, and Y. W. Teh. Functional regularisation for continual learning with gaussian processes. In Proceedings of the International Conference on Learning Representations (ICLR), 2020

    2020

  50. [59]

    C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. 2011

    2011

  51. [60]

    Y. Wang, Z. Huang, and X. Hong. S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning. Advances in Neural Information Processing Systems, 35: 0 5682--5695, 2022 a

    2022

  52. [61]

    Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y. Lee, X. Ren, G. Su, V. Perot, J. Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pages 631--648. Springer, 2022 b

    2022

  53. [62]

    Z. Wang, Z. Zhang, C.-Y. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139--149, 2022 c

    2022

  54. [63]

    Wortsman, V

    M. Wortsman, V. Ramanujan, R. Liu, A. Kembhavi, M. Rastegari, J. Yosinski, and A. Farhadi. Supermasks in superposition. In Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  55. [64]

    Xu and Z

    J. Xu and Z. Zhu. Reinforced continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2018

    2018

  56. [65]

    S. Yan, J. Xie, and X. He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014--3023, 2021

    2021

  57. [66]

    Zenke, B

    F. Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987--3995. PMLR, 2017

    2017

  58. [67]

    Zhang, J

    X. Zhang, J. Zhao, and Y. LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015

    2015

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.