Pith. sign in

REVIEW 4 major objections 4 minor 32 references

FCL-ViT: Task-Aware Attention Tuning for Continual Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Two-phase attention tuning beats CL rivals without replay memory

desk verdict The architecture is real and the TSB ablation is convincing, but the SOTA claim compares a task-aware method to class-incremental baselines, and that mismatch is load-bearing. read the letter →

arxiv 2412.02509 v3 pith:ZV4NS6RU submitted 2024-12-03 cs.AI

Anestis Kaimakamidis , Ioannis Pitas This is my paper · ORCID
classification cs.AI
keywords continuallearningvisiontransformerattentiontuningcatastrophicforgettingelasticweightconsolidationrehearsal-freetask-specificfeaturesfeedbackmechanism
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that continual learning can be handled without replay memory by making a Vision Transformer's attention itself task-aware. The proposed FCL-ViT runs every image through the same frozen Transformer twice: the first pass produces generic features, small Task-Specific Blocks convert those features into attention-steering vectors, and the second pass uses cross-attention to produce a representation tuned to the current task. The authors report that this design, regularized by Elastic Weight Consolidation, achieves top Last-task accuracy on ImageNet-100 with 10 tasks and stable average accuracy on CIFAR-100 across 10, 20, and 50 tasks while training only about 14.23 million parameters and storing no old examples. If true, the central payoff is a rehearsal-free continual classifier whose forgetting is controlled by attention steering rather than by memory or parameter expansion.

What carries the argument

The load-bearing identity is the two-phase attention replay over one frozen Transformer. In phase 1, patch embeddings pass through $d$ Tunable self-Attention Blocks in self-attention mode, giving $\mathbf{r} = \mathbf{f}_d$. Task-Specific Blocks map $\mathbf{r}$ through stacked linear-dropout layers into steering vectors $\mathbf{g}_j$, one per depth. In phase 2 the same blocks switch to cross-attention, with $\mathbf{Q}_j = \mathbf{W}_{qj}\mathbf{f}_{j-1}$ and $\mathbf{K}_j = \mathbf{W}_{kj}\mathbf{g}_j$, $\mathbf{V}_j = \mathbf{W}_{vj}\mathbf{g}_j$, so each layer's attention is steered by task-specific information. The trainable parameters are the TSB matrices $\mathbf{H}^1_j, \mathbf{H}^2_j$ and per-task linear classifiers; Elastic Weight Consolidation penalizes changes to important TSB weights using the diagonal Fisher Information Matrix. This is what lets the architecture adapt the same frozen backbone to each new task instead of expanding or replaying.

What would settle it

Retrain iCaRL, WA, DER, and DyTox+ on the same frozen ViT-Base backbone and training schedule as FCL-ViT and compare Last accuracy; if FCL-ViT no longer leads under matched backbone capacity, the central superiority claim collapses. A second check: replace the TSB steering vectors with a fixed linear projection of phase-1 features; if accuracy stays high, the task-specific cross-attention mechanism is not doing the load-bearing work.

Watch

Extended reading notes

Core claim

The central claim is that catastrophic forgetting can be substantially reduced by giving a frozen Vision Transformer a feedback loop that retunes attention to the task at hand. FCL-ViT's 12 Tunable self-Attention Blocks (TABs) stay frozen; during phase 1 they act as ordinary self-attention and yield generic feature vector $\mathbf{r}$. Task-Specific Blocks (TSBs), each a two-linear-layer module with dropout, read $\mathbf{r}$ and emit per-block steering vectors $\mathbf{g}_j$. In phase 2 the same TABs run again as cross-attention layers: the query comes from the current layer's input, the key and value come from $\mathbf{g}_j$, and the final vector $\mathbf{z}$ is what the per-task classifier sees. Training updates only the TSB weights and the current task's linear classifier, with an Elastic Weight Consolidation penalty that protects TSB parameters important for earlier tasks. The paper reports Last-task Top-1 accuracy of 71.80% on ImageNet-100 (10 tasks), above DyTox+ (69.10%) and DER (66.70%), and stable Last accuracy of 65.02%, 67.61%, and 67.72% on CIFAR-100 for 10, 20, and 50 tasks, all without rehearsal memory; removing the TSBs drops ImageNet-100 Last accuracy to 53.75%.

Load-bearing premise

The load-bearing premise is that the competing methods' benchmark scores were measured with backbones of comparable size to FCL-ViT's frozen ViT-Base, so the reported gains reflect the attention-tuning mechanism rather than a larger pretrained backbone.

Editorial extensions

If this is right

  • No-exemplar continual learning becomes competitive with replay-based methods: FCL-ViT reports Last accuracy 71.80% on ImageNet-100 10-task and stable 65-68% on CIFAR-100 across 10/20/50 tasks without storing any previous samples.
  • Task count can grow without architectural expansion: the 12 frozen backbone blocks and the TSBs are reused across all tasks; each new task adds only one linear classifier, so trainable-parameter growth is much smaller than in dynamic-expansion methods like DER.
  • The TSB mechanism, not the regularizer alone, is responsible for the forgetting control: removing the TSBs and fine-tuning the whole ViT with only EWC drops ImageNet-100 Last accuracy from 71.80% to 53.75%.
  • The method's accuracy is unusually stable as tasks increase from 10 to 50 on CIFAR-100 (Last 65.02% to 67.61% to 67.72%), whereas comparator methods degrade, so its advantage is expected to be largest in long task sequences.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the backbone is frozen and steering is a generic cross-attention operation, the same two-phase design should transfer to other Transformer backbones (smaller or larger ViTs, Swin-style models), but the paper only evaluates ViT-Base.
  • Editorial inference: the reported roughly 92% inference slowdown means the second pass is a real deployment cost; a natural extension would train a lightweight head to predict the TSB steering vectors directly from phase-1 features, collapsing the two passes into one.
  • Editorial inference: the BLAZE wildfire experiment shows a single domain-shift sequence, so the no-rehearsal claim has not yet been stress-tested on longer or more heterogeneous task orders; a multi-domain curriculum would be a harder test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes FCL-ViT, a Vision Transformer architecture for continual learning that combines a frozen ViT backbone (TABs) with trainable Task-Specific Blocks (TSBs) and Elastic Weight Consolidation (EWC). Inference proceeds in two phases: a generic feature extraction pass and a task-specific pass in which TSB outputs are injected through cross-attention into the TABs. A new linear classifier is added per task, and no rehearsal memory is used. The method is evaluated on ImageNet-100 and CIFAR-100 under several task splits, compared with iCaRL, WA, DER, and DyTox+, and ablated by removing the TSBs and by varying the EWC strength. The paper claims that FCL-ViT surpasses state-of-the-art continual learning performance while retaining few trainable parameters.

Significance. If the claims are substantiated, FCL-ViT is a useful contribution to rehearsal-free continual learning: the architecture is simple, the trainable parameter count is low, and the core ablation (with vs. without TSBs) shows a large improvement (71.80 vs. 53.75 Last accuracy on ImageNet-100). The paper also reports stable performance across increasing numbers of tasks, which is a desirable property. However, the headline state-of-the-art claim is currently not supported by the experimental comparison because of a protocol mismatch and unverified baseline comparability. The mechanism itself is plausible, but the empirical evidence needs to be placed on a sound footing. No equation is fitted to the target results, so circularity is not a concern; the main risks are experimental comparability and missing details.

major comments (4)
  1. [§3.1–3.2, Tables 1–2] The evaluation protocol is task-incremental for FCL-ViT but class-incremental for all quoted baselines. Section 3.1 states that the model is evaluated 'using the task index i', and Section 3.2 adds a Linear Block per task and selects the classifier via the task-specific head ŷ_k = H̃_k z. By contrast, iCaRL, WA, DER, and DyTox+ operate without task IDs and must discriminate among all classes seen so far. FCL-ViT only needs to discriminate within the current task, which is an easier problem. The reported Last accuracy comparisons are therefore not apples-to-apples, and the abstract's 'surpasses state-of-the-art' claim is not supported by Tables 1 and 2. The authors should either evaluate FCL-ViT in a class-incremental protocol or compare against task-aware baselines and explicitly restrict the claim to task-incremental learning.
  2. [§4.1, Tables 1–2] All baseline numbers are imported from [7] without disclosing the backbone used for each method. FCL-ViT uses a ViT-Base backbone with embedding dimension D=768. If iCaRL, WA, DER, or DyTox+ used smaller backbones (e.g., ViT-Small), the reported gains could be due to backbone capacity rather than the proposed TSB/TAB mechanism. The #TP column counts only trainable parameters and does not account for the frozen backbone's capacity. Please report the backbone specification for every baseline and, ideally, rerun at least DyTox+ with the same ViT-Base backbone so that the comparison is controlled for architecture capacity.
  3. [§4.2] The ablation that removes TSBs and trains all ViT weights with EWC shows a drop from 71.80 to 53.75 on ImageNet-100. This demonstrates that TSBs help relative to that particular full-fine-tuning baseline, but it does not show that the two-phase mechanism improves over a much simpler task-aware linear probe on the frozen ViT-Base features with per-task heads. Since the TABs are frozen and the per-task LB already provides task-specific classification, adding a linear-probe baseline is necessary to attribute the performance to attention tuning rather than to the quality of the frozen backbone features. This baseline is absent and is load-bearing for the paper's explanatory claim.
  4. [§3.2 and §4.1] Several experimental details required for reproducibility are missing: the value of α in Eq. (5) is never reported; the EWC Fisher Information Matrix computation is unspecified (number of samples used, diagonal vs. full approximation, and whether it is recomputed after each task); and the ImageNet-100 results in Table 1 have no variance or number of runs. Given the small differences in some comparisons (e.g., CIFAR-100 10-task Last: 65.02 vs. 65.22), these omissions prevent verification of the claimed improvements.
minor comments (4)
  1. [§3.2, Eq. (4)] The sentence defining θ_i,j and θ_{i-1,j} says they are trained for tasks T_i and T_{i+1}, respectively, which is inconsistent with the penalty term (θ_i,j,l − θ_{i−1,j,l})² that compares consecutive tasks; please correct to T_i and T_{i−1}.
  2. [Abstract and §1] The abstract says TABs and TSBs 'operate in both phases and are responsible for tuning the TABs attention'; this is ambiguous because only the TSBs tune the TAB attention. Please reword for clarity.
  3. [§4.1, Table 2 and Figure 2] The text says FCL-ViT 'outperforms other models for all CIFAR100 splits', but Table 2 shows that DER has higher Last accuracy on the 10-task split (65.22 vs. 65.02); please reconcile the claim with the table.
  4. [§4.3] The reported inference slowdown of 'approximately 92% slower' is given without measurement details; please state the hardware, batch size, and evaluation setup used for timing.

Circularity Check

0 steps flagged · score 0.0 of 10

The paper's performance claims are empirical measurements of an independent architecture; no derivation step reduces to its own inputs by construction or by self-citation.

full rationale

The paper's central claim is that the proposed FCL-ViT architecture achieves competitive or state-of-the-art continual learning accuracy with few trainable parameters. This claim rests on direct experimental measurements in Tables 1 and 2, ablation studies in Section 4.2, and hyperparameter sensitivity analysis in Figure 5. No equation in the paper is fitted against the reported target metric, and no 'prediction' is derived from a parameter that was itself fit to that same target. The only external results imported are benchmark numbers from DyTox [7], which are used as baselines for comparison rather than as premises of an argument, so that import does not constitute circularity. The discussion in Section 4.3 is explanatory and post hoc, but it is not used to generate the experimental numbers. The reader-identified concern that FCL-ViT uses task IDs and per-task linear blocks while quoted baselines may be class-incremental is a correctness or comparability issue about whether the comparison is fair, not a circularity issue: the paper's own architecture is still evaluated empirically. Similarly, the choice of a pretrained ViT-Base backbone is an experimental design choice, not a fitted input masquerading as a prediction. There are no self-citations that carry the load of the argument, no uniqueness theorem imported from the authors' prior work, and no ansatz that is smuggled in via citation. Accordingly, the honest finding is no significant circularity, with a score of 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 2 invented entities

The central claim rests on the specific TSB/TAB design and on EWC as a forgetting regularizer. The experimental comparison imports baseline numbers without disclosing backbone configurations, and several hyperparameters are fixed by search or left unspecified, so the reader must trust the authors' configuration choices.

free parameters (6)
  • EWC penalty lambda = best 100-200
    Ablated in Fig. 5; value materially changes stability-plasticity balance.
  • loss weight alpha = not reported
    Appears in Eq. 5 but no value is given in experiments.
  • dropout rate p = 0.5
    Chosen for TSBs; no ablation.
  • learning rate = 1e-3
    Selected by hyperparameter search.
  • batch size = 128
    Selected by hyperparameter search.
  • epochs = 100
    Fixed for all tasks; no schedule reported.
assumptions (5)
  • standard math Softmax attention and transformer equations
    Section 3.2 uses standard transformer formulations.
  • domain assumption Class-disjoint tasks and task index available at test
    Section 3.1 states classes do not overlap and evaluation uses the task index.
  • domain assumption Pretrained ViT features are good generic representations
    Section 3.2 initializes TAB weights from public pretrained weights and freezes them.
  • domain assumption EWC Gaussian posterior approximation is adequate
    Section 3.2 relies on Fisher Information Matrix and multivariate Gaussian approximation.
  • ad hoc to paper Baseline benchmark numbers from [7] are directly comparable
    Section 4.1 takes iCaRL, WA, DER, DyTox+ numbers from DyTox paper without disclosing backbones.
invented entities (2)
  • Tunable self-Attention Blocks (TABs)
    purpose: Frozen transformer blocks that run in two modes: self-attention for generic features and cross-attention with TSB outputs for task-tuned features.
    Architectural component defined in this paper; no external implementation or artifact released for verification.
  • Task Specific Blocks (TSBs)
    purpose: Small trainable linear blocks with dropout that generate per-depth key/value vectors from generic features to steer attention.
    Architectural component defined in this paper; no external implementation or artifact released for verification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FCL-ViT: Task-Aware Attention Tuning for Continual Learning." pith.science (2026). https://pith.science/paper/ZV4NS6RU

@misc{pith2026241202509,
  author       = {Pith},
  title        = {Pith review of: FCL-ViT: Task-Aware Attention Tuning for Continual Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZV4NS6RU}},
  note         = {Machine review of arXiv:2412.02509}
}
read the original abstract

Continual Learning (CL) involves adapting the prior Deep Neural Network (DNN) knowledge to new tasks, without forgetting the old ones. However, modern CL techniques focus on provisioning memory capabilities to existing DNN models rather than designing new ones that are able to adapt according to the task at hand. This paper presents the novel Feedback Continual Learning Vision Transformer (FCL-ViT) that uses a feedback mechanism to generate real-time dynamic attention features tailored to the current task. The FCL-ViT operates in two Phases. In phase 1, the generic image features are produced and determine where the Transformer should attend on the current image. In phase 2, task-specific image features are generated that leverage dynamic attention. To this end, Tunable self-Attention Blocks (TABs) and Task Specific Blocks (TSBs) are introduced that operate in both phases and are responsible for tuning the TABs attention, respectively. The FCL-ViT surpasses state-of-the-art performance on Continual Learning compared to benchmark methods, while retaining a small number of trainable DNN parameters.

Figures

Figures reproduced from arXiv: 2412.02509 by the authors.

Figure 1
Figure 1. FCL-ViT architecture. 3. FCL-ViT Methodology 3.1. Problem Setup The CL goal is to progressively update a single DNN image classification model to accommodate new classes as they become available. In a series of classification tasks 𝑇1 , 𝑇2 ,…, 𝑇𝑡 , task dataset 𝑖 = { (𝐱 𝑖 𝑗 , 𝑦𝑖 𝑗 ) }𝑁 𝑗=1 comprises 𝑁𝑖 samples, and 𝐱 𝑖 𝑗 and 𝑦 𝑖 𝑗 denote the 𝑗-th image and its corresponding label. One task may comprise data coming … view at source ↗
Figure 2
Figure 2. The Top-1 classification accuracy (%) is reported after learning each task. FCL-ViT outperforms other models for all CIFAR100 splits. which is the desired result from a CL method, as it measures the final average accuracy of all tasks. However, it presents a bit lower “Avg" classification accuracy than two of the competing methods. This means that these methods lever￾aged better the training dataset throughout the t… view at source ↗
Figure 4
Figure 4. FCL-ViT classification accuracy on the Imagenet-100 dataset with and without the TSBs. decreases. On the other hand, the highest 𝜆 values constrain the gradients during learning. Therefore, FCL-ViT fails to achieve high classification performance on new tasks, while retaining knowledge of previous tasks. The best FCL-ViT classification performances were attained with intermediate 𝜆 values ( 𝜆 = 100 and 𝜆 = 200) that… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Effect of hyperparameter 𝜆 on FCL-ViT classification accuracy 5. Conclusion In conclusion, this work introduces FCL-ViT, a robust Vision Transformer-based framework that addresses the unique challenges of Continual Learning (CL) through dynamic task-specific attention …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 20 canonical work pages

  1. [7]

    Dytox: Transformers for continual learning with dynamic token expansion, in: Proceedings IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp

    Douillard, A., Ramé, A., Couairon, G., Cord, M., 2022. Dytox: Transformers for continual learning with dynamic token expansion, in: Proceedings IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9285–9295

    work page 2022

  2. [1]

    End-to-end incremental learning, in: Proceedings of the European Conference on Computer Vision (ECCV), pp

    Castro, F.M., Marín-Jiménez, M.J., Guil, N., Schmid, C., Alahari, K., 2018. End-to-end incremental learning, in: Proceedings of the European Conference on Computer Vision (ECCV), pp. 233–248

    work page 2018

  3. [2]

    Coatnet: Marrying convolution and attention for all data sizes

    Dai, Z., Liu, H., Le, Q.V., Tan, M., 2021. Coatnet: Marrying convolution and attention for all data sizes. Advances in neural information processing systems 34, 3965–3977

    work page 2021

  4. [3]

    Imagenet: A large-scale hierarchical image database, in: 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Ieee

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L., 2009. Imagenet: A large-scale hierarchical image database, in: 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Ieee. pp. 248–255

    work page 2009

  5. [4]

    Bert:Pre-trainingofdeepbidirectionaltransformers for language understanding

    Devlin,J.,2018. Bert:Pre-trainingofdeepbidirectionaltransformers for language understanding. arXiv preprint arXiv:1810.04805

    arXiv 2018

  6. [5]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N., 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    arXiv 2020

  7. [6]

    Pod- net: Pooled outputs distillation for small-tasks incremental learning, in: Proceedings of the European Conference on Computer Vision (ECCV), pp

    Douillard, A., Cord, M., Ollion, C., Robert, T., Valle, E., 2020. Pod- net: Pooled outputs distillation for small-tasks incremental learning, in: Proceedings of the European Conference on Computer Vision (ECCV), pp. 86–102

    work page 2020

  8. [8]

    Catastrophic forgetting in connectionist net- works

    French, R.M., 1999. Catastrophic forgetting in connectionist net- works. Trends in cognitive sciences 3, 128–135

    work page 1999

Show all 32 references
  1. [9]

    Video action transformer network, in: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp

    Girdhar, R., Carreira, J., Doersch, C., Zisserman, A., 2019. Video action transformer network, in: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 244–253

    2019

  2. [10]

    Levit:avisiontransformerinconvnet’sclothing for faster inference, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp

    Graham, B., El-Nouby, A., Touvron, H., Stock, P., Joulin, A., Jégou, H.,Douze,M.,2021. Levit:avisiontransformerinconvnet’sclothing for faster inference, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12259–12269

    2021

  3. [11]

    Asurveyonvisualtransformer

    Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao,A.,Xu,C.,Xu,Y.,etal.,2020. Asurveyonvisualtransformer. arXiv preprint arXiv:2012.12556

    2020 arXiv

  4. [12]

    Hassani, A., Walton, S., Shah, N., Abuduweili, A., Li, J., Shi, H.,

  5. [13]

    Distillingtheknowledgeina neural network

    Hinton,G.,Vinyals,O.,Dean,J.,2015. Distillingtheknowledgeina neural network. arXiv preprint arXiv:1503.02531

    2015 arXiv

  6. [14]

    Learning a unifiedclassifierincrementallyviarebalancing,in:Proceedingsofthe IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp

    Hou, S., Pan, X., Loy, C.C., Wang, Z., Lin, D., 2019. Learning a unifiedclassifierincrementallyviarebalancing,in:Proceedingsofthe IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 831–839

    2019

  7. [15]

    Khan, S., Naseer, M., Hayat, M., Zamir, S.W., Khan, F.S., Shah, M.,

  8. [16]

    Overcoming catastrophic forgetting in neural net- works

    Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G.,Rusu,A.A.,Milan,K.,Quan,J.,Ramalho,T.,Grabska-Barwinska, A., et al., 2017. Overcoming catastrophic forgetting in neural net- works. Proceedings of the national academy of sciences 114, 3521– 3526

    2017

  9. [17]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al., 2009. Learning multiple layers of features from tiny images

    2009

  10. [18]

    Learning without forgetting

    Li, Z., Hoiem, D., 2017. Learning without forgetting. IEEE transac- tions on pattern analysis and machine intelligence 40, 2935–2947

    2017

  11. [19]

    Preserving earlier knowledge in continual learning with the help of all previous feature extractors

    Li, Z., Zhong, C., Liu, S., Wang, R., Zheng, W.S., 2021. Preserving earlier knowledge in continual learning with the help of all previous feature extractors. arXiv preprint arXiv:2104.13614

    2021 arXiv

  12. [20]

    icarl: Incremental classifier and representation learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp

    Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H., 2017. icarl: Incremental classifier and representation learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2001–2010

    2017

  13. [21]

    Overcoming catastrophicforgettingwithhardattentiontothetask,in:International conference on machine learning, PMLR

    Serra, J., Suris, D., Miron, M., Karatzoglou, A., 2018. Overcoming catastrophicforgettingwithhardattentiontothetask,in:International conference on machine learning, PMLR. pp. 4548–4557

    2018

  14. [22]

    Toast:Transferlearning via attention steering

    Shi,B.,Gai,S.,Darrell,T.,Wang,X.,2023. Toast:Transferlearning via attention steering. arXiv preprint arXiv:2305.15542

    2023 arXiv

  15. [23]

    Attention is all you need

    Vaswani, A., 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  16. [24]

    Scaling local self-attention for parameter effi- cientvisualbackbones,in:ProceedingsoftheIEEE/CVFConference on Computer Vision and Pattern Recognition (CVPR), pp

    Vaswani, A., Ramachandran, P., Srinivas, A., Parmar, N., Hechtman, B., Shlens, J., 2021. Scaling local self-attention for parameter effi- cientvisualbackbones,in:ProceedingsoftheIEEE/CVFConference on Computer Vision and Pattern Recognition (CVPR), pp. 12894– 12904

    2021

  17. [25]

    Dualprompt: Complementary prompting for rehearsal-free continual learning, in: European Conference on Computer Vision (ECCV), Springer

    Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.Y., Ren, X., Su, G., Perot, V., Dy, J., et al., 2022a. Dualprompt: Complementary prompting for rehearsal-free continual learning, in: European Conference on Computer Vision (ECCV), Springer. pp. 631–648

  18. [26]

    Learning to prompt for continual learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp

    Wang, Z., Zhang, Z., Lee, C.Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., Pfister, T., 2022b. Learning to prompt for continual learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 139–149

  19. [27]

    Co-scale conv-attentional image transformers, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp

    Xu, W., Xu, Y., Chang, T., Tu, Z., 2021. Co-scale conv-attentional image transformers, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9981–9990

    2021

  20. [28]

    Der: Dynamically expandable repre- sentation for class incremental learning, in: Proceedings IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp

    Yan, S., Xie, J., He, X., 2021. Der: Dynamically expandable repre- sentation for class incremental learning, in: Proceedings IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3014–3023

    2021

  21. [29]

    Tokens-to-token vit: Training vision trans- formers from scratch on imagenet, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp

    Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z.H., Tay, F.E., Feng, J., Yan, S., 2021. Tokens-to-token vit: Training vision trans- formers from scratch on imagenet, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 558–567

    2021

  22. [30]

    Maintaining discriminationandfairnessinclassincrementallearning,in:Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp

    Zhao, B., Xiao, X., Gan, G., Zhang, B., Xia, S.T., 2020. Maintaining discriminationandfairnessinclassincrementallearning,in:Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13208–13217. : Preprint submitted to Elsevier Page 8 of 8

    2020

  23. [2021]

    arXiv preprint arXiv:2104.05704

    Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:2104.05704 . : Preprint submitted to Elsevier Page 7 of 8

    arXiv

  24. [2022]

    ACM computing surveys (CSUR) 54, 1–41

    Transformers in vision: A survey. ACM computing surveys (CSUR) 54, 1–41

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.