The reviewed record of science sign in
Pith

arxiv: 2606.27372 · v1 · pith:4TLZJRLW · submitted 2026-06-25 · cs.CV

DnA: Denoising Attention for Visual Tasks

Ron Campos , Subhajit Maity , Xin Li , Srijan Das , Aritra Dutta This is my paper

Reviewed by Pith2026-06-26 05:02 UTCgrok-4.3pith:4TLZJRLWopen to challenge →

classification cs.CV
keywords denoising attentionmulti-head attentionvision transformersImageNet classificationvideo understandingsubspace separationprincipal anglessoftmax alternative
0
0 comments X

The pith

DnA improves visual attention by projecting positive and negative query interactions into two distinct subspaces with larger principal angles.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Denoising Attention (DnA) to address noisy patterns produced by standard softmax in multi-head attention for visual models. A positive query locates features belonging to the correct class while a negative query flags closely related but irrelevant features; these are then mapped into separate subspaces chosen to maximize principal angles between them. The authors report that this separation yields an absolute 0.8 percent accuracy lift on ImageNet-1K with a ViT-B backbone, plus 1.8 percent on video transformer tasks and 0.5 percent on video LLMs. A sympathetic reader would care because attention noise is a recurring bottleneck in transformer-based vision systems, and the method keeps the same backbone while altering only the attention computation. Extensive ablations are presented to justify the two-subspace design and the denoising effect.

Core claim

DnA modifies multi-head attention so that positive queries identify correct-class image features and negative queries identify closely associated but irrelevant features; the resulting interactions are projected into two distinct subspaces engineered to have larger principal angles, which the authors state promotes subspace separation and improved discriminability. On a ViT-B backbone this produces an absolute gain of 0.8 percent on ImageNet-1K, 1.8 percent on video-understanding tasks with video transformers, and 0.5 percent on video LLMs.

What carries the argument

Denoising Attention (DnA) mechanism that projects positive and negative query interactions into two distinct subspaces with larger principal angles

If this is right

  • Attention maps exhibit reduced noise and higher feature discriminability across image and video inputs.
  • Classification accuracy rises on ImageNet-1K without changing model size or training regime.
  • Video transformer and video LLM benchmarks record consistent positive gains, with the largest lift on video tasks.
  • Empirical analyses confirm that the two-subspace construction and the denoising step are both necessary for the observed improvements.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same positive-negative query split could be inserted into attention layers of non-vision transformers to test whether subspace separation helps language or multimodal models.
  • If the principal-angle construction is the active ingredient, replacing it with other geometric separation objectives might produce comparable or larger gains on the same benchmarks.
  • The reported improvements might allow smaller backbones to reach accuracy levels previously requiring larger models, which would be testable by scaling down ViT size while keeping DnA.
  • The approach might interact with existing regularization techniques such as attention dropout, an interaction the paper does not explore.

Load-bearing premise

That mapping the interactions into subspaces with larger principal angles will produce better separation and therefore higher discriminability.

What would settle it

Running the same ViT-B experiments on ImageNet-1K and finding that accuracy does not rise by the reported 0.8 percent or that measured principal angles show no correlation with the accuracy difference.

Figures

Figures reproduced from arXiv: 2606.27372 by Aritra Dutta, Ron Campos, Srijan Das, Subhajit Maity, Xin Li.

Figure 1
Figure 1. Figure 1: The original image from ImageNet-1K [18] shows the primary object, breast￾plate, along with secondary adversarial objects, person, helmet, etc. The traditional attention [20, 73] and differential attention [80] project the interactions onto a single V subspace, resulting in misallocation of attention to secondary objects. Our proposed DnA explicitly models positive and negative interactions using two sets … view at source ↗
Figure 2
Figure 2. Figure 2: Original images (center) from ImageNet-1K, their attention activation maps for traditional softmax only showing positive interactions (left), and our denoising attention, DnA (A Q±V ± h ) (right) showing both positive and negative interactions. (a) The softmax attention is dispersed and does not focus on the concerned object (hare), while our DnA focuses on the concerned object. (b) The softmax attention i… view at source ↗
Figure 3
Figure 3. Figure 3: A schematic comparison between traditional attention and the proposed de￾noising attention. (i) The multi-head attention uses a softmax activation on the rows of the scaled dot-product QhK⊤ √ h d and combines it with Vh. (ii) The proposed denoising attention A Q±V ± h , uses two sets of values, V + h and V − h , and two sets of queries Q + h and Q − h . We use softmax and softmin two scaled dot-products, Q… view at source ↗
Figure 4
Figure 4. Figure 4: (a) Top: Attention visualization for ViT-B [20] with softmax, differential at￾tention [80], and our DnA. The objects are dumbbells and a volleyball. (b) Bottom: Example of DnA outperforming baseline methods on egocentric video QA. Four frames are uniformly sampled; the object of interest is highlighted by a cyan box. The correct answer is in green, the incorrect in red. We initialize the positive projectio… view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of intruder dimension counts between DnA and differential atten￾tion ϵ = cos π 3 , k=10 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Average pairwise cosine distances under varying noise levels; see noise variance (σ) and PSNR in titles. Solid lines represent models evaluated under clean data (DnA and softmax), while dashed lines reflect artifacts of noisy data (DnAσ and softmaxσ ) for different noise, σ. DnAσ exhibits smaller accuracy drops than softmaxσ across all noise levels, indicating more diverse feature representations and robus… view at source ↗
Figure 8
Figure 8. Figure 8: Training loss, validation loss, and validation accuracy of ViT-B using [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Visualization of the inference results of ViT-Det on a ViT-B backbone (a) using DnA (top row) and (b) using softmax attention with the corresponding ground truths (bottom row); zoom in for a better view. For each sample, the left image is the prediction and the right image is the ground truth. C.4 Qualitative Analysis on Detection & Segmentation Figure 9a displays the qualitative performance of using DnA i… view at source ↗
Figure 10
Figure 10. Figure 10: Statistics of αh across different denoising designs. Each row represents the mean, maximum, minimum, and standard deviation of the learnable parameter, αh, for each layer of the network. actions. (e) Finally, the proposed DnA, A Q±V ± h , outperforms all baselines as it can model token interactions better with separate value subspaces for positive and negative interactions. (B) Is the performance gain of … view at source ↗
Figure 11
Figure 11. Figure 11: Mean head redundancy measured by pairwise cosine distance between atten￾tion weights across heads. Higher distance indicates lower redundancy. line. This suggests that different heads in the negative branch learn to identify distinct noise patterns. After layer 8, the negative branch becomes more redun￾dant than the baseline, indicating that once noise suppression is complete, the heads become more simila… view at source ↗
Figure 12
Figure 12. Figure 12: Attention visualization for ViT-B [20] using softmax attention, differential attention [80], and our DnA. From top to bottom, the objects are: cassette, chain link, cocktail shaker, Egyptian cat, swimming cap, and ladybug [PITH_FULL_IMAGE:figures/full_fig_p035_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Examples of DnA outperforming baseline methods on egocentric video QA. For each example, frames are uniformly sampled, with the object of interest in a cyan box. The correct answer is in green, the incorrect in red. observe a larger range between the values of αh, and a mean that exhibits less linear progression. Lastly, for Aˆ± h , we see that for the first 8 layers, the minimum αh actually takes on a ne… view at source ↗
Figure 14
Figure 14. Figure 14: Layer-wise entropy of attention weights for ViT-B using softmax attention (QK⊤) and both DnA attention branches (Q +K⊤ and Q −K⊤) on the ImageNet-1K validation set. Solid lines show mean entropy across all attention heads. Dashed lines represent the entropy of the Top-k% most attended tokens. coder layers. Attention entropy, measured as the Shannon entropy, H (see Theo￾rem 1), of the attention weights, qu… view at source ↗
read the original abstract

The softmax activation in multihead attention (MHA) is the de facto standard for attention-based models in visual perception tasks. However, standard softmax can produce noisy attention patterns that dilute relevant features and degrade its performance. In this paper, we propose Denoising Attention or DnA, in which, first, a positive query identifies which image features belong to the correct class, and a negative query identifies closely associated but irrelevant image features. DnA then projects these interactions into two distinct subspaces with larger principal angles, promoting subspace separation and improved discriminability. Using a ViT-B backbone, our proposed DnA achieves an absolute gain of 0.8% on ImageNet-1K compared to the baseline. We further show improvements across multiple visual understanding tasks, including video understanding with video transformers (1.8%) and video LLMs (0.5%). Our extensive empirical analyses justify the design choices involving two interacting subspaces and the denoising effect of DnA.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 0 minor

Summary. The paper proposes Denoising Attention (DnA) to address noisy attention patterns from standard softmax in multi-head attention for visual tasks. It introduces positive and negative queries to identify relevant and irrelevant image features, then projects their interactions into two distinct subspaces chosen for larger principal angles to promote separation and discriminability. Empirical claims include an absolute 0.8% gain on ImageNet-1K using a ViT-B backbone, 1.8% improvement on video understanding with video transformers, and 0.5% on video LLMs, supported by extensive empirical analyses justifying the two-subspace design.

Significance. If the reported accuracy gains prove reproducible and the subspace-separation mechanism is shown to be causal rather than incidental, DnA could represent a lightweight modification to attention that improves discriminability across image and video tasks. The absence of any derivation, principal-angle measurements, or controlled ablations in the provided description limits the assessed significance; no machine-checked proofs, open code, or parameter-free derivations are mentioned.

major comments (3)
  1. [Abstract] Abstract: The central claim attributes performance gains to projection into subspaces with larger principal angles promoting separation and discriminability, yet supplies no derivation, bound, measurement of principal angles versus standard MHA, or controlled experiment isolating this mechanism from ancillary changes (extra queries, altered computation). This assumption is load-bearing for explaining the 0.8% ImageNet-1K gain.
  2. [Abstract] Abstract: No ablation of the two-subspace design, error bars, or training protocol is described, preventing verification that the reported gains (0.8% ImageNet-1K, 1.8% video transformers, 0.5% video LLMs) arise from the proposed denoising effect rather than capacity or implementation differences.
  3. [Abstract] Abstract: The construction and training of the positive query (for correct-class features) and negative query (for irrelevant but associated features) are not specified, leaving the method's implementation details and reproducibility unclear despite being foundational to the subspace projection step.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive comments on our manuscript. Below we respond point-by-point to the major comments, clarifying details present in the full paper and indicating revisions we will make to improve clarity and reproducibility.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim attributes performance gains to projection into subspaces with larger principal angles promoting separation and discriminability, yet supplies no derivation, bound, measurement of principal angles versus standard MHA, or controlled experiment isolating this mechanism from ancillary changes (extra queries, altered computation). This assumption is load-bearing for explaining the 0.8% ImageNet-1K gain.

    Authors: The full manuscript motivates the subspace projection empirically through analyses of attention patterns and discriminability, rather than a theoretical derivation or bound. We agree that direct measurements of principal angles and a controlled ablation isolating the projection step would strengthen the causal link to the observed gains. We will add these measurements (comparing DnA subspaces to standard MHA) and the requested ablation in the revised manuscript. revision: yes

  2. Referee: [Abstract] Abstract: No ablation of the two-subspace design, error bars, or training protocol is described, preventing verification that the reported gains (0.8% ImageNet-1K, 1.8% video transformers, 0.5% video LLMs) arise from the proposed denoising effect rather than capacity or implementation differences.

    Authors: The manuscript contains extensive empirical analyses and ablations on the two-subspace design and denoising effect. However, we acknowledge that error bars and explicit training protocol details are not sufficiently highlighted. We will include error bars on all reported results and expand the training protocol description in the revision to facilitate verification. revision: yes

  3. Referee: [Abstract] Abstract: The construction and training of the positive query (for correct-class features) and negative query (for irrelevant but associated features) are not specified, leaving the method's implementation details and reproducibility unclear despite being foundational to the subspace projection step.

    Authors: The method section details the construction of the positive and negative queries, including their initialization, training objectives to identify correct-class versus associated irrelevant features, and integration with the subspace projection. To address the concern about the abstract, we will add a concise description of query construction in the abstract and introduction of the revised version. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical gains reported independently of asserted mechanism

full rationale

The paper defines DnA as a projection step into subspaces with larger principal angles and states that this promotes separation and discriminability, then reports measured accuracy improvements on ImageNet-1K and other tasks. No equations, fitted parameters, or self-citations are shown that reduce the reported gains to the assumption by construction. The gains are presented as external empirical outcomes rather than derived quantities, satisfying the default expectation of non-circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no free parameters, axioms, or invented entities can be extracted or audited.

pith-pipeline@v0.9.1-grok · 5702 in / 1071 out tokens · 26705 ms · 2026-06-26T05:02:21.061126+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

91 extracted references · 9 canonical work pages · 3 internal anchors

  1. [1]

    https://ai.meta.com/blog/meta-llama-3/ (2024)

    AI, M.: Introducing Meta Llama 3: The most capable openly available LLM to date. https://ai.meta.com/blog/meta-llama-3/ (2024)

    2024

  2. [2]

    In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision

    Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., Schmid, C.: ViViT: A Video Vision Transformer. In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision. pp. 6836–6846 (2021)

    2021

  3. [3]

    In: International Conference on Learning Representa- tions (2015)

    Bahdanau, D., Cho, K., Bengio, Y.: Neural Machine Translation by Jointly Learn- ing to Align and Translate. In: International Conference on Learning Representa- tions (2015)

    2015

  4. [4]

    Bertasius, G., Wang, H., Torresani, L.: Is Space-Time Attention All You Need for Video Understanding? In: International Conference on Machine Learning. pp. 833–842 (2021)

    2021

  5. [5]

    In: Advances in Neural Information Processing Systems

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language Models are Few- Shot Learners. In: Advances in Neural Information Processing Systems. pp. 1877– 1901 (2020)

    1901

  6. [6]

    Cadenhead, E., McGee, C., Li, X., Bergou, E.H., Dutta, A.: Beyond LoRA: Is Sparsity-Induced Adaptation Better? arXiv preprint arXiv:2606.13767 (2026)

    work page arXiv 2026

  7. [7]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

    Campos, R., Vayani, A., Kulkarni, P.P., Gupta, R., Zafar, A., Dutta, A., Shah, M.: GAEA: A Geolocation Aware Conversational Model. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 5236– 5246 (2026)

    2026

  8. [8]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Carreira, J., Zisserman, A.: Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6299–6308 (2017)

    2017

  9. [9]

    In: Proceedings of the ACM SIGFIDET (Now SIGMOD) Workshop on Data De- scription, Access and Control

    Chamberlin, D.D., Boyce, R.F.: SEQUEL: A Structured English Query Language. In: Proceedings of the ACM SIGFIDET (Now SIGMOD) Workshop on Data De- scription, Access and Control. pp. 249–264 (1974)

    1974

  10. [10]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chefer, H., Gur, S., Wolf, L.: Transformer Interpretability Beyond Attention Visu- alization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 782–791 (2021)

    2021

  11. [11]

    In: Annual Meeting of the Association for Computational Linguistics

    Chen, C., Yin, Y., Shang, L., Jiang, X., Qin, Y., Wang, F., Wang, Z., Chen, X., Liu, Z., Liu, Q.: bert2BERT: Towards Reusable Pretrained Language Models. In: Annual Meeting of the Association for Computational Linguistics. pp. 2134–2148 (2022)

    2022

  12. [12]

    In: International Conference on Learning Representations (2016)

    Chen, T., Goodfellow, I., Shlens, J.: Net2net: Accelerating Learning via Knowledge Transfer. In: International Conference on Learning Representations (2016)

    2016

  13. [13]

    In: International Conference on Learning Representations (2023)

    Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., Kolesnikov, A., Puigcerver, J., Ding, N., Rong, K., Akbari, H., Mishra, G., Xue, L., Thapliyal, A.V., Bradbury, J., Kuo, W., Seyedhosseini, M., Jia, C., Ayan, B.K., Ruiz, C.R., Steiner, A.P., An- gelova, A., Zhai, X., Houlsby, N...

    2023

  14. [14]

    International Economic Review44(2), 435–465 (2003)

    Christiano, L.J., Fitzgerald, T.J.: The Band Pass Filter. International Economic Review44(2), 435–465 (2003)

    2003

  15. [15]

    John Wiley & Sons, 2 edn

    Cover, T.M., Thomas, J.A.: Elements of Information Theory. John Wiley & Sons, 2 edn. (2006) Denoising Attention 17

    2006

  16. [16]

    In: International Conference on Learning Representations (2024)

    Darcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision Transformers Need Reg- isters. In: International Conference on Learning Representations (2024)

    2024

  17. [17]

    In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision

    Das, S., Dai, R., Koperski, M., Minciullo, L., Garattoni, L., Bremond, F., Francesca, G.: Toyota Smarthome: Real-World Activities of Daily Living. In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision. pp. 833– 842 (2019)

    2019

  18. [18]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2009)

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A Large- Scale Hierarchical Image Database. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2009)

    2009

  19. [19]

    In: Proceedings of the European Conference on Computer Vision

    Ding,M.,Xiao,B.,Codella,N.,Luo,P.,Wang,J.,Yuan,L.:DaViT:DualAttention Vision Transformers. In: Proceedings of the European Conference on Computer Vision. pp. 74–92 (2022)

    2022

  20. [20]

    In: International Conference on Learning Representations (2021)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In: International Conference on Learning Representations (2021)

    2021

  21. [21]

    In: International Conference on Machine Learning

    d’Ascoli, S., Touvron, H., Leavitt, M.L., Morcos, A.S., Biroli, G., Sagun, L.: Con- ViT: Improving Vision Transformers with Soft Convolutional Inductive Biases. In: International Conference on Machine Learning. pp. 2286–2296 (2021)

    2021

  22. [22]

    MIT Press (2016)

    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016)

    2016

  23. [23]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Grauman, K., Westbury, A., Torresani, L., Kitani, K., Malik, J., Afouras, T., Ashutosh, K., Baiyya, V., Bansal, S., Boote, B., Byrne, E., Chavis, Z., Chen, J., Cheng, F., Chu, F.J., Crane, S., Dasgupta, A., Dong, J., Escobar, M., Forigua, C., Gebreselasie, A., Haresh, S., Huang, J., Islam, M.M., Jain, S., Khirodkar, R., Kukreja, D., Liang, K.J., Liu, J.W....

    2024

  24. [24]

    In: Conference of the North American Chapter of the Association for Computational Linguistics

    Guo, Q., Qiu, X., Liu, P., Shao, Y., Xue, X., Zhang, Z.: Star-Transformer. In: Conference of the North American Chapter of the Association for Computational Linguistics. pp. 1315–1325 (2019)

    2019

  25. [25]

    In: Advances in Neural Information Processing Systems

    Han, D., Pu, Y., Xia, Z., Han, Y., Pan, X., Li, X., Lu, J., Song, S., Huang, G.: Bridging the Divide: Reconsidering Softmax and Linear Attention. In: Advances in Neural Information Processing Systems. pp. 79221–79245 (2024)

    2024

  26. [26]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2961–2969 (2017)

    2017

  27. [27]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016) 18 R. Campos et al

    2016

  28. [28]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al.: The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8340–8349 (2021)

    2021

  29. [29]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., Song, D.: Natural Adversarial Examples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15262–15271 (2021)

    2021

  30. [30]

    IEEE Transactions on Signal Processing64(8), 1933–1945 (2016)

    Huang, J., Qiu, Q., Calderbank, R.: The Role of Principal Angles in Subspace Classification. IEEE Transactions on Signal Processing64(8), 1933–1945 (2016)

    1933

  31. [31]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Hyeon-Woo, N., Yu-Ji, K., Heo, B., Han, D., Oh, S.J., Oh, T.H.: Scratching Visual Transformer’s Back with Uniform Attentionz. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5807–5818 (2023)

    2023

  32. [32]

    In: International Conference on Machine Learning

    Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., Carreira, J.: Per- ceiver: General Perception with Iterative Attention. In: International Conference on Machine Learning. pp. 4651–4664 (2021)

    2021

  33. [33]

    Physical Review 106(4), 620–630 (1957)

    Jaynes, E.T.: Information Theory and Statistical Mechanics. Physical Review 106(4), 620–630 (1957)

    1957

  34. [34]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment Anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4015–4026 (2023)

    2023

  35. [35]

    In: Proceedings of the Conference on Empirical Methods in Natural Language Processing

    Kobyzev,I.,Ghaddar,A.,Hu,D.,Chen,B.:IntegralTransformer:DenoisingAtten- tion, Not Too Much Not Too Little. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 2337–2354 (2025)

    2025

  36. [36]

    In: Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision

    Koohpayegani, S.A., Pirsiavash, H.: SimA: Simple Softmax-free Attention for Vi- sion Transformers. In: Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision. pp. 2607–2617 (2024)

    2024

  37. [37]

    In: Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing

    Kovaleva, O., Romanov, A., Rogers, A., Rumshisky, A.: Revealing the Dark Secrets of BERT. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing. pp. 4365–4374 (2019)

    2019

  38. [38]

    In: IEEE Workshop on 3D Representation and Recognition

    Krause, J., Deng, J., Stark, M., Fei-Fei, L.: Collecting a Large-Scale Dataset of Fine-Grained Cars. In: IEEE Workshop on 3D Representation and Recognition. Sydney, Australia (2013)

    2013

  39. [39]

    Krizhevsky, A.: Learning Multiple Layers of Features from Tiny Images. Tech. rep. (2009)

    2009

  40. [40]

    (ed.): Comprehensive Dictionary of Electrical Engineering

    Laplante, P.A. (ed.): Comprehensive Dictionary of Electrical Engineering. CRC Press, 2 edn. (2018)

    2018

  41. [41]

    Laurençon, H., Tronchon, L., Sanh, V.: What matters when building vision- language models? In: Advances in Neural Information Processing Systems. pp. 87874–87907 (2024)

    2024

  42. [42]

    In: Annual Meeting of the Association for Computational Linguistics

    Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoy- anov, V., Zettlemoyer, L.: BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In: Annual Meeting of the Association for Computational Linguistics. pp. 7871–7880 (2020)

    2020

  43. [43]

    Li, J., Li, D., Savarese, S., Hoi, S.: BLIP-2: Bootstrapping Language-Image Pre- trainingwithFrozenImageEncodersandLargeLanguageModels.In:International Conference on Machine Learning. pp. 19730–19742 (2023)

    2023

  44. [44]

    In: Proceedings of the European Conference on Com- puter Vision

    Li, Y., Mao, H., Girshick, R., He, K.: Exploring Plain Vision Transformer Back- bones for Object Detection. In: Proceedings of the European Conference on Com- puter Vision. pp. 280–296. Springer (2022) Denoising Attention 19

    2022

  45. [45]

    In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition

    Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature Pyramid Networks for Object Detection. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition. pp. 2117–2125 (2017)

    2017

  46. [46]

    In: Proceedings of the European Conference on Computer Vision

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common Objects in Context. In: Proceedings of the European Conference on Computer Vision. pp. 740–755 (2014)

    2014

  47. [47]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin Trans- former: Hierarchical Vision Transformer using Shifted Windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10012–10022 (2021)

    2021

  48. [48]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., Hu, H.: Video Swin Trans- former. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3202–3211 (2022)

    2022

  49. [49]

    In: Advances in Neural Information Processing Systems

    Lu, J., Yao, J., Zhang, J., Zhu, X., Xu, H., Gao, W., Xu, C., Xiang, T., Zhang, L.: SOFT: Softmax-free Transformer with Linear Complexity. In: Advances in Neural Information Processing Systems. pp. 21297–21309 (2021)

    2021

  50. [50]

    arXiv preprint arXiv:2411.07176 (2024)

    Lv, A., Xie, R., Li, S., Liao, J., Sun, X., Kang, Z., Wang, D., Yan, R.: More Ex- pressive Attention with Negative Weights. arXiv preprint arXiv:2411.07176 (2024)

    work page arXiv 2024

  51. [51]

    arXiv preprint arXiv:2212.12552 (2022)

    Ma, X., Wang, H., Qin, C., Li, K., Zhao, X., Fu, J., Fu, Y.: A Close Look at Spatial Modeling: From Attention to Convolution. arXiv preprint arXiv:2212.12552 (2022)

    work page arXiv 2022

  52. [52]

    Maity, S., Hitsman, K., Li, X., Dutta, A.: Kolmogorov-Arnold Attention: Is Learn- able Attention Better For Vision Transformers? arXiv preprint arXiv:2503.10632v2 (2025)

    work page arXiv 2025

  53. [53]

    In: International Conference on Learning Representations (2025)

    Meng, W., Luo, Y., Li, X., Jiang, D., Zhang, Z.: PolaFormer: Polarity-aware Lin- ear Attention for Vision Transformersn. In: International Conference on Learning Representations (2025)

    2025

  54. [54]

    https://ai.meta.com/blog/llama-3- 2-connect-2024-vision-edge-mobile-devices/ (2024)

    Meta AI: Llama 3.2: Vision and edge models. https://ai.meta.com/blog/llama-3- 2-connect-2024-vision-edge-mobile-devices/ (2024)

    2024

  55. [55]

    Transac- tions on Machine Learning Research (2026)

    Nair, P.: Softmax is1/2-Lipschitz: A Tight Bound Across Allℓp Norms. Transac- tions on Machine Learning Research (2026)

    2026

  56. [56]

    In: Advances in Neural Information Processing Systems

    Nguyen, T., Nguyen, T., Do, H., Nguyen, K., Saragadam, V., Pham, M., Nguyen, K.D., Ho, N., Osher, S.: Improving Transformer with an Admixture of Attention Heads. In: Advances in Neural Information Processing Systems. pp. 27937–27952 (2022)

    2022

  57. [57]

    In: Advances in Neural Information Processing Systems

    Nguyen, T., Pham, M., Nguyen, T., Nguyen, K., Osher, S., Ho, N.: FourierFormer: Transformer Meets Generalized Fourier Integral Theorem. In: Advances in Neural Information Processing Systems. pp. 29319–29335 (2022)

    2022

  58. [58]

    In: Advances in Neural Information Processing Systems

    Nguyen, T., Suliafu, V., Osher, S., Chen, L., Wang, B.: FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field attention. In: Advances in Neural Information Processing Systems. pp. 29449–29463 (2021)

    2021

  59. [59]

    In: IEEE International Conference on Acoustics, Speech and Signal Processing

    Nguyen, T.M., Nguyen, T., Bui, L., Do, H., Nguyen, D.K., Le, D.D., Tran-The, H., Ho, N., Osher, S.J., Baraniuk, R.G.: A Probabilistic Framework for Pruning Transformers Via a Finite Admixture of Keys. In: IEEE International Conference on Acoustics, Speech and Signal Processing. pp. 1–5 (2023)

    2023

  60. [60]

    Transactions on Machine Learning Research (2024) 20 R

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.Y., Li, S.W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: DINOv2: Learning robust visual feat...

    2024

  61. [61]

    In: International Conference on Machine Learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning Transferable Visual Models From Natural Language Supervision. In: International Conference on Machine Learning. pp. 8748–8763 (2021)

    2021

  62. [62]

    Journal of Machine Learning Research21(140), 1–67 (2020)

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the Limits of Transfer Learning with a Unified Text- to-Text Transformer. Journal of Machine Learning Research21(140), 1–67 (2020)

    2020

  63. [63]

    arXiv preprint arXiv:2406.19391 (2024)

    Rahimian, A.K., Govind, M.K., Maity, S., Reilly, D., Kümmerle, C., Das, S., Dutta, A.: Fibottention: Inceptive Visual Representation Learning with Diverse Attention Across Heads. arXiv preprint arXiv:2406.19391 (2024)

    work page arXiv 2024

  64. [64]

    In: Proceedings of the European Conference on Computer Vision (2026)

    Reilly, D., Govind, M.K., Xue, L., Das, S.: From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision. In: Proceedings of the European Conference on Computer Vision (2026)

    2026

  65. [65]

    In: Proceedings of the European Conference on Computer Vision (2026)

    Reilly, D., Govind, M.K., Xue, L., Das, S.: VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models. In: Proceedings of the European Conference on Computer Vision (2026)

    2026

  66. [66]

    arXiv preprint arXiv:2410.18613 (2024)

    Saratchandran, H., Zheng, J., Ji, Y., Zhang, W., Lucey, S.: Rethinking At- tention: Polynomial Alternatives to Softmax in Transformers. arXiv preprint arXiv:2410.18613 (2024)

    work page arXiv 2024

  67. [67]

    In: Proceedings of the ACM SIGMOD International Conference on Management of Data

    Selinger, P.G., Astrahan, M.M., Chamberlin, D.D., Lorie, R.A., Price, T.G.: Access Path Selection in a Relational Database Management System. In: Proceedings of the ACM SIGMOD International Conference on Management of Data. pp. 23—-34 (1979)

    1979

  68. [68]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Shahroudy, A., Liu, J., Ng, T.T., Wang, G.: NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysist. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1010–1019 (2016)

    2016

  69. [69]

    In: International Conference on Machine Learning

    Shi, H., Gao, J., Ren, X., Xu, H., Liang, X., Li, Z., Kwok, J.T.Y.: SparseBERT: Rethinking the Importance Analysis in Self-attention. In: International Conference on Machine Learning. pp. 9547–9557 (2021)

    2021

  70. [70]

    Shuttleworth, R., Andreas, J., Torralba, A., Sharma, P.: LoRA vs Full Fine-tuning: AnIllusionofEquivalence.In:AdvancesinNeuralInformationProcessingSystems. pp. 174627–174662 (2025)

    2025

  71. [71]

    Association for Computing Machinery Computing Surveys55(6), 1–28 (2022)

    Tay, Y., Dehghani, M., Bahri, D., Metzler, D.: Efficient Transformers: A Survey. Association for Computing Machinery Computing Surveys55(6), 1–28 (2022)

    2022

  72. [72]

    Gemini: A Family of Highly Capable Multimodal Models

    Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.B., Yu, J., Soricut, R., Schalk- wyk, J., Dai, A.M., Hauth, A., et al.: Gemini: A Family of Highly Capable Multi- modal Models. arXiv preprint arXiv:2312.11805 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  73. [73]

    In: International Conference on Machine Learning

    Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: International Conference on Machine Learning. pp. 10347–10357 (2021)

    2021

  74. [74]

    In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision

    Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., Jégou, H.: Going Deeper With Image Transformers. In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision. pp. 32–42 (2021)

    2021

  75. [75]

    In: Advances in Neural Infor- mation Processing Systems (2017)

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez,A.N.,Łukasz Kaiser, Polosukhin, I.: Attention Is All You Need. In: Advances in Neural Infor- mation Processing Systems (2017)

    2017

  76. [76]

    Linformer: Self-Attention with Linear Complexity

    Wang, S., Li, B., Khabsa, M., Fang, H., Ma, H.: Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 (2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2006

  77. [77]

    In: Advances in Neural Information Processing Systems (2019) Denoising Attention 21

    Xu, J., Sun, X., Zhang, Z., Zhao, G., Lin, J.: Understanding and Improving Layer Normalization. In: Advances in Neural Information Processing Systems (2019) Denoising Attention 21

    2019

  78. [78]

    Nature Communications16(1), 5509 (2025)

    Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Chen, C., Li, H., Zhao, W., He, Z., Chen, Q., Zhou, R., Zou, Z., Zhang, H., Hu, S., Zheng, Z., Zhou, J., Cai, J., Han, X., Zeng, G., Li, D., Liu, Z., Sun, M.: Efficient GPT-4V level multimodal large language model for deployment on edge devices. Nature Communications16(1), 5509 (2025)

    2025

  79. [79]

    In: International Conference on Learning Representations (2025)

    Ye, J., Xu, H., Liu, H., Hu, A., Yan, M., Qian, Q., Zhang, J., Huang, F., Zhou, J.: mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models. In: International Conference on Learning Representations (2025)

    2025

  80. [80]

    In: International Conference on Learning Representations (2025)

    Ye, T., Dong, L., Xia, Y., Sun, Y., Zhu, Y., Huang, G., Wei, F.: Differential Trans- former. In: International Conference on Learning Representations (2025)

    2025

Showing first 80 references.