Pith. sign in

REVIEW 4 major objections 4 minor 57 references

Block-based Symmetric Pruning and Fusion for Efficient Vision Transformers

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Joint query-and-key pruning plus similarity fusion raises DeiT-S accuracy to 81.8% while halving FLOPs.

desk verdict The 2D block-pruning/fusion idea is real and the ablation shows it helps, but the symmetric-attention speedup is mathematically false and the headline gains dissolve under apples-to-apples comparison. read the letter →

arxiv 2507.12125 v1 pith:5MWGBBNS submitted 2025-07-16 cs.CV

Yi-Kuan Hsieh , Jun-Wei Hsieh , Xin Li , Yu-Ming Chang , Yu-Chee Tseng This is my paper · ORCID
classification cs.CV
keywords visiontransformertokenpruningfusionblock-basedsymmetricattentionefficientinferenceImageNetclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that token reduction in vision transformers should be decided in two dimensions—both the query and key directions—rather than along one, and that pruned tokens should be folded into similar surviving tokens instead of being discarded. Its method, BSPF-ViT, splits the attention matrix into blocks, scores each entry with a learned 3x3 convolution that averages nearby attention values, keeps the top 50% of entries per block, and fuses the pruned entries into the most similar remaining query row. It also shares the query and key weight matrices so that the attention matrix is symmetric, allowing pruning to be computed on the upper triangle and mirrored to the lower triangle. The paper reports that this recipe raises ImageNet top-1 accuracy by 1.3% on DeiT-T and 2.0% on DeiT-S while cutting FLOPs by roughly half, and that similar gains occur when the method is attached to T2T-ViT, LV-ViT, and CaFormer-S18.

What carries the argument

The machinery is block-based 2D pruning on the attention matrix followed by similarity fusion, with a symmetry shortcut. Each block $A_{m,n}$ is smoothed by a learnable $3\times 3$ convolution (Eq. 3) so that a token's score includes its neighbors; the top 50% of entries are kept and the rest are marked pruned. Each pruned query row is matched to its most cosine-similar surviving row, and its pruned attention values are added into that row weighted by the cosine similarity times a Hamming similarity of the two pruning masks (Eq. 8). Because the model shares query and key weights, the paper treats the attention matrix as symmetric and performs the pruning score computation only on the upper triangle, mirroring the chosen positions to the lower triangle.

What would settle it

On a real DeiT-S block with shared query and key weights, feed any ImageNet batch, compute the full attention matrix, and compare off-diagonal pairs $A(i,j)$ and $A(j,i)$; if the two differ by more than a small tolerance on a substantial fraction of pairs, the upper-triangle mirroring shortcut is not exact, and the reported speedup would need to be re-evaluated as an approximation.

Watch

Extended reading notes

Core claim

The central claim is that joint, block-based pruning of both the query and key dimensions preserves the information that one-directional token pruning throws away, and that fusing pruned entries into matched neighbors can turn a pruned ViT into a strictly better classifier. Concretely, the paper claims that BSPF-ViT reaches 81.8% top-1 accuracy on DeiT-S (up from the 79.8% baseline) at 2.4 GFLOPs instead of 4.6 GFLOPs, and 73.5% on DeiT-T (up from 72.2%) at 0.6 GFLOPs instead of 1.3 GFLOPs. The same procedure is reported to improve T2T-ViT, LV-ViT, and CaFormer-S18 by 0.5–0.9% with a 10% extra speedup over MCTF, and to transfer to COCO object detection and instance segmentation through SHViT/Mask R-CNN.

Load-bearing premise

The load-bearing assumption is that making the query and key transformations share weights makes the attention matrix symmetric, so pruning computed on the upper triangle can be copied to the lower triangle exactly; if row-wise score normalization breaks this symmetry, the triangular speedup no longer performs the same pruning as the full matrix.

Editorial extensions

If this is right

  • If the reported numbers hold, aggressive token reduction no longer trades accuracy for speed: the pruned DeiT models beat their unpruned baselines while computing about half the attention.
  • The block-based, neighbor-aware scoring rule implies that pruning decisions should be made on smoothed attention patches, not on individual token scores, and that this smoothing is trainable.
  • Sharing query and key weights creates a symmetry that halves the pruning computation, so the speedup is tied to the model geometry rather than to hardware specifics.
  • The method's transfer to T2T-ViT, LV-ViT, CaFormer-S18, and COCO downstream tasks indicates that 2D block pruning and fusion is a generic add-on for vision transformers, not a fix for one architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension not pursued in the paper is to measure, with shared query and key weights, how much row-wise softmax actually breaks attention-matrix symmetry; if the break is small on natural images, the triangular shortcut is an accurate approximation, and if large, the shortcut changes which tokens survive.
  • The same block-pruning-plus-fusion recipe could translate to causal or language attention only with a directed treatment of the pruning mask, since the paper's symmetry argument relies on visual, bidirectional token influence.
  • Because the pruning ratio is fixed at 50% per block, a natural next step is an adaptive ratio per block or per layer, using the smoothed score distribution to decide how much of each block to drop; the paper itself mentions adaptive block sizes only as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes BSPF-ViT, a token-reduction method for vision transformers built on three components: chunk-based attention that restricts self-attention to blocks; block-based 2D pruning in which 50% of entries in each attention block are removed according to a locally smoothed score; and a similarity-based fusion step intended to transfer pruned information into retained tokens. The paper further claims that tying the query and key projection weights makes the attention matrix symmetric, allowing pruning decisions to be computed on the upper triangle and mirrored to the lower triangle. Experiments report accuracy gains on ImageNet for DeiT-T/S, T2T-ViT, LV-ViT, and CaFormer-S18, plus COCO detection/segmentation results.

Significance. If the headline results were measured against proper baselines and the symmetry argument were valid, the paper would offer a useful combination of local attention and 2D pruning with fusion. The experimental breadth is a genuine strength: the paper evaluates multiple architectures, downstream tasks, similarity metrics, pruning ratios, and the effect of neighborhood-aware pruning decisions. However, the central theoretical premise—that shared Q/K weights make the post-softmax attention matrix symmetric—is mathematically incorrect, the fusion equation as written cannot transfer pruned values, and the FLOPs claims are not compared apples-to-apples against the chunk-attention baseline. The stress-test concern therefore lands directly.

major comments (4)
  1. [Sec. 3.4, Eq. (1)] The symmetry argument is mathematically incorrect. With Wk = Wq, the pre-softmax score matrix S = X Wq Wq^T X^T is symmetric, but Eq. (1) applies a row-wise softmax: A(i,j) = exp(S_ij) / sum_k exp(S_ik), while A(j,i) = exp(S_ij) / sum_k exp(S_jk). These denominators differ, so A is not symmetric in general. Consequently the lower-triangular pruning positions cannot be obtained by mirroring upper-triangular pruning decisions, and the speedup rationale behind Fig. 8 and Algorithm 1 is unsupported. In addition, even if A were symmetric, the 3x3 convolution in Eq. (3) would preserve symmetry only if the learned kernel itself were symmetric, which is not stated.
  2. [Tables 1 and 8] The headline accuracy gains are not apples-to-apples. BSPF-ViT replaces standard full attention with chunk attention (Sec. 3.1), and Table 8 reports that chunk attention alone already reaches 81.4% top-1 at 2.4 GFLOPs on DeiT-S. The incremental contribution of block-based pruning and fusion over chunk attention is therefore 0.4 percentage points (81.8 vs. 81.4), not the 2.0 percentage points over the 79.8% full-attention baseline printed in Table 1. All comparisons in Table 1 should be made against the chunk-attention baseline, or the two effects should be decomposed explicitly. The same issue affects the DeiT-T and T2T-ViT rows.
  3. [Sec. 3.3, Eqs. (4)-(8), Algorithm 1] The fusion operation as specified cannot transfer pruned information. By Eq. (4), Ar and Ap are complementary masks: at any position (i,j), exactly one of Ar(i,j) or Ap(i,j) is nonzero. Line 14 of Algorithm 1 updates Ar only when Ar(ib,j)>0, which is precisely the case where Ap(ib,j)=0, so the pruned values are never added. Eq. (8) also writes to the ib-th row rather than to the original pruned row i, so the index mapping is unclear. The similarity-based fusion mechanism needs to be re-specified and its implementation clarified before the accuracy claims can be interpreted.
  4. [Secs. 3.1-3.2, Eq. (2)] The complexity analysis does not account for the cost of forming inter-chunk blocks. Chunk attention is O(N Omega) only when the C diagonal blocks are computed, but Sec. 3.2 defines A_{m,n} for arbitrary query chunk m and key chunk n (Eq. (2)) and then scores and prunes each block with a 3x3 convolution. With C^2 blocks of size Omega^2, this is O(N^2) work before any symmetry shortcut is applied. If the method actually computes only a sparse subset of inter-chunk blocks, that sparsity pattern must be stated and included in the FLOPs calculation; otherwise the reported 50% reduction in FLOPs relative to full self-attention is not justified.
minor comments (4)
  1. [Abstract vs. Sec. 1] The abstract states a 1.3% DeiT-T gain while Sec. 1 states 1.7%; these numbers should be reconciled.
  2. [Sec. 1, Fig. 3 caption] The text says 'Fig. 3 overviews the proposed VTPS-ViT method'; this appears to be a leftover name and should read BSPF-ViT.
  3. [Sec. 3.3, Eq. (7)] The phrase 'punning strings' should be 'pruning strings', and the definition of the length |Pi| for the Hamming distance should be stated explicitly.
  4. [Table 7] The heading contains the typos 'FLPOS' and 'ses'; the pruning-ratio scale should also be defined (e.g., fraction of entries retained or pruned) to match Sec. 3.2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: BSPF-ViT is an empirical comparison against external baselines; the Section 3.4 symmetry flaw is a correctness issue, not a circular reduction.

full rationale

BSPF-ViT is a purely empirical token-reduction method whose reported gains are comparisons against external baselines (DeiT, T2T-ViT, LV-ViT, SHViT on ImageNet and COCO). No step in the paper derives a result from a parameter fitted to that same result. The pruning ratio (0.5) is chosen by an ImageNet ablation (Table 7), but it is a hyperparameter of the method and is not relabeled as a prediction; the central comparison remains against independent published baselines. The symmetry argument in Section 3.4 is mathematically flawed rather than circular: with Wk=Wq the pre-softmax score S=XWqWq^TX^T is symmetric, but Eq. (1) applies row-wise softmax, so A(i,j)=exp(S_ij)/sum_k exp(S_ik) and A(j,i)=exp(S_ij)/sum_k exp(S_jk) have different denominators; hence A is not symmetric and the upper-triangle mirroring in Fig. 8 does not follow from Wk=Wq. That is an unsupported correctness claim about the mechanism, not a reduction of the paper's output to its input. There are no load-bearing self-citations, no imported uniqueness theorems, and no fitted parameter renamed as a prediction, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim relies on two problematic assumptions: the symmetry of the softmax attention matrix, which is false, and the implicit assumption that all off-diagonal attention blocks can be computed within the stated O(N*Omega) budget, which is unexplained.

free parameters (2)
  • Pruning ratio (keep ratio) = 0.5
    Selected through ImageNet ablation (Table 7) as the best accuracy/FLOPs trade-off; all headline results use this value.
  • Chunk size Omega (or number of chunks C) = not reported
    Defines the block structure in Section 3.1; the paper never states the value used in experiments, but it determines the FLOPs and the block pruning granularity.
assumptions (4)
  • ad hoc to paper Row-wise softmax of a symmetric score matrix yields a symmetric attention matrix
    Invoked in Section 3.4 to justify upper-triangular pruning; mathematically false, see weakest_assumption.
  • domain assumption Query and key projections can be shared without loss because in vision the influence of token i on j equals that of j on i
    Stated in Section 3.4 to motivate Wq=Wk; not validated.
  • domain assumption Adjacent attention entries are correlated in images, so a 3x3 local average of attention scores gives better pruning decisions
    Assumed in Section 3.2; supported only by a single ablation (Table 6) showing a 0.4 point gain.
  • ad hoc to paper All inter-chunk attention blocks A_{m,n} can be processed within the claimed O(N*Omega) complexity
    The paper claims chunk attention gives O(N*Omega) but describes forming C^2 blocks; the computational cost of computing and pruning off-diagonal blocks is not accounted for.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Block-based Symmetric Pruning and Fusion for Efficient Vision Transformers." pith.science (2026). https://pith.science/paper/5MWGBBNS

@misc{pith2026250712125,
  author       = {Pith},
  title        = {Pith review of: Block-based Symmetric Pruning and Fusion for Efficient Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5MWGBBNS}},
  note         = {Machine review of arXiv:2507.12125}
}
abstract

Vision Transformer (ViT) has achieved impressive results across various vision tasks, yet its high computational cost limits practical applications. Recent methods have aimed to reduce ViT's $O(n^2)$ complexity by pruning unimportant tokens. However, these techniques often sacrifice accuracy by independently pruning query (Q) and key (K) tokens, leading to performance degradation due to overlooked token interactions. To address this limitation, we introduce a novel {\bf Block-based Symmetric Pruning and Fusion} for efficient ViT (BSPF-ViT) that optimizes the pruning of Q/K tokens jointly. Unlike previous methods that consider only a single direction, our approach evaluates each token and its neighbors to decide which tokens to retain by taking token interaction into account. The retained tokens are compressed through a similarity fusion step, preserving key information while reducing computational costs. The shared weights of Q/K tokens create a symmetric attention matrix, allowing pruning only the upper triangular part for speed up. BSPF-ViT consistently outperforms state-of-the-art ViT methods at all pruning levels, increasing ImageNet classification accuracy by 1.3% on DeiT-T and 2.0% on DeiT-S, while reducing computational overhead by 50%. It achieves 40% speedup with improved accuracy across various ViTs.

Figures

Figures reproduced from arXiv: 2507.12125 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of ViT token reduction strategies: (a) The original attention map. (b) Result after 1D token pruning, where important information along the main diagonal is discarded (blue ovals). (c) Our proposed 2D block pruning effectively pre￾serves critical information, highlighted in red and blue ovals. cient token pruning. • Our method surpasses existing ViT token pruning ap￾proaches in performance, while signific… view at source ↗
Figure 3
Figure 3. The proposed block-based symmetric pruning and fusion for efficient ViT (BSPF-ViT) consists of three main steps: (a) Chunk-based attention (§ 3.1), which is modified from the traditional self-attention mechanism. (b) Block-based pruning (§ 3.2) is applied to the sparse regions, targeting sparse regions to determine which elements to retain or prune. (c) Matching and fusion (§ 3.3), where each row of the attention ma… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: (a) Chunk attention only considers self attention within [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: Each query finds its closest query and then contributes [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: An example calculating the pruning similarity. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: For a symmetric attention matrix, the pruning can be [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 41 canonical work pages

  1. [1]

    Learned queries for efficient local attention

    Moab Arar, Ariel Shamir, and Amit H Bermano. Learned queries for efficient local attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10841–10852, 2022. 2

    work page 2022

  2. [2]

    Multimae: Multi-modal multi-task masked autoen- coders

    Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi-task masked autoen- coders. In European Conference on Computer Vision, pages 348–367. Springer, 2022. 2

    work page 2022

  3. [3]

    Long- former: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Long- former: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. 1

    arXiv 2004

  4. [4]

    To- ken merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. To- ken merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. 2, 3, 6

    arXiv 2022

  5. [5]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European confer- ence on computer vision, pages 213–229. Springer, 2020. 1

    2020

  6. [6]

    Crossvit: Cross-attention multi-scale vision trans- former for image classification

    Chun-Fu (Richard) Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision trans- former for image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 357–366, 2021. 2

    work page 2021

  7. [7]

    Rethinking attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Do- han, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020. 2

    arXiv 2009

  8. [8]

    Twins: Revisiting the design of spatial attention in vision transformers

    Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haib- ing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in neural information processing systems, 34:9355–9366, 2021. 1, 2

    work page 2021

Show all 57 references
  1. [9]

    Cswin transformer: A general vision transformer backbone with cross-shaped windows

    Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CV...

    2022

  2. [10]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint a...

    2010 arXiv

  3. [11]

    RMT: Retentive Networks Meet Vision Trans- formers

    Qihang Fan, Huaibo Huang, Mingrui Chen, Hongmin Liu, and Ran He. RMT: Retentive Networks Meet Vision Trans- formers . In 2024 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR) , pages 5641–5651,

    2024

  4. [12]

    Adaptive token sampling for efficient vision transformers

    Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and J¨urgen Gall. Adaptive token sampling for efficient vision transformers. In European Conference on Computer Vision, pages 396–414...

    2022

  5. [13]

    Global context vision transformers

    Ali Hatamizadeh, Hongxu Yin, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Global context vision transformers. In International conference on machine learning, pages 12633– 12646. PMLR, 2023. 1, 2

    2023

  6. [14]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. 7, 8

    2017

  7. [15]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000– 16009, 2022. 2

    2022

  8. [16]

    Rethinking spa- tial dimensions of vision transformers

    Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spa- tial dimensions of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11936–11945, 2021. 2

    2021

  9. [17]

    Elsa: Ex- ploiting layer-wise n: M sparsity for vision transformer ac- celeration

    Ning-Chi Huang, Chi-Chih Chang, Wei-Cheng Lin, Endri Taka, Diana Marculescu, and Kai-Chiang Wu. Elsa: Ex- ploiting layer-wise n: M sparsity for vision transformer ac- celeration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8006–...

    2024

  10. [18]

    All tokens matter: Token labeling for training better vision transform- ers

    Zi-Hang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transform- ers. Advances in neural information processing systems, 34: 18590–18602, 2021. 2, 6

    2021

  11. [19]

    Chunkformer: Learning long time series with multi-stage chunked transformer.arXiv preprint arXiv:2112.15087, 2021

    Yue Ju, Alka Isac, and Yimin Nie. Chunkformer: Learning long time series with multi-stage chunked transformer.arXiv preprint arXiv:2112.15087, 2021. 4

    2021 arXiv

  12. [20]

    Token fusion: Bridging the gap between token pruning and token merging

    Minchul Kim, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. Token fusion: Bridging the gap between token pruning and token merging. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1383–1392, 2024. 3, 6

    2024

  13. [21]

    Reformer: The efficient transformer

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020. 1, 2

    2001 arXiv

  14. [22]

    Spvit: Enabling faster vision transformers via latency-aware soft token pruning

    Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In European conference on computer vision, pages 620–640. Springer, 2022. 2, 3, 6

    2022

  15. [23]

    Multi-criteria token fusion with one-step-ahead attention for efficient vision transformers

    Sanghyeok Lee, Joonmyung Choi, and Hyunwoo J Kim. Multi-criteria token fusion with one-step-ahead attention for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15741–15750, 2024. 2, 3, 6, 7

    2024

  16. [24]

    Re- 9 thinking vision transformers for mobilenet size and speed

    Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Re- 9 thinking vision transformers for mobilenet size and speed. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16889–16900, 2023. 8

    2023

  17. [25]

    Not all patches are what you need: Expediting vision transformers via token reorganiza- tions

    Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganiza- tions. arXiv preprint arXiv:2202.07800, 2022. 2, 3, 6

    2022 arXiv

  18. [26]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  19. [27]

    Scale-aware modulation meet transformer

    Weifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang, and Lian- wen Jin. Scale-aware modulation meet transformer. In 2023 IEEE/CVF International Conference on Computer Vi- sion (ICCV), pages 5992–6003, 2023. 1, 2

    2023

  20. [28]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 1, 2

    2021

  21. [29]

    Beyond attentive tokens: Incorporating to- ken importance and diversity for efficient vision transform- ers

    Sifan Long, Zhen Zhao, Jimin Pi, Shengsheng Wang, and Jingdong Wang. Beyond attentive tokens: Incorporating to- ken importance and diversity for efficient vision transform- ers. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 10334–10343,

  22. [30]

    Mega: moving average equipped gated atten- tion

    Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: moving average equipped gated atten- tion. arXiv preprint arXiv:2209.10655, 2022. 1, 4

    2022 arXiv

  23. [31]

    Token pooling in vision transformers for image classification

    Dmitrii Marin, Jen-Hao Rick Chang, Anurag Ranjan, An- ish Prabhu, Mohammad Rastegari, and Oncel Tuzel. Token pooling in vision transformers for image classification. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pages 12–21, 2023. 2, 3

    2023

  24. [32]

    Adavit: Adaptive vision transformers for efficient image recognition

    Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12309–12318, 2022. 1, 3

    2022

  25. [33]

    Ia-red: Interpretability-aware redundancy reduction for vision transformers

    Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogerio Feris, and Aude Oliva. Ia-red: Interpretability-aware redundancy reduction for vision transformers. Advances in Neural Information Processing Systems, 34:24898–24911, 2021. 1, 3

    2021

  26. [34]

    Interpretability-aware redundancy re- duction for vision transformers, 2023

    Bowen Pan, Rameswar Panda, Rogerio Schmidt Feris, and Aude Jeanne Oliva. Interpretability-aware redundancy re- duction for vision transformers, 2023. US Patent App. 17/559,053. 6

    2023

  27. [35]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  28. [36]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937–13949,

  29. [37]

    TransNeXt: Robust Foveal Visual Perception for Vision Transformers

    Dai Shi. TransNeXt: Robust Foveal Visual Perception for Vision Transformers . In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 17773–17783, 2024. 1, 3

    2024

  30. [38]

    Segmenter: Transformer for semantic segmenta- tion

    Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmenta- tion. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 7262–7272, 2021. 1

    2021

  31. [39]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. In International conference on machine learning , pages 10347–10357. PMLR, 2021. 1, 2, 6, 7

    2021

  32. [40]

    Going deeper with im- age transformers

    Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Herve Jegou. Going deeper with im- age transformers. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pages 32–42, 2021. 1, 2

    2021

  33. [41]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1

    2023 arXiv

  34. [42]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 4

    2017

  35. [43]

    Zero- tprune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers

    Hongjie Wang, Bhishma Dedhia, and Niraj K Jha. Zero- tprune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 16070–16079, 2024. 3, 6

    2024

  36. [44]

    Linformer: Self-attention with linear complexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. 1, 2

    2006 arXiv

  37. [45]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision , p...

    2021

  38. [46]

    Joint token pruning and squeezing towards more aggressive compression of vision transformers

    Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jia- jun Liang. Joint token pruning and squeezing towards more aggressive compression of vision transformers. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2092–2101, 2023. 2, 3

    2023

  39. [47]

    Denoising masked autoencoders help ro- bust classification

    QuanLin Wu, Hang Ye, Yuntian Gu, Huishuai Zhang, Liwei Wang, and Di He. Denoising masked autoencoders help ro- bust classification. arXiv preprint arXiv:2210.06983, 2022. 2

    2022 arXiv

  40. [48]

    Segformer: Simple and efficient design for semantic segmentation with transform- ers

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transform- ers. Advances in neural information processing systems, 34: 12077–12090, 2021. 1 10

    2021

  41. [49]

    Nystr¨omformer: A nystr¨om-based algorithm for approximat- ing self-attention

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystr¨omformer: A nystr¨om-based algorithm for approximat- ing self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 14138–14148, 2021. 2

    2021

  42. [50]

    Evo-vit: Slow-fast token evolution for dynamic vision transformer

    Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2964–2972, 2022. 6

    2022

  43. [51]

    Global vision trans- former pruning with hessian-aware saliency

    Huanrui Yang, Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision trans- former pruning with hessian-aware saliency. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18547–18557, 2023. 3

    2023

  44. [52]

    A-vit: Adaptive to- kens for efficient vision transformer

    Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive to- kens for efficient vision transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10809–10818, 2022. 1, 3, 6

    2022

  45. [53]

    Tokens-to-token vit: Training vision transformers from scratch on imagenet

    Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF in- ternational conference on computer vision , page...

  46. [54]

    Shvit: Single-head vision transformer with memory efficient macro design

    Seokju Yun and Youngmin Ro. Shvit: Single-head vision transformer with memory efficient macro design. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5756–5767, 2024. 2

    2024

  47. [55]

    Shifted chunk transformer for spatio-temporal representa- tional learning

    Xuefan Zha, Wentao Zhu, Lv Xun, Sen Yang, and Ji Liu. Shifted chunk transformer for spatio-temporal representa- tional learning. Advances in Neural Information Processing Systems, 34:11384–11396, 2021. 4

    2021

  48. [56]

    Scaling vision transformers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lu- cas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12104–12113, 2022. 2

    2022

  49. [57]

    Deformable detr: Deformable trans- formers for end-to-end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable trans- formers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020. 1 11

    2010 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.