Pith. sign in

REVIEW 4 major objections 6 minor 51 references

ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that grouping low-attention image patches into coherent chunks before pruning or merging—called ImagePiece—makes DeiT-S run 54% faster while improving ImageNet accuracy by 0.39%.

desk verdict The retokenization idea is neat and the compatibility results are useful, but the paper overclaims the accuracy gain because it never isolates the conv stem from the retokenization mechanism. read the letter →

arxiv 2412.16491 v1 pith:PU42ZCGM submitted 2024-12-21 cs.CV

Seungdong Yoa , Seungjun Lee , Hyeseung Cho , Bumsoo Kim , Woohyung Lim This is my paper · ORCID
classification cs.CV
keywords imagerecognitionvisiontransformerefficientinferencetokenpruningmergingretokenizationNetclassificationattentionscoring
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ImagePiece claims that the standard way of cutting tokens in vision transformers—pruning low-attention patches or merging similar patches—works on the wrong unit: an individual image patch often carries no meaning by itself, so it is discarded or smoothed away before it can contribute. The paper proposes treating image patches like subwords in NLP and regrouping the least-attended tokens into locally coherent chunks, using a MaxMatch-style procedure lifted from WordPiece. Those chunks are then scored again against the class token, and only the ones that have become meaningful survive. This re-tokenization, together with a lightweight overlapping-convolution stem, is claimed to make DeiT-S 54% faster on ImageNet while raising its top-1 accuracy by 0.39%, and to give an 8% accuracy margin over prior token-reduction baselines in hyper-speed regimes. If true, it means token reduction for ViTs should be a content-aware re-segmentation problem, not just a selection or averaging problem.

What carries the argument

The mechanism is a three-stage retokenization loop supervised by class-token attention, preceded by a local coherence bias module. The local coherence bias replaces the ViT patchify stem with overlapping 3x3 convolutions and a 1x1 convolution, raising the first-layer similarity among nearby low-attention tokens from 0.5293 to 0.8091 (Tab. 8). Then MaxMatch-style bottom-k grouping: from the bottom 30% of tokens by [CLS]-attention, tokens are divided into two groups A and B by alternating rank order; each A-token is bipartite-soft-matched to its most similar B-token and averaged into an abstraction. The merged tokens are re-scored, and those that have become attentive are retained while the rest are pruned (bottom 20%). This machinery is compatible as a drop-in replacement: applying ImagePiece to DynamicViT, EViT, and ToMe raises their ImageNet accuracy by 0.79% on average.

What would settle it

Train a DeiT-S variant that uses the same local coherence bias stem and the same training recipe, but keeps ordinary patch tokens and performs no ImagePiece grouping or pruning; if its ImageNet-1k top-1 accuracy matches or exceeds 80.22%, the central accuracy claim of the paper collapses to an architecture change.

Watch

Extended reading notes

Core claim

The central claim is that the standard ViT patch tokenizer produces tokens that are not atomic semantics: many patches are meaningless in isolation, so pruning or merging them prematurely destroys information. ImagePiece addresses this by repeatedly applying a three-step loop: score each token by its attention to the class token, take the bottom 30% of tokens, split them into two alternating groups, and merge each token from one group with its most similar partner in the other; then recompute attention on the merged 'abstractions' and discard those that remain inattentive. This lets the model delay its pruning decision until tokens have had a chance to become meaningful when grouped with local neighbors, and it preserves highly attentive tokens from being diluted by similarity-based merging. On ImageNet-1k with DeiT-S, the method reports 80.22% top-1 accuracy (up from 79.83% for DeiT-S) at 3891.9 img/s (up from 2531.1), a 54% speedup; with DeiT-Ti it reports 72.61% (up from 72.13%) at 9450.2 img/s. At a 2.51x speedup it claims to lose only 6.28% accuracy, while DynamicViT, EViT, and ToMe lose 20.05%, 13.99%, and 9.24% respectively.

Load-bearing premise

The load-bearing assumption is that the reported accuracy improvement over DeiT-S comes from the retokenization logic, but the model also changes the input stem to overlapping convolutions; without a control that uses the new stem but no retokenization, the gains could be architectural rather than due to token grouping.

Editorial extensions

If this is right

  • Token pruning in ViTs becomes safer: decisions to discard tokens are postponed until low-attention patches have been grouped with their local neighbors and re-scored, so fewer prematurely discarded tokens lose useful context.
  • ImagePiece is a plug-in rather than a replacement: attaching it to DynamicViT, EViT, and ToMe improves their ImageNet accuracy by 0.79% on average, so existing efficient-ViT methods can be upgraded without rewriting their pruning or merging logic.
  • At extreme compression (26 output tokens, 13% of the original 197), ImagePiece preserves accuracy comparable to baselines that keep two to three times as many tokens, pointing to substantially cheaper deployment for batch inference.
  • The local coherence bias stem is a separate, transferable ingredient: it raises first-layer similarity among low-attention tokens from 0.5293 to 0.8091, which can be adopted by other token-reduction schemes to improve merge quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported accuracy gain over DeiT-S rests on a single training run; a multi-seed replication would establish whether the +0.39% is systematic or partly random variation, since the paper reports no variance.
  • The retokenize-then-rescore loop is not specific to images; the same logic could be applied to other transformer inputs whose elementary tokens are context-dependent, such as point clouds or audio frames, with a locality bias appropriate to each domain.
  • Because the bottom-30% fraction and pruning ratio are fixed hyperparameters, an adaptive policy that varies how many tokens are grouped per image based on its attention distribution could push the speed-accuracy frontier further; the paper does not explore this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ImagePiece, a retokenization strategy for Vision Transformers that groups inattentive (bottom-k) tokens into semantically meaningful chunks via bipartite soft matching, reassesses their [CLS]-attention after merging, and prunes tokens that remain unattentive. To make nearby non-semantic patches more similar, the standard patchify stem is replaced by an overlapping-convolution local coherence bias module. On ImageNet-1k, the authors report that with DeiT-S the method reaches 80.22% top-1 accuracy at 3891.9 img/s, versus the DeiT-S baseline of 79.83% at 2531.1 img/s (a 54% speedup and +0.39% accuracy), and they further demonstrate compatibility with existing pruning and merging methods and robustness under hyper-speed inference and random masking.

Significance. The core idea of retokenizing image patches in analogy to subword tokenization is well motivated and potentially useful. The compatibility experiments (Table 9) and the analyses of token attentiveness and similarity during merging (Tables 5-8) provide interesting evidence about why merging inattentive tokens first can help. If the accuracy gain were properly isolated to the retokenization mechanism, this would be a meaningful contribution to efficient ViT inference. However, as the paper stands, the central accuracy claim is confounded with an architecture change, and no variance information is given, so the significance level is not yet established. The speed benefit of the method is plausible, but the advertised accuracy advantage requires additional controlled experiments.

major comments (4)
  1. [Ablation Study (near end) and Tables 1-2] The paper's headline result, +0.39% accuracy over DeiT-S while improving throughput by 54%, is not attributable to retokenization because the full model also replaces the standard patchify stem with an overlapping-convolution local coherence bias module. The only ablation removes this entire module, giving 79.81%, which is slightly below DeiT-S's 79.83%. This strongly suggests that the entire accuracy gain comes from the conv stem, not from the retokenization mechanism. The authors must add a model that uses the conv stem with the original DeiT tokenizer (no retokenization and no pruning) to separate the stem's contribution; without that control, the central accuracy claim is unsupported.
  2. [Tables 1, 2 and Section "Main Results"] The comparisons against DynamicViT, EViT, ToMe, and other baselines are unfair because those baselines use the standard patchify stem, whereas ImagePiece uses a different overlapping-convolution stem. Since early convolutional stems are known to improve ViT accuracy (the paper itself cites Xiao et al. 2021 but does not discuss this), the accuracy advantage over the baselines could be due to the stem rather than the retokenization strategy. To make the comparison meaningful, the authors should either integrate the same stem into the baselines or report a stem-only baseline (DeiT with conv stem, no token reduction) for all comparisons.
  3. [Section "ImagePiece", Subsection "Re-tokenizing Non-semantic Tokens"] The method description is underspecified for reproducibility. The three-stage retokenization procedure does not state at which transformer layers the retokenization is applied, how many retokenization iterations are run per layer, or how the hyperparameters (p = 0.3, similarity merging ratio = 0.08, pruning keep rate r = 0.8) are scheduled across layers. For example, Table 5 reports statistics at layers 2, 3, 5, 6, 8, and 9, but the text never explains why these layers are chosen. A precise algorithm description or pseudocode, plus a table of hyperparameter schedules, is needed before the experiments can be reproduced.
  4. [Tables 1-4 and the Abstract] The main claims of +0.39% accuracy and 54% speedup are reported without error bars or multiple-seed trials. ImageNet-1k training from scratch typically has run-to-run variation of at least ±0.2%, which is comparable to the reported accuracy improvement. The authors should report the mean and standard deviation over at least two or three seeds for the main models, and state the measurement conditions for throughput (e.g., batch size, hardware, software versions). Without such information, the accuracy gain cannot be distinguished from noise.
minor comments (6)
  1. [Ablation Study] When describing the ablation that removes the local coherence bias module, the paper should explicitly state that this returns to the standard patchify stem, so that readers understand the two confounded changes.
  2. [Table 4] The caption for Table 4 says "16×16 masks" but it is not clear whether each mask is a 16×16 pixel block or a patch, nor how many masks are applied for each column besides the listed counts. Please clarify the mask generation procedure.
  3. [Table 5] The header "Ratioinattn→attn(%)" is missing spacing and should be written as "Ratio (inattn→attn) (%)" for readability.
  4. [Section "Implementation details"] The local coherence bias module is described as "four 3×3 convolutions and a single 1×1 convolution" but the stride, padding, number of channels, and output resolution are not specified. Please provide the full architecture details.
  5. [Abstract and Section "Hyper-speed Inference Results"] The abstract's claim that the approach surpasses other baselines "by an accuracy over 8%" is vague; the Experiments section reports the average gap as 8.15% over three baselines. Please state the average explicitly in both places or give the range of per-baseline gaps.
  6. [Throughout] The paper uses "re-tokenization" and "retokenization" inconsistently; please pick one and use it consistently. There are also minor hyphenation errors such as "bottom-k" and "bottom-k" appearing inconsistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ImagePiece's claims rest on empirical ImageNet evaluation against external baselines; the missing stem-only ablation is an attribution confound, not a circular reduction.

full rationale

The paper's central claims are empirical, not derivational: accuracy and throughput are measured on ImageNet-1k and compared with external baselines (DynamicViT, EViT, ToMe, Token Pooling, Token Learner), and the method's hyperparameters (p=0.3, similarity ratio=0.08, pruning ratio=0.8) are fixed settings rather than parameters fitted to the test set. Equation (1) only defines the class-attention score used to rank tokens; it does not encode the outcome being claimed. The inattentive-to-attentive and similarity analyses in Tabs. 5-8 are descriptive observations about the trained system, not fitted quantities masquerading as predictions. The paper contains no load-bearing self-citation chain or imported uniqueness theorem; its references to prior work are standard external baselines and building blocks, and Tab. 9 provides independent compatibility evidence by plugging ImagePiece into existing methods. The most serious weakness, identified in the reader's take, is that the ablation removes the local-coherence-bias conv stem together with retokenization, so the reported +0.39% accuracy gain over DeiT-S is not cleanly attributable to retokenization alone. That is a genuine experimental-design and attribution concern, but it is not circularity: it does not reduce the claimed result to its own inputs by definition, and per the review rules it belongs under correctness risk rather than the circularity score. No circular step can be exhibited with the required specificity, so the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several hand-tuned hyperparameters, a domain assumption about CLS attention, and the local-coherence heuristic. No new physical entities are introduced.

free parameters (5)
  • p (non-semantic candidate proportion) = 0.3
    Bottom 30% of tokens are selected as candidates for retokenization; chosen by the authors and stated in Implementation details.
  • similarity merging ratio = 0.08
    Token pairs equal to 0.08 times the total number of tokens are merged from the non-semantic set; tuned for the speed and accuracy tradeoff.
  • pruning ratio r (keep rate) = 0.8
    Bottom 20% of tokens are discarded after retokenization; authors say these hyperparameters are 'specifically adjusted to further accelerate ViTs beyond the standard settings.'
  • local coherence bias module architecture = 4x 3x3 convolutions plus 1x1 convolution
    Replaces the patchify stem; kernel sizes, overlap, and depth are hand-picked rather than derived.
  • hyper-speed inference keep rates / merging ratios = not specified in main text
    For Fig. 3, models are evaluated at shifted keep rates without retraining, but the exact schedules for the proposed method are deferred to the supplements.
assumptions (4)
  • domain assumption The [CLS] attention score is a valid measure of token semantic importance.
    Used in Step I to select bottom-k tokens; inherited from EViT (Liang et al. 2022) and not independently established for merged tokens.
  • domain assumption Locally coherent patches, encouraged by overlapping convolutions, form semantically meaningful groups.
    Core inductive bias of the local coherence bias module; asserted rather than proven.
  • domain assumption Bipartite soft matching (ToMe) produces good merges when restricted to the bottom-k inattentive tokens.
    The method reuses ToMe's matching algorithm without analyzing its properties under this restricted setting.
  • domain assumption Throughput measured on a single RTX 3090 is a faithful proxy for inference efficiency comparisons.
    Batch size, software stack, and measurement procedure are not reported; comparisons against other papers' numbers may not be apples-to-apples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition." pith.science (2026). https://pith.science/paper/PU42ZCGM

@misc{pith2026241216491,
  author       = {Pith},
  title        = {Pith review of: ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PU42ZCGM}},
  note         = {Machine review of arXiv:2412.16491}
}
abstract

Vision Transformers (ViTs) have achieved remarkable success in various computer vision tasks. However, ViTs have a huge computational cost due to their inherent reliance on multi-head self-attention (MHSA), prompting efforts to accelerate ViTs for practical applications. To this end, recent works aim to reduce the number of tokens, mainly focusing on how to effectively prune or merge them. Nevertheless, since ViT tokens are generated from non-overlapping grid patches, they usually do not convey sufficient semantics, making it incompatible with efficient ViTs. To address this, we propose ImagePiece, a novel re-tokenization strategy for Vision Transformers. Following the MaxMatch strategy of NLP tokenization, ImagePiece groups semantically insufficient yet locally coherent tokens until they convey meaning. This simple retokenization is highly compatible with previous token reduction methods, being able to drastically narrow down relevant tokens, enhancing the inference speed of DeiT-S by 54% (nearly 1.5$\times$ faster) while achieving a 0.39% improvement in ImageNet classification accuracy. For hyper-speed inference scenarios (with 251% acceleration), our approach surpasses other baselines by an accuracy over 8%.

Figures

Figures reproduced from arXiv: 2412.16491 by the authors.

Figure 1
Figure 1. An illustration of the ImagePiece pipeline com [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall architecture of the proposed method. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of our ImagePiece with the patch to [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 23 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bolya, D.; Fu, C.-Y.; Dai, X.; Zhang, P.; Feichtenhofer, C.; and Hoffman, J. 2023. Token Merging: Your ViT But Faster. In The Eleventh International Conference on Learning Representations

    work page 2023

  4. [4]

    Bolya, D.; Fu, C.-Y.; Dai, X.; Zhang, P.; and Hoffman, J. 2022. Hydra attention: Efficient attention with many heads. In European Conference on Computer Vision, 35--49. Springer

    work page 2022

  5. [5]

    D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al

    Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877--1901

    2020

  6. [6]

    Chen, M.; Lin, M.; Li, K.; Shen, Y.; Wu, Y.; Chao, F.; and Ji, R. 2023. Cf-vit: A general coarse-to-fine method for vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 7042--7052

    work page 2023

  7. [7]

    Choromanski, K.; Likhosherstov, V.; Dohan, D.; Song, X.; Gane, A.; Sarlos, T.; Hawkins, P.; Davis, J.; Mohiuddin, A.; Kaiser, L.; et al. 2020. Rethinking attention with performers. arXiv preprint arXiv:2009.14794

    arXiv 2020

  8. [8]

    Chu, X.; Tian, Z.; Wang, Y.; Zhang, B.; Ren, H.; Wei, X.; Xia, H.; and Shen, C. 2021. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34: 9355--9366

    work page 2021

Show all 51 references
  1. [9]

    Dao, T.; Fu, D.; Ermon, S.; Rudra, A.; and R \'e , C. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 16344--16359

    2022

  2. [10]

    Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In CVPR, 248--255

    2009

  3. [11]

    Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  4. [12]

    Dong, X.; Bao, J.; Chen, D.; Zhang, W.; Yu, N.; Yuan, L.; Chen, D.; and Guo, B. 2022. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12124--12134

    2022

  5. [13]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR

    2021

  6. [14]

    Fan, H.; Xiong, B.; Mangalam, K.; Li, Y.; Yan, Z.; Malik, J.; and Feichtenhofer, C. 2021. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, 6824--6835

    2021

  7. [15]

    A.; Jafari, F

    Fayyaz, M.; Koohpayegani, S. A.; Jafari, F. R.; Sengupta, S.; Joze, H. R. V.; Sommerlade, E.; Pirsiavash, H.; and Gall, J. 2022. Adaptive token sampling for efficient vision transformers. In European Conference on Computer Vision, 396--414. Springer

    2022

  8. [16]

    R.; Raje, S.; Chakaravarthy, V.; Sabharwal, Y.; and Verma, A

    Goyal, S.; Choudhury, A. R.; Raje, S.; Chakaravarthy, V.; Sabharwal, Y.; and Verma, A. 2020. PoWER-BERT: Accelerating BERT inference via progressive word-vector elimination. In International Conference on Machine Learning, 3690--3699. PMLR

    2020

  9. [17]

    Graham, B.; El-Nouby, A.; Touvron, H.; Stock, P.; Joulin, A.; J \'e gou, H.; and Douze, M. 2021. Levit: a vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEE/CVF international conference on computer vision, 12259--12269

    2021

  10. [18]

    Kim, G.; and Cho, K. 2020. Length-adaptive transformer: Train once with length drop, use anytime with search. arXiv preprint arXiv:2010.07003

    2020 arXiv

  11. [19]

    Kim, S.; Shen, S.; Thorsley, D.; Gholami, A.; Kwon, W.; Hassoun, J.; and Keutzer, K. 2022. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 784--794

    2022

  12. [20]

    Kitaev, N.; Kaiser, .; and Levskaya, A. 2020. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451

    2020 arXiv

  13. [21]

    Kong, Z.; Dong, P.; Ma, X.; Meng, X.; Sun, M.; Niu, W.; Shen, X.; Yuan, G.; Ren, B.; Qin, M.; et al. 2022. Spvit: Enabling faster vision transformers via soft token pruning. ECCV

    2022

  14. [22]

    Kudo, T.; and Richardson, J. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226

    2018 arXiv

  15. [23]

    Lassance, C.; Maachou, M.; Park, J.; and Clinchant, S. 2021. A study on token pruning for colbert. arXiv preprint arXiv:2112.06540

    2021 arXiv

  16. [24]

    Li, Y.; Wu, C.-Y.; Fan, H.; Mangalam, K.; Xiong, B.; Malik, J.; and Feichtenhofer, C. 2022. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4804--4814

    2022

  17. [25]

    Liang, Y.; GE, C.; Tong, Z.; Song, Y.; Wang, J.; and Xie, P. 2022. EV iT: Expediting Vision Transformers via Token Reorganizations. In International Conference on Learning Representations

    2022

  18. [26]

    Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 10012--10022

    2021

  19. [27]

    Long, S.; Zhao, Z.; Pi, J.; Wang, S.; and Wang, J. 2023. Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10334--10343

    2023

  20. [28]

    R.; Ranjan, A.; Prabhu, A.; Rastegari, M.; and Tuzel, O

    Marin, D.; Chang, J.-H. R.; Ranjan, A.; Prabhu, A.; Rastegari, M.; and Tuzel, O. 2021. Token pooling in vision transformers. arXiv preprint arXiv:2110.03860

    2021 arXiv

  21. [29]

    Mehta, S.; and Rastegari, M. 2022. Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer. ICLR

    2022

  22. [30]

    Meng, L.; Li, H.; Chen, B.-C.; Lan, S.; Wu, Z.; Jiang, Y.-G.; and Lim, S.-N. 2022. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12309--12318

    2022

  23. [31]

    Michel, P.; Levy, O.; and Neubig, G. 2019. Are sixteen heads really better than one? Advances in neural information processing systems, 32

    2019

  24. [32]

    Pan, B.; Panda, R.; Jiang, Y.; Wang, Z.; Feris, R.; and Oliva, A. 2021. IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers. Advances in Neural Information Processing Systems, 34: 24898--24911

    2021

  25. [33]

    Pan, Z.; Cai, J.; and Zhuang, B. 2022. Fast vision transformers with hilo attention. Advances in Neural Information Processing Systems, 35: 14541--14554

    2022

  26. [34]

    Pan, Z.; Zhuang, B.; He, H.; Liu, J.; and Cai, J. 2022. Less is more: Pay less attention in vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2035--2043

    2022

  27. [35]

    Rao, Y.; Zhao, W.; Liu, B.; Lu, J.; Zhou, J.; and Hsieh, C.-J. 2021. DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification. In Beygelzimer, A.; Dauphin, Y.; Liang, P.; and Vaughan, J. W., eds., Advances in Neural Information Processing Systems

    2021

  28. [36]

    Ryoo, M.; Piergiovanni, A.; Arnab, A.; Dehghani, M.; and Angelova, A. 2021. Tokenlearner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems, 34: 12786--12797

    2021

  29. [37]

    Shen, Z.; Zhang, M.; Zhao, H.; Yi, S.; and Li, H. 2021. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, 3531--3539

    2021

  30. [38]

    Song, Z.; Xu, Y.; He, Z.; Jiang, L.; Jing, N.; and Liang, X. 2022. Cp-vit: Cascade vision transformer pruning via progressive sparsity prediction. arXiv preprint arXiv:2203.04570

    2022 arXiv

  31. [39]

    Touvron, H.; Cord, M.; Douze, M.; Massa, F.; Sablayrolles, A.; and J \'e gou, H. 2021 a . Training data-efficient image transformers & distillation through attention. In International conference on machine learning, 10347--10357. PMLR

    2021

  32. [40]

    Touvron, H.; Cord, M.; Sablayrolles, A.; Synnaeve, G.; and J \'e gou, H. 2021 b . Going deeper with image transformers. In Proceedings of the IEEE/CVF international conference on computer vision, 32--42

    2021

  33. [41]

    N.; Kaiser, L

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L. u.; and Polosukhin, I. 2017. Attention is All you Need. In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., Advances in Neural Infor...

    2017

  34. [42]

    Voita, E.; Talbot, D.; Moiseev, F.; Sennrich, R.; and Titov, I. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. ACL

    2019

  35. [43]

    Z.; Khabsa, M.; Fang, H.; and Ma, H

    Wang, S.; Li, B. Z.; Khabsa, M.; Fang, H.; and Ma, H. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768

    2020 arXiv

  36. [44]

    Wang, W.; Xie, E.; Li, X.; Fan, D.-P.; Song, K.; Liang, D.; Lu, T.; Luo, P.; and Shao, L. 2021. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, 568--578

    2021

  37. [45]

    Xiao, T.; Singh, M.; Mintun, E.; Darrell, T.; Doll \'a r, P.; and Girshick, R. 2021. Early convolutions help transformers see better. Advances in neural information processing systems, 34: 30392--30400

    2021

  38. [46]

    Xu, Y.; Zhang, Z.; Zhang, M.; Sheng, K.; Li, K.; Dong, W.; Zhang, L.; Xu, C.; and Sun, X. 2022. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2964--2972

    2022

  39. [47]

    R.; and Le, Q

    Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R. R.; and Le, Q. V. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32

    2019

  40. [48]

    M.; Mallya, A.; Kautz, J.; and Molchanov, P

    Yin, H.; Vahdat, A.; Alvarez, J. M.; Mallya, A.; Kautz, J.; and Molchanov, P. 2022. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10809--10818

    2022

  41. [49]

    Yu, H.; and Wu, J. 2023. A unified pruning framework for vision transformers. Science China Information Sciences, 66(7): 1--2

    2023

  42. [50]

    E.; Feng, J.; and Yan, S

    Yuan, L.; Chen, Y.; Wang, T.; Yu, W.; Shi, Y.; Jiang, Z.-H.; Tay, F. E.; Feng, J.; and Yan, S. 2021. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision, 558--567

    2021

  43. [51]

    Zhou, D.; Kang, B.; Jin, X.; Yang, L.; Lian, X.; Jiang, Z.; Hou, Q.; and Feng, J. 2021. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.11886

    2021 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.