Pith. sign in

REVIEW 5 major objections 6 minor 22 references

Training Noise Token Pruning

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a noise-allocation training signal — adding graded noise to tokens instead of dropping them — learns token relevance well enough to beat CLS-attention pruning on frozen vision transformers, with no CLS token required.

desk verdict Genuinely new noise-relaxation idea, honest limitations, but SOTA claim overreaches and the softmax normalization blurs the claimed mechanism. read the letter →

arxiv 2411.18092 v2 pith:MWNBJEHV submitted 2024-11-27 cs.CV

Mingxing Rao , Bohan Jiang , Daniel Moyer This is my paper · ORCID
classification cs.CV
keywords tokenpruningvisiontransformerinformationbottleneckadditivenoiserelaxationrate-distortionfrozenbackbonesimilarity-basedImageNet-1K
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a lightweight add-on module can learn which image patches a vision transformer can afford to lose, purely by adding graded noise to tokens during training: no hard dropping, no fine-tuning of the backbone, and no reliance on the CLS token. If the claim holds, the payoff is a cheap way to speed up any existing vision transformer — train one small head for 40 epochs on a frozen model, then at deployment rank tokens by the learned score, keep the top K, and remove redundant patches with a similarity pass. The reported evidence is strongest at aggressive keep rates, where the method is claimed to preserve accuracy far better than stochastic-dropout, token-merging, and CLS-attention baselines at matched FLOPs, while also working on a mean-pooled ViT where CLS-attention methods cannot be applied at all.

What carries the argument

The central object is the noise allocator: a linear map $W \in \mathbb{R}^{D \times 1}$ followed by a softmax that converts each token embedding into a relevance score $\alpha_i$, with training noise $\eta_i = (1 - \alpha_i)\varepsilon$ for $\varepsilon \sim \mathcal{N}(0, \beta I)$ and a fixed noise scale $\beta = 0.02$. Its companion at deployment is similarity-based pruning by random partition — tokens are split into two random groups, the closest pair across groups is found by embedding similarity, and the most similar pairs are pruned — which removes redundancy that element-wise relevance scores miss. The information-bottleneck framing ties the two together: the token embeddings stand in for the latent means and $(1 - \alpha_i)$ for the element-wise standard deviations of the Deep Variational Information Bottleneck, so the softmax-constrained noise budget is the rate term and classification accuracy is the relevance term.

What would settle it

Keep the full TNT evaluation protocol but replace the trained noise-allocator head with an untrained, randomly initialized head and compare top-$K$ accuracies at the same keep rates: if the gap between trained and untrained rankings is small at $K = 0.3$ or below, the 40-epoch noise training is not the source of the reported gains. A complementary check is to inspect the learned score distribution, since the softmax over roughly 196 tokens pushes each $\alpha_i$ near $1/N$; if the spread of $(1 - \alpha_i)$ is negligible, the training-time noise is nearly uniform and the learned scores cannot be carrying the pruning signal.

Watch

Extended reading notes

Core claim

TNT relaxes discrete token dropping into a continuous optimization problem: a per-token relevance score $\alpha_i$, computed by a linear layer and softmax attached to a frozen transformer block, controls the amplitude of Gaussian noise $\eta_i = (1 - \alpha_i)\varepsilon$, $\varepsilon \sim \mathcal{N}(0, \beta I)$, added to that token's embedding. Because the softmax keeps the total noise budget fixed, the network can only protect the most predictive tokens, and so $\alpha$ becomes a learned estimate of token relevance. At test time the noise is replaced by hard pruning: tokens are ranked by $\alpha$, the top $K$ are kept, and a random-partition similarity step removes the most redundant among them, recovering the discrete computational savings that the continuous training relaxation hides. The paper frames the procedure as an information-bottleneck trade-off in which the number of kept tokens is the transmission rate and classification accuracy is the relevance, with layer-wise normalization bounding the signal power so the noise budget acts as a genuine rate constraint. On ImageNet-1K with ViT and DeiT backbones — frozen except for the noise-allocator heads — the paper reports accuracy at matched compute that matches or exceeds the CLS-attention Top-K baseline while also succeeding on a mean-pooled ViT, and its largest margins come at the lowest token keep rates.

Load-bearing premise

A relevance ranking learned under gentle additive noise on a frozen backbone remains the correct ranking for removing whole tokens at test time, even when far more tokens are removed than the noise schedule ever simulated and the similarity-pruning step is applied without ever having been part of training.

Editorial extensions

If this is right

  • Pruning no longer needs a CLS token or attention supervision: any frozen ViT-style backbone can be retrofitted with a short-trained noise-allocator head, which is what lets the method run on mean-pooled, CLS-free models.
  • The paper's largest reported margins come at the most aggressive keep rates — multi-layer DeiT-B at about 5.9 GFLOPs holds 59.9% top-1 accuracy versus 56.0% for Top-K, 54.3% for EViT, 43.4% for ToMe, 19.8% for Zero-TP, and 11.2% for DynamicViT.
  • Because training is limited to the new head on a frozen backbone, the procedure can be applied to already-pretrained checkpoints rather than requiring a new model to be trained from scratch.
  • Test-time inference still enjoys discrete-dropping savings: the allocator scores tokens, drops the losers, and only the kept tokens flow through the rest of the network.
  • The information-bottleneck derivation reframes the accuracy/compute trade-off as a rate-distortion choice, giving a principled vocabulary for setting how many tokens each layer may keep.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same noise-allocator recipe should transfer to non-classification tasks — detection, segmentation, captioning — that use ViT-style backbones without a CLS token, since the only requirements are a frozen backbone and a differentiable task loss.
  • Beyond the paper: the training noise scale $\beta$ and the test-time keep rate $K$ are only loosely connected; sweeping $\beta$ per layer so the training-time rate constraint matches the deployment keep rate would likely sharpen the transfer of the learned ranking.
  • Beyond the paper: because redundancy removal is never simulated in training (a limitation the paper states), a soft, differentiable merge during training could teach the allocator to favor tokens with low mutual redundancy, directly addressing that limitation.
  • Beyond the paper: the rate-distortion framing suggests replacing the hand-tuned per-layer keep rates with a distortion-based allocation — give more kept tokens to layers where marginal accuracy loss is highest — which is a testable alternative schedule.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Training Noise Token (TNT) Pruning, a method for vision transformers that relaxes discrete token dropping into a continuous additive noise process. During training, a lightweight noise allocator computes per-token scores alpha via a softmax over a learned linear projection of token embeddings (Eq. 4) and perturbs each token with Gaussian noise scaled by (1 - alpha) (Eq. 5). At test time, the same alpha scores are used to rank tokens and prune the lowest-ranked ones, optionally followed by a similarity-based redundancy-removal step using random partitioning. The authors claim theoretical connections to Rate-Distortion / Variational Information Bottleneck, and evaluate on ImageNet with DeiT-T/S/B and ViT/16, reporting accuracy, GFLOPs, and throughput under single-layer and multi-layer pruning schedules. The paper includes extensive supplementary tables, an ablation study, qualitative visualizations, and a PyTorch implementation.

Significance. If the claims were substantiated, TNT would be a practically relevant contribution: it prunes tokens using only a small learned allocator on a frozen backbone, needs no CLS token, and could be applied to architectures without a classification token. The paper is commendable for providing code, extensive ImageNet experiments, qualitative pruning maps, and ablations over several design choices. However, the headline claims of state-of-the-art performance and of a principled information-theoretic method are not supported by the evidence in the manuscript. The paper's own tables show that TNT is frequently outperformed by EViT and DynamicViT in the low-token regimes where the authors claim superiority, and the softmax-based noise allocation has a structural property that undermines the claimed per-token relevance learning. The theoretical connection to the information bottleneck is asserted rather than derived. The practical value of the method is therefore uncertain outside a narrow set of configurations.

major comments (5)
  1. [Section 5 and Tables 6, 19-20] The abstract and conclusion claim that TNT 'demonstrates advantages over previous pruning methods' and 'excels in low-token retention rates.' These claims are contradicted by the paper's own data. In single-layer DeiT-B-Distil. at K=0.25 (Table 6), EViT achieves 73.54 while TNT achieves 70.76, and at K=0.2 EViT achieves 70.4 vs TNT 62.62. In multi-layer ViT/16 (Table 20), at GFLOPs ≈ 3.9 DynamicViT achieves 60.74 vs TNT 51.40, and at GFLOPs ≈ 3.6 DynamicViT achieves 50.15 vs TNT 39.81. The 'state-of-the-art' claim is therefore not supported by the authors' own measurements; the conclusion overstates the method's performance.
  2. [Section 3, Eqs. (4)-(5)] The softmax normalization in Eq. (4) makes alpha a probability distribution over all N tokens, so the sum of alpha_i is 1. For N=196, the average alpha is about 0.005, and the noise multiplier (1 - alpha_i) in Eq. (5) is approximately 0.995 for almost every token. At keep rates K=0.5 or K=0.25, most of the tokens retained at test time therefore received nearly the same noise magnitude during training as the tokens that were dropped. The gradient signal for ranking the retained tokens is consequently very weak, and the claim that alpha is a learned per-token relevance estimate is not supported by the mechanism as written. The reported gains over random dropping cannot be attributed to the noise-allocation mechanism without further analysis, especially given the acknowledged train/deploy gap in Section 4.4.
  3. [Section 3, 'Classical results...' and Section 1] The paper claims 'theoretical connections to Rate-Distortion literature' and positions TNT 'within the Information Bottleneck framework,' but no information-bottleneck objective is actually optimized. The training loss is the standard cross-entropy, there is no rate term, no mutual-information estimator, and the cited channel-capacity bound is not used to derive any result. The sentence 'This solution can also be directly mapped onto the Deep Variational Information Bottleneck' is an assertion, not a derivation. As a result, the information-theoretic framing is rhetorical rather than substantive, and the paper's methodological novelty rests entirely on the heuristic noise-allocation scheme.
  4. [Section 4.2, Tables 7, 11-12, 19-20] The multi-layer comparisons do not appear to be on equal footing. In Table 11 (DeiT-S-Distil. multi-layer), the EViT parameter is listed as ρ=.2 for every row while GFLOPs vary from 3.48 to 2.05, which suggests the EViT keep rate was held fixed rather than swept; this does not produce a genuine accuracy-computation trade-off curve for EViT. Additionally, in Table 19 (ViT/16 multi-layer), DynamicViT's GFLOPs are systematically higher than TNT's GFLOPs within the same nominal bucket (e.g., 7.23 vs 6.73 for 'GFLOPs ≈ 6.8'), so the comparison is not at matched compute. These issues undermine the claim that TNT 'consistently shows strong performance' in the multi-layer setting.
  5. [Section 4.4 and Figure 6] Section 4.4 admits that redundant tokens are not removed during training, and the code in Figure 6 adds noise at layers 1-5 during training, whereas deployment prunes at a single layer (Section 4.1) or at layers [3,4,5] (Table 4). Thus the alpha scores used at the pruning layer are trained under a distribution in which earlier layers also inject noise, but at test time no noise is added before pruning. The paper does not analyze how this train/deploy mismatch affects the validity of alpha as a relevance ranking for the deployment procedure, which is a load-bearing gap given that the method's only training signal is the noise allocation.
minor comments (6)
  1. [Table 3 caption] The caption says the ablation is for 'DeiT-S-Distil.' but the table rows list 'Deit-B-Distil.'; please correct the inconsistency.
  2. [References] Several references contain stray page numbers (e.g., [4] and [16] end with '1, 8, 9, 10, ...' and similar strings); please clean the bibliography.
  3. [Section 1 and Tables] EViT is cited inconsistently as both [10] and [14]; use a single reference consistently throughout.
  4. [Supplementary Figure 7] The supplementary uses 'TNT+MLP' for DeiT-Tiny, but the main text does not describe this variant; please explain the difference between TNT and TNT+MLP.
  5. [Section 4.1 and Section 4.2] The number of similarity-pruned tokens s is stated as 25 and 30 for DeiT and ViT respectively in Section 4.1, but as 40 for all experiments in Section 4.2; please clarify which setting applies to which configuration.
  6. [Section 3.1] The text says 'Both I(x, y) and I(s, x) measure interaction information'; this terminology is imprecise, as these are mutual information quantities, not interaction information.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the relevance scores are optimized against an external classification loss on training images and evaluated on held-out validation images, so the reported pruning accuracy is not equivalent to the training objective by construction.

full rationale

The paper's derivation chain was walked. Section 3 defines the learned scores via Eq. 4 (α_i = Softmax(W f)_i) and the training-time noise via Eq. 5 (η_i = (1-α_i)ε), but these equations do not define the test-time pruning outcome; they define a differentiable noise perturbation used during training. The α values are optimized by gradient descent against the standard classification cross-entropy on ImageNet training images, as shown in the public implementation excerpt (Fig. 6: x = x + 0.02 * noise before the head), so the relevance estimate is fitted to an external label signal rather than to the validation accuracy that is later reported. At test time, tokens are ranked and dropped on validation images whose labels never participated in training, and accuracy is measured against those labels; there is no test-time fitting and no prediction that reduces to the training objective by construction. The similarity-pruning stage is explicitly adapted from Zero-TP with disclosed modifications and is an additional pruning mechanism, not the source of the α relevance definition. The Rate-Distortion and Information Bottleneck discussion is presented as analogy and motivation ("more clearly illustrates the analogy to the Token Pruning problem"), not as a theorem that tautologically forces the empirical results. Section 4.4 itself admits a training/deployment mismatch: redundant tokens are not removed during training, and the method is not optimized for specific deployment keep-rates. That is a correctness or generalization concern, not circularity: the reported gains are empirical measurements against external baselines and ImageNet validation labels. No load-bearing self-citation was found: DynamicViT is by Yongming Rao et al., a different author group, and the variational IB references (Alemi et al., Tishby et al.) are external and independent. The softmax-normalization observation raised by some reviewers is a mechanistic critique of whether the learned α genuinely concentrates noise protection on kept tokens, but it does not make the reported accuracy a re-statement of the training loss; it is an empirical/mechanism risk rather than a circular-definition risk. Overall, the central claim is self-contained and falsifiable through the public code and ImageNet evaluations.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method depends on two hand-set hyperparameters (beta and the similarity-pruning budget s), on a loose mapping between token pruning and the information bottleneck that is asserted rather than derived, and on the transfer assumption that noise-trained alpha scores rank tokens correctly at test time when combined with a similarity-pruning stage not seen in training. No fundamentally new entities are introduced.

free parameters (2)
  • beta (noise amplitude) = 0.02 (fixed for all models)
    Hyperparameter controlling the scale of the added Gaussian noise; set to 0.02 for all base models without per-model tuning. The method's accuracy depends on this choice.
  • s (number of redundant tokens pruned via similarity pruning) = 25 (DeiT) / 30 (ViT) single-layer; 40 multi-layer
    Chosen by hand; the paper states 'We set s to be 25 and 30 for experiments involving DeiT and ViT' and 'For all experiments, s is set to 40' for multi-layer. These values affect the GFLOPs and accuracy.
assumptions (4)
  • domain assumption Token pruning can be modeled as an information bottleneck trade-off where token count corresponds to transmission rate and prediction accuracy corresponds to relevance.
    Invoked in Section 3 (Eqs. 1-3) to justify the method, but the mapping is analogical; no formal proof is given that minimizing cross-entropy with added noise solves a VIB objective.
  • standard math Classical information-theoretic capacity bound log(1 + SNR) applies per token, so reducing noise on high-alpha tokens preserves mutual information with the label.
    Cited from Cover (Section 3), but the application to token-level softmax scores is not derived.
  • domain assumption Training only the noise allocator for 40 epochs on frozen backbones yields alpha scores that transfer to test-time token dropping.
    This is the core transfer assumption; unsupported by ablations varying training length or backbone tuning.
  • ad hoc to paper Random-partition similarity pruning improves performance over sequential partitioning or token merging.
    Claimed in Section 3.1 and the ablation (Table 3) but no mechanism or significance test is given; the choice of s is hand-set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training Noise Token Pruning." pith.science (2026). https://pith.science/paper/MWNBJEHV

@misc{pith2026241118092,
  author       = {Pith},
  title        = {Pith review of: Training Noise Token Pruning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MWNBJEHV}},
  note         = {Machine review of arXiv:2411.18092}
}
read the original abstract

In the present work we present Training Noise Token (TNT) Pruning for vision transformers. Our method relaxes the discrete token dropping condition to continuous additive noise, providing smooth optimization in training, while retaining discrete dropping computational gains in deployment settings. We provide theoretical connections to Rate-Distortion literature, and empirical evaluations on the ImageNet dataset using ViT and DeiT architectures demonstrating TNT's advantages over previous pruning methods.

Figures

Figures reproduced from arXiv: 2411.18092 by the authors.

Figure 1
Figure 1. Training Noise Token Pruning (TNT). Our proposed method computes a relevance term αi for each token. In training (diagrammed at top), these terms dictate an amount of noise added to the token, while at test time they indicate pruning order. [14, 20] Rao et al. 2021 [16] introduces Dynamic ViT, which is no￾table as the first token pruning method for vision transformer [7]. It is prototypical of the stochastic dropout… view at source ↗
Figure 2
Figure 2. Noise Allocator block architecture: the block dia￾grammed above is injected into pre-trained models as a pruning layer. It takes the output of the previous Transformer block as input, then computes the noise signal terms α using a linear layer followed by a Softmax function. During training it samples Gaus￾sian noise conditioned on α for each token, then adds the noise to the token embeddings. At test time, tokens a… view at source ↗
Figure 3
Figure 3. Visualization of Token Pruning maps on ImageNet-1K: at [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Single Layer Pruning results: We plot the Top-1 Accuracy in the ImageNet-1k validation set for each of the pruning methods as a function of computational efficiency, in the top row measured by GFLOPs and in the bottom row measured by throughput, for single layer prunin…
Figure 5
Figure 5. Figure 5: Multi-layer Pruning results: We plot the Top-1 Accuracy in the ImageNet-1k validation set for each of the pruning methods as a function of computational efficiency, in the top row measured by GFLOPs and in the bottom row measured by throughput, for multi-layer pruning.…
Figure 6
Figure 6. Figure 6: Python implementation of VisionTransformerWithTNT class. Codes highlighted with brown are the main modifications. VisionTransformer class is taken from https://github.com/rwightman/pytorch-image-models/blob/master/ timm/models/vision_transformer.py. We make simple modi…
Figure 7
Figure 7. Figure 7 [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: More Visualization of Token Pruning maps on ImageNet-1K: at [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 12 canonical work pages

  1. [1]

    Deep variational information bottleneck

    Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv 8 preprint arXiv:1612.00410, 2016. 1, 2, 3

    arXiv 2016

  2. [2]

    Computation of channel capacity and rate- distortion functions

    Richard Blahut. Computation of channel capacity and rate- distortion functions. IEEE transactions on Information The- ory, 18(4):460–473, 1972. 2

    work page 1972

  3. [3]

    Token merging for fast sta- ble diffusion

    Daniel Bolya and Judy Hoffman. Token merging for fast sta- ble diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4599–4603,

    work page

  4. [4]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. 2, 4, 6, 7, 1, 8, 9, 10, 11, 12, 13, 14, 15, 16

    arXiv 2022

  5. [5]

    Elements of information theory

    Thomas M Cover. Elements of information theory . John Wiley & Sons, 1999. 3

    1999

  6. [6]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 4, 5

    work page 2009

  7. [7]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 1, 2, 4

    arXiv 2010

  8. [8]

    Adaptive token sampling for efficient vision transformers

    Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and J¨urgen Gall. Adaptive token sampling for efficient vision transformers. In European Conference on Computer Vision, pages 396–414. Springer, 2022. 1, 4, 5

    work page 2022

Show all 22 references
  1. [9]

    Power-bert: Accelerating bert inference via progres- sive word-vector elimination

    Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma. Power-bert: Accelerating bert inference via progres- sive word-vector elimination. In International Conference on Machine Learning, pages 3690–3699. PMLR, 2020. 1

    2020

  2. [10]

    Which tokens to use? investigating token reduction in vision transformers

    Joakim Bruslund Haurum, Sergio Escalera, Graham W Taylor, and Thomas B Moeslund. Which tokens to use? investigating token reduction in vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 773–783, 2023. 1, 2, 4, 6, 7

    2023

  3. [11]

    Length-adaptive trans- former: Train once with length drop, use anytime with search

    Gyuwan Kim and Kyunghyun Cho. Length-adaptive trans- former: Train once with length drop, use anytime with search. arXiv preprint arXiv:2010.07003, 2020. 1

    2010 arXiv

  4. [12]

    Learned token pruning for transformers

    Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 784–794, 2022. 1

    2022

  5. [13]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 3

    2013 arXiv

  6. [14]

    Not all patches are what you need: Expediting vision transformers via token reorganizations

    Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.07800 , 2022. 1, 2, 4, 7, 5, 6, 8, 9, 10, 11, 12, 13

    2022 arXiv

  7. [15]

    The pagerank citation ranking: Bringing order to the web

    Lawrence Page. The pagerank citation ranking: Bringing order to the web. Technical report, Technical Report, 1999. 2

    1999

  8. [16]

    Dynamicvit: Efficient vision trans- formers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision trans- formers with dynamic token sparsification. Advances in neu- ral information processing systems, 34:13937–13949, 2021. 1, 2, 4, 5, 7, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16

    2021

  9. [17]

    The information bottleneck method

    Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000. 1, 2

    2000 arXiv

  10. [18]

    Training data-efficient image transformers & distillation through atten- tion

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv´e J´egou. Training data-efficient image transformers & distillation through atten- tion. In International conference on machine learning, pages 10347–10357. PMLR, 2021. 1, 4, 7

    2021

  11. [19]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 1

    2017

  12. [20]

    Zero- tprune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers

    Hongjie Wang, Bhishma Dedhia, and Niraj K Jha. Zero- tprune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16070–16079, 2024. 1, 2, 4, 5...

    2024

  13. [21]

    Evo-vit: Slow-fast token evolution for dynamic vision transformer

    Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, pages 2964–2972, 2022

    2022

  14. [22]

    VisionTransformer- WithTNT

    Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10809–10818, 2022. 1, 2, 6 9 Training Noi...

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.