REVIEW 8 cited by
PPT: Token Pruning and Pooling for Efficient Vision Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Vision Transformers (ViTs) have emerged as powerful models in the field of computer vision, delivering superior performance across various vision tasks. However, the high computational complexity poses a significant barrier to their practical applications in real-world scenarios. Motivated by the fact that not all tokens contribute equally to the final predictions and fewer tokens bring less computational cost, reducing redundant tokens has become a prevailing paradigm for accelerating vision transformers. However, we argue that it is not optimal to either only reduce inattentive redundancy by token pruning, or only reduce duplicative redundancy by token merging. To this end, in this paper we propose a novel acceleration framework, namely token Pruning & Pooling Transformers (PPT), to adaptively tackle these two types of redundancy in different layers. By heuristically integrating both token pruning and token pooling techniques in ViTs without additional trainable parameters, PPT effectively reduces the model complexity while maintaining its predictive accuracy. For example, PPT reduces over 37% FLOPs and improves the throughput by over 45% for DeiT-S without any accuracy drop on the ImageNet dataset. The code is available at https://github.com/xjwu1024/PPT and https://github.com/mindspore-lab/models/
Forward citations
Cited by 8 Pith papers
-
Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective
Heterogeneity-aware depth pruning of attention and activation layers, guided by a polynomial model-accuracy predictor, delivers up to 1.58× speedup on DeiT-B and 5.19× when combined with width pruning.
-
Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Models
ACCM recovers information lost in high-rate visual token pruning by generating a question-guided caption from discarded tokens and selecting the best candidate, improving pruned LVLM accuracy with fewer FLOPs.
-
ToFe: Lagged Token Freezing and Reusing for Efficient Vision Transformer Inference
ToFe freezes unimportant ViT tokens temporarily and reuses them later, achieving about 50% FLOPs reduction with less than 2% top-1 accuracy drop on ImageNet.
-
ToSA: Token Merging with Spatial Awareness
A training-free token merging method that adds depth-derived spatial similarity to ToMe's bipartite soft matching, improving VQA accuracy at high token reduction rates.
-
Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers
GRAT accelerates pretrained diffusion transformers by grouping tokens and restricting each group's attention to neighboring blocks or criss-cross rows and columns, achieving large speedups with near-full-attention quality.
-
GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models
A training-free token pruning method that combines cosine-similarity saliency with greedy redundancy removal to preserve accuracy at high compression ratios.
-
Continual Learning for Generative AI: From LLMs to MLLMs and Beyond
A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.
-
Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Acceleration
Token reduction for vision transformers is recast as a matrix transformation, and a many-to-many weighted merging is used for training-free acceleration.
Discussion (0). Continue with ORCID to comment.