REVIEW 5 cited by
DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Attention is sparse in vision transformers. We observe the final prediction in vision transformers is only based on a subset of most informative tokens, which is sufficient for accurate image recognition. Based on this observation, we propose a dynamic token sparsification framework to prune redundant tokens progressively and dynamically based on the input. Specifically, we devise a lightweight prediction module to estimate the importance score of each token given the current features. The module is added to different layers to prune redundant tokens hierarchically. To optimize the prediction module in an end-to-end manner, we propose an attention masking strategy to differentiably prune a token by blocking its interactions with other tokens. Benefiting from the nature of self-attention, the unstructured sparse tokens are still hardware friendly, which makes our framework easy to achieve actual speed-up. By hierarchically pruning 66% of the input tokens, our method greatly reduces 31%~37% FLOPs and improves the throughput by over 40% while the drop of accuracy is within 0.5% for various vision transformers. Equipped with the dynamic token sparsification framework, DynamicViT models can achieve very competitive complexity/accuracy trade-offs compared to state-of-the-art CNNs and vision transformers on ImageNet. Code is available at https://github.com/raoyongming/DynamicViT
Forward citations
Cited by 5 Pith papers
-
CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models
CRAFT recursively merges video tokens with training-free similarity selection plus learnable gated fusion, retaining ~97% of average accuracy at 8x compression across six benchmarks.
-
Searching for Task-Specific Vision Paths: Evolutionary Block Pruning Across Vision-Language Models
Evolutionary search improves fixed-budget vision-block skipping, but capability-specific routes are unstable and fail to transfer across models and sources.
-
Foveation-Guided Dynamic Token Selection for Robust and Efficient Vision Transformers
FDT adds foveation and binary fixation modules to DeiT so multi-scale tokens are selected dynamically in one pass, improving ImageNet100 accuracy, MACs, and robustness without adversarial training.
-
Compact Vision Transformer by Reduction of Kernel Complexity
KCR-Transformer replaces standard transformer blocks with channel-pruned MLPs plus a truncated-nuclear-norm regularizer, reporting better ImageNet top-1 accuracy at lower FLOPs.
-
CascadeFormer: A Family of Two-stage Cascading Transformers for Skeleton-based Human Action Recognition
A masked-pretrained skeleton transformer with a second fine-tuning transformer and cross-attention fusion reaches 94.66% on Penn Action, 91.16% on N-UCLA, and 81.01%/88.17% on NTU RGB+D 60 cross-subject/cross-view.
Discussion (0). Sign in to comment.