REVIEW 6 cited by
CvT: Introducing Convolutions to Vision Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present in this paper a new architecture, named Convolutional vision Transformer (CvT), that improves Vision Transformer (ViT) in performance and efficiency by introducing convolutions into ViT to yield the best of both designs. This is accomplished through two primary modifications: a hierarchy of Transformers containing a new convolutional token embedding, and a convolutional Transformer block leveraging a convolutional projection. These changes introduce desirable properties of convolutional neural networks (CNNs) to the ViT architecture (\ie shift, scale, and distortion invariance) while maintaining the merits of Transformers (\ie dynamic attention, global context, and better generalization). We validate CvT by conducting extensive experiments, showing that this approach achieves state-of-the-art performance over other Vision Transformers and ResNets on ImageNet-1k, with fewer parameters and lower FLOPs. In addition, performance gains are maintained when pretrained on larger datasets (\eg ImageNet-22k) and fine-tuned to downstream tasks. Pre-trained on ImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7\% on the ImageNet-1k val set. Finally, our results show that the positional encoding, a crucial component in existing Vision Transformers, can be safely removed in our model, simplifying the design for higher resolution vision tasks. Code will be released at \url{https://github.com/leoxiaobin/CvT}.
Forward citations
Cited by 6 Pith papers
-
ClawEnvKit: Automatic Environment Generation for Claw-Like Agents
EVT improves the RMT backbone by using Euclidean-distance attention decay and 1D token grouping, achieving 86.6% top-1 on ImageNet-1K at 384×384 resolution.
-
JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment
JAM is a 530M-parameter flow-matching song generator that adds word- and phoneme-level timing control and duration control, achieving strong lyric fidelity and musicality scores when ground-truth timings are provided.
-
Dual-attention ResNet outperforms transformers in HER2 prediction on DCE-MRI
An attention-based ResNet reports AUC 0.74 for HER2 prediction from DCE-MRI, beating transformer baselines in this study, though non-matched aggregation and a post-hoc threshold weaken the comparison.
-
Single Domain Generalization in Diabetic Retinopathy: A Neuro-Symbolic Learning Approach
KG-DG fuses YOLO-derived lesion features with a frozen ViT via confidence-based fusion and claims gains in diabetic retinopathy domain generalization, but the central KL-divergence mechanism and the MDG headline are c...
-
Residual Transformer Fusion Network for Salt and Pepper Image Denoising
RTF-Net, combining residual blocks with a convolutional vision transformer, reports the highest PSNR for salt-and-pepper denoising on most tested images.
-
Enhancing compact convolutional transformers with super attention
A compact convolutional transformer replacing SDPA with token mixing attention reports higher CIFAR100 accuracy, but the method is essentially prior token mixing and the comparison lacks error bars, code, and a strong...
Discussion (0). Continue with ORCID to comment.