Pith. sign in

REVIEW 4 major objections 6 minor 34 references

MSCViT: A Small-size ViT architecture with Multi-Scale Self-Attention Mechanism for Tiny Datasets

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes MSCViT, a compact hybrid vision transformer that trains from scratch on tiny datasets and claims to outdo comparable CNNs and transformers, reaching 84.68% on CIFAR-100 with 14M parameters and no pretraining.

desk verdict A plausible hybrid ViT for small datasets, but the headline accuracy claim rests on a comparison table that contradicts its own 'same setting' statement. read the letter →

arxiv 2501.06040 v2 pith:E6M5UKD7 submitted 2025-01-10 cs.CV

Bowei Zhang , Yi Zhang This is my paper · ORCID
classification cs.CV
keywords VisionTransformerConvolutionalNeuralNetworksMulti-scaleself-attentionTinydatasetsWaveletconvolutionLocalfeatureextractionLightweightattention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a small Vision Transformer (ViT) can be trained from scratch on tiny datasets if it is redesigned around multi-scale attention and convolutional feature fusion. The authors propose MSCViT, which replaces positional encoding with a local feature extraction block, merges tokens at different rates per attention head, and injects wavelet-extracted high-frequency features into selected channels. On CIFAR-100, MSCViT-S reaches 84.68% top-1 accuracy with 14.0M parameters and 2.5 GFLOPs, beating comparable-size CNNs, hybrid models, and transformer baselines without any large-scale pretraining. The practical payoff is a parameter-efficient ViT for small-data applications where pretraining data or compute is scarce.

What carries the argument

The load-bearing idea is multi-scale token fusion inside each attention block, coupled with two local-feature injections. LMSSA spatially reduces K and V by different factors (8, 4, 2, 1) in different heads, so coarse heads capture global structure while fine heads retain detail, at roughly $4nd^2 + \sum_i 2n^2d/R_i^2$ FLOPs. LFE replaces the learnable positional encoding with a residual depthwise-convolution block that carries translation equivariance into the transformer. CFF selects a subset of channels, applies wavelet convolution followed by a convolution with stage-dependent kernel size, and concatenates the result with standard attention output. Together they let the network induce locality without pretraining and keep the token count low.

What would settle it

Retrain the unmarked baselines in Table 3 (e.g., PVT-T, CMT-Ti, BiFormer-T) on CIFAR-100 with exactly the MSCViT training recipe — DeiT augmentation, 300 epochs, AdamW, 224x224 resolution — and check whether any reaches 84.68% or higher; if so, the claim of universal superiority under the same setting is refuted.

Watch

Extended reading notes

Core claim

MSCViT is a hybrid vision transformer that outperforms existing models of similar size on tiny image datasets when trained from scratch. The architecture combines three modifications to the standard ViT block: a Local Feature Extraction (LFE) module that replaces positional encoding with a pair of depthwise convolutions; a Lightweight Multi-scale Self-Attention (LMSSA) module that applies different token fusion coefficients ($R=8,4,2,1$) across attention heads and reduces key/value spatial size with depthwise convolutions; and a Convolutional Feature Fusion (CFF) module that selects redundant channels and processes them with wavelet convolution to capture high-frequency shape information. On CIFAR-100 the small variant achieves 84.68% top-1 accuracy with 14.0M parameters and 2.5 GFLOPs, and the tiny variant reaches 80.11% with only 3.8M parameters and 0.5 GFLOPs, both without pretraining. The paper concludes that the gaps between ViT and CNNs on small data can be closed by architectural changes that inject locality and multi-granularity attention, rather than by larger data or pretraining.

Load-bearing premise

The central claim that MSCViT exceeds all similar-size models on tiny datasets assumes that every baseline in Table 3 was trained under the same recipe as MSCViT (DeiT-style augmentation, 300 epochs, AdamW, 224x224 input), even though some reported numbers are quoted from the original papers rather than reproduced, so unequal training settings could explain the accuracy gaps instead of architecture.

Editorial extensions

If this is right

  • On CIFAR-100, MSCViT-T (3.8M params) already surpasses several larger baselines, suggesting that small-data ViTs need not trade much accuracy for compactness.
  • The reported gains on Flowers102 and Chaoyang (medical) indicate the recipe transfers across domains with very different class counts and sample imbalances.
  • Replacing positional encoding with a convolution block did not degrade accuracy, so future ViT designs for small data can omit PE without penalty.
  • The multi-scale attention with fusion coefficients offers a cheap way to model scale: a 10.7% parameter reduction and 5.2% FLOP reduction for only a 0.18% accuracy drop (200-epoch setting).
  • On Tiny ImageNet, MSCViT-S reaches 72.11%, outperforming CMT, BiFormer, and Shunted baselines, supporting the same claim at slightly larger scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The success of wavelet-based CFF suggests a broader testable recipe: frequency-split convolutions in transformer channels may be a general inductive bias for fine-grained classification under scarce data.
  • Because the scaling factors and kernel sizes were hand-searched (8/4/2/1 and 5x5/3x3), a systematic ablation on fusion schedules might yield further gains or transfer to other dataset regimes.
  • MSCViT could plausibly be combined with modern self-supervised pretraining or knowledge distillation; the paper only trains from scratch, so the component's value under those regimes is unknown.
  • The medical Chaoyang result hints that the architecture may be well suited to histopathology-like tasks with few samples per class, but that would need dedicated evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MSCViT, a compact vision-transformer backbone designed for image classification on small ('tiny') datasets. The architecture combines three modules: a Local Feature Extraction (LFE) block that replaces positional encoding, a Lightweight Multi-scale Self-Attention (LMSSA) module that reduces token counts via per-head fusion coefficients, and a Convolutional Feature Fusion (CFF) module that mixes wavelet-based and standard convolutions on selected channels. The authors report top-1 accuracies of 80.11%, 83.44%, and 84.68% on CIFAR-100 for MSCViT-T, -XS, and -S with 3.8M, 7.8M, and 14.0M parameters and 0.5, 1.0, and 2.5 GFLOPs, respectively, all trained from scratch with DeiT augmentation. Results are also reported on CIFAR-10, Flowers102, Chaoyang, Oxford-IIIT Pet, and Tiny ImageNet, together with ablations of the three modules. The central claim is that MSCViT exceeds all comparable-size ViT and hybrid models on tiny datasets (Section 4.3.1).

Significance. If the reported comparisons are fair, the paper offers a parameter-efficient ViT variant that is competitive on small data, a practically useful direction. The paper has several strengths: it evaluates on multiple tiny datasets, includes ablations that support a contribution from each proposed module, reports computational costs explicitly, and trains from scratch without large-scale pretraining. The architecture is clearly described and the reported numbers are internally consistent. However, the load-bearing comparison claim currently rests on a table that is not fully self-consistent about training protocols, and the ablation baseline appears to coincide with an existing model, so the evidence for the headline superiority claim is not yet convincing. Reproducibility would also be materially improved by releasing code and reporting variance over multiple runs.

major comments (4)
  1. [Section 4.3.1, Table 3] The statement 'All methods are tested and compared under the same setting' is contradicted by the table itself: the entries for CCT-7/3x1, HSViT-C3A4, HSViT-C4A8, and Swin-T are marked with an asterisk, indicating they are taken from the original papers. Those papers use different training schedules, augmentations, and native input resolutions (for example, 32x32 or 64x64 for CIFAR), so their GFLOPs and accuracies are not on the same computational or training basis as the 224x224 MSCViT runs. Because the headline claim that MSCViT 'exceeds all other models of similar sizes' rests entirely on this table, that claim is unsupported until the asterisked baselines are retrained under the paper's exact protocol (Section 4.2) or the comparison is reworked to state the protocol differences explicitly and the claim is narrowed accordingly.
  2. [Table 5, row 'none'] The first row of the ablation table reports GFLOPs of 2.1 and accuracy of 81.66, which are exactly the values listed for Shunted-T in Table 3. This exact coincidence suggests that the ablation baseline is not the paper's own backbone as described in Section 3.1 but rather a Shunted-T model. The paper must specify the exact backbone used in Section 4.4.1; if it is Shunted-T, the authors need to justify why that is the appropriate base for isolating the contributions of LFE, LMSSA, and CFF, and the interpretation of the ablation attributions would need to be revised. As written, the ablation does not clearly demonstrate gains over the proposed architecture's own default configuration.
  3. [Tables 3-5, 7] All accuracy values are reported as single numbers with no error bars, no multiple seeds, and no significance tests. Several comparisons involve small margins, such as the 0.18-point difference in Table 7 and the 0.08-point difference in Table 6. On small datasets like CIFAR-100, single-run differences of this size are typically within run-to-run variance. The paper should report mean and standard deviation over at least three independent training runs for the main comparisons and the ablations; otherwise, the claimed margins, especially the fine-grained ones, are not statistically grounded.
  4. [Sections 3.5, 4.4.2, 4.4.3, Table 2] The design choices central to the architecture, including the fusion coefficients R_i in LMSSA, the CFF kernel-size schedule (5x5 in early stages and 3x3 in later stages), and the stage depths, appear to have been selected using experiments on CIFAR-100 (for example, Table 6 chooses the kernel schedule and Table 2 fixes the R_i values). CIFAR-100 is also the dataset on which the main superiority claim is made. This creates a selection loop: the architecture is tuned to the headline dataset. The paper should state explicitly whether the Flowers102, Chaoyang, Oxford-IIIT Pet, and Tiny ImageNet results were obtained with these hyperparameters fixed before seeing those test sets, or report a validation-based selection procedure. Without this clarification, the cross-dataset generalization claim is weakened.
minor comments (6)
  1. [Equations (5)-(6)] The notation 'i ∈ 1, 2, 3, 4' should be 'i ∈ {1, 2, 3, 4}', and the dimension description following Equation (4) is garbled: 'turning the dimension of the feature map X into HW R2 R2C' should be written using standard tensor shapes with explicit division symbols.
  2. [Equation (7)] There is a typo: 'Sof tmax' should be 'Softmax'. In addition, the sentence after Equation (6), 'The, the calculation', is missing a clause and should be rephrased.
  3. [Table 4] The Tiny ImageNet results are reported without the same-setting caveat that appears for Table 3; it is unclear whether the baseline numbers (cmt_Ti, cmt_XS, BiFormer_T, Shunted_T) were rerun under the paper's protocol or taken from their original papers. This should be clarified.
  4. [Section 4.4.4] The sentence 'In section, we analyze the impact of lightweight self-attention mechanism on LMSSA' is incomplete; it presumably refers to Section 4.4.3 but reads as a missing cross-reference.
  5. [Table 4 and Figure 3] The model name 'Hybird' in Table 4 is a typo for 'Hybrid', and the figure caption 'The comparison of the models sizes and accuracies' should be 'model sizes'.
  6. [Section 4.3.1 and Figure 4] The Grad-CAM discussion states 'we captures the attention' and 'we captures the attention scattered in the background', which should be 'we capture' or 'our model captures'. More importantly, the qualitative heatmap comparison would be stronger if accompanied by a quantitative metric, such as the average drop or increase in confidence over a set of test images.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MSCViT's design is evaluated against external tiny datasets, and the flagged baseline/comparability issues are experimental validity concerns rather than derivation-by-construction.

full rationale

The paper's central claims are empirical: MSCViT reports accuracies on CIFAR-10, CIFAR-100, Flowers102, Chaoyang, Oxford-IIIT Pet, and Tiny ImageNet trained from scratch. The design choices, such as the fusion coefficients Ri = 8, 4, 2, 1 and the 5x5/3x3 kernel schedule, are selected via ablations on CIFAR-100 before final evaluation; this is conventional hyperparameter selection, not a fitted parameter renamed as a prediction. The headline result is additionally checked on several other datasets, so it is not defined in terms of the fitted choices. There is no self-citation chain, no imported uniqueness theorem, and no equation that reduces to its own input. The most serious issues are experimental rather than circular: Table 3 claims "All methods are tested and compared under the same setting" while asterisked entries (CCT-7/3x1, HSViT-C3A4, HSViT-C4A8, Swin-T) are taken from original papers, and the "none" ablation row in Table 5 exactly matches the Shunted-T row in Table 3 (81.66% accuracy, 2.1 GFLOPs). Those concerns undermine the fairness and reproducibility of the comparison, but they do not constitute a circular derivation: the paper is not defining its target result in terms of its inputs. The acknowledged limitation that ablation results are lower because they were run on a single GPU is likewise a reporting caveat, not a circular step. No load-bearing premise reduces to a self-citation or to a quantity fitted from the target benchmark, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central empirical result depends on three hand-tuned architectural schedules (fusion coefficients, kernel sizes, depths) plus the assumption that convolution can supply inductive bias and replace positional encoding. The kernel-size schedule is tuned on CIFAR-100, the same dataset used for the headline accuracy, which is the main free-parameter concern. The modules themselves are not independently grounded outside the paper's own ablations.

free parameters (3)
  • Fusion coefficients R_i = R = 8, 4, 2, 1 across attention heads
    Chosen by hand as the multi-scale fusion schedule in LMSSA (Section 3.3); no sweep is reported, and the schedule is applied to all datasets.
  • CFF kernel-size schedule = 5x5 in shallow stages, 3x3 in deep stages
    Selected via CIFAR-100 ablation (Section 4.4.2, Table 6); the winning schedule is then used in the final model whose CIFAR-100 accuracy is the headline result.
  • Stage depths and widths = Depths [1,2,4,1], [1,1,3,2], [2,2,4,2]; widths 32 to 512 depending on variant
    Manual scaling choices for the T, XS, and S variants (Table 2); no automatic search or justification beyond matching the target parameter budgets.
assumptions (4)
  • domain assumption Convolutional inductive bias is required for ViT to learn local features on tiny datasets.
    Stated in Section 1 and 2.2 based on prior work (CvT, CMT, LocalViT); the paper treats it as a given rather than proving it.
  • ad hoc to paper Wavelet convolution selectively preserves high-frequency shape information in a way that improves classification on tiny datasets.
    CFF (Section 3.4) relies on this; there is no ablation comparing wavelet convolution to a standard convolution of the same cost, and no citation for wavelet convolution methods.
  • domain assumption Replacing positional encoding with depthwise convolution (LFE) preserves order information in the token sequence.
    Supported only by the internal ablation in Table 8; the claim that convolution translation invariance substitutes for explicit PE is an assumption about how the model represents position.
  • ad hoc to paper All baseline models in the comparison tables were trained under the same protocol as MSCViT.
    Section 4.2 claims the same setting, but Table 3 contains starred entries taken verbatim from other papers; parity for unstarred baselines is not documented.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MSCViT: A Small-size ViT architecture with Multi-Scale Self-Attention Mechanism for Tiny Datasets." pith.science (2026). https://pith.science/paper/E6M5UKD7

@misc{pith2026250106040,
  author       = {Pith},
  title        = {Pith review of: MSCViT: A Small-size ViT architecture with Multi-Scale Self-Attention Mechanism for Tiny Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E6M5UKD7}},
  note         = {Machine review of arXiv:2501.06040}
}
read the original abstract

Vision Transformer (ViT) has demonstrated significant potential in various vision tasks due to its strong ability in modelling long-range dependencies. However, such success is largely fueled by training on massive samples. In real applications, the large-scale datasets are not always available, and ViT performs worse than Convolutional Neural Networks (CNNs) if it is only trained on small scale dataset (called tiny dataset), since it requires large amount of training data to ensure its representational capacity. In this paper, a small-size ViT architecture with multi-scale self-attention mechanism and convolution blocks is presented (dubbed MSCViT) to model different scales of attention at each layer. Firstly, we introduced wavelet convolution, which selectively combines the high-frequency components obtained by frequency division with our convolution channel to extract local features. Then, a lightweight multi-head attention module is developed to reduce the number of tokens and computational costs. Finally, the positional encoding (PE) in the backbone is replaced by a local feature extraction module. Compared with the original ViT, it is parameter-efficient and is particularly suitable for tiny datasets. Extensive experiments have been conducted on tiny datasets, in which our model achieves an accuracy of 84.68% on CIFAR-100 with 14.0M parameters and 2.5 GFLOPs, without pre-training on large datasets.

Figures

Figures reproduced from arXiv: 2501.06040 by the authors.

Figure 1
Figure 1. Performance of MSCViT on CIFAR-10 and CIFAR￾100. MSCViT performs better than some models with similar structures. with other models with similar scales, our model is compet￾itive without pre-training on large datasets. In summary, the contribution of this paper can be summarized as follows: • A Local Feature Extraction (LFE) module has been pro￾posed, which is used to capture local information of the intermediate fe… view at source ↗
Figure 2
Figure 2. The overall architecture of the proposed MSCViT. 2.3. ViT for tiny datasets Despite the favorable results on large-scale dataset (e.g. ImageNet-1K), most of the existing models cannot compete with CNNs if they are only trained on tiny datasets (e.g. CIFAR-100). To bridge the performance gap, Liu, Yahui, et al. [21] introduced self-supervised style training strategies and loss functions to carry out training on tiny … view at source ↗
Figure 3
Figure 3. The comparison of the model sizes and accuracies among different methods. (including fusion of convolutions) for comparison. The ex￾perimental results are shown in both [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The comparison of heatmaps of different methods generated by Grad-CAM [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Visual demonstration of the functions of the proposed LFE, CFF and LMSSA modules [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 21 canonical work pages

  1. [1]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weis- senborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    arXiv 2010

  2. [2]

    Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34:12116–12128, 2021

    Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34:12116–12128, 2021

    2021

  3. [3]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  4. [4]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In2008 Sixth Indian Y.Zhang,B: Preprint submitted to Elsevier Page 9 of 10 MSCViT: A Small-size ViT architecture with Multi-Scale Self-Attention Mechanism for Tiny Datasets conference on computer vision, graphics & image processing , pages 722–729. I...

    work page 2008

  5. [5]

    Hard sample aware noise robust learning for histopathology image classification

    Chuang Zhu, Wenkai Chen, Ting Peng, Ying Wang, and Mulan Jin. Hard sample aware noise robust learning for histopathology image classification. IEEE transactions on medical imaging , 41(4):881– 894, 2021

    work page 2021

  6. [6]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. InInternational conference on machine learning, pages 10347–10357. PMLR, 2021

    2021

  7. [7]

    Tokens-to- token vit: Training vision transformers from scratch on imagenet

    LiYuan,YunpengChen,TaoWang,WeihaoYu,YujunShi,Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to- token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision, pages 558–567, 2021

    work page 2021

  8. [8]

    Deformabledetr:Deformabletransformersforend-to-endobject detection

    XizhouZhu,WeijieSu,LeweiLu,BinLi,XiaogangWang,andJifeng Dai. Deformabledetr:Deformabletransformersforend-to-endobject detection. arXiv preprint arXiv:2010.04159, 2020

    arXiv 2010

Show all 34 references
  1. [9]

    End-to-end object detec- tion with transformers

    NicolasCarion,FranciscoMassa,GabrielSynnaeve,NicolasUsunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detec- tion with transformers. InEuropean conference on computer vision, pages 213–229. Springer, 2020

    2020

  2. [10]

    Crossvit: Cross-attention multi-scale vision transformer for image classifica- tion

    Chun-FuRichardChen,QuanfuFan,andRameswarPanda. Crossvit: Cross-attention multi-scale vision transformer for image classifica- tion. In Proceedings of the IEEE/CVF international conference on computer vision, pages 357–366, 2021

    2021

  3. [11]

    Localvit: Bringing locality to vision transformers

    YaweiLi,KaiZhang,JiezhangCao,RaduTimofte,andLucVanGool. Localvit: Bringing locality to vision transformers. arXiv preprint arXiv:2104.05707, 2021

    2021 arXiv

  4. [12]

    Pyramid vision transformer: A versatile backbone for dense prediction without con- volutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without con- volutions. InProceedings of the IEEE/CVF international conference on computer vision, p...

    2021

  5. [13]

    Swintransformer:Hierarchicalvision transformerusingshiftedwindows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, StephenLin,andBainingGuo. Swintransformer:Hierarchicalvision transformerusingshiftedwindows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012–10022, 2021

    2021

  6. [14]

    Biformer: Vision transformer with bi-level routing attention

    Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, and Ryn- son WH Lau. Biformer: Vision transformer with bi-level routing attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10323–10333, 2023

    2023

  7. [15]

    Cvt: Introducing convolutions to vision transformers

    Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 22–31, 2021

    2021

  8. [16]

    Twins: Revisiting the design of spatial attention in vision transformers.Advances in neural information processing systems, 34:9355–9366, 2021

    Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers.Advances in neural information processing systems, 34:9355–9366, 2021

    2021

  9. [17]

    Maxvit: Multi-axis vision transformer

    Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. InEuropean conference on computer vision,pages459–

  10. [18]

    Coatnet: Marrying convolution and attention for all data sizes.Advances in neural information processing systems, 34:3965–3977, 2021

    Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes.Advances in neural information processing systems, 34:3965–3977, 2021

    2021

  11. [19]

    Escaping the big data paradigm with compact transformers.arXiv preprint arXiv:2104.05704, 2021

    Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers.arXiv preprint arXiv:2104.05704, 2021

    2021 arXiv

  12. [20]

    Conformer: Local features coupling global representations for visual recognition

    Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. Conformer: Local features coupling global representations for visual recognition. InProceedings of the IEEE/CVF international conference on computer vision , pages 367– 376, 2021

    2021

  13. [21]

    Efficient training of visual transformers with small datasets

    Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri, and Marco Nadai. Efficient training of visual transformers with small datasets. Advances in Neural Information Processing Systems , 34:23818–23830, 2021

    2021

  14. [22]

    Vision transformerforsmall-sizedatasets

    Seung Hoon Lee, Seunghyun Lee, and Byung Cheol Song. Vision transformerforsmall-sizedatasets. arXiv preprint arXiv:2112.13492, 2021

    2021 arXiv

  15. [23]

    Transmcgc: a recast vision transformer for small-scale image classification tasks

    Jian-Wen Xiang, Min-Rong Chen, Pei-Shan Li, Hao-Li Zou, Shi- Da Li, and Jun-Jie Huang. Transmcgc: a recast vision transformer for small-scale image classification tasks. Neural Computing and Applications, 35(10):7697–7718, 2023

    2023

  16. [24]

    Accumulated trivial attention matters in vision transformers on small datasets

    Xiangyu Chen, Qinghao Hu, Kaidong Li, Cuncong Zhong, and Guanghui Wang. Accumulated trivial attention matters in vision transformers on small datasets. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,pages3984– 3992, 2023

    2023

  17. [25]

    Early convolutions help transformers see bet- ter

    Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see bet- ter. Advances in neural information processing systems , 34:30392– 30400, 2021

    2021

  18. [26]

    Efficientvit: Memory efficient vision transformer withcascadedgroupattention

    Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuqing Yang, Han Hu, and Yixuan Yuan. Efficientvit: Memory efficient vision transformer withcascadedgroupattention. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition , pages 14420– 14430, 2023

    2023

  19. [27]

    Pvt v2: Improved baselines with pyramid vision transformer

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415–424, 2022

    2022

  20. [28]

    Moganet: Multi-order gated aggregation network

    Siyuan Li, Zedong Wang, Zicheng Liu, Cheng Tan, Haitao Lin, Di Wu, Zhiyuan Chen, Jiangbin Zheng, and Stan Z Li. Moganet: Multi-order gated aggregation network. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  21. [29]

    Visual attention network

    Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Visual attention network. Computational Visual Media, 9(4):733–752, 2023

    2023

  22. [30]

    Hsvit: Horizontally scalable vision transformer

    ChenhaoXu,Chang-TsunLi,CheePengLim,andDouglasCreighton. Hsvit: Horizontally scalable vision transformer. arXiv preprint arXiv:2404.05196, 2024

    2024 arXiv

  23. [31]

    Convit: Improving vision trans- formers with soft convolutional inductive biases

    Stéphaned’Ascoli,HugoTouvron,MatthewLLeavitt,AriSMorcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision trans- formers with soft convolutional inductive biases. InInternational conference on machine learning, pages 2286–2296. PMLR, 2021

    2021

  24. [32]

    Cmt: Convolutional neural networks meet vision transformers

    JianyuanGuo,KaiHan,HanWu,YehuiTang,XinghaoChen,Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12175–12185, 2022

    2022

  25. [33]

    Levit: a vision transformerinconvnet’sclothingforfasterinference

    BenjaminGraham,AlaaeldinEl-Nouby,HugoTouvron,PierreStock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformerinconvnet’sclothingforfasterinference. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12259–12269, 2021

    2021

  26. [34]

    Shunted self-attention via multi-scale token aggregation

    SuchengRen,DaquanZhou,ShengfengHe,JiashiFeng,andXinchao Wang. Shunted self-attention via multi-scale token aggregation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10853–10862, 2022. Y.Zhang,B: Preprint submitted to Elsevier Page...

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.