Pith. sign in

REVIEW 4 major objections 6 minor 64 references

MVFormer: Diversifying Feature Normalization and Token Mixing for Efficient Vision Transformers

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Three normalization views push vision transformers past conv-based rivals at equal cost.

desk verdict A well-run incremental engineering paper whose headline SOTA claim rests on single-run margins; MVN is Switchable Normalization restated, and the missing citation needs fixing. read the letter →

arxiv 2411.18995 v1 pith:XULYDNKJ submitted 2024-11-28 cs.CV

Jongseong Bae , Susang Kim , Minsu Cho , Ha Young Kim This is my paper · ORCID
classification cs.CV
keywords visiontransformermulti-viewnormalizationtokenmixingconvolution-basedViTMetaFormerimageclassificationobjectdetectionsemanticsegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that efficient vision transformers leave a resource on the table in the normalization step. It proposes MVFormer, which replaces the single normalization inside each MetaFormer block with a learnable weighted blend of batch, layer, and instance normalization, and replaces the token mixer with a three-scale depthwise convolution whose receptive fields shrink and shift across stages. On ImageNet-1K, MVFormer-T/S/B reach 83.4%, 84.3%, and 84.6% top-1 accuracy, edging out the previous convolution-based state of the art by 0.1 to 0.4 percentage points with equal or fewer parameters and MACs, and it also reports higher mAP on COCO object detection and instance segmentation and higher mIoU on ADE20K semantic segmentation under similar budgets. The paper's central claim is that diversifying both the normalization view and the mixing scale is what buys these gains.

What carries the argument

The central object is the MVFormer block, a MetaFormer block in which the token-mixer and MLP subblocks both use multi-view normalization (MVN) as their normalization layer. MVN computes three differently normalized copies of the input (batch, layer, and instance) and combines them with learnable per-channel weights before a single affine transform, giving the token mixer access to batch-level, channel-level, and sample-level statistical views at negligible parameter cost. The token mixer, MVTM, is an inverted separable convolution with the depthwise layer split into three channel groups of different kernel sizes; stage specificity sets the channel split ratio (from 50:50:0 in stage 1 to 0:50:50 in stage 4) and shrinks the global kernel size (55x55 down to 7x7) across stages, steering early stages toward local mixing and later stages toward global mixing.

What would settle it

Train MVFormer-T and ConvFormer-S18 from scratch on ImageNet-1K with the same code, the same DeiT-style recipe, and at least five random seeds, then compare the mean top-1 accuracy with confidence intervals. If the 95% confidence intervals overlap or the ordering flips, the central superiority claim is falsified.

Watch

Extended reading notes

Core claim

MVFormer outperforms state-of-the-art convolution-based vision transformers on image classification, object detection, instance segmentation, and semantic segmentation while staying within the same or lower parameter and MAC budgets. The design combines two ideas: multi-view normalization (MVN), a learnable weighted sum of batch-, layer-, and instance-normalized features within every MetaFormer subblock, and a multi-view token mixer (MVTM), a channel-wise depthwise convolution split into local (3x3), intermediate (7x7), and global filters whose channel ratios and global kernel sizes are tuned per stage.

Load-bearing premise

The claim that MVFormer outperforms prior convolution-based vision transformers depends on the assumption that the reported accuracy gaps of 0.1 to 0.4 percentage points over the closest baselines are larger than run-to-run training noise from a single training run.

Editorial extensions

If this is right

  • If the central claim holds, MVN is a cheap drop-in module: it improved top-1 accuracy by 0.2 percentage points when substituted for layer norm in Swin-T, ConvFormer-S18, ConvNeXt-T, and PoolFormer-S36, and by 0.2 points when substituted for batch norm in ResNet50.
  • MVTM's three-scale split is responsible for reliable gains beyond the baseline, and both stage-specific channel ratios and stage-specific global kernel sizes contribute independently; removing either or both degrades ImageNet top-1 accuracy by 0.08 to 0.15 points.
  • The strong downstream results imply the normalization and mixing diversity transfers well to dense prediction, not just classification: MVFormer-T under Mask R-CNN 1x reaches 46.2 box AP and 42.1 mask AP with fewer parameters and MACs than prior backbones.
  • At 384x384 fine-tuning, all three MVFormer variants keep or extend their advantage over ConvFormer, suggesting the benefits are not specific to 224x224 training.
  • The paper's ablation shows instance normalization alone hurts (78.83% top-1) but improves when combined with batch or layer norm, so the synergy among normalizations is what MVN is designed to exploit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because MVN also improves CNNs like ResNet50, the principle of mixing normalizations may be a general training stabilizer that does not depend on the token-mixing mechanism, so it could be tested in pure attention-based ViTs as well.
  • The derived benefit may stem mostly from the LN component, which dominates the learned weights in most blocks (per Figure 3); a controlled study that freezes LN and only adds a small BN or IN branch could isolate whether the synergy is genuinely mutual.
  • The stage-specific global kernel in stage 1 (55x55 on a 56x56 feature map) is effectively whole-image mixing, yet later stages use 7x7 on a 7x7 feature map, meaning the static convolution still cannot perform content-dependent global interactions; replacing the global filter with a lightweight attention mechanism in the last stage might further push efficiency and accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MVFormer, a convolution-based vision transformer built on the MetaFormer scheme, with two new components: Multi-View Normalization (MVN), which computes a learnable weighted sum of batch-, layer-, and instance-normalized features, and Multi-View Token Mixer (MVTM), a multiscale depthwise convolution with local, intermediate, and global filters whose channel split ratios and global kernel sizes vary by stage. The model is evaluated on ImageNet-1K classification (83.4/84.3/84.6% top-1 for T/S/B variants), COCO object detection and instance segmentation, and ADE20K semantic segmentation, reporting small gains over ConvFormer baselines at matched or lower parameters and MACs. Ablations on a smaller xT model support the individual contributions of MVN and MVTM, and the learned normalization weights are analyzed.

Significance. If reproducible, the paper offers a modest architectural insight: combining multiple normalization views with stage-specific multiscale convolution can improve the efficiency-accuracy trade-off of convolution-based ViTs. The MVN module's consistent +0.2% improvement when plugged into several existing architectures (Table 7) is a useful empirical result, and the ablations in Tables 5-8 are fairly detailed. However, the headline SOTA claim rests on single-run differences of 0.1-0.4% top-1 in Table 2, with no variance estimates or released code; this currently limits the significance to a suggestive result rather than a demonstrated improvement.

major comments (4)
  1. [Section 4.1, Table 2] The central claim that MVFormer-T/S/B outperforms ConvFormer-S18/S36/M36 relies on top-1 margins of 0.4%, 0.2%, and 0.1%, respectively. The paper reports a single run per configuration and provides no confidence intervals, multiple seeds, or released code. At this performance level, ImageNet-1K top-1 accuracy is known to vary by roughly 0.2-0.3% run-to-run under the DeiT-style recipe used here, so some of the reported margins are within the noise level. The authors should report mean +/- standard deviation over at least three independent runs for the main comparisons, or release code and checkpoints so that the results can be independently reproduced; without this, the 'outperforms SOTA' claim is not yet supported.
  2. [Section 4.1, Table 2] It is not stated whether the ConvFormer and other baseline numbers in Table 2 were retrained under the exact MVFormer recipe (300 epochs, batch size 4096, learning rate 4e-3, stochastic depth rates, RandAugment, Mixup/CutMix, and ResScale for the last two stages). If the baselines are taken from their original papers, differences in training recipes can easily shift top-1 accuracy by more than the reported margins; batch size and stochastic depth changes alone are known to move ImageNet top-1 by 0.1-0.3%. The authors must either retrain the relevant baselines under the same codebase and hyperparameters, or clearly state that published numbers are quoted and justify that the comparison is controlled.
  3. [Section 4.1 and Section 4.4] The recipe for the main MVFormer models includes ResScale in the last two stages, but no ablation isolates its contribution. The ablation study in Table 5 uses the MVFormer-xT model with the training configuration of Table 10, which does not list ResScale; therefore the gains of the full models in Table 2 relative to ConvFormer could be partly due to ResScale rather than to the proposed MVN and MVTM modules. The authors should either add an ablation of ResScale on the same models, or remove it from the main comparison.
  4. [Appendix F, Algorithms 1 and 2] The PyTorch-style pseudocode contains API errors: `nn.BatchNorm(affine_shape, affine=False)` and `nn.InstanceNorm(affine_shape, affine=False)` pass a shape where the number of features is expected, and `nn.LayerNorm(affine_shape, elementwise_affine=False)` may also be incorrect if `affine_shape` is a tuple. Since no code is released, this pseudocode is the only implementation reference and should be corrected, or the actual code should be provided.
minor comments (6)
  1. [Section 3.2.2] There is a typo, 'convonlution-based', which should be 'convolution-based'.
  2. [Table 1] The 'Global kernel size' row is garbled; it should be formatted to clearly show the per-stage values (e.g., 55x1/1x55, 27x1/1x27, 13x1/1x13, 7x7).
  3. [Abstract and Section 1] The terms 'state-of-the-art' and 'SOTA' are used despite the absence of error bars and the small margins; consider softening to 'competitive' unless statistical significance is demonstrated.
  4. [Section 4.4.3] The sentence 'MVN displays impressive generalization, significantly improving the original performance' uses 'significantly' without any statistical test; 'consistently improves by 0.2%' would be more accurate.
  5. [Section 2.1] The claim that MVN is 'the first study of a normalization integration paradigm in ViTs' is strong; please either temper it or support it with a more comprehensive review of related normalization-combination work.
  6. [Section 4.1 and Table 10] The main text says EMA is used when fine-tuning to 384x384 resolution, but Table 10 lists 'EMA decay rate None' in the training configuration; please clarify that EMA is applied only during fine-tuning.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MVFormer's reported gains are empirical outcomes, not predictions derived from fitted inputs or self-citations.

full rationale

The paper is an empirical architecture study: MVN (Eq. 6) and MVTM (Eqs. 7-11) are proposed modules, and all headline numbers are measured training results on ImageNet-1K, COCO, and ADE20K. The learnable normalization weights in MVN are fit during training, but the paper never renames those fitted weights as a prediction; Section 4.4.5 only reports the learned weights post hoc. The ablation studies choose module configurations and combinations by validation accuracy, which is standard model selection rather than circular reasoning, and the final SOTA claims are directly reported experimental accuracies rather than derived quantities. The only self-citation is reference [19], a related-work item on relational self-attention co-authored by Minsu Cho; it is cited merely as one example of attention-variant methods in the introduction and is not load-bearing for any conclusion. Concerns about the 0.1-0.4% margins, missing seeds, and whether baseline numbers were reproduced under identical recipes are legitimate reproducibility and benchmarking risks, but they are not circularity: the comparison does not reduce to the paper's own definitions, fitted parameters, or a self-citation chain.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claims rest on a small number of hand-designed architectural hyperparameters (stage-specific channel ratios and kernel sizes) and on standard empirical assumptions of deep learning (that validation accuracy on ImageNet transfers to other tasks, and that the training recipe is held fair). The learnable normalization weights are trained on the same benchmarks used for evaluation, so they do not provide independent confirmation of the underlying hypothesis.

free parameters (4)
  • Learnable normalization mixture weights (alpha_BN, alpha_LN, alpha_IN) = learned during training
    Introduced in Eq. 6; these per-channel weights are trained on the same datasets used for evaluation, so they are not independent predictions.
  • Stage-wise local:intermediate:global channel split ratios = 50:50:0, 25:50:25, 25:50:25, 0:50:50
    Hand-designed for each stage (Table 1) and validated via ablations on ImageNet-1K; these ratios directly control the receptive field distribution of the token mixer.
  • Stage-wise global filter kernel sizes = 55x1/1x55, 27x1/1x27, 13x1/1x13, 7x7
    Hand-designed per stage (Table 1) to adjust global mixing scope; selected based on performance on ImageNet validation.
  • Local and intermediate kernel sizes = 3x3 and 7x7
    Fixed choices for the local and intermediate branches of MVTM; not derived from first principles.
assumptions (3)
  • domain assumption Combining BN, LN, and IN yields feature diversity beneficial for token mixing.
    Motivated by a visual example in Fig. 1, but not rigorously derived; the paper assumes this diversity is useful.
  • domain assumption Stage-specific receptive field sizes improve efficiency and accuracy.
    Based on prior work [29,58] that local mixing is preferred in early stages and global in later stages; adopted without a proof for the specific ratios and kernels.
  • domain assumption The standard DeiT training recipe (300 epochs, augmentations, etc.) transfers fairly to the proposed architecture.
    The paper uses the same training hyperparameters as prior work, assuming the architecture does not require different regularization or optimization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MVFormer: Diversifying Feature Normalization and Token Mixing for Efficient Vision Transformers." pith.science (2026). https://pith.science/paper/XULYDNKJ

@misc{pith2026241118995,
  author       = {Pith},
  title        = {Pith review of: MVFormer: Diversifying Feature Normalization and Token Mixing for Efficient Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XULYDNKJ}},
  note         = {Machine review of arXiv:2411.18995}
}
read the original abstract

Active research is currently underway to enhance the efficiency of vision transformers (ViTs). Most studies have focused solely on effective token mixers, overlooking the potential relationship with normalization. To boost diverse feature learning, we propose two components: a normalization module called multi-view normalization (MVN) and a token mixer called multi-view token mixer (MVTM). The MVN integrates three differently normalized features via batch, layer, and instance normalization using a learnable weighted sum. Each normalization method outputs a different distribution, generating distinct features. Thus, the MVN is expected to offer diverse pattern information to the token mixer, resulting in beneficial synergy. The MVTM is a convolution-based multiscale token mixer with local, intermediate, and global filters, and it incorporates stage specificity by configuring various receptive fields for the token mixer at each stage, efficiently capturing ranges of visual patterns. We propose a novel ViT model, multi-vision transformer (MVFormer), adopting the MVN and MVTM in the MetaFormer block, the generalized ViT scheme. Our MVFormer outperforms state-of-the-art convolution-based ViTs on image classification, object detection, and instance and semantic segmentation with the same or lower parameters and MACs. Particularly, MVFormer variants, MVFormer-T, S, and B achieve 83.4%, 84.3%, and 84.6% top-1 accuracy, respectively, on ImageNet-1K benchmark.

Figures

Figures reproduced from arXiv: 2411.18995 by the authors.

Figure 1
Figure 1. Visualization of normalized images from BN, LN, IN, and their averages. These illustrate that BN and IN maintain the detailed spatial distribution of the input image, whereas LN overly smooths the image. We can intuitively observe the spatially smoothed output, including the local details, by taking a simple average. ers, to improve or substitute self-attention. Some stud￾ies [4, 10, 19, 22, 50, 54] have proposed at… view at source ↗
Figure 2
Figure 2. Overall architecture of the proposed MVFormer and MVFormer block. Similar to MetaFormer, each block of MVFormer adopts a hierarchical architecture with four stages. Each Stagej comprises Nj blocks with a feature dimension Cj . The MVFormer block consists of two main components, MVN and MVTM, which can be compared to the MetaFormer block. able weighted summation of three normalized features ob￾tained through BN, LN, … view at source ↗
Figure 3
Figure 3. The average values of αLN , αBN and αIN for each block in MVFormer-S [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: , we observe that the overall learning trends of all [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Activation maps generated by Grad-CAM for the ConvFormer-S18 and MVFormer-T models [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Visualization of normalized images from BN, LN, IN, and MVN [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 42 canonical work pages

  1. [1]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. Layer normalization. arXiv preprint arXiv:1607.06450,

    arXiv

  2. [2]

    Mmdetection: Open mmlab detection tool- box and benchmark

    Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection tool- box and benchmark. arXiv preprint arXiv:1906.07155, 2019. 6

    arXiv 1906

  3. [3]

    Cyclemlp: A mlp-like architecture for dense prediction

    Shoufa Chen, Enze Xie, Chongjian Ge, Ding Liang, and Ping Luo. Cyclemlp: A mlp-like architecture for dense prediction. arXiv preprint arXiv:2107.10224, 2021. 1

    arXiv 2021

  4. [4]

    Twins: Revisiting the design of spatial attention in vision transformers, 2021

    Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haib- ing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers, 2021. 1

    work page 2021

  5. [5]

    Mmsegmentation: Open- mmlab semantic segmentation toolbox and benchmark,

    MMSegmentation Contributors. Mmsegmentation: Open- mmlab semantic segmentation toolbox and benchmark,

  6. [6]

    Randaugment: Practical automated data augmen- tation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmen- tation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703, 2020. 5

    work page 2020

  7. [7]

    Coatnet: Marrying convolution and attention for all data sizes

    Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems , 34:3965–3977, 2021. 1

    work page 2021

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. 2, 5, 1

    work page 2009

Show all 64 references
  1. [9]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint a...

    2010 arXiv

  2. [10]

    Multiscale vision transformers

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feicht- enhofer. Multiscale vision transformers. arXiv preprint arXiv:2104.11227, 2021. 1, 5

    2021 arXiv

  3. [11]

    Segnext: Rethinking convolutional attention design for semantic segmentation,

    Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation,

  4. [12]

    Visual attention network

    Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Visual attention network. arXiv preprint arXiv:2202.09741, 2022. 3, 6, 7

    2022 arXiv

  5. [13]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 7, 8

    2016

  6. [14]

    Mask r-cnn, 2018

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn, 2018. 6

    2018

  7. [15]

    Deep networks with stochastic depth

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kil- ian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision , pages 646–661. Springer, 2016. 5

    2016

  8. [16]

    Arbitrary style transfer in real-time with adaptive instance normalization, 2017

    Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization, 2017. 2

    2017

  9. [17]

    Batch renormalization: Towards reducing minibatch dependence in batch-normalized models, 2017

    Sergey Ioffe. Batch renormalization: Towards reducing minibatch dependence in batch-normalized models, 2017. 2

    2017

  10. [18]

    Batch normalization: Accelerating deep network training by reducing internal co- variate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal co- variate shift. arXiv preprint arXiv:1502.03167, 2015. 1, 2, 3

    2015 arXiv

  11. [19]

    Relational self-attention: What’s missing in attention for video understanding

    Manjin Kim, Heeseung Kwon, Chunyu Wang, Suha Kwak, and Minsu Cho. Relational self-attention: What’s missing in attention for video understanding. Advances in Neural Information Processing Systems, 34:8046–8059, 2021. 1

    2021

  12. [20]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,

    arXiv

  13. [21]

    Panoptic feature pyramid networks

    Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Doll´ar. Panoptic feature pyramid networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6399–6408, 2019. 7

    2019

  14. [22]

    Mvitv2: Improved multiscale vision transformers for classification and detection

    Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Man- galam, Bo Xiong, Jitendra Malik, and Christoph Feichten- hofer. Mvitv2: Improved multiscale vision transformers for classification and detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2022

  15. [23]

    As- mlp: An axial shifted mlp architecture for vision

    Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao. As- mlp: An axial shifted mlp architecture for vision. arXiv preprint arXiv:2107.08391, 2021. 3

    2021 arXiv

  16. [24]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740–755. Springer, 2014. 6

    2014

  17. [25]

    Focal loss for dense object detection, 2018

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection, 2018. 6

    2018

  18. [26]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021. 1, 2, 3, 5, 6, 7, 8

    2021

  19. [27]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 11976–11986,

  20. [28]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 5

    2017 arXiv

  21. [29]

    How do vision transformers work?, 2022

    Namuk Park and Songkuk Kim. How do vision transformers work?, 2022. 1, 2, 5

    2022

  22. [30]

    Semantic image synthesis with spatially-adaptive nor- malization

    Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive nor- malization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2337–2346,

  23. [31]

    Pytorch: An im- perative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems ...

    2019

  24. [32]

    Acceleration of stochastic approximation by averaging

    Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838–855, 1992. 5

    1992

  25. [33]

    What makes for good tokenizers in vision transformer? IEEE Transactions on Pattern Analysis and Machine Intelligence,

    Shengju Qian, Yi Zhu, Wenbo Li, Mu Li, and Jiaya Jia. What makes for good tokenizers in vision transformer? IEEE Transactions on Pattern Analysis and Machine Intelligence,

  26. [34]

    Do vision trans- formers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34:12116–12128,

    Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision trans- formers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34:12116–12128,

  27. [35]

    Mobilenetv2: Inverted residuals and linear bottlenecks, 2019

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh- moginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks, 2019. 3, 4

    2019

  28. [36]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017. 1

    2017

  29. [37]

    Powernorm: Rethinking batch normaliza- tion in transformers

    Sheng Shen, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Powernorm: Rethinking batch normaliza- tion in transformers. In International Conference on Ma- chine Learning, pages 8741–8751. PMLR, 2020. 2

    2020

  30. [38]

    Normformer: Improved transformer pretraining with extra normalization

    Sam Shleifer, Jason Weston, and Myle Ott. Normformer: Improved transformer pretraining with extra normalization. arXiv preprint arXiv:2110.09456, 2021. 5

    2021 arXiv

  31. [39]

    Evalnorm: Esti- mating batch normalization statistics for evaluation, 2019

    Saurabh Singh and Abhinav Shrivastava. Evalnorm: Esti- mating batch normalization statistics for evaluation, 2019. 2

    2019

  32. [40]

    Rethinking the inception archi- tecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception archi- tecture for computer vision. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 2818–2826, 2016. 5

    2016

  33. [41]

    Mlp-mixer: An all-mlp architecture for vision

    Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lu- cas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems, 34:2...

  34. [42]

    Resmlp: Feedforward networks for image clas- sification with data-efficient training, 2021

    Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izac- ard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Herv´e J´egou. Resmlp: Feedforward networks for image clas- sification with data-efficient training, 2021. 3

    2021

  35. [43]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. In International Conference on Machine Learning , pages 10347–10357. PMLR, 2021. 2, 3, 5, 6

    2021

  36. [44]

    Maxvit: Multi-axis vision transformer, 2022

    Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer, 2022. 1

    2022

  37. [45]

    In- stance normalization: The missing ingredient for fast styliza- tion, 2017

    Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. In- stance normalization: The missing ingredient for fast styliza- tion, 2017. 2, 3, 1

    2017

  38. [46]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 1

    2017

  39. [47]

    When shift operation meets vision transformer: An extremely simple alternative to attention mechanism

    Guangting Wang, Yucheng Zhao, Chuanxin Tang, Chong Luo, and Wenjun Zeng. When shift operation meets vision transformer: An extremely simple alternative to attention mechanism. arXiv preprint arXiv:2201.10801, 2022. 1

    2022 arXiv

  40. [48]

    Ri- former: Keep your vision backbone effective but removing token mixer

    Jiahao Wang, Songyang Zhang, Yong Liu, Taiqiang Wu, Yu- jiu Yang, Xihui Liu, Kai Chen, Ping Luo, and Dahua Lin. Ri- former: Keep your vision backbone effective but removing token mixer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages...

    2023

  41. [49]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , p...

    2021

  42. [50]

    PVT v2: Improved baselines with pyramid vision transformer

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. PVT v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415–424, 2022. 1

    2022

  43. [51]

    Activemlp: An mlp-like architecture with active token mixer

    Guoqiang Wei, Zhizheng Zhang, Cuiling Lan, Yan Lu, and Zhibo Chen. Activemlp: An mlp-like architecture with active token mixer. arXiv preprint arXiv:2203.06108, 2022. 3

    2022 arXiv

  44. [52]

    Con- vnext v2: Co-designing and scaling convnets with masked autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Con- vnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16133– 161...

    2023

  45. [53]

    Towards stabilizing batch statistics in backward propagation of batch normalization, 2020

    Junjie Yan, Ruosi Wan, Xiangyu Zhang, Wei Zhang, Yichen Wei, and Jian Sun. Towards stabilizing batch statistics in backward propagation of batch normalization, 2020. 2

    2020

  46. [54]

    Focal self-attention for local-global interactions in vision transformers

    Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. arXiv preprint arXiv:2107.00641, 2021. 1, 5, 6

    2021 arXiv

  47. [55]

    Focal modulation networks, 2022

    Jianwei Yang, Chunyuan Li, Xiyang Dai, Lu Yuan, and Jian- feng Gao. Focal modulation networks, 2022. 1, 3, 6

    2022

  48. [56]

    Leveraging batch normalization for vision transformers

    Zhuliang Yao, Yue Cao, Yutong Lin, Ze Liu, Zheng Zhang, and Han Hu. Leveraging batch normalization for vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) , pages 413–422,

    2021

  49. [57]

    S2-mlp: Spatial-shift mlp architecture for vision

    Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. S2-mlp: Spatial-shift mlp architecture for vision. In Pro- ceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision, pages 297–306, 2022. 3

    2022

  50. [58]

    Metaformer is actually what you need for vision

    Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10819–10829, 2022. 1, 2, 3, 5, 6, 7, 8

    2022

  51. [59]

    Metaformer baselines for vision, 2022

    Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. Metaformer baselines for vision, 2022. 1, 2, 3, 4, 5, 6, 7, 8

    2022

  52. [60]

    Inceptionnext: When inception meets convnext, 2023

    Weihao Yu, Pan Zhou, Shuicheng Yan, and Xinchao Wang. Inceptionnext: When inception meets convnext, 2023. 1, 2, 3, 4, 6, 7

    2023

  53. [61]

    Cutmix: Regu- larization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regu- larization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 6023–6032, 2019. 5

    2019

  54. [62]

    mixup: Beyond empirical risk minimiza- tion

    Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion. arXiv preprint arXiv:1710.09412, 2017. 5

    2017 arXiv

  55. [63]

    Random erasing data augmentation

    Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceed- ings of the AAAI conference on artificial intelligence , pages 13001–13008, 2020. 5

    2020

  56. [64]

    Semantic under- standing of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302–321, 2019. 7 MVFormer: Diversifying Feature Normalization an...

    2019

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.