Pith. sign in

REVIEW 3 major objections 6 minor 36 references

Maximizing the Position Embedding for Vision Transformers with Global Average Pooling

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Position embeddings counterbalance token embeddings in vision transformers, and preserving that counterbalance through the final LayerNorm improves accuracy.

desk verdict A simple GAP fix gives small, consistent accuracy gains in ViTs, but the paper's counterbalancing mechanism is not actually supported until a random-bias control rules out a generic learnable-bias effect. read the letter →

arxiv 2502.02919 v1 pith:AECS45S7 submitted 2025-02-05 cs.CV cs.LG

Wonjun Lee , Bumsub Ham , Suhyun Kim This is my paper · ORCID
classification cs.CVcs.LG
keywords positionembeddingvisiontransformerglobalaveragepoolinglayer-wisecounterbalancingLayerNormalizationMPVGimageclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper addresses why global average pooling (GAP) and layer-wise position embedding (PE) hurt each other when combined in vision transformers, even though each helps alone. It claims that in a layer-wise structure the PE increasingly counterbalances the token embedding values as layers deepen, but after the last layer the token embedding retains a directional imbalance that no longer gets compensated. The proposed fix, MPVG, is a one-line change: add the initial position embedding to the LayerNorm just before the classification head. On ImageNet-1K this yields consistent gains over the default model, ranging from +0.27 percentage points on Swin-Ti to +1.37 on DeiT-Ti, and also improves CIFAR-100, COCO object detection, and ADE20K segmentation. The paper's central contribution is the diagnosis of the counterbalancing role of PE and the demonstration that maintaining it matters.

What carries the argument

The central object is the Layer-wise position-embedding structure (introduced by LaPE) combined with GAP, and the one-line modification in Eq. (11). In that structure each layer applies separate Layer Normalizations to token embedding and PE, with PE delivered hierarchically through the layers. MPVG adds the initial PE (after its own LayerNorm) to the Last LN's output before the classification head. The paper's diagnostic tool is the correlation coefficient between token and position embeddings at each layer, which grows more negative with depth (e.g., from -0.37 at layer 4 to -0.95 at layer 11 for DeiT-Ti), indicating the counterbalancing role. The Last-LN addition preserves that directionality after the final layer, and the paper argues this preserves features that would otherwise be lost when the Last LN alone removes high-value dimensions.

What would settle it

A concrete test: train a GAP vision transformer with layer-wise PE where the initial PE is added to the Last LN but with its sign flipped or with a learnable scalar multiplier; if the counterbalancing direction alone matters, the flipped sign should hurt, while a learnable scale should learn a positive value close to 1. If instead any additive vector to the Last LN helps (e.g., a random fixed vector), the benefit is not specific to the counterbalancing direction. Alternatively, measure the correlation between token and PE after the Last LN in MPVG: if the claim is right, the correlation should remain strongly negative after the Last LN, and models with higher post-LN negative correlation should show higher accuracy.

Watch

Extended reading notes

Core claim

The central claim is that position embedding performs a counterbalancing role in layer-wise vision transformers: as the layers deepen, the correlation between token embedding and PE becomes increasingly negative, and this negative correlation is what keeps the token embedding values in a usable range. The paper further claims that this counterbalancing is incomplete after the final layer, so the model should add the initial PE to the Last LayerNorm (Eq. 11, $y = \mathrm{LN}(x_{L+1}) + \mathrm{LN}'(pos_0)$). When this is done, MPVG outperforms the default architecture, LaPE, and an intermediate PVG variant across image classification, object detection, and semantic segmentation.

Load-bearing premise

The paper assumes that the increasingly negative correlation between token embedding and position embedding in deeper layers is evidence of a counterbalancing function that should be preserved after the final layer; this is inferred from post-hoc correlation plots and ablations, not from an experiment that isolates the causal direction or rules out other explanations such as LayerNorm statistics.

Editorial extensions

If this is right

  • Any vision transformer using GAP and layer-wise PE can be improved by a single addition of the initial PE to the final LayerNorm, with no extra parameters beyond the existing PE and LN.
  • The counterbalancing role of PE is not just an artifact but a functional mechanism that should be maintained after the last layer.
  • The inconsistency between GAP and layer-wise structures is explained and resolved, making the two compatible.
  • The gains transfer to dense prediction tasks (detection and segmentation) when the pretrained backbone uses MPVG.
  • The choice of which PE to add matters little: pos0 works best, but other layer-wise PE values still beat the baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same counterbalancing argument might apply to class-token models if the class token undergoes similar distributional shifts, although the paper reports incompatibility with class tokens; a test would be to add a learned or transformed PE to the Last LN in a class-token model.
  • The correlation diagnostic could serve as a training-time monitor: if the negative correlation flattens or reverses, the model may be losing positional or counterbalancing information.
  • The mechanism suggests that LayerNorm's beta parameter is a limited substitute for PE; architectures that reduce reliance on LN statistics might see larger or smaller effects.
  • A testable extension: apply the same Last-LN PE addition in language transformers or other sequence models that use GAP-like pooling, to see if the counterbalancing role generalizes beyond vision.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies the combination of global average pooling (GAP) with layer-wise position embedding (PE) delivery in vision transformers. It observes that while GAP and layer-wise PE each help individually, combining them hurts accuracy. The authors propose MPVG, which adds the initial position embedding pos0 to the 'Last LN' before the classification head (Eq. 11), and PVG, a variant without this addition. They report consistent top-1 accuracy gains over the default and over LaPE across DeiT, Swin, CeiT, and T2T on ImageNet-1K, on CIFAR-100, and on COCO/ADE20K dense-prediction tasks. The paper interprets these gains as evidence that PE counterbalances token embeddings in deep layers and that maintaining this counterbalancing directionality after the final layer is beneficial.

Significance. If the central claim holds, this is a simple, broadly applicable modification to vision transformers that use GAP, with a suggestive new interpretation of PE beyond positional encoding. The empirical results are internally consistent and cover multiple architectures and tasks, which is a clear strength of the paper. The correlation analyses in Figs. 2 and 4 are interesting and could motivate further study. However, the paper's mechanistic claim—that the identity of the added PE, rather than a generic learned bias, drives the improvement—is not supported by the provided experiments. The paper also selects architectural details and the PE source on the test set without a validation split and reports single runs. These issues are central to the contribution as stated.

major comments (3)
  1. [Eq. (11) and 'Maximizing the Position Embedding with GAP'] The paper does not include a control experiment that isolates whether the content of the added vector matters. After GAP, LN'(pos0) in Eq. (11) is constant across tokens, so it is functionally equivalent to adding a learnable bias to the pooled feature vector. Since the linear classification head already has a bias, any constant offset can be absorbed, and the observed gains could stem from the extra parameter, from optimization dynamics, or from the counterbalancing directionality claimed by the authors. To support the paper's central claim that PE performs a counterbalancing role, the authors should compare Eq. (11) against a same-shaped learnable vector, a random fixed vector, or a zero-initialized bias. Without such a control, the reported gains do not discriminate between the proposed mechanism and a generic bias-shift effect.
  2. [Tables 5 and 6, 'Ablation Study'] The architecture choices (excluding layer 0, hierarchical PE, adding PE to x before layer 0) and the choice of pos0 as the Last LN input are selected based on test-set top-1 accuracy on ImageNet-1K. For example, Table 6 shows differences of 0.2–0.3% between configurations, and Table 5 selects pos0 over pos5/pos8/pos11 by a margin of 0.1–0.2%. With single runs and no held-out validation split, these differences are within typical training noise, and the 'optimal' configuration may overfit the test set. The authors should either use a separate validation set for these selections or report multiple seeds with means and variances for all reported values.
  3. [Figs. 2, 4, 8 and the 'Analysis' section] The correlation analyses are purely descriptive and do not establish a causal counterbalancing role. The negative correlation between token and position embeddings after LayerNorm could arise trivially from the centering and scaling properties of LayerNorm, or from optimization constraints, rather than from a functional mechanism the model seeks to preserve. The paper's own appendix shows that the Last LN's beta parameter has high variance in PVG and low variance in MPVG, which is equally consistent with a simple bias-shift explanation. To support the counterbalancing hypothesis, the authors should provide a control that decorrelates the added PE from the counterbalancing direction (for example, adding a randomly sampled or sign-permuted PE) while keeping the same distributional statistics, and show that the gain disappears.
minor comments (6)
  1. [Table 5 caption and Eq. (7)] The caption defines posN as LN'_N(pos_{N-1}), but Eq. (7) defines pos_l = LN'_{l-1}(pos_{l-1}). The subscript is off by one and should be corrected to avoid confusion.
  2. [Figure 2 caption] The phrase 'the average value of token embedding' is ambiguous; it should clarify whether the heatmaps are averaged over tokens, batch samples, or both, and over which forward pass (training or validation).
  3. [Tables throughout] The paper reports only single-run top-1 accuracies without standard deviations or number of seeds. For a fair comparison of changes as small as 0.1%, at least three seeds should be reported, or the authors should state that the differences are beyond their evaluation noise.
  4. [Table 2 caption] The typo 'MVPG' in the Table 2 caption should read 'MPVG'.
  5. [Appendix Tables 8 and 9] The header 'Prei-train' should be 'Pretrain' (or 'Pre-train').
  6. [Introduction, contribution 3] The claim that MPVG is 'generally effective for vision transformers' is supported only for a limited set of architectures and training settings; the paper should soften this to 'effective across the evaluated architectures and tasks'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: MPVG is an empirical architecture change whose reported gains are independently measured.

full rationale

MPVG is presented as an empirical architecture modification: Eq. (11), y = LN(x_{L+1}) + LN'(pos0), adds the initial position embedding to the Last LN when GAP is used. The 'counterbalancing' narrative is inferred from correlation plots and then tested by explicit comparisons (PVG vs MPVG; Table 5 varying which pos_l is added; Fig. 5 non-layer-wise control), not by a derivation that assumes the conclusion. No parameter is fitted to the target accuracy and then renamed a prediction; the choice of pos0 in Table 5 is made by comparing test accuracies, which is post-hoc model selection rather than circularity. The paper does cite LaPE as the layer-wise prior work, but the authors are disjoint from the current authors, so there is no load-bearing self-citation chain. The strongest concern—that LN'(pos0) is token-invariant and, after GAP, is mathematically absorbable into the classification head bias, so the 'directionality' mechanism is not isolated by the experiments—is a missing-control/identifiability objection, not a circularity objection. MPVG's reported accuracy gains are independent empirical measurements and do not reduce by definition to the paper's inputs. Therefore no circular step is established.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the prior LaPE structure and three added assumptions: that negative correlation implies counterbalancing, that the final averaged token needs the initial PE, and that small single-run accuracy differences are meaningful. The architecture is tuned on the test set through the choices in Tables 5 and 6, which are effectively free parameters.

free parameters (4)
  • Last LN position embedding source = pos0 (initial PE)
    In Table 5, pos0 achieves 73.51% top-1 accuracy on DeiT-Ti, beating pos5 (73.39), pos8 (73.38), and pos11 (73.30); the paper adopts pos0. The qualitative benefit of adding any PE to the Last LN is robust, but the specific source is selected by test accuracy.
  • Layer 0 inclusion flag = excluded (PE not delivered to layer 0)
    Table 6 shows 73.51% with layer 0 excluded versus 73.31% with layer 0 included, so the paper excludes it.
  • Hierarchical PE delivery flag = hierarchical (pos_l = LN'_l(pos_{l-1}))
    Table 6 shows 73.51% for hierarchical versus 73.28% for non-hierarchical, so the paper adopts hierarchical delivery, though the paper notes the difference is small.
  • x+PE pre-addition flag = included (PE added to token embedding before layer 0)
    Table 6 shows 73.51% with x+PE versus 73.48% without, so the paper includes it.
assumptions (4)
  • domain assumption The LaPE layer-wise position embedding structure with independent LayerNorms for token and PE is a valid and beneficial baseline.
    MPVG builds directly on LaPE (Yu et al. 2023), and the paper does not re-derive its properties or validity.
  • ad hoc to paper Negative correlation between token and position embeddings in deep layers indicates a counterbalancing function.
    This is the central interpretive step, supported only by correlational plots in Figs 2 and 4, not by causal experiments.
  • ad hoc to paper The averaged token embedding after the last layer still needs the initial position embedding pos0 to maintain the counterbalance.
    Eq. (11) injects pos0 into the Last LN; Table 5 shows pos0 gives the best accuracy, but that selection is made on the test set.
  • domain assumption Single-run accuracy differences between 0.27 and 1.37 percentage points are meaningful without error bars or multiple seeds.
    Single-run reporting is common in the field, but the central claim rests on these margins, making the assumption load-bearing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Maximizing the Position Embedding for Vision Transformers with Global Average Pooling." pith.science (2026). https://pith.science/paper/AECS45S7

@misc{pith2026250202919,
  author       = {Pith},
  title        = {Pith review of: Maximizing the Position Embedding for Vision Transformers with Global Average Pooling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AECS45S7}},
  note         = {Machine review of arXiv:2502.02919}
}
read the original abstract

In vision transformers, position embedding (PE) plays a crucial role in capturing the order of tokens. However, in vision transformer structures, there is a limitation in the expressiveness of PE due to the structure where position embedding is simply added to the token embedding. A layer-wise method that delivers PE to each layer and applies independent Layer Normalizations for token embedding and PE has been adopted to overcome this limitation. In this paper, we identify the conflicting result that occurs in a layer-wise structure when using the global average pooling (GAP) method instead of the class token. To overcome this problem, we propose MPVG, which maximizes the effectiveness of PE in a layer-wise structure with GAP. Specifically, we identify that PE counterbalances token embedding values at each layer in a layer-wise structure. Furthermore, we recognize that the counterbalancing role of PE is insufficient in the layer-wise structure, and we address this by maximizing the effectiveness of PE through MPVG. Through experiments, we demonstrate that PE performs a counterbalancing role and that maintaining this counterbalancing directionality significantly impacts vision transformers. As a result, the experimental results show that MPVG outperforms existing methods across vision transformers on various tasks.

Figures

Figures reproduced from arXiv: 2502.02919 by the authors.

Figure 1
Figure 1. The conflicting result between the GAP method [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The heatmaps depict the characteristics of each layer in both the original structure and the Layer-wise structure with [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The overview of the various methods. (a) ViT. (b) LaPE (Yu et al. 2023). (c) PVG, an improved Layer-wise structure. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Correlation coefficient between token embedding [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Comparison of two methods on DeiT-Ti. (a) Struc [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Heatmaps(averaged over the batch size), beta and gamma values in the Last LN, and cosine similarity for each method. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: The visualizations and heatmaps before and after the Last LN in the PVG and MPVG methods are shown. These [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: The difference between delivering PE to the Last LN in Non-Layer-wise and Layer-wise structures. In (a), PE is [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 13 canonical work pages

  1. [1]

    L.; Kiros, J

    Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016. Layer normalization. arXiv preprint arXiv:1607.06450

    arXiv 2016

  2. [2]

    Bello, I.; Zoph, B.; Vaswani, A.; Shlens, J.; and Le, Q. V. 2019. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF international conference on computer vision, 3286--3295

    work page 2019

  3. [3]

    Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; and Zagoruyko, S. 2020. End-to-end object detection with transformers. In European conference on computer vision, 213--229. Springer

    2020

  4. [4]

    J.; Jin, R.; and Shou, M

    Chang, S.; Wang, P.; Lin, M.; Wang, F.; Zhang, D. J.; Jin, R.; and Shou, M. Z. 2023. Making vision transformers efficient from a token sparsification view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6195--6205

    work page 2023

  5. [5]

    C.; and Lin, D

    Chen, K.; Wang, J.; Pang, J.; Cao, Y.; Xiong, Y.; Li, X.; Sun, S.; Feng, W.; Liu, Z.; Xu, J.; Zhang, Z.; Cheng, D.; Zhu, C.; Cheng, T.; Zhao, Q.; Li, B.; Lu, X.; Zhu, R.; Wu, Y.; Dai, J.; Wang, J.; Shi, J.; Ouyang, W.; Loy, C. C.; and Lin, D. 2019. MMDetection : Open MMLab Detection Toolbox and Benchmark. arXiv preprint arXiv:1906.07155

    arXiv 2019

  6. [6]

    Chen, Z.; Duan, Y.; Wang, W.; He, J.; Lu, T.; Dai, J.; and Qiao, Y. 2022. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.08534

    arXiv 2022

  7. [7]

    Chu, X.; Tian, Z.; Wang, Y.; Zhang, B.; Ren, H.; Wei, X.; Xia, H.; and Shen, C. 2021 a . Twins: Revisiting the design of spatial attention in vision transformers. Advances in neural information processing systems, 34: 9355--9366

    work page 2021

  8. [8]

    Chu, X.; Tian, Z.; Zhang, B.; Wang, X.; and Shen, C. 2021 b . Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102.10882

    arXiv 2021

Show all 36 references
  1. [9]

    Contributors, M. 2020. MMSegmentation : OpenMMLab Semantic Segmentation Toolbox and Benchmark. https://github.com/open-mmlab/mmsegmentation

    2020

  2. [10]

    Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248--255. Ieee

    2009

  3. [11]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    2020 arXiv

  4. [12]

    Hassani, A.; Walton, S.; Shah, N.; Abuduweili, A.; Li, J.; and Shi, H. 2022. Escaping the Big Data Paradigm with Compact Transformers. arXiv:2104.05704

    2022 arXiv

  5. [13]

    He, K.; Gkioxari, G.; Doll \'a r, P.; and Girshick, R. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, 2961--2969

    2017

  6. [14]

    Heo, B.; Park, S.; Han, D.; and Yun, S. 2024. Rotary position embedding for vision transformer. arXiv preprint arXiv:2403.13298

    2024 arXiv

  7. [15]

    Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images

    2009

  8. [16]

    Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Doll \'a r, P.; and Zitnick, C. L. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 1...

    2014

  9. [17]

    Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, 10012--10022

    2021

  10. [18]

    Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations

    2019

  11. [19]

    Raghu, M.; Unterthiner, T.; Kornblith, S.; Zhang, C.; and Dosovitskiy, A. 2021. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34: 12116--12128

    2021

  12. [20]

    Shaw, P.; Uszkoreit, J.; and Vaswani, A. 2018. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155

    2018 arXiv

  13. [21]

    Strudel, R.; Garcia, R.; Laptev, I.; and Schmid, C. 2021. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, 7262--7272

    2021

  14. [22]

    Touvron, H.; Cord, M.; Douze, M.; Massa, F.; Sablayrolles, A.; and J \'e gou, H. 2021. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, 10347--10357. PMLR

    2021

  15. [23]

    N.; Kaiser, .; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  16. [24]

    Wang, Y.; Xu, Z.; Wang, X.; Shen, C.; Cheng, B.; Shen, H.; and Xia, H. 2021. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8741--8750

    2021

  17. [25]

    Wu, K.; Peng, H.; Chen, M.; Fu, J.; and Chao, H. 2021. Rethinking and improving relative position encoding for vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 10033--10041

    2021

  18. [26]

    Xiao, T.; Liu, Y.; Zhou, B.; Jiang, Y.; and Sun, J. 2018. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), 418--434

    2018

  19. [27]

    Xu, H.; Xiang, L.; Ye, H.; Yao, D.; Chu, P.; and Li, B. 2024. Permutation Equivariance of Transformers and Its Applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5987--5996

    2024

  20. [28]

    Yu, R.; Wang, Z.; Wang, Y.; Li, K.; Liu, C.; Duan, H.; Ji, X.; and Chen, J. 2023. LaPE: Layer-adaptive position embedding for vision transformers with independent layer normalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 5886--5896

    2023

  21. [29]

    Yuan, K.; Guo, S.; Liu, Z.; Zhou, A.; Yu, F.; and Wu, W. 2021 a . Incorporating convolution designs into visual transformers. In Proceedings of the IEEE/CVF international conference on computer vision, 579--588

    2021

  22. [30]

    E.; Feng, J.; and Yan, S

    Yuan, L.; Chen, Y.; Wang, T.; Yu, W.; Shi, Y.; Jiang, Z.-H.; Tay, F. E.; Feng, J.; and Yan, S. 2021 b . Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision, 558--567

    2021

  23. [31]

    Zhai, X.; Kolesnikov, A.; Houlsby, N.; and Beyer, L. 2022. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 12104--12113

    2022

  24. [32]

    H.; et al

    Zheng, S.; Lu, J.; Zhao, H.; Zhu, X.; Luo, Z.; Wang, Y.; Fu, Y.; Feng, J.; Xiang, T.; Torr, P. H.; et al. 2021. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern ...

    2021

  25. [33]

    Zhou, B.; Zhao, H.; Puig, X.; Xiao, T.; Fidler, S.; Barriuso, A.; and Torralba, A. 2019. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127: 302--321

    2019

  26. [34]

    Zhu, X.; Su, W.; Lu, L.; Li, B.; Wang, X.; and Dai, J. 2020. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159

    2020 arXiv

  27. [35]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  28. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.