Pith. sign in

REVIEW 4 cited by

Rotary Position Embedding for Vision Transformer

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.13298 v2 pith:AY72LLX7 submitted 2024-03-20 cs.CV cs.LG

Byeongho Heo , Song Park , Dongyoon Han , Sangdoo Yun This is my paper · ORCID
classification cs.CVcs.LG
keywords ropeperformancevisionanalysisembeddingextrapolationlanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Rotary Position Embedding (RoPE) performs remarkably on language models, especially for length extrapolation of Transformers. However, the impacts of RoPE on computer vision domains have been underexplored, even though RoPE appears capable of enhancing Vision Transformer (ViT) performance in a way similar to the language domain. This study provides a comprehensive analysis of RoPE when applied to ViTs, utilizing practical implementations of RoPE for 2D vision data. The analysis reveals that RoPE demonstrates impressive extrapolation performance, i.e., maintaining precision while increasing image resolution at inference. It eventually leads to performance improvement for ImageNet-1k, COCO detection, and ADE-20k segmentation. We believe this study provides thorough guidelines to apply RoPE into ViT, promising improved backbone performance with minimal extra computational overhead. Our code and pre-trained models are available at https://github.com/naver-ai/rope-vit

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Krause Synchronization Transformers

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    Krause Attention replaces global softmax self-attention with localized, distance-based bounded-confidence interactions to promote local synchronization, reduce complexity to linear in sequence length, and alleviate at...

  2. ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Trainable commuting angle matrices generalize Rotary Position Embedding and improve accuracy and resolution robustness on vision Transformers.

  3. Maximizing the Position Embedding for Vision Transformers with Global Average Pooling

    cs.CV 2025-02 conditional novelty 5.0 of 10

    Adding the position embedding to the final layer normalization, called MPVG, improves vision transformer top-1 accuracy by 0.27 to 1.37 percentage points across several GAP-based models and tasks.

  4. Directional Non-Commutative Monoidal Structures for Compositional Embeddings in Machine Learning

    cs.LG 2025-05 conditional novelty 4.0 of 10

    A D-axis composition algebra on (vector, matrix-power) tuples provides associative per-axis operators and an interchange law when axis matrices commute, recovering RoPE, affine embedding composition, and SSM-style rec...

Pith tools