Pith. sign in

REVIEW 4 major objections 3 minor 71 references

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING

T0 review · 4 major / 3 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces STRING, a family of translationally invariant position encodings that provably contains RoPE as a special case, and shows that its two efficient variants, Cayley-STRING and Circulant-STRING, outperform RoPE and…

desk verdict Solid theory framing a learnable reparameterization of RoPE; the empirical 'substantial gains' claim is not yet established because two key tables are confounded or max-seeded. read the letter →

arxiv 2502.02562 v1 pith:TJSS3BR2 submitted 2025-02-04 cs.LG cs.AIcs.CVcs.ROstat.ML

Connor Schenck , Isaac Reid , Mithun George Jacob , Alex Bewley , Joshua Ainslie , David Rendleman , Deepali Jain , Mohit Sharma
show 14 more authors
Avinava Dubey Ayzaan Wahid Sumeet Singh René Wagner Tianli Ding Chuyuan Fu Arunkumar Byravan Jake Varley Alexey Gritsenko Matthias Minderer Dmitry Kalashnikov Jonathan Tompson Vikas Sindhwani Krzysztof Choromanski
This is my paper · ORCID
classification cs.LGcs.AIcs.CVcs.ROstat.ML
keywords STRINGpositionencodingrotarytranslationinvarianceVisionTransformeropen-vocabularydetectionroboticmanipulationmatrixexponential
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

STRING is a new family of position encodings for Transformers that the authors prove to be the most general way to make attention depend on token positions through matrix multiplication while remaining exactly translation invariant. Rotary position encodings (RoPE), the standard in large language models, turn out to be a special case, and any STRING encoding is RoPE conjugated by a fixed learned orthogonal matrix. The paper reports consistent gains over RoPE and absolute position encodings on image classification, image-text retrieval, open-vocabulary object detection in 2D and 3D, and robotic manipulation, with the clearest advantages when tokens carry 2D or 3D coordinates such as depth. A practical reader should care because STRING adds only a small number of parameters to existing vision or language models while extending exact translation invariance to arbitrary coordinate dimensions.

What carries the argument

The central object is the matrix exponential of a linear combination of commuting skew-symmetric generators, $\exp(\sum_k L_k [\mathbf{r}_i]_k)$, which defines STRING. The proofs use the Lie-group fact that a one-parameter family of rotations is an exponential of an antisymmetric matrix: the group-like property $R(\mathbf{r}_i)^\top R(\mathbf{r}_j)=R(\mathbf{r}_j-\mathbf{r}_i)$ forces the generators to commute and the map on coordinate vectors to be linear. Theorem 3.4 reduces every STRING to RoPE in a learned orthogonal basis $P$, and the two efficient instantiations parameterize that basis or the generators directly: Cayley-STRING builds $P$ via the Cayley transform of a learnable antisymmetric matrix, and Circulant-STRING takes circulant generators whose exponentials act on tokens in $O(d\log d)$ time using the fast Fourier transform. The same framework extends to other Abelian Lie groups by replacing Cartesian coordinates with canonical coordinates.

What would settle it

An equal-footing evaluation that feeds RoPE and STRING the same RGB-D inputs with depth as a third coordinate, reports the mean (not the max) over at least three seeds on the 2D and 3D detection and robot manipulation tasks, and finds no advantage for STRING over a well-tuned RoPE with learnable frequencies would refute the practical superiority claim; the theoretical universality theorems would remain true because they are proven.

Watch

Extended reading notes

Core claim

The paper claims that the space of translationally invariant position encodings that act on tokens by matrix multiplication, are continuously differentiable, and satisfy $R(\mathbf{0})=I_d$, is exactly the STRING family $R(\mathbf{r}_i)=\exp(\sum_k L_k [\mathbf{r}_i]_k)$ with commuting skew-symmetric generators $L_k$ (Theorem 3.2), and that every such encoding factorizes as $R(\mathbf{r}_i)=P\,\mathrm{RoPE}(\mathbf{r}_i)P^\top$ with a fixed orthogonal matrix $P$ (Theorem 3.4). Because the orthogonal factors cancel in standard softmax attention, the effective model is learnable RoPE with a shared learned change of basis, which the paper presents as the mechanism behind its empirical wins. On image classification, WebLI-3D retrieval, OWL-ViT open-vocabulary detection in 2D and 3D, and simulated and real robot manipulation, the STRING variants (Cayley-STRING and Circulant-STRING) improve over RoPE and absolute encodings, with the largest margins on 3D tasks. Theorem 3.4 also implies the parameter overhead is minimal, since the basis $P$ is shared across all tokens.

Load-bearing premise

The reported performance gains rest on comparisons whose data and reporting choices can tilt the result: in the WebLI-3D retrieval evaluation the ViT baseline receives only RGB while STRING and RoPE receive RGB-D, and the 3D detection table reports the best of three random seeds, so the claimed superiority is only as strong as those equal-footing assumptions.

Editorial extensions

If this is right

  • Any future translationally invariant matrix-multiplication position encoding, including RoPE variants with different frequency schedules, is a STRING instance and thus inherits the efficient implementations of the paper.
  • STRING can be dropped into existing ViT and ViT with depth backbones, including pre-trained checkpoints, because depth is used to modulate attention rather than to add channels; the paper exploits this to fine-tune 2D backbones into 3D policies.
  • Exact translation invariance holds for position vectors of any dimension $d_c$, so the same encoding treats 2D image patches, 3D points, or point clouds without re-deriving the machinery.
  • The $O(d\log d)$ cost of Circulant-STRING keeps position encoding cheap for large batches and for robotics inputs where positions vary per example, avoiding the $O(d^3)$ cost of dense matrix exponentials.
  • Because STRING contains RoPE, the reported gains over RoPE come from learning the frequency-defining parameters as well as the basis, and these gains are preserved when the framework is applied to existing transformer architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the equivalence theorem locates STRING's advantage in optimization, not expressivity, since the space of attainable attention scores is identical to RoPE's; a clean ablation that freezes $P$ to a random orthogonal matrix per head would test whether the gains come from learning $P$ or from the shared-basis regularization effect.
  • Editorial inference: the canonical-coordinate extension suggests STRING could build rotation-invariant encodings for objects with pose, e.g., encoding azimuth and polar angles of point clouds; the paper raises this possibility but does not test it.
  • Editorial inference: because the WebLI-3D baseline receives RGB only while STRING and RoPE receive RGB-D, the reported retrieval gap could reflect the extra depth channel rather than the encoding itself; an input-matched comparison would isolate the encoding effect.
  • Editorial inference: the same Lie-group construction works for any Abelian transformation group, so STRING may transfer to equivariant attention for other structured coordinate spaces, such as in-plane rotations of images, beyond the translations studied in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper introduces STRING (Separable Translationally Invariant Position Encodings), a family of position encodings defined by exponentiated commuting skew-symmetric generators. It proves that any continuously differentiable matrix-multiplication position encoding satisfying the group-like translation-invariance property R(r_i)^T R(r_j) = R(r_j - r_i) with R(0)=I must have the STRING form (Theorem 3.2), shows that RoPE is a special case (Theorem 3.3), and proves that every STRING encoding is equal to RoPE up to an orthogonal change of basis (Theorem 3.4). Two practical instantiations are given: Cayley-STRING, using a learnable orthogonal matrix, and Circulant-STRING, using commutating circulant generators computed in O(d log d) time via FFT. The paper reports experiments on ImageNet and Places365 classification, WebLI-3D retrieval, 2D and 3D open-vocabulary detection, and simulated and real robot manipulation, claiming substantial gains over RoPE and absolute position encodings.

Significance. The theoretical analysis is a genuine contribution: it gives a clean characterization of the class of translationally invariant, separably applied, matrix-multiplication position encodings, and the proofs in Appendix A are concise and correct. The paper is also unusually honest in Theorem 3.4, showing that STRING and RoPE generate the same attention scores up to a change of basis, so any practical advantage must come from optimization rather than representational power. The empirical evaluation is broad and touches on tasks of real applied interest, including 3D detection and real-robot manipulation. However, the strongest empirical claims currently rest on a confounded input-modality comparison in Table 2 and on max-over-seeds reporting in Table 4, so the magnitude and attribution of the reported gains are not yet established. The theoretical contribution is strong enough to warrant revision, but the practical claims need to be re-evidenced or substantially tempered.

major comments (4)
  1. [Section 4.1 (Table 2); Appendix E.2] The retrieval comparison is not on equal footing. Appendix E.2 states that the ViT baseline used RGB data while all RoPE and STRING variants used RGB-D, with depth incorporated as a third coordinate for 3D position representations. Since STRING and RoPE receive an additional input channel, the reported mean recall gain of about 1.7 points (68.77 for ViT vs. 70.48 for Cayley-STRING) may be explained by the depth channel rather than by the position encoding. Please add an RGB-D ViT baseline or an equivalent depth-injection baseline so that all methods are compared with identical input modalities, or remove and reframe the retrieval claim.
  2. [Section 4.2.2 (Table 4); Appendix F.4] Table 4 reports the maximum over three random seeds. Appendix F.4 shows that one Circulant-STRING seed collapsed to 13.12% IOU while its reported maximum is 58.95%, and two ViT+RoPE-M seeds collapsed to 2.53% and 2.46%. Selecting the maximum systematically favors higher-variance methods and can manufacture performance gaps. Please report all individual runs, or at least mean and standard deviation or median, and state how many seeds failed to train; the data already present in Table 6 of Appendix F.4 should be reflected in the main-text analysis.
  3. [Section 3.2, Theorem 3.4; Section 5] The claim that STRING is 'more general' than RoPE needs precise qualification. Theorem 3.4 shows that every STRING encoding satisfies R(r) = P RoPE(r) P^T for a fixed orthogonal P. Because the attention logits are q_i^T R(r_i)^T R(r_j) k_j = (P^T q_i)^T RoPE(r_j - r_i) (P^T k_j), STRING and RoPE generate exactly the same family of attention scores up to a learned linear reparameterization of queries and keys. The correct contribution is therefore a learnable-basis parameterization with the same representational power as RoPE, not a larger class of mechanisms. The abstract, the key contributions, and the conclusion should state this explicitly rather than presenting STRING as more general in an unqualified way.
  4. [Section 4.1, Table 1; Section 4.2.1, Table 3] Tables 1 and 3 report single runs without confidence intervals or seed variance, yet the text highlights the first gain larger than 1% on ImageNet. Given that the margins in these tables are small (e.g., 1.18 points on ImageNet and 0.37 points on Places365 for the best STRING variant), the magnitude of the improvement is not established without some measure of variability. Please provide error bars, multiple seeds, or a statistical comparison for at least the main classification and 2D detection results.
minor comments (3)
  1. [Section 4.3, Table 5] The sentence 'We find that STRING has the best success rate across 11 of 13 tasks' is not supported by the tabulated numbers. STRING is best or tied in 10 of the 12 individual tasks and best on the MultiTask aggregate, while RoPE wins on MugOnPlate and PlateOnRack; the wording should be corrected to match the table.
  2. [Section 4.4.2, Figure 5] The caption reports '(with 2 seeds)' but the text says the mean and standard deviation were calculated from 35 evaluation runs. Please clarify whether the standard deviation is over evaluation runs or over seeds, and report seed-level variability separately.
  3. [Appendix E] For Circulant-STRING, the block size was chosen by sweeping over {4, 8, 16, 32, 64}. Please state whether this selection was made on a validation split and not on the test split, and whether the reported results use the selected block size.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: STRING's derivation from the group-like property is self-contained; Theorem 3.4 is an honest equivalence, and empirical protocol issues are confounds, not circular reasoning.

full rationale

No circularity found. Definition 3.1 specifies STRING as an exponential of commuting skew-symmetric generators; Theorem 3.2 independently proves that any continuously differentiable R with R(0)=I and R(r_i)^T R(r_j)=R(r_j-r_i) must have that exponential form via standard Lie-group facts, so the definition is not the input of the theorem. Theorems 3.3 and 3.4 are explicit algebraic equivalences (RoPE as a special generator choice; every STRING equal to RoPE up to an orthogonal basis P), and the paper states rather than hides the basis-change equivalence. The efficient variants (Cayley, Circulant) are parameterizations of P or of commuting generators, with Theorem 3.5 providing an FFT-based complexity proof. No downstream number is a fitted parameter relabeled as a prediction; learned generators are trained and evaluated on held-out tasks. The cited self-works are related-work pointers (e.g., Performers, masking, Diffusion-phi) and are not load-bearing for the universality claim. The RGB-only baseline for ViT in Table 2 and the max-over-seeds reporting in Table 4 are experimental confounds and variance-selection concerns, which belong to correctness risk, not circularity. The only mathematical caveat is Theorem 3.2's silent restriction to SO(d), which is a precision issue, not a circular step.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central mathematical claim rests on standard Lie group theory and on the definitional choice that translation invariance means R(r_i)^T R(r_j) = R(r_j - r_i) with R(0)=I. No new physical or ontological entities are proposed; STRING is a parameterization of existing mathematical objects (commuting skew-symmetric matrices, orthogonal matrices). The empirical claims rely on the unstated representativeness of the evaluation datasets. The only tunable hyperparameter explicitly swept for one method is the Circulant-STRING block size.

free parameters (2)
  • block_size (Circulant-STRING) = 16 for ImageNet/Places365, 32 for WebLI-3D
    Chosen by sweeping {4,8,16,32,64} and selecting the best per task; affects model capacity and reported performance. Location: Appendix E.
  • initialization wavelength for STRING frequencies = 100
    Borrowed from RoPE-Mixed [22] as the initial value for learned frequencies; a design choice that influences optimization. Location: Appendix E.
assumptions (4)
  • domain assumption R(0)=I and continuous differentiability are the defining properties of translation-invariant position encodings.
    Assumed in Definition 3.1 and Theorem 3.2; restricts the class of position encodings under consideration. Location: Section 3.1.
  • standard math Any continuous homomorphism from R^{d_c} to SO(d) is the exponential of a linear map into an abelian subalgebra of skew-symmetric matrices.
    Used in the proof of Theorem 3.2 (Appendix A.1); standard Lie group theory, see Hall, ref [20].
  • standard math Commuting skew-symmetric matrices are simultaneously orthogonally block-diagonalizable into 2x2 rotation blocks.
    Used in Theorem 3.4 (Appendix A.3) to show that every STRING is RoPE in a changed basis.
  • domain assumption The evaluation datasets (WebLI-3D filtered subset, synthetic 3D detection scenes, ALOHA simulation) are representative of real-world vision and robotics tasks.
    Underpins the claim that observed gains generalize beyond the specific benchmarks. Location: Sections 4.1-4.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning the RoPEs: Better 2D and 3D Position Encodings with STRING." pith.science (2026). https://pith.science/paper/TJSS3BR2

@misc{pith2026250202562,
  author       = {Pith},
  title        = {Pith review of: Learning the RoPEs: Better 2D and 3D Position Encodings with STRING},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TJSS3BR2}},
  note         = {Machine review of arXiv:2502.02562}
}
read the original abstract

We introduce STRING: Separable Translationally Invariant Position Encodings. STRING extends Rotary Position Encodings, a recently proposed and widely used algorithm in large language models, via a unifying theoretical framework. Importantly, STRING still provides exact translation invariance, including token coordinates of arbitrary dimensionality, whilst maintaining a low computational footprint. These properties are especially important in robotics, where efficient 3D token representation is key. We integrate STRING into Vision Transformers with RGB(-D) inputs (color plus optional depth), showing substantial gains, e.g. in open-vocabulary object detection and for robotics controllers. We complement our experiments with a rigorous mathematical analysis, proving the universality of our methods.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 43 canonical work pages

  1. [1]

    Round and round we go! what makes rotary positional encodings useful?arXiv preprint arXiv:2410.06205, 2024

    Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful?arXiv preprint arXiv:2410.06205, 2024

    arXiv 2024

  2. [2]

    PaliGemma: A versatile 3B VLM for transfer.arXiv preprint arXiv:2407.07726, 2024

    Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bau...

    arXiv 2024

  3. [3]

    On computing givens rotations reliably and efficiently.ACM Trans

    David Bindel, James Demmel, William Kahan, and Osni Marques. On computing givens rotations reliably and efficiently.ACM Trans. Math. Softw., 28(2):206–238, 2002. doi: 10.1145/567806. 567809. URL https://doi.org/10.1145/567806.567809

    arXiv 2002

  4. [4]

    Benchmarkinginmanipulationresearch: Theycbobjectandmodelsetandbenchmarking protocols

    Berk Calli, Aaron Walsman, Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. Benchmarkinginmanipulationresearch: Theycbobjectandmodelsetandbenchmarking protocols. arXiv preprint arXiv:1502.03143, 2015

    arXiv 2015

  5. [5]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities, 2024

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities, 2024. URLhttps://arxiv.org/abs/2401.12168

    arXiv 2024

  6. [6]

    A simple and effective positional encoding for transformers

    Pu-Chin Chen, Henry Tsai, Srinadh Bhojanapalli, Hyung Won Chung, Yin-Wen Chang, and Chun-Sung Ferng. A simple and effective positional encoding for transformers. In Marie- 13 Learning the RoPEs: Better 2D and 3D Position Encodings with STRING Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Proceedings of the 2021 Conference on...

    2021

  7. [7]

    Pali: A jointly- scaled multilingual language-image model, 2023

    XiChen,XiaoWang,SoravitChangpinyo,AJPiergiovanni,PiotrPadlewski,DanielSalz,Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury,WeichengKuo,MojtabaSeyedhosseini,ChaoJia,BurcuKaragolAyan,CarlosRiquelme, Andreas St...

    work page 2023

  8. [8]

    Ramadge, and Alexander Rudnicky

    Ta-Chung Chi, Ting-Han Fan, Peter J. Ramadge, and Alexander Rudnicky. KERPLE: kernelized relative positional embedding for length extrapolation. In Sanmi Koyejo, S. Mohamed, A. Agar- wal, Danielle Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, Neur...

    work page 2022

Show all 71 references
  1. [9]

    Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter J. Ramadge. Dissecting transformer length extrapolation via the lens of receptive field analysis. In Anna Rogers, Jordan L. Boyd- Graber, and Naoaki Okazaki, editors,Proceedings of the 61st Annual Meeting of the Associa...

    2023

  2. [10]

    Rethinking attention with performers.arXiv preprint arXiv:2009.14794, 2020

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers.arXiv preprint arXiv:2009.14794, 2020

    2009 arXiv

  3. [11]

    From block- toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked transformers

    KrzysztofChoromanski, HanLin, HaoxianChen, TianyiZhang, ArijitSehanobish, ValeriiLikhosh- erstov, Jack Parker-Holder, Tamas Sarlos, Adrian Weller, and Thomas Weingarten. From block- toeplitz matrices to differential equations on graphs: towards a general theory for scalable ma...

    2022

  4. [12]

    Abo: Dataset and benchmarks for real-world 3d object understanding

    Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, and Jitendra Malik. Abo: Dataset and benchmarks for real-world 3d object understanding. CVPR, 2022

    2022

  5. [13]

    Imagenet: A large-scale hierarchicalimagedatabase

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchicalimagedatabase. In 2009IEEEConferenceonComputerVisionandPatternRecognition , pages 248–255, 2009. doi: 10.1109/CVPR.2009.5206848

    2009

  6. [14]

    Fasma Diele, Luciano Lopez, and R. Peluso. The cayley transform in the numerical solution of unitary differential systems.Adv. Comput. Math., 8(4):317–334, 1998. doi: 10.1023/A: 1018908700358. URL https://doi.org/10.1023/A:1018908700358

    1998 doi

  7. [15]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkor- eit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  8. [16]

    Google scanned objects: A high-quality dataset of 3d scanned household items

    Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In2022 International Conference on Robotics and Automation (ICRA), pag...

    2022

  9. [17]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [18]

    Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  11. [19]

    Lvis: A dataset for large vocabulary instance seg- mentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance seg- mentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019

    2019

  12. [20]

    Springer, 2013

    Brian C Hall.Lie groups, Lie algebras, and representations. Springer, 2013

    2013

  13. [21]

    Bridging nonlinearities and stochastic regularizers with gaussian error linear units.CoRR, abs/1606.08415, 2016

    Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units.CoRR, abs/1606.08415, 2016. URLhttp://arxiv.org/abs/ 1606.08415

    2016 arXiv

  14. [22]

    Rotary position embedding for vision transformer

    Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. InEuropean Conference on Computer Vision, pages 289–305. Springer, 2025

    2025

  15. [23]

    Integrating generic sensor fusion algorithms with sound state representations through encapsulation of manifolds.CoRR, abs/1107.1119, 2011

    Christoph Hertzberg, René Wagner, Udo Frese, and Lutz Schröder. Integrating generic sensor fusion algorithms with sound state representations through encapsulation of manifolds.CoRR, abs/1107.1119, 2011. URLhttp://arxiv.org/abs/1107.1119

    2011 arXiv

  16. [24]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. InInternational conference on machine learning, pages 5156–5165. PMLR, 2020

    2020

  17. [25]

    The impact of positional encoding on length generalization in transformers

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Adva...

    2023

  18. [26]

    SHAPE: Shifted absolute position embedding for transformers

    Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, and Kentaro Inui. SHAPE: Shifted absolute position embedding for transformers. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Proceedings of the 2021 Conference on Empirical Methods in Natural La...

    2021

  19. [27]

    H.W. Kuhn. The hungarian method for the assignment problem.Naval Research Logistics Quarterly, 2(1–2):83–97, 1955

    1955

  20. [28]

    Functional interpolation for relative positions improves long context transformers

    Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontañón, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli. Functional interpolation for relative positions improves long context transformers. InThe Twelfth International Conferenc...

    2024

  21. [29]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings...

    2014

  22. [30]

    Dhillon, and Cho-Jui Hsieh

    Xuanqing Liu, Hsiang-Fu Yu, Inderjit S. Dhillon, and Cho-Jui Hsieh. Learning to encode position for transformer with continuous dynamical model. InProceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Pro...

    2020

  23. [31]

    Stable, fast and accurate: Kernelized attention with relative positional encoding

    Shengjie Luo, Shanda Li, Tianle Cai, Di He, Dinglan Peng, Shuxin Zheng, Guolin Ke, Liwei Wang, and Tie-Yan Liu. Stable, fast and accurate: Kernelized attention with relative positional encoding. Advances in Neural Information Processing Systems, 34:22795–22807, 2021

    2021

  24. [32]

    Simple open-vocabulary object detection with vision transformers.ECCV, 2022

    Austin Stone Maxim Neumann Dirk Weissenborn Alexey Dosovitskiy Aravindh Mahendran Anurag Arnab Mostafa Dehghani Zhuoran Shen Xiao Wang Xiaohua Zhai Thomas Kipf Neil Houlsby Matthias Minderer, Alexey Gritsenko. Simple open-vocabulary object detection with vision transformers.ECCV, 2022

    2022

  25. [33]

    Liere: Generalizing rotary position encodings.arXiv preprint arXiv:2406.10322, 2024

    Sophie Ostmeier, Brian Axelrod, Michael E Moseley, Akshay Chaudhari, and Curtis Langlotz. Liere: Generalizing rotary position encodings.arXiv preprint arXiv:2406.10322, 2024

    2024 arXiv

  26. [34]

    Smith, and Mike Lewis

    Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://...

    2022

  27. [35]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InInternational conf...

    2021

  28. [36]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.J. Mach. Learn. Res., 21:140:1–140:67, 2020. URLhttps://jmlr. org/pap...

    2020

  29. [37]

    Linear transformer topological masking with graph random features.arXiv preprint arXiv:2410.03462, 2024

    Isaac Reid, Kumar Avinava Dubey, Deepali Jain, Will Whitney, Amr Ahmed, Joshua Ainslie, Alex Bewley, Mithun Jacob, Aranyak Mehta, David Rendleman, et al. Linear transformer topological masking with graph random features.arXiv preprint arXiv:2410.03462, 2024

    arXiv

  30. [38]

    Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  31. [39]

    Recognition of distorted patterns by invariance kernels.Pattern Recognition, 24(10):959–967, 1991

    Jacob Rubinstein, Joseph Segman, and Yehoshua Zeevi. Recognition of distorted patterns by invariance kernels.Pattern Recognition, 24(10):959–967, 1991

    1991

  32. [40]

    Joseph Segman, Jacob Rubinstein, and Yehoshua Y Zeevi. The canonical coordinates method for pattern deformation: Theoretical and computational considerations.IEEE Transactions on Pattern Analysis & Machine Intelligence, 14(12):1171–1183, 1992. 16 Learning the RoPEs: Better 2D ...

    1992

  33. [41]

    Self-attention with relative position represen- tations

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position represen- tations. arXiv preprint arXiv:1803.02155, 2018

    2018 arXiv

  34. [42]

    Revisiting energy based models as policies: Ranking noise contrastive estimation and interpolating energy models.Transactions on Machine Learning Research, 2024

    Sumeet Singh, Stephen Tu, and Vikas Sindhwani. Revisiting energy based models as policies: Ranking noise contrastive estimation and interpolating energy models.Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id= JmKAYb7I00

    2024

  35. [43]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    2024

  36. [44]

    Equivariant transformer networks

    Kai Sheng Tai, Peter Bailis, and Gregory Valiant. Equivariant transformer networks. InInterna- tional Conference on Machine Learning, pages 6086–6095. PMLR, 2019

    2019

  37. [45]

    Early or late fusion matters: Efficient rgb-d fusion in vision transformers for 3d object recognition

    Georgios Tziafas and Hamidreza Kasaei. Early or late fusion matters: Efficient rgb-d fusion in vision transformers for 3d object recognition. In2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9558–9565, 2023

    2023

  38. [46]

    Divya Udayan, Veerababu Addanki, Sathvik Durgapu, Dhanvanth Reddy Yerramreddy, and Dorasanaiah Kolla

    J. Divya Udayan, Veerababu Addanki, Sathvik Durgapu, Dhanvanth Reddy Yerramreddy, and Dorasanaiah Kolla. Forward kinematics simulation of KUKA KR5 arc robot with robo analyzer. In Proceedings of the 2023 Fifteenth International Conference on Contemporary Computing, IC3-2023, N...

    2023

  39. [47]

    Unity, 2023

    Unity Technologies. Unity, 2023. URLhttps://unity.com/. Game development platform

    2023

  40. [48]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, ...

    2017

  41. [49]

    On position embeddings in BERT

    Benyou Wang, Lifeng Shang, Christina Lioma, Xin Jiang, Hao Yang, Qun Liu, and Jakob Grue Simonsen. On position embeddings in BERT. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://o...

    2021

  42. [50]

    Effective long-context scaling of foundation models.arXiv preprint arXiv:2309.16039, 2023

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models.arXiv preprint arXiv:2309.16039, 2023

    2023 arXiv

  43. [51]

    Depth anything v2, 2024

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2, 2024. URLhttps://arxiv.org/abs/2406.09414

    2024 arXiv

  44. [52]

    Sigmoid loss for language image pre-training.arXiv preprint arXiv:2303.15343, 2023

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training.arXiv preprint arXiv:2303.15343, 2023

    2023 arXiv

  45. [53]

    Length extrapolation of transformers: A survey from the perspective of position encoding.CoRR, abs/2312.17044,

    Liang Zhao, Xiaocheng Feng, Xiachong Feng, Bing Qin, and Ting Liu. Length extrapolation of transformers: A survey from the perspective of position encoding.CoRR, abs/2312.17044,

    arXiv

  46. [54]

    1 0 0 1 # if 𝑛 is even, 𝜃𝑛(−1)(𝑛−1)/2

    Tony Z Zhao, Jonathan Tompson, Danny Driess, Pete Florence, Kamyar Ghasemipour, Chelsea Finn, and Ayzaan Wahid. Aloha unleashed: A simple recipe for robot dexterity.arXiv preprint arXiv:2410.13126, 2024. 17 Learning the RoPEs: Better 2D and 3D Position Encodings with STRING Co...

    2024 arXiv

  47. [57]

    SingleInsertion: Use the left arm to grab the blue socket and the right arm to insert the block into the socket. 22 Learning the RoPEs: Better 2D and 3D Position Encodings with STRING Figure 9| Views from the wrist, overhead and table-level cameras in ALOHA sim.Top to bottom, ...

  48. [58]

    DoubleInsertion: After completing theSingleInsertion task, insert another block into the other end of the socket. 6-7. Functional Manipulation Benchmark (FMB) 1 and 2: Insert a yellow block into the recess of a red base

  49. [59]

    FruitBowl: Place all the fruits in the bowl

  50. [60]

    StorageBin: Place all the snack boxes in the storage bin. 10-11. HandOver{Banana/Pen}: Hand over the item and place it in the container

  51. [61]

    MultiTask aggregates results of all of the above tasks

    MugOnPlate Place the mug on the plate. MultiTask aggregates results of all of the above tasks. C. Aloha Real Tasks ALOHA-real models are first pre-trained with human-teleop data collected on 300 diverse tasks, which were crowd-sourced based on relevance with real-world scenari...

  52. [62]

    open-jar-lid: open the glass jar lid, handover to other hand and put on the table

  53. [63]

    bowl-in-rack: put the bowl into the drying rack

  54. [64]

    cup-in-rack: put the cup into the drying rack

  55. [65]

    banana-handover: put banana in bowl with handover

  56. [66]

    open-drawer: open the drawer

  57. [67]

    remove-gears: remove the gears from the nist-board

  58. [68]

    fold-dress: fold the dress

  59. [69]

    stack-cups: stack the cups 23 Learning the RoPEs: Better 2D and 3D Position Encodings with STRING Figure 10| More ALOHA Simulation Tasks.Top to bottom, left to right:FruitBowl, FMB-2, GlassOn- Rack, HandOverBanana, HandOverPen and StorageBin

  60. [70]

    pen-handover: put pen in container with handover

  61. [71]

    See: Fig

    take-phone-out: take phone out of purse Afterfine-tuning,themodelisthenevaluatedonthefollowing5tasks: bowl-in-rack, banana-handover, bowl-in-rack, fold-dress, remove-gears. See: Fig. 11forthevisualizationsofselectedALOHA real world tasks. D. WebLI-3D Dataset We lift the WebLI ...

  62. [2021]

    doi: 10.18653/v1/2021.emnlp-main.266

    Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.266. URL https://aclanthology.org/2021.emnlp-main.266/

    2021 doi

  63. [2023]

    doi: 10.48550/ARXIV.2312.17044

    doi arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.