Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

SeqPE: Transformer with Sequential Position Encoding

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SeqPE replaces positional lookup tables with a small Transformer that turns each position's digit string into an embedding, and reports better perplexity, exact match, and accuracy than ALiBi, RoPE, and learned absolute embeddings…

desk verdict A clever new position encoding scheme, but the central extrapolation claim is not actually tested because the encoder is trained on the evaluation range, and the abstract overstates the numbers. read the letter →

arxiv 2506.13277 v2 pith:LIYU2575 submitted 2025-06-16 cs.LG cs.AIcs.CLcs.CV

Huayang Li , Yahui Liu , Hongyu Sun , Deng Cai , Leyang Cui , Wei Bi , Peilin Zhao , Taro Watanabe This is my paper
classification cs.LGcs.AIcs.CLcs.CV
keywords positionencodingsequentialcontextlengthextrapolationresolutioncontrastivelearningknowledgedistillationgeneralizationtransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SeqPE changes the interface of position encoding: instead of looking up a fixed table of position vectors, the model writes each position as a left-padded digit sequence and feeds that sequence to a two-layer Transformer whose output is the position embedding. The paper's central claim is that this one mechanism generalizes across text and images, extends naturally to sequences far longer than the training context, and does so while matching or beating hand-designed alternatives such as ALiBi and RoPE. Two auxiliary losses make the learned embedding space well-behaved: a contrastive loss pulls numerically close positions together in embedding space, and a knowledge-distillation loss teaches out-of-range positions to copy the relative similarity structure of in-range positions. If the claim holds, position encoding becomes a single learnable component that needs no task-specific redesign for long contexts or higher-dimensional data.

What carries the argument

The load-bearing object is the sequential position encoder $f_\theta$, a two-layer causal Transformer that reads the digit-sequence representation of a position and returns its embedding. The representation combines three embedding types—digit token, slot/place-value position, and data dimension—so $(b^k)^n$ positions are represented by only $b+n+k$ embedding vectors, a logarithmic saving over lookup tables. The argument is carried by two regularizers on that embedding space: $L_\delta$, an InfoNCE-style contrastive loss that makes Euclidean-near positions have similar embeddings, and $L_{OOD}$, a KL-divergence distillation from in-distribution teacher positions to shifted student positions with a stop-gradient. The full training objective is $L = L_{\text{main}} + \alpha L_\delta + \beta L_{OOD}$, and at inference the embeddings for fixed positions are precomputed into a look-up table so the encoder adds only a one-time cost.

What would settle it

Retrain SeqPE with the position encoder's maximum position set equal to the training context (512 tokens for language modeling, a 14-by-14 grid for images) and evaluate at 16K tokens or 672-by-672 images; if perplexity and accuracy then collapse toward the baselines, the reported gains come from the encoder having seen those positions during training, whereas flat performance would confirm real extrapolation. A complementary check is to evaluate beyond the trained maximum, for example at 24K tokens or a 112-by-112 patch grid, and see whether the reported stability continues past the encoder's range.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that an n-dimensional position can be treated as a symbolic sequence and mapped through a lightweight sequential encoder to a hidden vector, unifying what are normally separate position-encoding designs. Concretely, position (2,3) in a 4-by-4 image is written as ('0','2','0','3') using digit-value, digit-slot, and data-dimension embeddings, and the [CLS] output of a two-layer causal Transformer is the position embedding. Two regularizers support this representation: $L_\delta$, a contrastive objective whose positive sample is the nearest position under Euclidean distance, and $L_{OOD}$, a KL-divergence distillation in which shifted (out-of-range) positions reproduce the pairwise similarity pattern of in-range teacher positions with stop-gradient. The reported numbers are average perplexity 18.95 on Wikitext-103 (versus 19.54 for ALiBi and 91.57 for learned APE), average perplexity 12.34 and exact match 13.9 on RULER-SQuAD, and average accuracy 80.1 on ImageNet with a ViT-S trained at 224-by-224, versus 77.8 for the strongest RoPE2D variant. The paper presents these as evidence that a unified, fully learnable position encoding can extrapolate in length and resolution without manual architectural changes.

Load-bearing premise

The central claim assumes the test positions are genuinely unseen by the position encoder, but the encoder is trained on positions up to a maximum of 20,000 for text and 100-by-100 for images, which already includes every test length and resolution reported; no control with that maximum set to the training context is given.

Editorial extensions

If this is right

  • A language model trained on 512-token Wikitext-103 keeps perplexity near 19 even when evaluated at 16K tokens, where ALiBi reaches 21.39 and learned absolute embeddings exceed 230.
  • After fine-tuning on 1024-token long-context QA data, the model maintains answer-span perplexity of 28.60 at 8K context and achieves the best average exact match among the compared position encodings.
  • A ViT-S trained on 224-by-224 images stays accurate up to 672-by-672 resolution, with 80.1 average accuracy versus 77.8 for the strongest RoPE2D variant.
  • The same sequential interface works for 1D text and 2D images, so adding a new modality costs only a data-dimension embedding rather than a new positional formula.
  • Position-embedding storage grows logarithmically with the number of positions, and inference can reuse a precomputed look-up table built from the encoder.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the paper is right, the contrastive loss is carrying the metric structure: without it the digit-sequence encoder is lexically biased ("100" lands closer to "1000" than to "123"), so the choice of distance function $\delta$ and of the sampling procedure should be as consequential as the encoder architecture itself.
  • The distillation objective can be read as learning approximate shift-equivariance: matching shifted positions' similarity patterns to in-distribution positions gives a learned relative-position behavior without any hand-coded relative formula, which may explain why the same encoder transfers to images.
  • The paper motivates hybrid-dimensional data such as interleaved text and images but reports only 1D and 2D experiments, so a direct test of the 'arbitrary dimensions' claim would be to train a multimodal model on mixed text and image positions using the same SeqPE code with two data-dimension embeddings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. SeqPE proposes a learnable position encoding scheme in which an n-dimensional position is converted into a left-padded digit sequence and mapped to a hidden vector by a small Transformer encoder. Two auxiliary losses regularize the resulting embedding space: a contrastive loss that aligns embedding distances with Euclidean position distances, and a knowledge-distillation loss that transfers similarity patterns from in-distribution teacher positions to shifted out-of-distribution positions. The method is evaluated on Wikitext-103 language modeling, RULER-SQuAD long-context question answering, and ImageNet-1K image classification, with the paper reporting gains over RoPE, ALiBi, APE, and related baselines, especially at longer contexts and higher image resolutions.

Significance. If the extrapolation and multi-dimensional generalization claims were supported, SeqPE would be a useful unified position-encoding framework: it replaces a linear-size lookup table with an encoder whose parameter count grows logarithmically with the number of positions, it can be precomputed at inference time, and it extends from 1D text to 2D images without manual architectural redesign. The two dimensionality-agnostic regularizers are conceptually clean, and the stated release of code, data, and checkpoints is a strength. However, the central extrapolation claim is not established by the experiments as run, because the position encoder is trained on positions covering the full evaluation range. The significance of the paper therefore remains conditional on additional controlled experiments.

major comments (4)
  1. [Appendix A.4 / Section 4] The hyperparameter Lmax is set to 20,000 for text and 100×100 for images (Appendix A.4), and 10% of main-task windows use shifted starts z in [0, Lmax − L) (Section 4). Since L_delta and L_OOD sample positions from [0, Lmax), the position encoder is directly trained on absolute positions up to 19,999 for text and 99×99 for images. The evaluations at 4K–16K tokens and at resolutions up to 672 pixels therefore fall inside the training range of the position encoder. These experiments do not test the paper's central claims of 'context length extrapolation' and generalization to 'unseen positions' for SeqPE; they only test the main task model's ability to use position embeddings that have already been optimized on those positions. A control with Lmax equal to the training context (512 tokens; 14×14 patches) is required to support the extrapolation claim.
  2. [Section 4 / Table 4] The 10% shifted-start augmentation is never ablated. All rows of Table 4 keep the settings of Section 4.1 and Section 4.3, which include the shift. Because RoPE and ALiBi are invariant to the shift (Appendix A.5), they cannot benefit from this augmentation, whereas SeqPE's position encoder is trained on the shifted positions. The reported gains could therefore be due to the augmentation rather than to the sequential encoder or the regularizers. The paper should include an ablation with the shift disabled and, for the extrapolation control, with Lmax equal to the training context length.
  3. [Section 4.3 / Table 3] The image classification baselines APE-LEARN, ROPE2D, and ROPE2D-MIX are 'reported from the model pre-trained by [19]', while SeqPE is trained under the recipe in Appendix B.4. Without retraining all methods under identical data, epochs, optimizer, and augmentation settings, the 2.1-point average accuracy advantage attributed to SeqPE may reflect training-recipe differences rather than the position encoding. Please report results for baselines trained in the same codebase and with the same budget.
  4. [Tables 1 and 2] No error bars, confidence intervals, or multiple seeds are reported. Several head-to-head margins are small or reversed at in-distribution lengths (e.g., Table 2: at 1K, ALiBi perplexity is 5.49 versus SeqPE's 5.56), so the claim that SeqPE 'surpasses' the baselines is not statistically supported by the presented evidence.
minor comments (6)
  1. [Full text, around Section 3] The manuscript contains several pages of unrelated material—repeated passages about Visual Transformer localization and a 'schematic representation of the VT architecture' with classes 'Bird/Ball/Car'—that do not belong to this paper. This material must be removed and its provenance checked.
  2. [Abstract / Table 1] The abstract states that SeqPE outperforms other baselines by 'at least 0.6 perplexity points' on Wikitext-103, but Table 1 shows an average gap of 0.59 (18.95 versus 19.54). Please reconcile the wording with the reported numbers.
  3. [Appendix A.3] The text refers to 'P and Q similarity matrices', but Equation (10) and the surrounding discussion use P and S for the teacher and student matrices. Please use consistent notation throughout.
  4. [Tables 1 and 3] Several table entries have missing spaces, e.g., 'APE-SIN23.37' in Table 1 and 'APE-SIN80.9' in Table 3.
  5. [Section 4 / Appendix A.5] The claim that the random shift 'ensures a fair comparison' is argued only for RoPE and ALiBi, which are shift-invariant. The treatment of APE-SIN and APE-LEARN under the shifted-start training is not described, so it is unclear whether those baselines saw the same data.
  6. [Section 5.2] There is a typo in 'image resolitions'; this should read 'resolutions'.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline extrapolation claim is not tested as stated: SeqPE's regularizers train on positions up to Lmax=20,000 (text) and 100×100 (image), so the 16K-token and 672-pixel evaluations fall inside the position encoder's fitted range.

  1. fitted input called prediction [Appendix A.4, Section 4.1, and Table 3]
    "In the text and image tasks, the training context length are 512 and 14×14, and the hyper-parameter Lmax for the two objectives are set to 20,000 and 100×100, respectively. ... we evaluate it on the test set with extended sequences with L∈[1024,16384] tokens. ... The training image resolution is 224×224 with patch size 16×16, while the testing resolutions are from 224×224 to 672×672."

    The paper's central claim is context-length extrapolation, but the two regularizers Lδ and LOOD sample positions from [0,Lmax) with Lmax=20,000 for text and 100×100 for image patches. Every evaluation position up to 16K tokens (text) and up to 42×42 patches (672-pixel images) is therefore inside the SeqPE encoder's training distribution. The abstract and conclusion attribute the gains to 'context length extrapolation' and 'effective generalization to sequences significantly longer than training examples,' yet the position encoder was directly regularized on those very positions.

full rationale

One load-bearing step is identified: the paper's extrapolation evaluation is not an extrapolation test for the SeqPE position encoder, because the auxiliary objectives train on positions up to Lmax=20,000 (text) and 100×100 (image), while the maximum test lengths are 16K tokens and 672-pixel images (42×42 patches). This makes the claimed context-length and resolution extrapolation partially reduce to the range of the regularizers, so the central 'extrapolation' result is partly fitted input called prediction. I did not find other circularity: the contrastive objective uses an external Euclidean distance function and the distillation objective uses in-distribution teacher representations, neither of which is defined in terms of the target metric; the main-task perplexity, EM, and accuracy results are measured on independent test sets. The paper does contain a self-citation to prior work by co-authors (reference [38] on masked jigsaw puzzle position embeddings), but it appears only in related work and is not load-bearing for the derivation. The method has independent content—the sequential position representation and the two regularizers are genuine construction choices—so the circularity is partial rather than total.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central method rests on several design choices (digit representation, Lmax, regularizer weights) that are not derived from first principles. The main confound is Lmax being larger than the evaluation range, which weakens the extrapolation claim. No new physical entities are introduced.

free parameters (5)
  • alpha (L_delta weight) = 0.1
    Hyperparameter selected on validation performance; affects balance between main task and distance regularization.
  • beta (L_OOD weight) = 0.1
    Hyperparameter selected on validation; controls distillation strength.
  • Lmax for position sampling = 20,000 (text); 100x100 (image)
    Sets the upper bound of positions the encoder sees during regularization; evaluation lengths stay below this value.
  • digits per dimension k = 5 (text); 2 (image)
    Length of each dimension's digit sequence; chosen based on maximum needed positions.
  • batch size for L_delta/L_OOD = 32
    Set based on guidance in Appendix A.4; affects training frequency of positions.
assumptions (4)
  • domain assumption Euclidean distance is a suitable predefined distance function for positions in arbitrary dimensions.
    Used in Equation 9 to define the positive pair p+; Section 3.3.
  • domain assumption A 2-layer causal Transformer with a CLS token can encode sequential position strings sufficiently well.
    Encoder architecture in Section 3.2; motivated by preliminary experiments.
  • ad hoc to paper Left-padded decimal digit representation with fixed k digits per dimension preserves place-value alignment.
    Core representation choice in Section 3.2; no theoretical guarantee.
  • domain assumption The knowledge distillation from in-distribution to shifted positions transfers useful relative patterns and does not collapse embeddings when combined with L_delta.
    Discussion in Section 3.4 and ablation in Table 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SeqPE: Transformer with Sequential Position Encoding." pith.science (2026). https://pith.science/paper/LIYU2575

@misc{pith2026250613277,
  author       = {Pith},
  title        = {Pith review of: SeqPE: Transformer with Sequential Position Encoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LIYU2575}},
  note         = {Machine review of arXiv:2506.13277}
}
abstract

Since self-attention layers in Transformers are permutation invariant by design, positional encodings must be explicitly incorporated to enable spatial understanding. However, fixed-size lookup tables used in traditional learnable position embeddings (PEs) limit extrapolation capabilities beyond pre-trained sequence lengths. Expert-designed methods such as ALiBi and RoPE, mitigate this limitation but demand extensive modifications for adapting to new modalities, underscoring fundamental challenges in adaptability and scalability. In this work, we present SeqPE, a unified and fully learnable position encoding framework that represents each $n$-dimensional position index as a symbolic sequence and employs a lightweight sequential position encoder to learn their embeddings in an end-to-end manner. To regularize SeqPE's embedding space, we introduce two complementary objectives: a contrastive objective that aligns embedding distances with a predefined position-distance function, and a knowledge distillation loss that anchors out-of-distribution position embeddings to in-distribution teacher representations, further enhancing extrapolation performance. Experiments across language modeling, long-context question answering, and 2D image classification demonstrate that SeqPE not only surpasses strong baselines in perplexity, exact match (EM), and accuracy--particularly under context length extrapolation--but also enables seamless generalization to multi-dimensional inputs without requiring manual architectural redesign. We release our code, data, and checkpoints at https://github.com/ghrua/seqpe.

Figures

Figures reproduced from arXiv: 2506.13277 by the authors.

Figure 1
Figure 1. A schematic representation of the VT architecture. (a) A typical second-generation VT. (b) Our localization MLP which takes as input (concatenated) pairs of final token embeddings. 3 Preliminaries A typical VT network takes as input an image split in a grid of (possibly overlapping) K ⇥ K patches. Each patch is projected in the input embedding space, obtaining a set of K ⇥ K input tokens.A VT is based on the Transfo… view at source ↗
Figure 2
Figure 2. Heatmap for the values of dot product between posi [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Heatmap for position embeddings of 2D images with [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The placeholder MULTI_DOCS represents the long context with multiple Wikipedia documents, QUESTION is the user question, and ANSWER is the target answer. We evaluate performance by measuring the perplexity of the gold answer or generating a hypothesis using the fine-tu…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RePo: Language Models with Context Re-Positioning

    cs.LG 2025-12 conditional novelty 7.0 of 10

    RePo, a learned per-head token position assigner, improves OLMo-2's performance on noisy-context, structured-data, and long-context tasks by 0.6-6.9 points while keeping general-task scores roughly unchanged.

Reference graph

Works this paper leans on

54 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [19]

    Rotary position embedding for vision trans- former

    Byeongho Heo, Song Park, Dongyoon Han, and Sang- doo Yun. Rotary position embedding for vision trans- former. InEuropean Conference on Computer Vision (ECCV), 2024

    work page 2024

  2. [1]

    The fal- con series of open language models.arXiv preprint arXiv:2311.16867, 2023

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The fal- con series of open language models.arXiv preprint arXiv:2311.16867, 2023

    arXiv 2023

  3. [2]

    Lex- ical generalization improves with larger models and longer training

    Elron Bandel, Yoav Goldberg, and Yanai Elazar. Lex- ical generalization improves with larger models and longer training. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors,Findings of the Association for Computational Linguistics: EMNLP, 2022

    work page 2022

  4. [3]

    Language models are few-shot learners.Ad- vances in Neural Information Processing Systems (NeurIPS), 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Ad- vances in Neural Information Processing Systems (NeurIPS), 2020

    work page 2020

  5. [4]

    CLEX: Continuous length PREPRINT 10 extrapolation for large language models

    Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. CLEX: Continuous length PREPRINT 10 extrapolation for large language models. InInternational Conference on Learning Representations (ICLR), 2024

    work page 2024

  6. [5]

    Neural ordinary differential equations.Advances in Neural Information Processing Systems (NeurIPS), 2018

    Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations.Advances in Neural Information Processing Systems (NeurIPS), 2018

    work page 2018

  7. [6]

    Extending context window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595, 2023

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595, 2023

    arXiv 2023

  8. [7]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. InInternational Conference on Machine Learning (ICML), 2020

    2020

Show all 54 references
  1. [8]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019

    2019

  2. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  3. [10]

    Vitar: Vision transformer with any resolution.arXiv preprint arXiv:2403.18361, 2024

    Qihang Fan, Quanzeng You, Xiaotian Han, Yongfei Liu, Yunzhe Tao, Huaibo Huang, Ran He, and Hongxia Yang. Vitar: Vision transformer with any resolution.arXiv preprint arXiv:2403.18361, 2024

    2024 arXiv

  4. [11]

    SimCSE: Simple contrastive learning of sentence embeddings

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Proceedings of the Confer- ence on Empirical Methods in Natural Language Processing (EMN...

    2021

  5. [12]

    Convolutional sequence to sequence learning

    Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. InInternational Conference on Machine Learning (ICML), pages 1243–1252, 2017

    2017

  6. [13]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Man- nat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  7. [14]

    Bootstrap your own latent-a new approach to self-supervised learn- ing.Advances in Neural Information Processing Systems (NeurIPS), 2020

    Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learn- ing.Advances in Neural ...

    2020

  8. [15]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  9. [16]

    Transformer language models without positional encodings still learn positional information.arXiv preprint arXiv:2203.16634, 2022

    Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information.arXiv preprint arXiv:2203.16634, 2022

    2022 arXiv

  10. [17]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020

    2020

  11. [18]

    Deberta: Decoding-enhanced bert with disentan- gled attention

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentan- gled attention. InInternational Conference on Learning Representations (ICLR), 2020

    2020

  12. [20]

    RULER: What’s the real context size of your long- context language models? InFirst Conference on Language Modeling, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shan- tanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What’s the real context size of your long- context language models? InFirst Conference on Language Modeling, 2024

    2024

  13. [21]

    Fourier position embedding: Enhancing attention’s periodic extension for length generalization.arXiv preprint arXiv:2412.17739, 2024

    Ermo Hua, Che Jiang, Xingtai Lv, Kaiyan Zhang, Ning Ding, Youbang Sun, Biqing Qi, Yuchen Fan, Xuekai Zhu, and Bowen Zhou. Fourier position embedding: Enhancing attention’s periodic extension for length generalization.arXiv preprint arXiv:2412.17739, 2024

    2024 arXiv

  14. [22]

    Improve transformer models with better relative position embeddings.arXiv preprint arXiv:2009.13658, 2020

    Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang. Improve transformer models with better relative position embeddings.arXiv preprint arXiv:2009.13658, 2020

    2009 arXiv

  15. [23]

    Adversarial examples for evaluating reading comprehension systems

    Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017

    2017

  16. [24]

    The impact of positional encoding on length generalization in transformers.Advances in Neural Information Process- ing Systems (NeurIPS), 2023

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers.Advances in Neural Information Process- ing Systems (NeurIPS), 2023

    2023

  17. [25]

    Rethinking posi- tional encoding in language pre-training

    Guolin Ke, Di He, and Tie-Yan Liu. Rethinking posi- tional encoding in language pre-training. InInternational Conference on Learning Representations

  18. [26]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gon- zalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles, ...

    2023

  19. [27]

    Aria: An open multi- modal native mixture-of-experts model.arXiv preprint arXiv:2410.05993, 2024

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang, Yanpeng Li, et al. Aria: An open multi- modal native mixture-of-experts model.arXiv preprint arXiv:2410.05993, 2024

    2024 arXiv

  20. [28]

    Learning to encode position for transformer with continuous dynamical model

    Xuanqing Liu, Hsiang-Fu Yu, Inderjit Dhillon, and Cho- Jui Hsieh. Learning to encode position for transformer with continuous dynamical model. InInternational Conference on Machine Learning (ICML), 2020

    2020

  21. [29]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InProceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV), 2021. PREPRINT 11

    2021

  22. [30]

    Base of rope bounds context length.arXiv preprint arXiv:2405.14591, 2024

    Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng Chen. Base of rope bounds context length.arXiv preprint arXiv:2405.14591, 2024

    2024 arXiv

  23. [31]

    Liere: Gen- eralizing rotary position encodings.arXiv preprint arXiv:2406.10322, 2024

    Sophie Ostmeier, Brian Axelrod, Michael E Moseley, Akshay Chaudhari, and Curtis Langlotz. Liere: Gen- eralizing rotary position encodings.arXiv preprint arXiv:2406.10322, 2024

    2024 arXiv

  24. [32]

    Yarn: Efficient context window extension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. InThe Twelfth International Conference on Learning Representations

  25. [33]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  26. [34]

    Learning transferable visual models from natural lan- guage supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. InInternational Conference on Machine Learning (I...

    2021

  27. [35]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  28. [36]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of Machine Learning Research, 21:1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of Machine Learning Research, 21:1–67, 2020

    2020

  29. [37]

    SQuAD: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors,Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016

    2016

  30. [38]

    Masked jigsaw puzzle: A versatile position embedding for vision transformers

    Bin Ren, Yahui Liu, Yue Song, Wei Bi, Rita Cucchiara, Nicu Sebe, and Wei Wang. Masked jigsaw puzzle: A versatile position embedding for vision transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  31. [39]

    Randomized positional en- codings boost length generalization of transformers

    Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. Randomized positional en- codings boost length generalization of transformers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Proceedings...

    2023

  32. [40]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexan- der C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge.International Journal of Computer Vision (IJ...

    2015

  33. [41]

    Self- attention with relative position representations.arXiv preprint arXiv:1803.02155, 2018

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self- attention with relative position representations.arXiv preprint arXiv:1803.02155, 2018

    2018 arXiv

  34. [42]

    Roformer: Enhanced trans- former with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced trans- former with rotary position embedding.Neurocomputing, 568:127063, 2024

    2024

  35. [43]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Fran- cisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Marina Meila and Tong Zhang, editors,International Conference on Machine Learning (ICML), 2021

    2021

  36. [44]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  37. [45]

    Neural discrete representation learning.Advances in Neural Information Processing Systems (NeurIPS), 2017

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  38. [46]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett,...

    2017

  39. [47]

    Encoding word order in complex embeddings

    Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. Encoding word order in complex embeddings. InInternational Conference on Learning Representations (ICLR), 2019

    2019

  40. [48]

    Scaling context, not parameters: Training a compact 7b language model for efficient long-context processing.arXiv preprint arXiv:2505.08651, 2025

    Chen Wu and Yin Song. Scaling context, not parameters: Training a compact 7b language model for efficient long-context processing.arXiv preprint arXiv:2505.08651, 2025

    2025 arXiv

  41. [49]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chen- gen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  42. [50]

    Rope to nope and back again: A new hybrid attention strategy.arXiv preprint arXiv:2501.18795, 2025

    Bowen Yang, Bharat Venkitesh, Dwarak Talupuru, Hangyu Lin, David Cairuz, Phil Blunsom, and Acyr Locatelli. Rope to nope and back again: A new hybrid attention strategy.arXiv preprint arXiv:2501.18795, 2025

    2025

  43. [51]

    Length extrapolation of transformers: A survey from the perspective of position encoding.arXiv preprint arXiv:2312.17044, 2023

    Liang Zhao, Xiaocheng Feng, Xiachong Feng, Bin Qin, and Ting Liu. Length extrapolation of transformers: A survey from the perspective of position encoding.arXiv preprint arXiv:2312.17044, 2023

    2023 arXiv

  44. [52]

    Gonzalez, Clark Bar- rett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Bar- rett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. In A. Globerson, L. Mackey, D. B...

    2024

  45. [53]

    Trans- fusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tiru- mala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Trans- fusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024

    2024 arXiv

  46. [54]

    2”, . . . , s0 k =“3

    Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. Pose: Efficient context window extension of llms via positional skip-wise training.arXiv preprint arXiv:2309.10400, 2023. PREPRINT 12 APPENDIXA IMPLEMENTATIONDETAILS OFSEQPE A.1 Example of 1D Posi...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.