Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper revives a pre-GPT-2 result: multi-layer autoregressive Transformers are fully position-sensitive, so explicit positional encodings are nonessential once the model has more than one layer.

desk verdict A useful corrective review with a nice explanatory figure, held back by an unproven and overbroad formal claim that should be qualified to generic parameters. read the letter →

arxiv 2501.00659 v2 pith:EOHBJ7MR submitted 2024-12-31 cs.LG cs.CL

Kazuki Irie This is my paper · ORCID
classification cs.LGcs.CL
keywords positionalencodingsautoregressivetransformersself-attentionpermutationinvariancepositionsensitivitylanguagemodelinglengthgeneralizationlinear
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to re-establish a result that was known to language-modeling practitioners around the time of GPT-2 but has been repeatedly rediscovered and misattributed since: deep autoregressive Transformers do not need explicit positional encodings. The claim is that any autoregressive Transformer with two or more layers is fully position-sensitive, meaning a change in the input at any position changes the model's output at that position and every later position, so permuted sequences can be distinguished without any position signal. The paper contrasts this with one-layer autoregressive models, whose last-step output is insensitive to permutation, and with non-autoregressive self-attention, which is outright permutation invariant. If true, this corrects a widespread misconception about a core component of language models and supports the practical finding that removing positional encodings is often harmless and can improve length generalization.

What carries the argument

The load-bearing object is the property the paper calls full position sensitivity, together with the cascade argument that produces it. A processor $f$ is fully position-sensitive if changing the input at position $i$ changes the output at every position $j \ge i$. The mechanism: in autoregressive self-attention without positional encodings, each layer processes the set of tokens seen so far; the first layer's output at position $t$ depends only on the prefix set, not on its order, but a difference in any prefix set changes the context available to the second layer at every later position. Stacking layers thus builds order sensitivity out of order-insensitive parts. The paper uses this to answer why non-autoregressive Transformers need positional encodings (permutation invariance), why one-layer autoregressive models need them (last position collapses permutations), why multi-layer models do not (full position sensitivity), and why the common belief otherwise is an incomplete reading of the original attention paper's 'no recurrence, no convolution' remark.

What would settle it

Take a two-layer autoregressive Transformer with fixed random weights and no positional encodings, feed it a sequence such as (a,b,c) and the permuted sequence (b,a,c), and compare the top-layer output at the last position. If the outputs are equal for any nonzero, non-tied weight configuration while the input prefixes differ, the paper's central claim fails; the expected outcome under the claim is that they differ for generic random weights. A systematic search over small random initializations would settle whether the property is generic or only typical.

Watch

Extended reading notes

Core claim

The central discovery is that a cascade of permutation-invariant set processors becomes a sequence processor in the autoregressive setting. Concretely, a Transformer layer without positional encodings, at time step $t$, sees the prefix set $\{x_1,\ldots,x_t\}$; the first layer's output at each position is therefore determined by which tokens have appeared so far, not by their order. Once two input sequences differ at any position, the second layer receives different sets at all subsequent positions, and by induction every later layer does too. The paper formalizes the needed property as full position sensitivity: for inputs $X, X'$, if $X_i \neq X'_i$ then $f(X)_j \neq f(X')_j$ for all $j \ge i$. Multi-layer autoregressive Transformers have this property; non-autoregressive self-attention and one-layer autoregressive self-attention do not, which is why the latter two require explicit positional encodings. The paper also notes that the argument is stated without an explicit proof or a generic-parameters qualification, and points to the empirical demonstration that 12-, 24-, and 42-layer language models train well without positional encodings.

Load-bearing premise

The argument assumes that whenever the set of tokens in a prefix changes, the first-layer self-attention output at that position changes, and that this difference propagates unchanged in kind through later layers to every future position; degenerate weights (zero, tied, or otherwise symmetric) can make outputs coincide.

Editorial extensions

If this is right

  • Multi-layer autoregressive Transformers can be trained without any explicit position vectors and still distinguish token-permuted sequences, so removing positional encodings is a valid architectural choice rather than an error.
  • One-layer autoregressive Transformers remain dependent on explicit positional encodings, because their output at the final position is invariant to prefix permutations.
  • The causal attention mask is itself a source of position information in deep models; a single masked self-attention layer's set-based computation already records how many tokens have passed, and deeper layers amplify that.
  • No-positional-encoding models can generalize to sequence lengths beyond those seen in training, with empirical evidence on formal languages and reasoning tasks.
  • Linear Transformers, which are equivalent to fast weight programmers with an identity transition, inherit the same one-layer-versus-multi-layer distinction with respect to positional encodings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the cascade argument suggests a general recipe for making any permutation-invariant encoder sequence-sensitive: feed it prefixes in an autoregressive stack and require at least two layers; the same logic should transfer to set transformers, graph networks, and other permutation-invariant architectures.
  • Editorial inference: the paper's full-position-sensitivity claim is made without a formal proof and without a generic-parameters caveat; one can read it as true for non-degenerate weight configurations, and a rigorous treatment would need to exclude zero, tied, or otherwise collapsed weights.
  • Editorial inference: a direct experimental test is to compare top-layer representations, at every position, for a two-layer randomly initialized Transformer fed a sequence and its token-swapped permutation; the paper's argument predicts differences at all future positions for almost all initializations.
  • Editorial inference: if the result becomes common knowledge, it may shift design practice toward simpler position-free baselines in language model infrastructure, reserving learned relative positional encodings for tasks where they measurably help rather than including them by default.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that explicit positional encodings (PEs) are nonessential for multi-layer autoregressive Transformers because a cascade of permutation-invariant set processors can become sequence-sensitive when each layer is applied autoregressively. It defines two properties, permutation invariance and full position-sensitivity, and then answers a sequence of questions about when PEs are needed, including the one-layer case, the multi-layer case, and the linear-Transformer case. The paper also reviews the literature on this topic, crediting Irie et al. (2019) with the original empirical demonstration and documenting later rediscoveries, and it closes with a metascience discussion of citation practices. The central empirical claim is supported by prior work, but the formal statement in Section 3 overreaches: the property of full position-sensitivity is asserted without a proof and without a generic-parameters qualification.

Significance. If the formal claim is properly qualified, the paper is a useful didactic and historical contribution: it gives an intuitive explanation of a result that is known but not widely disseminated, it explicitly defines the relevant notions, and it is honest about its limitations. The literature review and the quote-based documentation of rediscovery are valuable for the community. The main technical weakness is that the key theoretical statement, 'multi-layer autoregressive Transformers are fully position-sensitive,' is not established rigorously as stated; the paper itself notes that a proof could be provided but does not provide one. This prevents the paper from being a complete theoretical explanation, although the empirical support for the nonessentiality of PEs in practice remains credible.

major comments (3)
  1. [Sec. 3, Definition (2) and Question 3] The claim that multi-layer autoregressive Transformers are fully position-sensitive is false as an unqualified statement about the model class. For example, if all weight matrices in Eq. (1) are zero (Wq=Wk=Wv=0), the composed network maps every input to a constant output; for X=(a,b) and X'=(b,a) one has Xi≠X'i but f(X)_j=f(X')_j for all j, violating the definition. The paper needs to state the result for generic (e.g., Lebesgue-almost-all) parameters and for inputs with distinct token embeddings, and should supply a proof or a precise reference to one. As written, the assertion in Sec. 3, Q3 is load-bearing for the paper's central conclusion and cannot remain unqualified.
  2. [Sec. 3, Question 3 and Figure 1] The prose argument that the multi-layer model 'sees different contexts' at later positions conflates differing layer inputs with differing layer outputs. The cascade argument requires an injectivity or nondegeneracy condition at each layer: the fact that the context sets differ does not by itself guarantee that the self-attention outputs differ. Figure 1 is an instructive illustration, but it is not a proof. The paper should either add the missing mathematical argument or explicitly reframe the explanation as an informal sketch, with the formal statement restricted to a generic-parameter setting.
  3. [Sec. 3, Questions 3 and 5] The paper states that full position-sensitivity makes explicit PEs 'not needed,' but this is a statement about theoretical capacity, not about what trained models actually do. The paper later clarifies this distinction (Sec. 5), and I do not consider the omission fatal, but the main sections should be more careful: 'nonessential' can be read as a claim that PEs never help in practice, which the cited literature itself shows is not universally true (e.g., Haviv et al. 2022 and Scao et al. 2022 report slight degradation without PEs). The conclusion should consistently emphasize the capacity-level reading.
minor comments (6)
  1. [Sec. 2, Eq. (6)] The notation 'M = 1' is used for the all-ones matrix but is not explicitly defined; please state that 1 denotes the T×T matrix of all ones.
  2. [Sec. 1, footnote] The footnote uses 'petroplyph,' which appears to be a typo for 'petroglyph' as used in the title.
  3. [Sec. 3] The text contains typos: 'permutated' should be 'permuted,' and 'additonally' should be 'additionally.'
  4. [Sec. 5] 'non-symetric' should be 'non-symmetric,' and the use of 'c.f.' should be harmonized to the standard 'cf.'
  5. [Sec. 7] In the quote from Zaremba et al., the text says 'Pham et al. (2013)' while the reference list gives Pham et al. (2014); please harmonize the year.
  6. [Figure 1] The caption text refers to colors in the right diagram, but the figure may be rendered in grayscale; please make the caption self-contained or use additional visual cues such as patterns.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central explanation is structural and self-contained; the unproven 'fully position-sensitive' assertion is a rigor gap, not an input recycled as output.

full rationale

The claimed derivation is self-contained: Section 3 defines 'fully position-sensitive' as a formal output-difference property, connects it to the non-necessity of explicit positional encodings via an explicit implication, and then justifies the property for multi-layer autoregressive Transformers by the architectural cascade of set processors illustrated in Figure 1. No parameter is fitted to data and then renamed as a prediction; no equation is used to derive a result that was already assumed in the same form. The self-citations to Irie et al. (2019) in Sections 4-5 support the empirical and historical claim that no-PE Transformers perform well, and that claim is also confirmed by the independent works Haviv et al. (2022) and Kazemnejad et al. (2023), so the self-citation is not load-bearing for the mathematical explanation. The linear-Transformer discussion in Section 4 inherits the property through the derived equivalence in Appendix A rather than by definition. The main caveat is a rigor gap, not circularity: Section 3, Question 3 says 'While we could also provide a mathematical proof here' and then asserts without proof that multi-layer autoregressive Transformers are fully position-sensitive; as stated, this is false for degenerate parameters such as all-zero attention weights, and a generic-parameter qualification plus a real proof would be required. That is an unsupported theorem about the architecture, not an input of the derivation recycled as its output. Therefore, no significant circularity is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities. The argument relies on three domain assumptions: distinct tokens map to distinct embeddings; non-degenerate attention weights ensure different token sets yield different outputs; and the position-wise feedforward sublayer preserves the difference signal. These are reasonable for typical trained transformers but are not formally guaranteed.

assumptions (3)
  • domain assumption Different token sets at a prefix produce different self-attention outputs at that position.
    Used in Sec. 3 Q3 and Figure 1 to argue that layer-1 output differs when a token in the prefix changes; this fails for degenerate weights (e.g., zero attention weights).
  • domain assumption Token embeddings are injective for distinct tokens.
    The argument treats each token as a distinguishable element of the set; collisions would ruin the distinction.
  • domain assumption The position-wise feedforward network does not destroy the position-differentiating signal.
    Sec. 2 notes it processes each position independently; the paper assumes it preserves the differences from attention.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph." pith.science (2026). https://pith.science/paper/EOHBJ7MR

@misc{pith2026250100659,
  author       = {Pith},
  title        = {Pith review of: Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EOHBJ7MR}},
  note         = {Machine review of arXiv:2501.00659}
}
read the original abstract

Do autoregressive Transformer language models require explicit positional encodings (PEs)? The answer is 'no' provided they have more than one layer -- they can distinguish sequences with permuted tokens without the need for explicit PEs. This follows from the fact that a cascade of (permutation invariant) set processors can collectively exhibit sequence-sensitive behavior in the autoregressive setting. This property has been known since early efforts (contemporary with GPT-2) adopting the Transformer for language modeling. However, this result does not appear to have been well disseminated, leading to recent rediscoveries. This may be partially due to a sudden growth of the language modeling community after the advent of GPT-2/3, but perhaps also due to the lack of a clear explanation in prior work, despite being commonly understood by practitioners in the past. Here we review the long-forgotten explanation why explicit PEs are nonessential for multi-layer autoregressive Transformers (in contrast, one-layer models require PEs to discern order information of their inputs), as well as the origin of this result, and hope to re-establish it as a common knowledge.

Figures

Figures reproduced from arXiv: 2501.00659 by the authors.

Figure 1
Figure 1. A cascade of set processors can behave as a sequence processor. An illustration of autoregressive Transformers without explicit positional encodings for two input sequences: (a, b, c) and its permutation (b, a, c). The color in the right diagram highlights the differences in terms of context seen by each layer at each position (expressed as a set). With two (or more) layers, as soon as an input is different at one p… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings

    cs.LG 2025-09 conditional novelty 6.0 of 10

    PoPE, a polar-coordinate variant of RoPE that removes the content-phase interaction, improves Transformer sequence modeling and extrapolation across multiple domains.

Reference graph

Works this paper leans on

14 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [5]

    The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention. In Proc. Int. Conf. on Machine Learning (ICML), Baltimore, MD, USA. Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber

    work page

  2. [7]

    Preprint arXiv:2307.10169

    Challenges and applications of large language models. Preprint arXiv:2307.10169. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pap- pas, and François Fleuret. 2020. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proc. Int. Conf. on Machine Learning (ICML), Virtual only. Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Nat...

    arXiv 2020

  3. [9]

    Preprint arXiv:2308.12950

    Code Llama: Open foundation models for code. Preprint arXiv:2308.12950. Teven Le Scao, Thomas Wang, Daniel Hesslow, Stas Bekman, M. Saiful Bari, Stella Biderman, Hady Elsa- har, Niklas Muennighoff, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Lau- nay, and Iz Beltagy. 2022. What lang...

    arXiv 2022

  4. [10]

    Linear Transformers are secretly fast weight programmers. In Proc. Int. Conf. on Machine Learn- ing (ICML), Virtual only. Jürgen Schmidhuber. 1992. Learning to control fast- weight memories: An alternative to dynamic recur- rent networks. Neural Computation, 4(1):131–139. Jürgen Schmidhuber and Stefan Heil. 1994. Predictive coding with neural nets: Applic...

    work page 1992

  5. [11]

    Preprint arXiv:2403.07816

    Branch-train-MiX: Mixing expert llms into a mixture-of-experts llm. Preprint arXiv:2403.07816. Ilya Sutskever, James Martens, and Geoffrey E. Hinton

    arXiv

  6. [2000]

    A neural probabilistic language model. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 932–938, Denver, CO, USA. Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal

    work page

  7. [2011]

    Generating text with recurrent neural networks. In Proc. Int. Conf. on Machine Learning (ICML) , pages 1017–1024, Bellevue, W A, USA. Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov

    work page

  8. [2014]

    Recurrent neural network regularization. In Int. Conf. on Learning Representations (ICLR), San Diego, CA, USA. Albert Zeyer, Parnia Bahar, Kazuki Irie, Ralf Schlüter, and Hermann Ney. 2019. A comparison of trans- former and LSTM encoder decoder models for asr. In Proc. IEEE Automatic Speech Recog. and Under- standing Workshop (ASRU), Sentosa, Singapore. A...

    work page 2019

Show all 14 references
  1. [2019]

    Transformer dissection: An unified under- standing for transformer’s attention via the lens of kernel. In Proc. Conf. on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 4344– 4353, Hong Kong, China. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, ...

    2017

  2. [2020]

    On the ability and limitations of transformers to recognize formal languages. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 7096–7116, Virtual only. James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2017. Quasi-recurrent neu...

    2017 arXiv

  3. [2021]

    The devil is in the detail: Simple tricks im- prove systematic generalization of transformers. In Proc. Conf. on Empirical Methods in Natural Lan- guage Processing (EMNLP), Punta Cana, Dominican Republic. Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber

  4. [2022]

    The Neural Data Router: Adaptive control flow in Transformers improves systematic generalization. In Int. Conf. on Learning Representations (ICLR) , Virtual only. Hugo Cui, Freya Behrens, Florent Krzakala, and Lenka Zdeborová. 2024. A phase transition between posi- tional and ...

    2024 arXiv

  5. [2023]

    Practical computational power of linear trans- formers and their recurrent and self-referential ex- tensions. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP) , Sentosa, Singapore. Kazuki Irie, Alexander Gerstenberger, Ralf Schlüter, and Hermann Ney. ...

    2020

  6. [2024]

    The illusion of state in state-space models. In Proc. Int. Conf. on Machine Learning (ICML) , Vienna, Austria. Tomás Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recur- rent neural network based language model. In Proc. Interspeech, pages ...

    2010

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.