Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that RoPE's attention score entangles content ('what') with position ('where') through a phase term, and that removing that term—PoPE—improves perplexity, downstream accuracy, and zero-shot length extrapolation.

desk verdict PoPE is a plausible improvement to RoPE that deserves a serious referee, but the printed frequency schedule in Eq. 4 looks like a sign error that would invalidate the extrapolation results, and the causal story is under-tested because the channel count is not matched. read the letter →

arxiv 2509.10534 v3 pith:NGILMKJH submitted 2025-09-05 cs.LG cs.AIcs.CL

Anand Gopalakrishnan , Robert Csordás , Jürgen Schmidhuber , Michael C. Mozer This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords rotarypositionembeddingwhat-wheredisentanglementlengthextrapolationpositionalencodingattentionmechanismpolarcoordinatespointerarithmeticrelative
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the popular rotary position embedding (RoPE) couples the content of a query or key with its positional tuning: the phase of each 2D component shifts with the token's identity, so a query-key match is never purely about position or purely about content. It proposes PoPE, a minimal modification that strips the content-dependent phase out of the rotation, leaving an attention score that is a product of content magnitudes times a cosine of relative position. Across a diagnostic pointer-arithmetic task, music and genome modeling, and language modeling from 124M to 774M parameters, PoPE beats RoPE on loss and downstream accuracy, and it extrapolates zero-shot to sequences ten times longer than its training window. If correct, PoPE is a drop-in RoPE replacement that improves data efficiency, asymptotic accuracy, and context-length generalization.

What carries the argument

The key identity is Equation 2 versus Equation 5: RoPE's component score is mu_q mu_k cos((s-t)theta_c + phi_k - phi_q), where the content-dependent phases phi_k and phi_q act as learned offsets to the relative-position cosine; PoPE eliminates these phases, leaving mu_q mu_k cos((s-t)theta_c). Combined with a doubling of the number of frequency channels (one per element rather than one per 2D pair) and a softplus activation that guarantees non-negative magnitudes, this makes the position match orthogonal to the content match in each channel. The same polar-coordinate decomposition is also used to explain RoPE's extrapolation failure: at low frequencies, content-dependent phase shifts become

What would settle it

Train an RoPE variant that rotates every element with its own frequency (d channels instead of d/2), applies softplus to magnitudes, and uses zero phase offsets—exactly matching PoPE's capacity and nonlinearity while keeping RoPE's phase interaction. If this variant matches PoPE's perplexity, extrapolation, and Indirect Indexing accuracy, then the decoupling claim is unsupported.

Watch

Extended reading notes

Core claim

The central claim is that RoPE's attention score, written in polar coordinates as a sum over components of mu_q mu_k cos((s-t)theta_c + phi_k - phi_q), lets the key's and query's content phases (phi_k and phi_q) shift the position tuning of each frequency component, entangling the 'what' and 'where'. PoPE removes that interaction by giving every element its own frequency and using softplus-activated magnitudes with zero initial phases, yielding the score sum_c sigma(q_c) sigma(k_c) cos((s-t)theta_c). The paper demonstrates, on the Indirect Indexing task, that this decoupling turns a near-impossible task (RoPE at 11% accuracy) into one that is nearly solved (PoPE at 95%), and shows consistent

Load-bearing premise

The paper attributes PoPE's gains to removing the what-where phase interaction, but PoPE also doubles the number of frequency channels and adds a softplus nonlinearity; if those differences, rather than the decoupling, are what drive performance, the central explanation collapses.

Editorial extensions

If this is right

  • If PoPE's decoupling is the cause of its gains, position and content can be learned independently, which should improve sample efficiency on tasks requiring pointer arithmetic or exact positional indexing.
  • PoPE's stable extrapolation without fine-tuning suggests that context-length extension methods like YaRN, which re-tune frequencies after pretraining, may become unnecessary for models trained with PoPE.
  • The better high-frequency usage observed in PoPE's frequency heatmaps implies that content-carrying channels can stay active at short distances, potentially improving local pattern learning in music and genomics.
  • The consistent language-modeling gains from 124M to 774M parameters suggest the benefit does not vanish with scale, so the method transfers to larger frontier models as a direct RoPE substitute.
  • Because PoPE is a drop-in replacement implemented in a modified Flash Attention kernel, it can be adopted without changing the rest of the Transformer architecture or training recipe.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the confound explanation would be to train an RoPE variant with d frequency channels (rotating each element with its own theta, with softplus magnitudes) to match PoPE's channel count and nonlinearity; if that variant matches PoPE's performance, the decoupling story loses its causal force.
  • The paper's frequency-usage analysis suggests that PoPE's high-frequency channels carry content, which implies a learnable per-channel bias delta_c may be doing more than tuning offsets—it may be learning content-conditional position priors that RoPE gets implicitly through phi_k - phi_q.
  • The Indirect Indexing task is a crisp diagnostic that future positional-encoding methods could adopt as a standard test for what-where independence, which would make cross-paper comparisons of similar claims easier.
  • Because PoPE's extrapolation does not degrade with model size (whereas RoPE's does), the method may enable training on short sequences and deploying on much longer ones, which has practical value for long-document and long-context applications beyond what the paper explicitly reports.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper argues that RoPE entangles content ('what') and position ('where') because the relative-phase term φ_k−φ_q in Eq. (2) is content-dependent. It proposes PoPE, which represents each scalar component of query/key as a complex number whose magnitude is a softplus-transformed scalar and whose phase is purely positional (Eqs. 3–5), optionally adding a learnable per-frequency offset δ_c. Experiments compare PoPE with RoPE on a synthetic Indirect Indexing task, music (JSB, MAESTRO), human genome, and OpenWebText language modeling at 124M–774M, and report zero-shot length extrapolation to 10K on PG-19. The paper concludes that PoPE decouples what/where and improves data efficiency, asymptotic accuracy, and context-length generalization.

Significance. The potential value is high: PoPE is a simple drop-in RoPE alternative, the Indirect Indexing diagnostic is well motivated, and the perplexity gains, if reproducible, are consistent across model scales. The paper also provides a Triton flash-attention implementation and documents its computational overhead. However, the central causal attribution — that decoupling causes the observed gains — is not demonstrated because PoPE differs from RoPE in several uncontrolled ways, and the printed frequency schedule is internally inconsistent with the headline extrapolation result. These issues make the contribution currently contingent rather than established.

major comments (4)
  1. [§3, Eq. (4); Figure 2] Eq. (4) defines θ_c = θ^{(c-1)/d} with θ = 10,000, so θ_c ∈ [1, 10^4] rad/step and every channel has wavelength ≤ 2π. The score in Eq. (5) is then an oscillatory function of r = s−t with no low-frequency component; 10× zero-shot extrapolation (Fig. 2) is not reproducible from this definition. If the implementation actually used θ^{-(c-1)/d} (the usual negative-exponent schedule), Eq. (4) and the surrounding text are wrong and all empirical results inherit the discrepancy. Please correct the schedule and verify the reported extrapolation.
  2. [§3, Eq. (5); Figure 1 caption; Appendix B, Table 8] The causal interpretation is not isolated. PoPE differs from RoPE by (i) using d rotating channels instead of d/2, (ii) applying a softplus magnitude nonlinearity, and (iii) adding a learnable δ_c. Appendix B abletes softplus and δ, but never tests a RoPE baseline with d channels or a phase-removed RoPE with matched capacity. The observed gains may therefore be due to the doubled number of frequency channels or the nonlinearity rather than to decoupling 'what' and 'where'. A matched-channel RoPE control is needed to support the paper's central claim.
  3. [Tables 2–3 and captions] Tables 2–3 report 'Best NLL on the test split' with no number of seeds and no checkpoint-selection protocol. The reported gaps are small (MAESTRO 1.501 vs 1.486; HRG 4.217 vs 4.152), so without variance estimates or a validation-selected checkpoint procedure the cross-domain superiority claim is not well supported. Report mean±std over seeds or otherwise specify the selection protocol.
  4. [Abstract vs. Section 4] The abstract claims PoPE is compared 'even' to YaRN, but no YaRN experiment appears in Section 4 or Figure 2; YaRN is only discussed in Related Work. Either add the YaRN comparison or remove this claim from the abstract.
minor comments (4)
  1. [§3, after Eq. (6)] The bound statement 'we bound δ_c ... i.e. θ_c = min(max(θ_c, −2π), 0)' appears to refer to δ_c, not θ_c. Please correct the notation.
  2. [§3, Eq. (9)] The complex multiplication expansion contains '−i^2 y y'; the real part in Eq. (10) is correct, but the intermediate algebra should be written as xx' + yy' + i(xy' − yx').
  3. [Figure 2] The figure caption describes red/green curves but the plot appears to lack axis labels and a legend. Please make the figure self-contained.
  4. [§3, Eq. (4)] Once the exponent sign is fixed, define the 'base wavelength' θ clearly (e.g., whether it is the wavelength of the first channel or a base for a geometric schedule), to avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PoPE's decoupling is definitional, but all claimed improvements are measured on held-out data and are not fitted to targets.

full rationale

The paper's central construction is not circular in the sense of this review. Section 2 re-expresses the RoPE attention score in polar coordinates, exposing the content-dependent phase interaction phi_k - phi_q (Eq. 2). Section 3 defines PoPE by setting phases to position-only terms s heta_c and t heta_c, so the absence of the content-phase interaction in Eq. 5 is true by construction. However, this is a design choice, not a fitted prediction: the paper does not fit a parameter and then "predict" that same parameter. The learnable bias delta_c is part of the model trained on the actual language-modeling objective, and all reported gains (Indirect Indexing accuracy, NLL, perplexity, downstream accuracy, length extrapolation) are measured on held-out data with identical hyperparameters except for the positional encoding. No self-citation is load-bearing: citations to Schmidhuber (1992), Csordas et al. (2022), Irie et al. (2019/2025) appear only in related work, and no uniqueness theorem or prior result by the same authors is invoked to force the PoPE form. The possible frequency-range sign issue in Eq. 4 is a correctness/specification concern, not circularity. Thus the derivation chain is self-contained and the empirical comparisons are independent of the construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no invented physical entities. The central claim rests on a standard algebraic identity, on the modeling assumption that content is fully captured by softplus magnitudes, on the diagnostic task's representativeness, and on the assumption that the doubled frequency count does not explain the gains. The learnable bias delta and base wavelength theta are the main free parameters.

free parameters (2)
  • delta_c (per-frequency bias) = learned; values not reported
    Eq. 6 introduces a learnable offset delta_c for each frequency c. Its initialization affects in-distribution versus extrapolation performance (Section 3), and it is fit by training.
  • base wavelength theta = 10000 for all experiments
    Table 6 sets the base wavelength to 10000 for all experiments. Frequencies are theta^{(c-1)/d}, so this chosen hyperparameter controls the position encoding scale.
assumptions (4)
  • domain assumption Softplus-transformed magnitudes plus positional phases are a sufficient encoding, so content information survives in magnitudes alone.
    Eq. 3 sets magnitudes to softplus of key and query elements. The paper assumes this preserves the content information needed for attention, which is load-bearing for the decoupling claim but not proven.
  • ad hoc to paper The Indirect Indexing task is a valid operationalization of the need for independent what/where matching, and its difficulty transfers to real sequence tasks.
    Section 4 defines the diagnostic task. The paper generalizes from it to music, genome, and language without independent validation of the task's representativeness.
  • ad hoc to paper The doubled number of frequency channels in PoPE does not confer the observed advantages, making the comparison to RoPE a fair test of decoupling.
    Figure 1 notes PoPE uses twice the number of components as RoPE. No control with matched channel count is run, so this assumption is needed for the causal claim.
  • domain assumption The chosen base wavelength and bounded bias initialization are adequate, and their specific values do not drive the conclusions.
    Table 6 fixes theta and init ranges. No sensitivity analysis is reported, so the conclusions implicitly assume these hyperparameter choices are not decisive.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings." pith.science (2026). https://pith.science/paper/NGILMKJH

@misc{pith2026250910534,
  author       = {Pith},
  title        = {Pith review of: Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NGILMKJH}},
  note         = {Machine review of arXiv:2509.10534}
}
read the original abstract

The attention mechanism in a Transformer architecture matches key to query based on both content -- the what -- and position in a sequence -- the where. We present an analysis indicating that what and where are entangled in the popular RoPE rotary position embedding. This entanglement can impair performance particularly when decisions require independent matches on these two factors. We propose an improvement to RoPE, which we call Polar Coordinate Position Embeddings or PoPE, that eliminates the what-where confound. PoPE is far superior on a diagnostic task requiring indexing solely by position or by content. On autoregressive sequence modeling in music, genomic, and natural language domains, Transformers using PoPE as the positional encoding scheme outperform baselines using RoPE with respect to evaluation loss (perplexity) and downstream task performance. On language modeling, these gains persist across model scale, from 124M to 774M parameters. Crucially, PoPE shows strong zero-shot length extrapolation capabilities compared not only to RoPE but even a method designed for extrapolation, YaRN, which requires additional fine tuning and frequency interpolation.

Figures

Figures reproduced from arXiv: 2509.10534 by the authors.

Figure 1
Figure 1. Illustration compares how RoPE and PoPE encode relative positions via rotations of [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Length extrapolation at test-time on PG-19 dataset for different model sizes. We evaluate [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. 2-norm plotted over 2D RoPE ‘chunks’ of queries (left) and keys (right) in each layer [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Magnitude of each complex-valued features of queries (left) and keys (right) in each layer [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: 2-norm plotted over 2D RoPE components of queries (left) and keys (right) in each layer [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Magnitude of each complex-valued features of queries (left) and keys (right) in each layer [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    RoVE rotates value embeddings simultaneously with keys in attention to make values position-dependent, reframing RoPE as attentive convolution and reporting gains on long-context tasks in 124M and 354M GPT-2 models.

Reference graph

Works this paper leans on

14 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [1]

    Layer normalization.arXiv preprint arXiv:1607.06450,

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization.arXiv preprint arXiv:1607.06450,

    arXiv

  2. [2]

    We use a vocabulary size of 90 which includes the MIDI notes, silence and padding tokens

    We use a maximum sequence length of 2048 for training with 229/76/77 sequences present in the train/validation/test sets. We use a vocabulary size of 90 which includes the MIDI notes, silence and padding tokens. MAESTRO.The dataset contains about 200 hours of paired audio and MIDI recordings from ten years of International Piano-e-Competition. The MIDI da...

    work page

  3. [7]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    arXiv

  4. [9]

    LLaMA: Open and efficient foundation language models.Preprint arXiv:2302.13971,

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aur ´elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models.Preprint arXiv:2302.13971,

    arXiv

  5. [11]

    quantizations

    which was used to train GPT-2 (Radford et al., 2019). The training and validation splits roughly contain 9B and 4M tokens respectively and maximum sequence length of 1024 for pretraining. We use the GPT-2 tokenizer with a vocabulary size of 50257. Bach-Chorales.This dataset (JSB) consists of 4-part scored choral music, which are represented as a matrix wi...

    work page 2019

  6. [14]

    OpenWebText JSB MAESTRO HRG Embedding size 512 768/1024/1280 256 384 1024 Num

    Hyperparameter Indirect Idx. OpenWebText JSB MAESTRO HRG Embedding size 512 768/1024/1280 256 384 1024 Num. heads 8 12/16/20 8 8 16 Num. layers 8 12/16/36 6 6 16 Norm. type RMSNorm RMSNorm RMSNorm RMSNorm RMSNorm Base wavelength (θ) 10,000 10,000 10,000 10,000 10,000 Init. range forδ 2π 0/0/0 2π 2π 2π Dropout 0.0 0.0/0.0/0.0 0.2 0.1 0.1 A.3 TRAININGDETAIL...

    work page

  7. [328]

    We follow the preprocessing and tokenization procedures from the recent state-of-the-art model for genomic sequence modeling, the Nucleotide Transformer (Dalla-Torre et al., 2025)

    Human Reference Genome.The human reference genome (HRG) dataset was constructed by considering all autosomal and sex chromosomes sequences from reference assembly GRCh38/hg38 4 and reached a total of 3.2 billion nucleotides. We follow the preprocessing and tokenization procedures from the recent state-of-the-art model for genomic sequence modeling, the Nu...

    work page 2025

  8. [1996]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    arXiv

Show all 14 references
  1. [2012]

    Extending context window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595,

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595,

    arXiv

  2. [2019]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  3. [2020]

    Why are positional encodings nonessential for deep autoregressive transformers? revisiting a petroglyph.Preprint arXiv:2501.00659,

    Kazuki Irie. Why are positional encodings nonessential for deep autoregressive transformers? revisiting a petroglyph.Preprint arXiv:2501.00659,

    arXiv

  4. [2023]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457,

    arXiv

  5. [2024]

    Aaron Gokaslan and Vanya Cohen

    URLhttps://zenodo.org/records/12608602. Aaron Gokaslan and Vanya Cohen. Openwebtext corpus.http://Skylion007.github.io/ OpenWebTextCorpus,

  6. [2025]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram´e, Morgane Rivi`ere, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.