Pith Number

pith:DMFM5PS5

pith:2026:DMFM5PS5KZSTDL3W5PZ3UXDNKM

not attested not anchored not stored refs resolved

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably

Aram Galstyan, Eliu A Huerta, Hao Peng, Minyang Tian, Phillip Harris, Srikanth Ronanki, Subendhu Rongali, Yufeng Du

RoPE-based attention loses locality bias and token relevance consistency as context length grows, with failure probability approaching 0.5.

arxiv:2605.15514 v1 · 2026-05-15 · cs.CL · cs.AI · cs.LG

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{DMFM5PS5KZSTDL3W5PZ3UXDNKM}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

We prove that as context length increases, RoPE-based attention becomes unpredictable and loses its locality bias and consistency in token relevance, with the probability of failure approaching 0.5.

C2weakest assumption

The theoretical analysis abstracts away from the specific content of the context and depends only on its length.

C3one line summary

Proves that RoPE attention loses locality bias and token distinction in long contexts, approaching random behavior independent of content.

References

55 extracted · 55 resolved · 2 Pith anchors

[1] RoFormer: Enhanced Transformer with Rotary Position Embedding , author=. 2021 , eprint= 2021

[2] Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context , author=. 2024 , eprint= 2024

[3] Proceedings of the 41st International Conference on Machine Learning , articleno = 2024

[4] 2026 , institution = 2026

[5] International Conference on Learning Representations , year=

Receipt and verification

First computed	2026-05-20T00:01:02.609785Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

1b0acebe5d566531af76ebf3ba5c6d53364799f45aa2e706745bf1c784738d33

Aliases

arxiv: 2605.15514 · arxiv_version: 2605.15514v1 · doi: 10.48550/arxiv.2605.15514 · pith_short_12: DMFM5PS5KZST · pith_short_16: DMFM5PS5KZSTDL3W · pith_short_8: DMFM5PS5

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/DMFM5PS5KZSTDL3W5PZ3UXDNKM \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 1b0acebe5d566531af76ebf3ba5c6d53364799f45aa2e706745bf1c784738d33

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "310a4e2e0e2c9f087f8e9d1a8afe364b09bee276480ed57e149f56fabbfc59eb",
    "cross_cats_sorted": [
      "cs.AI",
      "cs.LG"
    ],
    "license": "http://creativecommons.org/licenses/by-nc-sa/4.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2026-05-15T01:16:16Z",
    "title_canon_sha256": "1e6babc51c35f5873e9ea13697cfd84dec7866f14de727051ee852cf1cbb42aa"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2605.15514",
    "kind": "arxiv",
    "version": 1
  }
}