Pith. sign in

REVIEW 4 cited by

Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2103.03404 v2 pith:CCQ4XKRD submitted 2021-03-05 cs.LG

Yihe Dong , Jean-Baptiste Cordonnier , Andreas Loukas This is my paper · ORCID
classification cs.LG
keywords attentionoutputarchitecturesconnectionsdoublyexponentiallymlpsself-attention
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Attention-based architectures have become ubiquitous in machine learning, yet our understanding of the reasons for their effectiveness remains limited. This work proposes a new way to understand self-attention networks: we show that their output can be decomposed into a sum of smaller terms, each involving the operation of a sequence of attention heads across layers. Using this decomposition, we prove that self-attention possesses a strong inductive bias towards "token uniformity". Specifically, without skip connections or multi-layer perceptrons (MLPs), the output converges doubly exponentially to a rank-1 matrix. On the other hand, skip connections and MLPs stop the output from degeneration. Our experiments verify the identified convergence phenomena on different variants of standard transformer architectures.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Transient Reserves, Sink Dampers, and the Failure of Eigenvalue Reasoning in the Attention Propagator

    cond-mat.dis-nn 2026-07 conditional novelty 7.0 of 10

    Resolvent analysis of trained causal attention shows sinks act as transient dampers, routing heads carry excess Kreiss reserve, and eigenvalue depth predictions fail by 7–11 orders of magnitude.

  2. Many-body Tipping Dynamics of ChatGPT-like AIs

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Tipping of ChatGPT-like AI to undesirable outputs is modeled as first-passage transport of a residual-state spin across an output-basin wall, with attention disorder controlling the crossing.

  3. Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KATA uses rank-one PSD feature maps to pack exponentially many nearly orthogonal keys at fixed interference, reaching near-softmax MQAR at 16× length with about a quarter of softmax's KV-cache entries.

  4. Spectral-LSH: Sub-Quadratic Prompt Compression via Krylov-Projected Locality-Sensitive Hashing

    cs.AI 2026-06 conditional novelty 6.0 of 10

    At compression ratios above 8x, grouping tokens via a Krylov-projected LSH of an implicit attention kernel preserves LLM output quality far better than block averaging, at a large preprocessing latency cost.

Pith tools