Pith. sign in

REVIEW 19 cited by

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2006.16236 v3 pith:XBCSNFJQ submitted 2020-06-29 cs.LG stat.ML

Angelos Katharopoulos , Apoorv Vyas , Nikolaos Pappas , François Fleuret This is my paper
classification cs.LGstat.ML
keywords transformersautoregressivelinearachievecomplexityleftlengthlong
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.

Discussion (0). Sign in to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation

    cs.LG 2025-09 unverdicted novelty 7.0 of 10

    Robust Filter Attention models self-attention as consistency-based state estimation under a linear SDE for token trajectories, matching standard attention complexity while showing lower perplexity and better zero-shot...

  2. Planning as Emergent Behavior in Reinforcement Learning with Relational Hidden States

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Relational hidden states anchored to environment states are what let a model-free RL agent plan, and a free-slot control without that anchoring shows no planning signatures.

  3. Infrared Organization and Critical Cognitive Field Formation in Transformer Dynamics

    cs.LG 2026-07 reject novelty 6.0 of 10

    Slow relaxation modes in Pythia transformers accumulate toward zero rate during training, yielding a near-flat infrared spectrum and 1/t memory kernels—but the claimed 'critical cognitive field formation' is not direc...

  4. Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

    cs.LG 2026-07 conditional novelty 6.0 of 10

    SDM sparsifies the Gated DeltaNet update rule to enable 1000x larger recurrent memory states at iso-FLOP, improving long-context recall and short-context reasoning over GDN and matching full attention at 8B scale.

  5. ELiTeFormer: An Efficient Transformer for FPGAs

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Hybrid linear attention plus ternary projections, co-designed with a multiplier-free PE, deliver 10× weight and 12.8× KV-cache compression with competitive MMLU and FPGA speedups over LLaMA 3 on A100.

  6. Memory-Managed Long-Context Attention: Bounded Editable Memory with a Hard Lifecycle and Calibrated Sparse Fallback

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Editable bounded memory slots with lifecycle control plus sparse fallback cover overwrite and no-signal long-context cases that pure fixed-state or pure sparse methods fail under controlled conditions.

  7. Structured Recurrent Mixers for Massively Parallelized Sequence Generation

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    Structured Recurrent Mixers provide a dual parallel-recurrent representation for sequence models, claiming superior training efficiency, information capacity, and inference throughput over linear complexity alternatives.

  8. Incremental Transformer Neural Processes

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A causally masked, KV-cached Transformer Neural Process (incTNP) updates its context in linear time and matches or exceeds standard TNP accuracy on tabular and weather-stream benchmarks.

  9. ICR-RL: Deep Reinforcement Learning via In-Context Regression

    cs.LG 2025-09 conditional novelty 6.0 of 10

    A pre-trained tabular regression transformer (TabPFN) can act as a Q-function approximator in fitted Q iteration, matching DQN on CartPole, MountainCar, and Acrobot without gradient updates.

  10. Customizing the Inductive Biases of Softmax Attention using Structured Matrices

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Structured-matrix scoring functions, BTT and MLR, let attention escape the low-rank bottleneck and add a distance-dependent compute bias, improving accuracy for fixed compute on regression, language modeling, and forecasting.

  11. Workflow-Based Evaluation of Music Generation Systems

    eess.AS 2025-06 conditional novelty 5.0 of 10

    A single-producer workflow evaluation of eight music AI tools finds they work as idea and sound generators but not as complete composers, and proposes a reusable framework.

  12. A Theoretical Study of (Hyper) Self-Attention through the Lens of Interactions: Representation, Training, Generalization

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Single-layer linear self-attention can represent, train on, and length-generalize pairwise interaction functions under data-versatility and exact-realizability assumptions, and the paper introduces higher-order HyperA...

  13. GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference

    cs.AI 2026-06 reject novelty 4.0 of 10

    GLIDE allocates softmax attention only to early/middle transformer layers and linearizes later layers, cutting KV-cache I/O by up to ~62× while retaining 92–96% of baseline accuracy.

  14. Cognitive Field Theory: Memory-Dressed Collective Dynamics of Intelligence

    q-bio.NC 2026-01 reject novelty 4.0 of 10

    The paper asserts that Hopfield networks, RNNs, transformers, and the author's FHRN model are all special cases of a single stochastic field equation whose collective time-scale spectrum governs cognition.

  15. Evaluation of Finetuned LLMs in AMR Parsing

    cs.CL 2025-08 conditional novelty 4.0 of 10

    Simple finetuning of LLaMA 3.2 reaches SMATCH F1 0.804 on the AMR 3.0 test set, matching the APT+Silver parser and coming within 0.05 of the Graphene state of the art.

  16. Scaling Context Requires Rethinking Attention

    cs.LG 2025-07 conditional novelty 4.0 of 10

    On 64k-token natural language training, power attention with degree 2 achieves lower loss per FLOP than both softmax attention and existing linear attention, and its GPU kernels run faster than Flash Attention at long...

  17. WaveLLDM: Design and Development of a Lightweight Latent Diffusion Model for Speech Enhancement and Restoration

    cs.SD 2025-08 conditional novelty 3.0 of 10

    WaveLLDM, a lightweight latent diffusion model with a neural codec, achieves low spectral distortion (LSD 0.48-0.60) on speech restoration but scores far below SOTA on PESQ and STOI.

  18. Rethinking Transformer Connectivity: TLinFormer, A Path to Exact, Full Context-Aware Linear Attention

    cs.LG 2025-08 reject novelty 3.0 of 10

    TLinFormer compresses long history into a fixed-size context state to make each full forward pass linear in sequence length, but the attention is not exact and per-token generation still costs O(N).

  19. A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents

    cs.AI 2025-10 unverdicted novelty 2.0 of 10

    A position paper proposing compact, domain-specific AI agents as the path to ≥1000× energy efficiency, without demonstrating the claim.

Pith tools