Pith. sign in

REVIEW 17 cited by

Repeat After Me: Transformers are Better than State Space Models at Copying

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.01032 v2 pith:BIPRLVCH submitted 2024-02-01 cs.LG cs.AIcs.CL

Samy Jelassi , David Brandfonbrener , Sham M. Kakade , Eran Malach This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords modelscopyinggssmsstatetransformerscontextspacetasks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Transformers are the dominant architecture for sequence modeling, but there is growing interest in models that use a fixed-size latent state that does not depend on the sequence length, which we refer to as "generalized state space models" (GSSMs). In this paper we show that while GSSMs are promising in terms of inference-time efficiency, they are limited compared to transformer models on tasks that require copying from the input context. We start with a theoretical analysis of the simple task of string copying and prove that a two layer transformer can copy strings of exponential length while GSSMs are fundamentally limited by their fixed-size latent state. Empirically, we find that transformers outperform GSSMs in terms of efficiency and generalization on synthetic tasks that require copying the context. Finally, we evaluate pretrained large language models and find that transformer models dramatically outperform state space models at copying and retrieving information from context. Taken together, these results suggest a fundamental gap between transformers and GSSMs on tasks of practical interest.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Learned replacement non-linearities show transformers are rarely optimal for algorithmic tasks, with benefits that are task-specific, while language/code gains are smaller and more transferable.

  2. Dynamic Chunking for End-to-End Hierarchical Sequence Modeling

    cs.LG 2025-07 conditional novelty 7.0 of 10

    A learned dynamic chunking hierarchy lets byte-level language models match or beat BPE-tokenized Transformers at matched compute, with larger gains on Chinese, code, and DNA.

  3. Learning Compositional Functions with Transformers from Easy-to-Hard Data

    cs.LG 2025-05 conditional novelty 7.0 of 10

    A transformer with O(log k) layers provably learns the k-fold permutation composition task in poly(N,k) samples with curriculum or mixed easy-to-hard data, despite an SQ lower bound requiring N^{Omega(k)} samples on h...

  4. DSSMs: State Space Models with Explicit Memory via Delay Differential Equations

    cs.LG 2026-07 conditional novelty 6.5 of 10

    Delay State Space Models augment diagonal SSMs with explicit delayed feedback, stable discrete parameterization, and FFT training, improving delayed-retrieval tasks and matching or beating S4D on most standard sequenc...

  5. DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling

    cs.LG 2026-08 conditional novelty 6.0 of 10

    DART augments Mamba-2 with attention over its own chunk state contributions, improving associative recall and retrieval with a smaller inference cache than token-level attention.

  6. Raven: High-Recall Sequence Modeling with Sparse Memory Routing

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Raven is a linear-time sequence model whose sparse, input-dependent routing writes tokens into dedicated memory slots, preserving long-context recall and extrapolating 16x beyond training length.

  7. Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KATA uses rank-one PSD feature maps to pack exponentially many nearly orthogonal keys at fixed interference, reaching near-softmax MQAR at 16× length with about a quarter of softmax's KV-cache entries.

  8. The Capability Convergence Hypothesis: Capability from Access Structure, Not Scale

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Long-context capability converges to hybrid architectures pairing O(1) state with a scalable index; pure scaling of SSMs or Transformers alone hits information, horizon, or circuit walls.

  9. Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A DP-means allocate-on-novelty cache matches full-attention associative recall while storing only distinct items, and a minimal novelty gate recovers the rule end-to-end.

  10. Towards Understanding What State Space Models Learn About Code

    cs.AI 2026-02 conditional novelty 6.0 of 10

    SSM code models capture code syntax and semantics better than Transformers before fine-tuning, forget short-range structure when fine-tuned on type inference, and an added high-frequency path or more kernels recovers ...

  11. Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers

    cs.CL 2025-12 conditional novelty 6.0 of 10

    Canon layers—residual 1-d causal convolutions over adjacent tokens—boost synthetic reasoning depth 2-4x, lift NoPE to RoPE level, and bring GLA up to Mamba2/GDN, with qualitative real-world confirmation.

  12. Controllably Efficient Language Models

    cs.LG 2025-11 conditional novelty 6.0 of 10

    A single transformer variant can compress past context into chunk summaries and use chunk size as a test-time knob to trade quality against speed and memory, outperforming many efficient baselines on recall benchmarks.

  13. Transformers Meet In-Context Learning: A Universal Approximation Theory

    cs.LG 2025-06 accept novelty 6.0 of 10

    A constructive theorem shows that transformers can perform in-context learning for any Barron-type function class by combining universal features with an emulated Lasso solver.

  14. Sparsified State-Space Models are Efficient Highway Networks

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Simba prunes tokens hierarchically in pre-trained SSMs, creating sparse upper layers that act as highways, improving the accuracy-FLOPs trade-off and long-context perplexity.

  15. Context by Distinct Information: An Auditable Dirichlet-Process Working Memory for Long, Redundant Context Streams

    cs.LG 2026-07 conditional novelty 5.5 of 10

    A Dirichlet-process novelty cache plus recurrence matches full-attention coupling at about half the attended tokens on long redundant streams, with inspectable retained slots.

  16. LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid

    cs.LG 2025-02 conditional novelty 5.0 of 10

    LASP-2 trains linear-attention transformers with long sequences by exchanging GPU memory states in one all-gather step, improving throughput over prior sequence-parallel methods.

  17. To model human linguistic prediction, make LLMs less superhuman

    cs.CL 2025-10 accept novelty 4.0 of 10

    LLMs' next-word predictions are too accurate to model human reading because their memory is superhuman, so cognitive modeling should build LLMs with human-like memory.

Pith tools