REVIEW 16 cited by
Transformers are Multi-State RNNs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Transformers are considered conceptually different from the previous generation of state-of-the-art NLP models - recurrent neural networks (RNNs). In this work, we demonstrate that decoder-only transformers can in fact be conceptualized as unbounded multi-state RNNs - an RNN variant with unlimited hidden state size. We further show that transformers can be converted into $\textit{bounded}$ multi-state RNNs by fixing the size of their hidden state, effectively compressing their key-value cache. We introduce a novel, training-free compression policy - $\textbf{T}$oken $\textbf{O}$mission $\textbf{V}$ia $\textbf{A}$ttention (TOVA). Our experiments with four long range tasks and several LLMs show that TOVA outperforms several baseline compression policies. Particularly, our results are nearly on par with the full model, using in some cases only $\frac{1}{8}$ of the original cache size, which translates to 4.8X higher throughput. Our results shed light on the connection between transformers and RNNs, and help mitigate one of LLMs' most painful computational bottlenecks - the size of their key-value cache. We publicly release our code at https://github.com/schwartz-lab-NLP/TOVA
Forward citations
Cited by 16 Pith papers
-
Error Certificates for KV-Cache Eviction via Randomized Design
Deterministic KV-cache eviction provably cannot self-diagnose its own error; Poisson-sampled eviction with a Hajek logit offset yields an empirically valid error certificate (0.97 coverage) that attributes, but does n...
-
Learning to Evict from Key-Value Cache
Per-head RL agents that rank KV-cache tokens by predicted future utility outperform heuristic eviction baselines and generalize zero-shot to new tasks.
-
Training-Free Hashing-Based Attention via Binary Principal Components
BinaryPC uses binary principal components of keys to pick the most relevant tokens for attention, preserving accuracy with a 2% budget.
-
MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference
Region-labeled tokens in LLM agent traces have order-of-magnitude different attention lifetimes, and a decay-plus-pinning eviction policy preserves system facts under fixed KV budgets while recency collapses as context grows.
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference
RAT+ pretrains a single dense recurrent-augmented attention model that supports flexible dilated sparse inference after short adaptation, matching dense accuracy at moderate dilation and losing only 1-3 points at high...
-
OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inference
Perturbation-based OBD saliency scores for KV cache eviction beat attention-only scoring on LLaMA/Qwen long-context benchmarks.
-
Think Clearly: Improving Reasoning via Redundant Token Pruning
A training-free test-time method prunes low-attention reasoning tokens from the KV cache, guided by an injected end-of-thinking token, and reports accuracy gains on math competition benchmarks.
-
Cartridges: Lightweight and general-purpose long context representations via self-study
A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.
-
Stem: Rethinking Causal Information Flow in Sparse Attention
Stem prunes causal attention with a position-decaying budget and a value-magnitude-aware score, matching dense accuracy on LongBench/RULER at a 25-31% compute budget.
-
TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference
TPLA splits the latent KV cache across tensor-parallel GPUs while keeping every head's full view, yielding 1.79x and 1.93x decode speedups on DeepSeek-V3 and Kimi-K2 at 32K context with modest accuracy loss.
-
LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
LaCache keeps a layer-dependent diagonal slice of the KV cache and iteratively compacts old entries, improving long-context perplexity and retrieval accuracy versus StreamingLLM at fixed cache sizes.
-
AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Models
AhaKV presents a KV cache eviction strategy that combines recent-window attention accumulation, an entropy-derived softmax temperature, and value-norm priors to reduce positional bias and improve long-context inferenc...
-
Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression
ScaleKV cuts KV cache memory for Visual Autoregressive text-to-image generation to 10% by classifying layers as drafters or refiners per scale and pruning low-attention tokens while keeping benchmark scores nearly unchanged.
-
InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU
A modular sparse-attention framework with CPU KV offloading and layer-wise RoPE adjustment runs Llama-class models on 3M-token contexts on a single 48GB GPU without training.
-
Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models
Dynamic chunking plus question-aware chunk selection improves long-context QA, but the headline numbers are partly inflated by choosing hyperparameters on the test benchmarks.
Discussion (0). Continue with ORCID to comment.