REVIEW 17 cited by
Zoology: Measuring and Improving Recall in Efficient Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Attention-free language models that combine gating and convolutions are growing in popularity due to their efficiency and increasingly competitive performance. To better understand these architectures, we pretrain a suite of 17 attention and "gated-convolution" language models, finding that SoTA gated-convolution architectures still underperform attention by up to 2.1 perplexity points on the Pile. In fine-grained analysis, we find 82% of the gap is explained by each model's ability to recall information that is previously mentioned in-context, e.g. "Hakuna Matata means no worries Hakuna Matata it means no" $\rightarrow$ "??". On this task, termed "associative recall", we find that attention outperforms gated-convolutions by a large margin: a 70M parameter attention model outperforms a 1.4 billion parameter gated-convolution model on associative recall. This is surprising because prior work shows gated convolutions can perfectly solve synthetic tests for AR capability. To close the gap between synthetics and real language, we develop a new formalization of the task called multi-query associative recall (MQAR) that better reflects actual language. We perform an empirical and theoretical study of MQAR that elucidates differences in the parameter-efficiency of attention and gated-convolution recall. Informed by our analysis, we evaluate simple convolution-attention hybrids and show that hybrids with input-dependent sparse attention patterns can close 97.4% of the gap to attention, while maintaining sub-quadratic scaling. Our code is accessible at: https://github.com/HazyResearch/zoology.
Forward citations
Cited by 17 Pith papers
-
Learning What Matters: Supervising Global Context Pruning with Causal Evidence Sets
Attention is an unreliable supervision target for context pruning; causal evidence sets recovered by masking produce routers that match or beat attention-distilled routers, especially on multi-hop and conflicting-fact tasks.
-
The Orthogonalized Read Is a Removable Training Scaffold for Recurrent Memory
The orthogonalized read on noisy recall is a removable training scaffold: it multiplies escape hazard ~6.4x during plateau training, but a stock mLSTM achieves full accuracy once the read is annealed away.
-
Decomposing Prediction Mechanisms for In-Context Recall
In a toy in-context recall task, label-based task initiation and observation-based continuation are distinct mechanisms with separate emergence times, and the same first-token versus second-token gap appears in an OLM...
-
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
MesaNet uses conjugate-gradient-optimal test-time regression in a chunkwise-parallelizable recurrent layer, achieving strong language modeling and benchmark performance at up to 1B scale.
-
DSSMs: State Space Models with Explicit Memory via Delay Differential Equations
Delay State Space Models augment diagonal SSMs with explicit delayed feedback, stable discrete parameterization, and FFT training, improving delayed-retrieval tasks and matching or beating S4D on most standard sequenc...
-
The Capability Convergence Hypothesis: Capability from Access Structure, Not Scale
Long-context capability converges to hybrid architectures pairing O(1) state with a scalable index; pure scaling of SSMs or Transformers alone hits information, horizon, or circuit walls.
-
Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention
A DP-means allocate-on-novelty cache matches full-attention associative recall while storing only distinct items, and a minimal novelty gate recovers the rule end-to-end.
-
Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modelling and State Tracking
KLA reparameterises information-form Kalman filtering as an associative Möbius scan, giving a parallelisable nonlinear state-space mixer that tracks A5 permutations with one or two layers.
-
Towards Understanding What State Space Models Learn About Code
SSM code models capture code syntax and semantics better than Transformers before fine-tuning, forget short-range structure when fine-tuned on type inference, and an added high-frequency path or more kernels recovers ...
-
Controllably Efficient Language Models
A single transformer variant can compress past context into chunk summaries and use chunk size as a test-time knob to trade quality against speed and memory, outperforming many efficient baselines on recall benchmarks.
-
Eigenvalues as a Metric for Memory Dynamics in Sequence Models
Eigenvalue spectra of attention and SSM dynamics show consistent signatures of memory retention and selective forgetting that align with task requirements.
-
HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modeling
A compact hybrid GDN+attention model distilled from Nucleotide Transformer v2 outperforms similarly sized models and, on several tasks, its 500x larger teacher.
-
Context by Distinct Information: An Auditable Dirichlet-Process Working Memory for Long, Redundant Context Streams
A Dirichlet-process novelty cache plus recurrence matches full-attention coupling at about half the attended tokens on long redundant streams, with inspectable retained slots.
-
Extrapolation by Association: Length Generalization Transfer in Transformers
Length generalization on a short-trained main task can be inherited from a longer-trained related auxiliary task trained jointly with it.
-
ATLAS: Learning to Optimally Memorize the Context at Test Time
A new recurrent memory module (Atlas) that optimizes its memory over a window of past tokens and uses Muon-style second-order updates, reporting gains over Transformers and existing linear RNNs on long-context tasks.
-
Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory
Decoupling the forget gate from the input gate in a discrete state-space recurrence lets one fixed-size state both preserve old bindings over long horizons and overwrite stale ones.
-
Convolution for Large Language Models
Adding a residual depthwise convolution (kernel 3) to QKV projections before attention raises average downstream accuracy in Qwen3-1.7B/4B by 1.6-3.8 points with negligible parameter cost.
Discussion (0). Continue with ORCID to comment.