Pith. sign in

REVIEW 9 major objections 6 minor 7 cited by

ATLAS: Learning to Optimally Memorize the Context at Test Time

T0 review · 9 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ATLAS claims a recurrent memory that memorizes context windows rather than single tokens, and reports an 80 percent score at 10 million tokens of context.

desk verdict Solid, inventive architecture paper with a genuine new idea in the Omega rule and Muon inner-loop, but the headline BABILong result rests on uncontrolled baseline comparisons and the strict-generalization claim is overstated. read the letter →

arxiv 2505.23735 v1 pith:DMW235KF submitted 2025-05-29 cs.CL cs.AI

Ali Behrouz , Zeman Li , Praneeth Kacham , Majid Daliri , Yuan Deng , Peilin Zhong , Meisam Razaviyayn , Vahab Mirrokni This is my paper
classification cs.CLcs.AI MSC 68T0768T50
keywords test-timememorizationassociativememorylong-contextlanguagemodelsslidingwindowupdaterecurrentneuralnetworkspolynomialkernelsMuonoptimizerDeepTransformers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that modern recurrent models fail at long-context understanding because they update memory only from the latest token, have fixed-size memories with limited capacity, and manage those memories with first-order gradient descent. ATLAS addresses all three limitations at once: its $\Omega$ rule updates memory from a sliding window of past tokens, polynomial feature maps lift the keys to raise memory capacity, and the Muon optimizer supplies near-second-order memory updates. The paper reports that ATLAS outperforms Transformers and recent linear RNNs on language modeling, common-sense reasoning, recall, and long-context benchmarks, including over 80 percent accuracy at a 10 million token context on BABILong. If the claims hold, they point to a linear-time architecture that memorizes context rather than tokens and extrapolates far beyond its training context.

What carries the argument

The central object is the $\Omega$ rule, a sliding-window memory update that solves min over the memory M of sum_{i=t-c+1}^{t} $gamma_i^{{(t)}}$ ||M(k_i) - v_i||$_2^{2}$, with c the window length and gamma input-dependent gates that prune or retain past tokens. With c=1 the rule reduces to the classic $\Delta$ rule, and with c equal to the full context it becomes global least-squares memorization, so $\Omega$ spans the spectrum between online and offline memory learning. Its computational engine is a chunked sliding-window masking scheme that parallelizes the gradient computation without materializing all c gradients. Two supporting mechanisms carry the argument: polynomial feature maps phi_p (and the exponential map phi_*) that increase the effective key dimension, and Muon with Newton-Schulz iterations that approximates second-order memory updates while staying parallelizable.

What would settle it

Rerun Titans, Gated DeltaNet, DeltaNet, and Transformer++ under the paper's exact training setup (FineWeb data, the same token budgets and 4K context length) and then fine-tune them on BABILong with the same protocol as ATLAS; if Titans matches or exceeds ATLAS's accuracy at 10 million tokens, or if ATLAS's advantage over the baselines disappears under identical conditions, the central claim fails. A more targeted check: on the BABILong 10M setting, directly compare ATLAS and Titans under identical fine-tuning and measure whether the 80 percent versus drop pattern reproduces.

Watch

Extended reading notes

Core claim

The central claim is that three disjoint design defects cause the long-context gap in modern recurrent models, and that fixing them yields a memory module that learns to memorize context at test time. The first fix is the $\Omega$ rule, which optimizes the memory objective over a window of c past tokens with input-dependent gates, so the module memorizes the local context instead of greedily storing individual tokens. The second is the use of polynomial feature maps on keys and queries, which provably raises the number of linearly independent key-value associations a fixed-size matrix memory can store from O(d_k) to O(d_k^p). The third is replacing gradient descent with the Muon optimizer, which approximates second-order information and, for ATLAS, is paired with a momentum state; the Newton-Schulz iteration count becomes an internal test-time compute knob. Building on the same analysis, the paper defines DeepTransformers, a family that replaces the polynomial map with the infinite-dimensional exponential kernel and thereby strictly generalizes the original softmax Transformer. Experiments across model sizes from 340M to 1.3B parameters show consistent gains over Transformer++, Titans, DeltaNet, and other recurrent baselines, with ATLAS maintaining about 80 percent accuracy at 10 million tokens on BABILong, where Titans degrades.

Load-bearing premise

The empirical comparisons assume that baseline numbers quoted from earlier papers, especially Titans and Gated DeltaNet, are directly comparable to the authors' own runs; if those baselines used different training data, token budgets, or fine-tuning procedures, the reported gains, including the 10 million token BABILong figure, would not be established.

Editorial extensions

If this is right

  • Modern recurrent architectures can be upgraded to context-window memorization without losing parallel training, using the chunked sliding-window masking described in the paper.
  • Fixed-size matrix memories store more associations when keys are lifted by polynomial kernels, since the capacity bound grows from O(d_k) to O(d_k^p).
  • DeepTransformers and the Omega-based Dot variant inherit Transformer expressivity while adding deep memory, yielding consistent gains over Transformer++ at both 760M and 1.3B scales.
  • ATLAS extrapolates to roughly four times its training context on RULER needle tasks and to 10 million tokens on BABILong, where the Titans baseline reportedly drops in accuracy.
  • Hybrid variants that interleave attention (MAG, MAL) and a gated-MLP memory (Atlas++) further improve perplexity and reasoning accuracy over the purely recurrent version.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Omega rule is a parametric counterpart of sliding-window attention, and the input-dependent gates gamma suggest an untested middle ground: learning per-token pruning policies that decide which historical tokens deserve memory updates, which could reduce noise in very long contexts where only a few needles matter.
  • The paper frames ATLAS as test-time memorization, but for BABILong it fine-tunes the model; an untested extension is attaching the ATLAS memory to a frozen pretrained transformer backbone to extend its effective context without any fine-tuning, which would be a stronger demonstration of the 'test-time' claim.
  • The Newton-Schulz iteration count k in Muon is exposed as an internal test-time compute knob; increasing k should improve recall on hard needles if the second-order benefit is real, a cheap experiment that directly tests the paper's memory-management argument.
  • Higher-degree polynomial lifting increases capacity but also raises computational cost; sketching methods for polynomial kernels could reduce that cost while retaining part of the capacity gain, potentially closing the efficiency gap with simpler linear attention.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

9 major / 6 minor

Summary. The paper introduces ATLAS, a recurrent long-term memory module that updates its memory by optimizing an attentional-bias objective over a sliding window of past tokens (the Omega rule), using polynomial feature maps for keys/queries and a Muon-style second-order update for the internal optimizer. The authors also present a family of Transformer-like architectures (DeepTransformers, SWDT, Dot) claimed to strictly generalize softmax attention, and they report experiments on language modeling, common-sense reasoning, recall, RULER, and BABILong, claiming consistent gains over Transformers and modern linear RNNs, including +80% accuracy at 10M context on BABILong. The theoretical sections aim to justify super-linear memory capacity via deep memory and polynomial lifting, and the paper includes parallelization schemes for the proposed recurrent updates.

Significance. If the empirical claims were established under controlled conditions, the paper would make a useful contribution: the Omega rule (context-window optimization of the memory objective) addresses a real limitation of online-update recurrent models, polynomial feature lifting is a concrete way to increase memory capacity, and using a second-order-style inner optimizer is a novel direction for recurrent memory management. The paper also proposes parallel training algorithms that could make such updates practical. However, the current evidence base is weakened by uncontrolled baseline comparisons and by several theory claims that are either definitional or not fully proved. The strongest parts of the manuscript are the architectural ideas and the extensive evaluation breadth; the weakest parts are the strict-generalization claim (which reduces to the softmax kernel identity) and the capacity bounds (which rest on assumptions such as full-rank lifted keys and a single linear region after perturbation). The central empirical headline, especially the BABILong +80% result, is not yet supported by matched comparisons or uncertainty estimates.

major comments (9)
  1. [Section 6 (Setup) and Appendix E] The main empirical claim that ATLAS outperforms Transformers and modern recurrent models is not established by controlled comparisons. As stated in Section 6, baseline results are 'reported by' prior papers rather than rerun in the paper's FineWeb setup, and Appendix E confirms that 760M baselines come from Behrouz et al. (2025)/Behrouz et al. (2024) and 1.3B baselines from Yang et al. (2024). Because token budgets, data mixtures, and hyperparameters may differ (the paper trains 760M models on 30B tokens and 1.3B models on 100B tokens, while some baselines may have used different budgets), the reported margins, e.g., Table 2 1.3B average of 57.62 vs. Titans 56.82, could be within run-to-run noise. The authors should rerun the baselines in their own setup, or at minimum provide per-task variance, matched token budgets, and a clear table of all training hyperparameters for every baseline.
  2. [Section 6.3, Figure 4] The headline BABILong result (+80% accuracy at 10M context) is shown only as a figure with no exact accuracies, no error bars, and no description of the fine-tuning protocol beyond 'we follow the original setup in the benchmark and fine-tune our model.' The text also states that ATLAS uses the MAC architecture 'but without persistent memory tokens,' which may not match the Titans baseline configuration. This is a load-bearing result for the paper's central claim, and without matched fine-tuning, exact numbers, and variance estimates, the +80% claim is not established. The authors should report the underlying numbers, confidence intervals, and the exact baseline configuration.
  3. [Section 4.2, Eq. (26)] The claim that DeepTransformers 'strictly generalize' Transformers is definitional rather than derived. Equation (26) follows immediately from the identity exp(q^T k) = phi*(q)^T phi*(k), which is a standard property of the exponential/Taylor feature map, not a new result. The paper should substantially temper this claim, for example by presenting it as a kernel-view reformulation rather than a 'strict generalization,' and by clarifying that the unnormalized output in Eq. (26) is not the softmax-normalized attention of Eq. (1).
  4. [Theorem 1 (Appendix C)] The proof of Theorem 1 does not justify the stated lower bound O(d_k d_v) and relies on the assumption that all keys fall in a single linear region 'after a small perturbation.' The proof only bounds the rank of a single affine transformation and does not show that the MLP can actually memorize O(d_k d_v) pairs with linearly independent keys; the lower bound is asserted without construction. The upper bound expression is also derived from a rank inequality that does not translate directly to a capacity upper bound for the full piecewise-linear function. The theorem should either be proved rigorously or removed and replaced with a precise statement about the representational rank of deep linear maps.
  5. [Proposition 2 (Section 3.1)] The capacity bound O(d_k^p) is presented as a statement about memory capacity, but it is only an upper bound on the dimension of the polynomial feature space, and it assumes that the lifted keys are linearly independent. In practice, polynomial-lifted keys are generically not linearly independent for arbitrary input points, so the bound does not directly characterize the number of key-value pairs a delta-rule memory can store. Additionally, Proposition 2 gives no lower bound and does not address the effect of gradient descent dynamics on the achievable capacity. The claim that polynomial mappings 'enhance memory capacity' should be qualified accordingly.
  6. [Section 6.1, Table 2] The reported improvements over the strongest baseline are small (e.g., 1.3B average 57.62 vs. Titans 56.82; 760M average 52.77 vs. 51.56), and many per-task differences are within a fraction of a point. Without confidence intervals or multiple seeds, it is difficult to conclude that ATLAS 'surpasses' these baselines on the language modeling and common-sense tasks. The paper should provide standard errors (or multiple runs) for at least the main comparisons, or explicitly state that the improvements are within plausible noise.
  7. [Section 5.1] The parallelization derivation for ATLAS contains an incomplete sentence ('with momentum but without , we have') and the notation in Eqs. (34)-(41) conflates the memory state M_t with the momentum accumulator S_t. As written, it is not clear how the Newton-Schulz operation is applied per token within a chunk when the momentum terms are computed in parallel. The authors should rewrite this section with clear definitions of all recurrent variables and a precise statement of the chunk-wise update.
  8. [Section 3.3, Eq. (16)] The chunk-wise update rule in Eq. (16) uses undefined notation (t', the meaning of the product of alpha terms, and the summation limits) and is not fully derived. Since the parallel training scheme is a claimed contribution, the authors should give a step-by-step derivation or a clearer algorithm block.
  9. [Section 4.1, Eqs. (17)-(18)] The connection between the nonparametric Nadaraya-Watson estimator and the sliding-window attention is written in a way that conflates the per-position query q with the windowed memory M. The notation M* in Eq. (17) is used both as a function to be minimized and as the resulting attention output; the authors should make the distinction explicit.
minor comments (6)
  1. [Table 1] The symbols in Table 1 (especially the * and the 'Without Normalization' footnote) are not defined in the table caption; please add a legend that explains all markers and abbreviations, including 'NS-5' and 'GD w/ M.'.
  2. [Section 6.2, Table 3] Table 3 reports S-NIAH results at 2K/4K/8K/16K, but the 16K column is missing for S-NIAH-W; please either add the result or explain why it is omitted.
  3. [Section 6.3, Figure 4] Figure 4 has no axis labels or legend description in the text; please add them and specify whether the y-axis is accuracy or another metric.
  4. [Appendix E, Table 7] The 'Peak LR' column lists values like 3e-3 and 1.5e-3, which are inconsistent with the main text's statement of LR 4e-4; please clarify which learning rate schedule and peak value were actually used for ATLAS and for the baselines.
  5. [Throughout] Several equations contain typographical issues (e.g., Eq. (16) uses 'alpha_t...alpha_t'' in a nonstandard way, and Eq. (29) has a misplaced product term). Please proofread the formulas for notational consistency.
  6. [Section 2, Definition 1] The definition of associative memory is attributed to Behrouz et al. (2025) but the formal definition is repeated in the paper without a clear statement of what is new in this work; please add a sentence distinguishing the contributions from the prior framework.

Circularity Check

1 steps flagged · score 4.0 of 10

A secondary theoretical claim (DeepTransformers strictly generalize Transformers) reduces to the defining exponential-kernel identity; the main Atlas results rest on experiments, with uncontrolled baseline imports as a correctness caveat.

  1. self definitional [Section 4.2, Equations (22)-(26)]
    "we define: φ∗(x) = [1; x; x^⊗2/√2!; ...], φ_p(x)=x^⊗p, (22), ... and so: exp(q_t^T k_t)=φ∗(q_t)^T φ∗(k_t). (23) ... M_t = M_{t-1}+v_t φ∗(k_t)^T = Σ v_i φ∗(k_i)^T ⇒ y_t = M_t φ∗(q_t) = Σ v_i exp(q_i^T k_i), (26), which matches the output of the unnormalized Transformers. Therefore, DeepTransformers are strict generalizations of Transformers with softmax attention (Vaswani et al. 2017)."

    φ∗ is chosen exactly so that Eq. 23 (Taylor expansion of exp) holds, and DeepTransformers is then obtained by substituting this feature map into the linear-attention recurrence. Eq. 26 restates that substitution, and the 'Therefore' conclusion follows immediately from the definition rather than from an independent argument. The claimed strict generalization is thus equivalent to the identity used to define φ∗; it is a construction/renaming of the unnormalized softmax kernel, and it drops the softmax normalization in Eq. 21. The Atlas empirical claims do not depend on this step.

full rationale

The main ATLAS story is empirical and self-contained: the Omega rule, polynomial feature maps, Muon-based memory updates, and the reported language-modeling, RULER, MAD, and BABILong results are evaluated against baselines rather than derived from the paper's own definitions. I found no fitted-input-called-prediction pattern and no load-bearing uniqueness theorem imported from the authors' prior work. The frequent citations to the authors' own Titans and Miras papers are normal self-citation: they supply baselines, notation, and framing, but the ATLAS improvements are not statistically forced by those citations. The one genuinely definitional step is the 'DeepTransformers strictly generalize Transformers' claim: the family is defined by replacing a polynomial kernel with the infinite-dimensional monomial feature map φ∗, whose defining property exp(q^T k)=φ∗(q)^T φ∗(k) immediately yields the unnormalized attention output in Eq. 26. This makes the generalization result an equivalence-by-construction rather than an independent derivation, and the paper even concedes that the matched output is unnormalized while attributing the result to softmax attention. Because this circular-by-construction step is a secondary theoretical contribution and does not infect the central empirical ATLAS comparisons, the score is moderate (4) rather than 6 or higher. Separately, the baseline numbers are imported from prior papers, including the authors' own Titans work, without matched reruns; that is a serious experimental-comparison limitation, but it is a correctness/control risk, not a circularity of the derivation chain.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

No new physical or conceptual entities are introduced without empirical support; the architectures (Atlas, OmegaNet, DeepTransformers, Dot) are the contributions and are tested on benchmarks. The paper's key free parameters are the polynomial degree, sliding window length, Newton-Schulz steps, and the short convolution size, none of which are fully pinned down for the headline results.

free parameters (4)
  • Polynomial degree p = not stated (ablation mentions 'w/o Polynomial Mapping' but no degree)
    Controls the order of the polynomial feature map, which the paper claims increases memory capacity as O(d_k^p). Chosen by hand and not reported for the main experiments.
  • Sliding window length c = not stated for main experiments; varied in Figure 5
    Local context length in the Omega rule. Determines how many past tokens are used when updating memory, and hence the trade-off between context memorization and computational cost.
  • Newton-Schulz steps k = 5 (Newton-Schulz5)
    Number of Newton-Schulz iterations in the Muon update of Atlas. The paper treats k as an 'internal test-time compute parameter' and states that larger k can improve memorization.
  • Short convolution kernel size = 4
    Architectural hyperparameter used in the backbone for keys, queries, and values (Section 5.1), fixed by hand to 4.
assumptions (6)
  • domain assumption Sequence models are associative memories optimized via an internal 'attentional bias' objective (Definition 1, from Behrouz, Razaviyayn, et al. 2025).
    The paper's entire framing and derivations assume this unification, which is taken from the authors' own prior work.
  • domain assumption Polynomial kernels approximate the softmax kernel via Taylor expansion (Eq. 5) and thus can substitute for exp(QK^T) without normalization.
    Used to justify replacing the exponential kernel with polynomial features; the approximation quality for long contexts is not analyzed.
  • ad hoc to paper In Theorem 1, all keys can be assumed to fall in a single linear region of the ReLU MLP 'after a small perturbation' (Appendix C).
    This is stated without proof and is required for the rank argument that gives the capacity lower bound.
  • domain assumption Muon's Newton-Schulz iteration converges to the nearest semi-orthogonal matrix as k goes to infinity, providing approximate second-order information (Jordan et al., 2024).
    The 'locally optimal memory' claim rests on this external convergence result, with finite k being 5.
  • standard math Attention can be written as the non-parametric solution of an l2 regression with Nadaraya-Watson estimators (Eq. 17-18).
    Standard equivalence, used to link SWA to the Omega rule.
  • standard math exp(q^T k) = phi*(q)^T phi*(k) for the infinite tensor-product feature map (Eq. 23).
    Standard identity underlying DeepTransformers; the step from this to 'strict generalization of softmax attention' is not valid because of normalization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ATLAS: Learning to Optimally Memorize the Context at Test Time." pith.science (2026). https://pith.science/paper/DMW235KF

@misc{pith2026250523735,
  author       = {Pith},
  title        = {Pith review of: ATLAS: Learning to Optimally Memorize the Context at Test Time},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DMW235KF}},
  note         = {Machine review of arXiv:2505.23735}
}
read the original abstract

Transformers have been established as the most popular backbones in sequence modeling, mainly due to their effectiveness in in-context retrieval tasks and the ability to learn at scale. Their quadratic memory and time complexity, however, bound their applicability in longer sequences and so has motivated researchers to explore effective alternative architectures such as modern recurrent neural networks (a.k.a long-term recurrent memory module). Despite their recent success in diverse downstream tasks, they struggle in tasks that requires long context understanding and extrapolation to longer sequences. We observe that these shortcomings come from three disjoint aspects in their design: (1) limited memory capacity that is bounded by the architecture of memory and feature mapping of the input; (2) online nature of update, i.e., optimizing the memory only with respect to the last input; and (3) less expressive management of their fixed-size memory. To enhance all these three aspects, we present ATLAS, a long-term memory module with high capacity that learns to memorize the context by optimizing the memory based on the current and past tokens, overcoming the online nature of long-term memory models. Building on this insight, we present a new family of Transformer-like architectures, called DeepTransformers, that are strict generalizations of the original Transformer architecture. Our experimental results on language modeling, common-sense reasoning, recall-intensive, and long-context understanding tasks show that ATLAS surpasses the performance of Transformers and recent linear recurrent models. ATLAS further improves the long context performance of Titans, achieving +80\% accuracy in 10M context length of BABILong benchmark.

Figures

Figures reproduced from arXiv: 2505.23735 by the authors.

Figure 1
Figure 1. Comparison of learning to memorize (Left) individual tokens, and (Right) the context. 1. Inner Loop: Where parameters of the memory module are optimized (i.e., 𝜽M = {𝑊1,𝑊2, . . . ,𝑊LM,...}). In the inner optimization loop, all other parameters from the model are considered hyperparameters and are fixed and not optimized. 2. Outer Loop: Where all other parameters of the model are optimized, such as linear projections… view at source ↗
Figure 2
Figure 2. The illustration of tokens dependencies in SWA and [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the Atlas’s (and our other variants’) architecture, and its hybrid counterpart with SWA. where 𝐺 is the gradient matrix, 𝐸 and Θ are diagonal matrices with value 𝜃 and 𝜂, and ⊙ is broadcasting. The main advantage of the above formulation (chunk wise recurrence) is that the recurrence of momentum is independent of the state of memory. That is, we can calculate all the momentum terms in the beginning … view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: The effect of local context length (i.e. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Loss curves for different setting with various hyperparameters [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: The results for associative mem￾ory recall [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Test-Time Memory: State-Space Optimal Control for LLM Reasoning

    cs.LG 2026-03 conditional novelty 7.0 of 10

    A symplectic LQR layer inserted as an adapter into pretrained LLMs yields large gains on MATH-500, AMC and AIME by solving a latent optimal-control problem at inference time.

  2. MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

    cs.LG 2025-06 conditional novelty 7.0 of 10

    MesaNet uses conjugate-gradient-optimal test-time regression in a chunkwise-parallelizable recurrent layer, achieving strong language modeling and benchmark performance at up to 1B scale.

  3. Black-Mamba: Biologically-Inspired Leaky Accumulation for Conceptual Knowledge under Distribution Drift

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Gating test-time memory writes on leaky accumulated surprisal preserves most of the adaptation benefit while roughly halving the number of updates.

  4. Cartridges: Lightweight and general-purpose long context representations via self-study

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.

  5. Memoir: Should a Model Write to Its Memory While It Thinks?

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Writing to fast memory during pondering slows associative-recall learning at a fixed budget, but does not reduce final performance once training is long enough.

  6. A Survey on Latent Reasoning

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A survey that organizes latent reasoning methods into vertical recurrence, horizontal recurrence, and infinite-depth diffusion, arguing that silent reasoning can beat explicit chain-of-thought.

  7. Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures

    cs.LG 2025-08 unverdicted novelty 3.0 of 10

    Memory-augmented Transformer research is organized into a three-axis taxonomy bridging neuroscience memory concepts to network designs, but no new result is produced.

Reference graph

Works this paper leans on

98 extracted references · 34 canonical work pages · cited by 7 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. “Gpt-4 technical report”. In:arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers

    Zeyuan Allen-Zhu. “Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers”. In: SSRN Electronic Journal (May 2025). https://ssrn.com/abstract=5240330

    2025

  3. [3]

    Zoology: Measuring and improving recall in efficient language models

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. “Zoology: Measuring and improving recall in efficient language models”. In: arXiv preprint arXiv:2312.04927 (2023)

    arXiv 2023

  4. [4]

    Simple linear attention language models balance the recall-throughput tradeoff

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, and Christopher Re. “Simple linear attention language models balance the recall-throughput tradeoff”. In: Forty-first International Conference on Machine Learning . 2024. url: https://openreview.net/forum?id=e93ffDcpH3

    2024

  5. [5]

    Language models enable simple systems for generating structured views of heterogeneous data lakes

    Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. “Language models enable simple systems for generating structured views of heterogeneous data lakes”. In: arXiv preprint arXiv:2304.09433 (2023)

    arXiv 2023

  6. [6]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. “Neural machine translation by jointly learning to align and translate”. In: arXiv preprint arXiv:1409.0473 (2014)

    arXiv 2014

  7. [7]

    On the capabilities of multilayer perceptrons

    Eric B Baum. “On the capabilities of multilayer perceptrons”. In: Journal of Complexity 4.3 (1988), pp. 193–215. issn: 0885-064X. doi: https://doi.org/10.1016/0885-064X(88)90020-9 . url: https://www.sciencedirect.com/ science/article/pii/0885064X88900209

    work page arXiv 1988

  8. [8]

    It’s All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization

    Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. “It’s All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization”. In: arXiv preprint arXiv:2504.13173 (2025)

    arXiv 2025

Show all 98 references
  1. [9]

    Titans: Learning to memorize at test time

    Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. “Titans: Learning to memorize at test time”. In: arXiv preprint arXiv:2501.00663 (2024)

    2024 arXiv

  2. [10]

    Low-rank bottleneck in multi-head attention models

    Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. “Low-rank bottleneck in multi-head attention models”. In: International conference on machine learning . PMLR. 2020, pp. 864–873

    2020

  3. [11]

    Birth of a transformer: A memory viewpoint

    Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. “Birth of a transformer: A memory viewpoint”. In: Advances in Neural Information Processing Systems 36 (2023), pp. 1560–1588

    2023

  4. [12]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. “Piqa: Reasoning about physical commonsense in natural language”. In: Proceedings of the AAAI conference on artificial intelligence . Vol. 34. 2020, pp. 7432–7439

    2020

  5. [13]

    BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. “BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions”. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Comput...

    2019

  6. [14]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. “Think you have solved question answering? try arc, the ai2 reasoning challenge”. In: arXiv preprint arXiv:1803.05457 (2018)

    2018 arXiv

  7. [15]

    Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition

    Thomas M. Cover. “Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition”. In: IEEE Transactions on Electronic Computers EC-14.3 (1965), pp. 326–334. doi: 10.1109/PGEC. 1965.264137

    1965

  8. [16]

    Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations

    Róbert Csordás, Christopher Potts, Christopher D Manning, and Atticus Geiger. “Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations”. In: Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP...

    2024

  9. [17]

    One-Minute Video Generation with Test-Time Training

    Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao, Youjin Song, Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, et al. “One-Minute Video Generation with Test-Time Training”. In: arXiv preprint arXiv:2504.05298 (2025)

    2025 arXiv

  10. [18]

    DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. “DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs”. In: arXiv preprint arXiv:1903.00161 (2019)

    2019 arXiv

  11. [19]

    Local polynomial modelling and its applications: monographs on statistics and applied probability 66

    Jianqing Fan. Local polynomial modelling and its applications: monographs on statistics and applied probability 66 . Routledge, 2018. 20

    2018

  12. [20]

    Towards scalable and stable paralleliza- tion of nonlinear rnns

    Xavier Gonzalez, Andrew Warrington, Jimmy Smith, and Scott Linderman. “Towards scalable and stable paralleliza- tion of nonlinear rnns”. In: Advances in Neural Information Processing Systems 37 (2024), pp. 5817–5849

    2024

  13. [21]

    Liquid Structural State-Space Models

    Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. “Liquid Structural State-Space Models”. In: The Eleventh International Conference on Learning Representations . 2023. url: https://openreview.net/forum?id=g4OTKRKfS7R

    2023

  14. [22]

    CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory

    Zexue He, Leonid Karlinsky, Donghyun Kim, Julian McAuley, Dmitry Krotov, and Rogerio Feris. “CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory”. In: arXiv preprint arXiv:2402.13449 (2024)

    2024 arXiv

  15. [23]

    The organization of behavior: A neuropsychological theory

    Donald Olding Hebb. The organization of behavior: A neuropsychological theory . Psychology press, 2005

    2005

  16. [24]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. “Gaussian error linear units (gelus)”. In: arXiv preprint arXiv:1606.08415 (2016)

    2016 arXiv

  17. [25]

    Neural networks and physical systems with emergent collective computational abilities

    John J Hopfield. “Neural networks and physical systems with emergent collective computational abilities.” In: Proceedings of the national academy of sciences 79.8 (1982), pp. 2554–2558

    1982

  18. [26]

    RULER: What’s the Real Context Size of Your Long-Context Language Models?

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. “RULER: What’s the Real Context Size of Your Long-Context Language Models?” In: First Conference on Language Modeling. 2024. url: https://openreview.net/forum?id=kIoBbc76Sy

    2024

  19. [27]

    Provably optimal memory capacity for modern hopfield models: Transformer-compatible dense associative memories as spherical codes

    Jerry Yao-Chieh Hu, Dennis Wu, and Han Liu. “Provably optimal memory capacity for modern hopfield models: Transformer-compatible dense associative memories as spherical codes”. In: arXiv preprint arXiv:2410.23126 (2024)

    2024 arXiv

  20. [28]

    Transformer quality in linear time

    Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. “Transformer quality in linear time”. In:International conference on machine learning. PMLR. 2022, pp. 9099–9117

    2022

  21. [29]

    Learning capability and storage capacity of two-hidden-layer feedforward networks

    Guang-Bin Huang. “Learning capability and storage capacity of two-hidden-layer feedforward networks”. In: IEEE Transactions on Neural Networks 14.2 (2003), pp. 274–281. doi: 10.1109/TNN.2003.809401

    2003

  22. [30]

    Going beyond linear transformers with recurrent fast weight programmers

    Kazuki Irie, Imanol Schlag, Robert Csordas, and Jurgen Schmidhuber. “Going beyond linear transformers with recurrent fast weight programmers”. In: Advances in neural information processing systems 34 (2021), pp. 7703–7717

    2021

  23. [31]

    Muon: An optimizer for hidden layers in neural networks

    Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks . 2024. url: https://kellerjordan.github.io/posts/muon/

    2024

  24. [32]

    PolySketchFormer: Fast Transformers via Sketching Polyno- mial Kernels

    Praneeth Kacham, Vahab Mirrokni, and Peilin Zhong. “PolySketchFormer: Fast Transformers via Sketching Polyno- mial Kernels”. In: Forty-first International Conference on Machine Learning . 2024. url: https://openreview.net/ forum?id=ghYrfdJfjK

    2024

  25. [33]

    PolySketchFormer: Fast Transformers via Sketching Polyno- mial Kernels

    Praneeth Kacham, Vahab Mirrokni, and Peilin Zhong. “PolySketchFormer: Fast Transformers via Sketching Polyno- mial Kernels”. In: Proceedings of the 41st International Conference on Machine Learning . Ed. by Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nu...

    2024

  26. [34]

    Gemma 3 technical report

    Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. “Gemma 3 technical report”. In: arXiv preprint arXiv:2503.19786 (2025)

    2025 arXiv

  27. [35]

    Karami and V

    M. Karami and V. Mirrokni. Lattice: Learning to Efficiently Compress the Memory . 2025

    2025

  28. [36]

    Finetuning Pretrained Transformers into RNNs

    Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, and Noah A. Smith. “Finetuning Pretrained Transformers into RNNs”. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Ed. by Mari...

    2021 doi

  29. [37]

    Transformers are rnns: Fast au- toregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. “Transformers are rnns: Fast au- toregressive transformers with linear attention”. In: International conference on machine learning . PMLR. 2020, pp. 5156–5165

    2020

  30. [38]

    Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension

    Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. “Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension”. In: Proceedings of the IEEE Conference on Computer Vision and Pattern ...

    2017

  31. [39]

    Hierarchical associative memory

    Dmitry Krotov. “Hierarchical associative memory”. In: arXiv preprint arXiv:2107.06446 (2021)

    2021 arXiv

  32. [40]

    Dense associative memory for pattern recognition

    Dmitry Krotov and John J Hopfield. “Dense associative memory for pattern recognition”. In: Advances in neural information processing systems 29 (2016)

    2016

  33. [41]

    BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack

    Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Igorevich Sorokin, Artyom Sorokin, and Mikhail Burtsev. “BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack”. In: The Thirty- 21 eight Conference on Neural Information Processing Systems...

    2024

  34. [42]

    Natural questions: a benchmark for question answering research

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. “Natural questions: a benchmark for question answering research”. In: Transactions of the Association for C...

    2019

  35. [43]

    A survey on long video generation: Challenges, methods, and prospects

    Chengxuan Li, Di Huang, Zeyu Lu, Yang Xiao, Qingqi Pei, and Lei Bai. “A survey on long video generation: Challenges, methods, and prospects”. In: arXiv preprint arXiv:2403.16407 (2024)

    2024 arXiv

  36. [44]

    On the expressive power of modern hopfield networks

    Xiaoyu Li, Yuanpeng Li, Yingyu Liang, Zhenmei Shi, and Zhao Song. “On the expressive power of modern hopfield networks”. In: arXiv preprint arXiv:2412.05562 (2024)

    2024

  37. [45]

    Parallelizing non-linear sequential models over the sequence length

    Yi Heng Lim, Qi Zhu, Joshua Selfridge, and Muhammad Firmansyah Kasim. “Parallelizing non-linear sequential models over the sequence length”. In: The Twelfth International Conference on Learning Representations . 2024. url: https://openreview.net/forum?id=E34AlVLN0v

    2024

  38. [46]

    Longhorn: State space models are amortized online learners

    Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, and Qiang Liu. “Longhorn: State space models are amortized online learners”. In: arXiv preprint arXiv:2407.14207 (2024)

    2024 arXiv

  39. [47]

    Lost in the middle: How language models use long contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. “Lost in the middle: How language models use long contexts”. In:Transactions of the Association for Computational Linguistics 12 (2024), pp. 157–173

    2024

  40. [48]

    Openceres: When open information extraction meets the semi-structured web

    Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. “Openceres: When open information extraction meets the semi-structured web”. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies...

    2019

  41. [49]

    Exponential capacity of dense associative memories

    Carlo Lucibello and Marc Mézard. “Exponential capacity of dense associative memories”. In: Physical Review Letters 132.7 (2024), p. 077301

    2024

  42. [50]

    Pointer Sentinel Mixture Models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. “Pointer Sentinel Mixture Models”. In: International Conference on Learning Representations . 2017. url: https://openreview.net/forum?id=Byj72udxe

    2017

  43. [51]

    The Illusion of State in State-Space Models

    William Merrill, Jackson Petty, and Ashish Sabharwal. “The Illusion of State in State-Space Models”. In: Forty-first International Conference on Machine Learning . 2024. url: https://openreview.net/forum?id=QZgo9JZpLq

    2024

  44. [52]

    On the number of linear regions of deep neural networks

    Guido Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. “On the number of linear regions of deep neural networks”. In: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2. NIPS’14. Montreal, Canada: MIT Press, 2014, pp...

    2014

  45. [53]

    Metalearned neural memory

    Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler. “Metalearned neural memory”. In: Advances in Neural Information Processing Systems 32 (2019)

    2019

  46. [54]

    Neural semantic encoders

    Tsendsuren Munkhdalai and Hong Yu. “Neural semantic encoders”. In: Proceedings of the conference. Association for Computational Linguistics. Meeting. Vol. 1. NIH Public Access. 2017, p. 397

    2017

  47. [55]

    The LAMBADA dataset: Word prediction requiring a broad discourse context

    Denis Paperno, German Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. “The LAMBADA dataset: Word prediction requiring a broad discourse context”. In: Proceedings of the 54th Annual Meeting o...

    2016 doi

  48. [56]

    Razvan Pascanu, Guido Montufar, and Yoshua Bengio.On the number of response regions of deep feed forward networks with piece-wise linear activations . 2014. arXiv: 1312.6098 [cs.LG]. url: https://arxiv.org/abs/1312.6098

    2014 arXiv

  49. [57]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydlicek, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. “The fineweb datasets: Decanting the web for the finest text data at scale”. In: Advances in Neural Information Processing Systems 37 (2024), pp. 30811–30849

    2024

  50. [58]

    RWKV: Reinventing RNNs for the Transformer Era

    Bo Peng, Eric Alcaide, Quentin Gregory Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Nguyen Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Ba...

    2023

  51. [59]

    Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence

    Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, et al. “Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence”. In: arXiv preprint arXiv:2404.05892 (2024). 22

    2024 arXiv

  52. [60]

    Rwkv-7" goose

    Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Haowen Hou, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, et al. “Rwkv-7" goose" with expressive dynamic state evolution”. In: arXiv preprint arXiv:2503.14456 (2025)

    2025 arXiv

  53. [61]

    Mechanistic design and scaling of hybrid architectures

    Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Bjorn Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Re, et al. “Mechanistic design and scaling of hybrid architectures”. In: arXiv preprint arXiv:2403.17844 (2024)

    2024 arXiv

  54. [62]

    Neural network capacity using delta rule

    DL Prados and SC Kak. “Neural network capacity using delta rule”. In: Electronics Letters 25.3 (1989), pp. 197–199

    1989

  55. [63]

    Squad: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. “Squad: 100,000+ questions for machine comprehension of text”. In: arXiv preprint arXiv:1606.05250 (2016)

    2016 arXiv

  56. [64]

    Hopfield Networks is All You Need

    Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleit- ner, Thomas Adler, David Kreil, Michael K Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. “Hopfield Networks is All You Need”. In: Internationa...

    2021

  57. [65]

    Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

    Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. “Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling”. In: arXiv preprint arXiv:2406.07522 (2024)

    2024 arXiv

  58. [66]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. “Winogrande: An adversarial winograd schema challenge at scale”. In: Communications of the ACM 64.9 (2021), pp. 99–106

    2021

  59. [67]

    Social IQa: Commonsense Reasoning about Social Interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. “Social IQa: Commonsense Reasoning about Social Interactions”. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natur...

    2019 doi

  60. [68]

    The dynamics of gradient descent for overparametrized neural networks

    Siddhartha Satpathi and Rayadurgam Srikant. “The dynamics of gradient descent for overparametrized neural networks”. In: Learning for Dynamics and Control . PMLR. 2021, pp. 373–384

    2021

  61. [69]

    Linear transformers are secretly fast weight programmers

    Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. “Linear transformers are secretly fast weight programmers”. In: International Conference on Machine Learning . PMLR. 2021, pp. 9355–9366

    2021

  62. [70]

    Learning to control fast-weight memories: An alternative to recurrent nets. Accepted for publication in

    JH Schmidhuber. “Learning to control fast-weight memories: An alternative to recurrent nets. Accepted for publication in”. In: Neural Computation (1992)

    1992

  63. [71]

    Reducing the ratio between learning complexity and number of time varying variables in fully recurrent nets

    Jürgen Schmidhuber. “Reducing the ratio between learning complexity and number of time varying variables in fully recurrent nets”. In: ICANN’93: Proceedings of the International Conference on Artificial Neural Networks Amsterdam, The Netherlands 13–16 September 1993 3 . Spring...

    1993

  64. [72]

    Long Short-term Memory

    Jürgen Schmidhuber and Sepp Hochreiter. “Long Short-term Memory”. In: Neural Computation MIT-Press (1997)

    1997

  65. [73]

    Implicit Language Models are RNNs: Balancing Parallelization and Expressivity

    Mark Schöne, Babak Rahmani, Heiner Kremer, Fabian Falck, Hitesh Ballani, and Jannes Gladrow. “Implicit Language Models are RNNs: Balancing Parallelization and Expressivity”. In: arXiv preprint arXiv:2502.07827 (2025)

    2025 arXiv

  66. [74]

    Adjustment of an inverse matrix corresponding to a change in one element of a given matrix

    Jack Sherman and Winifred J Morrison. “Adjustment of an inverse matrix corresponding to a change in one element of a given matrix”. In: The Annals of Mathematical Statistics 21.1 (1950), pp. 124–127

    1950

  67. [75]

    DeltaProduct: Increasing the Expressivity of DeltaNet Through Products of Householders

    Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riccardo Grazzi. “DeltaProduct: Increasing the Expressivity of DeltaNet Through Products of Householders”. In: arXiv preprint arXiv:2502.10297 (2025)

    2025

  68. [76]

    Simplified State Space Layers for Sequence Modeling

    Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. “Simplified State Space Layers for Sequence Modeling”. In: The Eleventh International Conference on Learning Representations . 2023. url: https://openreview.net/forum? id=Ai8Hw3AXqks

    2023

  69. [77]

    Learning to (learn at test time): Rnns with expressive hidden states

    Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al. “Learning to (learn at test time): Rnns with expressive hidden states”. In: arXiv preprint arXiv:2407.04620 (2024)

    2024 arXiv

  70. [78]

    Retentive network: A successor to transformer for large language models

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. “Retentive network: A successor to transformer for large language models”. In: arXiv preprint arXiv:2307.08621 (2023)

    2023 arXiv

  71. [79]

    Learning and memory: Basic principles, processes, and procedures

    W Scott Terry. Learning and memory: Basic principles, processes, and procedures . Routledge, 2017

    2017

  72. [80]

    On the resurgence of recurrent models for long sequences: Survey and research opportunities in the transformer era

    Matteo Tiezzi, Michele Casoni, Alessandro Betti, Tommaso Guidi, Marco Gori, and Stefano Melacci. “On the resurgence of recurrent models for long sequences: Survey and research opportunities in the transformer era”. In: arXiv preprint arXiv:2402.08132 (2024)

    2024 arXiv

  73. [81]

    Attention is All you Need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. “Attention is All you Need”. In:Advances in Neural Information Processing Systems. Ed. by I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S....

    2017

  74. [82]

    Uncovering mesa-optimization algorithms in transformers

    Johannes Von Oswald, Maximilian Schlegel, Alexander Meulemans, Seijin Kobayashi, Eyvind Niklasson, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Max Vladymyrov, et al. “Uncovering mesa-optimization algorithms in transformers”. In: arXiv preprint arXiv:2309.05858 (2023)

    2023 arXiv

  75. [83]

    Test-time regression: a unifying framework for designing sequence models with associative memory

    Ke Alexander Wang, Jiaxin Shi, and Emily B Fox. “Test-time regression: a unifying framework for designing sequence models with associative memory”. In: arXiv preprint arXiv:2501.12352 (2025)

    2025 arXiv

  76. [84]

    Rnns are not transformers (yet): The key bottleneck on in-context retrieval

    Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu. “Rnns are not transformers (yet): The key bottleneck on in-context retrieval”. In: arXiv preprint arXiv:2402.18510 (2024)

    2024 arXiv

  77. [85]

    Adaptive switching circuits

    Bernard Widrow and Marcian E Hoff. Adaptive switching circuits. 1988

    1988

  78. [86]

    Non-holographic associative memory

    David J Willshaw, O Peter Buneman, and Hugh Christopher Longuet-Higgins. “Non-holographic associative memory”. In: Nature 222.5197 (1969), pp. 960–962

    1969

  79. [87]

    Gated Delta Networks: Improving Mamba2 with Delta Rule

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. “Gated Delta Networks: Improving Mamba2 with Delta Rule”. In: arXiv preprint arXiv:2412.06464 (2024)

    2024 arXiv

  80. [88]

    Gated Linear Attention Transformers with Hardware-Efficient Training

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. “Gated Linear Attention Transformers with Hardware-Efficient Training”. In: Forty-first International Conference on Machine Learning . 2024. url: https: //openreview.net/forum?id=ia5XvxFUJT

    2024

  81. [89]

    Parallelizing linear transformers with the delta rule over sequence length

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. “Parallelizing linear transformers with the delta rule over sequence length”. In: Advances in Neural Information Processing Systems 37 (2024), pp. 115491–115522

    2024

  82. [90]

    HellaSwag: Can a Machine Really Finish Your Sentence?

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. “HellaSwag: Can a Machine Really Finish Your Sentence?” In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Ed. by Anna Korhonen, David Traum, and Lluis Marquez. F...

    2019 doi

  83. [91]

    An analysis of attention via the lens of exchangeability and latent variable models

    Yufeng Zhang, Boyi Liu, Qi Cai, Lingxiao Wang, and Zhaoran Wang. “An analysis of attention via the lens of exchangeability and latent variable models”. In: arXiv preprint arXiv:2212.14852 (2022). 24 A Additional Related Work Modern Linear Recurrent Neural Networks2. Recent res...

    2022 arXiv

  84. [92]

    In the inner optimization loop, all other parameters from the model are considered hyperparameters and are fixed and not optimized

    Inner Loop: Where parameters of the memory module are optimized (i.e., 𝜽M ={𝑊1,𝑊 2,...,𝑊 LM,...}). In the inner optimization loop, all other parameters from the model are considered hyperparameters and are fixed and not optimized

  85. [93]

    B.1 Examples As an example, one can define the linear attention as the optimization of dot-product similarity with gradient descent: i.e., ˜ℓ𝑡 :=⟨M𝑡−1k𝑡, v𝑡⟩

    Outer Loop: Where all other parameters of the model are optimized, such as linear projections, MLP layers, convolutions, etc. B.1 Examples As an example, one can define the linear attention as the optimization of dot-product similarity with gradient descent: i.e., ˜ℓ𝑡 :=⟨M𝑡−1k...

    2021

  86. [94]

    low-rank bottleneck

    Exact memorization requires solving the linear systemM𝐾 =𝑉 . Vectorizing the expression yields the system(𝐾⊤⊗𝐼𝑑𝑣)vec(M) = vec(𝑉), which has𝑚𝑑𝑣 scalar equations in𝑑𝑘𝑑𝑣 unknowns. When the keys are linearly independent, rank(𝐾) =𝑚, and hence the system matrix has full row rank𝑚𝑑𝑣...

    1988

  87. [95]

    D.4 Atlas In the Atlas, we use the same internal objective as OmegaNet but we optimize it using Muon optimizer (Jordan et al

    (56) In the above formulation,𝜙(.) is the polynomial feature mapping function. D.4 Atlas In the Atlas, we use the same internal objective as OmegaNet but we optimize it using Muon optimizer (Jordan et al

  88. [97]

    2025; Behrouz, Zhong, et al

    (58) E Experimental Details In our experimental setup we follow recent studies on linear recurrent models (Behrouz, Razaviyayn, et al. 2025; Behrouz, Zhong, et al. 2024; Yang, Kautz, et al. 2024), we use Wikitext (Merity et al. 2017), LMB (Paperno et al. 2016), PIQA (Bisk et a...

    2025

  89. [98]

    and Gated DeltaNet-H2 (Yang, Kautz, et al. 2024). Using the hybrid variants of MAG and MAL further improve the performance of Atlas, which shows the complementary role of recurrent long-term memory and attention. The Effect of Design. Comparing the performance of Atlas, OmegaN...

    2024

  90. [2024]

    That is, M𝑡 =𝛼𝑡M𝑡−1+ Newton-schulz5(S𝑡) (57) S𝑡 =𝜃𝑡S𝑡−1− 𝑡∑︁ 𝑖=𝑡−𝑐+1 𝜂(𝑡) 𝑖 ∇∥M𝑡−1(𝜙(k𝑖))− v𝑖∥2

    with weight decay. That is, M𝑡 =𝛼𝑡M𝑡−1+ Newton-schulz5(S𝑡) (57) S𝑡 =𝜃𝑡S𝑡−1− 𝑡∑︁ 𝑖=𝑡−𝑐+1 𝜂(𝑡) 𝑖 ∇∥M𝑡−1(𝜙(k𝑖))− v𝑖∥2

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.