Gated Associative Memory: A Parallel O(N) Architecture for Efficient Sequence Modeling

Rishiraj Acharya

REVIEW 3 major objections 6 minor 13 references

Gated Associative Memory: A Parallel O(N) Architecture for Efficient Sequence Modeling

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fully parallel, non-recurrent GAM block with a causal convolution and an associative memory bank can replace self-attention, running in linear time and achieving better validation perplexity than both a Transformer and Mamba on the tested

desk verdict The GAM paper is an honest small-scale study whose central 'global context' claim is not actually implemented; the memory pathway is a per-token lookup with no cross-token mixing. read the letter →

arxiv 2509.00605 v1 pith:OPP6FWF4 submitted 2025-08-30 cs.CL cs.LG

Rishiraj Acharya This is my paper · ORCID

classification cs.CLcs.LG

keywords sequencemodelingefficienttransformersassociativememorycausalconvolutionlinearcomplexitylanguagegatingmechanismstatespacemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that self-attention's quadratic cost is not needed for competitive language modeling. It proposes the Gated Associative Memory (GAM) block, which replaces attention with two parallel, fully parallelizable pathways: a causal convolution for local context and a softmax retrieval from a fixed learned memory bank for global context, fused by a learned gate. At sequence length 256 on WikiText-2, GAM trains about 11% faster than a similarly sized Transformer and 8% faster than Mamba, while reaching a lower validation perplexity (882.6 vs 919.0 and 1017.5); on TinyStories it also beats the Transformer. A single-block scaling benchmark shows GAM's time and memory doubling as sequence length doubles, whereas the Transformer quadruples and runs out of memory beyond 2048 tokens. If these results hold at longer contexts, GAM is a viable O(N) alternative to attention for sequence modeling.

What carries the argument

The load-bearing object is the GAM block: a causal depthwise convolution (local pathway) running in parallel with softmax retrieval from a fixed learned memory bank of 512 slots (global pathway), the two fused by per-token sigmoid gates computed from the same input. This replaces self-attention entirely, keeping the operation count linear in N and keeping every operation a parallelizable matmul or convolution. The memory bank is a matrix of learned prototype patterns; each token retrieves a weighted average of these prototypes, and the gate decides how much of that global context to mix with the local convolutional context.

What would settle it

Train GAM and a Transformer on a synthetic task that requires reading a token 1000 positions earlier (e.g., copy or flip at distance 1000) with sequence length 4096. If GAM's error grows with distance while the Transformer's does not, the fixed memory bank is not supplying true long-range content-based context.

Watch

Extended reading notes

Core claim

The central claim is that the GAM block, built from a causal depthwise convolution and a parallel associative-memory lookup over 512 learned slots, can stand in for multi-head self-attention without a loss in modeling quality. The global pathway computes a softmax over the memory bank for every token simultaneously, then aggregates the bank vectors; the local pathway gathers the previous k-1 tokens. A per-token sigmoid gate fuses the two streams. In the paper's experiments, this combination is faster and reaches lower validation perplexity than the Transformer on WikiText-2 and TinyStories, and lower perplexity than Mamba on WikiText-2. The ablation study shows that removing the gate and sim

Load-bearing premise

The load-bearing premise is that a fixed memory bank, queried independently for each token, supplies enough 'global' context that direct token-to-token interactions are unnecessary at the sequence lengths and tasks tested.

Editorial extensions

If this is right

At sequence lengths beyond a few thousand, GAM's time and memory should continue to grow roughly linearly, while a Transformer of similar size fails; the paper's single-block benchmark shows GAM handling 8192 tokens where the Transformer OOMs at 4096.
Because GAM has no recurrent scan, its training wall-clock advantage over Mamba can grow on hardware where matmul and conv are more optimized than sequential scans; the paper reports 7.8% faster epochs at length 256.
The ablation attributes most of the quality to the global memory pathway; therefore improving the memory bank (more slots, or richer retrieval) is the most direct lever for further perplexity gains.
The gate is essential; summing the two pathways without gates costs about 42 perplexity points on WikiText-2, so dynamic fusion is a core part of the mechanism, not an add-on.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's end-to-end experiments top out at sequence length 256, so the O(N) advantage appears only in a single-block benchmark; I would not treat the full-model long-context speed-up as demonstrated until GAM is trained end-to-end at lengths in the thousands.
The global pathway is a softmax over a fixed codebook; a natural stress test the paper does not run is to vary num_slots and measure perplexity on long-range tasks. My expectation is that slot count, not just sequence length, will become the scaling bottleneck.
The gate reuses the same input vector to produce both local and global weights, so it can trade off but not amplify total context; a gate that computes its own query from a separate projection might allow richer fusion.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know: the architecture is not what the abstract says. The 'global context pathway' (Eqs. 1-3) is applied row-wise to each token independently. It computes softmax(x_n M^T) M for each n, so there is no interaction between tokens. The only sequence mixing is the causal convolution with k=3; over 6 layers the receptive field is at most 13 tokens. So the paper's core claim that GAM models long-range dependencies is not backed by any mechanism.

What the paper does well: it gives a clear description of the block, implements it from scratch, and reports comparable wall-clock times and perplexities against Transformer and Mamba on WikiText-2 and TinyStories. The scaling benchmark on a single block shows the expected O(N) behavior. The ablation is sensible: removing the gate or either pathway degrades PPL, which is evidence the two parts matter. The arithmetic checks out, and the empirical curves are internally consistent.

The soft spots are in proportion to the claim. First, the memory retrieval is exactly single-head cross-attention to a fixed learned key/value matrix, as in Perceiver, Set Transformer, or Hopfield layers. None of these are cited. That does not make the paper worthless, but it is not a new mechanism. Second, 'global' is a misuse: the memory bank is shared across tokens but does not let tokens exchange information; it is a learned per-token nonlinearity. The claim that it captures long-range dependencies is unsupported. Third, the experiments are thin: one seed, 5 epochs, 256-token contexts, no long-range task, no error bars. The speed advantage at N=256 is modest and partly reflects the fact that GAM has fewer parameters than the Transformer baseline. Fourth, no code or data is released, so the results are not independently reproducible.

Who is this for? Someone building a quick baseline for efficient sequence modeling might find the simple design useful, but the paper overstates its contribution. It deserves a serious referee who can push for a corrected description, prior-art citations, long-context evaluation, and error bars. I would not cite it in my own work as a novel architecture, but it is a reasonable data point that a convolution-plus-memory block can match a Transformer at short context.

Referee Report

3 major / 6 minor

Summary. The paper proposes GAM, a non-recurrent sequence model that replaces self-attention with two parallel pathways: a causal depthwise convolution (local context) and an associative memory retrieval from a fixed learned matrix (global context), fused by a per-token gate. The authors derive O(N) complexity, report wall-clock training time and validation perplexity on WikiText-2 (against Transformer and Mamba) and TinyStories (against Transformer), and include a scaling benchmark and ablations. The central claims are: (i) linear complexity and full parallelizability; (ii) global, content-based context modeling; (iii) consistently faster training and superior or competitive perplexity relative to the baselines.

Significance. If the claims held, GAM would be a useful O(N) alternative to attention for sequence modeling. The linear-complexity arithmetic is straightforward and the scaling benchmark in Table 2 is consistent with O(N) for the tested range. However, the paper does not establish the 'global context' claim: the memory pathway has no token-token interaction, and the only cross-token mechanism is a small-kernel convolution. In addition, the experimental evidence is limited to 5 epochs, a 256-token context, and single runs without error bars. The combination of a local convolution with a per-token associative memory is worth exploring, but the headline architectural contribution and the empirical superiority claims are not yet substantiated.

major comments (3)

[§3.3, Eqs. (1)–(3)] The 'global' pathway is not global. For each token n, GlobalContext_n = softmax(x_n M^T) M depends only on x_n and the fixed matrix M; ∂GlobalContext_n/∂x_m = 0 for all m ≠ n. The only cross-token operation in the block is the causal convolution in §3.2 with k=3; after L=6 blocks the receptive field is at most 2L+1 = 13 tokens. Therefore statements in the abstract, §1, and §3.3 that this pathway models 'global, content-based patterns' or 'long-range ... dependencies' are not supported by the architecture. The ablation called 'Global Only' in Table 3 is more accurately described as a per-token learned nonlinearity over a fixed codebook. The authors should either revise the central claim to 'per-token content-based retrieval' and add experiments on tasks that actually require long-range dependencies (e.g., LRA or long-context language modeling), or add a mechanism with genuine cross-token
[§5, Table 1 and §4.3] The empirical superiority claim rests on a single run per model: no seeds, no error bars, and only 5 epochs of training. The reported differences on TinyStories (23.55 vs 23.15 PPL) and WikiText-2 (918.99 vs 882.57) are modest and could easily lie within run-to-run variance. Moreover, all experiments use a 256-token context, and no long-range task is reported, so these numbers cannot speak to the 'global context' claim. The authors should report multiple seeds with standard deviations and evaluate on at least one long-context benchmark before claiming consistent superiority.
[§5.4, Table 3] The ablation table contains internal inconsistencies. First, GAM (Full) is listed with best val PPL 900.84, while Table 1 reports final val PPL 882.57 for the same configuration and dataset; since the best PPL cannot be worse than the final PPL, at least one of the two numbers is erroneous or comes from a different experimental setup. Second, GAM (Sum Fusion) and GAM (Global Only) are both listed with 19.4M parameters even though Sum Fusion includes both the convolution and the memory pathways and should have strictly more parameters than Global Only. These issues undermine the ablation's implied effect of the gating mechanism; the authors should correct the table and clarify the parameter accounting, or remove the parameter column.

minor comments (6)

[Abstract / throughout] There are typographical artifacts such as 'WikiT ext-2' and 'T rim' in §3.2; these should be cleaned up.
[§2] The text refers to 'TCNs (Temporal Convolutional Networks)' but gives no citation; a reference would help place the local-pathway design.
[§3.4, Eq. (4)] The gating linear layer's output dimension is not specified. Since it is split into two halves that multiply d-dimensional context vectors, the output should be 2d; please state this explicitly.
[§5, Table 1] The claim of outperforming 'both baselines' applies only to WikiText-2; Mamba is not evaluated on TinyStories, so the abstract's wording overstates the comparison.
[§5.2, Table 2 and Figure 2] The scaling benchmark is useful, but Figure 2 would benefit from labeled axes and a legend that clearly distinguishes the two models; also, error bars over repeated measurements would strengthen the linear-scaling conclusion.
[§1 / §6] The paper says the implementation is 'complete' and available? No code link is provided. Adding a repository link would make the results reproducible and is standard for empirical papers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the architecture, complexity analysis, and perplexity measurements are self-contained; the 'global context' critique is a correctness concern, not circular reasoning.

full rationale

The paper's derivation chain is self-contained. The GAM block is defined by explicit equations (1)-(6): causal convolution in §3.2 and associative memory retrieval in §3.3 using a fixed learned matrix M, fused by gates. The O(N) complexity claim follows directly from the operation counts (convolution O(N·k·d), memory retrieval O(N·num_slots·d)), and the scaling benchmark in §5.2 independently measures this behavior; it is an empirical measurement, not a fitted artifact. Perplexity results in Tables 1 and 3 are empirical training outcomes: no parameter is fitted to force a pre-specified perplexity, and no 'prediction' is statistically forced from a fitted subset. There are no self-citations or imported uniqueness theorems: all references are to external prior work. The skeptical concern that the 'global' pathway is per-token and lacks cross-token dependence is a substantive architectural/correctness critique, not circularity: it does not make any derived quantity equivalent to an input by construction. The paper's own admission that long-sequence benchmarks are future work (§6) further confirms that the long-range claim is an untested interpretation rather than a circular derivation.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The only hand-chosen architectural constants are the memory slot count and convolution kernel size; both are reasonable standard values. The performance claims rest on domain assumptions about benchmark representativeness and single-seed training, which are standard but untested here. No new entities are postulated.

free parameters (2)

num_slots = 512
Memory bank size chosen equal to d_model; no sensitivity analysis. It controls the capacity of the global retrieval.
kernel_size = 3
Causal convolution window; small standard choice, no ablation across kernel sizes.

assumptions (3)

domain assumption Cross-entropy/perplexity on held-out validation is the right measure of the model's language modeling quality.
Assumed throughout Section 4.4.
domain assumption The 5-epoch, context-length-256, single-seed training setup is sufficient to compare architectures.
Used in Sections 4.3 and 5; no evidence beyond one run.
domain assumption The implementations of the baselines are fair and comparably optimized.
Stated as 'similar scale and hyperparameters' in Section 4.2; not independently verified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Gated Associative Memory: A Parallel O(N) Architecture for Efficient Sequence Modeling." pith.science (2026). https://pith.science/paper/OPP6FWF4

@misc{pith2026250900605,
  author       = {Pith},
  title        = {Pith review of: Gated Associative Memory: A Parallel O(N) Architecture for Efficient Sequence Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OPP6FWF4}},
  note         = {Machine review of arXiv:2509.00605}
}

read the original abstract

The Transformer architecture, underpinned by the self-attention mechanism, has become the de facto standard for sequence modeling tasks. However, its core computational primitive scales quadratically with sequence length (O(N^2)), creating a significant bottleneck for processing long contexts. In this paper, we propose the Gated Associative Memory (GAM) network, a novel, fully parallel architecture for sequence modeling that exhibits linear complexity (O(N)) with respect to sequence length. The GAM block replaces the self-attention layer with two parallel pathways: a causal convolution to efficiently capture local, position-dependent context, and a parallel associative memory retrieval mechanism to model global, content-based patterns. These pathways are dynamically fused using a gating mechanism, allowing the model to flexibly combine local and global information for each token. We implement GAM from scratch and conduct a rigorous comparative analysis against a standard Transformer model and a modern linear-time baseline (Mamba) on the WikiText-2 benchmark, as well as against the Transformer on the TinyStories dataset. Our experiments demonstrate that GAM is consistently faster, outperforming both baselines on training speed, and achieves a superior or competitive final validation perplexity across all datasets, establishing it as a promising and efficient alternative for sequence modeling.

Figures

Figures reproduced from arXiv: 2509.00605 by the authors.

**Figure 1.** The GAM Block. The input x first passes through a Layer Normalization. It then splits into three branches. The first branch is a residual connection. The second branch computes local and global context in parallel. The local context is generated by a Causal 1D Convolution. The global context is generated by querying a learnable Memory Bank. The outputs of these two pathways are combined via a learned gate. The gated… view at source ↗

**Figure 2.** GAM vs. Transformer Scaling Comparison. (Left) Average forward-backward pass time vs. sequence length. (Right) Peak GPU memory usage vs. sequence length. Both axes are on a logarithmic scale. The Transformer’s quadratic growth is evident in the steep upward curve, while GAM exhibits clear linear scaling. The Transformer fails due to out-ofmemory errors beyond a sequence length of 2048 in this setup. At a short sequ… view at source ↗

**Figure 3.** Training dynamics on the WikiText-2 dataset. (a) Validation perplexity, (b) validation loss, and (c) wall-clock time per epoch. The plots show GAM (in blue) achieving a lower final perplexity and consistently faster epoch times compared to both the Transformer (in red) and Mamba (in green) baselines [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Training dynamics on the TinyStories dataset. (a) Validation perplexity, (b) validation loss, and (c) wall-clock time per epoch. GAM (in green) demonstrates a faster learning trajectory and maintains a significant efficiency advantage throughout the 5 epochs compared t…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 11 linked inside Pith

[1]

Peters, and Arman Cohan

Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document trans- former, 2020. URL https://arxiv.org/abs/2004.05150

arXiv 2020
[2]

Rethinking attention with performers, 2022

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers, 2022. URL https://arxiv.org/abs/2009.14794

arXiv 2022
[3]

Tinystories: How small can language models be and still speak coherent english?, 2023

Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023. URL https://arxiv.org/abs/2305.07759

arXiv 2023
[4]

Mamba: Linear-time sequence modeling with selective state spaces,

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces,
[5]

Efficiently modeling long sequences with structured state spaces, 2022

Albert Gu, Karan Goel, and Christopher R´ e. Efficiently modeling long sequences with structured state spaces, 2022. URL https://arxiv.org/abs/2111.00396

arXiv 2022
[6]

Long short-term memory.Neural Computation, 9(8):1735–1780, 1997

Sepp Hochreiter and J¨ urgen Schmidhuber. Long short-term memory.Neural Computation, 9(8):1735–1780, 1997. doi: 10.1162/neco.1997.9.8.1735

doi:10.1162/neco.1997.9.8.1735 1997
[7]

Decoupled weight decay regularization, 2019

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

arXiv 2019
[8]

Pointer sentinel mixture models, 2016

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL https://arxiv.org/abs/1609.07843

arXiv 2016

Show all 13 references

[9]

Long range arena: A benchmark for efficient transformers, 2020

Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers, 2020. URL https://arxiv.org/abs/2011.04006

2020 arXiv
[10]

Efficient transformers: A survey, 2022

Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, 2022. URL https://arxiv.org/abs/2009.06732

2022 arXiv
[11]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https: //arxiv.org/abs/1706.03762

2023 arXiv
[12]

Li, Madian Khabsa, Han Fang, and Hao Ma

Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self- attention with linear complexity, 2020. URL https://arxiv.org/abs/2006.04768. 11

2020 arXiv
[2024]

URL https://arxiv.org/abs/2312.00752

arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Peters, and Arman Cohan

Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document trans- former, 2020. URL https://arxiv.org/abs/2004.05150

arXiv 2020

[2] [2]

Rethinking attention with performers, 2022

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers, 2022. URL https://arxiv.org/abs/2009.14794

arXiv 2022

[3] [3]

Tinystories: How small can language models be and still speak coherent english?, 2023

Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023. URL https://arxiv.org/abs/2305.07759

arXiv 2023

[4] [4]

Mamba: Linear-time sequence modeling with selective state spaces,

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces,

[5] [5]

Efficiently modeling long sequences with structured state spaces, 2022

Albert Gu, Karan Goel, and Christopher R´ e. Efficiently modeling long sequences with structured state spaces, 2022. URL https://arxiv.org/abs/2111.00396

arXiv 2022

[6] [6]

Long short-term memory.Neural Computation, 9(8):1735–1780, 1997

Sepp Hochreiter and J¨ urgen Schmidhuber. Long short-term memory.Neural Computation, 9(8):1735–1780, 1997. doi: 10.1162/neco.1997.9.8.1735

doi:10.1162/neco.1997.9.8.1735 1997

[7] [7]

Decoupled weight decay regularization, 2019

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

arXiv 2019

[8] [8]

Pointer sentinel mixture models, 2016

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL https://arxiv.org/abs/1609.07843

arXiv 2016