Pith. sign in

REVIEW 2 major objections 4 minor 95 references

All LLM memory mechanisms fit on three axes: representation, update dynamics, and persistence.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:34 UTC pith:MPNOQ2DS

load-bearing objection A genuinely useful survey taxonomy, but the MoE-as-explicit-memory classification contradicts the paper's own scope and definition; fix that and it deserves publication. the 2 major comments →

arxiv 2607.25380 v1 pith:MPNOQ2DS submitted 2026-07-28 cs.CL

Memory for Large Language Models

Sining Zhoubian , Dan Zhang , Evgeny Kharlamov , Jie Tang This is my paper
classification cs.CL
keywords large language modelsmemory taxonomyimplicit memoryexplicit memoryhybrid architecturestest-time trainingretrieval-augmented modelsupdate dynamics
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This survey argues that memory in large language models has become a primary architectural design dimension, not a byproduct of scale. It proposes a taxonomy that classifies every model-level memory mechanism along three largely orthogonal axes—representation (implicit vs. explicit), update dynamics (offline vs. online), and persistence (short-term vs. long-term)—and claims that seemingly disparate mechanisms (attention caches, recurrent states, test-time writable parameters, mixture-of-experts routing, and lookup stores) address the same underlying questions: what is stored, when and how it is updated, and how long it lasts. The payoff is a shared vocabulary for comparing scattered research. A sympathetic reader would care because the field is fragmented, and the taxonomy turns isolated results into a map of design choices and trade-offs.

Core claim

The paper's central claim is that memory is shifting from an implicit byproduct of computation to an explicit, controllable architectural dimension, and that this spectrum can be organized by three orthogonal axes: representation, update dynamics, and persistence. Explicit memory is defined by an independent and controllable interface for storage or retrieval, whereas implicit memory is coupled to the forward computation graph and lacks such an interface; the paper also refines update dynamics into fine-grained rules—optimization-based writing, state-transition updates, signal-gated routing, admission/eviction/consolidation, and objective-induced structural updates. It applies this framework

What carries the argument

The organizing machinery is the taxonomy itself (Table I), built on three axes—representation, update dynamics, and persistence—plus the finer-grained view of update rules (Table II). The load-bearing distinction is the implicit/explicit boundary, defined by whether a mechanism exposes an independent, controllable interface for read/write/lookup; persisting across contexts and addressable storage count as explicit, while computation-coupled states such as KV caches and recurrent hidden states count as implicit. The taxonomy's work is to map every mechanism into a common design space so that trade-offs and unexplored combinations become visible.

Load-bearing premise

The taxonomy assumes that whether a mechanism is explicit memory is an intrinsic property—the presence of an independent, controllable interface—and that this line can be drawn sharply; if controllability is a matter of degree, or if a mechanism's memory role changes with training and usage (as the paper's own treatment of MoE and static weights suggests), the binary axes and the orthogonality claim are not secure.

What would settle it

A convincing falsifier would be a demonstration that a mechanism classified as implicit—say, a standard Transformer KV cache—can be steered at inference (e.g., via learned eviction) to achieve the same cross-session persistence and targeted recall as an explicitly designed memory module, without any independent read/write interface; alternatively, the paper's own cited equivalence between test-time training with KV binding and linear attention, if shown to extend to surprise-driven memory modules like Titans, would reduce the explicit/implicit distinction to a reformulation rather than a diffe

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Researchers can compare attention caches, recurrent states, test-time trained parameters, and lookup stores on equal footing, making design trade-offs explicit rather than implicit.
  • Hybrid architectures become describable as compositions of memory axes, clarifying which component supplies high-fidelity recall and which supplies compressed global context.
  • Evaluation can be decomposed into orthogonal dimensions—capacity, fidelity, persistence, efficiency—instead of conflating memory with context length or parameter count.
  • The map exposes under-explored cells in the design space, such as explicit short-term memory or offline long-term implicit memory, guiding future architecture search.
  • The framework provides a shared language for coordinating memory subsystems and for framing stability–plasticity and interference issues as memory-management problems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If controllability is a matter of degree rather than kind, the binary implicit/explicit axis may be better treated as a continuous spectrum; treating it as a gradient would preserve most of the taxonomy while removing its sharpest edge.
  • The survey's classification of mixture-of-experts as memory—despite static weights being excluded elsewhere—suggests conditional computation generally is a memory mechanism; a testable extension would be to check whether expert routing shows memory-like interference and consolidation under sequential task exposure.
  • The update-rule refinement could double as a generative design tool: systematically combining storage substrates (implicit/explicit) with update rules (state-transition, signal-gated, optimization-based) and persistence levels would enumerate the design space and highlight cells no current model occupies.
  • Because the paper defines long-term memory by effective horizon rather than physical size, evaluation metrics could be built around information-retention half-life—how long a stored item remains retrievable under interference—rather than recall at a single context length.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This survey proposes an architecture-centric taxonomy of memory in large language models. Memory is characterized along three purportedly orthogonal axes: representation (implicit vs. explicit), update dynamics (offline vs. online), and persistence (short-term vs. long-term). The paper then reviews implicit mechanisms (attention, sparse/selective attention, recurrent states), explicit mechanisms (parameterized modules, lookup stores, MoE, multi-timescale updates), hybrid architectures, system-level efficiency, and evaluation. The central claim is that this taxonomy provides a unified vocabulary for comparing attention caches, recurrent states, test-time writable parameters, lookup stores, and hybrids.

Significance. The paper covers a broad and relevant corpus, and its comparative tables (Tables I-III) and fine-grained update-rule taxonomy (Table II) provide a useful organizing device for a fragmented literature. Its positioning relative to earlier memory surveys is clear. However, the contribution depends on the soundness of the implicit/explicit distinction and the asserted orthogonality of the axes. As written, the classification of MoE systems in Table I and Section IV-C contradicts the scope defined in Section II-A(c), and no formal criterion is given for 'designed and operated as memory.' These issues affect the central claim and require revision.

major comments (2)
  1. [§II-A(c), §IV-C, Table I] The survey explicitly states that 'we do not discuss static model parameters learned during pretraining' and defines explicit memory in §II-A(b) as a distinct component with operations 'not reducible to standard forward computation or offline backpropagation.' MoE experts are static after pretraining and are selected by a learned router during ordinary forward computation. Yet Table I lists Switch Transformer, GLaM, Mixtral, and DeepSeek-MoE under 'Explicit Memory,' and §IV-C calls each expert 'a persistent parameterized memory block.' No criterion is provided that would admit MoE experts while excluding dense FFN parameters, which are equally persistent parameterized transformations conditioned on the input. This internal inconsistency undermines the central implicit/explicit axis. Please either remove MoE from explicit memory (or from Table I) or supply a formal criterion and revise th
  2. [§II-C and §II-A(b)] The orthogonality of the three axes is asserted ('These three axes are largely orthogonal') rather than demonstrated. In particular, the explicit/implicit axis is defined by 'an independent and controllable interface,' but controllability and independence are not formally specified; without such a criterion the phrase 'designed and operated as memory' is circular. Please provide definitions that make each axis independently variable, with examples showing, e.g., explicit offline vs. explicit online, and implicit short-term vs. implicit long-term mechanisms. This is load-bearing because the abstract and introduction promise three orthogonal axes.
minor comments (4)
  1. [§IV-C] Typo: 'an distinct intermediate position' should be 'a distinct intermediate position'.
  2. [§IV-B(c)] The Engram discussion correctly distinguishes MoE sparsity from memory sparsity, but this distinction is not carried forward into §IV-C; reconciling these two sections would improve coherence.
  3. [§II] A short glossary defining 'interface,' 'controllable,' 'addressable,' and 'effective horizon' would make the taxonomy easier to apply and test.
  4. [Table I] The persistence labels for recurrent models (e.g., Mamba, RWKV) are all 'Long-Term,' but the text defines long-term by 'effective horizon' without giving operational criteria; a brief justification for representative entries would help.

Circularity Check

0 steps flagged

No significant circularity: the survey proposes a stipulated taxonomy and fits no parameters; no derived result reduces to its inputs.

full rationale

This paper is a survey/taxonomy paper, not a derivation or empirical study. Its central output is a classification scheme along three axes (representation, update dynamics, persistence) defined in Section II. The axes are stipulated definitions, not results derived from data or from equations. No parameter is fitted, no quantity is predicted, and no empirical claim is used to justify the taxonomy. The paper explicitly positions itself relative to prior surveys (Section II-D, Table III) and does not rely on a load-bearing self-citation chain: the cited prior memory surveys are external works, and the taxonomy's classifications are not justified by invoking a uniqueness theorem or an author's prior result. The only notable concern is an internal-consistency issue in the treatment of MoE as 'offline explicit memory' (Section IV-C) despite Section II-A(c) excluding static pretrained weights; however, that is a definitional-coherence critique, not circular reasoning. Because there is no derivation chain in which an output is assumed in the input, the circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The survey introduces no free parameters or invented physical entities. Its contributions are conceptual, resting on definitional assumptions about what counts as memory and on the asserted orthogonality of its axes.

axioms (3)
  • domain assumption Model-level memory excludes agent-level or prompt-based memory orchestrated through external systems.
    Section II scope clarification. If agent-level memory were included, the representation axis and persistence definitions would need additional categories.
  • domain assumption Explicit memory is defined by an independent, controllable read/write interface rather than by its implementation form.
    Section II-A(b) uses this definition to classify all systems in Table I, treating controllability as a binary property rather than a graded one.
  • domain assumption The three axes—representation, update dynamics, persistence—are largely orthogonal.
    Section II-C asserts orthogonality but provides no formal argument; the paper concedes interplay between axes, which weakens the claim.

pith-pipeline@v1.3.0-alltime-deepseek · 29120 in / 8753 out tokens · 93750 ms · 2026-08-01T02:34:27.325428+00:00 · methodology

0 comments
read the original abstract

Memory has evolved into a foundational architectural dimension in large language models (LLMs), shifting from an implicit byproduct of computation to a spectrum of explicit, controllable mechanisms. While recent advances introduce diverse strategies---spanning transient attention, recurrent state dynamics, parameter-efficient adaptations, and scalable lookup storage---this rapid evolution has led to a highly fragmented research landscape. In this survey, we present a systematic, architecture-centric taxonomy of memory in LLMs. Our framework characterizes memory along three orthogonal axes: representation (implicit versus explicit), update dynamics (offline versus online), and persistence (short-term versus long-term). We further formalize the granular mechanisms dictating memory writing, routing, state transitions, and consolidation. This unified perspective elucidates the conceptual boundaries between computation-coupled and independently addressable memory, effectively bridging disparate architectural paradigms. Additionally, we critically analyze hybrid memory architectures, system-level efficiency trade-offs, and multi-dimensional evaluation methodologies. By consolidating these scattered advancements into a cohesive framework, this survey charts the trajectory of memory-centric LLM design and provides a principled foundation for future innovations in scalable and adaptive language modeling.

Figures

Figures reproduced from arXiv: 2607.25380 by Dan Zhang, Evgeny Kharlamov, Jie Tang, Sining Zhoubian.

Figure 1
Figure 1. Figure 1: The evolution of memory in LLMs. As memory transitions from an implicit byproduct of computation to an explicit architectural design dimension [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Main content and taxonomy of memory for LLMs. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of implicit memory via computation dynamics. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of explicit memory via addressable and adaptive storage. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

95 extracted references · 51 linked inside Pith

  1. [1]

    Generating long sequences with sparse transformers,

    R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” 2019. [Online]. Available: https://arxiv.org/abs/1904.10509

    Pith/arXiv arXiv 2019

  2. [2]

    Big bird: Transformers for longer sequences,

    M. Zaheer, G. Guruganesh, A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed, “Big bird: Transformers for longer sequences,” 2021. [Online]. Available: https://arxiv.org/abs/2007.14062

    Pith/arXiv arXiv 2021

  3. [3]

    Longformer: The long- document transformer,

    I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long- document transformer,” 2020. [Online]. Available: https://arxiv.org/abs/ 2004.05150

    Pith/arXiv arXiv 2020

  4. [4]

    Mamba: Linear-time sequence modeling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,”arXiv preprint arXiv:2312.00752, 2024. [Online]. Available: https://arxiv.org/abs/2312.00752

    Pith/arXiv arXiv 2024

  5. [5]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality,

    T. Dao and A. Gu, “Transformers are ssms: Generalized models and efficient algorithms through structured state space duality,” in Proceedings of the 41st International Conference on Machine Learning,

  6. [6]

    Jamba: A hybrid transformer-mamba language model,

    O. Lieber, B. Lenz, H. Bata, G. Cohen, J. Osin, I. Dalmedigos, E. Safahi, S. Meirom, Y . Belinkov, S. Shalev-Shwartz, O. Abend, R. Alon, T. Asida, A. Bergman, R. Glozman, M. Gokhman, A. Manevich, N. Ratner, N. Rozen, E. Shwartz, M. Zusman, and Y . Shoham, “Jamba: A hybrid transformer-mamba language model,”

  7. [7]

    Jamba-1.5: Hybrid transformer-mamba models at scale,

    J. Team, B. Lenz, A. Arazi, A. Bergman, A. Manevich, B. Peleg, B. Aviram, C. Almagor, C. Fridman, D. Padnos, D. Gissin, D. Jannai, D. Muhlgay, D. Zimberg, E. M. Gerber, E. Dolev, E. Krakovsky, E. Safahi, E. Schwartz, G. Cohen, G. Shachaf, H. Rozenblum, H. Bata, I. Blass, I. Magar, I. Dalmedigos, J. Osin, J. Fadlon, M. Rozman, M. Danos, M. Gokhman, M. Zusm...

    Pith/arXiv arXiv 2024

  8. [8]

    Available: https://arxiv.org/abs/2403.19887

    [Online]. Available: https://arxiv.org/abs/2403.19887

    Pith/arXiv arXiv

  9. [9]

    End-to-end test-time training for long context,

    A. Tandon, K. Dalal, X. Li, D. Koceja, M. Rød, S. Buchanan, X. Wang, J. Leskovec, S. Koyejo, T. Hashimoto, C. Guestrin, J. McCaleb, Y . Choi, and Y . Sun, “End-to-end test-time training for long context,”

  10. [10]

    Titans: Learning to memorize at test time,

    A. Behrouzet al., “Titans: Learning to memorize at test time,” arXiv preprint arXiv:2501.00663, 2025. [Online]. Available: https: //arxiv.org/abs/2501.00663 PREPRINT 19

    Pith/arXiv arXiv 2025

  11. [11]

    Nested learning: The illusion of deep learning architectures,

    A. Behrouz, M. Razaviyayn, P. Zhong, and V . Mirrokni, “Nested learning: The illusion of deep learning architectures,” 2025. [Online]. Available: https://arxiv.org/abs/2512.24695

    arXiv 2025

  12. [13]

    Conditional memory via scalable lookup: A new axis of sparsity for large language models,

    X. Cheng, W. Zeng, D. Dai, Q. Chen, B. Wang, Z. Xie, K. Huang, X. Yu, Z. Hao, Y . Li, H. Zhang, H. Zhang, D. Zhao, and W. Liang, “Conditional memory via scalable lookup: A new axis of sparsity for large language models,” 2026. [Online]. Available: https://arxiv.org/abs/2601.07372

    Pith/arXiv arXiv 2026

  13. [14]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023. [Online]. Available: https://arxiv.org/abs/1706.03762

    Pith/arXiv arXiv 2023

  14. [16]

    Available: https://arxiv.org/abs/2401.06066

    [Online]. Available: https://arxiv.org/abs/2401.06066

    Pith/arXiv arXiv

  15. [17]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Rouxet al., “Mixtral of experts,”arXiv preprint arXiv:2401.04088, 2024. [Online]. Available: https://arxiv.org/abs/2401.04088

    Pith/arXiv arXiv 2024

  16. [18]

    Samba: Simple hybrid state space models for efficient unlimited context language modeling,

    L. Ren, Y . Liu, Y . Lu, Y . Shenet al., “Samba: Simple hybrid state space models for efficient unlimited context language modeling,” arXiv preprint arXiv:2406.07522, 2024. [Online]. Available: https: //arxiv.org/abs/2406.07522

    Pith/arXiv arXiv 2024

  17. [19]

    Selective attention improves transformer,

    Y . Leviathan, M. Kalman, and Y . Matias, “Selective attention improves transformer,”arXiv preprint arXiv:2410.02703, 2024. [Online]. Available: https://arxiv.org/abs/2410.02703

    Pith/arXiv arXiv 2024

  18. [20]

    Lighttransfer: Your long-context llm is secretly a hybrid model with effortless adaptation,

    X. Zhang, F. Zhang, C. Du, C. Du, T. Panget al., “Lighttransfer: Your long-context llm is secretly a hybrid model with effortless adaptation,”arXiv preprint arXiv:2410.13846, 2024. [Online]. Available: https://arxiv.org/abs/2410.13846

    Pith/arXiv arXiv 2024

  19. [21]

    Retentive network: A successor to transformer for large language models,

    Y . Sun, L. Dong, S. Huang, S. Ma, Y . Xia, J. Xue, J. Wang, and F. Wei, “Retentive network: A successor to transformer for large language models,”arXiv preprint arXiv:2307.08621, 2023. [Online]. Available: https://arxiv.org/abs/2307.08621

    Pith/arXiv arXiv 2023

  20. [22]

    Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence,

    B. Peng, D. Goldstein, Q. Anthony, A. Albalaket al., “Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence,” arXiv preprint arXiv:2404.05892, 2024. [Online]. Available: https: //arxiv.org/abs/2404.05892

    Pith/arXiv arXiv 2024

  21. [23]

    Gated delta networks: Improving mamba2 with delta rule,

    S. Yanget al., “Gated delta networks: Improving mamba2 with delta rule,”arXiv preprint arXiv:2412.06464, 2025. [Online]. Available: https://arxiv.org/abs/2412.06464

    Pith/arXiv arXiv 2025

  22. [24]

    RWKV-7 “goose

    B. Peng, R. Zhang, D. Goldstein, E. Alcaide, X. Du, H. Hou, J. Lin, J. Liu, J. Lu, W. Merrill, G. Song, K. Tan, S. Utpala, N. Wilce, J. S. Wind, T. Wu, D. Wuttke, and C. Zhou-Zheng, “RWKV-7 “goose” with expressive dynamic state evolution,” 2025. [Online]. Available: https://arxiv.org/abs/2503.14456

    Pith/arXiv arXiv 2025

  23. [25]

    Log-linear attention,

    H. Guo, S. Yang, T. C. Goel, E. P. Xing, T. Dao, and Y .- J. Kim, “Log-linear attention,” 2025. [Online]. Available: https: //arxiv.org/abs/2506.04761

    arXiv 2025

  24. [26]

    Expansion span: Combining fading memory and retrieval in hybrid state space models,

    E. Nunez, L. Zancato, B. Bowman, A. Golatkar, and S. Soatto, “Expansion span: Combining fading memory and retrieval in hybrid state space models,” inProceedings of the International Conference on Neuro-symbolic Systems, ser. Proceedings of Machine Learning Research, 2025, pp. 570–596. [Online]. Available: https: //proceedings.mlr.press/v288/nunez25a.html

    2025

  25. [27]

    MoBA: Mixture of block attention for long-context LLMs,

    E. Lu, Z. Jiang, J. Liu, Y . Du, T. Jiang, C. Hong, S. Liu, W. He, E. Yuan, Y . Wang, Z. Huang, H. Yuan, S. Xu, X. Xu, G. Lai, Y . Chen, H. Zheng, J. Yan, J. Su, Y . Wu, N. Y . Zhang, Z. Yang, X. Zhou, M. Zhang, and J. Qiu, “MoBA: Mixture of block attention for long-context LLMs,”

  26. [28]

    Available: https://arxiv.org/abs/2502.13189

    [Online]. Available: https://arxiv.org/abs/2502.13189

    Pith/arXiv arXiv

  27. [29]

    Rattention: Towards the minimal sliding window size in local-global attention models,

    Apple Machine Learning Research, “Rattention: Towards the minimal sliding window size in local-global attention models,”arXiv preprint,

  28. [30]

    Mamba-3: Improved sequence modeling using state space principles,

    A. Lahoti, K. Y . Li, B. Chen, C. Wang, A. Bick, J. Z. Kolter, T. Dao, and A. Gu, “Mamba-3: Improved sequence modeling using state space principles,” 2026. [Online]. Available: https://arxiv.org/abs/2603.15569

    arXiv 2026

  29. [31]

    Gated deltanet-2: Decoupling erase and write in linear attention,

    A. Hatamizadeh, Y . Choi, and J. Kautz, “Gated deltanet-2: Decoupling erase and write in linear attention,” 2026. [Online]. Available: https://arxiv.org/abs/2605.22791

    Pith/arXiv arXiv 2026

  30. [32]

    Kimi linear: An expressive, efficient attention architecture,

    K. Team, Y . Zhang, Z. Lin, X. Yao, J. Hu, F. Meng, C. Liu, X. Men, S. Yang, Z. Li, W. Li, E. Lu, W. Liu, Y . Chen, W. Xu, L. Yu, Y . Wang, Y . Fan, L. Zhong, E. Yuan, D. Zhang, Y . Zhang, T. Y . Liu, H. Wang, S. Fang, W. He, S. Liu, Y . Li, J. Su, J. Qiu, B. Pang, J. Yan, Z. Jiang, W. Huang, B. Yin, J. You, C. Wei, Z. Wang, C. Hong, Y . Chen, G. Chen, Y ...

    Pith/arXiv arXiv 2025

  31. [33]

    Mom: Linear sequence modeling with mixture-of-memories,

    J. Du, W. Sun, D. Lan, J. Hu, and Y . Cheng, “Mom: Linear sequence modeling with mixture-of-memories,” 2025. [Online]. Available: https://arxiv.org/abs/2502.13685

    arXiv 2025

  32. [34]

    Blending complementary memory systems in hybrid quadratic-linear transformers,

    K. Irie, M. Yau, and S. J. Gershman, “Blending complementary memory systems in hybrid quadratic-linear transformers,” 2025. [Online]. Available: https://arxiv.org/abs/2506.00744

    arXiv 2025

  33. [35]

    Next-latent prediction transformers learn compact world models,

    J. Teoh, M. Tomar, K. Ahn, E. S. Hu, T. Pearce, P. Sharma, A. Krishnamurthy, R. Islam, A. Lamb, and J. Langford, “Next-latent prediction transformers learn compact world models,” 2026. [Online]. Available: https://arxiv.org/abs/2511.05963

    Pith/arXiv arXiv 2026

  34. [36]

    Available: https://machinelearning.apple.com/research/ rattention

    [Online]. Available: https://machinelearning.apple.com/research/ rattention

  35. [37]

    Memory caching: Rnns with growing memory,

    A. Behrouz, Z. Li, Y . Deng, P. Zhong, M. Razaviyayn, and V . Mirrokni, “Memory caching: Rnns with growing memory,” 2026. [Online]. Available: https://arxiv.org/abs/2602.24281

    arXiv 2026

  36. [38]

    When to think fast and slow? amor: Adaptive entropy gate for hybrid models,

    H. Zheng and C. Shani, “When to think fast and slow? amor: Adaptive entropy gate for hybrid models,” 2026. [Online]. Available: https://arxiv.org/abs/2602.13215

    Pith/arXiv arXiv 2026

  37. [39]

    Kaczmarz linear attention,

    J. Zou, R. Ren, and Y . Liu, “Kaczmarz linear attention,” 2026. [Online]. Available: https://arxiv.org/abs/2605.08587

    Pith/arXiv arXiv 2026

  38. [40]

    Kalman linear attention: Parallel bayesian filtering for efficient language modelling and state tracking,

    V . Shaj, C. Barker, A. Scannell, A. Szecsenyi, E. J. Crowley, and A. Storkey, “Kalman linear attention: Parallel bayesian filtering for efficient language modelling and state tracking,” 2026. [Online]. Available: https://arxiv.org/abs/2602.10743

    Pith/arXiv arXiv 2026

  39. [41]

    Gated kalmanet: A fading memory layer through test-time ridge regression,

    L. Peng, A. Chattopadhyay, L. Zancato, E. Nunez, W. Xia, and S. Soatto, “Gated kalmanet: A fading memory layer through test-time ridge regression,” 2026. [Online]. Available: https://arxiv.org/abs/2511.21016

    Pith/arXiv arXiv 2026

  40. [42]

    Glam: Efficient scaling of language models with mixture-of-experts,

    N. Du, Y . Huang, A. M. Dai, S. Tong, D. Lepikhin, Y . Xu, M. Krikun, Y . Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. Bosma, Z. Zhou, T. Wang, Y . E. Wang, K. Webster, M. Pellat, K. Robinson, K. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V . Le, Y . Wu, Z. Chen, and C. Cui, “Glam: Efficient scaling of language models with mixture-of-experts,” 20...

    Pith/arXiv arXiv 2022

  41. [43]

    Hypermlp: An integrated perspective for sequence modeling,

    J. Lu and S. Yang, “Hypermlp: An integrated perspective for sequence modeling,” 2026. [Online]. Available: https://arxiv.org/abs/2602.12601

    arXiv 2026

  42. [44]

    Memoryllm: Towards self-updatable large language models,

    Y . Wang, Y . Gao, X. Chen, H. Jiang, S. Li, J. Yang, Q. Yin, Z. Li, X. Li, B. Yin, J. Shang, and J. McAuley, “Memoryllm: Towards self-updatable large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2402.04624

    Pith/arXiv arXiv 2024

  43. [45]

    Lm2: Large memory models,

    J. Kang, W. Wu, F. Christianos, A. J. Chan, F. Greenlee, G. Thomas, M. Purtorab, and A. Toulis, “Lm2: Large memory models,” 2025. [Online]. Available: https://arxiv.org/abs/2502.06049

    Pith/arXiv arXiv 2025

  44. [46]

    Hybrid associative memories,

    L. Lufkin, T. Figliolia, B. Millidge, and K. Krishnamurthy, “Hybrid associative memories,” 2026. [Online]. Available: https://arxiv.org/abs/ 2603.22325

    arXiv 2026

  45. [47]

    Generalization through memorization: Nearest neighbor language models,

    U. Khandelwal, O. Levy, D. Jurafsky, L. Zettlemoyer, and M. Lewis, “Generalization through memorization: Nearest neighbor language models,”ICLR, 2021. [Online]. Available: https://arxiv.org/abs/1911. 00172

    2021

  46. [48]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” 2022. [Online]. Available: https://arxiv.org/abs/2101.03961

    Pith/arXiv arXiv 2022

  47. [49]

    Gated differentiable working memory for long-context language modeling,

    L. Mei, S. Liu, Y . Wang, Y . Ge, B. Bi, J. Yao, J. Wan, Z. Yin, J. Guo, and X. Cheng, “Gated differentiable working memory for long-context language modeling,” 2026. [Online]. Available: https://arxiv.org/abs/2601.12906

    arXiv 2026

  48. [50]

    Decouple knowledge from paramters for plug-and-play language modeling,

    X. Cheng, Y . Lin, X. Chen, D. Zhao, and R. Yan, “Decouple knowledge from paramters for plug-and-play language modeling,” in Findings of the Association for Computational Linguistics: ACL 2023. Toronto, Canada: Association for Computational Linguistics, 2023, pp. 14 288–14 308. [Online]. Available: https://aclanthology.org/2023. findings-acl.901/

    2023

  49. [51]

    Bottlenecked transformers: Periodic kv cache consolidation for generalised reasoning,

    A. Oomerjee, Z. Fountas, H. Bou-Ammar, and J. Wang, “Bottlenecked transformers: Periodic kv cache consolidation for generalised reasoning,”

  50. [52]

    Priming: Hybrid state space models from pre-trained transformers,

    A. Chattopadhyay, E. Nunez, P. Kaul, B. Bowman, E. Becker, L. Zancato, D. Thomas, W. Xia, and S. Soatto, “Priming: Hybrid state space models from pre-trained transformers,” 2026. [Online]. Available: https://arxiv.org/abs/2605.08301

    Pith/arXiv arXiv 2026

  51. [53]

    ExplicitLM: Decoupling knowledge from parameters via explicit memory banks,

    C. Yu, Z. Lu, C. Zheng, C. Wang, Y . Zhang, and Z. Jin, “ExplicitLM: Decoupling knowledge from parameters via explicit memory banks,”

  52. [54]

    Available: https://arxiv.org/abs/2511.01581

    [Online]. Available: https://arxiv.org/abs/2511.01581

    arXiv

  53. [55]

    Hydra: A modular architecture for efficient long-context reasoning,

    S. Chaudhary, D. Patel, M. Chaudhary, and B. Browning, “Hydra: A modular architecture for efficient long-context reasoning,” 2025. [Online]. Available: https://arxiv.org/abs/2508.15099

    arXiv 2025

  54. [56]

    In-place test-time training,

    G. Feng, S. Luo, K. Hua, G. Zhang, D. He, W. Huang, and T. Cai, “In-place test-time training,” 2026. [Online]. Available: https://arxiv.org/abs/2604.06169 PREPRINT 20

    Pith/arXiv arXiv 2026

  55. [57]

    From storage to experience: A survey on the evolution of LLM agent memory mechanisms,

    J. Luo, Y . Tian, C. Cao, Z. Luo, H. Lin, K. Li, C. Kong, R. Yang, and J. Ma, “From storage to experience: A survey on the evolution of LLM agent memory mechanisms,” inFindings of the Association for Computational Linguistics: ACL 2026. San Diego, California, United States: Association for Computational Linguistics, 2026, pp. 41 622–41 652. [Online]. Avai...

    2026

  56. [58]

    Memoryllm: Plug-n-play interpretable feed-forward memory for transformers,

    A. Jaiswal, L. Hannah, H.-B. Kim, D. Hoang, A. Kundu, M. Farajtabar, and M. Cho, “Memoryllm: Plug-n-play interpretable feed-forward memory for transformers,” 2026. [Online]. Available: https://arxiv.org/ abs/2602.00398

    arXiv 2026

  57. [59]

    Ruler: What’s the real context size of your long-context language models?

    C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y . Zhang, and B. Ginsburg, “Ruler: What’s the real context size of your long-context language models?” 2024. [Online]. Available: https://arxiv.org/abs/2404.06654

    Pith/arXiv arXiv 2024

  58. [60]

    ∞bench: Extending long context evaluation beyond 100k tokens,

    X. Zhang, Y . Chen, S. Hu, Z. Xu, J. Chen, M. K. Hao, X. Han, Z. L. Thai, S. Wang, Z. Liu, and M. Sun, “∞bench: Extending long context evaluation beyond 100k tokens,” 2024. [Online]. Available: https://arxiv.org/abs/2402.13718

    Pith/arXiv arXiv 2024

  59. [61]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention,

    J. Yuan, H. Gao, D. Dai, J. Luo, L. Zhao, Z. Zhang, Z. Xie, Y . Wei, L. Wang, Z. Xiao, Y . Wang, C. Ruan, M. Zhang, W. Liang, and W. Zeng, “Native sparse attention: Hardware-aligned and natively trainable sparse attention,” 2025. [Online]. Available: https://arxiv.org/abs/2502.11089

    Pith/arXiv arXiv 2025

  60. [62]

    A survey on the memory mechanism of large language model based agents,

    Z. Zhang, X. Bo, C. Ma, R. Li, X. Chen, Q. Dai, J. Zhu, Z. Dong, and J.-R. Wen, “A survey on the memory mechanism of large language model based agents,” 2024. [Online]. Available: https://arxiv.org/abs/2404.13501

    Pith/arXiv arXiv 2024

  61. [63]

    From human memory to AI memory: A survey on memory mechanisms in the era of LLMs,

    Y . Wu, S. Liang, C. Zhang, Y . Wang, Y . Zhang, H. Guo, R. Tang, and Y . Liu, “From human memory to AI memory: A survey on memory mechanisms in the era of LLMs,” 2025. [Online]. Available: https://arxiv.org/abs/2504.15965

    Pith/arXiv arXiv 2025

  62. [64]

    Rethinking memory in llm based agents: Representations, operations, and emerging topics,

    Y . Du, W. Huang, D. Zheng, Z. Wang, S. Montella, M. Lapata, K.-F. Wong, and J. Z. Pan, “Rethinking memory in llm based agents: Representations, operations, and emerging topics,” 2025. [Online]. Available: https://arxiv.org/abs/2505.00675

    arXiv 2025

  63. [65]

    Memory in large language models: Mechanisms, evaluation and evolution,

    D. Zhang, W. Li, K. Song, J. Lu, G. Li, L. Yang, and S. Li, “Memory in large language models: Mechanisms, evaluation and evolution,”

  64. [66]

    Available: https://arxiv.org/abs/2509.18868

    [Online]. Available: https://arxiv.org/abs/2509.18868

    arXiv

  65. [67]

    Comba: Improving bilinear RNNs with closed-loop control,

    J. Hu, Y . Pan, J. Du, D. Lan, X. Tang, Q. Wen, Y . Liang, and W. Sun, “Comba: Improving bilinear RNNs with closed-loop control,” 2025. [Online]. Available: https://arxiv.org/abs/2506.02475

    arXiv 2025

  66. [68]

    DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model,

    DeepSeek-AI, “DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model,” 2024. [Online]. Available: https: //arxiv.org/abs/2405.04434

    Pith/arXiv arXiv 2024

  67. [69]

    Using fast weights to attend to the recent past,

    J. Ba, G. Hinton, V . Mnih, J. Z. Leibo, and C. Ionescu, “Using fast weights to attend to the recent past,” 2016. [Online]. Available: https://arxiv.org/abs/1610.06258

    Pith/arXiv arXiv 2016

  68. [70]

    Meta-learning with memory-augmented neural networks,

    A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap, “Meta-learning with memory-augmented neural networks,” inProceedings of The 33rd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. F. Balcan and K. Q. Weinberger, Eds., vol. 48. New York, New York, USA: PMLR, 20–22 Jun 2016, pp. 1842–1850. [...

    2016

  69. [71]

    Hybrid computing using a neural network with dynamic external memory,

    A. Graveset al., “Hybrid computing using a neural network with dynamic external memory,”Nature, vol. 538, pp. 471–476, 2016

    2016

  70. [72]

    Finding structure in time,

    J. L. Elman, “Finding structure in time,”Cognitive Science, vol. 14, no. 2, pp. 179–211, 1990. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/036402139090002E

    arXiv 1990

  71. [73]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,”Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997

    1997

  72. [74]

    Transformers are RNNs: Fast autoregressive transformers with linear attention,

    A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, “Transformers are RNNs: Fast autoregressive transformers with linear attention,” inProceedings of the 37th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol. 119. PMLR, 2020, pp. 5156–5165. [Online]. Available: https://proceed...

    2020

  73. [75]

    Gated linear attention transformers with hardware-efficient training,

    S. Yang, B. Wang, Y . Shen, R. Panda, and Y . Kim, “Gated linear attention transformers with hardware-efficient training,” inProceedings of the 41st International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 235. PMLR, 2024, pp. 56 501–56 523. [Online]. Available: https://proceedings.mlr.press/v235/yang24ab.html

    2024

  74. [76]

    DeltaProduct: Improving state-tracking in linear RNNs via householder products,

    J. Siems, T. Carstensen, A. Zela, F. Hutter, M. Pontil, and R. Grazzi, “DeltaProduct: Improving state-tracking in linear RNNs via householder products,” 2025. [Online]. Available: https://arxiv.org/abs/2502.10297

    arXiv 2025

  75. [77]

    Olmo hybrid: From theory to practice and back,

    W. Merrill, Y . Li, T. Romero, A. Svete, C. Costello, P. Dasigi, D. Groeneveld, D. Heineman, B. Kuehl, N. Lambert, C. Li, K. Lo, S. Malik, D. Matusz, B. Minixhofer, J. Morrison, L. Soldaini, F. Timbers, P. Walsh, N. A. Smith, H. Hajishirzi, and A. Sabharwal, “Olmo hybrid: From theory to practice and back,” 2026. [Online]. Available: https://arxiv.org/abs/...

    Pith/arXiv arXiv 2026

  76. [78]

    Learning to control fast-weight memories: An alterna- tive to dynamic recurrent networks,

    J. Schmidhuber, “Learning to control fast-weight memories: An alterna- tive to dynamic recurrent networks,”Neural Computation, vol. 4, no. 1, pp. 131–139, 1992

    1992

  77. [79]

    Falcon-h1: A family of hybrid-head language models redefining efficiency and performance,

    J. Zuo, M. Velikanov, I. Chahed, Y . Belkada, D. E. Rhayem, G. Kunsch, H. Hacid, H. Yous, B. Farhat, I. Khadraoui, M. Farooq, G. Campesan, R. Cojocaru, Y . Djilali, S. Hu, I. Chaabane, P. Khanna, M. E. A. Seddik, N. D. Huynh, P. L. Khac, L. AlQadi, B. Mokeddem, M. Chami, A. Abubaker, M. Lubinets, K. Piskorski, and S. Frikha, “Falcon-h1: A family of hybrid...

    Pith/arXiv arXiv 2025

  78. [80]

    A systematic analysis of hybrid linear attention,

    D. Wang, R.-J. Zhu, S. Abreu, Y . Shan, T. Kergan, Y . Pan, Y . Chou, Z. Li, J. Wu, G. Zhang, W. Huang, and J. Eshraghian, “A systematic analysis of hybrid linear attention,” 2026. [Online]. Available: https://arxiv.org/abs/2507.06457

    Pith/arXiv arXiv 2026

  79. [81]

    Commvq: Commutative vector quantization for kv cache compression,

    J. Li, Y . Zhang, M. Y . Hassanet al., “Commvq: Commutative vector quantization for kv cache compression,” inProceedings of the 42nd International Conference on Machine Learning (ICML),

  80. [82]

    Test-time training with KV binding is secretly linear attention,

    J. Liu, S. Elflein, O. Litany, Z. Gojcic, and R. Li, “Test-time training with KV binding is secretly linear attention,” 2026. [Online]. Available: https://arxiv.org/abs/2602.21204

    Pith/arXiv arXiv 2026

Showing first 80 references.