Pith. sign in

REVIEW 4 major objections 5 minor 84 references

Foundation models can keep multi-step memory as evolving internal states that store and recall through ordinary forward computation alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 21:54 UTC pith:NXFFK7UH

load-bearing objection Solid early prototype of parametric native memory with real short-horizon gains—and the authors’ own capacity curves already bound how far Def. 1 currently goes. the 4 major comments →

arxiv 2607.26760 v1 pith:NXFFK7UH submitted 2026-07-29 cs.CL cs.LG

Metis: Memory Foundation Model

Zeyu Zhang , Ziliang Guo , Yihang Sun , Xichong Zhang , Xixuan Hao , Zehao Lin , Yang Zhang , Xiaoyan Zhao
show 9 more authors
Tong Shen Bo Tang Zhi-Qin John Xu Junchi Yan Haofen Wang Xu Chen Feiyu Xiong Zhiyu Li Tat-Seng Chua
This is my paper
classification cs.CL cs.LG
keywords memory foundation modelsnative memoryMetismemory attentionparametric memory statemid-trainingagent memoryfast weights
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that agent memory need not live in external stores and retrieval pipelines. Instead, a foundation model can hold a persistent native memory state inside its backbone and run remember, forget, update, and reflect operations as part of the same forward pass that produces the next answer. The authors build Metis, a prototype that compresses past interactions into fixed-size dense parameters, reads them with a dedicated memory-attention path, and updates them with a gradient-free forward write while all trained weights stay frozen. Mid-training on large synthetic memory trajectories teaches those procedures end to end. If the approach holds, multi-turn agents can retain history without replaying long contexts, discrete retrieval stages, or test-time weight updates, with clear gains in coupling, optimizability, and latency.

Core claim

A memory foundation model is an autoregressive model whose parameters at each interaction step already encode prior steps as a native memory state, and whose next state is produced autonomously inside the forward computation from the current input and output. Metis shows this is workable in practice: with frozen backbone weights, fixed-size layerwise memory matrices updated by a forward write and read by memory attention, the model answers memory-operation and memory-based QA questions without any external context, beating other no-context parametric and test-time-adaptation baselines on the reported benchmarks.

What carries the argument

The Metis block: each Transformer layer gains a local dense memory matrix and normalization vector (the native memory state) plus a hyper-memory block that scores, selects, and projects hidden states into a gated/discounted write; utilization is a parallel memory-attention readout mixed with ordinary causal attention.

Load-bearing premise

A fixed-size dense matrix updated by compressed forward writes can keep enough of online history, and later read the right pieces, for general multi-step use.

What would settle it

On long trajectories with growing step count or tokens per update, no-context Metis accuracy on held-out remember/update/forget/reflect and multi-hop QA should stay near full-context performance and above strong no-context baselines; a collapse toward empty-memory or baseline levels, or severe bleed of stored noise into unrelated general tasks, would falsify the claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agent memory can be trained end-to-end like other native skills, without discrete retrieval or blocked gradients through external stores.
  • Online memory maintenance needs only a forward pass with frozen weights, avoiding test-time optimizers and growing textual contexts.
  • History cost can stay roughly constant in state size rather than linear in interaction length, improving tail latency and per-session storage.
  • Scaling the backbone and memory-specific mid-training data can raise native memory skill, though fixed-size compression still limits very long horizons.
  • Hybrid designs that combine native states with external memory remain a natural next path when pure compression is insufficient.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Clean gating of when memory should stay silent may matter as much as write fidelity, given how loaded memory can hurt strict instruction-following.
  • Low-rank structure in the memory matrices suggests storage can shrink further without redesigning the architecture.
  • The same mid-training recipe could be tried on other decoder-only families to test whether native memory is backbone-agnostic.
  • Long-horizon agents may need hierarchical or multi-timescale native states before external memory can be retired rather than only complemented.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper defines memory foundation models as autoregressive models whose parameters θ_t carry a persistent native memory state across multi-step interactions, with storage and utilization executed inside forward computation (Definition 1). It proposes Metis: Metis blocks with local dense memory M^(l)_t, S^(l)_t and hyper-memory projections, memory attention fused with causal attention (Eqs. 2–5), and mid-training via reconstruction, operation, and regularization objectives on large synthetic multi-turn corpora. Online memory updates are gradient-free (forward-only); learned weights stay frozen at inference. Experiments on MemOps, LoCoMo (Gold), NextMem, OOD ATM-Bench/MemDaily, ablations, capacity, general-capability, low-rank storage, and efficiency appendices show consistent no-context gains over Qwen, Temp-LoRA, and δ-Mem, with frank reporting of long-horizon collapse and active-stage interference. Code and checkpoints are released.

Significance. If the prototype holds under the paper’s own caveats, this is a substantive first architectural and training recipe for parametric, instruction-driven memory inside frozen foundation-model backbones—distinct from external RAG, MANNs-as-separate-controllers, and pure TTT. Strengths include a clear formalization (state vs procedure), end-to-end mid-training without online gradients, parallelizable storage/utilization analysis, extensive ablations (adaptive aggregation, QKN, OQ, GDU), OOD transfer on ATM-Bench, storage/latency appendices, and public artifacts. The work is valuable as an early systems/ML contribution even if fixed-size associative memory cannot yet replace external memory for long-horizon agents.

major comments (4)
  1. [Definition 1; Eqs. 2–5; §3.5; Fig. 3; Tables 5–6, 11] Definition 1 and the abstract/intro frame native memory as a path toward eliminating external context memory, but the load-bearing premise—that fixed-size M^(l)_t/S^(l)_t updated by discounted or GDU outer-product writes (Eqs. 2–3 and GDU variant) plus synthetic mid-training support selective multi-step readout—is only validated in short-horizon regimes. Figure 3 shows rapid step- and trajectory-level collapse; Tables 5–6 and 11 leave large gaps to full context; §3.5 error terms ε1–ε3 grow when non-target steps leak. The manuscript already notes limitations, but the central claim should be scoped explicitly to short-horizon residual parametric memory (or state conditions under which Def. 1 is claimed), not left as a general agent-memory substitute.
  2. [§4.1; Tables 1–2, 5–6] Primary data are synthesized from 27 benchmarks that overlap the evaluation families (LoCoMo, LongMemEval, etc.; Tables 1–2). The paper states exact QA behaviors are not leaked, yet seed facts and operation skeletons come from the same sources. Without a clear held-out-source split or a contamination audit (e.g., entity/fact overlap statistics between train seeds and LoCoMo/MemOps test items), the no-context gains on those suites are harder to interpret as fully general native procedures. ATM-Bench/MemDaily help, but the main tables need either stricter decontamination reporting or primary metrics restricted to fully disjoint sources.
  3. [§6.6; Table 9; Eq. 5] Table 9 (Active Stage) shows large degradation once irrelevant memory is loaded (IFEval −22.18; smaller drops on MMLU-Pro/GSM8K/MMMLU). For a native memory procedure that is always in the forward path (Eq. 5 mix with γ), this indicates weak gating between memory and task computation. This is load-bearing for the claim that memory procedures are autonomously and appropriately applied: either strengthen the regularization/gating story with quantitative memory-off metrics, or treat interference as a first-class limitation of the current architecture rather than a secondary general-capability note.
  4. [§2.3; §5.3; Table 5] Forgetting remains weak on external MemOps (Gold) even for Metis-27B (Table 5: Forget 10.91 vs much higher remember/update/reflect on the internal test set). Given that forget/update are core native procedures in the formalization (§2.3) and training objectives (§5.3), the discrepancy between synthetic test forget scores and MemOps forget scores needs analysis (distribution shift, operation realization in latent space, or evaluation sensitivity). Without it, the claim of acquired native forget/update procedures is only partially supported outside the training distribution.
minor comments (5)
  1. [§3.3; §5.1; Eq. 13] γ, λ, ρ, τ, K_min and the task-weight annealing schedule (Eq. 13) are free parameters; a compact hyperparameter table in the main text or appendix would aid reproducibility beyond the training config paragraph.
  2. [Figure 1] Figure 1 is conceptually useful but dense; labeling which row corresponds to architecture vs optimization vs efficiency more explicitly would help non-specialist readers.
  3. [§3.3; Appendix C] GDU is adopted over linear update with the full rule deferred to Appendix C; a one-line formal statement of the GDU update next to Eq. 2 would make the main architecture self-contained.
  4. [Front matter; §7] Date line “July 30, 2026” and arXiv stamp are fine for a preprint, but journal submission should normalize chronology and ensure baseline citations (Temp-LoRA, δ-Mem, MemGen) are version-stable.
  5. [§6.1; Appendix F.3] LLM-as-a-judge (gpt-4.1-mini, median of three) is reasonable; reporting inter-judge agreement or a small human spot-check on MemOps/LoCoMo would strengthen metric trust.

Circularity Check

0 steps flagged

No load-bearing circular derivation; Metis is an architecture-plus-mid-training prototype with empirical claims, not a closed-form prediction forced by its inputs.

full rationale

The paper’s chain is definitional framing → architecture (Metis blocks, memory attention, GDU/linear updates) → synthetic multi-step supervision → NLL mid-training → no-context empirical comparison. Definition 1 names the target capability (persistent parametric state + autonomous store/utilize via forward pass); Sections 3–5 then implement and train that design. That is normal systems work, not self-definitional circularity: the non-vacuous claims are empirical (Tables 5–6, 8; capacity Fig. 3; general-capability Table 9) and are tested against external no-context controls, full/partial context upper bounds, Temp-LoRA, and δ-Mem. The theoretical development (§3.4–3.5) approximates virtual-prefix attention by a fixed-size outer-product memory plus error terms ε1–ε3; it does not fit a constant and relabel it a prediction. Related-work citations (FWP, MANN, TTT, authors’ MemOps/NextMem) situate the prototype; none is invoked as a uniqueness theorem that forces the architecture or forbids alternatives. Mild contamination risk exists because primary data is synthesized from benchmark families that later appear in evaluation (e.g. LoCoMo, LongMemEval in Table 1), but the paper states it does not leak exact QA pairs, and that issue is data hygiene—not a derivation that equals its inputs by construction. Score 1 reflects only that minor proximity of train sources to eval families, not a circular proof.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

Load-bearing content is architectural and empirical, not a short axiom list. Claims rest on Transformer causal LMs, FWP-style associative memory writes, synthetic supervision as a proxy for online memory procedures, and fixed-rank latent states as adequate stores. Free knobs (λ, γ, ρ, ranks, sampling weights, GDU choice) shape capacity and behavior; invented entities are the native memory state/procedures and Metis block factorization.

free parameters (5)
  • Memory mix γ and discount λ = γ∈[0,1]; λ discount in Eqs. 2–3 (exact defaults not uniquely fixed in main text)
    Balance original vs memory attention and exponential forgetting in state updates; chosen design hyperparameters that directly control retention/interference.
  • Top-ρ aggregation threshold / K_min / temperature τ = ρ-quantile selection with K_min clip (Sec. 3.3)
    Controls which tokens enter the memory write; ablation w/o SA collapses performance, so these selection knobs are load-bearing.
  • Task-weighted sampler schedule w^s_τ, w^e_τ = Linear anneal over epochs (Sec. 5.1)
    Curriculum over reconstruction/operation/regularization subsets (Eq. 13) determines what procedures are learned; annealed by hand.
  • Low-rank retention k for storage = k=64 recovers ~99.9% overall in Table 10
    Optional SVD rank for compressing M; performance plateaus near k=64 on reported suites.
  • Mid-training steps / early stopping = 14k / 8k / 14k steps by model size
    4B 14k steps; 9B early-stopped 8k; 27B 14k (~0.4 epoch)—selection affects reported peaks.
axioms (5)
  • domain assumption Causal Transformer forward computation plus residual attention is an adequate substrate for both language modeling and memory read/write.
    Sec. 3 builds Metis blocks into standard pre-norm Transformer layers without replacing full attention within the step.
  • domain assumption Online interaction information (not offline pretraining knowledge) is the proper scope of 'memory,' representable as dynamic parameters aligned in mid-training.
    Sec. 2.2 strict definition excluding offline knowledge from memory.
  • ad hoc to paper Synthetic multi-turn trajectories with explicit operation labels teach generalizable native remember/forget/update/reflect procedures.
    Sec. 4–5; central training assumption; OOD mixed and capacity results test but do not fully validate generality.
  • ad hoc to paper Linear/GDN associative updates of fixed d_k×d_v matrices approximate useful cross-step memory attention (error terms ε1–ε3 small when cross-similarities are low).
    Sec. 3.4–3.5 theoretical insight; relies on query decoupling and sparsity of relevance.
  • standard math Standard softmax attention, next-token NLL, and AdamW mid-training with frozen backbone are valid optimization background.
    Sec. 3.1 and 5; conventional LM training facts.
invented entities (3)
  • Memory foundation model / native memory state & native memory procedure no independent evidence
    purpose: Define the target class: persistent parametric state θ_t/M_t plus autonomous store/use inside forward passes (Def. 1).
    Framing entities organizing the contribution; operationalized only through Metis.
  • Metis block (local memory M,S + hyper memory projections + memory attention) no independent evidence
    purpose: Concrete mechanism to write compressed history and residual-read it beside causal attention.
    Core invented module; supported by ablations and benchmarks inside the paper, not external prior measurement of this exact block.
  • Primary/auxiliary memory-operation synthetic corpora and reconstruction/operation/regularization objectives no independent evidence
    purpose: Induce semantic memory ops and robustness without external controllers.
    Training apparatus constructed for this work from public seeds.

pith-pipeline@v1.2.0-daily-grok45 · 48467 in / 4077 out tokens · 83641 ms · 2026-07-30T21:54:18.306205+00:00 · methodology

0 comments
read the original abstract

Recent advances in AI agents have increasingly internalized native capabilities into their underlying foundation models, giving rise to multimodal foundation models and large reasoning models. However, agent memory is still primarily implemented through external modules, leaving the native memory capability largely unexplored. In this paper, we take a first step toward this direction by introducing memory foundation models, which empower foundation models with native memory capabilities. We formalize native memory from two perspectives: a persistent and dynamically evolving memory state within the backbone, and native memory procedures that autonomously store and utilize information through model computation. We show that native memory offers advantages in architecture, end-to-end optimization, and efficiency. Based on this formulation, we propose Metis, the first prototype of memory foundation models. Metis introduces a new architecture that equips a foundation model with a native memory state, allowing historical information to be compressed into the model and accessed through memory attention. We construct large-scale memory-specific training data and introduce multiple optimization objectives to acquire these native memory procedures through mid-training. The online memory maintenance of Metis is gradient-free, and the memory update requires only a forward pass. At inference time, all learned model weights remain frozen, while the native memory states are autonomously transformed through standard forward computation. Through extensive experiments, we show that Metis exhibits native memory capabilities and further provide a detailed analysis of its strengths, limitations, and behaviors. To facilitate future research on memory foundation models, we release our project and model checkpoints.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

84 extracted references · 35 linked inside Pith

  1. [1]

    A survey of large language models.arXivpreprint arXiv:2303.18223, 1(2):1–124, 2023

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models.arXivpreprint arXiv:2303.18223, 1(2):1–124, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Large language models: A survey.arXivpreprint arXiv:2402.06196, 2024

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey.arXivpreprint arXiv:2402.06196, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    A survey on large language models for code generation

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. ACMTransactionsonSoftwareEngineering andMethodology, 35(2):1–72, 2026

    2026

  4. [4]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  5. [5]

    Code llama: Open foundation models for code.arXivpreprint arXiv:2308.12950, 2023

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code.arXivpreprint arXiv:2308.12950, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Toward large reasoning models: A survey of reinforced reasoning with large language models.Patterns, 6(10), 2025

    Fengli Xu, Qianyue Hao, Chenyang Shao, Zefang Zong, Yu Li, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, et al. Toward large reasoning models: A survey of reinforced reasoning with large language models.Patterns, 6(10), 2025

    2025

  7. [7]

    Chain-of-thought prompting elicits reasoning in large language models.Advancesinneuralinformationprocessingsystems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advancesinneuralinformationprocessingsystems, 35:24824–24837, 2022

    2022

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    A survey on the memory mechanism of large language model-based agents.ACMTransactionsonInformationSystems, 43(6):1–47, 2025

    Zeyu Zhang, Quanyu Dai, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model-based agents.ACMTransactionsonInformationSystems, 43(6):1–47, 2025

    2025

  10. [10]

    Memory in the age of ai agents.arXivpreprint arXiv:2512.13564, 2025

    YuyangHu,ShichunLiu,YanweiYue,GuibinZhang,BoyangLiu,FangyiZhu,JiahangLin,HonglinGuo,ShihanDou,Zhiheng Xi, et al. Memory in the age of ai agents.arXivpreprint arXiv:2512.13564, 2025. 27

    Pith/arXiv arXiv 2025

  11. [11]

    Memorybank: Enhancing large language models with long-term memory

    Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. InProceedingsofthe AAAIconferenceonartificial intelligence, volume 38, pages 19724–19731, 2024

    2024

  12. [12]

    A-mem: Agentic memory for llm agents

    Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for llm agents. Advancesin NeuralInformationProcessingSystems, 38:17577–17604, 2026

    2026

  13. [13]

    Memgpt: Towards llms as operating systems, 2024

    CharlesPacker,SarahWooders,KevinLin,VivianFang,ShishirG.Patil,IonStoica,andJosephE.Gonzalez. Memgpt: Towards llms as operating systems, 2024

    2024

  14. [14]

    Memos: A memory os for ai system.arXivpreprint arXiv:2507.03724, 2025

    Zhiyu Li, Chenyang Xi, Chunyu Li, Ding Chen, Boyu Chen, Shichao Song, Simin Niu, Hanyu Wang, Jiawei Yang, Chen Tang, et al. Memos: A memory os for ai system.arXivpreprint arXiv:2507.03724, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Learn to memorize: Optimizing llm-based agents with adaptive memory framework.arXivpreprint arXiv:2508.16629, 2025

    Zeyu Zhang, Quanyu Dai, Rui Li, Xiaohe Bo, Xu Chen, and Zhenhua Dong. Learn to memorize: Optimizing llm-based agents with adaptive memory framework.arXivpreprint arXiv:2508.16629, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Memrl: Self-evolving agents via runtime reinforcement learning on episodic memory.arXiv preprint arXiv:2601.03192, 2026

    Shengtao Zhang, Jiaqian Wang, Ruiwen Zhou, Junwei Liao, Yuchen Feng, Zhuo Li, Yujie Zheng, Weinan Zhang, Ying Wen, Zhiyu Li, et al. Memrl: Self-evolving agents via runtime reinforcement learning on episodic memory.arXiv preprint arXiv:2601.03192, 2026

    Pith/arXiv arXiv 2026

  17. [17]

    Memfactory: Unified inference & training framework for agent memory

    Ziliang Guo, Ziheng Li, Bo Tang, Feiyu Xiong, and Zhiyu Li. Memfactory: Unified inference & training framework for agent memory. arXivpreprint arXiv:2603.29493, 2026

    Pith/arXiv arXiv 2026

  18. [18]

    Using fast weights to attend to the recent past

    Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. Advancesin neuralinformationprocessingsystems, 29, 2016

    2016

  19. [19]

    Neural turing machines.arXivpreprint arXiv:1410.5401, 2014

    Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines.arXivpreprint arXiv:1410.5401, 2014

    Pith/arXiv arXiv 2014

  20. [20]

    In-place test-time training.arXiv preprint arXiv:2604.06169, 2026

    Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang, Di He, Wenhao Huang, and Tianle Cai. In-place test-time training.arXiv preprint arXiv:2604.06169, 2026

    Pith/arXiv arXiv 2026

  21. [21]

    Memgen: Weaving generative latent memory for self-evolving agents.arXiv preprint arXiv:2509.24704, 2025

    Guibin Zhang, Muxin Fu, and Shuicheng Yan. Memgen: Weaving generative latent memory for self-evolving agents.arXiv preprint arXiv:2509.24704, 2025

    arXiv 2025

  22. [22]

    Jingdi Lei, Di Zhang, Junxian Li, Weida Wang, Kaixuan Fan, Xiang Liu, Qihan Liu, Xiaoteng Ma, Baian Chen, and Soujanya Poria.𝛿-mem: Efficient online memory for large language models.arXivpreprint arXiv:2605.12357, 2026

    Pith/arXiv arXiv 2026

  23. [23]

    Memo: Memory as a model.arXivpreprint arXiv:2605.15156, 2026

    Ryan Wei Heng Quek, Sanghyuk Lee, Alfred Wei Lun Leong, Arun Verma, Alok Prakash, Nancy F Chen, Bryan Kian Hsiang Low, Daniela Rus, and Armando Solar-Lezama. Memo: Memory as a model.arXivpreprint arXiv:2605.15156, 2026

    Pith/arXiv arXiv 2026

  24. [24]

    How lora remembers? a parametric memory law for llm finetuning.arXivpreprint arXiv:2605.30260, 2026

    Ziwen Xu, Haiwen Hong, Linsong Yu, Benglei Cui, Longtao Huang, Hui Xue, and Ningyu Zhang. How lora remembers? a parametric memory law for llm finetuning.arXivpreprint arXiv:2605.30260, 2026

    Pith/arXiv arXiv 2026

  25. [25]

    Memory3: Language modeling with explicit memory

    Hongkang Yang, Zehao Lin, Wenjin Wang, Hao Wu, Zhiyu Li, Bo Tang, Wenqiang Wei, Jinbo Wang, Zeyun Tang, Shichao Song, Chenyang Xi, Yu Yu, Kai Chen, Feiyu Xiong, Linpeng Tang, and Weinan E. Memory3: Language modeling with explicit memory. Journal ofMachineLearning, 3(3):300–346, 2024

    2024

  26. [26]

    Gated delta networks: Improving mamba2 with delta rule

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. InInternational Conferenceon Learning Representations, volume 2025, pages 29687–29707, 2025

    2025

  27. [27]

    Evaluating very long- term conversational memory of llm agents

    Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long- term conversational memory of llm agents. InProceedingsofthe62ndAnnualMeetingoftheAssociationforComputational Linguistics(Volume1: LongPapers), pages 13851–13870, 2024

    2024

  28. [28]

    Longmemeval: Benchmarkingchatassistants on long-term interactive memory.arXivpreprint arXiv:2410.10813, 2024

    DiWu,HongweiWang,WenhaoYu,YuweiZhang,Kai-WeiChang,andDongYu. Longmemeval: Benchmarkingchatassistants on long-term interactive memory.arXivpreprint arXiv:2410.10813, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Needle in a haystack - pressure testing llms

    Gregory Kamradt. Needle in a haystack - pressure testing llms. GitHub repository, 2023

    2023

  30. [30]

    Ruler: What’s the real context size of your long-context language models?arXivpreprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models?arXivpreprint arXiv:2404.06654, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. InProceedingsofthe62ndannualmeeting of the associationforcomputational linguistics(volume1: Longpapers), pages 3119–3137, 2024

    2024

  32. [32]

    InProceedings of the 62nd Annual Meeting of the Association forComputational Linguistics(Volume1: Long Papers), pages 15262–15277, 2024

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al.∞bench: Extending long context evaluation beyond 100k tokens. InProceedings of the 62nd Annual Meeting of the Association forComputational Linguistics(Volume1: Long Papers), pages 15262–15277, 2024. 28

    2024

  33. [33]

    L-eval: Instituting standardized evaluation for long context language models

    Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. InProceedings of the 62nd Annual Meeting of the Association forComputational Linguistics(Volume1: Long Papers), pages 14388–14411, 2024

    2024

  34. [34]

    Babilong: Testing the limits of llms with long context reasoning-in-a-haystack.Advances in Neural Information Processing Systems, 37:106519–106554, 2024

    Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack.Advances in Neural Information Processing Systems, 37:106519–106554, 2024

    2024

  35. [35]

    Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models

    Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models. InProceedings of the 2024 Joint International Conference on Computational Linguistics,LanguageResourcesandEvaluation(LREC-COLING2024), pages 2086–2099, 2024

    2024

  36. [36]

    Natural questions: a benchmark for question answering research.Transactionsof the Association forComputational Linguistics, 7:453–466, 2019

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research.Transactionsof the Association forComputational Linguistics, 7:453–466, 2019

    2019

  37. [37]

    How long can context length of open-source llms truly promise? InNeurIPS2023 Workshopon Instruction Tuningand Instruction Following, 2023

    DachengLi,RulinShao,AnzeXie,YingSheng,LianminZheng,JosephGonzalez,IonStoica,XuezheMa,andHaoZhang. How long can context length of open-source llms truly promise? InNeurIPS2023 Workshopon Instruction Tuningand Instruction Following, 2023

    2023

  38. [38]

    Zero-shot relation extraction via reading comprehension

    Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Proceedings ofthe 21stConferenceonComputationalNaturalLanguageLearning (CoNLL 2017), pages 333–342, 2017

    2017

  39. [39]

    Evaluating the ripple effects of knowledge editing in language models.Transactionsof theAssociation forComputational Linguistics, 12:283–298, 2024

    Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models.Transactionsof theAssociation forComputational Linguistics, 12:283–298, 2024

    2024

  40. [40]

    A comprehensive study of knowledge editing for large language models.arXivpreprint arXiv:2401.01286, 2024

    Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models.arXivpreprint arXiv:2401.01286, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models

    Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models. InProceedingsofthe2022 Conferenceon Empirical Methodsin NaturalLanguageProcessing, pages 6237–6250, 2022

    2022

  42. [42]

    Tofu: A task of fictitious unlearning for llms

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms. arXivpreprint arXiv:2401.06121, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    The wmdp benchmark: Measuring and reducing malicious use with unlearning.arXiv preprint arXiv:2403.03218, 2024

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning.arXiv preprint arXiv:2403.03218, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Muse: Machine unlearning six-way evaluation for language models

    Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. InInternational Conference onLearning Representations, volume 2025, pages 27797–27818, 2025

    2025

  45. [45]

    Rwku: Benchmarking real-world knowledge unlearning for large language models.Advances in Neural Information Processing Systems, 37:98213–98263, 2024

    Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, Jun Zhao, et al. Rwku: Benchmarking real-world knowledge unlearning for large language models.Advances in Neural Information Processing Systems, 37:98213–98263, 2024

    2024

  46. [46]

    Who’s harry potter? approximate unlearning in llms, arxiv.arXiv preprint arXiv:2310.02238, 2023

    R Eldan and M Russinovich. Who’s harry potter? approximate unlearning in llms, arxiv.arXiv preprint arXiv:2310.02238, 2023

    Pith/arXiv arXiv 2023

  47. [47]

    BLUR: A benchmark for LLM unlearning robust to forget-retain overlap, 2026

    Shengyuan Hu, Neil Kale, Pratiksha Thaker, Yiwei Fu, Steven Wu, and Virginia Smith. BLUR: A benchmark for LLM unlearning robust to forget-retain overlap, 2026

    2026

  48. [48]

    Unlearning that lasts: Utility-preserving, robust, and almost irreversible forgetting in llms.arXivpreprint arXiv:2509.02820, 2025

    Naman Deep Singh, Maximilian Müller, Francesco Croce, and Matthias Hein. Unlearning that lasts: Utility-preserving, robust, and almost irreversible forgetting in llms.arXivpreprint arXiv:2509.02820, 2025

    Pith/arXiv arXiv 2025

  49. [49]

    Clear: Character unlearning in textual and visual modalities

    AlexeyDontsov, DmitriiKorzh, Alexey Zhavoronkin, BorisMikheev, DenisBobkov, Aibek Alanov, OlegRogov, IvanOseledets, and Elena Tutubalina. Clear: Character unlearning in textual and visual modalities. InFindings of the Association for Computational Linguistics: ACL2025, pages 20582–20603, 2025

    2025

  50. [50]

    Locating and editing factual associations in gpt.Advancesin neural informationprocessing systems, 35:17359–17372, 2022

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt.Advancesin neural informationprocessing systems, 35:17359–17372, 2022. 29

    2022

  51. [51]

    Musique: Multihop questions via single-hop question composition.Transactionsofthe AssociationforComputational Linguistics, 10:539–554, 2022

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition.Transactionsofthe AssociationforComputational Linguistics, 10:539–554, 2022

    2022

  52. [52]

    Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.Transactions of the Association for Computational Linguistics, 9:346–361, 2021

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.Transactions of the Association for Computational Linguistics, 9:346–361, 2021

    2021

  53. [53]

    Measuring and narrowing the compositionality gap in language models

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. InFindingsoftheAssociationforComputationalLinguistics: EMNLP2023, pages 5687–5711, 2023

    2023

  54. [54]

    A survey on long-term memory security in llm agents: Attacks, defenses, and governance across the memory lifecycle, 2026

    Zehao Lin, Xixuan Hao, Renyu Fu, Shaobo Cui, Kai Chen, Chunyu Li, Zhiyu Li, and Feiyu Xiong. A survey on long-term memory security in llm agents: Attacks, defenses, and governance across the memory lifecycle, 2026

    2026

  55. [55]

    Memops: Benchmarking lifecycle memory operations in long-horizon conversations, 2026

    Xixuan Hao, Zeyu Zhang, Zehao Lin, Yihang Sun, Ziliang Guo, Xichong Zhang, Yuxuan Liang, Feiyu Xiong, and Zhiyu Li. Memops: Benchmarking lifecycle memory operations in long-horizon conversations, 2026

    2026

  56. [56]

    Nextmem: Towards latent factual memory for llm-based agents.arXivpreprint arXiv:2603.15634, 2026

    Zeyu Zhang, Rui Li, Xiaoyan Zhao, Yang Zhang, Wenjie Wang, Xu Chen, and Tat-Seng Chua. Nextmem: Towards latent factual memory for llm-based agents.arXivpreprint arXiv:2603.15634, 2026

    arXiv 2026

  57. [57]

    Squad: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. InProceedingsof the 2016conferenceonempirical methods innaturallanguageprocessing, pages 2383–2392, 2016

    2016

  58. [58]

    Hotpotqa: Adatasetfordiverse, explainablemulti-hopquestionanswering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: Adatasetfordiverse, explainablemulti-hopquestionanswering. In Proceedingsofthe2018conferenceonempirical methods innatural languageprocessing, pages 2369–2380, 2018

    2018

  59. [59]

    Qwen3.5: Towards native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026

    2026

  60. [60]

    Retrieval-augmented generation for knowledge-intensive nlp tasks.Advancesinneural informationprocessing systems, 33:9459–9474, 2020

    PatrickLewis,EthanPerez,AleksandraPiktus,FabioPetroni,VladimirKarpukhin,NamanGoyal,HeinrichKüttler,MikeLewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks.Advancesinneural informationprocessing systems, 33:9459–9474, 2020

    2020

  61. [61]

    With greater text comes greater necessity: Inference-time training helps long text generation

    Yan Wang, Dongyang Ma, and Deng Cai. With greater text comes greater necessity: Inference-time training helps long text generation. arXivpreprint arXiv:2401.11504, 2024

    Pith/arXiv arXiv 2024

  62. [62]

    According to me: Long-term personalized referential memory qa.arXivpreprint arXiv:2603.01990, 2026

    Jingbiao Mei, Jinghong Chen, Guangyu Yang, Xinyu Hou, Margaret Li, and Bill Byrne. According to me: Long-term personalized referential memory qa.arXivpreprint arXiv:2603.01990, 2026

    arXiv 2026

  63. [63]

    Memsim: A bayesian simulator for evaluating memory of llm-based personal assistants.Advances in Neural Information ProcessingSystems, 38:90475–90511, 2026

    Zeyu Zhang, Quanyu Dai, Luyu Chen, Zeren Jiang, Rui Li, Jieming Zhu, Xu Chen, Yi Xie, Zhenhua Dong, and Ji-Rong Wen. Memsim: A bayesian simulator for evaluating memory of llm-based personal assistants.Advances in Neural Information ProcessingSystems, 38:90475–90511, 2026

    2026

  64. [64]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.arXivpreprint arXiv:2406.01574, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.arXivpreprint arXiv:2406.01574, 2024

    Pith/arXiv arXiv 2024

  65. [65]

    Instruction- following evaluation for large language models.arXivpreprint arXiv:2311.07911, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction- following evaluation for large language models.arXivpreprint arXiv:2311.07911, 2023

    Pith/arXiv arXiv 2023

  66. [66]

    Training verifiers to solve math word problems.arXivpreprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXivpreprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  67. [67]

    Measuring massive multitask language understanding.arXivpreprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXivpreprint arXiv:2009.03300, 2020

    Pith/arXiv arXiv 2009

  68. [68]

    Deep research agents: A systematic examination and roadmap.arXivpreprint arXiv:2506.18096, 2025

    Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Huichi Zhou, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, et al. Deep research agents: A systematic examination and roadmap.arXivpreprint arXiv:2506.18096, 2025

    Pith/arXiv arXiv 2025

  69. [69]

    Deepresearcher: Scaling deep research via reinforcement learning in real-world environments

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. InProceedings of the 2025 Conference on Empirical Methods in NaturalLanguageProcessing, pages 414–431, 2025

    2025

  70. [70]

    Deepresearchbench: Acomprehensivebenchmark for deep research agents.arXivpreprint arXiv:2506.11763, 2025

    MingxuanDu,BenfengXu,ChiweiZhu,XiaoruiWang,andZhendongMao. Deepresearchbench: Acomprehensivebenchmark for deep research agents.arXivpreprint arXiv:2506.11763, 2025. 30

    Pith/arXiv arXiv 2025

  71. [71]

    A survey on in-context learning

    QingxiuDong, LeiLi, DamaiDai, CeZheng, JingyuanMa, RuiLi, HemingXia, JingjingXu, ZhiyongWu, BaobaoChang, etal. A survey on in-context learning. InProceedingsofthe2024conferenceonempirical methodsinnaturallanguageprocessing, pages 1107–1128, 2024

    2024

  72. [72]

    From isolated conversations to hierarchical schemas: Dynamic tree memory representation for llms

    Alireza Rezazadeh, Zichao Li, Wei Wei, and Yujia Bao. From isolated conversations to hierarchical schemas: Dynamic tree memory representation for llms. InInternational Conference on Learning Representations, volume 2025, pages 990–1023, 2025

    2025

  73. [73]

    Locas: Your models are principled initializers of locally-supported parametric memories.arXivpreprint arXiv:2602.05085, 2026

    Sidi Lu, Zhenwen Liang, Dongyang Ma, Yan Wang, Haitao Mi, and Dong Yu. Locas: Your models are principled initializers of locally-supported parametric memories.arXivpreprint arXiv:2602.05085, 2026

    arXiv 2026

  74. [74]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. InInternational conferenceon machinelearning, pages 5156–5165. PMLR, 2020

    2020

  75. [75]

    Lineartransformersaresecretlyfastweightprogrammers

    ImanolSchlag,KazukiIrie,andJürgenSchmidhuber. Lineartransformersaresecretlyfastweightprogrammers. In International conferenceonmachinelearning, pages 9355–9366. PMLR, 2021

    2021

  76. [76]

    Retentive network: A successor to transformer for large language models.arXivpreprint arXiv:2307.08621, 2023

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models.arXivpreprint arXiv:2307.08621, 2023

    Pith/arXiv arXiv 2023

  77. [77]

    Rwkv: Reinventingrnnsforthetransformerera

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung,LeonDerczynski,etal. Rwkv: Reinventingrnnsforthetransformerera. In Findingsoftheassociationforcomputational linguistics: EMNLP 2023, pages 14048–14077, 2023

    2023

  78. [78]

    Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396, 2021

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396, 2021

    Pith/arXiv arXiv 2021

  79. [79]

    Mamba: Linear-time sequence modeling with selective state spaces.arXivpreprint arXiv:2312.00752, 2023

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXivpreprint arXiv:2312.00752, 2023

    Pith/arXiv arXiv 2023

  80. [80]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXivpreprint arXiv:2405.21060, 2024

    Pith/arXiv arXiv 2024

Showing first 80 references.