Pith. sign in

REVIEW 2 major objections 5 minor 98 references

Short-horizon memory tests mis-rank the systems long-lived agents need: architecture rankings flip as history lengthens.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:11 UTC pith:YXPRLSBU

load-bearing objection A genuinely useful ground-truth-first memory benchmark with a plausible tenure crossover, but the renderer-fidelity audit has a timing hole that could shift the headline—worth serious peer review. the 2 major comments →

arxiv 2607.21962 v1 pith:YXPRLSBU submitted 2026-07-24 cs.CL

Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings

Quentin Spencer This is my paper
classification cs.CL
keywords agent memoryground-truth-first benchmarktenure crossovermemory architectureevictionprovenanceinjection resistancewrite-stage quality
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that agent-memory architecture rankings invert with history length, so evaluations that only test short interactions can pick the wrong system for a long-lived assistant. The authors build a benchmark where ground truth is planted before any text exists—facts carry validity intervals, volatility classes, and source channels—then render chat and email from those facts. Using this instrument, they measure five memory architectures at three weeks and nine weeks. The budgeted curated-map memory wins at three weeks but falls behind a provenance-typed graph and a layered hybrid by nine weeks because eviction consumes the oldest content. The paper also argues that write-stage quality strongly correlates with downstream accuracy and that injection resistance tracks whether provenance boundaries survive representation.

Core claim

The central claim is a measured ranking inversion: a budgeted curated-map memory that leads at week 3 (81.2% vs. 75.9% for the graph) loses recall of evicted early content by week 9 (96.3% to 72.2% on early-chapter probes), while a provenance-typed graph rises to 90.4% and a layered hybrid to 93.2%. Under complete cross-family re-judging, the graph-over-map interaction is positive for all six users (exact p = 0.031), and the week-9 margin is significant (57/18 discordant, p < 10^-4). The mechanism is eviction: a full-rendered-history baseline holds early-chapter recall at 160/162, showing that old content survives when nothing is evicted. The paper further claims that write-stage fact qualit

What carries the argument

The instrument is a ground-truth-first pipeline: a seeded life-script sampler emits facts with validity intervals, volatility classes, and sent/received channel provenance before any text exists; an LLM renderer writes chat and email from per-event fact manifests; a fidelity verifier confirms every planted fact appears; and questions are instantiated mechanically from the script, so gold answers are script-valid by construction. As-of-date question sets at weeks 3, 6, and 9 make the tenure crossover measurable, and a deterministic answer-text classifier separates wrong-specific answers from abstentions to expose confident-confabulation behavior.

Load-bearing premise

The rendered corpus faithfully realizes the planted life script: the fidelity verifier checks that planted facts appear, but not that the renderer avoids adding or omitting content in ways the open-book oracle tolerates, so script-valid gold answers could mis-measure what a memory system actually remembers.

What would settle it

Re-run the evaluation with a two-sided renderer audit (planted-fact recall plus unsupported-fact precision) and human adjudication of rendered support; if unsupported renderer additions are common and oracle-tolerant, or if a 12-week horizon with fresh users fails to reproduce the graph-over-map crossover under a second answerer, the central claim is weakened.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Short-horizon agent-memory benchmarks can mis-rank the architectures a long-lived assistant needs; evaluations should report accuracy as a function of history length.
  • Budgeted/curated memory stores need eviction policies that protect early-epoch content, or they will degrade exactly where long-horizon recall matters.
  • A layered architecture—unbounded provenance-preserving store, LLM-curated budgeted view, focused per-question retrieval—is top-tier at both horizons in this corpus.
  • Write-stage quality is a strong correlate of downstream memory accuracy: weakly written facts fail at roughly 24% versus 2% for clean writes, with fact-level alignment between write errors and QA misses.
  • Injection resistance in this harness tracks whether provenance boundaries survive representation: claim-typed graph edges and intact source framing both blocked all planted probes, while flattened assertional stores leaked on a subset.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the crossover generalizes, single-horizon leaderboards may systematically select systems that degrade with age; tenure curves should become a standard reporting axis for memory evaluations.
  • The eviction mechanism suggests a testable design rule: evict by volatility class and recency-weighted value rather than a uniform budget, and measure early-chapter recall under that policy.
  • The write-quality association implies that a write-time audit against planted ground truth could serve as an early warning gate in production memory systems, flagging weak facts before they cause downstream failures.
  • The provenance-quarantine finding points to a concrete ablation: hold retrieval and serialization fixed and vary only whether third-party claims are stored as typed claim edges versus flattened assertions, to test whether representation itself, not topology, is the causal property.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a ground-truth-first benchmark for long-lived agent memory: a seeded life-script sampler emits facts with validity intervals, volatility classes, and provenance before any text exists; an LLM renderer writes chat/email from per-event fact manifests; a fidelity verifier checks planted-fact presence; and questions are instantiated mechanically from the script, with gold answers script-valid by construction and separately checked for answerability. The reported corpus contains ~380 questions across 15 types. The paper benchmarks five memory architectures, a no-memory control, a full-rendered-history baseline, and a recency-window baseline at three-, six-, and nine-week horizons. The central empirical claim is a tenure crossover: the budgeted curated-map memory leads at week 3, but by week 9 the provenance-typed graph and a layered hybrid overtake it, with the graph–map interaction positive for all six users under a complete cross-family re-judging (exact p = 0.031). Secondary findings concern write-stage quality as a correlate of downstream failures, injection resistance tracking provenance-boundary preservation, and a confident-confabulation failure mode. The layered design is released as Veracium.

Significance. If the tenure crossover is valid, it is a significant, concrete demonstration that short-horizon benchmarks can mis-rank memory architectures, and the ground-truth-first instrument is a genuinely useful methodological contribution. The protocol is careful: exact McNemar tests, user-cluster bootstrap CIs, complete cross-family re-judging of 8,985 answers, explicit disclosure of the resolution floor (p = 0.063) for the primary-judge interaction, per-user and per-replicate distributions, and a full-history baseline that directly tests the short-horizon compression concern. The paper is unusually transparent about its limitations, including benchmark-informed iteration, six-cluster tenure statistics, and injection-probe bounds. The commitment to release the generator, harness, verdicts, logs, and re-derivation scripts is a strength, though the artifact package is currently listed as 'in preparation.' The main weakness is that the central claim depends on an unverified link between the planted script and the rendered corpus: the fidelity verifier checks only planted-fact presence, and the answerability audit is not checkpoint-restricted. Until that link is established, the crossover rem

major comments (2)
  1. [§3.1, §3.3, §7 (renderer fidelity / answerability)] The central claim presupposes that each as-of-date question is answerable from text available at that checkpoint and that gold reflects the planted script. §3.1 says the fidelity verifier checks planted-fact presence only; §3.3's answerability audit gives an oracle the complete rendered corpus with no time restriction. A week-3 question can pass even if its only textual support appears at week 9, and can pass via renderer-invented content because grounding in the planted fact is not checked. The paper reports one such case and lists a two-sided renderer audit as future work (§7). If early facts were rendered late, or unsupported additions survived in unbounded stores after budgeted stores evicted them, the map's week-3 lead and graph's week-9 rise could be inflated independently of actual memory. Cross-family re-judging cannot fix invalid gold. Please run a checkpoint-restricted answerab
  2. [§3.2 / §3.3 (corpus size vs audited questions)] §3.2 reports 383 validated questions; §3.3 reports an answerability audit of 215/220 generated questions (97.7%). Please reconcile: were the 108 tenure as-of-date questions included in that audit? If not, the crossover's gold answers lack answerability validation for the exact questions used. The five exclusions in §3.3 all stem from the oracle refusing third-party-only evidence, which is precisely the provenance-trust behavior most relevant to the graph backend; the tenure subset should be audited under the same protocol and its exclusion rate reported.
minor comments (5)
  1. [§5.4 / abstract] 'Judge-robust' should be phrased as 'robust to one alternative LLM judge,' since the primary judge's interaction is at the resolution floor (p = 0.063) and human adjudication is still pending.
  2. [§5.3 / §7] Hybrid v2's short-horizon 96.8% is explicitly a development-set number. The abstract's 'performs best among the memory systems' should carry the same benchmark-informed caveat.
  3. [Reproducibility statement] The artifact package is 'in preparation' and lacks a release tag/DOI; the released-code claims are promises at this stage. Please supply the archive link before archival publication.
  4. [§2.4 and references] Two distinct works are named 'MemAudit' (Tan et al. post-hoc auditing; Bhargava/Sobral Barrento write-quality protocol). Please disambiguate to avoid confusion.
  5. [Appendix B] The abstention/wrong-specific regex is acknowledged as coarse; the planned blinded human adjudication should report this classifier's agreement with human labels.

Circularity Check

0 steps flagged

No significant circularity; the central crossover is an empirical interaction grounded in script-defined ground truth, not a fitted or self-cited derivation.

full rationale

Walking the claimed derivation chain: ground truth is generated before any text exists by a seeded life-script sampler, and gold answers are instantiated mechanically from the script, so they are defined independently of the rendered corpus and of every memory backend. The renderer and answerability stages are quality checks, not fitting steps: §3.1 explicitly acknowledges that the fidelity verifier checks planted-fact presence rather than renderer precision, and §3.3's open-book oracle is described as a partial backstop; these are validity limitations, not equations that define the outcome. The tenure crossover is an empirical interaction computed from judged answers at two horizons, with the mechanism isolated by early-chapter probes and a full-history baseline; no parameter is fitted to the week-9 ranking and then reported as a prediction. The one design iteration that touched the same corpus — hybrid v2's short-horizon score — is explicitly labeled benchmark-informed (§5.3, §7), and the crossover uses six users generated after the architecture was fixed, giving it a meaningful held-out character. The write-quality association and injection findings are measured correlations/observations with acknowledged shared-evaluator and scope caveats (§5.5, §5.7), not circular derivations. There is no self-citation chain or imported uniqueness theorem doing load-bearing work. The residual threats (renderer additions, judge style interactions, single answerer) are external-validity risks and belong under correctness risk, not circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 2 invented entities

The empirical claims rest on design choices (corpus scale, horizons, budgets, probe templates) and on assumptions about renderer fidelity, LLM-judge validity, synthetic-user independence, and pretraining non-leakage. These are mostly disclosed by the paper; none is fitted to a target result, but several are load-bearing and would benefit from sensitivity analysis and human adjudication.

free parameters (5)
  • Tenure corpus size = 6 users × 3 replicates × 3 checkpoints (n=324/cell)
    Sample size chosen by the author; only six user clusters exist, and the paper's own primary-judge interaction test is resolution-limited (p=0.063).
  • History horizons and corpus scale = 3, 6, 9 weeks; short horizon ~30–60 records/user
    Hand-chosen; the short horizon deliberately sits below ConvoMem's ~150-conversation threshold, which compresses short-horizon deltas.
  • Eviction budget / read budget of memory systems = hybrid v2 mean ~1,807 tokens/question; curated map 1,183→1,182 tok/q
    The budget determines eviction timing and therefore the crossover; no sensitivity analysis over budget values is reported.
  • Injection probe set = 14 unique questions, 42 answer trials, 6-rung ladder
    Hand-designed and template-grade; the authors note 0/14 unique failures bounds the per-probe assertion rate only below ~19%.
  • Question templates and fact mapping = 15 question types, 20 templates; 17 of 20 mechanically mapped to audit keys
    The write-quality association covers the mechanically mapped subset; abstention and injection templates have no stored-fact key.
axioms (6)
  • domain assumption The rendered corpus faithfully realizes the planted life script; the fidelity verifier checks planted-fact presence, not renderer additions.
    §3.1/§3.3: the verifier checks presence only, and the answerability audit is a partial backstop that can succeed despite unsupported additions.
  • domain assumption The versioned LLM judge is a valid measure of answer correctness for differential claims.
    §3.4/§7: judge error interacting with answer style is backend-correlated; only 3 errors in ~200 hand-spot-checked verdicts; human adjudication pending.
  • domain assumption Six tenure users are independent generator draws sufficient for cluster-level inference.
    §7: same-archetype users share a chapter-index work-fact skeleton, archetype draw was concentrated 4/2/0, and bootstrap coverage is discrete at n=6.
  • domain assumption Fictionalized synthetic entities do not leak usable priors into the answerer or judge.
    §3.1: based on MemGym contamination findings; the corpus is fictional, but the renderer, answerer, and judge may still have genre-level priors.
  • domain assumption Provider-default decoding with unseeded API calls yields three valid stochastic replicates.
    §3.4: providers expose no random-seed control, so replicates are repeated API runs; decoding defaults are not archived.
  • domain assumption The cross-family judge (DeepSeek v3.2) is an independent re-judging of the same answers.
    §7/Appendix B: overall agreement 87.9%, Cohen's κ=0.60, with asymmetric strictness; there is no human ground truth yet.
invented entities (2)
  • Veracium layered provenance-aware memory library no independent evidence
    purpose: Operationalizes the winning design: unbounded provenance-preserving store, LLM-curated budgeted wiki, per-question retrieval, and an assertability gate.
    New software artifact released by the paper; no external validation yet beyond the paper's own harness and an internal smoke test.
  • Assertability gate no independent evidence
    purpose: Read-time mechanism that refuses to assert facts whose only support is third-party-authored evidence, closing the confident-confabulation and summarization-laundering channels.
    Motivated by the observed injection leak and the confident-error pattern in §5.7; implemented in Veracium but not independently evaluated.

pith-pipeline@v1.3.0-alltime-deepseek · 21638 in / 10531 out tokens · 109570 ms · 2026-08-01T06:11:57.864366+00:00 · methodology

0 comments
read the original abstract

Benchmarks for LLM-agent memory typically generate conversations first and extract answer keys afterwards -- with documented label-error and contamination problems -- and they overwhelmingly measure short interaction histories. We invert the pipeline: a seeded life-script sampler emits facts with validity intervals, volatility classes, and source channels before any text exists; an LLM renderer writes chat and email from per-event fact manifests; a fidelity verifier confirms every planted fact; and questions are instantiated mechanically from the script, so gold answers are script-valid by construction and separately validated for answerability. The synthetic, fictionalized corpus (~380 questions, 15 types) embeds features absent from the benchmarks we survey: per-fact validity intervals, sent/received trust distinctions, injection probes in a benign harness, and as-of-date question sets. Benchmarking five memory architectures against a no-memory control (fixed answerer, versioned LLM judge, three replicates, two horizons), we find backend rankings invert with history length: the budgeted curated-map memory that leads at three weeks loses recall of evicted content by nine weeks (96% to 72%) while a provenance-typed graph rises to 90%; the inversion is positive for all six users under complete cross-family re-judging (exact p=0.031). A full-rendered-history baseline ties or exceeds the best memory system at the short horizon but shows no judge-independent advantage at nine weeks, at about twice the read cost. Write-stage quality strongly correlates with downstream quality (weakly-written facts fail 24% vs 2%), and injection resistance tracked whether provenance boundaries survive representation. A layered architecture performs best among the memory systems in both regimes (96.8% short-horizon) and is released as Veracium, an open-source library, with the corpus generator and harness.

Figures

Figures reproduced from arXiv: 2607.21962 by Quentin Spencer.

Figure 1
Figure 1. Figure 1: The tenure crossover: aggregate accuracy by history length (6 users [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Early-chapter (evicted-content) recall by history length: the three [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

98 extracted references · 43 linked inside Pith

  1. [1]

    MEMAUDIT : An exact package-oracle evaluation protocol for budgeted long-term LLM memory writing

    Nishant Bhargava and Rodrigo Sobral Barrento. MEMAUDIT : An exact package-oracle evaluation protocol for budgeted long-term LLM memory writing. arXiv preprint arXiv:2605.02199, 2026. URL https://arxiv.org/abs/2605.02199

    Pith/arXiv arXiv 2026

  2. [2]

    AgentPoison : Red-teaming LLM agents via poisoning memory or knowledge bases

    Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. AgentPoison : Red-teaming LLM agents via poisoning memory or knowledge bases. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https://arxiv.org/abs/2407.12784

    Pith/arXiv arXiv 2024

  3. [3]

    Mem0 : Building production-ready AI agents with scalable long-term memory

    Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0 : Building production-ready AI agents with scalable long-term memory. arXiv preprint arXiv:2504.19413, 2025. URL https://arxiv.org/abs/2504.19413

    Pith/arXiv arXiv 2025

  4. [4]

    From untrusted input to trusted memory: A systematic study of memory poisoning attacks in LLM agents

    Pritam Dash, Tongyu Ge, Aditi Jain, Tanmay Shah, and Zhiwei Shang. From untrusted input to trusted memory: A systematic study of memory poisoning attacks in LLM agents. arXiv preprint arXiv:2606.04329, 2026. URL https://arxiv.org/abs/2606.04329

    Pith/arXiv arXiv 2026

  5. [5]

    MemTrace : Tracing and attributing errors in large language model memory systems

    Xinle Deng, Ruobin Zhong, Hujin Peng, Xiaoben Lu, Yanzhe Wu, Guang Li, Buqiang Xu, Yunzhi Yao, Jizhan Fang, Haoliang Cao, Junjie Guo, Yuan Yuan, Ziqing Ma, Yuanqiang Yu, Rui Hu, Baohua Dong, Hangcheng Zhu, and Ningyu Zhang. MemTrace : Tracing and attributing errors in large language model memory systems. arXiv preprint arXiv:2605.28732, 2026. URL https://...

    Pith/arXiv arXiv 2026

  6. [6]

    SmartSearch : How ranking beats structure for conversational memory retrieval

    Jesper Derehag, Carlos Calva, and Timmy Ghiurau. SmartSearch : How ranking beats structure for conversational memory retrieval. arXiv preprint arXiv:2603.15599, 2026. URL https://arxiv.org/abs/2603.15599

    arXiv 2026

  7. [7]

    Detecting hallucinations in large language models using semantic entropy

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630 0 (8017): 0 625--630, 2024. URL https://www.nature.com/articles/s41586-024-07421-0

    2024

  8. [8]

    OrgForge-IT : A verifiable synthetic benchmark for LLM -based insider threat detection

    Jeffrey Flynt. OrgForge-IT : A verifiable synthetic benchmark for LLM -based insider threat detection. arXiv preprint arXiv:2603.22499, 2026. URL https://arxiv.org/abs/2603.22499

    arXiv 2026

  9. [9]

    Not what you've signed up for: Compromising real-world LLM -integrated applications with indirect prompt injection

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you've signed up for: Compromising real-world LLM -integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (AISec), 2023. URL https://arxiv.org/abs/2302.12173

    Pith/arXiv arXiv 2023

  10. [10]

    PEEK : Context map as an orientation cache for long-context LLM agents

    Zhuohan Gu, Qizheng Zhang, Omar Khattab, and Samuel Madden. PEEK : Context map as an orientation cache for long-context LLM agents. arXiv preprint arXiv:2605.19932, 2026. URL https://arxiv.org/abs/2605.19932

    Pith/arXiv arXiv 2026

  11. [12]

    MemoryArena : Benchmarking agent memory in interdependent multi-session agentic tasks

    Zexue He, Yu Wang, Churan Zhi, Yuanzhe Hu, Tzu-Ping Chen, Lang Yin, Ze Chen, Tong Arthur Wu, Siru Ouyang, Zihan Wang, Jiaxin Pei, Julian McAuley, Yejin Choi, and Alex Pentland. MemoryArena : Benchmarking agent memory in interdependent multi-session agentic tasks. arXiv preprint arXiv:2602.16313, 2026. URL https://arxiv.org/abs/2602.16313

    arXiv 2026

  12. [13]

    Evaluating memory in LLM agents via incremental multi-turn interactions

    Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in LLM agents via incremental multi-turn interactions. arXiv preprint arXiv:2507.05257, 2025. URL https://arxiv.org/abs/2507.05257. MemoryAgentBench; ICLR 2026

    Pith/arXiv arXiv 2025

  13. [16]

    Governing evolving memory in LLM agents: Risks, mechanisms, and the stability and safety governed memory ( SSGM ) framework

    Chingkwun Lam, Jiaxin Li, Lingfei Zhang, and Kuo Zhao. Governing evolving memory in LLM agents: Risks, mechanisms, and the stability and safety governed memory ( SSGM ) framework. arXiv preprint arXiv:2603.11768, 2026. URL https://arxiv.org/abs/2603.11768

    Pith/arXiv arXiv 2026

  14. [18]

    A survey on long-term memory security in LLM agents: Attacks, defenses, and governance across the memory lifecycle

    Zehao Lin, Xixuan Hao, Renyu Fu, Shaobo Cui, Kai Chen, Chunyu Li, Zhiyu Li, and Feiyu Xiong. A survey on long-term memory security in LLM agents: Attacks, defenses, and governance across the memory lifecycle. arXiv preprint arXiv:2604.16548, 2026. URL https://arxiv.org/abs/2604.16548

    Pith/arXiv arXiv 2026

  15. [19]

    Securing LLM -agent long-term memory against poisoning: Non-malleable, origin-bound authority with machine-checked guarantees

    Yedidel Louck. Securing LLM -agent long-term memory against poisoning: Non-malleable, origin-bound authority with machine-checked guarantees. arXiv preprint arXiv:2606.24322, 2026. URL https://arxiv.org/abs/2606.24322

    Pith/arXiv arXiv 2026

  16. [21]

    Evaluating very long-term conversational memory of LLM agents

    Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fung. Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. URL https://arxiv.org/abs/2402.17753

    Pith/arXiv arXiv 2024

  17. [22]

    AI memory benchmarks in 2026: LoCoMo , LongMemEval & BEAM

    Mem0 . AI memory benchmarks in 2026: LoCoMo , LongMemEval & BEAM . Industry report (blog), 2026. URL https://mem0.ai/blog/ai-memory-benchmarks-in-2026

    2026

  18. [23]

    Locating and editing factual associations in GPT

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT . In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https://arxiv.org/abs/2202.05262

    Pith/arXiv arXiv 2022

  19. [26]

    LLM01 :2025 prompt injection, 2025

    OWASP GenAI Security Project . LLM01 :2025 prompt injection, 2025. URL https://genai.owasp.org/llmrisk/llm01-prompt-injection/

    2025

  20. [27]

    Patil, Ion Stoica, and Joseph E

    Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. MemGPT : Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560, 2023. URL https://arxiv.org/abs/2310.08560

    Pith/arXiv arXiv 2023

  21. [28]

    Convomem benchmark: Why your first 150 conversations don't need RAG

    Egor Pakhomov, Erik Nijkamp, and Caiming Xiong. Convomem benchmark: Why your first 150 conversations don't need RAG . arXiv preprint arXiv:2511.10523, 2025. URL https://arxiv.org/abs/2511.10523

    arXiv 2025

  22. [30]

    Supersede: Diagnosing and training the memory-update gap in LLM agents

    Vedant Patel. Supersede: Diagnosing and training the memory-update gap in LLM agents. arXiv preprint arXiv:2606.27472, 2026. URL https://arxiv.org/abs/2606.27472

    Pith/arXiv arXiv 2026

  23. [31]

    We audited LoCoMo : 6.4\ accepts up to 63\ Blog post; reproducible audit: github.com/dial481/locomo-audit, 2026

    Penfield Labs . We audited LoCoMo : 6.4\ accepts up to 63\ Blog post; reproducible audit: github.com/dial481/locomo-audit, 2026. URL https://penfieldlabs.substack.com/p/we-audited-locomo-64-of-the-answer

    2026

  24. [32]

    Self-improving memory for agents (``brain'')

    Perplexity AI . Self-improving memory for agents (``brain''). Perplexity blog, 2026. URL https://www.perplexity.ai/hub/blog/self-improving-memory-for-agents

    2026

  25. [33]

    Hidden in memory: Sleeper memory poisoning in LLM agents

    Sidharth Pulipaka, Stanislau Hlebik, Leonidas Raghav, Sahar Abdelnabi, Vyas Raina, Ivaxi Sheth, and Mario Fritz. Hidden in memory: Sleeper memory poisoning in LLM agents. arXiv preprint arXiv:2605.15338, 2026. URL https://arxiv.org/abs/2605.15338

    Pith/arXiv arXiv 2026

  26. [34]

    Zep : A temporal knowledge graph architecture for agent memory

    Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. Zep : A temporal knowledge graph architecture for agent memory. arXiv preprint arXiv:2501.13956, 2025. URL https://arxiv.org/abs/2501.13956

    Pith/arXiv arXiv 2025

  27. [35]

    Don't ask the LLM to track freshness: A deterministic recipe for memory conflict resolution

    Vikas Reddy and Sumanth Challaram. Don't ask the LLM to track freshness: A deterministic recipe for memory conflict resolution. arXiv preprint arXiv:2606.01435, 2026. URL https://arxiv.org/abs/2606.01435

    Pith/arXiv arXiv 2026

  28. [36]

    Scalable and reliable evaluation of AI knowledge retrieval systems: RIKER and the coherent simulated universe

    JV Roig. Scalable and reliable evaluation of AI knowledge retrieval systems: RIKER and the coherent simulated universe. arXiv preprint arXiv:2601.08847, 2026. URL https://arxiv.org/abs/2601.08847

    arXiv 2026

  29. [38]

    Mem2ActBench : A benchmark for evaluating long-term memory utilization in task-oriented autonomous agents

    Yiting Shen, Kun Li, Wei Zhou, and Songlin Hu. Mem2ActBench : A benchmark for evaluating long-term memory utilization in task-oriented autonomous agents. arXiv preprint arXiv:2601.19935, 2026. URL https://arxiv.org/abs/2601.19935

    arXiv 2026

  30. [39]

    Evaluating memory structure in LLM agents

    Alina Shutova, Alexandra Olenina, Ivan Vinogradov, and Anton Sinitsin. Evaluating memory structure in LLM agents. arXiv preprint arXiv:2602.11243, 2026. URL https://arxiv.org/abs/2602.11243

    Pith/arXiv arXiv 2026

  31. [40]

    TriMem : Rethinking how to remember---beyond atomic facts in lifelong LLM agent memory

    Jingwei Sun, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. TriMem : Rethinking how to remember---beyond atomic facts in lifelong LLM agent memory. arXiv preprint arXiv:2605.19952, 2026. URL https://arxiv.org/abs/2605.19952

    Pith/arXiv arXiv 2026

  32. [41]

    MemAudit : Post-hoc auditing of poisoned agent memory via causal attribution and structural anomaly detection

    Zhewen Tan, Yilun Yao, Huiyan Jin, Wenhan Yu, Guoan Wang, Mengyuan Fan, Liang Lu, Feng Liu, Xiangzheng Zhang, Duohe Ma, Tong Yang, and Lin Sun. MemAudit : Post-hoc auditing of poisoned agent memory via causal attribution and structural anomaly detection. arXiv preprint arXiv:2605.23723, 2026. URL https://arxiv.org/abs/2605.23723

    Pith/arXiv arXiv 2026

  33. [42]

    Beyond a million tokens: Benchmarking and enhancing long-term memory in LLMs

    Mohammad Tavakoli, Alireza Salemi, Carrie Ye, Mohamed Abdalla, Hamed Zamani, and J Ross Mitchell. Beyond a million tokens: Benchmarking and enhancing long-term memory in LLMs . arXiv preprint arXiv:2510.27246, 2025. URL https://arxiv.org/abs/2510.27246

    arXiv 2025

  34. [43]

    MemDelta : Controlled baselines and hidden confounds in agent memory evaluation

    Kuan Wang. MemDelta : Controlled baselines and hidden confounds in agent memory evaluation. arXiv preprint arXiv:2606.29914, 2026. URL https://arxiv.org/abs/2606.29914

    Pith/arXiv arXiv 2026

  35. [46]

    LongMemEval : Benchmarking chat assistants on long-term interactive memory

    Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. LongMemEval : Benchmarking chat assistants on long-term interactive memory. In International Conference on Learning Representations (ICLR), 2025. URL https://arxiv.org/abs/2410.10813

    Pith/arXiv arXiv 2025

  36. [48]

    A-Mem : Agentic memory for LLM agents

    Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-Mem : Agentic memory for LLM agents. arXiv preprint arXiv:2502.12110, 2025. URL https://arxiv.org/abs/2502.12110

    Pith/arXiv arXiv 2025

  37. [49]

    Wujiang Xu, Yu Wang, Kai Mei, Kaiqu Liang, Zhenting Wang, Mingyu Jin, Han Zhang, Shi-Xiong Zhang, Wenyue Hua, Sambit Sahu, and Dimitris N. Metaxas. MemGym : a long-horizon memory environment for LLM agents. arXiv preprint arXiv:2605.20833, 2026. URL https://arxiv.org/abs/2605.20833

    Pith/arXiv arXiv 2026

  38. [50]

    Temporal validity in retrieval memory: Eliminating stale-fact errors for AI agents over evolving knowledge

    Neeraj Yadav. Temporal validity in retrieval memory: Eliminating stale-fact errors for AI agents over evolving knowledge. arXiv preprint arXiv:2606.26511, 2026. URL https://arxiv.org/abs/2606.26511

    Pith/arXiv arXiv 2026

  39. [51]

    Diagnosing retrieval vs

    Boqin Yuan, Yue Su, and Kun Yao. Diagnosing retrieval vs. utilization bottlenecks in LLM agent memory. arXiv preprint arXiv:2603.02473, 2026. URL https://arxiv.org/abs/2603.02473

    Pith/arXiv arXiv 2026

  40. [52]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM -as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://arxiv.org/abs/2306.05685

    Pith/arXiv arXiv 2023

  41. [53]

    MemoryBank : Enhancing large language models with long-term memory

    Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. MemoryBank : Enhancing large language models with long-term memory. Proceedings of the AAAI Conference on Artificial Intelligence, 2024. URL https://arxiv.org/abs/2305.10250

    Pith/arXiv arXiv 2024

  42. [54]

    Manning, Christopher Potts, and Danqi Chen

    Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. MQuAKE : Assessing knowledge editing in language models via multi-hop questions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. URL https://arxiv.org/abs/2305.14795

    Pith/arXiv arXiv 2023

  43. [57]

    arXiv preprint arXiv:2604.11364 , year =

    The Missing Knowledge Layer in Cognitive Architectures for AI Agents , author =. arXiv preprint arXiv:2604.11364 , year =

    Pith/arXiv arXiv

  44. [58]

    Evaluating Very Long-Term Conversational Memory of

    Maharana, Adyasha and Lee, Dong-Ho and Tulyakov, Sergey and Bansal, Mohit and Barbieri, Francesco and Fung, Yuwei , booktitle =. Evaluating Very Long-Term Conversational Memory of. 2024 , eprint =

    2024

  45. [59]

    , journal=

    Xu, Wujiang and Wang, Yu and Mei, Kai and Liang, Kaiqu and Wang, Zhenting and Jin, Mingyu and Zhang, Han and Zhang, Shi-Xiong and Hua, Wenyue and Sahu, Sambit and Metaxas, Dimitris N. , journal=. 2026 , url =

    2026

  46. [60]

    2025 , url =

    Chhikara, Prateek and Khant, Dev and Aryan, Saket and Singh, Taranjeet and Yadav, Deshraj , journal=. 2025 , url =

    2025

  47. [61]

    2025 , url =

    Rasmussen, Preston and Paliychuk, Pavlo and Beauvais, Travis and Ryan, Jack and Chalef, Daniel , journal=. 2025 , url =

    2025

  48. [62]

    Evaluating Memory in

    Hu, Yuanzhe and Wang, Yu and McAuley, Julian , journal=. Evaluating Memory in. 2025 , url =

    2025

  49. [63]

    Don't Ask the

    Reddy, Vikas and Challaram, Sumanth , journal=. Don't Ask the. 2026 , url =

    2026

  50. [64]

    2026 , url =

    Bhargava, Nishant and Sobral Barrento, Rodrigo , journal=. 2026 , url =

    2026

  51. [65]

    2026 , url =

    Sun, Jingwei and Zhu, Jianing and Yao, Jiangchao and Liu, Tongliang and Han, Bo , journal=. 2026 , url =

    2026

  52. [66]

    Convomem Benchmark: Why Your First 150 Conversations Don't Need

    Pakhomov, Egor and Nijkamp, Erik and Xiong, Caiming , journal=. Convomem Benchmark: Why Your First 150 Conversations Don't Need. 2025 , url =

    2025

  53. [67]

    2026 , url =

    Gu, Zhuohan and Zhang, Qizheng and Khattab, Omar and Madden, Samuel , journal=. 2026 , url =

    2026

  54. [68]

    2026 , url =

    Tan, Zhewen and Yao, Yilun and Jin, Huiyan and Yu, Wenhan and Wang, Guoan and Fan, Mengyuan and Lu, Liang and Liu, Feng and Zhang, Xiangzheng and Ma, Duohe and Yang, Tong and Sun, Lin , journal=. 2026 , url =

    2026

  55. [69]

    2026 , url =

    Self-improving Memory for Agents (``Brain'') , author =. 2026 , url =

    2026

  56. [70]

    Not What You've Signed Up For: Compromising Real-World

    Greshake, Kai and Abdelnabi, Sahar and Mishra, Shailesh and Endres, Christoph and Holz, Thorsten and Fritz, Mario , booktitle =. Not What You've Signed Up For: Compromising Real-World. 2023 , eprint =

    2023

  57. [71]

    2024 , eprint =

    Chen, Zhaorun and Xiang, Zhen and Xiao, Chaowei and Song, Dawn and Li, Bo , booktitle =. 2024 , eprint =

    2024

  58. [72]

    and Stoica, Ion and Gonzalez, Joseph E

    Packer, Charles and Wooders, Sarah and Lin, Kevin and Fang, Vivian and Patil, Shishir G. and Stoica, Ion and Gonzalez, Joseph E. , journal=. 2023 , url =

    2023

  59. [73]

    Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST) , year =

    Generative Agents: Interactive Simulacra of Human Behavior , author =. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST) , year =. 2304.03442 , url =

    Pith/arXiv arXiv

  60. [74]

    2024 , eprint =

    Zhong, Wanjun and Guo, Lianghong and Gao, Qiqi and Ye, He and Wang, Yanlin , journal=. 2024 , eprint =

    2024

  61. [75]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Guti\'. Advances in Neural Information Processing Systems (NeurIPS) , year =. 2405.14831 , url =

    Pith/arXiv arXiv

  62. [76]

    2025 , url =

    Xu, Wujiang and Liang, Zujie and Mei, Kai and Gao, Hang and Tan, Juntao and Zhang, Yongfeng , journal=. 2025 , url =

    2025

  63. [77]

    and Potts, Christopher and Chen, Danqi , booktitle =

    Zhong, Zexuan and Wu, Zhengxuan and Manning, Christopher D. and Potts, Christopher and Chen, Danqi , booktitle =. 2023 , eprint =

    2023

  64. [78]

    Locating and Editing Factual Associations in

    Meng, Kevin and Bau, David and Andonian, Alex and Belinkov, Yonatan , booktitle =. Locating and Editing Factual Associations in. 2022 , eprint =

    2022

  65. [79]

    International Conference on Learning Representations (ICLR) , year =

    Mass-Editing Memory in a Transformer , author =. International Conference on Learning Representations (ICLR) , year =. 2210.07229 , url =

    Pith/arXiv arXiv

  66. [80]

    Nature , volume =

    Detecting Hallucinations in Large Language Models Using Semantic Entropy , author =. Nature , volume =. 2024 , url =

    2024

  67. [81]

    arXiv preprint arXiv:2207.05221 , year =

    Language Models (Mostly) Know What They Know , author =. arXiv preprint arXiv:2207.05221 , year =

    Pith/arXiv arXiv

  68. [82]

    arXiv preprint arXiv:2407.18418 , year =

    Know Your Limits: A Survey of Abstention in Large Language Models , author =. arXiv preprint arXiv:2407.18418 , year =

    Pith/arXiv arXiv

  69. [83]

    2025 , eprint =

    Wu, Di and Wang, Hongwei and Yu, Wenhao and Zhang, Yuwei and Chang, Kai-Wei and Yu, Dong , booktitle =. 2025 , eprint =

    2025

  70. [84]

    and Zhang, Hao and Gonzalez, Joseph E

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , booktitle =. Judging. 2023 , eprint =

    2023

  71. [85]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , year =

    Large Language Models Are Not Fair Evaluators , author =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , year =. 2305.17926 , url =

    Pith/arXiv arXiv

  72. [86]

    arXiv preprint arXiv:2406.04244 , year =

    Benchmark Data Contamination of Large Language Models: A Survey , author =. arXiv preprint arXiv:2406.04244 , year =

    Pith/arXiv arXiv

  73. [87]

    Scalable and Reliable Evaluation of

    Roig, JV , journal=. Scalable and Reliable Evaluation of. 2026 , url =

    2026

  74. [88]

    2026 , url =

    Flynt, Jeffrey , journal=. 2026 , url =

    2026

  75. [89]

    Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in

    Tavakoli, Mohammad and Salemi, Alireza and Ye, Carrie and Abdalla, Mohamed and Zamani, Hamed and Mitchell, J Ross , journal=. Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in. 2025 , url =

    2025

  76. [90]

    2026 , url =

    Derehag, Jesper and Calva, Carlos and Ghiurau, Timmy , journal=. 2026 , url =

    2026

  77. [91]

    Diagnosing Retrieval vs

    Yuan, Boqin and Su, Yue and Yao, Kun , journal=. Diagnosing Retrieval vs. Utilization Bottlenecks in. 2026 , url =

    2026

  78. [92]

    arXiv preprint arXiv:2602.19320 , year =

    Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations , author =. arXiv preprint arXiv:2602.19320 , year =

    Pith/arXiv arXiv

  79. [93]

    arXiv preprint arXiv:2602.17913 , year =

    From Lossy to Verified: A Provenance-Aware Tiered Memory for Agents , author =. arXiv preprint arXiv:2602.17913 , year =

    arXiv

  80. [94]

    From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in

    Dash, Pritam and Ge, Tongyu and Jain, Aditi and Shah, Tanmay and Shang, Zhiwei , journal=. From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in. 2026 , url =

    2026

Showing first 80 references.