pith. sign in

arxiv: 2605.28732 · v1 · pith:NT6X72U6new · submitted 2026-05-27 · 💻 cs.CL · cs.AI· cs.LG

MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems

Xinle Deng , Ruobin Zhong , Hujin Peng , Xiaoben Lu , Yanzhe Wu , Guang Li , Buqiang Xu , Yunzhi Yao
show 10 more authors
Jizhan Fang Haoliang Cao Junjie Guo Yuan Yuan Ziqing Ma Yuanqiang Yu Rui Hu Baohua Dong Hangcheng Zhu Ningyu Zhang
This is my paper

Pith reviewed 2026-06-29 12:23 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.LG
keywords LLM memory systemserror tracingmemory evolution graphsMemTraceBenchroot-cause attributionprompt optimizationinformation flowRAG
0
0 comments X

The pith

MemTrace converts LLM memory pipelines into graphs to trace information flow, locate failure causes, and auto-fix prompts for gains up to 7.62%.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models rely on memory to sustain reasoning across long inputs, yet these systems often corrupt or lose information without clear explanations. The work turns memory operations into executable graphs that record how data enters, combines, or disappears at each step. These graphs allow an attribution procedure to walk backward through subgraphs until it isolates the exact operation responsible for a given failure. The resulting signals then drive iterative prompt changes that repair the faults inside a closed loop. On a new benchmark built from systems such as RAG and long-context setups, the approach reveals that errors follow repeatable patterns rather than occurring at random.

Core claim

Memory pipelines can be represented as executable memory evolution graphs that make operational information flow traceable; an iterative subgraph-tracing attribution method then identifies the root operation causing any failed case; these attributions expose systematic failure modes such as information loss and retrieval misalignment; feeding the signals into prompt optimization creates a closed loop that corrects the faults and raises end-task performance.

What carries the argument

executable memory evolution graphs that represent memory pipelines and support fine-grained tracing of operation-level information flow

If this is right

  • Memory failures arise systematically from operation-level problems such as information loss and retrieval misalignment.
  • Iterative tracing of operation subgraphs isolates the precise root cause for any observed failure.
  • Attribution signals can be fed directly into prompt optimization to form an automatic correction loop.
  • End-task performance improves by up to 7.62 percent once the identified faults are corrected.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same graph representation could be applied to trace errors inside other LLM pipelines such as tool-use chains or multi-agent workflows.
  • MemTraceBench offers a reusable test bed for comparing the reliability of future memory architectures.
  • Closed-loop attribution may reduce the amount of human inspection needed when deploying memory-augmented models in production.

Load-bearing premise

Converting a memory pipeline into a graph accurately records how information actually moves and lets the tracing method correctly name the operation that caused a failure.

What would settle it

A controlled test in which the attribution method names one operation as the root cause, the corresponding prompt is edited, and the original failure either persists or the task score does not rise.

Figures

Figures reproduced from arXiv: 2605.28732 by Baohua Dong, Buqiang Xu, Guang Li, Hangcheng Zhu, Haoliang Cao, Hujin Peng, Jizhan Fang, Junjie Guo, Ningyu Zhang, Rui Hu, Ruobin Zhong, Xiaoben Lu, Xinle Deng, Yanzhe Wu, Yuanqiang Yu, Yuan Yuan, Yunzhi Yao, Ziqing Ma.

Figure 1
Figure 1. Figure 1: Framework for automatic diagnosis of LLM memory systems. We first execute a memory system to construct an execution graph. Given a failed case, MemTrace performs step-by-step tracing over this graph to locate the faulty operation. This framework is general across different memory systems and enables faster failure attribution than human experts. experience for future decisions, memory has be￾come widely ad… view at source ↗
Figure 2
Figure 2. Figure 2: An illustrative workflow of MemTrace. The initial to-explore list contains v1 and v10. Starting from v1, the agent inspects the operation subgraph corresponding to the operation o1, and finds that o1 correctly extracts the key user facts. The agent then adds v3 to the list to inspect subsequent operations. By continuing this graph exploration process, the agent identifies the faulty operation o3 in the thi… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of error distribution in MemTraceBench. find that the judge is overly strict. It penalizes re￾sponses that are essentially correct but either overly verbose or lacking sufficient specificity. High-quality annotation is intrinsically difficult on long-horizon memory benchmarks. Despite careful human verification, all three datasets con￾tain some annotation errors (see Figures 10 and 11). We find th… view at source ↗
Figure 4
Figure 4. Figure 4: a, smartcomment records the runtime exe￾cution graph of the memory system, and MemTrace performs credit assignment on this graph to local￾ize the earliest decisive faulty operation. Once that operation is identified, prompt optimiza￾tion reduces to a local problem: we only need to invoke an off-the-shelf optimizer on the small set of prompts participating in that operation. This sidesteps the difficulties … view at source ↗
Figure 5
Figure 5. Figure 5: The overview of dataset construction process. We define seven error types, five of which are specific to memory systems. We insert smartcomment-related code into each memory system and run these systems on public datasets to collect execution graphs and failed cases. For each failed case, once our annotators confirm that it is not caused by annotation errors or LLM-as-a-Judge errors, they identify the erro… view at source ↗
Figure 6
Figure 6. Figure 6: The annotation interface with each thumbnail clickable and linking to its corresponding full-size visualization. The left panel shows the entry point to the annotation interface, with the full-size visualization shown in [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Additional dataset analysis. (a) Token distribution of execution graph logs for each memory system. (b) Pairwise disagreement rates among annotators. Darker colors indicate higher disagreement. The disagreement between annotators 9q0hiycg and lyloqrja cannot be computed because their annotated cases do not overlap.       ! !"  #!    "$"         #         !… view at source ↗
Figure 9
Figure 9. Figure 9: An LLM-as-a-Judge error case. Due to ambiguity between the total required points and the remaining points needed, the prediction provides both correct values (“300 total” and “100 remaining”), but the LLM judge marks it incorrect. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: An Annotation case from LoCoMo. The evidence contains only five supported wins, but the golden answer expects six. Question I mentioned an investment for a competition four weeks ago? What did I buy? Source Evidence: • User (2023-03-04 13:12:00): I actually got my own set of sculpting tools, including a modeling tool set, a wire cutter, and a sculpting mat today. I'm excited to experiment with these new t… view at source ↗
Figure 11
Figure 11. Figure 11: An Annotation case from LongMemEval. The evidence confirms that the user obtains sculpting tools, but does not support the claim that the purchase is a competition-related investment. The question should instead remove the investment-related wording and ask a simpler supported query such as “What did I buy four weeks ago”. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: An instrumentation example. We insert two smartcomment statements, highlighted in green, into the method _delete_memory of the class Memory in the Mem0 source code to record the deletion of a memory unit. The method is extracted for presentation, with surrounding code omitted and indentation adjusted for readability. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Full-size visualization of the annotation interface entry point. This figure corresponds to the left thumbnail in the overview shown in [PITH_FULL_IMAGE:figures/full_fig_p032_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Full-size visualization of the annotation submission view. This figure corresponds to the middle thumbnail in the overview shown in [PITH_FULL_IMAGE:figures/full_fig_p033_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Full-size visualization of the execution graph exploration interface. This figure corresponds to the right thumbnail in the overview shown in [PITH_FULL_IMAGE:figures/full_fig_p033_15.png] view at source ↗
read the original abstract

Memory is essential for enabling large language models to support long-horizon reasoning, yet existing memory systems remain unreliable and difficult to debug. Tracing memory's dynamic evolution is crucial to understand how information is synthesized, propagated, or corrupted over time. In this work, we study the new problem of error tracing and attribution in LLM memory systems. We propose a novel framework that transforms memory pipelines into executable memory evolution graphs, enabling fine-grained tracing of operational information flow. We then construct MemTraceBench, a benchmark collected from representative memory systems such as Long-Context, RAG, Mem0, and EverMemOS, to systematically study memory failure modes. We further introduce an automatic attribution method that iteratively traces operation subgraphs to pinpoint the root cause of any failed case. Our analysis reveals that memory failures are systematic, stemming from operation-level issues like information loss and retrieval misalignment. Crucially, we leverage these fine-grained attribution signals to guide downstream prompt optimization, establishing a closed-loop system that automatically corrects faults and boosts end-task performance by up to 7.62%. Code will be released at https://github.com/zjunlp/MemTrace.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces MemTrace, a framework for tracing and attributing errors in LLM memory systems. It transforms memory pipelines into executable memory evolution graphs to enable fine-grained tracing of operational information flow, constructs MemTraceBench from representative systems including Long-Context, RAG, Mem0, and EverMemOS, develops an automatic attribution method that iteratively traces operation subgraphs to identify root causes such as information loss and retrieval misalignment, and applies these signals in a closed-loop prompt optimization system that boosts end-task performance by up to 7.62%. Code release is planned.

Significance. If the attribution method is shown to be reliable, the work could meaningfully improve debugging of LLM memory systems for long-horizon reasoning. The construction of MemTraceBench and planned code release are explicit strengths supporting reproducibility and follow-on research.

major comments (2)
  1. [Abstract] Abstract: the central claim that fine-grained attribution signals from memory evolution graphs enable a closed-loop system boosting performance by up to 7.62% is load-bearing, yet no precision/recall, human agreement, or ablation results are reported to show that corrections based on these attributions outperform generic prompt optimization or that the graphs faithfully capture dynamic flow.
  2. [Abstract] The automatic attribution method (iterative subgraph tracing) is presented as identifying operation-level failure causes, but without ground-truth validation on MemTraceBench or comparison to alternative tracing approaches, it is unclear whether the reported gains can be attributed to the framework rather than other factors.
minor comments (1)
  1. [Abstract] The abstract mentions 'systematic' failure modes but does not specify the criteria or statistical tests used to establish systematicity across the benchmark.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed feedback. We address the major comments point by point below, agreeing that additional validation will strengthen the claims regarding the attribution method and performance gains.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that fine-grained attribution signals from memory evolution graphs enable a closed-loop system boosting performance by up to 7.62% is load-bearing, yet no precision/recall, human agreement, or ablation results are reported to show that corrections based on these attributions outperform generic prompt optimization or that the graphs faithfully capture dynamic flow.

    Authors: We agree that the abstract's performance claim would benefit from explicit supporting metrics. The manuscript presents case studies showing how evolution graphs trace information flow and how attributions guide corrections, but does not include aggregate precision/recall or human agreement. In the revised manuscript we will add these evaluations on MemTraceBench along with an ablation comparing attribution-guided optimization against generic prompt optimization baselines. revision: yes

  2. Referee: [Abstract] The automatic attribution method (iterative subgraph tracing) is presented as identifying operation-level failure causes, but without ground-truth validation on MemTraceBench or comparison to alternative tracing approaches, it is unclear whether the reported gains can be attributed to the framework rather than other factors.

    Authors: MemTraceBench is built from systems with documented failure modes, and the attribution method is illustrated on those cases. However, the manuscript does not report quantitative ground-truth validation or head-to-head comparisons with other tracing techniques. We will incorporate such validation (using operation-level labels in the benchmark) and comparisons to alternative approaches in the revision to clarify the source of the observed gains. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The provided abstract and text describe an empirical framework for memory error tracing via executable graphs, benchmark construction from existing systems, an iterative subgraph attribution method, and downstream prompt optimization yielding up to 7.62% gains. No equations, derivations, or self-referential definitions are present that reduce any claimed result to its inputs by construction. No self-citations, ansatzes, or fitted inputs presented as predictions appear. The performance figure is reported as an experimental outcome from applying the method, not a tautological or load-bearing self-defined quantity. The derivation chain is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

Abstract-only; ledger populated from stated elements in the summary. No explicit free parameters named. One domain assumption and one invented construct identified.

axioms (1)
  • domain assumption Memory pipelines in LLMs can be represented as sequences of discrete operations whose information flow can be tracked.
    Required for the graph transformation step described in the abstract.
invented entities (1)
  • memory evolution graphs no independent evidence
    purpose: Transform memory pipelines into executable structures for fine-grained tracing of operational information flow.
    New representational construct introduced by the framework.

pith-pipeline@v0.9.1-grok · 5797 in / 1225 out tokens · 28289 ms · 2026-06-29T12:23:13.251677+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 10 canonical work pages · 7 internal anchors

  1. [1]

    Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation

    Monitoring reasoning models for misbehav- ior and the risks of promoting obfuscation.ArXiv, abs/2503.11926. Yuanchen Bei, Tianxin Wei, Xuying Ning, Yanjun Zhao, Zhining Liu, Xiao Lin, Yada Zhu, Hendrik Hamann, Jingrui He, and Hanghang Tong. 2026. Mem-gallery: Benchmarking multimodal long-term conversational memory for mllm agents.ArXiv, abs/2601.03515. Ha...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  2. [2]

    Trace is the next autodiff: Generative optimiza- tion with rich feedback, execution traces, and llms. InAdvances in Neural Information Processing Sys- tems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj ...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo

    Who is introducing the failure? automatically attributing failures of multi-agent systems via spec- trum analysis.ArXiv, abs/2509.13782. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. 2024. A survey on llm-as-a-judge.ArXiv, abs/2411.15594. Bernal Jime...

    work page arXiv 2024

  4. [4]

    Chuanrui Hu, Xingze Gao, Zuyi Zhou, Dannong Xu, Yi Bai, Xintong Li, Hui Zhang, Tong Li, Chong Zhang, Lidong Bing, and Yafeng Deng

    Memoryarena: Benchmarking agent memory in interdependent multi-session agentic tasks.ArXiv, abs/2602.16313. Chuanrui Hu, Xingze Gao, Zuyi Zhou, Dannong Xu, Yi Bai, Xintong Li, Hui Zhang, Tong Li, Chong Zhang, Lidong Bing, and Yafeng Deng. 2026a. Ev- ermemos: A self-organizing memory operating sys- tem for structured long-horizon reasoning.ArXiv, abs/2601....

    work page arXiv 2025

  5. [5]

    LLMs Get Lost In Multi-Turn Conversation

    ACM. Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. 2025. Llms get lost in multi-turn conversation.ArXiv, abs/2505.06120. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. 2026. Meta- harness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052. Patrick Lewis, Ethan Perez, Al...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  6. [6]

    InThe Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024

    Let’s verify step by step. InThe Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Review.net. Jiaqi Liu, Yaofeng Su, Peng Xia, Siwei Han, Zeyu Zheng, Cihang Xie, Mingyu Ding, and Huaxiu Yao

    2024

  7. [7]

    SimpleMem: Efficient Lifelong Memory for LLM Agents

    Simplemem: Efficient lifelong memory for llm agents.ArXiv, abs/2601.02553. Junming Liu, Yifei Sun, Weihua Cheng, Haodong Lei, Yirong Chen, Licheng Wen, Xuemeng Yang, Daocheng Fu, Pinlong Cai, Nianchen Deng, Yi Yu, Shuyue Hu, Botian Shi, and Ding Wang. 2025. Mem- verse: Multimodal memory for lifelong learning agents.ArXiv, abs/2512.03627. Nelson F. Liu, Ke...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  8. [8]

    Evaluating very long-term conversational memory of LLM agents. InProceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 13851–13870. Association for Computational Lin- guistics. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D

    mlsys.org. Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning

  10. [10]

    UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning

    RAPTOR: recursive abstractive processing for tree-organized retrieval. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. 12 Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving AI tasks with chatgpt and its friends in hug...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    ArXiv, abs/2602.23701

    From flat logs to causal graphs: Hierarchical failure attribution for llm-based multi-agent systems. ArXiv, abs/2602.23701. Yu Wang and Xi Chen. 2025. Mirix: Multi-agent memory system for llm-based agents.ArXiv, abs/2507.07957. Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Gra- ham Neubig. 2025d. Agent workflow memory. In Forty-second International Con...

    work page arXiv 2025

  12. [12]

    A-MEM: Agentic Memory for LLM Agents

    Association for Computational Linguistics. Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. 2025. A-mem: Agentic memory for llm agents.ArXiv, abs/2502.12110. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V . Le, Denny Zhou, and Xinyun Chen. 2024a. Large language models as optimizers. In The Twelfth International Confe...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [13]

    message", 5 comment=

    offers a more fine-grained automatic eval- uation by assessing the accuracy of memory ex- traction and memory updating. However, it mainly checks whether target memories can be found in the current memory store through retrieval, which may not always reflect the true system behavior. It also cannot reveal when an error is introduced or which operation cau...

    2024