Pith. sign in

REVIEW 5 major objections 5 minor 96 references

ContextWeave: A Real-World Workflow Benchmark

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Recalling prior work improves language agents' performance on real office workflows: the strongest memory component raises Workspace Score from 68.08 to 78.20 and Preference Score from 41.50 to 70.60.

desk verdict A genuinely useful longitudinal agent-memory benchmark whose headline numbers are all LLM-judged; the benchmark is worth serious review, but the central effect sizes need human-calibrated validation before they are quoted. read the letter →

arxiv 2608.04830 v1 pith:27MIGULY submitted 2026-08-05 cs.AI

Bo Wang , Yuqian Yao , Enxi Wang , Luozhijie Jin , Yang Liu , Yiran Suo , Yuxuan Cai , Enyu Zhou
show 20 more authors
Yufei Gao Honglin Guo Tianyu Huai Li Ji Zhikai Lei Bufan Li Lizhi Lin Jinxiu Liu Jie Yang Jiazheng Zhou Maosen Zhou Pengfang Qian Shichun Liu Guanshan Liu Hao Zheng Yunhao Yu Hang Yan Jihua Kang Xinchi Chen Xipeng Qiu
This is my paper · ORCID
classification cs.AI
keywords agentmemoryworkflowbenchmarklongitudinalevaluationin-contextexperiencepreferencealignmentworkspacequalitydiagnosticsLLMagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ContextWeave is a benchmark that tests whether language agents perform better on realistic office tasks when they can recall their past work. Built from the multi-month, privacy-anonymized workflows of 14 people, it turns 1,005 real document-editing sessions into 568 executable evaluation tasks with Docker environments and task-specific rubrics. The paper's central claim is that access to prior experience improves downstream task performance, with every one of five base models benefiting from recall in both workspace quality and preference alignment. It further argues that concrete in-context experience helps agents continue workflows better than compact summaries, while carrying a higher risk of misleading recall. A sympathetic reader would care because most existing memory benchmarks only test retrieval or question-answering, not whether memory actually makes an agent's final work better.

What carries the argument

The central object is the sequential task stream $D=(T_1,\dots,T_n)$ with the memory-induced gain $\Delta R_M(T_i)=R_M(T_i)-R(T_i)$, measured as the difference between an agent's performance on task $T_i$ with and without access to its preceding task histories. The operative machinery is the benchmark construction pipeline: real worklogs are segmented into tasks, anonymized with structure-preserving replacements, reconstructed into instructions plus containerized observations or controlled mock APIs, and aligned via rewritten trajectories so that the pre-task environment matches the original workflow state. This controlled trajectory alignment is what makes the memory gain attributable to recalled experience rather than to drift in earlier executions.

What would settle it

Take a random sample of, say, 100 of the 568 core tasks, run the no-recall and strongest with-recall conditions, and have human annotators who do not know which output has memory blindly rate workspace usefulness and preference alignment; if human-rated gains are substantially smaller than the GPT-5.5-rated gains (68.08 to 78.20 and 41.50 to 70.60), the central claim that memory improves downstream performance in real workflows would be called into question.

Watch

Extended reading notes

Core claim

The paper's discovery is that memory in language agents is best evaluated as downstream performance on real, longitudinally ordered workflows, not as retrieval accuracy. Under a fixed agent harness and model, adding recalled prior experience increases Workspace Score from 68.08 to 78.20 and Preference Score from 41.50 to 70.60 with the strongest memory component (a-mem), and the with-recall trajectory wins against the no-recall trajectory 72.70% of the time. Across five base models using mem0, recall improves both scores for every model, with Workspace gains ranging from +2.19 to +5.61 and Preference gains from +5.55 to +9.61. The mechanism it identifies is actionability: memory that preserves concrete paths, constraints, artifacts, and procedures lets agents reallocate effort from exploration to execution, while shorter summaries preserve conclusions but lose the working details needed to continue.

Load-bearing premise

The paper assumes that GPT-5.5 rubric grading is a trustworthy measure of workspace quality and participant-preference alignment; if that grading does not match human judgment, the reported memory gains may not reflect real usefulness.

Editorial extensions

If this is right

  • Memory systems should be scored by whether they improve final workspace quality and preference alignment, not by retrieval metrics alone.
  • Curated in-context experience, meaning full task traces rather than distilled summaries, should be the stronger default for workflow-continuation tasks.
  • Models differ in how reliably they use recalled information, so a memory system that works for one base model may not transfer to another.
  • Because richer recall raises the rate of memory-induced problems, memory design needs to balance actionability against robustness to misleading recall.
  • The 568-task, 14-participant benchmark provides a reproducible testbed for comparing agent memory components and base models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is a fixed-token-budget comparison of summary versus in-context memory, which the paper itself flags as future work; if compact summaries perform on par when budgets match, the actionability advantage would shrink.
  • The diagnostic metrics (solvability, continuity, memory-induced rate) could be combined into a single memory-utility index that predicts downstream score gains, letting developers tune memory without running full Docker evaluations.
  • The benchmark's GPT-5.5 grading introduces a potential confound: observed memory gains may partly reflect that the grader prefers outputs resembling the reference trajectory, so human-calibrated rubrics would strengthen the causal claim.
  • The trajectory-alignment procedure may hide exactly the kind of environment drift that long-term memory is meant to cope with, so real-world gains could differ from benchmark gains.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper presents ContextWeave, a benchmark that reconstructs multi-month, privacy-anonymized office workflows of 14 participants into 1,005 executable Docker tasks (568 core evaluation tasks). For each target task, an agent is run with or without access to prior task history, and downstream outcomes are scored by GPT-5.5 using a Workspace Score and a Preference Score, alongside four diagnostics (relevance, continuity, solvability, hallucination robustness). Experiments with six memory components under a fixed execution model and with five base models under mem0 report that recall improves both outcome scores for every tested model, with the strongest configuration raising the Workspace Score from 68.08 to 78.20 and the Preference Score from 41.50 to 70.60. The paper further argues that in-context experience memory outperforms compact summary memory while being more susceptible to misleading recall.

Significance. If the reported effects survive validation, ContextWeave is a valuable community resource: it provides real temporal dependencies, executable environments, a controlled paired protocol, and multi-faceted diagnostics, and the public repository is a clear strength. The no-recall baseline anchors the comparison, and the fact that conditions differ only by injected context is a sound experimental skeleton. The main risk is that every headline number is an LLM-judged point estimate with limited human calibration, so the benchmark's central quantitative claims are not yet established.

major comments (5)
  1. [Section 4.5, 5.2.1, 6, B.7.1] All downstream scores are GPT-5.5 rubric outputs, and the Workspace grader is explicitly instructed to treat the reference image as 'roughly around the 80-point level' and to derive task requirements from it. The Limitations section concedes that rubric calibration is ongoing and human validation is only a coarse-grained sampled subset. Since every delta in Tables 1 and 2 is computed from these scores, the central claim that memory improves downstream performance needs per-rubric human-model agreement (e.g., Cohen's kappa or ICC), bootstrap confidence intervals for the headline scores, and a human pairwise preference validation on a random sample of with-recall vs. without-recall trajectories.
  2. [Sections 4.3-4.4 vs. B.7.1] With-recall conditions consume trajectories that were reconstructed and rewritten backward from the reference document diff via 'result-aligned trajectory' rewriting, while the Workspace grader derives its requirements from the same reference-derived workspace. This is not a definitional circularity because the no-recall condition anchors the comparison, but it means the measured gain could partly reflect self-consistency between an LLM judge and an LLM-generated reference rather than quality that transfers to human users. Please test this by (i) rerunning grading with a different judge model and (ii) comparing model-based deltas to human expert ratings on a held-out sample.
  3. [Tables 1 and 2] No uncertainty quantification is reported. The tables give single point estimates with no per-task variance, confidence intervals, or paired significance tests, so the claim that recall improves every tested base model cannot be assessed; for example, the Workspace deltas in Table 2 range from +2.19 to +5.61 and may be within noise. Please report bootstrap confidence intervals and paired tests across the 568 core tasks for all downstream deltas.
  4. [Section 5.2.3 and abstract] The claim that in-context experience memory is more effective than compact summaries is confounded because the memory components differ in retrieval strategy and context length, as the paper itself acknowledges only in the last sentence of Section 5.2.3. The abstract and Section 6 present this comparison as a finding. Please either soften the wording or add a controlled comparison at a fixed token budget.
  5. [Section 4.4] The benchmark's realism claim rests on the unexamined assumption that result-aligned trajectory rewriting preserves the realistic memory signal of prior experience. Because all memory components consume these rewritten trajectories, this assumption is load-bearing for external validity. Please provide evidence that rewritten trajectories remain natural and representative, for example by measuring divergence between the original and rewritten trajectories and by human reading of a sample.
minor comments (5)
  1. [Section 6 and Appendix B] The Limitations section repeatedly refers to 'MemoryBench' and the appendix prompts use 'Membench' instead of 'ContextWeave'; please make the naming consistent throughout.
  2. [Figure 2 caption] The caption says the temporal panels use 1,007 task IDs, while the paper reports 1,005 total tasks and 568 core tasks; please reconcile these counts.
  3. [Section 4.5] The Preference Score normalization is described only as 'weighted rubric scores normalized to 0-100'; please specify the weighting and normalization formula, since this metric drives a headline claim.
  4. [Tables 1 and 2] The Solvability metric is reported as 'Problem Solvability Rate' in Table 1 and as 'Recall-Solvable Problems' in Table 2; the relationship between these two quantities should be stated explicitly.
  5. [Section 5.2.1] The cosine-similarity threshold of 0.65 for relevance diagnostics is presented as a fixed choice; please justify it and report sensitivity to the threshold.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the headline memory gains are measured against an external no-recall baseline with independent memory components, and the target-task reference is not fed into the with-recall condition.

full rationale

The paper's central claim is that recalled prior experience improves downstream Workspace and Preference scores. The comparison is anchored by an explicit no-recall control: 'We compare a no-recall baseline, in which no historical context is injected, with with-recall conditions augmented by different memory components,' and all conditions are said to 'use the same task instruction, initial workspace, model, and tool permissions; they differ only in the recalled context provided to the agent.' The memory components (mem0, langmem, a-mem, etc.) are external systems with their own retrieval and representation logic, so the with-recall scores are not derived from the metric definitions by construction. The Workspace Score references the 'validated post-task workspace obtained during task reconstruction,' and the Preference Score references participant-specific rubrics; neither metric uses the memory input as its definition, and the target task's own reference output is not included in the historical experience supplied to the agent. A with-recall agent must still convert recalled prior trajectories into concrete workspace edits and document writes, so the reported gains (68.08 to 78.20 Workspace; 41.50 to 70.60 Preference) are empirical outcomes rather than identities. The reconstruction pipeline does rewrite prior trajectories so that 'applying this trajectory produces the same final workspace changes as recorded in the worklog,' but those aligned trajectories describe earlier tasks, not the current evaluation target. The GPT-5.5 rubric grading, with the Workspace grader instructed to treat the reference image as 'the solid reference baseline ... roughly around the 80-point level,' is a validity and calibration concern, and the paper itself concedes that 'rubric calibration is still ongoing' with only 'coarse-grained human judgments for a sampled subset of tasks.' That limitation bears on whether the scores reflect human judgment, not on whether the scores are definitionally equal to their inputs. No load-bearing self-citation, uniqueness theorem, or ansatz-smuggling chain appears; the 'MemoryBench' naming in the Limitations section is a typographical inconsistency, not a circular step. The derivation chain is therefore self-contained with respect to circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No invented physical entities appear. The central assumptions are about dataset representativeness, reconstruction fidelity, LLM grading validity, and the faithfulness of trajectory rewriting; all are acknowledged or partially acknowledged in the paper itself.

free parameters (2)
  • Cosine similarity threshold for relevance diagnostics = 0.65
    Hand-set threshold for embedding-based recall/precision in Section 5.2.1; it affects diagnostic metrics, not the headline Workspace or Preference scores.
  • Workspace Score normalization anchors = 60 minimum completion, 80 reference-level
    Designer-chosen anchors in Section 4.5 that define the score scale; they shape interpretation but not the relative memory versus no-memory comparison.
assumptions (4)
  • domain assumption Document-editing event logs from one open-source project are a valid stand-in for realistic longitudinal office workflows.
    Used throughout Section 4.1 to ground the benchmark; the paper itself notes limited domains and participants in its Limitations, so this assumption caps generalization.
  • domain assumption Privacy-preserving anonymization and reconstruction preserve the task structure, temporal dependencies, and relevant state of the original workflows.
    Sections 4.2 and 4.3 rely on this to claim real-world validity; no external verification of reconstruction fidelity is provided.
  • domain assumption GPT-5.5-based rubric grading is a reliable proxy for human judgment about workspace quality and user preferences.
    Sections 4.5 and 5.2.1 use LLM judges for all metrics; the Limitations state rubric calibration is ongoing and human validation is coarse-grained and sampled.
  • ad hoc to paper Result-aligned trajectory rewriting preserves the realistic memory signal of prior experience.
    Section 4.4 rewrites agent trajectories to match reference workspace outcomes; memory components ingest these rewritten traces, so the historical experience is partly constructed by the benchmark pipeline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ContextWeave: A Real-World Workflow Benchmark." pith.science (2026). https://pith.science/paper/27MIGULY

@misc{pith2026260804830,
  author       = {Pith},
  title        = {Pith review of: ContextWeave: A Real-World Workflow Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/27MIGULY}},
  note         = {Machine review of arXiv:2608.04830}
}
read the original abstract

Memory is essential as language agents move from isolated tasks to long-horizon, stateful workflows, yet existing evaluations often reduce it to retrieval or question answering. We introduce ContextWeave, a longitudinal benchmark that evaluates whether recalled experience improves downstream agent performance in realistic office-work streams. ContextWeave reconstructs privacy-preserved, multi-month workflows of 14 participants into 1,005 executable tasks, including 568 core evaluation tasks, with instructions, containerized environments, trajectories, and task-specific rubrics. It measures workspace quality and alignment with participant-specific preferences, complemented by diagnostics of relevance, continuity, solvability, and robustness to misleading recall. Across six memory components under a fixed model, the strongest configuration raises Workspace Score from 68.08 to 78.20 and Preference Score from 41.50 to 70.60. With a fixed memory component, recall improves both outcomes for all five tested base models, although gains vary substantially. Our analysis shows that actionable, experience-rich memory supports workflow continuation and reduces redundant exploration more effectively than compact summaries, while it can also be more susceptible to misleading recall. These findings motivate memory systems that optimize not only retrieval relevance but also reliable use during execution.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

96 extracted references · 63 canonical work pages

  1. [1]

    Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. InAdvances in Neural Information Processing Systems, volume 35, 2022. URLhttp...

    2022

  2. [2]

    Code Llama: Open foundation models for code.CoRR, abs/2308.12950,2023

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thoma...

    arXiv doi:10.48550/arxiv.2308.12950 2023

  3. [3]

    Jiang, Jia Deng, Stella Biderman, and Sean Welleck

    Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id= 4WnqRR915j

    2024

  4. [4]

    DeepSeekMath: Pushingthelimitsofmathematicalreasoninginopenlanguagemodels.CoRR, abs/2402.03300,2024

    ZhihongShao,PeiyiWang,QihaoZhu,RunxinXu,JunxiaoSong,XiaoBi,HaoweiZhang,MingchuanZhang,Y.K.Li, Y.Wu,andDayaGuo. DeepSeekMath: Pushingthelimitsofmathematicalreasoninginopenlanguagemodels.CoRR, abs/2402.03300,2024. doi: 10.48550/ARXIV.2402.03300. URLhttps://doi.org/10.48550/arXiv.2402.03300

    arXiv doi:10.48550/arxiv.2402.03300 2024

  5. [5]

    GAIA: A benchmark for general AI assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: A benchmark for general AI assistants. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps: //openreview.net/forum?id=fibxvahvs3

    2024

  6. [6]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk,DanielFried,UriAlon,andGrahamNeubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk,DanielFried,UriAlon,andGrahamNeubig. WebArena: Arealisticwebenvironmentforbuildingautonomous agents. InThe Twelfth International Conference on Learning Representations, ICLR 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=oKn9c6ytLx

    2024

  7. [7]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=VTF8yNQM66

    2024

  8. [8]

    RULER: What’s the real context size of your long-context language models?CoRR, abs/2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models?CoRR, abs/2404.06654, 2024. doi: 10.48550/arXiv.2404.06654. URLhttps://doi.org/10.48550/arXiv.2404.06654

    arXiv doi:10.48550/arxiv.2404.06654 2024

Show all 96 references
  1. [9]

    LongBench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. InProceedings of the 62nd Annual Meeting ...

    2024 doi

  2. [10]

    Evaluating very long-term conversational memory of LLM agents

    Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of LLM agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p...

    2024 doi

  3. [11]

    LongMemEval: Benchmarking chat assistants on long-term interactive memory

    Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. LongMemEval: Benchmarking chat assistants on long-term interactive memory. InThe Thirteenth International Conference on Learning Representations, ICLR 2025. OpenReview.net, 2025. URLhttps://openreview.net...

    2025

  4. [12]

    Reflex- ion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflex- ion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems, volume 36, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/h...

    2023

  5. [13]

    ExpeL: LLM agents are experiential learners

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632–19642,

  6. [14]

    Learning and memory.Proceedings of the National Academy of Sciences of the United States of America, 97(23):12403–12404, 2000

    Hideyuki Okano, Tomoo Hirano, and Evan Balaban. Learning and memory.Proceedings of the National Academy of Sciences of the United States of America, 97(23):12403–12404, 2000. doi: 10.1073/pnas.210381897

    doi

  7. [15]

    Kandel, Yadin Dudai, and Mark R

    Eric R. Kandel, Yadin Dudai, and Mark R. Mayford. The Molecular and Systems Biology of Memory.Cell, 157(1): 163–186, 2014. doi: 10.1016/j.cell.2014.03.001

    2014 doi

  8. [16]

    Artificial Hippocampus Networks for Efficient Long-Context Modeling.CoRR, abs/2510.07318, 2025

    Yunhao Fang, Weihao Yu, Shu Zhong, Qinghao Ye, Xuehan Xiong, and Lai Wei. Artificial Hippocampus Networks for Efficient Long-Context Modeling.CoRR, abs/2510.07318, 2025. doi: 10.48550/arXiv.2510.07318. URLhttps: //doi.org/10.48550/arXiv.2510.07318

    2025 doi

  9. [17]

    DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks.CoRR, abs/2607.07946, 2026

    Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks.CoRR, abs/2607.07946, 2026. doi: 10.48550/arXiv.2607.07946. URL https://doi.org/10.48550/arXiv.2607.07946

    2026 doi arXiv

  10. [18]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 ConferenceonEmpiricalMethodsinNaturalLanguageProcessing,pa...

    2018

  11. [19]

    ALFWorld: Aligning text and embodied environments for interactive learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. InInternational Conference on Learning Representations, 2021. URLhttps://openreview.net/forum?id=...

    2021

  12. [20]

    Memento: Fine-tuning LLM agents without fine-tuning LLMs.CoRR, abs/2508.16153,

    Huichi Zhou, Yihang Chen, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang, and Jun Wang. Memento: Fine-tuning LLM agents without fine-tuning LLMs.CoRR, abs/2508.16153,

    arXiv

  13. [21]

    Memento-Skills: Let agents design agents.CoRR, abs/2603.18743, 2026

    Huichi Zhou, Siyuan Guo, Anjie Liu, Zhongwei Yu, Ziqin Gong, Bowen Zhao, Zhixun Chen, Menglong Zhang, Yihang Chen, Jinsong Li, Runyu Yang, Qiangbin Liu, Xinlei Yu, Jianmin Zhou, Na Wang, Chunyang Sun, and Jun Wang. Memento-Skills: Let agents design agents.CoRR, abs/2603.18743,...

    2026 doi

  14. [22]

    Memoryarena: Benchmarking agent memory in interdependent multi-session agentic tasks.arXiv preprint arXiv:2602.16313, 2026

    Zexue He, Yu Wang, Churan Zhi, Yuanzhe Hu, Tzu-Ping Chen, Lang Yin, Ze Chen, Tong Arthur Wu, Siru Ouyang, Zihan Wang, Jiaxin Pei, Julian McAuley, Yejin Choi, and Alex Pentland. Memoryarena: Benchmarking agent memory in interdependent multi-session agentic tasks.arXiv preprint ...

    2026

  15. [23]

    MemBench: Towards more comprehensive evaluation on the memory of LLM-based agents

    Haoran Tan, Zeyu Zhang, Chen Ma, Xu Chen, Quanyu Dai, and Zhenhua Dong. MemBench: Towards more comprehensive evaluation on the memory of LLM-based agents. InFindings of the Association for Computational Linguistics: ACL 2025, pages 19336–19352, Vienna, Austria, 2025. Associati...

    2025 doi

  16. [24]

    Evaluating memory in LLM agents via incremental multi-turn interactions

    Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in LLM agents via incremental multi-turn interactions. InThe Fourteenth International Conference on Learning Representations, 2026. URLhttps://openreview. net/forum?id=DT7JyQC3MR

    2026

  17. [25]

    MemoryBench: A benchmark for memory and continual learning in LLM systems.arXiv preprint arXiv:2510.17281, 2025

    Qingyao Ai, Yichen Tang, Changyue Wang, Jianming Long, Weihang Su, and Yiqun Liu. MemoryBench: A benchmark for memory and continual learning in LLM systems.arXiv preprint arXiv:2510.17281, 2025. URL https://arxiv.org/abs/2510.17281

    2025 arXiv

  18. [26]

    MemoryBank: Enhancinglargelanguagemodels with long-term memory

    WanjunZhong,LianghongGuo,QiqiGao,HeYe,andYanlinWang. MemoryBank: Enhancinglargelanguagemodels with long-term memory. InThirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, pages 19724–19731. AAAI Press, 2024. doi: 10.1609/AAAI.V38I17.29946. URLhttps://doi.org/1...

    2024 doi

  19. [27]

    Mem0: Building production-ready AI agents with scalable long-term memory

    Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready AI agents with scalable long-term memory. InECAI 2025 - 28th European Conference on Artificial Intelligence, volume 16 413 ofFrontiers in Artificial Intelligence and A...

    2025 doi

  20. [28]

    LangMem: Modularmemoryforagenticsystems,2025

    LangChain. LangMem: Modularmemoryforagenticsystems,2025. URL https://github.com/langchain-ai/ langmem. Software repository

    2025

  21. [29]

    A- MEM: Agentic memory for LLM agents

    Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A- MEM: Agentic memory for LLM agents. InAdvances in Neural Information Processing Sys- tems, volume 38, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/ 19909c36f51abc4856b4560aff...

    2025

  22. [30]

    MemOS:AmemoryOSforAIsystem.CoRR,abs/2507.03724,

    Zhiyu Li, Shichao Song, Chenyang Xi, Hanyu Wang, Chen Tang, Simin Niu, Ding Chen, Jiawei Yang, Chunyu Li, Qingchen Yu, Jihao Zhao, Yezhaohui Wang, Peng Liu, Zehao Lin, Pengyuan Wang, Jiahao Huo, Tianyi Chen, Kai Chen,KehangLi,ZhenTao,JunpengRen,HuayiLai,HaoWu,BoTang,ZhenrenWan...

    arXiv

  23. [31]

    Prompt design:

    OpenAI. Codex CLI: A coding agent for the terminal. Computer software, May 2026. URL https: //github.com/openai/codex/releases/tag/rust-v0.134.0. Version 0.134.0, Git tag rust-v0.134.0, commit a75c443fdb64db48c3cf4bdb247c7ee52c0144c9. 17 Appendix A Data Statistics We report th...

    2026

  24. [34]

    URLhttps://doi.org/10.48550/arXiv.2507.03724

    doi: 10.48550/ARXIV.2507.03724. URLhttps://doi.org/10.48550/arXiv.2507.03724

    doi arXiv

  25. [36]

    Inspect each candidate task together with its document snapshots, local markdown diff, temporal context, and neighboring tasks

  26. [37]

    The target unit is a coherent and relatively atomic work objective that could naturally be delegated to an agent

    Verify whether the candidate should remain separate or be merged with adjacent tasks. The target unit is a coherent and relatively atomic work objective that could naturally be delegated to an agent

  27. [38]

    Mark whether the task belongs to the participant’s main workflow

  28. [39]

    Write a concise task description that captures the work objective, boundary, and expected outcome

  29. [40]

    Write a metric description that identifies the evaluation focus for the task

  30. [41]

    Assign applicable agentic tags, such as coding, planning, debugging, data analysis, writing, tool use, review, or deep research

  31. [42]

    Select applicable participant-level preference metrics

  32. [43]

    annotation_status

    Markwhetheradditionalfilesorintermediateresourcesareneeded. Ifso,describetherequiredresource traits, format, granularity, and content pattern. Instruction Generation Prompt { "annotation_status": "", "task_description": "", "metric_description": "", "selected_preference_metric...

  33. [44]

    ‘worklog_observation‘: What definite content, process records, or pending signals are in the worklog

  34. [45]

    This must be open text; do not use type enumeration

    ‘expected_agent_action‘: What the agent should actually perform, deduced from the worklog. This must be open text; do not use type enumeration

  35. [46]

    ‘instruction_gap‘: The difference between the current ‘agent_task‘ and ‘expected_agent_action‘, such as weakening, offsetting, changing the task, or auditing without execution

  36. [47]

    ‘instruction_problem‘: Where the current instruction is wrong; point out the task structure that was incorrectly rewritten, rather than criticizing the specific execution result

  37. [48]

    ‘preserve_work_semantics‘: Which work themes, task stages, or action relationships should be preserved after the correction

  38. [49]

    ‘avoid_instruction_patterns‘: Which instruction patterns should be avoided after the revision

  39. [50]

    - ‘suggested_reconstruct_guidance‘ should be a natural language compressed version of ‘revise_note‘, containing three types of information:

    ‘rewrite_strategy‘: How to rewrite the instruction into a new, executable one. - ‘suggested_reconstruct_guidance‘ should be a natural language compressed version of ‘revise_note‘, containing three types of information:

  40. [51]

    What is ‘expected_agent_action‘?

  41. [52]

    What are the differences between the current instruction and ‘expected_agent_action‘?

  42. [53]

    X must be retained as successful / Y as verified

    How should it be rewritten, and which instruction patterns should be avoided? -Neither‘revise_note‘nor‘suggested_reconstruct_guidance‘shouldhardcodespecificstatusconclusions in the worklog as mandatory inherited facts; avoid writing "X must be retained as successful / Y as ver...

  43. [54]

    The rewritten trajectory must execute the task from scratch like the real agent, without appearing to know the final document answer in advance

  44. [55]

    There must be a natural target file write action at the end, and the written content must be verbatim equal to the after_markdown

  45. [56]

    based on the real diff,

    Before writing the target file, sufficient evidence must have naturally generated in the trajectory; the workspace must also have corresponding supporting files or original changed files. Important requirements: - Do not use phrases that reveal the benchmark, such as "based on...

  46. [57]

    - Identify the task type: research, analysis, code/config/environment work, prompt/rule/pipeline work, data work, or mixed

    Read the task context and reference document change to understand the actual task. - Identify the task type: research, analysis, code/config/environment work, prompt/rule/pipeline work, data work, or mixed. - Identify the core deliverable and the quality signal the workspace s...

  47. [58]

    - Treat it as the solid reference baseline for this task, roughly around the 80-point level

    Inspect ‘bundle.docker.reference_image_ref‘ first. - Treat it as the solid reference baseline for this task, roughly around the 80-point level. - It is not perfect and does not need to be copied. - Extract what it does well as content/task-quality requirements, not as path or ...

  48. [59]

    - ‘minimum_requirements‘: what a barely passing workspace must accomplish for this task

    Write ‘./output/criteria.json‘ before inspecting the candidate. - ‘minimum_requirements‘: what a barely passing workspace must accomplish for this task. - ‘reference_requirements‘: what the reference workspace accomplishes at a solid level. - ‘better_requirements‘: what would ...

  49. [60]

    - Find what the candidate actually produced or changed

    Inspect ‘bundle.docker.candidate_image_ref‘. - Find what the candidate actually produced or changed. - File paths are evidence only; they do not define success unless the task specifically requires a path

  50. [61]

    - Compare content and task quality, not whether the same file name exists

    Directly compare the candidate workspace against the reference workspace. - Compare content and task quality, not whether the same file name exists. - Ask whether the candidate is below minimum, near minimum, weaker than reference, close to reference, or better than reference

  51. [62]

    task": "short task summary

    Score the candidate using all three anchors: task requirements, the minimum requirements, and the direct reference comparison. Docker inspection: - Use Docker commands to inspect ‘/workspace‘ in both images. - Read relevant files, not just filenames. - Inspect code, scripts, c...

  52. [63]

    Identify the participant’s primary document: the path of the main, persistent worklog or work record

  53. [64]

    preserve the Three.js demo link

    Derive approximately 30 global preferences from all reference Markdown changes. Each preference must state a clear direction, not merely name a dimension. Input files: - ‘./input/reference_subtasks.json‘ - Lists all subtasks. - Each subtask contains task information, root pref...

  54. [65]

    - If GT updates the explicit main worklog, prefer the candidate that writes that main worklog

    Durable record / target behavior - Prefer the candidate that updates the same durable record type as GT. - If GT updates the explicit main worklog, prefer the candidate that writes that main worklog. A candidate that only writes a standalone artifact can still get content cred...

  55. [66]

    - Prefer mixed Chinese-English technical note-taking, short bullets, 49 lightweight status wording, and local worklog rhythm when GT uses them

    Worklog language style - Prefer concise rough worklog notes when GT is concise. - Prefer mixed Chinese-English technical note-taking, short bullets, 49 lightweight status wording, and local worklog rhythm when GT uses them. - Penalize converting a short daily/worklog note into...

  56. [67]

    still checking

    Judgment/status/uncertainty style - Prefer candidates that preserve tentative wording, TODO/pending/running status, blocked/failed checks, "still checking" language, rough comparative judgments, and non-overconfident conclusions when GT uses them. - Penalize unsupported comple...

  57. [68]

    - If GT is a short worklog continuation, a very long standalone document may be worse even if technically useful

    Proportional detail and scope control - Prefer the candidate whose amount of detail matches the reference change. - If GT is a short worklog continuation, a very long standalone document may be worse even if technically useful. - If GT is an artifact/spec, a longer artifact ma...

  58. [69]

    module":

    Root metric support - Use selected root preference metrics as high-level constraints. Prefer the candidate that better supports those metrics in a way visible from the message log. - If selected root metrics are empty, use global person-level preferences instead of blocking. -...

  59. [70]

    Read both task cards to understand each objective and deliverable

  60. [71]

    Read both message logs to determine what the agent actually inspected, ran, changed, verified, and summarized

  61. [72]

    Ask whether remembering ‘previous_task‘ would materially change the approach, eliminate irrelevant search, enable reuse of an existing artifact, avoid repeated validation, preserve an important boundary, or produce a more accurate conclusion

  62. [73]

    If the answer cannot be stated beyond shared file, path, topic, historical preservation, or formatting continuity, mark the pair not relevant

  63. [74]

    If relevant, explicitly reason through: concrete fact from the previous task -> concrete need in the target task -> effect on execution or output

  64. [75]

    Writeoneconcise‘evidence‘statementdescribingwhichfactfrom‘previous_task‘affects‘target_task‘

  65. [76]

    Keep it short but understandable; do not repeat background

    In ‘memory_needed‘, state the essential information that must be remembered. Keep it short but understandable; do not repeat background

  66. [77]

    If relevant, extract supporting excerpts from the original ‘previous_task.message_log‘

  67. [78]

    theuserisworkingonaproject,

    If not relevant, set ‘evidence‘ and ‘memory_needed‘ to empty strings and ‘excerpts‘ to an empty array. Excerpt requirements: - Extract excerpts only from ‘previous_task.message_log‘, never from ‘target_task‘. - Preserve the source wording as closely as possible. Do not rewrite...

  68. [79]

    Recall is absent, almost entirely unrelated, or actively misleading for the current task

  69. [80]

    Recall has weak/generic relevance; useful anchors are sparse or buried in noise

  70. [81]

    Recall contains some relevant prior context, but important anchors are missing, vague, or mixed with substantial noise

  71. [82]

    Recall contains clearly relevant prior artifacts/decisions/worklog context with manageable noise

  72. [83]

    dimension

    Recall is highly relevant and specific, naming the key prior tasks/artifacts/decisions needed for this subtask. Return exactly this JSON object: { "dimension": "memory_relevance", "category": "context", "score": "integer 1-5", "relevant_prior_tasks": [ {"id": "subtask_xxxxorhi...

  73. [84]

    With_recall is less familiar: follows wrong local context, misses obvious files, or has more avoidable local detours than without_recall

  74. [85]

    Little environment familiarity advantage; both mostly rely on broad scans or with_recall is only slightly more targeted

  75. [86]

    Some familiarity: with_recall uses a few relevant local anchors earlier, but without_recall recovers them with similar effort

  76. [87]

    Clear familiarity: with_recall navigates to relevant project files/artifacts more directly and avoids notable detours

  77. [88]

    dimension

    Strong familiarity: with_recall immediately or near-immediately uses precise local anchors/conven- tions and without_recall shows clear reconstruction cost. Return exactly this JSON object: { "dimension": "environment_familiarity", "category": "ability", "score": "integer 1-5"...

  78. [89]

    Severe memory-induced hallucination or misuse; with_recall is materially harmed

  79. [90]

    Clear risk or some harmful overfitting to irrelevant/stale memory

  80. [91]

    Some uncertainty/noise; with_recall mostly avoids harm but makes weakly grounded historical assumptions

  81. [92]

    Low risk; with_recall is mostly grounded and ignores irrelevant recall appropriately

  82. [93]

    dimension

    Very low risk; with_recall carefully grounds memory use, verifies important claims, and avoids overuse of weak recall. Return exactly this JSON object: { "dimension": "hallucination_risk", "category": "context", "score": "integer 1-5", "risk_level": "low|medium|high", 59 "grou...

  83. [94]

    The recalled memory contains incorrect, outdated, misleading, or ambiguous information relevant to the problem

  84. [95]

    The execution trace shows that the agent actually relied on or adopted that information

  85. [96]

    problem_index

    This reliance caused or materially contributed to the observed problem. Mark it as false when the memory merely contains questionable information that the agent did not use, when the connection is only speculative, or when the problem is a general coding, command, dependency, ...

  86. [2018]

    doi: 10.18653/v1/D18-1259

    Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. URLhttps://aclanthology. org/D18-1259/

    doi

  87. [2024]

    URLhttps://doi.org/10.1609/aaai.v38i17.29936

    doi: 10.1609/AAAI.V38I17.29936. URLhttps://doi.org/10.1609/aaai.v38i17.29936

    doi

  88. [2025]

    URLhttps://doi.org/10.48550/arXiv.2508.16153

    doi: 10.48550/arXiv.2508.16153. URLhttps://doi.org/10.48550/arXiv.2508.16153

    doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.