Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

The paper argues that fixed, hand-designed memory pipelines are the bottleneck for LLM agents, and that a lightweight learned controller can replace them, improving task success while using fewer tokens.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 04:43 UTC pith:BX7L5KF5

load-bearing objection MemCon is a genuinely useful idea—a learned, lightweight memory controller with zero extra LLM calls—but the single-run evaluation doesn't yet support the 'consistent gains' headline. the 4 major comments →

arxiv 2607.13591 v1 pith:BX7L5KF5 submitted 2026-07-15 cs.CL cs.AI

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents

Eric Hanchen Jiang , Zhi Zhang , Yuchen Wu , Levina Li , Dong Liu , Xiao Liang , Rui Sun , Yubei Li
show 6 more authors
Edward Sun Haozheng Luo Zhaolu Kang Aylin Caliskan Kai-Wei Chang Ying Nian Wu
This is my paper
classification cs.CL cs.AI
keywords memory managementLLM agentscontextual banditUCB explorationadaptive retrievalplan injectiononline learningbackend-agnostic wrapper
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's central claim is that static, hand-designed memory pipelines are a bottleneck for LLM agents, because no single retrieval setting works across all stages of learning, all task types, and all states of the memory store. It argues that memory access should be a controlled, learnable process: a lightweight controller that decides when, what, and how much to retrieve, when to inject a distilled plan, when to re-retrieve with a different query, and when to consolidate or forget. The proposed solution wraps any existing memory backend, learns online from binary task success using a tabular contextual bandit with UCB exploration and warm-start priors, and is claimed to converge within tens of tasks, with no pretraining and no extra LLM calls. Across six benchmarks, three agent frameworks, and three LLM backbones, the paper reports consistent accuracy gains (up to 15.2 points) with 5–20% lower token use. If true, agent memory ceases to be a fixed hyperparameter and becomes an adaptive layer any backend can inherit.

Core claim

At the center of the paper is a reframing: instead of calling a fixed retrieve routine with hard-coded top-k and hop depth, the agent treats each memory operation as an action in a small Markov decision process. The state summarizes task progress (goal type, step phase, stuck flag, objects held, locations visited) and memory status (size, plan availability, learning phase), discretized to a few hundred keys. The action set ranges from shallow to deep retrieval, plan injection, re-retrieval with alternative queries, consolidation, forgetting, and no-op. The controller learns a tabular UCB policy online, updating Q-values by reverse-discounted Monte-Carlo credit from the terminal success/failu

What carries the argument

Memory MDP: the central object. It is a formulation of memory management as a sequential decision problem, with a discretized state key representing task and memory status, a finite action space of (memory operation, parameters) pairs, and a terminal reward shaped by success plus an efficiency bonus. The learner is a tabular contextual bandit with UCB exploration, warm-start priors, and reverse-discounted Monte-Carlo updates. The backend-agnostic wrapper makes the controller applicable to any memory system that exposes retrieve/store endpoints, with consolidation and forget silently no-oping when unsupported.

Load-bearing premise

The gains and the formal convergence bound depend on the assumption that, after discretizing the state, each memory decision's reward distribution is stable over time and independent across episodes; in a real deployment the memory store grows, consolidates, and is forgotten, so that stability is not guaranteed.

What would settle it

Log per-state, per-action success rates across a long task stream with an intentional mid-stream change of task distribution (for example, household tasks followed by science tasks). If the same state–action pair's reward drifts systematically with memory size or task mix, the stationarity assumption behind the O(log T) regret and 'tens of tasks' sample-complexity claim is violated, and the reliability of the learned policy under drift is not established.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any memory backend—flat vector store, skill library, summarization memory, latent-token memory, or graph memory—can inherit adaptive control by being wrapped, with no changes to its internal storage.
  • Memory control costs no extra LLM calls: decisions are table lookups, so the adaptive layer adds near-zero inference latency and no token overhead.
  • The policy learns from a single binary success/failure signal per task, so it requires no human labels, no pretraining, and no reward engineering beyond end-of-task feedback.
  • Success gains and token savings co-occur: the policy learns to suppress retrieval when memory is sparse or irrelevant and to substitute plan injection for expensive nearest-neighbor lookup when a template exists.
  • The largest relative gains appear under stronger LLMs, where the binding constraint is whether the right experience surfaces at the right time; this suggests memory control becomes more valuable as model capability grows.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The same control layer could be transplanted to retrieval-augmented generation over evolving corpora, where the actions become query formulation, chunk count, and re-query decisions rather than memory-store operations.
  • Editorial inference: Because the reward includes an efficiency weight, the learned policy implicitly encodes a speed–accuracy trade-off; exposing that weight as a user dial would let deployments choose how token-frugal the agent should be.
  • Editorial inference: The convergence guarantee rests on per-state reward stationarity, so a natural stress test is a task stream with a deliberate mid-stream distribution shift; if gains erode exactly at the shift, a non-stationary bandit or drift detector would be the next step.
  • Editorial inference: The paper's single deployment run per cell leaves run-to-run variance unmeasured; repeating the full stream several times would give confidence intervals and test whether the 'tens of tasks' convergence is stable across seeds.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MEMCON, a lightweight learned controller that wraps any LLM-agent memory backend and decides when/how much to retrieve, when to inject a distilled plan, when to re-retrieve, and when to consolidate/forget. Memory operations are cast as a Memory MDP with a small discretized state space, and the policy is a tabular contextual bandit with UCB exploration, warm-start priors, reverse-discounted terminal reward credit, and no extra LLM calls. The authors evaluate on 6 benchmarks, 3 agent frameworks, and 3 LLM backbones against up to 9 memory baselines, report accuracy and token-cost numbers, provide a component ablation, and give a theoretical regret/convergence analysis in Appendix D.

Significance. If the empirical claims held, this would be a practically valuable contribution: a backend-agnostic, zero-pretraining, zero-extra-LLM-call controller that improves task success while reducing token use would be an attractive plug-in for LLM agents. The paper has real strengths: it releases code, provides detailed hyperparameter and prompt appendices, includes a component ablation that isolates the learned controller from hand-added augmentations, and is transparent about the single-run protocol. The central idea is plausible and worth pursuing. However, the headline claim of 'consistent, framework-/benchmark-/backbone-agnostic accuracy gains' is not statistically supported as presented, and the formal guarantees in Appendix D rest on assumptions that the system itself violates.

major comments (4)
  1. [§4.1, App. A.4, Table 1] The central empirical claim — consistent accuracy gains across 54 cells — rests on a single deployment run per configuration, with no seed averaging and no confidence intervals. Benchmark sizes are 100–200 tasks, so binomial standard errors are roughly 2–5 points at the reported success rates. Several headline differences are within that noise: SciWorld Lobster 38.0 vs 34.0 (n=100, SE≈4.8), TriviaQA 71.5 vs 70.5 (n=200, SE≈2.3), WebWalkerQA 20.6 vs 19.0 (n=200, SE≈2.0), DeepSeek Lobster-ALFWorld 86.7 vs 84.3 (n=134, SE≈3.0). Because the Q-table persists across tasks and learning is online, results also depend on task order; no random orderings or repeated runs are reported. The 'consistent' claim therefore is not statistically supported. I ask for repeated runs with shuffled task orders, or at minimum confidence intervals/bootstrap estimates, and for the 'converges within tens of tasks'
  2. [§4.1, Table 1 vs Tables 3–4] Table 1 (GPT-4.1-mini) does not include LatentMem, although §4.1 lists it among the nine memory baselines and Tables 3–4 include it for Sonnet-4 and DeepSeek-V3.2. LatentMem is often the strongest fixed-pipeline baseline in those tables (e.g., Sonnet-4 Lobster-ScienceWorld 82.0 vs MEMCON 81.6; DeepSeek GAIA 33.1 vs 32.8). Excluding it from the headline table changes the baseline set, alters the 'top S/A' counts, and prevents a fair cross-backbone comparison. Please add LatentMem to Table 1 or justify its omission.
  3. [Appendix D (Assumption D.3, Theorem D.8, Remark D.16)] The regret and convergence guarantees require Assumption D.3: the reward distribution for each (φ,a) is time-invariant and draws are conditionally independent across episodes. This is violated by the design itself: the memory backend accumulates, consolidates, and forgets content over the task stream, so D_{φ,a} drifts. Remark D.16 explicitly chooses a constant step size to 'track slowly drifting reward distributions', which forfeits the stationarity needed for Lemma D.7, Theorem D.8, and Theorem D.13. In addition, the same terminal reward is used for every (φ_j,a_j) in an episode with a discount exponent depending on |ep| and j, so draws are not conditionally independent given (φ,a). The proof of Theorem D.8 also switches between c=W/√2 and c=W√2 while the implementation uses c=1.4; the remark that constants remain valid 'up to a re-choice of the universal pre-factor' is not a proof. Mo
  4. [Appendix D.7, Eq. (33)] Proposition D.20's warm-start regret bound is not established. The proof replaces the pull count N_a by the 'effective sample size' m_eff = m + σ²/σ₀² in the UCB1 argument and then subtracts the prior mass from the expected-count bound. This does not follow from the UCB1 three-event decomposition, and Eq. (33) can become negative, which is impossible for a regret upper bound. The Gaussian-prior model is also in tension with the bounded rewards of Definition D.4. Please either provide a correct derivation or remove this proposition.
minor comments (4)
  1. [§4.2, Q1] The reported top-S/A counts for GPT-4.1-mini (4 of 6 under Lobster, 2 under LangGraph, 3 under Agent-FW, 9 of 18 overall) do not match Table 1. Counting ties as top, MEMCON is top in 6/6 (Lobster), 3/6 (LangGraph), and 5/6 (Agent-FW); if ties are excluded, the counts are 5/6, 2/6, and 4/6. Please correct the counts and state the tie-handling convention.
  2. [§4.3, Table 2] The component ablation is referred to as 'Table 4' in the text but is labelled Table 2 in the manuscript. Also, the text says the two augmented operations contribute '+1.5 each' on ALFWorld, whereas Table 2 shows +1.5 for plan injection, +0.8 for goal decomposition, and +0.7 for the final combined step. Please align text and table.
  3. [Table 5 and §4.2 Q2] The token-savings claim needs a precise comparison basis. In Table 5, MEMCON's token cost in several QA cells is above Empty and some baselines (e.g., Lobster-WebWalkerQA 164 vs 153 for Empty; Lobster-GAIA 440 vs 399 for ExpBank). The '5–20% token reduction' appears to refer only to selected strong-memory baselines, not to all cells. Please state the comparison basis and report dispersion or error bars for token costs.
  4. [Abstract and §4.2] The abstract says 'by up to 15.2 points', while §4.2 says '5–30+ point gains over the no-memory baseline'. Neither number is tied to a specific cell in the tables. Please reconcile these figures and identify the source of the 15.2-point margin.

Circularity Check

0 steps flagged

No significant circularity: the learned controller is trained on external task-success rewards and evaluated on external benchmarks; no claim reduces to a fitted parameter, self-citation, or definitional identity.

full rationale

The paper's derivation chain is self-contained rather than circular. The central claim—that a lightweight learned controller improves task success while reducing tokens—is supported by an online contextual-bandit policy whose Q-values are updated from the externally observed task reward r(τ_i) in Eq. (5) and Eq. (25). No equation defines the reported success rates in terms of the policy's own inputs; the reward is task completion from the environment, not a fitted quantity. Hyperparameters (α, γ, c) are selected on a held-out 30-task ALFWorld subset, explicitly 'independent of the evaluation tasks' (Appendix A.4), and the main tables compare MemCon against nine fixed memory baselines on the same benchmarks. The component ablation attributes most of the gain to the learned UCB controller, and the augmented operations are shown to be optional. The theoretical analysis in Appendix D is a standard UCB/Q-learning regret argument; its stationarity assumption (Assumption D.3) is acknowledged as a simplification in Remark D.16, which is a correctness/robustness concern rather than a circular reduction. The paper contains no load-bearing self-citations, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. The single-run, no-seed-averaging evaluation and the task-order dependence of the persistent Q-table are statistical-evidence limitations, not circularity under the stated rules. Therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

No new physical or speculative entities are introduced. The paper's load-bearing assumptions are the standard bandit machinery, the stationarity assumption that contradicts deployment drift, the sufficiency of the coarse state abstraction, and the clean wrapper interface. The free parameters are mostly hand-set priors, reward constants, discretization thresholds, action sizes, and per-backbone tuned policy hyperparameters; the empirical central claim depends on all of them.

free parameters (7)
  • UCB exploration coefficient c = 1.4 / 1.2 / 1.6 per backbone
    Selected per backbone by grid search on a held-out 30-task ALFWorld split (App. A.4); enters Eq. (6) and the regret analysis.
  • Q-learning step size α = 0.15 / 0.12 / 0.18 per backbone
    Selected per backbone by the same grid search; controls update size in Eq. (7)/Eq. (25).
  • Within-episode discount γ = 0.9 / 0.92 / 0.88 per backbone
    Selected per backbone; controls reverse-discounted credit assignment in Eq. (7).
  • Warm-start priors Q0 = RETRIEVE +0.5, PLANINJECT +0.3, RE-RETRIEVE +0.1, CONSOLIDATE 0.0, FORGET −0.1, NOOP −0.2
    Hand-set interpretable priors encode the paper's domain claims; fixed across backbones and not fitted to evaluation tasks.
  • Reward shape constants = r_succ=1.0, r_fail=0.5, λ=0.3, Tmax=30 (interactive)
    Hand-chosen reward shaping in Eq. (5); affects the learned Q-values and the efficiency/success tradeoff.
  • State discretization thresholds = phase bins <8/8-17/≥18; cold≤15 tasks; stuck after 2 repeats; caps hold≤2, visited/3≤4, mem_size/10≤5
    Hand-designed coarse bins in Eq. (8); define the state space on which the whole tabular policy lives.
  • Action-set sizes = 9 actions; top_k∈{1,2,3}, insight_k∈{2,3,5,8}, hop∈{0,1,2}
    Hand-chosen representative configurations in Table 7; the policy can only choose among these, so the action set partly encodes the paper's four regimes.
axioms (6)
  • standard math Standard UCB/Hoeffding/Robbins-Monro machinery (Appendix D)
    The regret and convergence proofs rely on classical bandit and stochastic-approximation results; these are unproved background, not paper-specific postulates.
  • ad hoc to paper Assumption D.3: per-(state, action) reward distribution is time-invariant and draws are conditionally independent across episodes
    Load-bearing for the bandit decomposition and O(log T) regret bound, but the memory backend evolves over tasks; Remark D.16 admits drift, so the assumption is violated in deployment.
  • domain assumption Episode-level binary success plus efficiency bonus is a sufficient learning signal for memory management
    Eq. (5) assumes task success/failure and efficiency cleanly credit or punish memory operations; if success depends mostly on the LLM or environment noise, Q-values mis-credit actions.
  • domain assumption The discretized state key ϕ(s) preserves the information needed to choose optimal memory operations
    Eq. (8) bins goal type, phase, stuck flag, memory size, plan availability, etc.; if the coarse bins discard relevant context, the per-state policy cannot be optimal.
  • domain assumption The wrapper can intercept retrieve/store/maintain hooks without changing the semantics of the inner backend
    The backend-agnostic claim relies on a clean two-method interface and silent no-ops for unsupported hooks; real backends may not degrade gracefully under changed top-k/hop parameters.
  • ad hoc to paper Treating the sequential Memory MDP as a contextual bandit with terminal Monte-Carlo reward is valid
    Actions within an episode affect the LLM's behavior and hence future states, but the analysis ignores within-episode temporal credit except through reverse discounting; the proof handles only the per-state bandit decomposition.

pith-pipeline@v1.3.0-alltime-deepseek · 32398 in / 12606 out tokens · 116797 ms · 2026-08-02T04:43:13.022756+00:00 · methodology

0 comments
read the original abstract

Large Language Model (LLM) agents increasingly rely on external memory systems to accumulate experience across tasks. Yet nearly all existing approaches, from graph-structured memories to reflective insight stores, access memory through fixed, hand-designed heuristics. We argue that this static view of memory is a core bottleneck for agentic learning because optimal memory behavior is fundamentally context-dependent. The early stages of the tasks, benefit from minimal retrieval because memory is sparse; recurring goal types benefit from plan reuse rather than generic nearest-neighbor lookup; stuck agents benefit from re-retrieval with alternative queries; and across long task streams, the memory store itself must be consolidated and pruned to remain useful. We present Memory as a Controlled Process (MemCon), a framework that models memory operations as a Markov Decision Process and learns an online policy that adaptively decides when, what, and how much to retrieve, when to inject a distilled plan, and when to consolidate or forget. MemCon is backend-agnostic: it wraps any existing memory implementation, learns from task-by-task binary feedback with no pretraining and no additional LLM calls, and uses a lightweight tabular contextual bandit with UCB exploration that converges within tens of tasks. Across 6 benchmarks, 3 agent frameworks, and 3 LLM backbones, MemCon consistently outperforms multiple memory baselines by up to 15.2 points in task success while reducing token consumption by 5--20%.

Figures

Figures reproduced from arXiv: 2607.13591 by Aylin Caliskan, Dong Liu, Edward Sun, Eric Hanchen Jiang, Haozheng Luo, Kai-Wei Chang, Levina Li, Rui Sun, Xiao Liang, Ying Nian Wu, Yubei Li, Yuchen Wu, Zhaolu Kang, Zhi Zhang.

Figure 1
Figure 1. Figure 1: Overview of MEMCON. (Left) Task streams from ALFWorld, PDDL, and ScienceWorld are executed through three agent frameworks (Lobster, LangGraph, Agent-FW) sharing one LLM backbone. (Middle) The Memory MDP runs four steps per retrieval: extract a compact state ϕ(s) from task + memory signals; select an action via the UCB policy over Q(ϕ, a); the backend-agnostic wrapper issues retrieval to the inner backend w… view at source ↗
Figure 2
Figure 2. Figure 2: Step-by-step case study on two qualitatively different queries. Left (Case 1, WebWalkerQA): a multi-hop financial query that requires the agent to navigate the web and identify a specific date. In Step 2 the controller is in the cold learning phase (Na(ϕ) = 0), so the UCB bonus is +∞ and the policy explores with a shallow RETRIEVE action. After Step 3 reports is stuck=True, the controller switches to RE-RE… view at source ↗
Figure 3
Figure 3. Figure 3: Token cost vs. task success across memories. Each panel plots, for one (framework, benchmark) pair, every memory baseline as a bubble whose horizontal position is mean S/A (%), vertical position is mean per-task input tokens (in thousands or units), and area is proportional to per-task token cost. MEMCON sits on the bottom-right of all three panels: it achieves the highest or near-highest S/A while using f… view at source ↗
Figure 5
Figure 5. Figure 5: Single-knob sensitivity (efficiency weight λ & action-preset). Left: sweeping the efficiency-bonus weight λ shows that GAIA peaks at the default λ= 0.3; both removing the bonus (λ = 0) and over-weighting it (λ ≥ 0.5) hurt by ∼2 points. Right: swapping the default 9-action preset for hand-tuned alternatives (retrieval heavy, plan first, compact); the default preset is the best on average. Variant ALFWorld G… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. From Cognitive Architectures to Language Agents: A Mechanism-Level Review of Lineage, Convergence, and Migration Gaps

    cs.AI 2026-07 conditional novelty 6.0

    Coding each mechanism for evidence of lineage and implementation depth, the review closes one candidate gap (GraSP) and isolates five residual control bundles for language agents.

Reference graph

Works this paper leans on

79 extracted references · 19 linked inside Pith · cited by 1 Pith paper

  1. [1]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  2. [2]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  3. [3]

    A survey on large language model based autonomous agents.Frontiers of Computer Science, 2024

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. A survey on large language model based autonomous agents.Frontiers of Computer Science, 2024

    2024

  4. [4]

    The rise and potential of large language model based agents: A survey.arXiv preprint arXiv:2309.07864, 2023

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey.arXiv preprint arXiv:2309.07864, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    O’Brien, Carrie J

    Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. InACM Symposium on User Interface Software and Technology (UIST), 2023

    2023

  6. [6]

    Re- flexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  7. [7]

    ExpeL: LLM agents are experiential learners

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. 2024

    2024

  8. [8]

    V oyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended embodied agent with large language models. 2024

    2024

  9. [9]

    G-Memory: Tracing hierarchical memory for multi-agent systems.arXiv preprint arXiv:2506.07398, 2025

    Guibin Zhang, Muxin Yue, Xiangguo Li, Jiaxuan Ran, Rui Song, Ran Cheng, Zheng Wang, and Shirui Pan. G-Memory: Tracing hierarchical memory for multi-agent systems.arXiv preprint arXiv:2506.07398, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Latent memory: Distilling cross-task experience into learnable tokens for language agents.arXiv preprint arXiv:2509.18432, 2025

    Haoran Ou, Jianyu Li, Weiran Chen, Yuxin Liu, Ting Sun, and Dian Yu. Latent memory: Distilling cross-task experience into learnable tokens for language agents.arXiv preprint arXiv:2509.18432, 2025

    arXiv 2025

  11. [11]

    Patil, Ion Stoica, and Joseph E

    Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. MemGPT: Towards llms as operating systems.arXiv preprint arXiv:2310.08560, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Mem0: Building production-ready ai agents with scalable long-term memory.arXiv preprint arXiv:2504.19413, 2025

    Prateek Chhikara, Deshraj Khant, Saket Aryan, Taranjeet Singh, and Deepak Yadav. Mem0: Building production-ready ai agents with scalable long-term memory.arXiv preprint arXiv:2504.19413, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    MemP: Procedural memory from trajectories for language agents.arXiv preprint arXiv:2508.06433, 2025

    Bingbing Wu, Xian Yang, Xinyan Chen, Jiayin Liu, Haowen Li, Yu Su, and Yu Zhang. MemP: Procedural memory from trajectories for language agents.arXiv preprint arXiv:2508.06433, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Finite-time analysis of the multiarmed bandit problem.Machine Learning, 47:235–256, 2002

    Peter Auer, Nicol `o Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem.Machine Learning, 47:235–256, 2002. 10

    2002

  15. [15]

    Schapire

    Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. InInternational Conference on World Wide Web (WWW), 2010

    2010

  16. [16]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. MIT Press, 2nd edition, 2018

    2018

  17. [17]

    MemoryBank: Enhancing large language models with long-term memory

    Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. MemoryBank: Enhancing large language models with long-term memory. 2024

    2024

  18. [18]

    ALFWorld: Aligning text and embodied environments for interactive learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre C ˆot´e, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  19. [19]

    ScienceWorld: Is your agent smarter than a 5th grader? InConference on Empirical Methods in Natural Language Processing (EMNLP), 2022

    Ruoyao Wang, Peter Jansen, Marc-Alexandre Cˆot´e, and Prithviraj Ammanabrolu. ScienceWorld: Is your agent smarter than a 5th grader? InConference on Empirical Methods in Natural Language Processing (EMNLP), 2022

    2022

  20. [20]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. InAnnual Meeting of the Association for Computational Linguistics (ACL), 2017

    2017

  21. [21]

    WebWalker: Benchmarking llms in web traversal.arXiv preprint arXiv:2501.07572, 2025

    Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, and Fei Huang. WebWalker: Benchmarking llms in web traversal.arXiv preprint arXiv:2501.07572, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    GAIA: A benchmark for general ai assistants.arXiv preprint arXiv:2311.12983, 2023

    Gr´egoire Mialon, Cl´ementine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: A benchmark for general ai assistants.arXiv preprint arXiv:2311.12983, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    LangGraph: Controllable cognitive architectures for agentic applications

    Harrison Chase and LangChain Team. LangGraph: Controllable cognitive architectures for agentic applications. 2024

    2024

  24. [24]

    MetaGPT: Meta programming for a multi-agent collaborative framework

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and J¨urgen Schmidhuber. MetaGPT: Meta programming for a multi-agent collaborative framework. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  25. [25]

    ChatDev: Communicative agents for software development

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. ChatDev: Communicative agents for software development. InAnnual Meeting of the Association for Computational Linguistics (ACL), 2024

    2024

  26. [26]

    OAgents: Open-source agents toward reliable experiential learning

    Chen Qian, Yaxi Wang, Zhouyi Wang, Weize Chen, Yufan Zhou, Ran Yu, Chen Zhou, Zhiyuan Liu, and Maosong Sun. OAgents: Open-source agents toward reliable experiential learning. arXiv preprint arXiv:2506.13815, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Leibo, Karl Tuyls, and Thore Graepel

    Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decomposition networks for cooperative multi-agent learning based on team reward. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2018

    2018

  28. [28]

    QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning

    Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. InInternational Conference on Machine Learning (ICML), 2018

    2018

  29. [29]

    Multi-agent actor-critic for mixed cooperative-competitive environments

    Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. InAdvances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  30. [30]

    Counterfactual multi-agent policy gradients

    Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. InAAAI Conference on Artificial Intelligence, 2018. 11

    2018

  31. [31]

    The surprising effectiveness of PPO in cooperative multi-agent games

    Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. 2022

    2022

  32. [32]

    Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. S. Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the StarCraft multi-agent challenge?arXiv preprint arXiv:2011.09533, 2020

    Pith/arXiv arXiv 2011

  33. [33]

    Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft multi-agent challenge. InInternational Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2019

    2019

  34. [34]

    Foerster, Yannis M

    Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2016

    2016

  35. [35]

    Multi-agent reinforcement learning is a sequence modeling problem

    Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  36. [36]

    Multi-agent deep reinforcement learning: A survey.Artificial Intelligence Review, 55:895–943, 2022

    Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: A survey.Artificial Intelligence Review, 55:895–943, 2022

    2022

  37. [37]

    Multi-agent reinforcement learning: A selective overview of theories and algorithms.Handbook of Reinforcement Learning and Control, pages 321–384, 2021

    Kaiqing Zhang, Zhuoran Yang, and Tamer Ba s ¸ar. Multi-agent reinforcement learning: A selective overview of theories and algorithms.Handbook of Reinforcement Learning and Control, pages 321–384, 2021

    2021

  38. [38]

    Pablo Hernandez-Leal, Bilal Kartal, and Matthew E. Taylor. A survey and critique of multiagent deep reinforcement learning.Autonomous Agents and Multi-Agent Systems, 33:750–797, 2019

    2019

  39. [39]

    LLM-augmented multi-agent reinforcement learning for collaborative task allocation.arXiv preprint arXiv:2403.08282, 2024

    Yan Meng, Sen Liu, Li Zhang, and Hang Xu. LLM-augmented multi-agent reinforcement learning for collaborative task allocation.arXiv preprint arXiv:2403.08282, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Bandit algorithms

    Tor Lattimore and Csaba Szepesv ´ari. Bandit algorithms. 2020

    2020

  41. [41]

    Christopher J. C. H. Watkins and Peter Dayan. Q-learning. InMachine Learning, volume 8, pages 279–292, 1992

    1992

  42. [42]

    White, Doug Burger, and Chi Wang

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023

    Pith/arXiv arXiv 2023

  43. [43]

    CAMEL: Communicative agents for “mind” exploration of large language model society

    Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: Communicative agents for “mind” exploration of large language model society. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  44. [44]

    AgentVerse: Facilitating multi-agent collaboration and exploring emergent behaviors

    Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. AgentVerse: Facilitating multi-agent collaboration and exploring emergent behaviors. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  45. [45]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. InInternational Conference on Machine Learning (ICML), 2024

    2024

  46. [46]

    Encouraging divergent thinking in large language mod- els through multi-agent debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language mod- els through multi-agent debate. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2024

    2024

  47. [47]

    Ashley, R ´obert Csord´as, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, et al

    Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R. Ashley, R ´obert Csord´as, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, et al. Mindstorms in natural language-based societies of mind.arXiv preprint arXiv:2305.17066, 2023. 12

    arXiv 2023

  48. [48]

    ChatLLM network: More brains, more intelligence

    Rui Hao, Linmei Hu, Weijian Qi, Qingliu Wu, Yirui Zhang, and Liqiang Nie. ChatLLM network: More brains, more intelligence. 2023

    2023

  49. [49]

    Dynamic LLM-agent network: An LLM-agent collaboration framework with agent team optimization

    Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic LLM-agent network: An LLM-agent collaboration framework with agent team optimization. InConference on Language Modeling (COLM), 2024

    2024

  50. [50]

    GPTSwarm: Language agents as optimizable graphs

    Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and J¨urgen Schmidhuber. GPTSwarm: Language agents as optimizable graphs. 2024

    2024

  51. [51]

    Chawla, Olaf Wiest, and Xiangliang Zhang

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V . Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges.International Joint Conference on Artificial Intelligence (IJCAI), 2024

    2024

  52. [52]

    Multi-agent collaboration: Harnessing the power of intelligent LLM agents.arXiv preprint arXiv:2306.03314, 2023

    Yashar Talebirad and Amirhossein Nadiri. Multi-agent collaboration: Harnessing the power of intelligent LLM agents.arXiv preprint arXiv:2306.03314, 2023

    Pith/arXiv arXiv 2023

  53. [53]

    Mem- LLM: Finetuning llms to use an explicit read-write memory.arXiv preprint arXiv:2404.11672, 2024

    Ali Modarressi, Abdullatif K¨oksal, Ayyoob Imani, Mohsen Fayyaz, and Hinrich Sch¨utze. Mem- LLM: Finetuning llms to use an explicit read-write memory.arXiv preprint arXiv:2404.11672, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    Think-in-memory: Recalling and post-thinking enable llms with long-term memory.arXiv preprint arXiv:2311.08719, 2023

    Lei Liu, Xiaoyan Yang, Yue Shen, Binbin Hu, Zhiqiang Zhang, Jinjie Gu, and Guannan Zhang. Think-in-memory: Recalling and post-thinking enable llms with long-term memory.arXiv preprint arXiv:2311.08719, 2023

    Pith/arXiv arXiv 2023

  55. [55]

    Memory-assisted prompt editing to improve gpt-3 after deployment

    Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve gpt-3 after deployment. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2022

    2022

  56. [56]

    Agent workflow memory

    Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arXiv preprint arXiv:2409.07429, 2024

    Pith/arXiv arXiv 2024

  57. [57]

    Procmem: Learning reusable procedural memory from experience via non-parametric ppo for llm agents.arXiv preprint arXiv:2602.01869, 2025

    Qirui Mi, Zhijian Ma, Mengyue Yang, Haoxuan Li, Yisen Wang, Haifeng Zhang, and Jun Wang. Procmem: Learning reusable procedural memory from experience via non-parametric ppo for llm agents.arXiv preprint arXiv:2602.01869, 2025

    Pith/arXiv arXiv 2025

  58. [58]

    Memskill: Learning and evolving memory skills for self-evolving agents.arXiv preprint arXiv:2602.02474, 2025

    Haozhen Zhang, Quanyu Long, Jianzhu Bao, Tao Feng, Weizhi Zhang, Haodong Yue, and Wenya Wang. Memskill: Learning and evolving memory skills for self-evolving agents.arXiv preprint arXiv:2602.02474, 2025

    Pith/arXiv arXiv 2025

  59. [59]

    HiAgent: Hierarchical working memory management for long-horizon agent tasks.arXiv preprint arXiv:2408.09559, 2024

    Mengkang Hu, Tianxing Chen, Qiguang Zou, Yuheng Liu, Yuxi Sun, Fei Mi, Yitao Liang, Jie Fu, and Yuanjing Mao. HiAgent: Hierarchical working memory management for long-horizon agent tasks.arXiv preprint arXiv:2408.09559, 2024

    Pith/arXiv arXiv 2024

  60. [60]

    CLIN: A continually learning language agent for rapid task adaptation and generalization

    Bodhisattwa Prasad Majumder, Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Callison-Burch, and Peter Clark. CLIN: A continually learning language agent for rapid task adaptation and generalization. InConference on Language Modeling (COLM), 2024

    2024

  61. [61]

    Retrieval-augmented generation for knowledge-intensive NLP tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨aschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. InAdvances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  62. [62]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas O ˘guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2020

    2020

  63. [63]

    Self-RAG: Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. 2024

    2024

  64. [64]

    PlanBench: An extensible benchmark for evaluating large language models on planning and reasoning about change

    Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. PlanBench: An extensible benchmark for evaluating large language models on planning and reasoning about change. 2023. 13

    2023

  65. [65]

    LLM+P: Empowering large language models with optimal planning proficiency.arXiv preprint arXiv:2304.11477, 2023

    Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. LLM+P: Empowering large language models with optimal planning proficiency.arXiv preprint arXiv:2304.11477, 2023

    Pith/arXiv arXiv 2023

  66. [66]

    default"). Additional presets (

    Malte Helmert. The fast downward planning system.Journal of Artificial Intelligence Research (JAIR), 26:191–246, 2006. 14 A Experimental Setup Details This appendix expands the abbreviated description in §4.1 with full benchmark sizes, framework descriptions, baseline mechanisms, and the hyperparameter-search protocol. A.1 Benchmarks Interactive decision-...

    2006

  67. [67]

    Step 3 (bounding Pr[E1(t)∪ E2(t)]).We bound each of Pr[E1(t)] and Pr[E2(t)] using Lemma D.7

    Hence E3(t) cannot hold once armahas been pulledutimes. Step 3 (bounding Pr[E1(t)∪ E2(t)]).We bound each of Pr[E1(t)] and Pr[E2(t)] using Lemma D.7. ForE 2(t), conditioning onN a(t) =sand applying (11) withε=c p lnN(t)/sgives Pr ˆµa(t)> µa +c q lnN(t) s Na(t) =s ≤exp − 2s·c 2 lnN(t)/s W 2 =N(t) −2c2/W 2 . With c=W/ √ 2, 2c2/W 2 = 1, so the per-arm upper-t...

  68. [68]

    The posterior mean afterm≥0observations is ˜µt(ϕ, a) := σ−2 0 Q0(ϕ, a) +m σ−2 ˆµt(ϕ, a) σ−2 0 +m σ−2 ,(32) with posterior variance Var[µ|ˆµ t, m] = σ−2 0 +mσ −2 −1

    on µ(ϕ, a), and suppose observations Gi ∼ N(µ(ϕ, a), σ2) (or, more generally, sub-Gaussian with parameter σ2). The posterior mean afterm≥0observations is ˜µt(ϕ, a) := σ−2 0 Q0(ϕ, a) +m σ−2 ˆµt(ϕ, a) σ−2 0 +m σ−2 ,(32) with posterior variance Var[µ|ˆµ t, m] = σ−2 0 +mσ −2 −1 . Replacing ˆµt(ϕ, a)by ˜µt(ϕ, a)and N (t) a (ϕ) by the effective sample size meff...

  69. [69]

    ∆min. Proof. Step 1 (posterior computation).Given the prior µ(ϕ, a)∼ N(Q0(ϕ, a), σ2

  70. [70]

    effective

    and i.i.d. Gaus- sian observationsG i |µ∼ N(µ, σ2)fori= 1, . . . , m, the likelihood is p(G1, . . . , Gm |µ)∝exp − 1 2σ2 mX i=1 (Gi −µ) 2 ! ∝exp − m 2σ2 (µ−ˆµt)2 , where ˆµt = 1 m P i Gi is the observed empirical mean. Multiplying by the Gaussian prior and completing the square gives a Gaussian posterior with mean and variance E[µ|G 1:m] = σ−2 0 Q0 +mσ −2...

  71. [71]

    in/on" together, never

    put a in/on b. (always write "in/on" together, never "in" or "on" alone.)

  72. [72]

    Thought" or

    think: xxx - You must check carefully whether your output command is consistent with the allowed commands above. Any output not among the listed commands is rejected. ScienceWorld solver (excerpt). 28 ScienceWorld system prompt You are a helpful assistant to do scientific experiments in a text-based environment. In the environment, there are several rooms...

  73. [73]

    Find & take first object→put it at target

  74. [74]

    [Reference: single-put plan] 1.{step 1} 2.{step 2}

    Find & take second object→put it at target. [Reference: single-put plan] 1.{step 1} 2.{step 2} ... The second block reuses the generalized plan for the simpler single-object variant (put) when one has been learned, providing an additional hint to the LLM agent. E.3 Baseline Memory-Module Prompts For full reproducibility we summarize the memory-write/re-ra...

  75. [75]

    Try to summarize the trajectory in no more than 6 sentences

  76. [76]

    Generative / ExperienceBank relevance scorer

    Your response should be a single line of text. Generative / ExperienceBank relevance scorer. Relevance-scoring system/user prompts System: You are an agent designed to score the relevance between two pieces of text. User: You will be given a successful case and an ongoing task. Do not summarize either case; evaluate how relevant and helpful the successful...

  77. [77]

    Phase-based summarization: organize records into logical phases and extract key steps

  78. [78]

    Task relevance: explain what has been completed and what remains

  79. [79]

    If intermediate states are incorrect or irrelevant, filter or correct them to make the summary more accurate

    Clarity and conciseness: precise language, no unnecessary details. If intermediate states are incorrect or irrelevant, filter or correct them to make the summary more accurate. OAgents rule induction. Rule-comparison system prompt for OAgents You are an advanced reasoning agent that derives general rules from examples. You will receive one successful tria...