REVIEW 16 cited by
MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents
read the original abstract
Recent works have highlighted the significance of memory mechanisms in LLM-based agents, which enable them to store observed information and adapt to dynamic environments. However, evaluating their memory capabilities still remains challenges. Previous evaluations are commonly limited by the diversity of memory levels and interactive scenarios. They also lack comprehensive metrics to reflect the memory capabilities from multiple aspects. To address these problems, in this paper, we construct a more comprehensive dataset and benchmark to evaluate the memory capability of LLM-based agents. Our dataset incorporates factual memory and reflective memory as different levels, and proposes participation and observation as various interactive scenarios. Based on our dataset, we present a benchmark, named MemBench, to evaluate the memory capability of LLM-based agents from multiple aspects, including their effectiveness, efficiency, and capacity. To benefit the research community, we release our dataset and project at https://github.com/import-myself/Membench.
Forward citations
Cited by 16 Pith papers
-
Supersede: Diagnosing and Training the Memory-Update Gap in LLM Agents
The supersession gap in LLM agents—failing to use current facts and discard superseded ones—is a distinct failure not fixed by scale or memory size, but improvable via RL training on a new environment.
-
Memory-Induced Tool-Drift in LLM Agents
Biased long-term memories in LLM agents cause measurable deviations in tool parameters across 105 scenarios, seven models, and 608 real tools, persisting under standard memory architectures.
-
Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory
Memory for long-horizon agents should preserve distinctions that affect decisions under a fixed budget, not descriptive features, yielding an exact forgetting boundary and a new online learner DeMem with regret guarantees.
-
NeuroState-Bench: A Human-Calibrated Benchmark for Commitment Integrity in LLM Agent Profiles
NeuroState-Bench is a human-calibrated benchmark with 144 tasks and 306 side-query probes showing that commitment integrity in LLM agent profiles diverges from task success, with 31 of 32 profiles changing rank under ...
-
Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory
Evo-Memory is a new benchmark for self-evolving memory in LLM agents across task streams, with baseline ExpRAG and proposed ReMem method that integrates reasoning, actions, and memory updates for continual improvement.
-
Memory in the Loop: In-Process Retrieval as Extended Working Memory for Language Agents
Store latency, not architecture, gates per-step memory access; in-process ~100 µs stores make memory-in-the-loop feasible and causally reduce redundant agent actions.
-
Supra Cognitive Modes: A Routed Architecture for Agent Memory
A per-query mode router over a shared memory substrate scores 84.87% LoCoMo factoid, 68.61% abstention, 61.49% MemoryAgentBench, 86.00% LongMemEval, with all causal and efficiency claims explicitly withheld.
-
Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventions
No existing agent benchmark protocol replays a fixed temporal change across varied persistent user states while measuring cross-component failure propagation, creating a concrete design gap for personal-agent evaluation.
-
Memory in the Loop: In-Process Retrieval as Extended Working Memory for Language Agents
Memory-in-the-loop becomes viable when the store lives in-process (~100µs), and under a fixed per-turn memory budget, store speed causally determines whether an agent repeats actions.
-
Escaping the Self-Confirmation Trap: An Execute-Distill-Verify Paradigm for Agentic Experience Learning
EDV decouples execution, distillation by a third-party agent, and consensus verification to filter erroneous trajectories in LLM agent experience learning, outperforming baselines on tau2-bench, Mind2Web, and MMTB.
-
NeuroState-Bench: A Human-Calibrated Benchmark for Commitment Integrity in LLM Agent Profiles
NeuroState-Bench supplies human-calibrated tasks and probes that measure commitment integrity in LLM agents and shows this measure diverges from ordinary task success.
-
Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory
Evo-Memory is a new streaming benchmark and evaluation framework for self-evolving memory in LLM agents, unifying over ten memory modules and introducing the ReMem pipeline for continual improvement on multi-turn and ...
-
Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agents
Memory-R2 proposes LoGo-GRPO to fix unfair trajectory comparisons in RL training of memory-augmented LLM agents by combining global end-to-end rewards with local rerollouts from identical memory states.
-
From Coarse to Fine: Self-Adaptive Hierarchical Planning for LLM Agents
AdaPlan-H enables LLM agents to generate self-adaptive hierarchical plans that adjust detail level to task difficulty, improving success rates in multi-step tasks.
-
StateFuse: Deterministic Conflict-Preserving Memory for Multi-Agent Systems
StateFuse is a conflict-preserving agent memory contract built on standard CRDT merge that surfaces contradictions explicitly and enables semantic correction handles, without improving accuracy over flat multi-value b...
-
Persistent AI Agents in Academic Research: A Single-Investigator Implementation Case Study
A self-observed case study of one persistent AI agent in academic research finds cache-dominant token usage and recommends artifact-level evaluation metrics.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.