pith. machine review for the scientific record. sign in

arxiv: 2605.12477 · v1 · submitted 2026-05-12 · 💻 cs.LG · cs.CL

Recognition: no theorem link

MEME: Multi-entity & Evolving Memory Evaluation

Alexander Rubinstein, Arnas Uselis, Sangdoo Yun, Seokwon Jung, Seong Joon Oh

Pith reviewed 2026-05-13 05:07 UTC · model grok-4.3

classification 💻 cs.LG cs.CL
keywords LLM agentsmemory systemsbenchmarkmulti-entity memoryevolving memorydependency reasoningpersistent environments
0
0 comments X

The pith

Current memory systems for LLM agents fail at reasoning over dependencies between multiple evolving entities, even when static retrieval works.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper creates the MEME benchmark with six tasks that together test memory across single-entity and multi-entity updates, as well as evolving dependencies and deletions over time. It runs six memory systems from three different paradigms on 100 controlled episodes and shows that every system collapses on the dependency-reasoning tasks Cascade and Absence, reaching only 3% and 1% average accuracy. This failure holds even after prompt optimization, deeper retrieval, noise reduction, and the use of stronger models. Only one expensive file-based configuration with a top-tier model improves results, but at roughly 70 times the normal cost.

Core claim

All tested memory systems, regardless of paradigm, prove unable to perform dependency reasoning when multiple entities change state across sessions; Cascade and Absence tasks expose average accuracies of 3% and 1% while static retrieval remains adequate.

What carries the argument

The MEME benchmark's six tasks, especially the dependency-reasoning tasks Cascade and Absence, which require tracking how updates to one entity affect others and how missing information propagates.

If this is right

  • Standard memory paradigms cannot scale to agent environments that require tracking dependencies across entities.
  • Common engineering fixes such as prompt tuning or stronger base models leave the core failure untouched.
  • Closing the gap currently demands configurations whose cost grows by an order of magnitude.
  • Benchmarks limited to single-entity updates miss the dominant failure mode in longer-lived agent settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Agent designers may need hybrid memory that combines vector retrieval with explicit symbolic tracking of entity relationships.
  • The cost-performance trade-off shown here could limit deployment of autonomous agents in domains that require reliable long-term memory.
  • Extending the benchmark to include real-world interaction logs would test whether the controlled episodes generalize.

Load-bearing premise

The six tasks and 100 controlled episodes capture the essential difficulties that appear in real persistent multi-entity environments.

What would settle it

A practical, low-cost memory system that reaches substantially higher accuracy on Cascade and Absence tasks under default conditions would show the collapse is not inherent to current approaches.

Figures

Figures reproduced from arXiv: 2605.12477 by Alexander Rubinstein, Arnas Uselis, Sangdoo Yun, Seokwon Jung, Seong Joon Oh.

Figure 1
Figure 1. Figure 1: MEME’s taxonomy of memory op￾erations along two dimensions: entity scope (single vs. multi-entity) and temporal dynamics (static vs. evolving), with six tasks distributed across the four quadrants. As Large Language Models (LLMs) increasingly serve as agents that interact with users across many sessions, accurately storing, updating, and reasoning over past interactions has become essential [17]. For insta… view at source ↗
Figure 2
Figure 2. Figure 2: Examples of the six MEME task types across three categories: [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Marginal effect of each evaluation axis on [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: State of two failing systems (Graphiti, Karpathy Wiki) and the closure case (MD-flat [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Two interventions external to the memory architecture: (a) prompt optimization (DSPy SIMBA, [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Full generated session for Episode 1, Session 1 (Fact Introduction) of the Personal Life domain. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Rejection examples per conflict type, with conflicting phrases in [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Third-person conversion prompt (Personal Life). [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Third-person conversion prompt (Software Project). [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: User LLM system prompt for self-chat conversation generation. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Assistant LLM system prompt for self-chat conversation generation (Personal Life). [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Assistant LLM system prompt for self-chat conversation generation (Software Project). [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Annotation verification prompt (Layer 1). [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Gemini semantic audit prompt (Layer 2). D.3 Ingestion and Retrieval Prompts (MD-flat) MD-flat is the only memory system whose ingestion and retrieval prompts are designed by the benchmark authors; the other systems use their own built-in logic without modification. The inges￾tion prompt processes each session and updates memory.md; the retrieval prompt reads back from memory.md at query time. You are a pe… view at source ↗
Figure 15
Figure 15. Figure 15: MD-flat ingestion prompt. The agent operates through a tool-calling loop (max 5 rounds) with [PITH_FULL_IMAGE:figures/full_fig_p021_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: MD-flat retrieval prompt. At query time the agent runs a tool-calling loop with only [PITH_FULL_IMAGE:figures/full_fig_p021_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Unified answer prompt shared across all memory systems. [PITH_FULL_IMAGE:figures/full_fig_p022_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Judge prompt: Before-phase (common). You are a judge evaluating whether an AI agent correctly recalls a sequence of historical values for an entity that changed multiple times. QUESTION: {question} GOLD (ordered history values, comma-separated): {gold} AGENT: {agent_answer} The user’s information changed multiple times. The question asks for the FULL HISTORY in chronological order (earliest to latest). Ru… view at source ↗
Figure 19
Figure 19. Figure 19: Judge prompt: Tracking. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Judge prompt: Aggregation. You are a judge evaluating whether an AI agent correctly recalls an EXACT value verbatim. QUESTION: {question} GOLD: {gold} AGENT: {agent_answer} Does the agent’s answer contain the exact value? Rules: – Check if the gold value appears VERBATIM as a substring in the agent’s answer – If the gold value is fully contained in the answer, even with extra words before/after it → YES –… view at source ↗
Figure 21
Figure 21. Figure 21: Judge prompt: Exact Recall. Note: implemented as deterministic substring match at runtime; this [PITH_FULL_IMAGE:figures/full_fig_p023_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Judge prompt: Deletion. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Judge prompt: Cascade. You are a judge evaluating whether an AI agent appropriately handles UNCERTAINTY after an upstream entity changed. QUESTION: {question} GOLD: {gold} AGENT: {agent_answer} The upstream entity changed, making the previous answer potentially invalid. The agent should express uncertainty. Rules: – Agent says “I don’t know” / “not sure” / “Unknown” / “None” / expresses uncertainty → YES … view at source ↗
Figure 24
Figure 24. Figure 24: Judge prompt: Absence. For Aggregation and Tracking tasks, partial credit is computed in addition to binary pass/fail. Aggre￾gation records the number of target values present out of the total. Tracking records the number of history values in correct chronological order using a position-based algorithm. D.6 Filler Filtering Prompt You are checking if a filler conversation conflicts with a known gold fact.… view at source ↗
Figure 25
Figure 25. Figure 25: Filler conflict judgment prompt (GPT-4o-mini). [PITH_FULL_IMAGE:figures/full_fig_p024_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: MD-flat: SIMBA-appended advice blocks for the ingest and retrieve prompts. [PITH_FULL_IMAGE:figures/full_fig_p025_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Graphiti: SIMBA-appended advice blocks for the three optimized prompts. [PITH_FULL_IMAGE:figures/full_fig_p026_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Karpathy Wiki: SIMBA-appended advice blocks for the three optimized prompts. [PITH_FULL_IMAGE:figures/full_fig_p027_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Per-task accuracy under three filler conditions (no filler, 32K, 128K) for each system. Deletion, [PITH_FULL_IMAGE:figures/full_fig_p028_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: State of the remaining four memory systems on episode sw_033, traced across encoding, maintenance, [PITH_FULL_IMAGE:figures/full_fig_p031_30.png] view at source ↗
read the original abstract

LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions. While prior benchmarks evaluate only single-entity updates, MEME defines six tasks spanning the full space defined by the multi-entity and evolving axes, including three not scored by prior work: Cascade and Absence (dependency reasoning) and Deletion (post-removal state). Evaluating six memory systems spanning three memory paradigms on 100 controlled episodes, we find that all systems collapse on dependency reasoning under the default configuration (Cascade: 3%, Absence: 1% in average accuracy) despite adequate static retrieval performance. Prompt optimization, deeper retrieval, reduced filler noise, and most stronger LLMs fail to close this gap. Only a file-based agent paired with Claude Opus 4.7 as its internal LLM partially closes the gap, but at ~70x the baseline cost, indicating closure currently depends on configurations that are not practical at scale. Code and data are available on the project page: https://seokwonjung-jay.github.io/meme-eval/.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces MEME, a benchmark for LLM-agent memory systems in persistent multi-entity and evolving environments. It defines six tasks spanning the multi-entity and evolving axes (including three novel tasks: Cascade and Absence for dependency reasoning, and Deletion for post-removal state), evaluates six memory systems across three paradigms on 100 controlled episodes, and reports that all systems collapse on dependency reasoning (Cascade: 3%, Absence: 1% average accuracy) despite adequate static retrieval. Prompt optimization, deeper retrieval, reduced filler noise, and most stronger LLMs fail to close the gap; only a high-cost file-based agent with Claude Opus 4.7 shows partial improvement.

Significance. If the 100 episodes and task definitions accurately instantiate the core difficulties of real-world persistent multi-entity agent environments, the work would be significant in demonstrating a systematic limitation of current memory paradigms on dependency reasoning and in motivating new architectures. The public release of code and data is a clear strength for reproducibility.

major comments (3)
  1. [Abstract] Abstract: The central empirical claim that all six systems collapse on dependency reasoning (Cascade 3%, Absence 1%) is load-bearing, yet the manuscript supplies no details on episode construction, statistical variance across the 100 episodes, exact definitions of the six memory systems, or the quantitative criteria used to establish 'adequate static retrieval performance'. These omissions prevent assessment of whether the reported numbers are robust or sensitive to implementation choices.
  2. [Abstract / Task definitions] The interpretation that current memory paradigms fundamentally fail at dependency reasoning requires that the Cascade, Absence, and Deletion tasks (and their 100 controlled episodes) represent the core challenges agents encounter in naturalistic multi-entity evolving settings. The manuscript provides no external validation—such as comparison against real agent interaction traces, expert review of dependency-chain realism, or sensitivity analysis to filler noise and update patterns—to rule out the possibility that the observed collapse is an artifact of the benchmark design.
  3. [Abstract / Evaluation] The claim that prompt optimization, deeper retrieval, reduced filler noise, and most stronger LLMs 'fail to close this gap' is presented without reporting the specific interventions tested, their quantitative effects on Cascade/Absence accuracy, or the exact set of stronger LLMs evaluated. This makes it impossible to determine the scope of the failure or whether the gap is truly unbridgeable within practical configurations.
minor comments (2)
  1. [Abstract] The abstract refers to 'three memory paradigms' without naming them; an explicit list in the introduction or a table summarizing the six systems would improve clarity.
  2. [Abstract] The paper states that code and data are available on the project page; confirming that the repository includes the exact episode generator, task definitions, and evaluation scripts would strengthen the reproducibility claim.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for the detailed and constructive feedback. We address each major comment below and indicate where revisions will be made to improve the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central empirical claim that all six systems collapse on dependency reasoning (Cascade 3%, Absence 1%) is load-bearing, yet the manuscript supplies no details on episode construction, statistical variance across the 100 episodes, exact definitions of the six memory systems, or the quantitative criteria used to establish 'adequate static retrieval performance'. These omissions prevent assessment of whether the reported numbers are robust or sensitive to implementation choices.

    Authors: We agree the abstract is too concise and omits key context. The full manuscript details episode construction and the 100 controlled episodes in the benchmark design section, reports statistical variance (standard deviations) alongside the accuracy figures in the evaluation tables, defines the six memory systems in the systems under test section, and specifies the criteria for adequate static retrieval based on control-task performance in the results analysis. In the revision we will expand the abstract with brief descriptions of episode construction, variance reporting, system definitions, and the static-retrieval criterion. revision: yes

  2. Referee: [Abstract / Task definitions] The interpretation that current memory paradigms fundamentally fail at dependency reasoning requires that the Cascade, Absence, and Deletion tasks (and their 100 controlled episodes) represent the core challenges agents encounter in naturalistic multi-entity evolving settings. The manuscript provides no external validation—such as comparison against real agent interaction traces, expert review of dependency-chain realism, or sensitivity analysis to filler noise and update patterns—to rule out the possibility that the observed collapse is an artifact of the benchmark design.

    Authors: The tasks were constructed to isolate the multi-entity and evolving axes defined in the problem formulation section, with controlled episodes that systematically vary dependency chains, filler content, and update patterns. We did not conduct external validation against real agent traces or expert review of realism. We will add an explicit limitations paragraph discussing this design choice and include any available sensitivity results on filler noise and update patterns. revision: partial

  3. Referee: [Abstract / Evaluation] The claim that prompt optimization, deeper retrieval, reduced filler noise, and most stronger LLMs 'fail to close this gap' is presented without reporting the specific interventions tested, their quantitative effects on Cascade/Absence accuracy, or the exact set of stronger LLMs evaluated. This makes it impossible to determine the scope of the failure or whether the gap is truly unbridgeable within practical configurations.

    Authors: The abstract summarizes findings from the ablation experiments. In the revision we will add a concise table or paragraph reporting the specific interventions tested, their measured effects on Cascade and Absence accuracy, and the exact set of stronger LLMs evaluated, so readers can assess the scope of the observed gap. revision: yes

standing simulated objections not resolved
  • External validation of task realism via real agent interaction traces or expert review, which was not performed in the current study.

Circularity Check

0 steps flagged

No circularity: direct empirical benchmark with measured results

full rationale

The paper is a controlled empirical evaluation that defines six tasks (Cascade, Absence, Deletion, etc.) and measures accuracy of six memory systems across 100 episodes. No derivation chain, equations, or first-principles predictions exist; reported numbers (e.g., Cascade 3%, Absence 1%) are direct experimental outputs, not quantities fitted or renamed from inputs. Task construction and episode design are explicit and independent of the measured outcomes. No self-citations are invoked to justify core claims, and no ansatz or uniqueness theorem reduces the results to prior author work. The evaluation is self-contained against its own benchmark definition.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper is an empirical benchmark study. It defines evaluation tasks along two axes but introduces no free parameters, mathematical axioms, or postulated physical entities.

pith-pipeline@v0.9.0 · 5495 in / 1082 out tokens · 73518 ms · 2026-05-13T05:07:46.047339+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 37 canonical work pages · 6 internal anchors

  1. [1]

    Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory

    Chhikara, P., Khant, D., Aryan, S., Singh, T., and Yadav, D. Mem0: Building production-ready AI agents with scalable long-term memory.arXiv preprint arXiv:2504.19413, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20(1):37–46, 1960

    Cohen, J. A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20(1):37–46, 1960

    work page 1960

  3. [3]

    From Local to Global: A Graph RAG Approach to Query-Focused Summarization

    Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., Metropolitansky, D., Ness, R. O., and Larson, J. From local to global: A graph RAG approach to query-focused summarization.arXiv preprint arXiv:2404.16130, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    RULER: What's the Real Context Size of Your Long-Context Language Models?

    Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y ., and Ginsburg, B. RULER: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    Evaluating memory in LLM agents via incremental multi-turn interactions

    Hu, Y ., Wang, Y ., and McAuley, J. Evaluating memory in LLM agents via incremental multi-turn interactions. InThe Fourteenth International Conference on Learning Representations (ICLR), 2026

    work page 2026

  6. [6]

    Unsupervised dense information retrieval with contrastive learning.Transactions on Machine Learning Research, 2022

    Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., and Grave, E. Unsupervised dense information retrieval with contrastive learning.Transactions on Machine Learning Research, 2022

    work page 2022

  7. [7]

    LLM knowledge base

    Karpathy, A. LLM knowledge base. https://gist.github.com/karpathy/ 442a6bf555914893e9891c11519de94f, 2026

    work page 2026

  8. [8]

    T., Moazam, H., Miller, H., Zaharia, M., and Potts, C

    Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., Miller, H., Zaharia, M., and Potts, C. DSPy: Compiling declarative lan- guage model calls into self-improving pipelines. InThe Twelfth International Conference on Learning Representations (ICLR), 2024

    work page 2024

  9. [9]

    Lù, X. H. BM25S: Orders of magnitude faster lexical search via eager sparse scoring.arXiv preprint arXiv:2407.03618, 2024

    work page arXiv 2024

  10. [10]

    Evaluating Very Long-Term Conversational Memory of LLM Agents

    Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., and Fang, Y . Evaluating very long-term conversational memory of LLM agents.arXiv preprint arXiv:2402.17753, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    A., Yoon, S., and Schütze, H

    Modarressi, A., Deilamsalehy, H., Dernoncourt, F., Bui, T., Rossi, R. A., Yoon, S., and Schütze, H. NoLiMa: Long-context evaluation beyond literal matching. InProceedings of the 42nd International Conference on Machine Learning (ICML), 2025

    work page 2025

  12. [12]

    MemGPT: Towards LLMs as Operating Systems

    Packer, C., Wooders, S., Lin, K., Fang, V ., Patil, S. G., Stoica, I., and Gonzalez, J. E. MemGPT: Towards LLMs as operating systems.arXiv preprint arXiv:2310.08560, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  13. [13]

    Zep: A Temporal Knowledge Graph Architecture for Agent Memory

    Rasmussen, P., Paliychuk, P., Beauvais, T., Ryan, J., and Chalef, D. Zep: A temporal knowledge graph architecture for agent memory.arXiv preprint arXiv:2501.13956, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  14. [14]

    ShareGPT52K.https://huggingface.co/datasets/RyokoAI/ShareGPT52K, 2023

    RyokoAI. ShareGPT52K.https://huggingface.co/datasets/RyokoAI/ShareGPT52K, 2023

    work page 2023

  15. [15]

    MemBench: Towards more comprehensive evaluation on the memory of LLM-based agents

    Tan, H., Zhang, Z., Ma, C., Chen, X., Dai, Q., and Dong, Z. MemBench: Towards more comprehensive evaluation on the memory of LLM-based agents. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 19336–19352, 2025

    work page 2025

  16. [16]

    LongMemEval: Benchmarking chat assistants on long-term interactive memory

    Wu, D., Wang, H., Yu, W., Zhang, Y ., Chang, K.-W., and Yu, D. LongMemEval: Benchmarking chat assistants on long-term interactive memory. InThe Thirteenth International Conference on Learning Representations (ICLR), 2025

    work page 2025

  17. [17]

    A survey on the memory mechanism of large language model based agents.arXiv preprint arXiv:2404.13501, 2024

    Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., and Wen, J.-R. A survey on the memory mechanism of large language model based agents.arXiv preprint arXiv:2404.13501, 2024

    work page arXiv 2024

  18. [18]

    W., Salakhutdinov, R., and Manning, C

    Yang, Z., Qi, P., Zhang, S., Bengio, Y ., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. HotpotQA: A dataset for diverse, explainable multi-hop question answering. InProceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018

    work page 2018

  19. [19]

    D., Potts, C., and Chen, D

    Zhong, Z., Wu, Z., Manning, C. D., Potts, C., and Chen, D. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. InProceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    work page 2023

  20. [20]

    Evaluating the ripple effects of knowledge editing in language models.Transactions of the Association for Computational Linguistics, 12:283–298, 2024

    Cohen, R., Biran, E., Yoran, O., Globerson, A., and Geva, M. Evaluating the ripple effects of knowledge editing in language models.Transactions of the Association for Computational Linguistics, 12:283–298, 2024. 10

    work page 2024

  21. [21]

    MuSiQue: Multihop questions via single hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. MuSiQue: Multihop questions via single hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    work page 2022

  22. [22]

    Retrieval-augmented generation for knowledge-intensive NLP tasks

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V ., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. Retrieval-augmented generation for knowledge-intensive NLP tasks. InAdvances in Neural Information Processing Systems (NeurIPS), 2020

    work page 2020

  23. [23]

    {target_fact} — this depends on where {source_entity_phrase}; if I move, this would change

    Zheng, L., Chiang, W.-L., Sheng, Y ., Zhuang, S., Wu, Z., Zhuang, Y ., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 11 A Operational Costs We report token usage at three pipeline stages:Ingest(LL...

    work page 2023

  24. [24]

    Enumerate all possible gold-fact sentences by applying each entity’s value pool to its fact template

    work page

  25. [25]

    For each gold fact, retrieve the top-K=10candidate fillers using a hybrid of BM25 lexical scoring andtext-embedding-3-smalldense similarity

    work page

  26. [26]

    The judge flags three conflict types:A(CONTRADICTION),B(ALTERNATIVE), andC (ENTITY_CONFUSION)

    Judge each (gold fact, filler) pair with GPT-4o-mini using the prompt in Section D.6. The judge flags three conflict types:A(CONTRADICTION),B(ALTERNATIVE), andC (ENTITY_CONFUSION). Fillers flagged for any type are removed from the pool

    work page

  27. [27]

    Hi! How can I help you today?

    At episode assembly time, a keyword-based blocklist derived from the current episode’s gold entities acts as a final safety net. 14 Assistant:“Hi! How can I help you today?” User:“My hobby is pottery. I’ve been getting into it more lately. Do you know any good techniques for beginners?” Assistant:“One effective technique for beginners is hand-building, wh...

    work page

  28. [28]

    My health condition is

    Each fact must include its entity keyword (e.g., “My health condition is...”, “My financial goal is...”). The entity concept must appear in your message

    work page

  29. [29]

    depends on

    If a fact mentions a dependency (“depends on”, “determined by”, “if X changes”), you MUST preserve the full dependency language including the conditional part. The reader must understand that if the source changes, the target would change too. Do NOT weaken “depends on X; if X changes, this would change” to just “tied to X” or “since we use X”

    work page

  30. [30]

    If” (conditional), use ONLY “will

    If a fact starts with “If” (conditional), use ONLY “will” as the modal: “If X changes, Y will be Z”. NEVER use “would/might/probably/consider”. NEVER state it as accomplished (“I switched to Z”)

    work page

  31. [31]

    I don’t have personal experiences

    If a fact is marked with [CHANGED], state the new value as a definitive current fact. Do NOT hedge or speculate. You are a HUMAN user, not an AI. Never say anything that implies you are an AI, a language model, or that you have a training data cutoff. Never repeat a previous message verbatim. Each message must be unique. When ALL facts have been conveyed ...

    work page

  32. [32]

    VERBALIZATION ACCURACY Is the gold fact’s value present in the conversation (exactly or close paraphrase)?

    work page

  33. [33]

    if” conditional structure. Trigger entity (dependency_source) must be explicitly mentioned. Modal must be “will

    IF-THEN CONDITIONAL FORM (only for facts with is_if_then=true) Must contain “if” conditional structure. Trigger entity (dependency_source) must be explicitly mentioned. Modal must be “will” (not “would/might/probably”) → flag WEAK_MODAL. Must NOT be stated as accomplished fact→flag CONDITIONAL_VIOLATED

    work page

  34. [34]

    What’s my financial goal?

    ENTITY KEYWORD PRESENCE The question asks about an entity using certain words (e.g., “What’s my financial goal?”). Those words or a natural synonym must appear in the conversation where the fact is introduced. FAIL if entity concept never mentioned (e.g., “I’m paying off debt” without “financial goal”)

    work page

  35. [35]

    depends on

    CASCADE-U DEPENDENCY (only for facts with has_dependency=true AND is_if_then=false) Is the causal link STRONGLY explicit? The message must contain language like “depends on”, “determined by”, or “if X changes, Y would change”. FAIL if dependency is only implied (“since we use X”, “tied to X”) rather than explicitly conditional (“depends on X; if X changes...

    work page

  36. [36]

    Yes — ” prefix? (flag PREFIX_ISSUE) Output JSON array of issues: {“entity

    GOLD ANSWER FORMAT Does before-delete gold have “Yes — ” prefix? (flag PREFIX_ISSUE) Output JSON array of issues: {“entity”: “...”, “fact_index”: N, “check”: “VERBALIZATION|IF_THEN|KEYWORD|CASCADE_U_DEP|GOLD_FORMAT”, “severity”: “HIGH|MEDIUM|LOW”, “issue”: “brief description”} Severity: HIGH = makes task unsolvable. MEDIUM = may cause false pass/fail. LOW...

    work page 2023

  37. [37]

    Weekly report recipient is Hyunwoo Nam (assigned by team lead Seokjin Kang). If the team lead changes, the recipient will be James Lee

    on all six main-table systems and 100 episodes; internal LLM held at gpt-4.1-mini. Trivial-pass filtering applied to Cascade, Absence, Deletion.Boldmarks the best per task column. System Answering LLM ER Agg Tr Del Cas Abs Overall BM25 gpt-4.1-mini1.000.05 0.160.270.02 0.00 0.25 Sonnet 4 0.70 0.09 0.11 0.19 0.01 0.12 0.20 text-embedding-3-small gpt-4.1-mi...

    work page 2023