{"id":"15f2dfba-c399-4f0a-8cb4-252d9e3dcf9c","arxiv_id":"2608.10108","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MESA learns a query-adaptive subset of five memory structures for long-horizon agents, beating AMA-Agent by 8.5 points on AMA-Bench while using 41% fewer evidence tokens than reading all structures.","lead":"MESA is a method that lets a long-horizon AI agent pick which types of stored memory to read for each new question, instead of always reading all of them. On the AMA-Bench agent-memory test it raises accuracy by 8.5 points over the previous best system while cutting evidence tokens by 41%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MESA's selector is given gold QA-type labels at test time; the claimed task-adaptive gain may be an artifact of this privileged annotation rather than of learned selection.","rationale":"The reader's verdict is CONDITIONAL with MODERATE confidence, identifying validation-set policy search as the weakest assumption and listing QA-type context as a secondary issue. My read agrees the paper is broadly sound, but the single most load-bearing threat is different: the selector is explicitly given gold QA-type annotations at test time. The paper's own Fig. 5 shows learned rules that branch on 'Causal inference' and 'state/update' - both are QA-type labels from the benchmark, not surface text. Baselines do not receive these labels, so the headline comparisons are not controlled for this input. The existing ablations (Table 3) vary the learning machinery but never vary the selector's input features, so the contribution of the privileged label is unidentified. This is more consequential than validation-set generalization because even a perfectly generalizable validation-selection protocol would still inherit the label leakage. The concrete test - rerunning with QA-type removed - is cheap, decisive, and directly addresses the central claim. Since the concern is unconfirmed and the paper already carries a conditional verdict, the verdict stays CONDITIONAL/UNCHANGED; acceptance should require the QA-type ablation and a significance check on the all-structure contrast.","tokens_in":25558,"tokens_out":14295,"duration_ms":155034,"concrete_test":"Re-run the full five-split MESA optimization on AMA-Bench with the selector context c_i stripped of the gold QA-type label t_{i,j} (Appendix A), leaving question text, domain metadata, and query-conditioned evidence previews unchanged; report per-split paired differences. If mean overall accuracy remains near 65.1 +/- 0.8, the label is not load-bearing; if it drops to the all-structure level (63.7) or the gap to AMA-Agent falls below 3 points, the headline result is driven by privileged annotations and the selection claim is unsupported. Optionally, also compare against a variant that replaces the gold label with an LLM-predicted QA type.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MESA's query-adaptive selection beats full fusion depends on the selector's inputs being available at deployment. Section 4.1 and Appendix A define the selector context c_i to include 'task, domain, and QA-type information,' and the accepted policies in Fig. 5 explicitly condition on gold QA-type labels ('Causal inference / text-to-sql', 'state/update'). These labels (recall, causal inference, state updating, state abstraction) are dataset annotations, not user-provided text. None of the baselines, including AMA-Agent, receive them. Because Sec. 3's own sweep shows the best subset varies by QA type, a selector that reads the gold type has a privileged signal that can explain the 8.5-point gain over AMA-Agent and even the 1.4-point gain over the all-structure configuration. The paper never ablates this input: no experiment removes QA-type from c_i. If the measured advantage collapses when the label is withheld, the paper's central claim is an artifact of annotation leakage rather than learned selection. This is a concrete, testable threat to the main result, and it is not covered by the existing ablations in Table 3.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MESA, a framework for query-adaptive selection among five pre-built memory structures (summary, temporal store, knowledge graph, vector database, raw episodic traces) for long-horizon agent memory. A learned selector, trained by an LLM-proposer harness search with UCB-guided scheduling and answer-level feedback, chooses a non-empty subset of structures per query; the selected evidence is fused and passed to a frozen answer model. On AMA-Bench, MESA reports 65.1% overall accuracy versus 56.6% for AMA-Agent, 63.7% for the all-structure configuration, and 52.6% for the long-context reader, while using 11.0k versus 18.7k evidence tokens per query. The paper also reports an exhaustive sweep over all 31 non-empty structure subsets, finding that the best subset is usually neither a single structure nor the full union and varies by task; additional experiments on LoCoMo and multiple ablation and robustness tables are included.","tokens_in":25701,"tokens_out":5368,"duration_ms":51457,"significance":"If the main result holds, the paper makes a useful contribution: it provides a controlled comparison of memory-structure composition strategies and shows that a learned, query-adaptive subset can beat both single-structure routing and full fusion while reducing token cost. Strengths include the exhaustive 31-subset sweep, experiments with two answer-model backbones, judge-robustness checks, a second benchmark (LoCoMo), and explicitly inspectable final policies, with a clear statement that test outcomes are not used during policy selection. The central risk is that the selector's context includes gold QA-type and capability annotations that are not available to baselines or in deployment; the paper does not ablate this input. A second concern is the small validation set used for policy selection. Both are testable and addressable within the manuscript's scope.","major_comments":[{"comment":"The selector context c_i is defined to include 'task, domain, and QA-type information,' and the accepted policies in Fig. 5 explicitly branch on gold QA-type and capability labels (e.g., 'Causal inference / text-to-sql -> {S,T,R}', 'state/update -> {S,T,G,V}', 'abstract -> all five'). These QA-type and capability labels are benchmark annotations, not user-visible query text, and no baseline receives them. Because the controlled sweep in Sec. 3 shows that the winning subset varies by QA type, a selector that reads the gold type has a privileged signal that could explain part of the 8.5-point gain over AMA-Agent and the 1.4-point gain over the all-structure configuration. Table 9 partially addresses coarse domain/capability-conditioned lookup, but it does not isolate the QA-type channel. Please add an ablation that removes QA-type and capability labels from c_i and from the proposer's rule language, and report the resulting accuracy on the same five splits; this is the key test of whether the measured advantage is learned selection or annotation leakage.","section":"§4.1, Fig. 5, Appendix D"},{"comment":"The final policy is chosen as the argmax of the regularized validation objective over an archive produced by 30 iterations with roughly three candidates each, while the validation split contains only about 20% of 208 episodes (approximately 42 episodes). The reported test numbers are therefore the result of validation-based selection over roughly 90 candidate policies. This is a realistic risk of selection bias. Please report the per-split relationship between validation and test scores for all archive policies, or compare the argmax selection rule with alternative rules (e.g., median validation policy, top-3 ensemble, or a stability criterion) to show that the 65.1% headline is not an artifact of choosing the best-looking validation policy.","section":"Algorithm 1, §5.1"},{"comment":"Several component-level claims are not supported by the reported variability. MESA (65.1 ± 0.8) versus the all-structure configuration (63.7 ± 1.0) and versus w/o UCB (64.3 ± 2.2) are within one standard deviation; the statements that removing UCB 'lowers accuracy' and that UCB 'stabilizes the search' rely on overlapping intervals. Please report paired per-split differences with significance tests or bootstrap confidence intervals before interpreting the ablation rows. This does not undermine the headline 8.5-point gain over AMA-Agent, but it does weaken the specific attribution of gains to the UCB component and the claimed accuracy advantage over full fusion.","section":"Table 3"}],"minor_comments":[{"comment":"Appendix D says the selector may access 'the question, task metadata, and query-conditioned evidence previews' but does not mention QA-type, while §4.1 and Fig. 5 make clear that QA-type is used; please reconcile these descriptions.","section":"Appendix D vs. §4.1"},{"comment":"The LoCoMo gain of +1.8 F1 over EMem reported in Table 2 is within the five-fold standard deviations reported in Table 7 (MESA 49.0 ± 3.3, EMem 47.0 ± 4.6). Please state clearly whether the main LoCoMo comparison is on one fixed split and add a paired significance statement over the five conversation-level splits.","section":"Table 2 vs. Table 7"},{"comment":"The last sentence of F.3 contains a typo: 'a acceptable' should be 'an acceptable'.","section":"Appendix F.3"},{"comment":"The label 'Accepted TEST (post-hoc)' in Fig. 5 is confusing; the footnote clarifies that these test accuracies were generated after optimization and never exposed during selection, so the caption should say so directly.","section":"Fig. 5 caption"},{"comment":"The hyperparameters λ and β are fixed across all settings; reporting a small sensitivity analysis (e.g., λ = 0 and β = 0.04/0.16) would make the calibration argument in B.2 more convincing.","section":"Appendix B.2"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the gold QA-type/capability annotation leakage described in Major Comment 1; if the requested ablation collapses the gain over AMA-Agent, the central claim would need substantial reframing. The validation-selection concern in Major Comment 2 is also important given the small 2:2:6 split. The authors evaluate on AMA-Bench and against AMA-Agent, both cited to Zhao et al. 2026, which appears to involve the same group as the first author; this is not a correctness issue, but the paper should acknowledge the provenance and, ideally, include an externally maintained baseline implementation or an independent third-party evaluation to strengthen the comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: a well-executed empirical paper with a genuinely useful formulation, but the headline result is weaker than it looks and one load-bearing input is never ablated. The central idea deserves engagement; the paper needs another experiment.\n\nWhat's new: structure-level dynamic selection across heterogeneous memory views, plus the controlled 31-subset sweep showing intermediate subsets often beat single-structure and all-structure access. Prior work fuses all, routes to one, or selects within one schema; framing selection across five distinct structures with answer-level feedback is a real contribution. The structures themselves are borrowed, but the problem is new.\n\nWhat it does well: extensive evaluation—two backbones, three judges, five splits with means and standard deviations, ablations for random, zero-shot LLM, route-to-one, and each learning component. Learned policies are interpretable, a genuine plus. Data handling is careful and the paper is mostly explicit about what is fixed.\n\nSoft spots:\n1. The stress-test concern holds. The selector context is allowed to include 'task, domain, QA-type' and the accepted policies in Fig. 5 explicitly branch on 'Causal inference / text-to-sql' and 'state/update.' These are dataset annotations, not user text. The paper never ablates QA-type from the context. Table 9 compares against category-conditioned fixed policies, which suggests labels alone don't explain everything, but those margins (0.7-1.3 points) are small and untested for significance. This is load-bearing.\n2. The abstract's 8.5% over the 'strongest baseline' bundles adding the knowledge-graph structure with learned selection. The isolated gain over all-structure is 1.4 points, with 41% fewer tokens. The efficiency result is good but less dramatic.\n3. Component ablations (w/o priors, w/o UCB) are within one standard deviation of MESA and no significance tests are given. Minor.\n4. No code or full proposer prompts are released; exact replication is not possible.\n\nCitation pattern looks fine; AMA-Agent is a natural baseline and there is no obvious circularity.\n\nWho this is for: people building long-horizon agent memory systems. They'll get a useful taxonomy, a careful evaluation, and a warning about metadata leakage.\n\nRecommendation: send to peer review; ask for the QA-type ablation and significance tests. With those, the paper could be solid.","headline":"Useful formulation and extensive experiments, but the missing QA-type ablation leaves the main gain possibly explained by annotation metadata.","tokens_in":26366,"tokens_out":6104,"would_cite":true,"duration_ms":57784,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a learned, query-adaptive selection among complementary memory structures outperforms both single-structure routing and full fusion for long-horizon agent memory.","keywords":["long-horizon agents","agent memory","memory structure selection","query-adaptive retrieval","harness optimization","evidence composition","LLM agents","multi-structure memory"],"falsifier":"Run the same experiments with a 1:1:8 train/validation/test split or with leave-one-out selection over episodes, and check whether the validation-selected policy still beats the all-structure configuration on the test set; if the advantage reverses or disappears, the central claim fails.","tokens_in":1568,"feed_emoji":"🧠","tokens_out":1896,"duration_ms":78738,"temperature":0.7,"pith_summary":"Long-horizon agents accumulate long, heterogeneous histories, and a later answer often depends on evidence buried deep in an earlier trajectory. The paper argues that the right way to read that memory is neither a single fixed structure nor every available structure, but a query-dependent subset of complementary structures. It supports this with a controlled sweep over all non-empty subsets of five trajectory representations, then introduces MESA, a framework that learns such a subset-selection policy from end-to-end answer-level feedback while keeping all memory builders, retrievers, and the answer model frozen. On AMA-Bench, MESA reaches 65.1% accuracy, outperforming the strongest prior baseline by 8.5 percentage points and the all-structure configuration by 1.4 points, while using 11.0k evidence tokens per query versus 18.7k for reading all structures. If right, memory access becomes a learned per-query composition problem rather than a fixed pipeline.","feed_headline":"Learned memory-subset selection beats reading everything","feed_subtitle":"MESA reaches 65.1% accuracy on AMA-Bench with 41% fewer evidence tokens than the all-structure baseline.","key_machinery":"The load-bearing object is the selector policy $\\rho$ that maps a query $q_i$ and observable context $c_i$ to a non-empty binary vector $z_i \\in \\{0,1\\}^K$ over $K=5$ memory structures (summary, temporal store, knowledge graph, vector database, and raw episodic trace). The optimizer, called prior-guided harness optimization, keeps all builders, retrievers, composer, and the answer model fixed and searches only over executable selection policies generated under twenty prior directions organized into five axes: demand inference, backend value estimation, subset construction, evidence awareness, and robustness and cost shaping. Candidate policies are fitted on training traces, scored on validation with the regularized objective $\\tilde{J}_{\\mathrm{val}}(\\rho)=J_{\\mathrm{val}}(\\rho)-\\lambda C_{\\mathrm{val}}(\\rho)$, and a UCB rule $\\mathrm{UCB}(a)=\\bar{\\tilde{J}}_{\\mathrm{val}}(a)+\\beta\\sqrt{\\log(N+2)/(n_a+1)}$ schedules which proposal direction the LLM proposer explores. This machinery converts sparse, answer-level correctness feedback into a concrete, inspectable routing rule.","core_discovery":"The central claim is that the optimal memory configuration for a long-horizon agent is typically neither a single structure nor the full union, but an intermediate, query-dependent subset. In a controlled analysis on AMA-Bench, the best composition varied across domains and memory capabilities, so no fixed subset was universally optimal. MESA learns an explicit selection policy that maps a query and observable context to a non-empty subset of five structures, and this learned policy outperforms all baselines on AMA-Bench with two different answer-model backbones while also improving F1 on the conversational LoCoMo benchmark. The paper attributes the gains to composing complementary evidence views rather than to any single strong representation, since the strongest single structure alone reaches only 57.7% accuracy, while MESA reaches 65.1%.","pith_inferences":["Inference: The intermediate-subset finding suggests a general design principle: memory access can be treated as per-query feature selection, and new memory structures should be evaluated by their marginal contribution to selection rather than in isolation.","Inference: Because MESA freezes memory builders and retrievers, jointly optimizing construction with selection could yield larger gains and may shift the optimal subset toward fewer or different structures than the five studied here.","Inference: The validation-based selection protocol is the main risk; a natural stress test is nested cross-validation or a much larger validation budget to see whether the reported margin persists.","Inference: The explicit rules learned by MESA could be distilled into a small deterministic router that runs without an LLM proposer at inference time, further reducing cost and latency."],"forward_implications":["If MESA is right, agent memory systems should expose multiple complementary structures and learn per-query selection instead of using fixed routing or full fusion.","The 41.2% token reduction at higher accuracy indicates that adaptive selection is a cost-saving strategy, not a trade-off between quality and efficiency.","Accuracy gains are concentrated in domains that require combining global state with exact identifiers and temporal locations, such as software engineering and text-to-SQL, suggesting that composition of complementary structures is the mechanism.","The framework transfers from agent trajectories to long-term conversations, so the principle is not limited to episodic agent memory.","Learned policies remain explicit, keyword-based rules, making memory-access decisions auditable and easy to inspect."],"supporting_citations":[{"why":"Supplies the AMA-Bench episodes, the LLM-judge protocol, and the strongest prior baseline that MESA must beat.","marker":"(Zhao et al. 2026)"},{"why":"Supplies the OpenIE graph-construction pipeline and graph-retrieval mechanism that MESA reuses for its knowledge-graph structure.","marker":"(Gutiérrez et al. 2024)"},{"why":"Provides the dense embedding encoder used for the vector-database structure and for item-level retrieval baselines.","marker":"(Zhang et al. 2025)"},{"why":"Supplies the UCB rule that MESA uses to balance exploration and exploitation across proposal directions.","marker":"(Auer, Cesa-Bianchi, and Fischer 2002)"},{"why":"Provides the bandit-based adaptive operator selection principle that motivates treating each prior direction as a UCB arm.","marker":"(Fialho et al. 2010)"},{"why":"Supplies the LoCoMo conversational-memory benchmark used to test whether the framework extends beyond agent trajectories.","marker":"(Maharana et al. 2024)"}],"fun_headline_variants":["Adaptive memory subset beats full history","MESA picks query-specific memory mixes","Optimal memory is a tailored subset, not everything","Learn to choose memory structures per query","Query-adaptive memory selection cuts tokens by 41%"],"cache_read_input_tokens":28416,"weakest_assumption_plain":"The load-bearing premise is that picking the policy with the best regularized score on a validation set of roughly 20 percent of episodes will keep its advantage on held-out episodes, even though the search only tries about 90 candidate policies.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive memory subset beats full history","MESA picks query-specific memory mixes","Optimal memory is a tailored subset, not everything","Learn to choose memory structures per query","Query-adaptive memory selection cuts tokens by 41%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1278,"prompt_tokens":959,"completion_tokens":319,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":251}},"tokens_in":575,"tokens_out":319,"duration_ms":3648,"temperature":1.0,"reasoning_tokens":251,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:11:47.281478+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same experiments with a 1:1:8 train/validation/test split or with leave-one-out selection over episodes, and check whether the validation-selected policy still beats the all-structure configuration on the test set; if the advantage reverses or disappears, the central claim fails.","supporting_citations":[],"review_version":1}