{"id":"9cc244a9-bb3f-477b-a3f5-7b9e27d31cc3","arxiv_id":"2605.22142","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A per-item Q-learning policy that explicitly decides which observed knowledge-graph facts to keep or drop outperforms fixed heuristics and sequence-memory baselines on the RoomKG benchmark at memory capacity 128.","lead":"This paper trains a reinforcement learning agent to decide, for each new fact it observes, whether to save that fact in long-term memory or throw it away, in a knowledge-graph environment where the agent can only see part of the world at a time. A smart generalist might read it because it turns a memory-management question into a learnable and inspectable decision problem, and shows interpretable keep/drop behavior on a benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TD update's stochastic index-wise item matching across timesteps introduces alignment noise, leaving learned Q-values potentially driven by arbitrary item pairings rather than true transfer values; the paper's own 'practical rather than theoretically complete' caveat (§7) leaves this unquantified.","rationale":"We reviewed the manuscript carefully. The central claim is empirical: on RoomKG capacity 128, learned keep/drop transfer decisions outperform symbolic and neural baselines. The most fundamental threat to this claim is not the baseline suite or statistics, but the training signal itself. Section 3.3 and Algorithm 1 define a per-item TD update where bootstrap targets for item j at time t are taken from item j at time t+1 after a stochastic shuffle, with length capped at min(n_b, n'_b). Because short-term sets are local observations at different states, the paired items usually represent different facts; the target therefore does not correspond to the same persistent fact, so the Q-values are updated toward arbitrary next-state values. The same transition, when resampled from replay, yields different random pairings, injecting further variance. The paper explicitly acknowledges this as 'practical rather than theoretically complete' (Conclusion §7), but provides no analysis of how this affects learning. If this alignment noise dominates, the reported improvement could be driven by spurious correlations rather than by accurate value estimates. This is more load-bearing than the lack of significance tests or stronger baselines because it questions whether the method actually learns what it claims. Our proposed test replaces the random pairing with content-based alignment (matching identical triples across consecutive steps), which would directly quantify the impact. Should the test show a large performance shift, the paper would need to either justify the random matching or adjust the claims; if the shift is small, the concern is mitigated and conditional acceptance can proceed. Thus we agree with the reader's weakest_assumption and recommend no change to the verdict.","tokens_in":13723,"tokens_out":12253,"duration_ms":142231,"concrete_test":"Modify Algorithm 1's matching step to use content-based alignment: for each sampled transition, identify triples that appear in both M_short_b and M_short_{b+1} (exact match on (h,r,t)) and pair these persistent facts; pair the remaining unmatched items randomly (or exclude them from the TD loss). Retrain the best configuration (GCN+Local-STM) on RoomKG capacity 128 with the same seeds and hyperparameters, and compare test QA scores to the reported 38.920±3.090. A shift greater than ~3 points (one seed std) would indicate the stochastic index-wise matching is load-bearing; a shift within noise would mitigate the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central methodological weakness is the TD update's item matching (§3.3, Algorithm 1). For a sampled transition, the target for item j at time t is r_t + γ(1−d_b) max_a Q_θ̄(M_{t+1}, j, a), where j results from a stochastic shuffle and ℓ_b = min(|M_short_t|, |M_short_{t+1}|). Because M_short_t and M_short_{t+1} are different observation sets (agent moves, objects move), the matched items are generally unrelated facts; the bootstrap target is the value of an arbitrary next-state fact, not the future value of the item whose keep/drop decision is being trained. Replaying the same transition with different random pairings adds further noise. Thus the Q-function's training signal is dominated by alignment noise, not by the true causal effect of keeping/dropping a specific triple. The authors acknowledge this only as a practical simplification ('practical rather than theoretically complete', Conclusion §7) and provide no sensitivity analysis. If the alignment noise is not benign, the reported improvement over Novel-Only (38.920 vs 31.960) could be an artifact of spurious correlations (e.g., at_location facts frequently co-occur with reward) rather than evidence that learned transfer decisions select the right facts. This directly threatens the paper's central claim that learned transfer decisions outperform symbolic baselines.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies explicit short-term-to-long-term memory transfer in a temporal knowledge-graph POMDP. It frames each observed RDF triple as a keep/drop decision before long-term insertion and learns these decisions with a shared-parameter per-item Q-network. To handle variable-sized short-term sets, the authors introduce a 'practical' TD update that matches items across consecutive steps by stochastic index-wise pairing. On the RoomKG benchmark at long-term memory capacity 128, the best learned variant (GCN+Local-STM) reaches 38.920±3.090 on held-out test QA score, compared with 31.960±1.255 for the strongest symbolic baseline (Novel-Only) and lower scores for LSTM/Transformer baselines. The paper also reports a step-level behavioral analysis indicating that the learned policy prefers at_location facts and drops directional map links.","tokens_in":14138,"tokens_out":11027,"duration_ms":129917,"significance":"If the results hold, the paper contributes a useful, interpretable neuro-symbolic mechanism for memory transfer, with controlled comparisons, open-source code, and explicit recognition of its scope. The per-item Q-learning design and the auditable keep/drop behavior are of interest. However, the central methodological contribution—the TD matching—is acknowledged in §7 as 'practical rather than theoretically complete,' and it is exactly the component that must make the learned policy meaningful. Without a corrected algorithm or a convincing control, the reported gains remain suggestive rather than established.","major_comments":[{"comment":"The TD target for matched item j is y_{b,j}=r_b+γ(1−d_b) max_a Q_{θ̄}(M_{b+1},j,a). Since M_short is randomly ordered, the j-th current item and the j-th next-state item are generally unrelated. In expectation over the stochastic matching, the target for each current item is r_b+γ times the average next-item max-Q, which is independent of the current item's identity. Hence the per-item loss cannot provide an item-specific keep/drop learning signal. The paper calls this 'practical rather than theoretically complete' (§7) but provides no sensitivity analysis. This threatens the central claim and the behavioral interpretation in §5.2. Please fix the matching or provide a control experiment showing that random pairing is benign (e.g., identity-based matching on persistent triples, or a random-target ablation).","section":"§3.3, Algorithm 1"},{"comment":"The behavioral analysis is based on a single trained model (stated at the start of §5.2). Given the TD-matching issue above, the observed keep/drop pattern (e.g., at_location kept 98/100 times, directional links mostly dropped) could reflect a global keep-rate bias or initialization rather than learned per-item values. Please report per-seed statistics and, ideally, show a correlation between the trained Q-values and the reported keep/drop decisions.","section":"§5.2"}],"minor_comments":[{"comment":"With n=5 seeds and the best result's standard deviation (±3.090), the margin over Novel-Only is about 4.7 standard errors, but a formal paired significance test and/or per-seed results would make the comparison more robust.","section":"Table 1, §4.1"},{"comment":"The phrase 'aligned default training settings used in prior work' is vague; please specify the exact training budgets and hyperparameters for the LSTM/Transformer baselines, or point to a table/appendix with these details.","section":"§4.2"},{"comment":"The collapsed memory-state plot with counts is difficult to verify; consider a supplementary table listing kept/dropped fact counts by relation type, ideally averaged over all seeds.","section":"Figure 1"},{"comment":"The loss L(θ) is defined but not numbered; number it and refer to it from §3.3 for clarity.","section":"Algorithm 1"},{"comment":"Minor formatting issue: 'V ossen' in the Kim et al. (2023) entry has an extra space. Please also check consistency of arXiv/URL formatting.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent about its limitations and provides code, but the TD-matching flaw is central to the method. The authors' own caveat in §7 does not resolve the issue; a revision must either correct the algorithm or provide a strong empirical control (e.g., identity matching or random-target ablation). I would not support acceptance in the current form. The benchmark is the authors' own; while the evaluation is controlled, the provenance is clearly cited, which is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper treats short-term-to-long-term memory transfer as a per-triple keep/drop RL decision in a temporal KG environment. That framing is new relative to the RoomKG prior work and to latent-memory baselines, and the empirical claim is supported at face value: on the held-out split at capacity 128, the learned GCN+Local-STM policy gets 38.9 vs 31.96 for the best symbolic baseline, with non-overlapping error bars. The paper also ships code and gives a detailed behavioral analysis showing the policy keeps navigation/query-relevant at_location facts and drops many direction links. That's real evidence and the authors deserve credit for the honest reporting including a transparent n=5 budget.\n\nThe main soft spot is the TD update over matched items. Since short-term sets have no canonical order, the bootstrap target pairs each item with a randomly co-indexed item in the next state; the target is not the future value of the same fact. The authors call this 'practical rather than theoretically complete' in §7, and that is an accurate description, but it leaves an unquantified risk: the learned Q-values could be shaped by alignment noise rather than by the true transfer value. I don't think it sinks the paper—the greedy policy still behaves sensibly, and the margin over Novel-Only is fairly large relative to the variance—but a referee should ask for an ablation that replaces the stochastic matching with something more identity-aware, or at least a sensitivity analysis.\n\nTwo more moderate concerns. The baselines do not include the closest related work the paper cites (Eberhard et al. 2025, Icarte et al. 2020); those are memory-action formulations, and a comparison would strengthen the claim. And the evaluation is a single environment at a single capacity. The paper explicitly scopes itself as a mechanism-validation study, so I don't hold that against it, but it does limit how far the conclusion generalizes.\n\nOverall: for someone working on memory management in neuro-symbolic RL, this is a useful and readable contribution. It deserves a serious peer review, and I'd like to see the matching issue addressed head-on. My own verdict would be 'conditional accept after revision,' not a desk reject.","headline":"A plausible mechanism study of explicit keep/drop memory transfer in RoomKG, with a real methodological caveat in the TD matching that the authors acknowledge; worth refereeing.","tokens_in":14616,"tokens_out":3813,"would_cite":true,"duration_ms":43245,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Learned keep/drop decisions for knowledge-graph memory beat fixed symbolic rules and sequence-memory baselines on RoomKG at capacity 128.","keywords":["neuro-symbolic reinforcement learning","partial observability","knowledge-graph memory","memory transfer","long-term memory","per-item Q-learning","RoomKG","temporal knowledge graphs"],"falsifier":"Run the same RoomKG configuration but replace the index-wise matched TD update with an identity-aware update that aligns short-term triples by their content across consecutive steps; if the held-out QA gap changes materially, the stochastic index-wise matching is a load-bearing part of the result, and if it stays the same, the matching is benign.","tokens_in":13620,"feed_emoji":"🧠","tokens_out":8690,"duration_ms":90222,"temperature":0.7,"pith_summary":"Under partial observability, an agent sees only a local slice of the world each step, so what it stores in long-term memory determines what it can later answer. This paper argues that short-term-to-long-term memory transfer should be an explicit learned decision: for every observed fact (a knowledge-graph triple like 'agent at location'), the agent chooses keep or drop before insertion into a capacity-limited memory. Because the number of candidate facts changes every step, the method scores each fact independently with shared parameters — a per-item Q-learning design — and trains it with a matched temporal-difference update across consecutive steps. On the RoomKG benchmark (a grid-world knowledge-graph environment) at long-term capacity 128, the learned policy outperforms symbolic transfer heuristics and sequence-based neural baselines, with the best variant scoring 38.9 ± 3.1 versus 32.0 ± 1.3 on held-out question answering. The policy's behavior is interpretable: it keeps navigation- and query-relevant facts and drops most room-direction links.","feed_headline":"Learned keep/drop memory choices beat fixed rules by 7 QA points","feed_subtitle":"In a memory-limited knowledge-graph task, an RL agent learns which facts to keep and beats fixed baselines.","key_machinery":"The mechanism is a per-item Q-learning design for variable-cardinality short-term memory: a shared-parameter Q-network (with a graph-neural-network encoder over the current memory graph) maps each short-term triple to a two-element keep/drop Q-vector, so the transfer action dimension adapts to the number of observed facts at each step. The matching temporal-difference update is the training device that makes this work: since short-term items form a set with no stable order, consecutive transitions are paired index-wise up to the smaller set size after stochastic shuffling, and the TD loss is averaged over matched items. This gives dense off-policy supervision without a fixed action index, at","core_discovery":"The paper's central claim is that transfer of symbolic facts from short-term to long-term memory should be treated as an explicit reinforcement-learning decision, and that per-item keep/drop choices outperform both fixed symbolic heuristics and opaque sequence-memory models in a memory-constrained POMDP. The method parameterizes one Q-function shared across items: for each short-term triple it outputs keep/drop Q-values, and variable-sized short-term buffers are handled by matching consecutive transitions index-wise after stochastic shuffling. On RoomKG with long-term capacity 128, the best configuration (a graph-convolutional encoder with a local short-term-only policy) scores 38.920±3.090","pith_inferences":["This suggests that the index-wise matched TD update, which the authors call practical rather than theoretically complete, is a candidate source of bias; an identity-aware matching (pairing the same triple across steps) is a direct way to test how much of the gain is alignment-driven.","A broader design principle follows: in memory-constrained POMDPs, explicit per-item decisions over symbolic facts may combine the sample efficiency of learned policies with the interpretability of symbolic memory, which could transfer to dialogue state tracking or open-world navigation.","Because the QA, exploration, and eviction policies are fixed, the learned transfer policy is optimized for those particular downstream components; joint end-to-end training of transfer with the other memory components might shift what the optimal keep/drop policy looks like.","The RoomKG result at capacity 128 leaves open whether the advantage persists at smaller or larger capacities; testing a capacity sweep would map where learned transfer decisions matter most."],"forward_implications":["Learned keep/drop transfer improves held-out QA scores over fixed transfer heuristics (novel-only, always-transfer, random) at long-term memory capacity 128.","The best learned variant uses only short-term input with local per-item decisions, showing that in this regime simple local transfer beats pooled global decisions.","Fact-level analysis of the trained policy reveals a stable, interpretable strategy — keep navigation- and query-relevant facts, drop most map-direction links — so memory management can be audited at the triple level.","With non-transfer components held fixed, the performance gap is attributable to the transfer policy itself rather than to QA, exploration, or eviction choices."],"fun_headline_variants":["RL learns which facts to keep for KG memory, beats fixed rules","Memory-tight KG? Learn keep/drop, gain 7 QA points","Agent chooses what to remember, outperforms static baselines","Keep/drop as RL decision: 7-point win in knowledge-graph QA","Forget wisely: RL memory policy beats heuristics in KG QA"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the training procedure aligns a short-term fact at one step with a fact at the next step by position after shuffling rather than by identity; if those two positions often hold different facts, the learned keep/drop values are driven by alignment noise, and the authors themselves note this matching is 'practical rather than theoretically complete.'","fun_headline_variants_meta":{"raw":{"variants":["RL learns which facts to keep for KG memory, beats fixed rules","Memory-tight KG? Learn keep/drop, gain 7 QA points","Agent chooses what to remember, outperforms static baselines","Keep/drop as RL decision: 7-point win in knowledge-graph QA","Forget wisely: RL memory policy beats heuristics in KG QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1183,"prompt_tokens":706,"completion_tokens":477,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":383}},"tokens_in":450,"tokens_out":477,"duration_ms":5992,"temperature":1.0,"reasoning_tokens":383,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T13:23:06.170013+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same RoomKG configuration but replace the index-wise matched TD update with an identity-aware update that aligns short-term triples by their content across consecutive steps; if the held-out QA gap changes materially, the stochastic index-wise matching is a load-bearing part of the result, and if it stays the same, the matching is benign.","supporting_citations":[],"review_version":3}