{"id":"72c70914-dab2-422a-83fe-6fcee3d05374","arxiv_id":"2607.19397","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Replacing the hard target-copy in DQN with a Q-value-sensitivity-weighted merge of the last K network copies yields competitive Atari performance, but only marginally beats architecture-matched baselines.","lead":"This paper proposes a new way to update the target network in Deep Q-Networks, blending the recent weights of the network based on which parameters matter most for Q-values instead of copying the newest weights outright. The method matches or slightly beats strong baselines on Atari, though most improvements are small and the architecture comparisons are not fully controlled.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control isolates the sensitivity-weighted merge: without uniform-parameter-averaging/Polyak baselines, the reported gains do not establish that Q-value sensitivity weighting (Eq. 7) — rather than any parameter smoothing — drives MM-DQN's improvements.","rationale":"The reader's verdict is CONDITIONAL, and my concern does not move it to a different category: it sharpens one reason for conditionality. The reader's named weakest assumption was copy compatibility and the unvalidated N=32 sensitivity proxy; my load-bearing concern is the missing uniform-weight/Polyak control, which is the most direct way to test whether Eq. (7)'s sensitivity weighting is actually responsible for the reported gains. This is related but not identical, hence partial agreement. The concern is not internal inconsistency: Eqs. (7)-(8) are well specified and matched to the stated rule. The issue is underdetermination of the central mechanism claim. The paper does report a memory-size ablation and a descriptive Q-value/return diagnostic, but neither removes sensitivity weighting while keeping the merge structure fixed. I also note that the abstract's 'beats' language is stronger than the pairwise counts support, especially against PQN (28-26) and DQN+LayerNorm (29-25), but this is a presentation issue; the missing control is the substantive gap. If the proposed uniform-weight/Polyak control were run and MM-DQN clearly dominated, the mechanism claim would be substantially strengthened; if not, the paper would need to reposition its contribution as 'parameter averaging helps' rather than 'sensitivity-weighted merging helps.' Therefore the conditional verdict should stand, pending this control and ideally significance testing on the 54-game counts.","tokens_in":13655,"tokens_out":3890,"duration_ms":47263,"concrete_test":"Run the exact MM-DQN protocol on the 54-game suite (or a pre-registered 20-game subset) with Eq. (8) modified so W^{(k)}_{ℓ,j}=1 for all k,ℓ,j (uniform parameter averaging with the same λ=0.1 and K=3), and also with a Polyak soft update whose smoothing is matched to the equivalent effective horizon. If uniform averaging or Polyak matches MM-DQN's win counts and mean rank, the sensitivity weighting is not the active ingredient; if MM-DQN clearly dominates both, the mechanism claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"MM-DQN's only departure from a hard target copy is Eq. (8), and the novel ingredient is the per-parameter sensitivity W of Eq. (7). Yet none of the reported baselines shares the parameter-merge structure with uniform weights or a soft (Polyak) update. Averaged DQN averages Q-value predictions, not parameters; DQN+LayerNorm and vanilla DQN use hard copies; PQN has no target network. Thus the positive results over DQN cannot be attributed to sensitivity weighting: any K-copy parameter average with a recency prior could plausibly smooth bootstrap targets and improve stability, and Eq. (8) with W≡1 is exactly such an average. The paper's mechanism claim—preserving useful value-function structure—is supported only by the descriptive Q-value/return plot of Figure 4 over six games, not by an ablation that removes the sensitivity weights. This is a load-bearing gap because the central conceptual contribution and the title's 'Sensitivity Weighted' claim rest on W mattering, not merely on parameter averaging being helpful. The pairwise counts against PQN (28-26) and the matched-architecture baseline (29-25) are near chance, so the decisive evidence for sensitivity weighting would need to come from such a control; it is absent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Memory Merge DQN (MM-DQN), an alternative target-network update for DQN. Rather than periodically copying the newest online network into the target, the method stores the last K online copies and merges them per-parameter using Q-value sensitivity weights (Eq. 7) plus a recency prior constant (Eq. 8). The authors evaluate MM-DQN on 54 Atari games for 10M steps with four matched seeds against vanilla DQN, Averaged DQN, DQN with layer normalisation, and PQN with gradient clipping. They report game-level rankings, pairwise win counts, large-improvement counts, a memory-size ablation on two games, and a Q-value-versus-return calibration diagnostic. The central claim is that sensitivity-weighted parameter merging stabilises learning and improves final performance, especially by preserving useful value-function structure that hard target copies discard.","tokens_in":13842,"tokens_out":6780,"duration_ms":78680,"significance":"If the central claim were established, the paper would show that target-network update rules are a meaningful, underused design lever in value-based reinforcement learning, and it would connect importance-weighted parameter merging ideas from supervised learning to bootstrapped value learning. The paper has genuine strengths: it avoids raw cross-game mean scores in favour of per-game rankings, uses carefully matched seeds across algorithms and ablations, provides a clear pseudocode description, reports a memory-size ablation, and adds a Q-value/return calibration diagnostic. It also states the important implementation difference (PQN receiving gradient clipping) explicitly. However, the current evidence does not isolate the proposed sensitivity-weighting mechanism, and the headline comparative claims are not supported by the reported statistics. The missing uniform-weight/Polyak control is a load-bearing gap that the paper would need to fill before the title's 'Sensitivity Weighted' claim can be accepted.","major_comments":[{"comment":"The abstract claims MM-DQN 'beats DQN, Averaged DQN, and PQN (with gradient clipping)', but the reported data do not support this against PQN. Table 2 shows MM-DQN vs PQN only 28–26, which is within chance under any paired sign test; no significance testing is reported. Table 1 shows PQN has the better mean rank (2.54 vs 2.65) and far more Top-2 finishes (34 vs 28). With only four seeds per game, pairwise game counts are also noisy. The paper should provide paired statistical comparisons (bootstrap or permutation over games/seeds) and, unless those tests reject equality, the text should describe the result as 'competitive' rather than 'beats'.","section":"Abstract, §4.1, Tables 1–2"},{"comment":"The strong pairwise result against vanilla DQN (41–13) is confounded with architecture. Vanilla DQN uses ReLU and no layer normalisation, while MM-DQN uses GELU and layer normalisation. The 41–13 split therefore cannot be attributed to the merge update. The matched-architecture baseline is DQN with layer normalisation, and against that baseline the pairwise count is only 29–25, again without significance testing. The paper should treat DQN+LayerNorm as the primary architecture-matched control and report the uncertainty of that comparison.","section":"§3.2, Table 5, Table 2"},{"comment":"No experiment isolates the sensitivity weighting itself. The only algorithmic difference from a hard target copy is Eq. (8); setting W≡1 in Eq. (8) yields a plain K-copy parameter average with a recency prior. None of the reported baselines shares this structure: Averaged DQN averages Q-value predictions rather than parameters, DQN and DQN+LayerNorm use hard copies, and PQN has no target network. Thus the positive results could be explained by any form of parameter smoothing. Adding a uniform-weight merge baseline (W≡1) and, ideally, a Polyak soft-target baseline is necessary to support the paper's central claim that Q-value sensitivity weighting is what matters. This is not an optional robustness check; it is the decisive control for the title and the main contribution.","section":"§3.1, Eq. (7)–(8); §4.1, Table 2"},{"comment":"Two assumptions are acknowledged but never measured. First, the text states that 'parameter merging is meaningful only when the stored networks remain sufficiently compatible', but no evidence is provided that consecutive 1K-update copies are in fact compatible under the chosen architecture. Second, the sensitivity estimate uses N=32 replay-buffer next states and anchor actions from the newest copy; this is an unvalidated proxy for parameter importance over the whole target interval. The paper should report sensitivity of the results to N and to the sample set, or compare Eq. (7) with alternative importance measures. Without this, the mechanism story in §4.2 remains illustrative rather than supported.","section":"§3.2, Eq. (5)–(7); Algorithm 1"}],"minor_comments":[{"comment":"The DQN+LayerNorm row appears malformed ('6 11.1% 2643 12.63'); the columns for Top 3 and Last place are not aligned. Please regenerate the table so that all counts are legible.","section":"Table 1"},{"comment":"The diagnostic would be reproducible only if the number of samples, the way 'average predicted Q-value' is computed, and whether points pool seeds or time steps are specified. Please add these details to the caption or text.","section":"Figure 4"},{"comment":"There are minor language issues: 'applying this across into actor-critic methods' is ungrammatical, and §3.2 contains a doubled period after 'transitions..'.","section":"§5"},{"comment":"The '≥50% better than DQN' criterion is not defined for games with negative or near-zero DQN scores. Please state the formula (e.g., relative to absolute DQN score) or exclude such games.","section":"Table 3"},{"comment":"Reference [9] is cited as a workshop paper; if a peer-reviewed version exists, it would be helpful to cite that version as well.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is clearly written and the underlying idea is worth investigating, but the empirical case is currently too weak for the abstract's claims. The decisive missing experiment is a uniform-weight (or Polyak) parameter-merge control; without it, the 'sensitivity weighted' contribution is not established. I would be willing to review a revised version that adds that ablation, reports paired significance tests, and appropriately tempers the PQN comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is clean: replace the hard target copy with a sensitivity-weighted parameter merge over recent online copies. Eq. 8 is simple, well-defined, and the paper is transparent about its evaluation. The problem is that the empirical claims outrun the evidence.\n\nWhat's genuinely new: Averaged DQN averages Q-values, SWA averages weights without sensitivity weighting, and nobody has used Q-value sensitivity to construct a target network. The paper also does a nice job of reporting per-game rankings and pairwise counts rather than relying on raw score averages. The ablation on K and the discussion of layer normalization as a compatibility mechanism show clear thinking.\n\nBut the headline \"beats PQN\" is not supported: 28-26 out of 54 games is essentially chance, and PQN actually has a better mean rank and more Top-2 finishes. The 41-13 against vanilla DQN is confounded by the addition of GELU and layer normalization; the matched-architecture control is only 29-25. There is no significance testing, only four seeds, and K=3 was chosen on two games that are part of the 54-game evaluation, so the default configuration is partially fitted to the test set. Most importantly, no baseline shares the parameter-merge structure with uniform weights or a Polyak update, so the \"sensitivity weighted\" part of the contribution is not isolated. The mechanism claim—preserving useful value structure—rests on a descriptive Q-value vs. return plot over six games; it's suggestive, not a falsifiable test. The paper itself concedes that parameter merging depends on stored networks remaining compatible, but it never measures that compatibility.\n\nNone of this is fatal; it's all addressable. Add significance testing, run uniform-weight and Polyak controls, tune K on a held-out set, and release code and per-seed scores. With those changes, the idea could be a real contribution to target-network design.\n\nThis paper deserves a serious referee—it's a well-posed idea with honest reporting—but it needs major revisions before the central claim can be accepted.","headline":"Sensible, well-specified target-merge idea that is currently oversold: the pairwise evidence against the strongest baselines is near chance and no ablation isolates sensitivity weighting.","tokens_in":14497,"tokens_out":1735,"would_cite":false,"duration_ms":19917,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing the hard target copy in DQN with a sensitivity-weighted merge of recent network copies improves final performance across 54 Atari games and gives Memory Merge DQN the most first-place results.","keywords":["deep reinforcement learning","DQN","target networks","Q-learning","parameter merging","weight averaging","Atari","value-based reinforcement learning"],"falsifier":"Run the same 54-game evaluation with Eq. (8) but with the sensitivity weights replaced by uniform constants, keeping the same stored copies, anchor actions, and recency prior; if uniform merging matches or beats Memory Merge DQN, then the Q-value sensitivity signal is not what drives the reported gains.","tokens_in":13364,"feed_emoji":"🎮","tokens_out":6207,"duration_ms":64354,"temperature":0.7,"pith_summary":"Deep Q-network agents stabilise learning by holding a fixed target network and periodically copying the online network into it. This paper argues that those hard copies throw away useful value-function structure and that the target should instead be built by merging several recent copies of the online network, weighting each parameter by how strongly it affects current Q-values. The proposed Memory Merge DQN keeps a short rolling memory of network copies and produces the next target as a sensitivity-weighted average, with a small recency bias toward the newest copy. Across 54 Atari games it wins 21 games, beats vanilla DQN on 41, and edges out both an averaged-Q baseline and a strong parallel Q-learning baseline on pairwise counts. If the result holds, target network design deserves attention as an active mechanism for preserving value structure, not just a stability patch.","feed_headline":"Merging past DQN weights beats hard target copies in 54 Atari games","feed_subtitle":"A target built from merged recent networks takes 21 first-place scores and preserves useful value structure.","key_machinery":"The central object is the sensitivity-weighted parameter merge used to build the target network. Instead of setting target parameters to the newest online copy, the update combines K stored copies with per-parameter weights from Eq. (7): each weight is the average, over N=32 sampled next states, of the squared gradient of Q with respect to that parameter, evaluated after fixing the greedy action chosen by the newest copy. The merged target is a weighted average, given in Eq. (8), with a small recency prior that keeps the target near the newest parameters. Fixing the anchor action before computing gradients is what makes weights comparable across copies; layer normalisation is invoked as the","core_discovery":"The paper claims that the target network in DQN does not have to be a copy of the newest online parameters, and that replacing hard copying with a merge of recent copies improves both stability and final performance. Memory Merge DQN stores the K most recent online networks, samples a small set of non-terminal next states from the replay buffer, and lets the newest copy choose the greedy action at those states. For every stored copy, each parameter is scored by the average squared gradient of that copy's Q-value at those anchored actions; this is the copy's Q-value sensitivity. The next target parameter is the weighted average of the stored copies' parameters, with these sensitivities as wei","pith_inferences":["The paper leaves the compatibility assumption untested; a direct measure of representation alignment between stored copies would let practitioners identify when Memory Merge should be disabled.","A natural extension is to apply the same sensitivity-weighted merge to actor-critic soft updates, where scalar Polyak averaging currently ignores per-parameter importance.","The large gains in sparse-reward games suggest a possible interaction with primacy bias: preserving early useful structure may be a softer alternative to plasticity resets."],"forward_implications":["Target network updates are a design lever for value-based RL: the choice of how the target is built affects final performance, not just training stability.","On 54 Atari games, Memory Merge DQN gets 21 first-place final scores, beats vanilla DQN on 41 games, and has 28 pairwise wins versus 26 losses against PQN with gradient clipping.","Large wins occur in games with sparse or delayed rewards, such as Elevator Action and Skiing, where preserving useful value-function history appears most beneficial.","Memory size has an optimum around K=3; very long memories can hurt final performance in some games, showing that retaining too much history can become counterproductive.","The Q-value versus return plots suggest the merge preserves high value estimates in games where they are behaviourally justified, rather than simply suppressing Q-value scale."],"fun_headline_variants":["Merge recent DQN weights, not just copy latest, for Atari","Q-sensitivity weighted target merges stabilize DQN learning","Memory Merge DQN: recency plus Q-sensitivity beats hard copies","Weighted merging of recent DQN targets improves Atari scores","DQN target update: merge recent params by Q-sensitivity"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that successive online-network copies are compatible enough that a parameter-level merge produces a well-calibrated target; the paper relies on layer normalisation for this but never measures copy-to-copy compatibility.","fun_headline_variants_meta":{"raw":{"variants":["Merge recent DQN weights, not just copy latest, for Atari","Q-sensitivity weighted target merges stabilize DQN learning","Memory Merge DQN: recency plus Q-sensitivity beats hard copies","Weighted merging of recent DQN targets improves Atari scores","DQN target update: merge recent params by Q-sensitivity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001336,"raw_usage":{"total_tokens":5315,"prompt_tokens":839,"completion_tokens":4476,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":4383}},"tokens_in":583,"tokens_out":4476,"duration_ms":32903,"temperature":1.0,"reasoning_tokens":4383,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T08:45:26.392196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 54-game evaluation with Eq. (8) but with the sensitivity weights replaced by uniform constants, keeping the same stored copies, anchor actions, and recency prior; if uniform merging matches or beats Memory Merge DQN, then the Q-value sensitivity signal is not what drives the reported gains.","supporting_citations":[],"review_version":1}