Pith. sign in

REVIEW 4 major objections 5 minor 23 references

Memory Merge DQN: Sensitivity Weighted Target Updates for Stable Value Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Replacing the hard target copy in DQN with a sensitivity-weighted merge of recent network copies improves final performance across 54 Atari games and gives Memory Merge DQN the most first-place results.

desk verdict Sensible, well-specified target-merge idea that is currently oversold: the pairwise evidence against the strongest baselines is near chance and no ablation isolates sensitivity weighting. read the letter →

arxiv 2607.19397 v1 pith:ZNTGZ7IR submitted 2026-07-04 cs.LG

Adrian Ly , Richard Dazeley , Peter Vamplew , Sunil Aryal , Francisco Cruz This is my paper
classification cs.LG
keywords deepreinforcementlearningDQNtargetnetworksQ-learningparametermergingweightaveragingAtarivalue-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deep Q-network agents stabilise learning by holding a fixed target network and periodically copying the online network into it. This paper argues that those hard copies throw away useful value-function structure and that the target should instead be built by merging several recent copies of the online network, weighting each parameter by how strongly it affects current Q-values. The proposed Memory Merge DQN keeps a short rolling memory of network copies and produces the next target as a sensitivity-weighted average, with a small recency bias toward the newest copy. Across 54 Atari games it wins 21 games, beats vanilla DQN on 41, and edges out both an averaged-Q baseline and a strong parallel Q-learning baseline on pairwise counts. If the result holds, target network design deserves attention as an active mechanism for preserving value structure, not just a stability patch.

What carries the argument

The central object is the sensitivity-weighted parameter merge used to build the target network. Instead of setting target parameters to the newest online copy, the update combines K stored copies with per-parameter weights from Eq. (7): each weight is the average, over N=32 sampled next states, of the squared gradient of Q with respect to that parameter, evaluated after fixing the greedy action chosen by the newest copy. The merged target is a weighted average, given in Eq. (8), with a small recency prior that keeps the target near the newest parameters. Fixing the anchor action before computing gradients is what makes weights comparable across copies; layer normalisation is invoked as the

What would settle it

Run the same 54-game evaluation with Eq. (8) but with the sensitivity weights replaced by uniform constants, keeping the same stored copies, anchor actions, and recency prior; if uniform merging matches or beats Memory Merge DQN, then the Q-value sensitivity signal is not what drives the reported gains.

Watch

Extended reading notes

Core claim

The paper claims that the target network in DQN does not have to be a copy of the newest online parameters, and that replacing hard copying with a merge of recent copies improves both stability and final performance. Memory Merge DQN stores the K most recent online networks, samples a small set of non-terminal next states from the replay buffer, and lets the newest copy choose the greedy action at those states. For every stored copy, each parameter is scored by the average squared gradient of that copy's Q-value at those anchored actions; this is the copy's Q-value sensitivity. The next target parameter is the weighted average of the stored copies' parameters, with these sensitivities as wei

Load-bearing premise

The load-bearing premise is that successive online-network copies are compatible enough that a parameter-level merge produces a well-calibrated target; the paper relies on layer normalisation for this but never measures copy-to-copy compatibility.

Editorial extensions

If this is right

  • Target network updates are a design lever for value-based RL: the choice of how the target is built affects final performance, not just training stability.
  • On 54 Atari games, Memory Merge DQN gets 21 first-place final scores, beats vanilla DQN on 41 games, and has 28 pairwise wins versus 26 losses against PQN with gradient clipping.
  • Large wins occur in games with sparse or delayed rewards, such as Elevator Action and Skiing, where preserving useful value-function history appears most beneficial.
  • Memory size has an optimum around K=3; very long memories can hurt final performance in some games, showing that retaining too much history can become counterproductive.
  • The Q-value versus return plots suggest the merge preserves high value estimates in games where they are behaviourally justified, rather than simply suppressing Q-value scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves the compatibility assumption untested; a direct measure of representation alignment between stored copies would let practitioners identify when Memory Merge should be disabled.
  • A natural extension is to apply the same sensitivity-weighted merge to actor-critic soft updates, where scalar Polyak averaging currently ignores per-parameter importance.
  • The large gains in sparse-reward games suggest a possible interaction with primacy bias: preserving early useful structure may be a softer alternative to plasticity resets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Memory Merge DQN (MM-DQN), an alternative target-network update for DQN. Rather than periodically copying the newest online network into the target, the method stores the last K online copies and merges them per-parameter using Q-value sensitivity weights (Eq. 7) plus a recency prior constant (Eq. 8). The authors evaluate MM-DQN on 54 Atari games for 10M steps with four matched seeds against vanilla DQN, Averaged DQN, DQN with layer normalisation, and PQN with gradient clipping. They report game-level rankings, pairwise win counts, large-improvement counts, a memory-size ablation on two games, and a Q-value-versus-return calibration diagnostic. The central claim is that sensitivity-weighted parameter merging stabilises learning and improves final performance, especially by preserving useful value-function structure that hard target copies discard.

Significance. If the central claim were established, the paper would show that target-network update rules are a meaningful, underused design lever in value-based reinforcement learning, and it would connect importance-weighted parameter merging ideas from supervised learning to bootstrapped value learning. The paper has genuine strengths: it avoids raw cross-game mean scores in favour of per-game rankings, uses carefully matched seeds across algorithms and ablations, provides a clear pseudocode description, reports a memory-size ablation, and adds a Q-value/return calibration diagnostic. It also states the important implementation difference (PQN receiving gradient clipping) explicitly. However, the current evidence does not isolate the proposed sensitivity-weighting mechanism, and the headline comparative claims are not supported by the reported statistics. The missing uniform-weight/Polyak control is a load-bearing gap that the paper would need to fill before the title's 'Sensitivity Weighted' claim can be accepted.

major comments (4)
  1. [Abstract, §4.1, Tables 1–2] The abstract claims MM-DQN 'beats DQN, Averaged DQN, and PQN (with gradient clipping)', but the reported data do not support this against PQN. Table 2 shows MM-DQN vs PQN only 28–26, which is within chance under any paired sign test; no significance testing is reported. Table 1 shows PQN has the better mean rank (2.54 vs 2.65) and far more Top-2 finishes (34 vs 28). With only four seeds per game, pairwise game counts are also noisy. The paper should provide paired statistical comparisons (bootstrap or permutation over games/seeds) and, unless those tests reject equality, the text should describe the result as 'competitive' rather than 'beats'.
  2. [§3.2, Table 5, Table 2] The strong pairwise result against vanilla DQN (41–13) is confounded with architecture. Vanilla DQN uses ReLU and no layer normalisation, while MM-DQN uses GELU and layer normalisation. The 41–13 split therefore cannot be attributed to the merge update. The matched-architecture baseline is DQN with layer normalisation, and against that baseline the pairwise count is only 29–25, again without significance testing. The paper should treat DQN+LayerNorm as the primary architecture-matched control and report the uncertainty of that comparison.
  3. [§3.1, Eq. (7)–(8); §4.1, Table 2] No experiment isolates the sensitivity weighting itself. The only algorithmic difference from a hard target copy is Eq. (8); setting W≡1 in Eq. (8) yields a plain K-copy parameter average with a recency prior. None of the reported baselines shares this structure: Averaged DQN averages Q-value predictions rather than parameters, DQN and DQN+LayerNorm use hard copies, and PQN has no target network. Thus the positive results could be explained by any form of parameter smoothing. Adding a uniform-weight merge baseline (W≡1) and, ideally, a Polyak soft-target baseline is necessary to support the paper's central claim that Q-value sensitivity weighting is what matters. This is not an optional robustness check; it is the decisive control for the title and the main contribution.
  4. [§3.2, Eq. (5)–(7); Algorithm 1] Two assumptions are acknowledged but never measured. First, the text states that 'parameter merging is meaningful only when the stored networks remain sufficiently compatible', but no evidence is provided that consecutive 1K-update copies are in fact compatible under the chosen architecture. Second, the sensitivity estimate uses N=32 replay-buffer next states and anchor actions from the newest copy; this is an unvalidated proxy for parameter importance over the whole target interval. The paper should report sensitivity of the results to N and to the sample set, or compare Eq. (7) with alternative importance measures. Without this, the mechanism story in §4.2 remains illustrative rather than supported.
minor comments (5)
  1. [Table 1] The DQN+LayerNorm row appears malformed ('6 11.1% 2643 12.63'); the columns for Top 3 and Last place are not aligned. Please regenerate the table so that all counts are legible.
  2. [Figure 4] The diagnostic would be reproducible only if the number of samples, the way 'average predicted Q-value' is computed, and whether points pool seeds or time steps are specified. Please add these details to the caption or text.
  3. [§5] There are minor language issues: 'applying this across into actor-critic methods' is ungrammatical, and §3.2 contains a doubled period after 'transitions..'.
  4. [Table 3] The '≥50% better than DQN' criterion is not defined for games with negative or near-zero DQN scores. Please state the formula (e.g., relative to absolute DQN score) or exclude such games.
  5. [References] Reference [9] is cited as a workshop paper; if a peer-reviewed version exists, it would be helpful to cite that version as well.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the MM-DQN merge rule (Eqs. 7-8) is a stated heuristic evaluated against external Atari benchmarks with matched seeds; no load-bearing self-citation or definitional reduction found.

full rationale

The paper's derivation chain contains no step in which a claimed prediction reduces to its own inputs. Eq. (7) defines Q-value sensitivity as a squared-gradient statistic on a 32-sample replay set, and Eq. (8) defines the target as a sensitivity-weighted average of K recent online copies with a fixed recency prior lambda=0.1; both are honestly presented as a heuristic ('The method is inspired by Fisher Weight Model Merging, but uses Q-value sensitivity rather than Fisher information as the weighting signal'). The claimed outcomes - first-place counts, pairwise wins, and large improvements over DQN across 54 Atari games - are measured externally and are not entailed by Eqs. (7)-(8); no fitted parameter is later renamed as a prediction, and there is no identity such as theta^- = y_t. The reference list contains no work by the present authors, so no self-citation chain is load-bearing and no 'uniqueness theorem' is imported; the only imported ingredient (Fisher-weighted merging, ref. [14]) is openly acknowledged as inspiration rather than proof. The genuinely soft points are validity concerns, not circularity: (i) K=3 is chosen via an ablation on Zaxxon and River Raid, which are then in the 54-game evaluation set, a mild hyperparameter-selection leak; (ii) no uniform-parameter-average or Polyak baseline is run, so the specific claim that sensitivity weighting (rather than any parameter smoothing) drives the gains is underdetermined - the 29-25 and 28-26 pairwise margins against DQN+LayerNorm and PQN are near chance; and (iii) Figure 4's Q-value-versus-return plots are descriptive evidence for the mechanism, not a causal test. None of these makes the central result equivalent to its inputs by construction, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on three hand-set hyperparameters (K=3, lambda=0.1, N=32), a compatibility premise delegated to layer normalization (stated by the authors), and an unvalidated gradient-sensitivity proxy for parameter importance. No new physical or mathematical entities are introduced. K was selected on two games that later appear in the 54-game evaluation, the main selection-bias item.

free parameters (3)
  • K (memory size) = 3
    Rolling number of stored online-network copies merged into the target. Selected as default from a 2-game ablation on Zaxxon and River Raid (§3.2.1), both of which are in the 54-game evaluation set (Table 4) — a mild selection leak.
  • lambda_l (recency prior per parameter group) = 0.1
    Extra weight given to the newest copy in Eq. (8). Paper states it was not tuned and is fixed across environments, but it is a hand-set constant that shapes every merged target.
  • N (sensitivity sample count) = 32
    Number of non-terminal next-state observations used to estimate the squared-gradient sensitivity weights in Eq. (7). Hand-set; no ablation or justification.
assumptions (4)
  • domain assumption Successive online copies remain representationally compatible enough that parameter averaging yields a calibrated target network
    Stated in §3.2 ('Parameter merging is meaningful only when the stored networks remain sufficiently compatible'); layer normalization is invoked as the mechanism enforcing this, but no measurement of copy compatibility is provided.
  • ad hoc to paper Squared Q-value gradients on N=32 sampled next-states are a reliable proxy for parameter importance
    Eq. (7) defines the entire merge weighting on this proxy; no validation that 32 samples suffice or that first-order sensitivity tracks long-horizon value-function importance.
  • domain assumption Anchoring all copies to the newest copy's greedy action makes sensitivity weights comparable across copies
    §3.1 argues this avoids conflating sensitivity with policy disagreement; it is a methodological choice with no theoretical support.
  • standard math Standard DQN machinery (replay buffer, 1K target interval, MSE TD loss) is a suitable substrate
    Eqs. (1)-(2) inherit standard DQN assumptions and convergence heuristics; not the paper's contribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Memory Merge DQN: Sensitivity Weighted Target Updates for Stable Value Learning." pith.science (2026). https://pith.science/paper/ZNTGZ7IR

@misc{pith2026260719397,
  author       = {Pith},
  title        = {Pith review of: Memory Merge DQN: Sensitivity Weighted Target Updates for Stable Value Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZNTGZ7IR}},
  note         = {Machine review of arXiv:2607.19397}
}
read the original abstract

Deep Q-networks use target networks to stabilise bootstrapped value learning, but the standard hard copy update also introduces a tradeoff. Holding the target network fixed, improves short term stability, yet each hard update abruptly replaces the target parameters with the newest online network and discards recent parameter history. This can produce sudden changes in the bootstrap target and may remove value function structure that remains useful later in training. This paper introduces Memory Merge DQN, a target network update mechanism that maintains a short memory of recent historical online network copies and constructs the target network by merging network parameters based on the Q-value sensitivity rather than copying only the newest online network. Memory Merge gives greater influence to parameters that remain locally important for current Q-value behaviour, while using a recency prior to keep the merged target close to the latest online parameters. The method is inspired by Fisher Weight Model Merging, but uses Q-value sensitivity rather than Fisher information as the weighting signal. This paper evaluates Memory Merge DQN on Atari environments against DQN, Averaged DQN, DQN with layer normalisation, and PQN (with gradient clipping). The results show that Memory Merge DQN is highly competitive and it achieves the largest number of first place final performance results among the evaluated methods, beats DQN, Averaged DQN, and PQN (with gradient clipping), and produces substantial gains in several games where preserving useful value-function parameters appears beneficial. These findings suggest that selectively merging recent parameter weights and history can improve the stability and final performance of DQN agents, and that target network design is an important mechanism for preserving useful value function structure during long horizon value learning.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 10 linked inside Pith

  1. [1]

    nature518(7540), 529–533 (2015)

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G.,et al.: Human-level control through deep reinforcement learning. nature518(7540), 529–533 (2015)

    2015

  2. [2]

    arXiv preprint arXiv:1312.5602 (2013)

    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)

    arXiv 2013

  3. [3]

    In: Learning for Dynamics and Control, pp

    Fan, J., Wang, Z., Xie, Y., Yang, Z.: A theoretical analysis of deep q-learning. In: Learning for Dynamics and Control, pp. 486–489 (2020). PMLR

    2020

  4. [4]

    arXiv preprint arXiv:1901.07510 (2019)

    Hernandez-Garcia, J.F., Sutton, R.S.: Understanding multi-step deep rein- forcement learning: A systematic study of the dqn target. arXiv preprint arXiv:1901.07510 (2019)

    arXiv 1901

  5. [5]

    arXiv preprint arXiv:1812.02648 (2018)

    Van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., Modayil, J.: Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02648 (2018)

    arXiv 2018

  6. [6]

    arXiv preprint arXiv:2204.09560 (2022)

    Lyle, C., Rowland, M., Dabney, W.: Understanding and preventing capacity loss in reinforcement learning. arXiv preprint arXiv:2204.09560 (2022)

    arXiv 2022

  7. [7]

    In: International Conference on Machine Learning, pp

    Sokar, G., Agarwal, R., Castro, P.S., Evci, U.: The dormant neuron phenomenon in deep reinforcement learning. In: International Conference on Machine Learning, pp. 32145–32168 (2023). PMLR

    2023

  8. [8]

    In: International Conference on Machine Learning, pp

    Nikishin, E., Schwarzer, M., D’Oro, P., Bacon, P.-L., Courville, A.: The primacy bias in deep reinforcement learning. In: International Conference on Machine Learning, pp. 16828–16847 (2022). PMLR

    2022

Show all 23 references
  1. [9]

    In: Uncertainty in Artificial Intelligence Workshop on Uncertainty in Deep Learning (2018)

    Nikishin, E., Izmailov, P., Athiwaratkun, B., Podoprikhin, D., Garipov, T., Shvechikov, P., Vetrov, D., Wilson, A.G.: Improving stability in deep reinforce- ment learning with weight averaging. In: Uncertainty in Artificial Intelligence Workshop on Uncertainty in Deep Learning (2018)

    2018

  2. [10]

    arXiv preprint arXiv:1509.02971 (2015)

    Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)

    2015 arXiv

  3. [11]

    In: International Conference on Machine Learning, pp

    Fujimoto, S., Hoof, H., Meger, D.: Addressing function approximation error in actor-critic methods. In: International Conference on Machine Learning, pp. 1587–1596 (2018). PMLR

    2018

  4. [12]

    In: 17 International Conference on Machine Learning, pp

    Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: 17 International Conference on Machine Learning, pp. 1861–1870 (2018). Pmlr

    2018

  5. [13]

    In: International Conference on Machine Learning, pp

    Anschel, O., Baram, N., Shimkin, N.: Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In: International Conference on Machine Learning, pp. 176–185 (2017). PMLR

    2017

  6. [14]

    Advances in Neural Information Processing Systems35, 17703–17716 (2022)

    Matena, M.S., Raffel, C.: Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems35, 17703–17716 (2022)

    2022

  7. [15]

    arXiv preprint arXiv:2407.04811 (2024)

    Gallici, M., Fellows, M., Ellis, B., Pou, B., Masmitja, I., Foerster, J.N., Martin, M.: Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.04811 (2024)

    2024 arXiv

  8. [16]

    arXiv preprint arXiv:1803.05407 (2018)

    Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., Wilson, A.G.: Aver- aging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407 (2018)

    2018 arXiv

  9. [17]

    In: Proceedings of the European Conference on Computer Vision (ECCV), pp

    Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: Learning what (not) to forget. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 139–154 (2018)

    2018

  10. [18]

    Journal of Machine Learning Research23(274), 1–18 (2022)

    Huang, S., Dossa, R.F.J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., Ara´ ujo, J.G.M.: Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research23(274), 1–18 (2022)

    2022

  11. [19]

    Advances in Neural Information Processing Systems38(2026)

    Towers, M., Kwiatkowski, A., Balis, J., De Cola, G., Deleu, T., Goul˜ ao, M., Andreas, K., Krimmel, M., Kg, A., Perez-Vicente, R., et al.: Gymnasium: A standard interface for reinforcement learning environments. Advances in Neural Information Processing Systems38(2026)

    2026

  12. [20]

    arXiv preprint arXiv:1607.06450 (2016)

    Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)

    2016 arXiv

  13. [21]

    arXiv preprint arXiv:2403.00514 (2024)

    Nauman, M., Bortkiewicz, M., Mi lo´ s, P., Trzci´ nski, T., Ostaszewski, M., Cygan, M.: Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. arXiv preprint arXiv:2403.00514 (2024)

    2024 arXiv

  14. [22]

    arXiv preprint arXiv:2506.03758 (2025)

    Palenicek, D., Vogt, F., Peters, J.: Scaling crossq with weight normalization. arXiv preprint arXiv:2506.03758 (2025)

    2025 arXiv

  15. [23]

    Bhatt, A., Argus, M., Amiranashvili, A., Brox, T.: Crossnorm: On normalization for off-policy reinforcement learning (2019) A Appendix 18 T able 4: Final 200K time-steps Atari Scores Games Averaged DQN DQN DQN (LayerNorm) MM-DQN PQN Alien 1333.96 1304.18 1157.9 2141.83 1634.87...

    2019

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.