REVIEW 4 major objections 5 minor 21 references
Ensemble Elastic DQN: A Step Dependent Ensemble Approach for Reducing Overestimation in Deep Value-Based Reinforcement Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read EEDQN shows that using the ensemble mean for one-step targets and the ensemble minimum for multi-step targets reduces overestimation bias in deep Q-learning while maintaining or improving final returns.
desk verdict A plausible step-dependent ensemble idea, but the algorithm as printed cannot reproduce the reported results — needs a corrected pseudocode, reconciled tables, and released code. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a horizon-dependent ensemble target. For a bootstrap length d, the target is $y = R + \gamma^d \mathrm{Avg}_i \hat{Q}_i(s', a)$ when $d=0$ and $y = R + \gamma^d \mathrm{Min}_i \hat{Q}_i(s', a)$ when $d>0$. The length $d$ itself is chosen by a threshold $h = \mathrm{Avg}(B) + \mathrm{StdDev}(B)/\sqrt{n}$ applied to the absolute difference between the ensemble-mean Q-values of consecutive states; if the difference exceeds $h$, the return is cut and stored, otherwise the agent keeps accumulating rewards. This replaces the clustering-based similarity test of Elastic Step DQN with a rule that is cheap enough to scale to MinAtar, and it is the pairing of the elastic horizon with the step-dependent aggregation that the paper identifies as the source of its results.
What would settle it
Run EEDQN in an environment with controlled reward noise (for example, adding zero-mean noise to MinAtar rewards) and measure whether the average elastic step length becomes uncorrelated with the true transition stability; if it does, the Q-difference threshold has lost its signal and the observed gains should vanish.
Extended reading notes
Core claim
The central claim is that treating single-step and multi-step bootstrap targets differently with an ensemble is an effective way to tame overestimation. For a single-step experience the target uses the ensemble mean, keeping the update informed by all members; for multi-step experiences the target uses the ensemble minimum, penalizing the most optimistic member and preventing bootstrap exaggerations from compounding over longer horizons. On the MinAtar benchmark, EEDQN records the highest average final return in four of five environments (SpaceInvaders, Asterix, Breakout, Freeway) and is second to MaxMin DQN in Seaquest, while being one of only two algorithms, along with MaxMin DQN, that keep normalised Q-values below the theoretical maximum in every environment. The ablation across seven aggregation configurations shows no single rule wins everywhere, which the authors read as evidence that the interaction between adaptive return length and ensemble aggregation is nontrivial and environment-dependent.
Load-bearing premise
The load-bearing premise is that the absolute difference between ensemble-mean Q-values of consecutive states reliably signals whether the environment is stable enough to extend the bootstrap horizon, even though those Q-values come from the networks being trained and can themselves be miscalibrated.
Editorial extensions
If this is right
- Overestimation can be reduced without making every update uniformly conservative: one-step updates keep the ensemble mean, and only longer returns switch to the ensemble minimum, preserving learning signal where the bootstrap is shortest.
- The elastic step mechanism and the ensemble aggregation genuinely interact; the ablation was needed to see that neither component alone reaches the top performance in all five games.
- Aggregation conservatism should be treated as a tunable, environment-dependent quantity, not a fixed design choice, because the best rule differs across Seaquest, SpaceInvaders, Asterix, Breakout, and Freeway.
- EEDQN's normalised Q-values stay below the theoretical ceiling in all five environments while DQN, DDQN, AvgDQN, and ESDQN exceed it at some point, so the method's bias reduction is observable in a direct Q-value measure, not only in returns.
Reading between the lines
- A direct test of the stability proxy would be to compare the elastic step lengths chosen by the Q-difference threshold with lengths chosen by an oracle that knows true state similarity; in stochastic or aliased environments the two would likely diverge, which would localise where the method gains or loses.
- The step-dependent aggregation idea (mean on short horizons, min on long ones) could be carried to actor-critic and continuous-control algorithms that suffer the same overestimation problem; the paper does not explore that transfer.
- Because overestimation peaks in early-to-mid training and fades near convergence, a testable variant would schedule the aggregation, using the mean longer into training or annealing from min back toward mean, to see whether early optimism can be exploited without final-performance loss.
- The threshold formula uses the standard error of the buffer, which shrinks as the buffer fills, so the rule's sensitivity changes with buffer size; tuning state-memory size may be as important as choosing the aggregation rule, a dimension the paper leaves implicit.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Ensemble Elastic DQN (EEDQN), a value-based deep RL algorithm that combines adaptive elastic multi-step returns with ensemble aggregation: one-step targets use the ensemble mean, while longer elastic returns use the ensemble minimum. The method replaces the clustering-based similarity test of Elastic Step DQN with a Q-value-difference threshold computed from a buffer of absolute ensemble-mean differences. The authors evaluate EEDQN on five MinAtar environments against DQN, Double DQN, Averaged DQN, MaxMin DQN, and Elastic Step DQN, using 10 seeds, permutation tests, and a normalized Q-value overestimation analysis. They also run an ablation over seven ensemble aggregation configurations. The main empirical claim is that EEDQN achieves the highest final return in four of five environments and keeps Q-values below the theoretical maximum in all environments.
Significance. If the algorithm were implemented as described and the empirical results were reproducible, the paper would offer a useful combination of two heuristic families in value-based RL, with a scalable threshold rule that improves on the clustering step of Elastic Step DQN. The experimental protocol has notable strengths: ten seeds per algorithm, permutation tests, an aggregation ablation, and a comparison of normalized Q-values against the theoretical upper bound. These are appropriate tools for the question. However, the contribution is currently not assessable because the central algorithm specification in Algorithm 4 is internally inconsistent and cannot generate the reported experience tuples, and because several of the reported result tables contradict each other. Until these load-bearing issues are resolved, the empirical claims cannot be attributed to the stated method.
major comments (4)
- [Algorithm 4, lines 11-27] Algorithm 4 as printed cannot produce the transitions used for learning. In each inner-loop iteration, line 11 calls Step(s_t, ...) with the start state of the current segment, so when d>0 the environment step is taken from s_t rather than from the current environment state. If the threshold breaks on the first step (d=0), line 15 stores R_t=0, discarding the actual reward r_t; the reward is only added in the else branch on line 19. The multi-step target in Algorithm 1 then evaluates the ensemble on a stale action a_t selected at s_t instead of an action appropriate for state s_{t+d+1}, and line 27 updates s_t to s_{t+d+1} even though only one environment step was performed. These defects make the update rule in the paper unreproducible from the pseudocode, and the empirical results in Tables 2-3 cannot be attributed to the algorithm as specified.
- [Tables 2 and 3] The two main result tables report different values for the same EEDQN condition under the same metric. Table 2 lists EEDQN Breakout as 27.24, while Table 3 lists EEDQN Breakout as 25.77. Both tables claim to report mean reward over the last 100 episodes. This discrepancy is not explained, and it leaves the reader unable to determine which set of numbers is definitive. A single reported result with two conflicting values undermines the central quantitative claim.
- [Section 4.3 and Table 2] The ablation text states that MeanEEDQN 'excelled in SpaceInvaders', but Table 2 shows ConvexEEDQN2 with the highest SpaceInvaders value (105.88), EEDQN at 101.42, and MeanEEDQN at 100.07. This is an internal contradiction between the narrative and the data table. The ablation conclusions are also inconsistent with the table: the text says EEDQN had the highest final performance in three environments, but then attributes the SpaceInvaders best result to MeanEEDQN, which the table does not support. The aggregation ablation is a central part of the paper's evidence, so this contradiction needs to be fixed and the correct winner identified.
- [Section 3.2, Algorithm 2] The threshold rule h = Avg(B) + StdDev(B)/sqrt(n) is introduced without validation. The same Q-networks being trained generate the absolute differences z that populate B and appear in the threshold, so the elastic horizon is not independent of value-estimation error; if the value estimates are poor, the step-length decision is also poor. The manuscript does not provide evidence that this proxy is reliable, nor does it report sensitivity to buffer size H. At minimum, the authors should report how often one-step vs. multi-step transitions occur under this rule and how sensitive results are to H, so that the adaptive mechanism is actually characterized.
minor comments (5)
- [Table 1] Table 1 contains a typo in the learning rate row ("0,.00025") and reports slightly different epsilon values across algorithms (0.0003125, 0.00031, 0.0003); the paper should use one consistent value or explain the differences.
- [Section 4.1] The text refers to "MinMaxDQN" in the list of baselines, while the method later described is MaxMin DQN; the naming should be made consistent throughout.
- [Algorithm 4 and Algorithm 1] Algorithm 1's ComputeTarget uses the action a_t for both the one-step and multi-step cases, but the text describes evaluating the target at s_{t+d+1}; the pseudocode and the surrounding explanation of the target need to be aligned, especially after the state-advancement issue is fixed.
- [Tables 2 and 3] The p-value columns in Table 3 contain typos ("p-valuie") and one p-value "<0.03" that is not a standard reporting convention; the table header and significance notation should be cleaned up.
- [Abstract and Section 1] The abstract calls the method "Ensemble Elastic DQN" and then "Ensemble Elastic Step DQN" inconsistently; the later sections use these names interchangeably, which is confusing for a reader trying to identify the exact algorithm.
Circularity Check
No significant circularity: the paper's claims are empirical comparisons on external MinAtar benchmarks, with no fitted parameter relabeled as a prediction and no load-bearing self-citation.
full rationale
EEDQN is presented as a heuristic algorithm rather than as a derivation from assumptions, and its headline claims (final returns, normalized Q-values) are measured against the MinAtar benchmark rather than implied by the construction. The threshold h = Avg(B) + StdDev(B)/sqrt(n) in Section 3.2 uses Q-value differences produced by the same networks that form the ensemble targets, so the elastic horizon is influenced by the current value estimates; this is a feedback loop in the algorithm, but it is not a definitional reduction of the reported results to the inputs, and no constant is fitted to the outcomes. The self-citations [LDV+23, LDV+24] supply the prior ESDQN baseline and motivation, but EEDQN is a separate aggregation rule evaluated against independently implemented baselines, and no uniqueness theorem or ansatz is imported from those citations to force the conclusion. Reproducibility concerns in Algorithm 4 and Table 2/3 conflicts are correctness risks, not circularity.
Assumptions & free parameters
free parameters (3)
- ensemble size N =
2
- threshold statistic h =
Avg(B)+StdDev(B)/sqrt(n)
- state difference buffer size =
10000
assumptions (5)
- standard math Maximisation over noisy Q estimates causes overestimation bias in DQN
- ad hoc to paper Absolute difference of ensemble-mean Q-values between consecutive states is a valid proxy for state similarity and determines when to stop bootstrapping
- domain assumption Ensemble members provide sufficiently independent estimates for min/mean aggregation to reduce overestimation
- domain assumption The single action a_t selected at the start of the trajectory is an appropriate action to evaluate at the future state s_{t+d+1} in the target
- domain assumption MinAtar performance and |Q|max normalization are adequate evidence for overestimation behavior in general deep RL
Cite this review
Pith. "Pith review of Ensemble Elastic DQN: A Step Dependent Ensemble Approach for Reducing Overestimation in Deep Value-Based Reinforcement Learning." pith.science (2026). https://pith.science/paper/2QMYX45V
@misc{pith2026250605716,
author = {Pith},
title = {Pith review of: Ensemble Elastic DQN: A Step Dependent Ensemble Approach for Reducing Overestimation in Deep Value-Based Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/2QMYX45V}},
note = {Machine review of arXiv:2506.05716}
}
read the original abstract
Deep Q-Networks (DQN) can suffer from overestimation bias because bootstrapped targets use a maximisation operation over noisy value estimates. Ensemble-based methods and multi-step methods have each been used to improve the stability and sample efficiency of value-based reinforcement learning, but their interaction remains less well understood. This paper introduces Ensemble Elastic DQN (EEDQN), a value-based reinforcement learning algorithm that combines adaptive elastic multi-step returns with ensemble-based target aggregation. EEDQN replaces the clustering-based state similarity test used in earlier Elastic Step DQN with a lightweight Q-value difference rule, making adaptive return construction simpler to apply in discrete control settings. The method then applies horizon dependent ensemble aggregation, one-step targets use the ensemble mean, while longer elastic returns use the ensemble minimum. This design aims to reduce extreme optimistic bootstrap estimates without making every update uniformly conservative. We evaluate EEDQN on five MinAtar environments against DQN, Double DQN, Averaged DQN, MaxMin DQN, and Elastic Step DQN. EEDQN achieves the highest final return in four of the five environments and remains competitive with conservative ensemble baselines. An aggregation rule ablation shows that the best degree of conservatism is environment dependent, suggesting that adaptive return length and ensemble aggregation interact in non-trivial ways.
Figures
Reference graph
Works this paper leans on
-
[1]
Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In International conference on machine learning , pages 176--185. PMLR, 2017
work page 2017
-
[2]
Gang Chen and Victoria Huang. Ensemble reinforcement learning in continuous spaces--a hierarchical multi-step approach for policy training. arXiv preprint arXiv:2209.14488 , 2022
work page Pith review arXiv 2022
-
[3]
Mushroomrl: Simplifying reinforcement learning research
Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Mushroomrl: Simplifying reinforcement learning research. Journal of Machine Learning Research , 22(131):1--5, 2021
work page 2021
-
[4]
Addressing function approximation error in actor-critic methods
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587--1596. PMLR, 2018
work page 2018
-
[5]
Understanding multi-step deep reinforcement learning: A systematic study of the dqn target
J Fernando Hernandez-Garcia and Richard S Sutton. Understanding multi-step deep reinforcement learning: A systematic study of the dqn target. arXiv preprint arXiv:1901.07510 , 2019
arXiv 1901
-
[6]
Wd3: Taming the estimation bias in deep reinforcement learning
Qiang He and Xinwen Hou. Wd3: Taming the estimation bias in deep reinforcement learning. In 2020 IEEE 32nd international conference on tools with artificial intelligence (ICTAI) , pages 391--398. IEEE, 2020
work page 2020
-
[7]
Rainbow: Combining improvements in deep reinforcement learning
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018
work page 2018
-
[8]
Bring color to deep q-networks: Limitations and improvements of dqn leading to rainbow dqn
Jonas J \"a ger, Felix Helfenstein, and Fabian Scharf. Bring color to deep q-networks: Limitations and improvements of dqn leading to rainbow dqn. In Reinforcement Learning Algorithms: Analysis and Applications , pages 135--149. Springer, 2021
work page 2021
Show all 21 references
-
[9]
Elastic step ddpg: Multi-step reinforcement learning for improved sample efficiency
Adrian Ly, Richard Dazeley, Peter Vamplew, Francisco Cruz, and Sunil Aryal. Elastic step ddpg: Multi-step reinforcement learning for improved sample efficiency. In 2023 International Joint Conference on Neural Networks (IJCNN) , pages 01--06. IEEE, 2023
2023
-
[10]
Elastic step dqn: A novel multi-step algorithm to alleviate overestimation in deep q-networks
Adrian Ly, Richard Dazeley, Peter Vamplew, Francisco Cruz, and Sunil Aryal. Elastic step dqn: A novel multi-step algorithm to alleviate overestimation in deep q-networks. Neurocomputing , 576:127170, 2024
2024
-
[11]
Maxmin q-learning: Controlling the estimation bias of q-learning
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Controlling the estimation bias of q-learning. arXiv preprint arXiv:2002.06487 , 2020
2002 arXiv
-
[12]
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529--533, 2015
2015
-
[13]
Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning
Michal Nauman, Micha Bortkiewicz, Piotr Mi o \'s , Tomasz Trzci \'n ski, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. arXiv preprint arXiv:2403.00514 , 2024
2024 arXiv
-
[14]
Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research
Johan S Obando-Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. arXiv preprint arXiv:2011.14826 , 2020
2011 arXiv
-
[15]
Issues in using function approximation for reinforcement learning
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 connectionist models summer school , pages 255--263. Psychology Press, 2014
1993
-
[16]
Deep reinforcement learning and the deadly triad
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02648 , 2018
2018 arXiv
-
[17]
Deep reinforcement learning with double q-learning
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence , volume 30, 2016
2016
-
[18]
Controlling underestimation bias in reinforcement learning via quasi-median operation
Wei Wei, Yujia Zhang, Jiye Liang, Lin Li, and Yyuze Li. Controlling underestimation bias in reinforcement learning via quasi-median operation. In Proceedings of the AAAI conference on artificial intelligence , volume 36, pages 8621--8628, 2022
2022
-
[19]
Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments
Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.03176 , 2019
1903 arXiv
-
[20]
A novel multi-step q-learning method to improve data efficiency for deep reinforcement learning
Yinlong Yuan, Zhu Liang Yu, Zhenghui Gu, Yao Yeboah, Wu Wei, Xiaoyan Deng, Jingcong Li, and Yuanqing Li. A novel multi-step q-learning method to improve data efficiency for deep reinforcement learning. Knowledge-Based Systems , 175:107--117, 2019
2019
-
[21]
beta-dqn: Improving deep q-learning by evolving the behavior
Hongming Zhang, Fengshuo Bai, Chenjun Xiao, Chao Gao, Bo Xu, and Martin M \"u ller. beta-dqn: Improving deep q-learning by evolving the behavior. arXiv preprint arXiv:2501.00913 , 2025
2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.