{"id":"9f2c1c8b-0ce0-498c-a742-46dbe3405416","arxiv_id":"2506.05716","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"EEDQN sets its bootstrap horizon using a Q-value difference threshold and then uses the ensemble mean for one-step targets and the ensemble minimum for multi-step targets, reporting the best final score on four of five MinAtar games.","lead":"The paper combines two known fixes to deep Q-learning, adaptive multi-step returns and ensembles of Q-networks, into one algorithm. This matters because overestimated Q-values are a common cause of unstable training, and the new rule is cheap to add to existing DQN code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 4 as printed cannot reproduce the reported results: the accumulation loop steps from s_t instead of the current state, one-step transitions store R_t=0, and multi-step targets use a stale action; reported tables also conflict.","rationale":"The reader's declared weakest assumption is the threshold rule in Algorithm 2 (Q-value difference as a stability proxy). I agree that this is a real concern, but it is secondary: the threshold rule only matters if the surrounding training loop is correctly specified. As printed, Algorithm 4 cannot be executed as written—the environment is stepped from the original s_t rather than the current state, the one-step stored return omits the immediate reward, and the multi-step target uses an action from the wrong state. These are not tuning issues; they change the objective being optimized. If the reported results were produced by a literal implementation of the pseudocode, the algorithm would behave differently from any sensible n-step ensemble method; if they were produced by a corrected implementation, the paper omits the actual algorithm. Either way, the central claim is unsupported by the manuscript. The table inconsistencies (Breakout 27.24 vs 25.77; ablation text vs Table 2 on SpaceInvaders) reinforce this by showing the reported numbers are internally inconsistent. I therefore agree with the reader's REJECT verdict, though through a different primary mechanism than the one highlighted as the weakest assumption. A corrected pseudocode and released code could move this toward CONDITIONAL, but as presented the evidence does not support acceptance.","tokens_in":77,"tokens_out":4652,"duration_ms":88831,"concrete_test":"Implement Algorithm 4 exactly as written, including Algorithm 3's env.Step(a_t) called from s_t in line 11 and the R_t=0 store for d=0, and run on MinAtar Breakout with the reported hyperparameters for 10 seeds. If the agent fails to learn or learns only when the code silently deviates from the listing (e.g., by stepping from the current state and accumulating r_t), the central claim is unsupported. As a second check, instrument the replay buffer to print the target for a d=0 transition: the listed algorithm stores R_t=0, so the target has no reward term; a corrected algorithm stores r_t. Also re-run the final-100-episode evaluation of EEDQN on Breakout and confirm whether the value is 27.24 (Table 2) or 25.77 (Table 3); any mismatch indicates the reported numbers are not from a single fixed evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—EEDQN outperforms baselines and controls overestimation—is not attributable to the algorithm as specified. Algorithm 4's accumulation loop contains concrete defects that make the listed update rule unreproducible. (1) In each outer iteration, line 11 calls Step(s_t,...) where s_t is the start of the current trajectory, not the environment's current state once d>0; therefore the transition used for the threshold and target is generated from the wrong state. (2) If the threshold breaks on the first step (d=0), line 15 stores R_t=0, so one-step transitions have zero reward; the current reward r_t is only added in the else branch. (3) ComputeTarget(d, s_{t+d+1}, a_t, R_t) evaluates the ensemble on the stale action a_t selected at s_t, rather than an action appropriate for the multi-step horizon. These defects mean the 'EEDQN' whose results appear in Tables 2-3 is not the algorithm in Algorithm 4 unless an unstated corrected implementation was used. Additionally, Table 2 and Table 3 report different Breakout final returns for EEDQN (27.24 vs 25.77) under the same metric, and the Section 4.3 ablation text says MeanEEDQN excelled in SpaceInvaders while Table 2 shows EEDQN (101.42) above MeanEEDQN (100.07). Without released code or a corrected pseudocode, the empirical results cannot be verified against the stated method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Ensemble Elastic DQN (EEDQN), a value-based deep RL algorithm that combines adaptive elastic multi-step returns with ensemble aggregation: one-step targets use the ensemble mean, while longer elastic returns use the ensemble minimum. The method replaces the clustering-based similarity test of Elastic Step DQN with a Q-value-difference threshold computed from a buffer of absolute ensemble-mean differences. The authors evaluate EEDQN on five MinAtar environments against DQN, Double DQN, Averaged DQN, MaxMin DQN, and Elastic Step DQN, using 10 seeds, permutation tests, and a normalized Q-value overestimation analysis. They also run an ablation over seven ensemble aggregation configurations. The main empirical claim is that EEDQN achieves the highest final return in four of five environments and keeps Q-values below the theoretical maximum in all environments.","tokens_in":8819,"tokens_out":4220,"duration_ms":40764,"significance":"If the algorithm were implemented as described and the empirical results were reproducible, the paper would offer a useful combination of two heuristic families in value-based RL, with a scalable threshold rule that improves on the clustering step of Elastic Step DQN. The experimental protocol has notable strengths: ten seeds per algorithm, permutation tests, an aggregation ablation, and a comparison of normalized Q-values against the theoretical upper bound. These are appropriate tools for the question. However, the contribution is currently not assessable because the central algorithm specification in Algorithm 4 is internally inconsistent and cannot generate the reported experience tuples, and because several of the reported result tables contradict each other. Until these load-bearing issues are resolved, the empirical claims cannot be attributed to the stated method.","major_comments":[{"comment":"Algorithm 4 as printed cannot produce the transitions used for learning. In each inner-loop iteration, line 11 calls Step(s_t, ...) with the start state of the current segment, so when d>0 the environment step is taken from s_t rather than from the current environment state. If the threshold breaks on the first step (d=0), line 15 stores R_t=0, discarding the actual reward r_t; the reward is only added in the else branch on line 19. The multi-step target in Algorithm 1 then evaluates the ensemble on a stale action a_t selected at s_t instead of an action appropriate for state s_{t+d+1}, and line 27 updates s_t to s_{t+d+1} even though only one environment step was performed. These defects make the update rule in the paper unreproducible from the pseudocode, and the empirical results in Tables 2-3 cannot be attributed to the algorithm as specified.","section":"Algorithm 4, lines 11-27"},{"comment":"The two main result tables report different values for the same EEDQN condition under the same metric. Table 2 lists EEDQN Breakout as 27.24, while Table 3 lists EEDQN Breakout as 25.77. Both tables claim to report mean reward over the last 100 episodes. This discrepancy is not explained, and it leaves the reader unable to determine which set of numbers is definitive. A single reported result with two conflicting values undermines the central quantitative claim.","section":"Tables 2 and 3"},{"comment":"The ablation text states that MeanEEDQN 'excelled in SpaceInvaders', but Table 2 shows ConvexEEDQN2 with the highest SpaceInvaders value (105.88), EEDQN at 101.42, and MeanEEDQN at 100.07. This is an internal contradiction between the narrative and the data table. The ablation conclusions are also inconsistent with the table: the text says EEDQN had the highest final performance in three environments, but then attributes the SpaceInvaders best result to MeanEEDQN, which the table does not support. The aggregation ablation is a central part of the paper's evidence, so this contradiction needs to be fixed and the correct winner identified.","section":"Section 4.3 and Table 2"},{"comment":"The threshold rule h = Avg(B) + StdDev(B)/sqrt(n) is introduced without validation. The same Q-networks being trained generate the absolute differences z that populate B and appear in the threshold, so the elastic horizon is not independent of value-estimation error; if the value estimates are poor, the step-length decision is also poor. The manuscript does not provide evidence that this proxy is reliable, nor does it report sensitivity to buffer size H. At minimum, the authors should report how often one-step vs. multi-step transitions occur under this rule and how sensitive results are to H, so that the adaptive mechanism is actually characterized.","section":"Section 3.2, Algorithm 2"}],"minor_comments":[{"comment":"Table 1 contains a typo in the learning rate row (\"0,.00025\") and reports slightly different epsilon values across algorithms (0.0003125, 0.00031, 0.0003); the paper should use one consistent value or explain the differences.","section":"Table 1"},{"comment":"The text refers to \"MinMaxDQN\" in the list of baselines, while the method later described is MaxMin DQN; the naming should be made consistent throughout.","section":"Section 4.1"},{"comment":"Algorithm 1's ComputeTarget uses the action a_t for both the one-step and multi-step cases, but the text describes evaluating the target at s_{t+d+1}; the pseudocode and the surrounding explanation of the target need to be aligned, especially after the state-advancement issue is fixed.","section":"Algorithm 4 and Algorithm 1"},{"comment":"The p-value columns in Table 3 contain typos (\"p-valuie\") and one p-value \"<0.03\" that is not a standard reporting convention; the table header and significance notation should be cleaned up.","section":"Tables 2 and 3"},{"comment":"The abstract calls the method \"Ensemble Elastic DQN\" and then \"Ensemble Elastic Step DQN\" inconsistently; the later sections use these names interchangeably, which is confusing for a reader trying to identify the exact algorithm.","section":"Abstract and Section 1"}],"recommendation":"reject","confidential_remarks":"The core problem is not stylistic but substantive: the algorithm as printed cannot generate the data that support the paper's claims, and the result tables contradict one another. A corrected pseudocode and a released reference implementation might make the contribution verifiable, but in the current form the central claim is not attributable to the stated method. This is a load-bearing issue rather than a presentation issue, and it would require a substantial revision plus new experimental validation rather than a small edit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: the paper has a real new combination—horizon-dependent ensemble aggregation (mean for one-step, min for multi-step) and a Q-value-difference threshold in place of clustering—and the ablation study is a sensible attempt to see which design choice matters. But as submitted, the algorithm listing is internally inconsistent in a way that makes the reported results unattributable to the stated method. The stress-test is not off-base; reading Algorithm 4, the accumulation loop steps from s_t on every iteration, so once d>0 the transition is generated from the wrong state; if the threshold fires on the first step, the stored transition has R_t=0; and the multi-step target evaluates the ensemble on the stale action chosen at s_t. Those aren't cosmetic typos—they change the update rule. On top of that, Table 2 and Table 3 disagree about EEDQN's Breakout score (27.24 vs 25.77), and the Section 4.3 text says MeanEEDQN excelled in SpaceInvaders while Table 2 shows EEDQN slightly ahead. No code or data are released, so I can't check whether a corrected implementation produced the numbers.\n\nCredit where it's due: the paper is clearly motivated, uses sensible baselines (DQN, DDQN, AvgDQN, MaxMin, ESDQN), runs 10 seeds, and uses permutation tests. The overestimation plot (Figure 2) is a useful comparison, and the finding that EEDQN and MaxMin stay below the theoretical max in all environments is the most robust empirical signal. The threshold heuristic itself is plausible—using the mean plus one standard error of recent Q-difference magnitudes is a reasonable lightweight similarity test—though it is entirely heuristic and unvalidated against the clustering alternative.\n\nThe soft spot is not minor; it's load-bearing. If the algorithm is fixed, the paper could be a modest but citable contribution to the value-based RL toolbox. As it stands, the central claim—EEDQN outperforms baselines and controls overestimation—cannot be attributed to the method as written. The right move is to reject or desk-reject with clear instructions to correct the pseudocode, reconcile the tables, and release code. I would not send this to peer review in its current form, but I'd be willing to look at a corrected version.","headline":"A plausible step-dependent ensemble idea, but the algorithm as printed cannot reproduce the reported results — needs a corrected pseudocode, reconciled tables, and released code.","tokens_in":9380,"tokens_out":3784,"would_cite":false,"duration_ms":35503,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EEDQN shows that using the ensemble mean for one-step targets and the ensemble minimum for multi-step targets reduces overestimation bias in deep Q-learning while maintaining or improving final returns.","keywords":["reinforcement learning","overestimation bias","ensemble learning","multi-step returns","elastic step","deep Q-networks","MinAtar","value-based methods"],"falsifier":"Run EEDQN in an environment with controlled reward noise (for example, adding zero-mean noise to MinAtar rewards) and measure whether the average elastic step length becomes uncorrelated with the true transition stability; if it does, the Q-difference threshold has lost its signal and the observed gains should vanish.","tokens_in":8287,"feed_emoji":"🎮","tokens_out":6069,"duration_ms":54795,"temperature":0.7,"pith_summary":"EEDQN tests whether multi-step and ensemble methods, usually studied separately, can be combined to fight overestimation bias in deep Q-learning without paying a uniform conservatism penalty. The algorithm extends Elastic Step DQN by replacing its clustering-based state-similarity test with a lightweight Q-value-difference threshold, and by using different ensemble aggregations for different return horizons: the average of the ensemble for one-step targets, the minimum for longer elastic returns. On five MinAtar games it achieves the best final return in four, never crosses the theoretical Q-value ceiling, and its ablation shows that the best degree of conservatism depends on the environment. The paper argues that systematically combining algorithmic improvements can yield gains that neither ingredient alone provides.","feed_headline":"Ensemble elastic DQN tops four of five MinAtar games","feed_subtitle":"Mixing ensemble average and minimum estimates by return length cuts optimistic bias while keeping sample efficiency.","key_machinery":"The mechanism is a horizon-dependent ensemble target. For a bootstrap length d, the target is $y = R + \\gamma^d \\mathrm{Avg}_i \\hat{Q}_i(s', a)$ when $d=0$ and $y = R + \\gamma^d \\mathrm{Min}_i \\hat{Q}_i(s', a)$ when $d>0$. The length $d$ itself is chosen by a threshold $h = \\mathrm{Avg}(B) + \\mathrm{StdDev}(B)/\\sqrt{n}$ applied to the absolute difference between the ensemble-mean Q-values of consecutive states; if the difference exceeds $h$, the return is cut and stored, otherwise the agent keeps accumulating rewards. This replaces the clustering-based similarity test of Elastic Step DQN with a rule that is cheap enough to scale to MinAtar, and it is the pairing of the elastic horizon with the step-dependent aggregation that the paper identifies as the source of its results.","core_discovery":"The central claim is that treating single-step and multi-step bootstrap targets differently with an ensemble is an effective way to tame overestimation. For a single-step experience the target uses the ensemble mean, keeping the update informed by all members; for multi-step experiences the target uses the ensemble minimum, penalizing the most optimistic member and preventing bootstrap exaggerations from compounding over longer horizons. On the MinAtar benchmark, EEDQN records the highest average final return in four of five environments (SpaceInvaders, Asterix, Breakout, Freeway) and is second to MaxMin DQN in Seaquest, while being one of only two algorithms, along with MaxMin DQN, that keep normalised Q-values below the theoretical maximum in every environment. The ablation across seven aggregation configurations shows no single rule wins everywhere, which the authors read as evidence that the interaction between adaptive return length and ensemble aggregation is nontrivial and environment-dependent.","pith_inferences":["A direct test of the stability proxy would be to compare the elastic step lengths chosen by the Q-difference threshold with lengths chosen by an oracle that knows true state similarity; in stochastic or aliased environments the two would likely diverge, which would localise where the method gains or loses.","The step-dependent aggregation idea (mean on short horizons, min on long ones) could be carried to actor-critic and continuous-control algorithms that suffer the same overestimation problem; the paper does not explore that transfer.","Because overestimation peaks in early-to-mid training and fades near convergence, a testable variant would schedule the aggregation, using the mean longer into training or annealing from min back toward mean, to see whether early optimism can be exploited without final-performance loss.","The threshold formula uses the standard error of the buffer, which shrinks as the buffer fills, so the rule's sensitivity changes with buffer size; tuning state-memory size may be as important as choosing the aggregation rule, a dimension the paper leaves implicit."],"forward_implications":["Overestimation can be reduced without making every update uniformly conservative: one-step updates keep the ensemble mean, and only longer returns switch to the ensemble minimum, preserving learning signal where the bootstrap is shortest.","The elastic step mechanism and the ensemble aggregation genuinely interact; the ablation was needed to see that neither component alone reaches the top performance in all five games.","Aggregation conservatism should be treated as a tunable, environment-dependent quantity, not a fixed design choice, because the best rule differs across Seaquest, SpaceInvaders, Asterix, Breakout, and Freeway.","EEDQN's normalised Q-values stay below the theoretical ceiling in all five environments while DQN, DDQN, AvgDQN, and ESDQN exceed it at some point, so the method's bias reduction is observable in a direct Q-value measure, not only in returns."],"supporting_citations":[{"why":"Supplies the elastic step mechanism and the baseline ESDQN that EEDQN extends and outperforms.","marker":"[LDV+24]"},{"why":"Provides the ensemble-mean aggregation idea used for single-step targets and the Averaged DQN baseline.","marker":"[ABS17]"},{"why":"Provides the ensemble-minimum aggregation idea used for multi-step targets and the MaxMin DQN baseline.","marker":"[LPFW20]"},{"why":"Establishes DQN and the overestimation-bias problem that EEDQN is designed to address.","marker":"[MKS+15]"},{"why":"Defines the MinAtar benchmark environments used for all experiments.","marker":"[YT19]"},{"why":"Supplies the hyperparameter settings and n-step DQN context that anchor the experimental setup.","marker":"[OCC20]"}],"fun_headline_variants":["Ensemble elastic DQN reduces overestimation on MinAtar","EEDQN mixes mean and min targets to trim optimistic bias","Step-dependent ensemble aggregation wins four of five MinAtar games","Elastic multi-step returns plus ensemble min cut DQN bias","Q-value difference rule simplifies adaptive return length in EEDQN"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the absolute difference between ensemble-mean Q-values of consecutive states reliably signals whether the environment is stable enough to extend the bootstrap horizon, even though those Q-values come from the networks being trained and can themselves be miscalibrated.","fun_headline_variants_meta":{"raw":{"variants":["Ensemble elastic DQN reduces overestimation on MinAtar","EEDQN mixes mean and min targets to trim optimistic bias","Step-dependent ensemble aggregation wins four of five MinAtar games","Elastic multi-step returns plus ensemble min cut DQN bias","Q-value difference rule simplifies adaptive return length in EEDQN"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1487,"prompt_tokens":983,"completion_tokens":504,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":413}},"tokens_in":599,"tokens_out":504,"duration_ms":5528,"temperature":1.0,"reasoning_tokens":413,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:13:39.709407+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EEDQN in an environment with controlled reward noise (for example, adding zero-mean noise to MinAtar rewards) and measure whether the average elastic step length becomes uncorrelated with the true transition stability; if it does, the Q-difference threshold has lost its signal and the observed gains should vanish.","supporting_citations":[],"review_version":1}