{"id":"3649cc24-dffd-4259-831a-2aa23234214a","arxiv_id":"2506.19478","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ADDQ uses the sample variance of distributional value estimates to adaptively blend Q-learning and double Q-learning, reducing bias in tabular, Atari, and MuJoCo experiments.","lead":"This paper presents ADDQ, a small modification to distributional reinforcement learning that mixes ordinary Q-learning with double Q-learning depending on how uncertain the agent is about each state-action value. The aim is to reduce the over-optimistic value estimates that slow down many reinforcement learning algorithms.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The variance proxy used to set β estimates only aleatoric variance; it does not capture the k- and N-dependence of overestimation that Prop. 2.1 identifies.","rationale":"The paper proposes ADDQ, a locally adaptive interpolation between Q-learning and double Q-learning, with the mixing weight β determined by the relative sample variance S²_rel(s,a) of the distributional return estimate. The core claim is that S²_rel faithfully indicates when QL overestimation is harmful, so that a small β (more double Q) is used exactly where overestimation skews learning. This claim is load-bearing: if the proxy is wrong, the method reduces to a heuristic mixing rule with no advantage over constant interpolation. The reader identified the same weakness: the sample variance is not established as a faithful proxy for overestimation bias, especially under function approximation. I agree, and I sharpen the concern by showing that Proposition 2.2 itself demonstrates the proxy is incomplete even in the tabular bandit. The sample variance of the return distribution equals the sample variance of the selected arm's rewards, which estimates σ² and is independent of k and N, while Proposition 2.1 shows overestimation grows with σ√(log k)/√N. Thus the adaptive mechanism cannot respond to two of the three drivers it is supposed to control. This does not disprove the empirical gains—ADDQ may still help on Atari through the σ-component or through a generic interpolation effect—but it means the theoretical motivation does not cover the proposed mechanism. The paper is otherwise honest and well-executed: the convergence theorem (Theorem 3.1) is a real contribution, the ablation studies and limitations are candid, and the code is provided. The conditional verdict remains appropriate: the paper should be accepted only if the proxy is validated or the claims are narrowed. No verdict change is needed.","tokens_in":35378,"tokens_out":8234,"duration_ms":89701,"concrete_test":"Run the two-sided bandit MDP (Section 2.2 and Appendix B) with σ₁ = σ₂ = 1, μ₁ = −0.1, μ₂ = 0.1, but with k₁ varying from 2 to 20 while k₂ = 5, so the reward variances are identical and S²_rel(s₀,·) ≈ 1 for both actions, yielding β = 0.5 for both by Eq. (1). Compare ADDQ's bias and correct-action rate against Q-learning and against a constant-β = 0.5 baseline. If ADDQ does not outperform constant β = 0.5 and if its bias does not track the overestimation growth with k₁ predicted by Proposition 2.1, then the variance proxy fails to capture the action-count component of overestimation, and the central adaptive mechanism lacks support even in the tabular regime.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism assumes S²(η_t(s,a)) is a faithful proxy for the local overestimation bias of Q-learning, so that S²_rel in Eq. (1) can decide when to down-weight the QL target. The paper's own theory undercuts this. Proposition 2.1 shows the overestimation bias in the bandit MDP scales as σ·√(log k)/√N, i.e., it depends on reward variance σ², action count k, and sample size N. Proposition 2.2, however, shows that the sample variance of the distributional return estimate is exactly the sample variance of the selected arm's iid rewards: σ²/(N−1)·χ²_{N−1}, with expectation σ² and variance 2σ⁴/(N−1). It therefore estimates only the aleatoric variance σ²; it does not shrink with N and does not depend on k. Consequently, S²_rel cannot detect the k- and N-driven components of overestimation that Proposition 2.1 identifies. This is not merely a function-approximation issue: the proxy is incomplete in the very tabular setting used to motivate it. If two actions have identical reward variance but different action counts or different visit counts, Eq. (1) assigns them the same β, so ADDQ cannot adapt to the skew Proposition 2.1 predicts. The claimed mechanism thus rests on an unproven correlation between return-distribution spread and estimator bias, and the paper's theory does not supply it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ADDQ, a modification of distributional Q-learning in which the Q-learning target and the double Q-learning target are mixed with a state-action-dependent weight β, chosen from the relative sample variance of the estimated return distributions. The authors provide two bandit-style propositions (a lower bound on Q-learning overestimation and an exact chi-squared law for the sample variance of the return estimate), a convergence theorem for the tabular categorical setting, grid-world and two-sided-bandit experiments, and deep RL experiments with C51 and QRDQN on Atari and with quantile SAC on MuJoCo. The central claim is that the local spread of the estimated return distribution identifies when Q-learning updates are prone to skewed overestimation, so that mixing in the double estimator locally improves bias and stability.","tokens_in":35633,"tokens_out":16805,"duration_ms":169335,"significance":"If the mechanism were fully established, ADDQ would be a cheap, few-line improvement to existing distributional algorithms, with an actual convergence proof in the tabular categorical case and a reproducible code base. The paper's strengths include a detailed convergence proof (Appendix F), an ablation over β schedules (Appendix C.2), the use of RLiable statistics for the deep experiments, and the fact that the symmetric β rule in Eq. (1) does satisfy the coupling condition lim |β^A_t − β^B_t| = 0 of Theorem 3.1. However, as detailed in the major comments, the theoretical support for the core variance-bias link is incomplete, and one step of the convergence proof as written contains a gap. The contribution is therefore promising but not yet fully supported.","major_comments":[{"comment":"The central mechanism is not established by the two bandit propositions. Proposition 2.2 shows that, in the idealized bandit, the sample variance of the return estimate is σ²/(N−1)χ²_{N−1}, with expectation σ², while Proposition 2.1 shows that the overestimation bias is proportional to σ√(log k)/√N. The sample variance therefore contains no information about the action count k and only a shrinking fluctuation component about N; two actions with identical reward variance but different k or different visit counts will receive the same β from Eq. (1), so ADDQ cannot adapt to the k- and N-driven skew that the paper's own lower bound identifies. Additionally, Proposition 2.2 is derived for the empirical measure (1/N)Σδ_{X_i} after a one-shot bootstrap, whereas Algorithm 2 computes S² from the categorical or quantile approximations η^A_t(s,a) before the update; the exact chi-squared law does not transfer to the projected fixed-atom representations used in the grid-world and Atari experiments. The statement in Section 2.4 that the agent 'does have access to the σ²χ²_N/N-distributed sample variance by computing sums of the atoms' is therefore not justified for the algorithm actually run.","section":"Section 2.4, Propositions 2.1 and 2.2, Eq. (1)"},{"comment":"The proof of convergence of Q^A_t and Q^B_t to Q* contains a gap in the bound on |E[\\tilde F_n | F_n]|. The text first bounds one of the terms by ||X^BA_n||∞ + 1/2||X^BA_n|| and then 'combines' the terms as though the coefficient were only ||X^BA_n||∞. With the displayed bound, the effective contraction coefficient in Lemma F.3 can exceed 1 for γ close to 1, so the SARSA-trick argument does not go through as written. The gap is repairable: using the maximality of a* with respect to Q^A and of b* with respect to Q^B gives the sharper bound |E[Q^A_n(S_{n+1},b*) − Q^B_n(S_{n+1},a*)|F_n]| ≤ ||X^BA_n||∞. The proof should be corrected before the theorem is relied upon.","section":"Appendix F, proof of Theorem F.2, Step 1"},{"comment":"There is a timing mismatch between the theory and the algorithm's use of the variance proxy. In the bandit analysis, the quantity whose distribution is computed in Proposition 2.2 is the sample variance of the return estimate after the bootstrap propagation from the next state. In Algorithm 2, however, β for updating (s,a) is determined from η^A_old and η^B_old before the update, and the paper does not specify or analyze how the pre-update sample variance at (s,a) relates to the overestimation introduced by the max at the next state (s',a*). The paper should either prove a statement about this pre-update proxy or explicitly acknowledge that the deep and sequential tabular use of the proxy is a heuristic not covered by the bandit propositions.","section":"Section 3.2 and Algorithm 2"}],"minor_comments":[{"comment":"The prose in Section 3.1 says that when a lot of uncertainty is present the algorithm uses large β, but Eq. (1) assigns β=0.25 to high relative variance and β=0.75 to low relative variance; the text should be reworded to avoid this apparent contradiction.","section":"Section 3.1 vs Eq. (1)"},{"comment":"The definition S²(ν) = (1/(n−1))Σ p_i(a_i − M(ν))² with n atoms is a weighted variance of the atoms, not the classical sample variance of observations; the statistical analogy should be stated more carefully, especially because the chi-squared law of Proposition 2.2 relies on equal weights 1/N rather than on the categorical parametrization.","section":"Definitions in Section 3.2"},{"comment":"The phrase 'the left side has been explored N k1 times' is ambiguous; from Theorem A.2 it appears to mean N episodes with k1 actions each, i.e., N k1 total arm pulls, and this should be stated explicitly.","section":"Proposition 2.1"},{"comment":"There are several typographical issues: 'Theoerem' in the heading of Appendix F, 'prevers' in Section 4, 'red off' in Section 3.2, 'Q-learing' in Appendix A, and 'chosing' in the proof of Theorem A.2.","section":"Typos"},{"comment":"The Atari and MuJoCo results are presented through learning curves and RLiable plots; adding a table with mean, median, and interquartile mean of normalized scores would make the aggregate comparison easier to verify.","section":"Experimental reporting"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful empirical recipe and a serious convergence proof, but the advertised theoretical evidence for the variance-based mixing rule is not currently established, and one step of the main proof needs fixing. I would ask the authors to either derive a proxy that captures the k- and N-dependence of the bias, or clearly reframe the variance rule as a heuristic and provide a more direct empirical justification for the (s,a)-level pre-update variance. With those changes, the paper could be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: ADDQ is a genuinely simple idea—use the local sample variance of the return distribution to interpolate between QL and double QL targets—and it is evaluated honestly. The novelty is real: the specific adaptive rule based on relative sample variances is not in the cited literature, and the paper adds a convergence proof for the tabular categorical setting with a detailed proof. The empirical work is above the usual bar: 10 Atari games, 10 seeds, RLiable statistics, ablations on the beta thresholds, comparisons to Maxmin/EBQL/REDQ in the grid world, and released code. The MuJoCo section is candid: ADDQ improves plain and double critics but does not beat clipping.\n\nThe soft spot is the theoretical motivation. Proposition 2.1 shows overestimation bias scales as sigma sqrt(log k)/sqrt N, but Proposition 2.2 shows the sample variance of the return estimate is just the selected arm's reward sample variance, which has expectation sigma-squared and does not shrink with N or depend on k. So the proxy in Eq. (1) cannot detect the k- and N-driven components of overestimation that the paper's own theory identifies. The stress-test note is right: this is not merely a function-approximation issue, it is present in the motivating tabular setting. The authors are somewhat careful—they call the beta rule 'exemplary' and leave adaptive thresholds to future work—but they still claim the propositions motivate the rule. That overstates the connection.\n\nDoes this sink the paper? No. The empirical gains are the actual claim, and they are consistent across games and parametrizations. The convergence theorem does not depend on the proxy being a faithful bias estimate. So the paper would be stronger if it either validated the variance-overestimation correlation directly in a controlled setting or simply dropped the strong wording and presented ADDQ as a heuristic that happens to work. The results stand either way.\n\nThis is a paper worth sending to review. A good referee should push on the proxy assumption and ask for a direct test of the mechanism, but the core contribution—a few-line improvement to C51/QRDQN with honest, reproducible experiments—deserves to see the light. I'd cite it if I work in this area.","headline":"A simple, honest drop-in overestimation-control trick for distributional RL, with solid if small experiments and a theoretical motivation that overstates the mechanism.","tokens_in":36228,"tokens_out":3628,"would_cite":true,"duration_ms":38707,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that the local spread of a distributional return estimate can be read as an overestimation signal, and that mixing Q-learning with double Q-learning according to that signal makes value learning more stable and less…","keywords":["reinforcement learning","Q-learning","double Q-learning","distributional reinforcement learning","overestimation bias","local adaptive control","sample variance","categorical and quantile distributions"],"falsifier":"Build a two-sided bandit whose two sides have equal reward variance but very different numbers of actions: the paper's own lower bound predicts stronger overestimation on the many-action side, while the variance ratio that sets $\\beta$ stays, in expectation, the same on both sides. If ADDQ does not shift its update toward double Q-learning on the many-action side and the overestimation persists, the variance signal is not the right local switch.","tokens_in":35142,"feed_emoji":"🕹️","tokens_out":10377,"duration_ms":105578,"temperature":0.7,"pith_summary":"This paper tries to establish that a distributional Q-learning agent can detect, locally and online, when its own Q-learning update is about to overestimate, and should then mix in the double Q-learning target instead. The proposed rule, ADDQ, compares an action's sample variance in the learned return distribution with the average over actions and uses that ratio to pick a mixing weight between Q-learning and double Q-learning. The claim matters because overestimation is harmful mainly when it is uneven across actions, so a fixed bias-reduction recipe for all states is the wrong tool; an automatic local switch should remove environment-specific tuning. The paper backs the rule with variance calculations for a Gaussian bandit, proves convergence in the tabular categorical setting, and reports lower bias and more stable learning in bandit, gridworld, Atari, and MuJoCo experiments.","feed_headline":"Use return variance to choose Q-learning vs double Q-learning","feed_subtitle":"ADDQ mixes the two updates per state, cutting overestimation bias in distributional RL without retuning.","key_machinery":"The load-bearing object is the relative sample variance $S^2_{\\mathrm{rel}}(s,a)=S^2_{s,a}/S^2_s$, computed from the atoms of the two learned return-distribution estimates; $S^2_{s,a}$ averages the sample variances of $\\eta^A$ and $\\eta^B$ for that action, and $S^2_s$ is the average across actions. This ratio is the local uncertainty signal, and the three-bin rule in Equation (1) converts it into the interpolation weight $\\beta$ for the mixture target. The distributional push-forward update $b_{r,\\gamma\\#}$ carries the signal: the target measure is the push-forward of $\\beta\\eta^A+(1-\\beta)\\eta^B$, so the chosen $\\beta$ continuously shifts the update from Q-learning-like to double-Q-learning-like.","core_discovery":"The central claim is that the sample variance of the distributional return estimate $\\eta_t(s,a)$ is a usable local proxy for the overestimation risk of the Q-learning max, and that mixing Q-learning with double Q-learning according to the relative variance beats either estimator used globally. In the ADDQ update, two categorical return distributions $\\eta^A,\\eta^B$ are kept; for the greedy action $a^*$ the target measure is $\\beta\\,\\eta^A(s',a^*) + (1-\\beta)\\,\\eta^B(s',a^*)$, with $\\beta=0.75,0.5,0.25$ according as the action's relative sample variance $S^2_{s,a}/S^2_s$ is below, near, or above the cross-action average. The variance signal is motivated by bandit calculations: the Q-learning bias grows like $\\sigma\\sqrt{\\log k}/\\sqrt{N}$, while the sample variance of the return estimate is $\\frac{\\sigma^2}{N-1}\\chi^2_{N-1}$-distributed and computable from the atoms. Theorem 3.1 states that in the tabular categorical setting, under Robbins-Monro step sizes and with $\\beta$ sequences that depend only on the past and satisfy $|\\beta^A_t-\\beta^B_t|\\to 0$ almost surely, the induced Q-values converge almost surely to $Q^*$; with a unique optimal policy the return distributions converge to a categorical fixed point whose greedy policy is optimal. Function approximation is deliberately left out of the theory.","pith_inferences":["Our inference: the same variance signal could replace fixed ensemble sizes or truncation counts, giving every state its own bias-variance trade-off instead of one global setting; the paper names this direction as future work for REDQ and TQC.","Our inference: because the sample-variance distribution in Proposition 2.2 does not depend on the number of actions $k$, while the overestimation bound grows with $\\sqrt{\\log k}$, an environment with equal variances but very different action counts is the sharpest available test of whether variance alone is the right switch.","Our inference: the fixed thresholds are the main tunable part of ADDQ; a learned map from the full shape of the return distribution, rather than just its variance, to $\\beta$ would be the natural next step and could remove the residual hyperparameter sensitivity.","Our inference: the theory is built on Gaussian rewards and cyclic exploration, so the most informative empirical check is to vary the tail shape of rewards while holding variance fixed; if the variance proxy still predicts bias, the mechanism is more general than the paper's proofs establish."],"forward_implications":["In the tabular categorical setting, ADDQ retains a convergence guarantee: the induced Q-values reach $Q^*$ almost surely whenever the two $\\beta$ chains merge asymptotically and step sizes satisfy the usual stochastic-approximation conditions.","Because the modification is confined to the target definition, existing categorical and quantile distributional algorithms can implement ADDQ with a few lines of code and without new hyperparameters beyond the three thresholds.","On the gridworld, ADDQ's total estimation bias is smaller than plain Q-learning, plain double Q-learning, Maxmin, ensemble bootstrapped Q-learning, and REDQ in the paper's comparisons, and the threshold choice is reported as harmless across an ablation study.","Across 10 Atari environments with both categorical and quantile parametrizations, ADDQ runs never fail completely and show higher probability-of-improvement and normalized-score aggregates than distributional DQN and its double-estimator variant.","For MuJoCo critic estimation, ADDQ improves single and double quantile critics but is not competitive with clipped double critics; the paper presents this as an expected boundary rather than a failure mode."],"supporting_citations":[{"why":"Supplies categorical distributional DQN (C51), the base algorithm ADDQ modifies.","marker":"(Bellemare et al., 2017)"},{"why":"Supplies the categorical distributional Q-learning convergence framework and the measure-mixture target used in ADDQ's proof.","marker":"(Rowland et al., 2018)"},{"why":"Supplies double Q-learning, the second estimator ADDQ interpolates toward when variance is high.","marker":"(van Hasselt, 2010)"},{"why":"Establishes the overestimation-from-function-approximation problem that motivates local bias control.","marker":"(Thrun & Schwartz, 1993)"},{"why":"Supplies the SARSA-trick stochastic-approximation lemma that carries the convergence proof.","marker":"(Singh et al., 2000)"},{"why":"Supplies weighted double Q-learning, the constant-weight baseline that ADDQ generalizes with variance-based local weights.","marker":"(Zhang et al., 2017)"},{"why":"Supplies quantile distributional reinforcement learning, the second architecture where ADDQ is tested.","marker":"(Dabney et al., 2018)"},{"why":"Supplies the clipped double estimator used as the strong MuJoCo baseline that ADDQ does not beat.","marker":"(Fujimoto et al., 2018)"}],"fun_headline_variants":["Adaptive variance chooses Q vs double Q in distributional RL","Variance-based mixing of Q and double Q cuts overestimation","Use return variance to adapt overestimation control per state","Mix Q and double Q updates by local sample variance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an action's relative sample variance in the learned return distribution is a faithful local proxy for how much Q-learning would overestimate at that action, so the mixing weight derived from that variance still does the right thing when neural networks approximate the distributions.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive variance chooses Q vs double Q in distributional RL","Variance-based mixing of Q and double Q cuts overestimation","Use return variance to adapt overestimation control per state","Mix Q and double Q updates by local sample variance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1553,"prompt_tokens":983,"completion_tokens":570,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":501}},"tokens_in":599,"tokens_out":570,"duration_ms":6437,"temperature":1.0,"reasoning_tokens":501,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:06:24.622521+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a two-sided bandit whose two sides have equal reward variance but very different numbers of actions: the paper's own lower bound predicts stronger overestimation on the many-action side, while the variance ratio that sets $\\beta$ stays, in expectation, the same on both sides. If ADDQ does not shift its update toward double Q-learning on the many-action side and the overestimation persists, the variance signal is not the right local switch.","supporting_citations":[],"review_version":1}