REVIEW 3 major objections 5 minor 18 references
ADDQ: Adaptive Distributional Double Q-Learning
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper argues that the local spread of a distributional return estimate can be read as an overestimation signal, and that mixing Q-learning with double Q-learning according to that signal makes value learning more stable and less…
desk verdict A simple, honest drop-in overestimation-control trick for distributional RL, with solid if small experiments and a theoretical motivation that overstates the mechanism. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the relative sample variance $S^2_{\mathrm{rel}}(s,a)=S^2_{s,a}/S^2_s$, computed from the atoms of the two learned return-distribution estimates; $S^2_{s,a}$ averages the sample variances of $\eta^A$ and $\eta^B$ for that action, and $S^2_s$ is the average across actions. This ratio is the local uncertainty signal, and the three-bin rule in Equation (1) converts it into the interpolation weight $\beta$ for the mixture target. The distributional push-forward update $b_{r,\gamma\#}$ carries the signal: the target measure is the push-forward of $\beta\eta^A+(1-\beta)\eta^B$, so the chosen $\beta$ continuously shifts the update from Q-learning-like to double-Q-learning-like.
What would settle it
Build a two-sided bandit whose two sides have equal reward variance but very different numbers of actions: the paper's own lower bound predicts stronger overestimation on the many-action side, while the variance ratio that sets $\beta$ stays, in expectation, the same on both sides. If ADDQ does not shift its update toward double Q-learning on the many-action side and the overestimation persists, the variance signal is not the right local switch.
Extended reading notes
Core claim
The central claim is that the sample variance of the distributional return estimate $\eta_t(s,a)$ is a usable local proxy for the overestimation risk of the Q-learning max, and that mixing Q-learning with double Q-learning according to the relative variance beats either estimator used globally. In the ADDQ update, two categorical return distributions $\eta^A,\eta^B$ are kept; for the greedy action $a^*$ the target measure is $\beta\,\eta^A(s',a^*) + (1-\beta)\,\eta^B(s',a^*)$, with $\beta=0.75,0.5,0.25$ according as the action's relative sample variance $S^2_{s,a}/S^2_s$ is below, near, or above the cross-action average. The variance signal is motivated by bandit calculations: the Q-learning bias grows like $\sigma\sqrt{\log k}/\sqrt{N}$, while the sample variance of the return estimate is $\frac{\sigma^2}{N-1}\chi^2_{N-1}$-distributed and computable from the atoms. Theorem 3.1 states that in the tabular categorical setting, under Robbins-Monro step sizes and with $\beta$ sequences that depend only on the past and satisfy $|\beta^A_t-\beta^B_t|\to 0$ almost surely, the induced Q-values converge almost surely to $Q^*$; with a unique optimal policy the return distributions converge to a categorical fixed point whose greedy policy is optimal. Function approximation is deliberately left out of the theory.
Load-bearing premise
The load-bearing premise is that an action's relative sample variance in the learned return distribution is a faithful local proxy for how much Q-learning would overestimate at that action, so the mixing weight derived from that variance still does the right thing when neural networks approximate the distributions.
Editorial extensions
If this is right
- In the tabular categorical setting, ADDQ retains a convergence guarantee: the induced Q-values reach $Q^*$ almost surely whenever the two $\beta$ chains merge asymptotically and step sizes satisfy the usual stochastic-approximation conditions.
- Because the modification is confined to the target definition, existing categorical and quantile distributional algorithms can implement ADDQ with a few lines of code and without new hyperparameters beyond the three thresholds.
- On the gridworld, ADDQ's total estimation bias is smaller than plain Q-learning, plain double Q-learning, Maxmin, ensemble bootstrapped Q-learning, and REDQ in the paper's comparisons, and the threshold choice is reported as harmless across an ablation study.
- Across 10 Atari environments with both categorical and quantile parametrizations, ADDQ runs never fail completely and show higher probability-of-improvement and normalized-score aggregates than distributional DQN and its double-estimator variant.
- For MuJoCo critic estimation, ADDQ improves single and double quantile critics but is not competitive with clipped double critics; the paper presents this as an expected boundary rather than a failure mode.
Reading between the lines
- Our inference: the same variance signal could replace fixed ensemble sizes or truncation counts, giving every state its own bias-variance trade-off instead of one global setting; the paper names this direction as future work for REDQ and TQC.
- Our inference: because the sample-variance distribution in Proposition 2.2 does not depend on the number of actions $k$, while the overestimation bound grows with $\sqrt{\log k}$, an environment with equal variances but very different action counts is the sharpest available test of whether variance alone is the right switch.
- Our inference: the fixed thresholds are the main tunable part of ADDQ; a learned map from the full shape of the return distribution, rather than just its variance, to $\beta$ would be the natural next step and could remove the residual hyperparameter sensitivity.
- Our inference: the theory is built on Gaussian rewards and cyclic exploration, so the most informative empirical check is to vary the tail shape of rewards while holding variance fixed; if the variance proxy still predicts bias, the mechanism is more general than the paper's proofs establish.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ADDQ, a modification of distributional Q-learning in which the Q-learning target and the double Q-learning target are mixed with a state-action-dependent weight β, chosen from the relative sample variance of the estimated return distributions. The authors provide two bandit-style propositions (a lower bound on Q-learning overestimation and an exact chi-squared law for the sample variance of the return estimate), a convergence theorem for the tabular categorical setting, grid-world and two-sided-bandit experiments, and deep RL experiments with C51 and QRDQN on Atari and with quantile SAC on MuJoCo. The central claim is that the local spread of the estimated return distribution identifies when Q-learning updates are prone to skewed overestimation, so that mixing in the double estimator locally improves bias and stability.
Significance. If the mechanism were fully established, ADDQ would be a cheap, few-line improvement to existing distributional algorithms, with an actual convergence proof in the tabular categorical case and a reproducible code base. The paper's strengths include a detailed convergence proof (Appendix F), an ablation over β schedules (Appendix C.2), the use of RLiable statistics for the deep experiments, and the fact that the symmetric β rule in Eq. (1) does satisfy the coupling condition lim |β^A_t − β^B_t| = 0 of Theorem 3.1. However, as detailed in the major comments, the theoretical support for the core variance-bias link is incomplete, and one step of the convergence proof as written contains a gap. The contribution is therefore promising but not yet fully supported.
major comments (3)
- [Section 2.4, Propositions 2.1 and 2.2, Eq. (1)] The central mechanism is not established by the two bandit propositions. Proposition 2.2 shows that, in the idealized bandit, the sample variance of the return estimate is σ²/(N−1)χ²_{N−1}, with expectation σ², while Proposition 2.1 shows that the overestimation bias is proportional to σ√(log k)/√N. The sample variance therefore contains no information about the action count k and only a shrinking fluctuation component about N; two actions with identical reward variance but different k or different visit counts will receive the same β from Eq. (1), so ADDQ cannot adapt to the k- and N-driven skew that the paper's own lower bound identifies. Additionally, Proposition 2.2 is derived for the empirical measure (1/N)Σδ_{X_i} after a one-shot bootstrap, whereas Algorithm 2 computes S² from the categorical or quantile approximations η^A_t(s,a) before the update; the exact chi-squared law does not transfer to the projected fixed-atom representations used in the grid-world and Atari experiments. The statement in Section 2.4 that the agent 'does have access to the σ²χ²_N/N-distributed sample variance by computing sums of the atoms' is therefore not justified for the algorithm actually run.
- [Appendix F, proof of Theorem F.2, Step 1] The proof of convergence of Q^A_t and Q^B_t to Q* contains a gap in the bound on |E[\tilde F_n | F_n]|. The text first bounds one of the terms by ||X^BA_n||∞ + 1/2||X^BA_n|| and then 'combines' the terms as though the coefficient were only ||X^BA_n||∞. With the displayed bound, the effective contraction coefficient in Lemma F.3 can exceed 1 for γ close to 1, so the SARSA-trick argument does not go through as written. The gap is repairable: using the maximality of a* with respect to Q^A and of b* with respect to Q^B gives the sharper bound |E[Q^A_n(S_{n+1},b*) − Q^B_n(S_{n+1},a*)|F_n]| ≤ ||X^BA_n||∞. The proof should be corrected before the theorem is relied upon.
- [Section 3.2 and Algorithm 2] There is a timing mismatch between the theory and the algorithm's use of the variance proxy. In the bandit analysis, the quantity whose distribution is computed in Proposition 2.2 is the sample variance of the return estimate after the bootstrap propagation from the next state. In Algorithm 2, however, β for updating (s,a) is determined from η^A_old and η^B_old before the update, and the paper does not specify or analyze how the pre-update sample variance at (s,a) relates to the overestimation introduced by the max at the next state (s',a*). The paper should either prove a statement about this pre-update proxy or explicitly acknowledge that the deep and sequential tabular use of the proxy is a heuristic not covered by the bandit propositions.
minor comments (5)
- [Section 3.1 vs Eq. (1)] The prose in Section 3.1 says that when a lot of uncertainty is present the algorithm uses large β, but Eq. (1) assigns β=0.25 to high relative variance and β=0.75 to low relative variance; the text should be reworded to avoid this apparent contradiction.
- [Definitions in Section 3.2] The definition S²(ν) = (1/(n−1))Σ p_i(a_i − M(ν))² with n atoms is a weighted variance of the atoms, not the classical sample variance of observations; the statistical analogy should be stated more carefully, especially because the chi-squared law of Proposition 2.2 relies on equal weights 1/N rather than on the categorical parametrization.
- [Proposition 2.1] The phrase 'the left side has been explored N k1 times' is ambiguous; from Theorem A.2 it appears to mean N episodes with k1 actions each, i.e., N k1 total arm pulls, and this should be stated explicitly.
- [Typos] There are several typographical issues: 'Theoerem' in the heading of Appendix F, 'prevers' in Section 4, 'red off' in Section 3.2, 'Q-learing' in Appendix A, and 'chosing' in the proof of Theorem A.2.
- [Experimental reporting] The Atari and MuJoCo results are presented through learning curves and RLiable plots; adding a table with mean, median, and interquartile mean of normalized scores would make the aggregate comparison easier to verify.
Circularity Check
No significant circularity: the bandit propositions are standalone, the beta schedule is an explicit heuristic, and the convergence theorem does not assume the empirical gains.
full rationale
The load-bearing chain is not circular. Proposition 2.1 derives a lower bound on Q-learning overestimation in a Gaussian bandit MDP using external Sudakov-Fernique comparisons and a target-matrix/cyclic-exploration reduction; it does not assume ADDQ's adaptive weighting. Proposition 2.2 computes the law of the sample variance of the distributional return estimate as sigma^2/(N-1) times a chi-squared variable; this is a standard statistics calculation, independent of the algorithm's success. Equation (1) is explicitly an 'exemplary choice' and the paper states 'There are many other possibilities to choose beta but we decided to fix this example for all our experiments', with thresholds acknowledged as hyperparameters. The beta schedule is therefore a heuristic motivated by, but not derived from, the propositions. The variance proxy's failure to capture the k- and N-dependence identified in Proposition 2.1 is a validity limitation of that heuristic, not a circular reduction, because the paper nowhere claims S^2_rel is mathematically equivalent to the overestimation bound. Theorem 3.1 treats beta as an arbitrary past-dependent sequence satisfying a symmetry condition and does not use the variance proxy's faithfulness in the convergence argument. No parameter is fitted to the evaluation metric and then renamed a prediction, and there are no load-bearing self-citations. Any concerns about the variance proxy should be assessed as correctness/robustness risks, not circularity.
Assumptions & free parameters
free parameters (1)
- Beta thresholds and values in Equation (1) =
0.75 / 0.5 / 0.25 with thresholds 0.75 and 1.25
assumptions (5)
- domain assumption Gaussian reward distributions for the bandit propositions
- domain assumption Return distributions are initialized and stay supported within [θ1, θm], and the true value interval lies inside it
- standard math Robbins-Monro step-size conditions and random symmetric updates of A and B
- ad hoc to paper The sample variance of the return distribution indicates local overestimation bias
- standard math Sudakov-Fernique comparison inequality and Gaussian maxima bounds
Cite this review
Pith. "Pith review of ADDQ: Adaptive Distributional Double Q-Learning." pith.science (2026). https://pith.science/paper/SZCEQXMG
@misc{pith2026250619478,
author = {Pith},
title = {Pith review of: ADDQ: Adaptive Distributional Double Q-Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/SZCEQXMG}},
note = {Machine review of arXiv:2506.19478}
}
abstract
Bias problems in the estimation of $Q$-values are a well-known obstacle that slows down convergence of $Q$-learning and actor-critic methods. One of the reasons of the success of modern RL algorithms is partially a direct or indirect overestimation reduction mechanism. We propose an easy to implement method built on top of distributional reinforcement learning (DRL) algorithms to deal with the overestimation in a locally adaptive way. Our framework is simple to implement, existing distributional algorithms can be improved with a few lines of code. We provide theoretical evidence and use double $Q$-learning to show how to include locally adaptive overestimation control in existing algorithms. Experiments are provided for tabular, Atari, and MuJoCo environments.
Figures
Figures from the paper (18 more)
Reference graph
Works this paper leans on
-
[1]
N(µ 2, σ2 2)) distributed rewards
(resp. N(µ 2, σ2 2)) distributed rewards. Suppose µ1 > µ2 are small but σ2 1 ≪σ 2 2 and/or k1 ≪k 2. The MDP is delicate as the following can happen using QL (overestimation) and DQL (global overestimation reduction). In QL the agent will believe for a long time that ”right” is the optimal action in s0. In DQL the agent will believe for a long time that ”d...
-
[2]
with concentration depending on k1 (resp. k2). The local overestimation control of β (based on relative sample variances) from (1) thus compares σ2 1 to σ2 2 and suggests to mitigate overestimation of ˆQ(s0,”right”) . In our algorithm we use double Q-learning to mitigate, the same idea can of course be integrated into other algorithms (such as changing nu...
work page 2020
-
[3]
the choice of updatingη A orη B is random and independent of all previous random variables
-
[4]
the sequences(β A t )t∈N,(β B t )t∈N only depend on the past and fulfilllim t→∞ |βA t −β B t |= 0almost surely. If additionally the MDP has a unique optimal policy π∗, then (ηA t ),(η B t ) converge almost surely in ¯ℓ2 to some limit η∗ C ∈ FC,m and the greedy policy with respect toη ∗ C is the optimal policy. Note that the algorithm and proof uses βA/B t...
work page 2010
-
[10]
the step sizes αt(s, a)almost surely fulfill the Robbins-Monro conditionsP∞ t=0 αt(s, a) =∞and P∞ t=0 α2 t (s, a)<∞
-
[14]
, dsuch that Fn is Fn+1-measurable and for alli= 1,
a stochastic process (Fn)n∈N ⊂R d with the coordinates Fi,n for i= 1, . . . , dsuch that Fn is Fn+1-measurable and for alli= 1, . . . , d ∥E[Fn|Fn]∥∞ ≤κ∥X n∥∞ +c n andV[F i,n|Fn]≤K(1 +κ∥X n∥∞)2 n≥1, where κ∈[0,1) , an adapted, stochastic process (cn)n∈N ⊂R + that converges to 0 almost surely and some constant K >0. 39 ADDQ: Adaptive Distributional DoubleQ...
-
[15]
, dis adapted with ∞X n=1 αi,n =∞and ∞X n=1 α2 i,n <∞a.s
the non-negative stochastic process (αn)n∈N ⊂R d, with the coordinates αi,n ∈[0,1] for i= 1, . . . , dis adapted with ∞X n=1 αi,n =∞and ∞X n=1 α2 i,n <∞a.s.. Then, for any F0-measurable initial condition X0 the stochastic process (Xn)n∈N ⊂R d with coordinates Xi,n for i= 1, . . . , dthat is recursively defined by Xi,n+1 = (1−α i,n)Xi,n +α i,nFi,n, n∈N, co...
work page 2018
-
[16]
the step sizesα t(s, a)fulfill the Robbins-Monro conditions: • P∞ t=0 αt(s, a) =∞ • P∞ t=0 α2 t (s, a)<∞,
Show all 18 references
-
[17]
rewards are bounded in[R min, Rmax]and[ Rmin 1−γ , Rmax 1−γ ]⊆[θ 1, θm],
-
[18]
the choice of updatingη A orη B is random and independent of all other previous random variables The above result is only relevant for the proof of Theorem F.2, as policy evaluation with a double estimator is not of interest. Note that convergence of categorical temporal diffe...
2018
-
[19]
Proof of Thoerem F .2.Step 1: Convergence of mean values toQ ∗ The proof mainly follows (Rowland et al., 2018) and (van Hasselt, 2010)
=∥ν 1 −ν ′ 1∥2 ℓ2 +∥ν 2 −ν ′ 2∥2 ℓ2 + 2⟨ν1 −ν ′ 1, ν2 −ν ′ 2⟩ holding by bilinearity of the inner product. Proof of Thoerem F .2.Step 1: Convergence of mean values toQ ∗ The proof mainly follows (Rowland et al., 2018) and (van Hasselt, 2010). Let the filtration be given by Ft ...
2018
-
[2000]
Sudakov, V
doi: 10.1023/A:1007678930559. Sudakov, V . N. Gaussian random processes, and measures of solid angles in Hilbert space.Dokl. Akad. Nauk SSSR, 197:43–45, 1971. ISSN 0002-3264. Sudakov, V . N. Geometric problems of the theory of infinite- dimensional probability distributions.Tr...
1971 doi
-
[2010]
cc/paper_files/paper/2010/file/ 091d584fced301b442654dd8c23b3fc9-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2010/file/ 091d584fced301b442654dd8c23b3fc9-Paper. pdf. van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double q-learning. Preprint available on arXiv:1509.06461, 2015. URL http://arxiv.org/abs/15...
2010 arXiv
-
[2013]
Bellemare, M
ISSN 1076-9757. Bellemare, M. G., Dabney, W., and Munos, R. A distribu- tional perspective on reinforcement learning. InProceed- ings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, pp. 449–458. JMLR.org, 2017. Bellemare, M. G., Dabney, W., and R...
2017 arXiv
-
[2017]
Badia, A., Piot, B., Kapturowski, S., Sprechmann, P., Vitvit- skyi, A., Guo, D., and Blundell, C
URL https://proceedings.mlr.press/ v70/anschel17a.html. Badia, A., Piot, B., Kapturowski, S., Sprechmann, P., Vitvit- skyi, A., Guo, D., and Blundell, C. Agent57: Outper- forming the atari human benchmark, 03 2020. URL http://arxiv.org/abs/2003.13350. Bellemare, M. G., Naddaf,...
2020 arXiv
-
[2018]
Thrun, S
URL http://incompleteideas.net/ book/the-book-2nd.html. Thrun, S. and Schwartz, A. Issues in using function approximation for reinforcement learning. In Mozer, M., Smolensky, P., Touretzky, D., Elman, J., and Weigend, A. (eds.),Proceedings of the 1993 Connectionist Models Summ...
1993
-
[2019]
Mnih, V ., Kavukcuoglu, K., Silver, D., Rusu, A
URL http://dblp.uni-trier.de/db/ conf/icml/icml2019.html#MavrinYKWY19. Mnih, V ., Kavukcuoglu, K., Silver, D., Rusu, A. A., Ve- ness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, ...
2015 arXiv
-
[2021]
cc/paper_files/paper/2021/file/ f514cec81cb148559cf475e7426eed5e-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2021/file/ f514cec81cb148559cf475e7426eed5e-Paper. pdf. Anschel, O., Baram, N., and Shimkin, N. Averaged-DQN: Variance reduction and stabilization for deep reinforce- ment learning. In Precup, D. and Teh, Y . W. (eds.),Pro-...
2021
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.