{"id":"94af8e9b-8764-490e-90ea-076935df349b","arxiv_id":"2507.09523","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"QV-learning has an expected contraction for on-policy prediction, QVMAX is biased for off-policy control, and a new regularized dueling algorithm (RDQ) outperforms Dueling DQN on MinAtar.","lead":"This paper analyzes two families of reinforcement learning algorithms that learn state values alongside action values, and introduces a new variant that outperforms Dueling DQN on MinAtar games. It combines theoretical convergence results with empirical comparisons to clarify when learning two value functions helps.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MinAtar RDQ-vs-Dueling-DQN comparison changes two things at once: it removes the mean-advantage identifiability term and adds an untuned l2 penalty; since the penalty is a generic output regularizer, the reported gains may not support the paper's minimum-norm mechanism, and no ablation…","rationale":"The paper has three kinds of claims: contraction theory (Theorem 3.1), bias identification (Propositions 3.2 and 3.3), and the empirical superiority of RDQ over Dueling DQN (Section 4.3). The theory is internally consistent; I re-derived the joint operator in Theorem 3.1 and its sup-norm contraction (each row of A sums to gamma < 1) and found no error. The empirical claim is the one identified as strongest by the Reader, and its least secure premise is causal attribution. Soft RDQ differs from Dueling DQN in two coupled ways: the Q-reconstruction (no mean subtraction) and the additional l2 penalty on V and Adv outputs. The paper explains the gain via the geometric minimum-norm idea of Figure 4, but the direct empirical support for that mechanism is thin: in the tabular setting where the two factors are separable (Hard RDQ versus Dueling Q-learning), Hard RDQ's advantage is \"slight\" and occurs only in a \"small majority\" of MDP instances. The deep setting, where the largest gains occur, is exactly the setting where the clean invariant (Eq. 15) does not carry over, so the min-norm story is heuristic there. Meanwhile the l2 penalty is a known generic stabilizer in deep RL: it pushes function outputs toward zero, dampening the magnitude of TD errors and the scale of bootstrapped targets under Adam. Because the paper does not run Dueling DQN with the same penalty, the alternative hypothesis, that regularization rather than the dueling-specific decomposition drives the improvement, remains entirely plausible and consistent with the published curves. This concern is concrete and settleable: the proposed 2x2 ablation uses the existing codebase and hyperparameters, only toggling the mean term and the penalty. If arm C (Dueling DQN plus penalty) matches arm B (Soft RDQ), contribution 3's mechanism claim is refuted even though RDQ would still outperform the original Dueling DQN baseline; the paper's framing would then need to change to \"output-weight decay helps dueling networks,\" a much weaker claim. If arm B beats arm C across a beta sweep, the confound is resolved in the paper's favor. The lack of statistical testing (\"significantly\") is a lesser concern; with 30 seeds and visible CI separation, the informal claim is defensible, but the ablation is essential. The Reader's weakest assumption overlaps only partially: the Reader flagged the fairness of the comparison (untuned beta, tuned baselines), while my concern is the confound within the algorithmic change itself. The condition for acceptance should therefore be the 2x2 ablation (or an equivalent isolation of the l2-regularization effect) rather than only additional tuning or statistical tests.","tokens_in":15546,"tokens_out":14561,"duration_ms":154357,"concrete_test":"Run a 2x2 ablation in MinAtar (all five games, same seeds, architecture, and hyperparameters as Section 4.3): (A) Dueling DQN (Eq. 8, no penalty); (B) Soft RDQ (Eq. 11, with penalty); (C) Dueling DQN with the l2 penalty of Eq. 12 added to its loss; and (D) Soft RDQ with the mean-advantage term re-inserted into the Q reconstruction. Sweep beta over {1e-4, 1e-3, 1e-2} for arms B and C. If C matches B at 10M steps, the reported gain is generic l2 regularization, and the paper's minimum-norm/no-identifiability mechanism is not supported. If B clearly beats C, and B remains best across the beta range, the mechanism attribution survives. Report mean returns with 95% confidence intervals and effect sizes at 10M steps.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 compares Soft RDQ against Dueling DQN in MinAtar while varying two factors simultaneously: the Q-reconstruction formula (Eq. 8 with mean subtraction versus Eq. 11 without it) and the loss (Eq. 16 versus Eq. 16 plus the l2 penalty of Eq. 12). The paper's narrative attributes the large gains to the minimum-norm solution that emerges when the identifiability term is removed (Figure 4, \"may require fewer updates to converge\"), but the l2 penalty on the outputs V(s) and Adv(s,a) is exactly output-layer weight decay, a well-known stabilizer for deep RL that shrinks the scale of bootstrapped targets. Nothing in the paper rules out the hypothesis that the entire MinAtar advantage of RDQ comes from this generic regularization, which would equally help Dueling DQN if added to its loss. The evidence offered for the min-norm mechanism is weak where the factors are separable: in the tabular control experiment (Section 4.2), Hard RDQ (no penalty, no mean term) only \"slightly outperforms Dueling Q-learning in a small majority of cases\" (Figure 5, right). Moreover, the invariant (Eq. 15) that defines Hard RDQ's minimum-norm property does not extend to function approximation, and the authors state there is no simple extension; thus the deep result rests entirely on the untuned penalty with beta = 1e-3 (Section 4.3), with no sensitivity analysis. The paper honestly notes beta was not tuned, but that caveat does not address the confound. As a result, the load-bearing premise of contribution 3, that the algorithmic change, specifically the removal of the identifiability term via an l2 penalty, is what improves performance, is untested.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies temporal-difference methods that learn both state-value and action-value functions, dividing them into QV-learning (directly updating Q and V) and AV-learning (decomposing Q into V plus an advantage function). The main theoretical results are: (i) Theorem 3.1, showing that the expected QV-learning update is an affine contraction with fixed point [q_b; v_b]; (ii) Proposition 3.2, showing that QVMAX is biased in off-policy control; and (iii) Proposition 3.3, giving the unique fixed point of a newly proposed bias-corrected variant, BC-QVMAX. The paper then formalizes tabular Dueling Q-learning and introduces Regularized Dueling Q-learning (RDQ), which replaces the mean-advantage identifiability subtraction with an l2 penalty; in MinAtar experiments, Soft RDQ is reported to significantly outperform DQN and Dueling DQN under the same network architecture.","tokens_in":15934,"tokens_out":14245,"duration_ms":150967,"significance":"The theoretical analysis is clean and fills a real gap: to my knowledge, no prior published convergence proof for QV-learning exists, and the bias of QVMAX had not been identified. The tabular experiments are careful, with step-size sweeps and confidence intervals, and the code is publicly available. The RDQ idea is interesting and the paper provides a clear geometric motivation. However, the deep RL result, which is the paper's headline empirical claim, is compromised by a confounded comparison: Soft RDQ differs from Dueling DQN in both the output decomposition and the presence of an l2 penalty, and no ablation isolates the mechanism. As a result, the support for the minimum-norm explanation is weak, though the algorithm itself may still be valuable.","major_comments":[{"comment":"The MinAtar comparison of Soft RDQ against Dueling DQN changes two factors at once: Q(s,a) is computed without the mean-advantage subtraction (Eq. 11 versus Eq. 8), and the loss includes the l2 penalty of Eq. (12). The l2 penalty is a generic output regularizer, so the reported gains cannot be attributed to the minimum-norm mechanism without additional controls. The tabular Hard RDQ, which removes the mean term but has no penalty, only slightly outperforms Dueling Q-learning in a small majority of cases (Figure 5, right), indicating that removing the identifiability term alone does not explain the large deep RL improvements. The authors should add ablations: (a) Dueling DQN with the same l2 penalty added to its loss, (b) Soft RDQ with the mean-advantage term retained, and (c) a sensitivity analysis over beta. This is necessary to support the paper's central claim that RDQ's advantage comes from the proposed minimum-norm decomposition rather than from generic output regularization.","section":"Section 4.3, Eq. (12), Figure 6"},{"comment":"The abstract states that 'both families are more efficient than Expected Sarsa in the prediction setting,' but the prediction-setting experiment in Section 3.1 evaluates QV-learning only; no prediction experiment for an AV-learning method is presented. This overstates the evidence. Please either add a prediction experiment for an AV-learning method or revise the abstract to say that QV-learning was found to be more efficient than Expected Sarsa in prediction, and that the control experiments showed benefits for AV-learning.","section":"Abstract"},{"comment":"The tabular Soft RDQ update is inconsistent with the loss in Eq. (12). With the penalty coefficient beta in the loss and step size alpha, ordinary gradient descent would give a decay factor of (1 - alpha * beta), not (1 - beta). Since alpha is typically much smaller than 1, the two formulations behave very differently. This does not affect the Hard RDQ (beta = 0) invariant or the deep RL implementation, which uses the stated loss with Adam, but the derivation as written is unclear and should be corrected or clarified.","section":"Section 4.2, Eqs. (13)-(14)"}],"minor_comments":[{"comment":"The phrase 'significantly outperforms' is used without a formal statistical test; consider reporting a paired test or explicitly discussing the overlap of the 95% confidence intervals at the evaluation points.","section":"Section 4.3"},{"comment":"The claim that QV-learning 'scales better' rests on a visual comparison of slopes; consider reporting the fitted slopes or a statistical comparison to make the claim quantitative.","section":"Figure 1 (right)"},{"comment":"The invariant V(s) = sum_a Adv(s,a) holds only up to the initialization difference V0(s) - sum_a Adv0(s,a); this is stated in the footnote but should be mentioned in the main text for clarity.","section":"Section 4.2, Eq. (15)"},{"comment":"The proof relies on operator notation from Daley (2025), a PhD thesis; since the notation is defined in the appendix, the paper is self-contained, but the reliance on a non-archival reference is slightly unusual and worth noting.","section":"Appendix A, Theorem 3.1 proof"}],"recommendation":"major_revision","confidential_remarks":"The theoretical portions of this paper are solid and novel, and the topic is a good fit for the RLJ. The main concern is the deep RL claim, which is the headline empirical contribution: the comparison is confounded and the paper needs an ablation to support the proposed mechanism. The abstract overclaim about AV-learning in the prediction setting should also be corrected. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the theory half is solid and useful; the RDQ deep RL claim is plausible but the stress-test concern is right—the MinAtar comparison is confounded and the mechanism isn't tested. I'd accept for peer review with a requested ablation.\n\nThe genuinely new stuff: expected contraction for QV-learning, which was missing from the literature; the QVMAX bias result and BC-QVMAX; and the tabular derivation of dueling Q-learning plus RDQ. The proofs are simple but correct, and the paper is unusually candid about the limits of its empirical evidence. The tabular experiments support the theory.\n\nThe soft spot is Section 4.3. RDQ changes the dueling loss in two ways at once—drops the mean-advantage term and adds an l2 penalty on the outputs. The paper credits the minimum-norm property, but the l2 penalty is a known stabilizer in deep RL, and nothing shows the gain isn't just that penalty. The tabular Hard RDQ, without the penalty, only slightly beats dueling Q-learning in a majority of cases, so the deep result is the main evidence, and it's not decomposed. Also, 'significantly' is used without a statistical test; likely true given the CIs, but a test would be proper.\n\nNone of this sinks the paper. The theoretical contributions stand. But the abstract's headline claim about RDQ is stronger than the evidence. A referee should ask for an ablation—add the same l2 penalty to the standard dueling loss—or a sensitivity sweep over beta, or both.\n\nI'd take this paper to a reading group, and I'd cite the QVMAX bias result. It deserves serious refereeing, with the RDQ section as the focus.","headline":"Solid theory on QV-learning; the RDQ deep RL claim is plausible but undersupported because the MinAtar comparison is confounded and the minimum-norm mechanism is not actually tested.","tokens_in":16494,"tokens_out":3249,"would_cite":true,"duration_ms":37779,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bootstrapping from a learned state value can be sound, and in the dueling family it is also faster: the paper proves QV-learning converges and introduces RDQ, which beats Dueling DQN on all five MinAtar games.","keywords":["TD learning","QV-learning","AV-learning","dueling architectures","advantage estimation","regularized dueling Q-learning","convergence analysis","MinAtar"],"falsifier":"Sweep $\\beta$ over a wide range in the same MinAtar setup while keeping Dueling DQN's architecture and tuned hyperparameters fixed; if some $\\beta$ lets Dueling DQN match or exceed RDQ, the claim that RDQ significantly outperforms it is weakened. In addition, a tabular MDP with a known non-greedy behavior policy should show QVMAX asymptoting to a biased fixed point with $\\|q^*-Q\\|$ bounded away from zero, as Proposition 3.2 predicts.","tokens_in":15380,"feed_emoji":"🎮","tokens_out":11936,"duration_ms":122143,"temperature":0.7,"pith_summary":"This paper asks when it pays to learn a state-value function alongside an action-value function in temporal-difference reinforcement learning, instead of learning only $Q(s,a)$. It separates such algorithms into QV-learning, where the two functions bootstrap from each other, and AV-learning, where the action value is decomposed as $Q(s,a)=V(s)+\\mathrm{Adv}(s,a)$. The authors prove that the expected QV-learning update is a contraction whose unique fixed point is the correct on-policy pair, show that the standard QV off-policy variant QVMAX is biased, and introduce a corrected version. They then propose Regularized Dueling Q-learning (RDQ), which replaces the mean-advantage subtraction of dueling networks with an $\\ell^2$ penalty, and report that RDQ significantly outperforms Dueling DQN on all five MinAtar games with the same network architecture and hyperparameters.","feed_headline":"Regularized dueling Q-learning beats Dueling DQN on MinAtar","feed_subtitle":"Same architecture and hyperparameters; the l2-regularized advantage split improves scores on all five games.","key_machinery":"At the center of the convergence result is the joint QV operator $H([q;v]) = b + A[q;v]$ with $b=[r; E_b r]$ and $A=\\gamma [0\\ P;\\ 0\\ E_b P]$. Because every row of $A$ is a probability distribution scaled by $\\gamma$, the operator norm is $\\|A\\|_\\infty = \\gamma$, so $H$ is a maximum-norm contraction and, by the fixed-point theorem, has unique fixed point $[q_b; v_b]$. At the center of the AV-learning result is the decomposition $Q(s,a)=V(s)+\\mathrm{Adv}(s,a)$ together with the $\\ell^2$ penalty $\\frac{1}{2}V(s)^2 + \\frac{1}{2}\\sum_a \\mathrm{Adv}(s,a)^2$, which replaces the mean-advantage subtraction of the original dueling network. The $\\ell^2$ penalty selects the point on the solution line closest to zero, the minimum-norm solution, and in the tabular setting the penalty-free invariant $V(s)=\\sum_a \\mathrm{Adv}(s,a)$ gives the same geometric mechanism for Hard RDQ.","core_discovery":"The paper's central claim is that learning a state-value function as an intermediate step in learning action values can be sound and useful, but the two families of methods behave differently. For QV-learning, the expected update is an affine joint operator $H([q;v]) = b + A[q;v]$, with $b=[r; E_b r]$ and $A=\\gamma [0\\ P;\\ 0\\ E_b P]$, and since $\\|A\\|_\\infty=\\gamma<1$, the operator is a contraction whose unique fixed point is the on-policy pair $[q_b; v_b]$; this is the paper's proof that both value estimates progress together. For off-policy control it shows QVMAX is biased because its $V$ update conditions on the state only, so $[q^*; v^*]$ is not a fixed point, and it gives a corrected version, BC-QVMAX, whose unique fixed point is $[q^*; v^*]$ but which behaves like a lagged Q-learning. The AV-learning analysis derives a tabular version of Dueling Q-learning and shows that subtracting the mean advantage is only one way to make the decomposition identifiable; RDQ instead uses an $\\ell^2$ penalty on $V(s)$ and $\\mathrm{Adv}(s,a)$ to find a minimum-norm solution of $Q(s,a)=V(s)+\\mathrm{Adv}(s,a)$. In the deep RL setting, RDQ with the same dueling architecture and the same tuned hyperparameters as Dueling DQN outperforms Dueling DQN on all five MinAtar games, supporting the paper's overall finding that state-value learning helps in prediction for both families, but in control only AV-learning delivers major gains over Q-learning.","pith_inferences":["The success of RDQ suggests that the dueling architecture's benefit comes less from estimating true advantages than from providing a lower-dimensional parameterization, so any penalty that anchors $V$ and $\\mathrm{Adv}$ near zero may produce similar gains.","The QV-learning contraction proof likely extends to other asymmetric pairings of value functions, since the argument only requires one function to serve as a bootstrap target under a fixed behavior policy.","A per-game sweep of the regularization strength $\\beta$ would clarify whether Dueling DQN's mean-advantage subtraction is actively harmful or merely different; the paper reports only one untuned value of $\\beta$."],"forward_implications":["QV-learning converges to the on-policy pair $(q_b, v_b)$ even though $V$ never needs to converge first; the joint operator is a contraction, so both value estimates make progress together.","QVMAX is biased for off-policy control: its state-value update is conditioned only on the state, so its fixed point is not $(q^*, v^*)$, and any use of QVMAX inherits this bias.","BC-QVMAX restores the correct fixed point $(q^*, v^*)$, but its smoothing makes it essentially a delayed, learned approximation of Q-learning, so it does not deliver a sample-efficiency benefit over Q-learning.","In the AV-learning family, dueling methods can substantially outperform Q-learning in tabular control, and Hard RDQ's minimum-norm path is at least as efficient as mean-advantage subtraction.","In the MinAtar deep RL benchmark, RDQ with the same dueling architecture and the same tuned hyperparameters as Dueling DQN achieves higher mean scores in all five games.","Because RDQ's $\\ell^2$ penalty is added to the loss rather than hard-coded into the network, the same regularization can be combined with other dueling-style losses without changing the network architecture."],"supporting_citations":[{"why":"introduced QV-learning; the paper supplies the missing convergence proof by showing its expected joint update is a contraction.","marker":"Wiering (2005)"},{"why":"introduced QVMAX; the paper identifies its off-policy bias and constructs a corrected variant.","marker":"Wiering & van Hasselt (2009)"},{"why":"introduced the dueling decomposition and the mean-advantage subtraction that RDQ replaces; also provided Dueling DQN as the main control baseline.","marker":"Wang et al. (2016)"},{"why":"provided the MinAtar benchmark and the base convolutional architecture used in the deep RL experiments.","marker":"Young & Tian (2019)"},{"why":"supplied the tuned hyperparameters and baseline configurations for DQN and Dueling DQN in MinAtar.","marker":"Obando-Ceron & Castro (2021)"},{"why":"the fixed-point theorem used to conclude that the QV contraction has a unique fixed point.","marker":"Banach (1922)"},{"why":"the standard conditions under which the contraction result extends to asynchronous stochastic updates.","marker":"Bertsekas & Tsitsiklis (1996)"},{"why":"introduced Expected Sarsa, the single-action-value baseline against which both families are measured in the prediction setting.","marker":"John (1994)"}],"fun_headline_variants":["RDQ beats Dueling DQN on all five MinAtar games","Only AV-learning improves control over Q-learning","State-value TD method outperforms Dueling DQN with same settings","New algorithm: regularized advantage split improves MinAtar scores","QV-learning bias shown; AV-learning wins in control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The MinAtar comparison is clean only if the single untuned regularization strength $\\beta=10^{-3}$ is suitable for all five games and if the tuned DQN and Dueling DQN baselines were not disadvantaged by the implementation; otherwise the reported advantage could come from tuning rather than from the algorithmic change.","fun_headline_variants_meta":{"raw":{"variants":["RDQ beats Dueling DQN on all five MinAtar games","Only AV-learning improves control over Q-learning","State-value TD method outperforms Dueling DQN with same settings","New algorithm: regularized advantage split improves MinAtar scores","QV-learning bias shown; AV-learning wins in control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001844,"raw_usage":{"total_tokens":7337,"prompt_tokens":1125,"completion_tokens":6212,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":741,"completion_tokens_details":{"reasoning_tokens":6125}},"tokens_in":741,"tokens_out":6212,"duration_ms":49029,"temperature":1.0,"reasoning_tokens":6125,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:52:00.385067+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sweep $\\beta$ over a wide range in the same MinAtar setup while keeping Dueling DQN's architecture and tuned hyperparameters fixed; if some $\\beta$ lets Dueling DQN match or exceed RDQ, the claim that RDQ significantly outperforms it is weakened. In addition, a tabular MDP with a known non-greedy behavior policy should show QVMAX asymptoting to a biased fixed point with $\\|q^*-Q\\|$ bounded away from zero, as Proposition 3.2 predicts.","supporting_citations":[{"cited_title":"Dueling network architectures for deep reinforcement learning","cited_arxiv_id":null,"evidence_quote":"introduced the dueling decomposition and the mean-advantage subtraction that RDQ replaces; also provided Dueling DQN as the main control baseline."},{"cited_title":"Revisiting Rainbow : Promoting more insightful and inclusive deep reinforcement learning research","cited_arxiv_id":null,"evidence_quote":"supplied the tuned hyperparameters and baseline configurations for DQN and Dueling DQN in MinAtar."},{"cited_title":"Sur les op \\'e rations dans les ensembles abstraits et leur application aux \\'e quations int \\'e grales","cited_arxiv_id":null,"evidence_quote":"the fixed-point theorem used to conclude that the QV contraction has a unique fixed point."}],"review_version":1}