{"id":"5a0fb3bc-3a00-443f-aee6-5706605727e7","arxiv_id":"2608.12973","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Quantile temporal difference learning iterates satisfy a functional central limit theorem, enabling an online, covariance-free confidence interval for quantile return estimates.","lead":"This paper proves that quantile temporal difference learning, a widely used algorithm for estimating return distributions in reinforcement learning, has well-behaved statistical fluctuations, and it uses that fact to build a confidence-interval method that runs online with constant memory. The new inference tool lets practitioners quantify uncertainty in estimated quantile returns without storing the whole learning trajectory.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FCLT and the online inference procedure rest on Assumption 2 and an unproved imported lemma; if boundary-degenerate quantile differences occur, the local linearization and the Brownian limit are unsupported.","rationale":"The paper's central claim is a conditional one: under Assumptions 1–2 and the standard decreasing step-size schedule used in Theorem 4.1, the averaged QTD iterates satisfy an FCLT and the random-scaling statistic is asymptotically pivotal. Within those conditions the derivation is coherent: the synchronous and asynchronous embeddings into Theorem 4.1 have the right mean-field structure, the covariance algebra for the asynchronous case is consistent, and the continuous-mapping argument for Theorem 3.3 is standard. The most fragile point is not the algebra but the smoothness/degeneracy condition that makes the local linearization (9) valid. Assumption 2 is exactly the condition that keeps the reward CDF away from its non-differentiable endpoints at the fixed-point quantiles, and it is also what makes Lemma 4.3's positivity argument work. The paper flags this as a technical condition, but it is load-bearing rather than cosmetic: if it fails, h can have a kink at θ_m and the Brownian limit is not obtained by the stated proof. In addition, the proof of uniqueness (Lemma 4.1) and the repulsiveness of G (Lemma 4.3) rely on Lemma A.2 imported from the authors' own unpublished preprint [6]; this is a missing support point that a reviewer cannot independently check from the submitted text. These observations do not demonstrate that the theorem is false under the stated assumptions; they show that the assumptions are more restrictive than one might hope and that a key supporting lemma is unverified. The reader's CONDITIONAL verdict already captures exactly this level of concern, so no change in the verdict is warranted; the appropriate next step is to require the authors to include a proof of Lemma A.2 or to relax and re-prove the smoothness condition, and ideally to add a numerical check at a boundary-degenerate configuration.","tokens_in":13332,"tokens_out":30913,"duration_ms":301728,"concrete_test":"Provide a self-contained proof of Lemma A.2 of Cheng et al. [6] under Assumptions 1–2, or exhibit a counterexample. In parallel, run a simulation in a single-state MDP with reward density uniform on [0,1] (so p(0)=p(1)=1) and discount γ chosen so that for some i,j, θ_m(s,i)−γθ_m(s,j)=0; with step sizes α_t=c(t+1)^{-3/4}, compute c^T(θ̄_T−θ_m)/√(c^T V̂_T c) over many seeds and compare its empirical distribution to V. If the distribution shifts away from V, the boundary-degenerate case breaks the pivotal inference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Assumption 2 (Section 2.5) excludes θ_m(s,i) − γθ_m(s',j) ∈ {0,1} for reachable state pairs. The proof needs this to ensure that the reward CDF F_{s,a} is differentiable at the arguments appearing in h and that the density p(Tπη_m)(s)(θ_m(s,i)) is strictly positive, so that G is uniformly repulsive (Lemma 4.3) and h satisfies the quadratic local linearization (9) of Theorem 4.1. If Assumption 2 fails and p(0) > 0 or p(1) > 0, F_{s,a} has a kink at the reward-support endpoint, h is not differentiable at θ_m, and the FCLT limit in Theorems 3.1–3.2 is not justified; the averaged iterates may have a non-Brownian limit or a different rate. The decisive steps (Lemma 4.1 and Lemma 4.3) both cite Lemma A.2 of the authors' earlier preprint [6] for the positivity and continuity statement, but no proof is reproduced. Thus the central claim of valid online inference is only as secure as this imported, unverified boundary condition.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies statistical inference for quantile temporal difference learning (QTD) in tabular, discounted MDPs with a generative model. It states functional central limit theorems for synchronous and asynchronous QTD (Theorems 3.1 and 3.2), showing that normalized partial sums of the QTD iterates converge weakly to rescaled Brownian motions with explicit covariance matrices G^{-1}\\Gamma_{\\mathrm{syn}}G^{-J} and (D_\\mu G)^{-1}\\Gamma_{\\mathrm{asyn}}(D_\\mu G)^{-J}. Based on these FCLTs, Theorem 3.3 and Corollary 3.1 propose an online, random-scaling-based confidence interval for linear functionals c^T\\theta_m, using a pivotal limit V=B(1)/[\\int_0^1(B(u)-uB(1))^2du]^{1/2} that does not require estimating the asymptotic covariance. The proofs use a general stochastic-approximation FCLT (Theorem 4.1) applied to a local linearization of the QTD fixed-point equation, together with lemmas establishing the unique zero, almost-sure convergence, and uniform repulsivity of the Jacobian G and D_\\mu G.","tokens_in":13539,"tokens_out":4760,"duration_ms":48231,"significance":"If the results are correct, this is a valuable contribution to distributional reinforcement learning: it provides the first distributional FCLT for QTD iterates under a generative model, gives explicit covariance characterizations, and constructs an asymptotically pivotal statistic that can be updated recursively without storing the whole trajectory. The paper is transparent about the main technical assumptions and gives a clear proof architecture via a general SA FCLT. The proposed inference procedure is attractive because it avoids covariance estimation and is fully online. The main caveats are that the theorem statements omit conditions used in the proofs, and the decisive regularity lemma for the quantile Bellman operator is imported from an earlier preprint without proof.","major_comments":[{"comment":"The statements of the main theorems do not include the step-size condition required by the proof. Theorem 4.1 explicitly assumes \\alpha_t=c(t+1)^{-a} with 1/2<a<1, and the proofs of Theorems 3.1 and 3.2 apply Theorem 4.1, but the FCLT statements in Section 3 do not mention any condition on \\alpha_t. Since the FCLT and the resulting inference procedure depend on this schedule, the theorems should state the condition or prove a broader result. In addition, the proof of Theorem 3.2 asserts 'Since \\mu(s)>0 for every s\\in S, D_\\mu is invertible', but the theorem statement does not include this full-support condition on \\mu.","section":"Theorems 3.1, 3.2, and 3.3"},{"comment":"The central regularity step is imported rather than proved. Lemma 4.1 and Lemma 4.3 both rely on Lemma A.2 of Cheng et al. [6] to assert that p(T^\\pi\\eta_m)(s)(\\theta_m(s,i))>0 and that the relevant density is continuous at the quantile locations. This positivity is used to prove that G and D_\\mu G are uniformly repulsive and to obtain the quadratic local linearization in Equation (9), which is the key condition for Theorem 4.1. Because the Brownian limit in Theorems 3.1 and 3.2 is only as secure as this imported lemma, the authors should either reproduce the lemma and its proof in an appendix or state the needed positivity condition as an explicit assumption. The current reliance on an unproved statement about exactly the boundary cases excluded by Assumption 2 is a load-bearing gap.","section":"Section 4, Lemmas 4.1 and 4.3; Appendix B"},{"comment":"The proof of the general FCLT is not fully self-contained at the two decisive points where martingale convergence is used. The convergence of I_4 to G^{-1}\\Gamma^{1/2}B(\\cdot) is delegated to Lemma A.3 of Li et al. [14], and the bound for I_2 uses Lemma 4 of Li et al. [15], but neither lemma is stated in the manuscript and the verification of their conditions is not provided. Since these are exactly the steps that produce the Brownian-motion limit, the authors should either state and prove the required lemmas in the appendix or give a complete argument that the quoted results apply.","section":"Appendix B, proof of Theorem 4.1"}],"minor_comments":[{"comment":"The displayed definition of A_t^i contains an undefined index j in the product; it should be written as a sum over k with a product over l from i to k-1 (or another explicit index range), and the formula should be corrected.","section":"Appendix B, definition of A_t^i"},{"comment":"The definition of P^\\pi(s'|s) is not fully explicit; it should be written as \\sum_{a\\in A}\\pi(a|s)P(s'|s,a) so that the reachability condition is unambiguous.","section":"Section 2.5, Assumption 2"},{"comment":"The recursive updates for \\bar\\theta_t, A_t, b_t, and q_t do not specify initial values; the paper should state \\bar\\theta_0, A_0, b_0, and q_0 explicitly to make the online procedure implementable.","section":"Section 3.2, online updates"},{"comment":"The abstract says 'averaged iterates of QTD converge weakly to a rescaled Brownian motion', but the mathematical object in Theorems 3.1 and 3.2 is the normalized partial-sum process, not the averaged iterate itself; the wording should be aligned with the theorem statements.","section":"Abstract and Section 1.1"}],"recommendation":"major_revision","confidential_remarks":"The paper relies heavily on the authors' own arXiv preprint [6] for the crucial positivity and continuity lemma, and that preprint is cited as a 2026 arXiv paper rather than a published or otherwise independently verified source. The editor may wish to verify that the lemma is correct and will appear in a peer-reviewed form, or require the authors to include a proof in this manuscript. The missing step-size and full-support conditions in the theorem statements are easy to fix but should be handled before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about this paper: it gives the first functional central limit theorems for quantile temporal difference learning and an online random-scaling inference procedure that avoids estimating the asymptotic covariance. The core derivation is sound under the stated technical conditions, and this is a real step beyond Rowland et al.'s convergence results. But the main theorems as written are missing the step-size condition and the full-support condition, and the proof relies on an imported lemma from the authors' own earlier preprint. Treat it as conditional.\n\nWhat's new: Theorems 3.1 and 3.2 establish weak convergence of the averaged QTD iterates to rescaled Brownian motion, with explicit covariance matrices, for both synchronous and asynchronous updates. Theorem 3.3 gives an asymptotically pivotal statistic for linear functionals, computed online with O(d^2) memory. The non-smooth indicator structure is handled via Assumption 2, which ensures differentiability at the fixed point, and the local linearization argument is clean.\n\nWhat the paper does well: It identifies the genuine obstacle (non-smoothness of QTD) and shows how to get around it with a boundary condition and a general FCLT for stochastic approximation. The proof sketch is readable and follows the standard template. The covariance formulas are explicit and the noise conditions check out. No numerical experiments, but for a theory paper that's acceptable.\n\nThe soft spots are real but addressable. First, the theorem statements don't include the step-size condition (α_t = c(t+1)^{-a}, 1/2<a<1) that Theorem 4.1 requires, and the asynchronous theorem doesn't state μ(s)>0 for all s. The proofs use these conditions, so the theorems claim more than proven. Second, the decisive Lemma 4.3 (uniform repulsiveness of G) relies on Lemma A.2 of the authors' earlier preprint [6], which is not reproduced. The stress-test note is right that Assumption 2 is load-bearing: if boundary-degenerate quantile differences occur, the density can hit zero or the CDF can have a kink, and the quadratic local linearization may fail. This doesn't break the central claim under the intended conditions, but it does mean the result is only as secure as that unverified lemma.\n\nI'd send this to a serious referee. The central result is novel and likely correct under the stated assumptions; the gaps are fixable with a fuller proof of the imported lemma and explicit conditions in the theorems. A referee should pressure-test Assumption 2 and the proof of Lemma 4.1, but the paper is worth the time.\n\nBring it to reading group if anyone cares about DRL theory or online inference for stochastic approximation. Recommendation: conditional accept, with the authors required to state all conditions and supply the missing lemma proof.","headline":"First FCLTs and online inference for quantile TD learning, with a sound core but theorem statements that omit needed conditions and a proof that leans on an unverified imported lemma.","tokens_in":14110,"tokens_out":2741,"would_cite":true,"duration_ms":25480,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62L20","60F17","62F12"],"pacs":[],"model":"deepseek-v4-flash","headline":"Averaged quantile TD iterates obey a functional central limit theorem, giving valid online confidence intervals for linear functionals of the return distribution without covariance estimation.","keywords":["quantile temporal difference learning","distributional reinforcement learning","functional central limit theorem","online statistical inference","random scaling","generative model","stochastic approximation","confidence interval"],"falsifier":"Simulate the synchronous QTD algorithm on a tabular environment satisfying Assumptions 1 and 2, pick a fixed linear functional $c$, and compare the empirical distribution of $c^\\top(\\bar\\theta_T-\\theta_m)/\\sqrt{c^\\top V_T c}$ with the pivotal $V$ over many runs; a mismatch beyond Monte Carlo error would refute Theorem 3.3. The sharpest test is to construct a reward density for which $\\theta_m(s,i)-\\gamma\\theta_m(s',j)$ equals 0 or 1 for some reachable pair and watch the studentized statistic leave the pivotal family at that boundary.","tokens_in":13057,"feed_emoji":"🎲","tokens_out":8265,"duration_ms":77252,"temperature":0.7,"pith_summary":"Quantile temporal difference learning (QTD) represents a return distribution by a fixed number of quantiles and updates them through stochastic indicator-based rules. This paper proves that, under a generative model, the averaged iterates of both synchronous and asynchronous QTD converge weakly to rescaled Brownian motions, with explicit covariance matrices. Building on that, it constructs a random-scaling statistic for any linear functional of the learned quantile locations whose limit is a single pivotal distribution, independent of the problem's covariance. The resulting confidence intervals have asymptotically exact coverage and can be updated online with constant memory. If the claim is right, quantile-based distributional reinforcement learning gains valid uncertainty quantification without covariance estimation.","feed_headline":"Quantile TD iterates converge to Brownian motion","feed_subtitle":"A new statistic gives valid confidence intervals for return quantiles on the fly, with no covariance estimation.","key_machinery":"At the center is the quantile-projected distributional Bellman operator $\\Pi_m T^\\pi$ and its unique fixed point $\\eta_m$, represented by quantile locations $\\theta_m$. The proof rewrites QTD as a stochastic approximation $\\theta^{(t)}=\\theta^{(t-1)}-\\alpha_{t-1}(h(\\theta^{(t-1)})+\\epsilon_{t-1})$ whose mean field $h$ has zero $\\theta_m$ and Jacobian $G$ at $\\theta_m$; Assumptions 1 and 2 make $h$ smooth enough there. The inference engine is the random-scaling matrix $V_T=T^{-3}\\sum_{t=1}^{T}(\\sum_{k=1}^{t}\\theta^{(k)}-t\\bar\\theta_T)(\\cdot)^\\top$, which is computed recursively from $\\bar\\theta_T$ and which, under the FCLT, converges to the integrated squared Brownian bridge that defines the pivotal $V$.","core_discovery":"The central discovery is that QTD's non-smooth, indicator-driven update is nonetheless locally linearizable at the fixed point: the Jacobian $G$ of the quantile-projected Bellman mean field is uniformly repulsive and the linearization error is quadratic, so a general functional central limit theorem for stochastic approximation applies. The paper proves $(1/\\sqrt{T})\\sum_{t=0}^{\\lfloor Tu\\rfloor}(\\theta^{(t)}-\\theta_m) \\Rightarrow G^{-1}\\Gamma_{syn}^{1/2}B(\\cdot)$ for synchronous QTD and the analogous limit with $(D_\\mu G)^{-1}\\Gamma_{asyn}^{1/2}$ for asynchronous QTD, where $\\Gamma_{syn}$ and $\\Gamma_{asyn}$ are the conditional covariance matrices of the updates. The consequence is that for any nonzero linear functional $c^\\top \\theta_m$, the studentized statistic $c^\\top(\\bar\\theta_T-\\theta_m)/\\sqrt{c^\\top V_T c}$ converges to the pivotal random variable $V=B(1)/[\\int_0^1(B(u)-uB(1))^2du]^{1/2}$, and the interval $c^\\top\\bar\\theta_T \\pm q_{\\alpha/2}\\sqrt{c^\\top V_T c}$ has asymptotic coverage $1-\\alpha$.","pith_inferences":["The same random-scaling construction would likely apply to other quantile-based distributional updates whose mean fields are locally smooth, but the paper only proves it for tabular QTD under a generative model.","One could extend the scalar functional result to joint confidence regions by studentizing a vector of linear functionals, though only the scalar case is proved.","The explicit boundary condition on $\\theta_m$ differences suggests a diagnostic: in environments whose reward boundaries align with shifted quantiles, the pivotal limit should visibly fail."],"forward_implications":["For any fixed linear functional of the quantile locations, an asymptotically valid confidence interval can be formed without estimating the asymptotic covariance.","Synchronous and asynchronous QTD share the same pivotal limit, so online inference is insensitive to the update schedule.","The statistic uses only recursive quantities, so memory requirements stay constant while the trajectory grows.","The result gives explicit asymptotic covariance matrices for averaged QTD iterates, allowing efficiency comparisons with other distributional temporal difference variants."],"supporting_citations":[{"why":"Introduces QTD and the quantile regression update that the paper analyzes.","marker":"[8]"},{"why":"Proves almost sure convergence of QTD iterates, used as Lemma 4.2 in the proof.","marker":"[23]"},{"why":"Supplies the contraction and non-expansiveness properties of the distributional Bellman and quantile projection operators.","marker":"[3]"},{"why":"Provides the averaging and gain-control lemmas used in the functional central limit theorem proof.","marker":"[20]"},{"why":"Its Lemma A.2 is invoked to ensure positive reward density at the quantile coordinates and the smoothness used with Assumption 2.","marker":"[6]"},{"why":"Supplies the martingale maximal inequality used to control a remainder term in the proof.","marker":"[15]"},{"why":"Provides the Lyapunov lemma used to show summability of linearization errors.","marker":"[14]"}],"fun_headline_variants":["Online inference for quantile TD without covariance estimation","Quantile TD confidence intervals via Brownian motion","Pivotal statistics for distributional RL, computed online","Functional CLT for QTD enables memory-light inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that at the fixed point no Bellman-shifted quantile location difference lands exactly on 0 or 1; this boundary-degeneracy condition is what makes the projected Bellman operator smooth enough for the proof, and without it the Jacobian argument can fail.","fun_headline_variants_meta":{"raw":{"variants":["Online inference for quantile TD without covariance estimation","Quantile TD confidence intervals via Brownian motion","Pivotal statistics for distributional RL, computed online","Functional CLT for QTD enables memory-light inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1335,"prompt_tokens":923,"completion_tokens":412,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":350}},"tokens_in":539,"tokens_out":412,"duration_ms":4740,"temperature":1.0,"reasoning_tokens":350,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:23:58.155126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate the synchronous QTD algorithm on a tabular environment satisfying Assumptions 1 and 2, pick a fixed linear functional $c$, and compare the empirical distribution of $c^\\top(\\bar\\theta_T-\\theta_m)/\\sqrt{c^\\top V_T c}$ with the pivotal $V$ over many runs; a mismatch beyond Monte Carlo error would refute Theorem 3.3. The sharpest test is to construct a reward density for which $\\theta_m(s,i)-\\gamma\\theta_m(s',j)$ equals 0 or 1 for some reachable pair and watch the studentized statistic leave the pivotal family at that boundary.","supporting_citations":[{"cited_title":"Statistical Efficiency and Inference of Quantile Distributional Reinforcement Learning","cited_arxiv_id":"2607.08444","evidence_quote":"Its Lemma A.2 is invoked to ensure positive reward density at the quantile coordinates and the smoothness used with Assumption 2."}],"review_version":1}