Pith. sign in

REVIEW 5 major objections 5 minor 44 references

Coverage Analysis for Digital Cousin Selection -- Improving Multi-Environment Q-Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper derives probabilistic coverage bounds for multi-environment Q-learning and uses them to select synthetic environments, cutting average policy error by up to 65% versus partial ordering and running 95% faster than exhaustive…

desk verdict Empirically promising but theoretically shaky: the ordering rule that powers CCQ is unproved and, in the paper's own settings, reduces to a random tie-break. read the letter →

arxiv 2411.08360 v1 pith:L7F4VW7E submitted 2024-11-13 cs.LG eess.SP

Talha Bozkus , Tara Javidi , Urbashi Mitra This is my paper · ORCID
classification cs.LGeess.SP MSC 68T0590C40
keywords coveragecoefficientmulti-environmentmixedQ-learningn-hopensembleenvironmentselectionMarkovdecisionprocessespolicyerrorrandomnetworkgraphs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a probabilistic coverage analysis of multi-environment mixed Q-learning (MEMQ) can identify which synthetic 'cousin' environments are worth training on, and that the resulting selection rule is near optimal. The authors derive upper and lower bounds on the expectation and variance of the coverage coefficient for individual environments, for the ensemble policy, and for arbitrary action-space sizes. From these bounds they extract a simple ratio test that orders environments by their Q-learning estimation-error variance, and wrap it in an algorithm, coverage-based ensemble Q-learning (CCQ), that picks the best K environments before running nEQL. If the claims hold, MEMQ users can replace heuristic partial ordering and expensive exhaustive search with a cheap, provably motivated preselection step that improves accuracy and complexity.

What carries the argument

The central object is the coverage coefficient $C^{\pi}(s,a)=d^{\pi}(s,a)/v(s,a)$, the ratio of the policy's occupancy measure to the exploration distribution, and its logarithm, which the paper bounds probabilistically. The proof machinery combines a Taylor-series approximation for the expectation and variance of logarithms of positive random variables, the linear action-selection policy $d^{\pi}(s,a)=Q(s,a)/\sum_{a'}Q(s,a')$, a ratio-smoothness assumption $\theta$ on optimal Q-functions, and the distributional error model $Q^{(n)}-Q^{*}\sim D_n(\mu_n,\lambda_n^2/3)$. The load-bearing comparison tool is the ratio test of Proposition 6, $f(\gamma,n,m)=((1-\gamma^n)(1-\gamma^{m-1}))/((1-\gamma^m)(1-\gamma^{n-1}))$, which orders $\lambda_n$ and $\lambda_m$ using only the discount factor and the cost bounds $c_{\min},c_{\max}$, with $\zeta$ interpolating between the two thresholds in the inconclusive region. This test is what turns coverage bounds into a concrete environment-selection algorithm.

What would settle it

Estimate the Q-function error variance for environments $n=1,\ldots,10$ on a small random-graph MDP using many independent runs, and compare the empirical ordering of $\lambda_n$ with the order predicted by Proposition 6 from $f(\gamma,n,m)$ and the cost bounds; for $n=1$ the appendix formula gives zero variance while Assumption 1 requires a positive value, so the comparison must treat $n=1$ separately and would reveal whether the identification holds.

Watch

Extended reading notes

Core claim

The paper's central claim is that the utility of a synthetic environment in nEQL is governed by its estimation-error variance $\lambda_n$, and that these variances can be ordered by evaluating $f(\gamma,n,m)=((1-\gamma^n)(1-\gamma^{m-1}))/((1-\gamma^m)(1-\gamma^{n-1}))$ against thresholds built from $c_{\min}$ and $c_{\max}$. Under Assumptions 1-6, Propositions 1-4 bound the expectation and variance of $\ln C^{\pi}(s,a)$ for individual, ensemble, and K-environment policies, and show the bounds tighten as $\lambda_n$ shrinks, as $\theta$ approaches 1, and as $|A|$ grows. Proposition 5 establishes that the original environment has the smallest $\lambda_n$. Proposition 6 supplies the complete ordering rule, and Algorithm 1 (CCQ) sorts all candidate environments by this rule and runs nEQL on the first K. Numerically, the authors report that CCQ cuts average policy error by up to 65% versus partial ordering with comparable runtime, is 95% faster than exhaustive search, and achieves 60% lower APE than several prior MEMQ and ensemble Q-learning baselines; the simulations also show coverage-based ordering identifies at least 80% of the optimal environment set for larger K.

Load-bearing premise

The load-bearing premise is that the estimation-error variance $\lambda_n$ in the Q-function error model equals the variance computed from the cost-based formula in Appendix E for every environment, including $n=1$; if that identification fails, the ratio test and the sorting step in Algorithm 1 lose their theoretical foundation.

Editorial extensions

If this is right

  • CCQ provides near-optimal environment selection: in the reported simulations it returns the exhaustive-search set for $K\le 6$ and identifies at least 80% of the optimal environments for larger $K$, while partial-ordering errors grow with $K$.
  • The ordering step costs at most $\binom{K_{\mathrm{total}}}{2}$ comparisons and does not depend on $K$, so the dominant cost shifts to the Q-learning itself.
  • Because the ensemble bounds are governed by the worst environment, reducing the largest $\lambda_n$ is the most efficient way to tighten coverage and improve accuracy.
  • The variance-based ordering remains valid as the action space grows, because the bounds generalize to arbitrary $|A|$ and tighten with $|A|$.
  • The simulated coverage coefficient stays inside the predicted interval as learning converges, so the bounds can be used as a convergence diagnostic for nEQL.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the ratio test depends only on $\gamma$, $n$, $m$, and the cost bounds, the environment order could be precomputed once per MDP family and reused across different transition matrices; the paper does not test this transfer.
  • A direct finite-MDP check of the Appendix E variance formula against empirical Q-error variance would settle the identification issue, and for $n=1$ it would need a separate definition since the formula vanishes there.
  • The coverage interval could be turned into a stopping rule: stop training once every state-action pair's empirical $\ln C$ is inside its bound, trading some accuracy for runtime.
  • The same variance-ordering idea may transfer to multi-agent MEMQ settings, since the ordering is agent-agnostic; the paper lists multi-agent MEMQ as future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper develops a probabilistic coverage analysis for multi-environment mixed Q-learning (MEMQ), focusing on n-hop ensemble Q-learning (nEQL). It derives upper and lower bounds on the expectation and variance of the logarithm of coverage coefficients for individual environments (Propositions 1 and 4), for the ensemble policy (Propositions 2 and 3), and generalizes the bounds to an arbitrary number of actions. The authors then propose Proposition 6, a rule to order environments by their estimation-error variance parameters λ_n using only the discount factor γ and cost bounds, and embed this ordering in a new algorithm, CCQ (Algorithm 1), which selects the K best environments out of K_total. Numerical experiments on random graphs with varying structure, sparsity, directionality, and regularity claim that CCQ achieves up to 65% less average policy error (APE) than partial ordering and is 95% faster than exhaustive search, and that the theoretical bounds match simulations.

Significance. If the theoretical results were fully established, the paper would provide a principled and computationally efficient method for environment selection in multi-environment Q-learning, a topic with potential practical value for large-scale reinforcement learning. The empirical study is broad, covering four graph properties and comparing against exhaustive search, partial ordering, and several reinforcement learning baselines. The authors also provide detailed proofs in the appendices and explicitly state assumptions. However, the central theoretical step connecting the coverage bounds to the environment-ordering rule is not rigorously established, and the numerical verification is partly circular because the λ_n parameters are estimated from the same simulations used to validate the bounds. These issues currently undermine the claimed theoretical foundation of CCQ.

major comments (5)
  1. [Appendix E, Eqs. (58)-(59)] The identification of the Assumption 1 variance parameter λ_n^2/3 with the expression (c_{π*}^{(n)}(s))^2 (γ(1-γ^n)/(1-γ^{n-1}))^2 is not proved. Assumption 1 concerns the variance of the online Q-learning error Q_t^{(n)}(s,a) - Q*(s,a), whereas Eq. (56) is a deterministic bound from [21] on the difference between the optimal Q-value and the n-hop Q-function under the optimal policy. Assumption 6 (uniformity of D_n) gives the variance of a uniform distribution on an interval, but it does not establish that the random variable in Eq. (58) is the same as the error in Eq. (7). Without this identification, the ratio test in Proposition 6 has no demonstrated foundation.
  2. [Proposition 6, Eq. (18)] The function f(γ,n,m) = (1-γ^n)(1-γ^{m-1})/((1-γ^m)(1-γ^{n-1})) is undefined for n=1 or m=1 because the denominator contains the factor 1-γ^{n-1} (respectively 1-γ^{m-1}), which is zero when the index equals 1. Since Algorithm 1 line 2 sorts λ_n for n=1,...,K_total, the rule cannot compare the original environment (n=1) with any other environment. The sorting step is therefore incomplete as stated.
  3. [Proposition 6, Eqs. (61)-(62)] For the numerical settings reported in Section IV-B (γ=0.9, cmin=0.5, cmax=1), every pair of environments with n,m>1 and n≠m gives f(γ,n,m) strictly inside the inconclusive interval (cmin/cmax, cmax/cmin) = (0.5, 2). Specifically, with a_n=(1-0.9^n)/(1-0.9^{n-1}), f=a_n/a_m, and a_n ranges from about 1.9 down to 1, so f ranges from about 0.526 to 1.9. Thus Eq. (61) never yields a definite comparison under the paper's own parameter settings, and the ordering produced in Algorithm 1 is actually determined by the random ζ tie-breaker. The claim that Proposition 6 provides a complete ordering is therefore unsupported for the configurations evaluated.
  4. [Proposition 3, Eqs. (12)-(13)] The function f(λ,u) appearing in the bounds of Proposition 3 is never specified. The text states only that 'f is some function of λ and u' and refers to properties derived in [21]. As written, Proposition 3 does not state a concrete, checkable bound; it cannot be verified or used without an explicit definition of f.
  5. [Section IV-E and Tables I-II] The numerical validation of the bounds is partly circular: the values λ_1,...,λ_5 used to plot the theoretical bounds are estimated as sample variances of the Q-function errors from the same simulations that produce the simulated coverage coefficients shown in Fig. 2. Agreement between the bounds and the simulated curves is therefore a goodness-of-fit check rather than an independent verification of the theory. Additionally, although each experiment is averaged over 100 runs, Tables I and II report only point estimates of APE and runtime without error bars or confidence intervals, making it difficult to assess whether the claimed 65% improvement over partial ordering is statistically significant.
minor comments (5)
  1. [Section II-C] The notation π_it for the estimated ensemble policy is used in Propositions 2 and 3 but is not defined in the system model; it would help to define it explicitly before first use.
  2. [Fig. 1a] The caption refers to 'correct detection rates' but the text does not define what constitutes a correct detection; please clarify the metric and how it is computed.
  3. [Table II] The statement that coverage-based ordering maintains 'similar runtime complexity' to partial ordering is only true for the S-S-U-R configuration; for S-D-D-R and U-S-U-I the coverage-based runtime is 40% and 20% higher, respectively, so the claim should be qualified.
  4. [Appendix A, Eq. (20)] The second-order Taylor approximations for E[ln X] and V[ln X] are stated to be accurate when μ/σ > 1.5, but the paper does not verify this condition for the numerical experiments where the bounds are plotted; a brief check of this condition would strengthen the validation.
  5. [Section III-B] In the displayed equations (9) and (15), the lower-bound expressions are written with 'maxt ... u' notation; this appears to be a typesetting artifact, but it should be presented as max{..., 0} for consistency with standard notation.

Circularity Check

2 steps flagged · score 6.0 of 10

The environment-ordering rule in Proposition 6 is the λ_n identification of Eq. (59) restated as a ratio test, and the bound validation fits λ_n from the same simulations; headline APE gains remain external, giving partial circularity.

  1. self definitional [Appendix D Eq. (59); Appendix E Eqs. (60)-(62); Algorithm 1 line 2]
    "Using the uniformity of D_n and the variance of the uniform distribution, we obtain: V[Q*(s)−Q^(n)_π*(s)] = (c^(n)_π*(s))^2 (γ/(1−γ)·(1−γ^n)/(1−γ^{n−1}))^2 = λ_n^2/3. (59)"

    Assumption 1 (Eq. 7) already defines λ_n^2/3 as the variance of the Q-function error distribution D_n. Eq. (59) then asserts this same variance equals a cost-based bound expression imported from [21] (Eq. 56) plus a uniformity ansatz. Proposition 6's ratio test f(γ,n,m) is exactly the ratio of those bound expressions, so the 'prediction' λ_n < λ_m iff f > threshold is an algebraic restatement of the identification at Eq. (59), not a consequence of the coverage bounds. Algorithm 1 line 2 then sorts λ_n using this restated identification; the ordering is thus the input assumption re-exported as a theorem. The empirical APE comparisons against exhaustive search are external, so this step is partial, not total, circularity.

  2. fitted input called prediction [Section IV-E, 'Numerical validation of bounds']
    "The parameters λn are estimated using the sample variance of the Q-function errors of individual environments over time. We also numerically verify that Assumption 2 is reasonable as µ1 = 0.02, µ2 = 0.08, µ3 = 0.04, µ4 = 0.08 and µ5 = 0.10."

    The section is titled 'Numerical validation of bounds' and the abstract and conclusions claim the assumptions and bounds are numerically verified. But the bounds in Propositions 1-4 are parameterized by λ_n (e.g., Eqs. (8)-(9)); taking λ_n from the sample variance of Q-function errors of the same runs that generate the simulated ln C makes the interval bounds track the fitted variance. The observed agreement therefore reflects the fitted input rather than an independent prediction. This self-referential validation does not affect the headline APE results, which are comparisons to partial ordering and exhaustive search.

full rationale

The strongest circularity is in Proposition 6 and Algorithm 1: the variance parameter λ_n is introduced as a free distributional parameter in Assumption 1, then fixed by Eq. (59) to a bound expression from the authors' prior work [21], and the proposed environment ordering is simply the ratio of that same expression. Thus the theoretical ordering reduces by construction to the imported identification, rather than following from the new coverage-coefficient bounds. A secondary issue is that the numerical 'verification' of the bounds estimates λ_n from the same simulations whose coverage coefficients are being tested, making that validation partially self-referential. However, the headline empirical claims (65% less APE than partial ordering, 95% faster than exhaustive search, and near-optimal environment selection in Tables II-III) are benchmarked against exhaustive search and other algorithms, which provides independent support for the practical method. For that reason the paper is not fully circular: the central empirical result has content beyond the assumption, and the score is 6 rather than 8.

Assumptions & free parameters 4 free parameters · 8 assumptions · 0 invented entities

The central claim rests on a posited Q-error distribution, the authors' own prior MEMQ results, and several hand-chosen or estimated parameters. The numerical verification reuses the same estimated λ_n values that parameterize the bounds, so part of the validation is not independent. No new physical or formal entity is introduced.

free parameters (4)
  • λ_n (per-environment Q-error variance parameter) = Estimated in experiments: λ1=0.24, λ2=0.98, λ3=1.24, λ4=1.44, λ5=1.30 (Section IV-E)
    Central to the theoretical bounds and to the environment-ordering rule. In experiments it is estimated from sample variances of the same Q-function errors, not independently predicted.
  • θ (Q-function ratio bound) = θ=12 in Section IV-E
    Assumption 4 bounds the ratio of optimal Q-functions; all printed bounds depend on θ, and in the numerical validation it is computed from Q* rather than independently selected.
  • ζ (decision-rule weight) = ζ ~ uniform(0,1) in experiments
    Proposition 6 uses ζ to resolve inconclusive cost-ratio regions. The paper gives no principled method to set it, and the random choice introduces stochasticity into the ordering.
  • u (nEQL update ratio) = u=0.5 in Section IV-E; u_t=1−e^{−t/c4} with c4 fine-tuned elsewhere
    Proposition 2 and Proposition 3 bounds depend on u, and the algorithm inherits u as a hyperparameter tuned as in [21].
assumptions (8)
  • domain assumption Assumption 1: Q_t^(n)(s,a) − Q*(s,a) follows D_n(μ_n, λ_n²/3) (Eq. (7)).
    All bounds and ordering results depend on this posited distribution and its variance parameter; it is not derived from Q-learning dynamics.
  • domain assumption Assumption 2: all D_n are zero-mean (μ_n=0).
    The authors state this may not always hold in practice and is used only to facilitate analysis, weakening its status as a general result.
  • domain assumption Assumption 3: linear action-selection d_t^π(s,a)=Q_t(s,a)/Σ_i Q_t(s,a_i) (Eq. (16)).
    A design choice used in every expectation and variance derivation; the paper claims softmax gives similar results without proof in the main text.
  • domain assumption Assumption 4: 1/θ ≤ Q*(s,a_k1)/Q*(s,a_k2) ≤ θ (Eq. (17)).
    This ratio bound is needed to convert ϵ_k expressions into the printed bounds; θ is a hand-chosen or computed parameter.
  • standard math Assumption 5: access to a state-action exploration distribution v.
    Common in RL coverage analysis; v appears in the denominator of the coverage coefficient and in all bound expressions.
  • ad hoc to paper Assumption 6: D_n is uniform for all n.
    Used in Appendix E to identify λ_n with a cost-based variance formula. The authors say other distributions give similar results but do not prove it here.
  • ad hoc to paper Second-order Taylor approximations for E[ln X] and V[ln X] (Eq. (20)), valid only when μ/σ > 1.5.
    The paper states the accuracy condition but never verifies it for all environments, state-action pairs, or time steps used in the bounds.
  • domain assumption Prior convergence properties from [21], including Eq. (40)-(41) and Proposition 3 of [21].
    Propositions 2, 5 and 6 rely on these self-cited results, which are not independently re-derived or machine-checked here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Coverage Analysis for Digital Cousin Selection -- Improving Multi-Environment Q-Learning." pith.science (2026). https://pith.science/paper/L7F4VW7E

@misc{pith2026241108360,
  author       = {Pith},
  title        = {Pith review of: Coverage Analysis for Digital Cousin Selection -- Improving Multi-Environment Q-Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L7F4VW7E}},
  note         = {Machine review of arXiv:2411.08360}
}
read the original abstract

Q-learning is widely employed for optimizing various large-dimensional networks with unknown system dynamics. Recent advancements include multi-environment mixed Q-learning (MEMQ) algorithms, which utilize multiple independent Q-learning algorithms across multiple, structurally related but distinct environments and outperform several state-of-the-art Q-learning algorithms in terms of accuracy, complexity, and robustness. We herein conduct a comprehensive probabilistic coverage analysis to ensure optimal data coverage conditions for MEMQ algorithms. First, we derive upper and lower bounds on the expectation and variance of different coverage coefficients (CC) for MEMQ algorithms. Leveraging these bounds, we develop a simple way of comparing the utilities of multiple environments in MEMQ algorithms. This approach appears to be near optimal versus our previously proposed partial ordering approach. We also present a novel CC-based MEMQ algorithm to improve the accuracy and complexity of existing MEMQ algorithms. Numerical experiments are conducted using random network graphs with four different graph properties. Our algorithm can reduce the average policy error (APE) by 65% compared to partial ordering and is 95% faster than the exhaustive search. It also achieves 60% less APE than several state-of-the-art reinforcement learning and prior MEMQ algorithms. Additionally, we numerically verify the theoretical results and show their scalability with the action-space size.

Figures

Figures reproduced from arXiv: 2411.08360 by the authors.

Figure 1
Figure 1. Numerical results. K. Table III shows the order of environments each method outputs as a function of K. The exhaustive search always finds the optimal set of environments (at the expense of very high runtime complexity). Coverage-based ordering outputs the optimal set of environments for all K ď 6, with only one mistake for larger K, whereas partial ordering makes errors for K ě 5, with errors increasing as K increa… view at source ↗
Figure 2
Figure 2. Simulation of theoretical results size using the S-S-U-R model. The state-action space is constructed by increasing the state-space size and action￾space size comparably (for example, p|S|, |A|q “ p1000, 2q Ñ p2000, 3q). We employ four algorithms: (i) n-hop Ensemble Q-Learning (nEQL) [21], (ii) Ensemble-Synthetic-Q-Learning (ESQL), (iii) Double Q-learning (DQ) and (iv) MaxMin￾Q-Learning (MMQ). The first two are MEMQ… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 29 canonical work pages

  1. [21]

    Multi-timescale ensemble q-learning for markov decision process policy optimization

    Talha Bozkus and Urbashi Mitra. Multi-timescale ensemble q-learning for markov decision process policy optimization. IEEE Transactions on Signal Processing, 72:1427–1442, 2024

    work page 2024

  2. [1]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  3. [2]

    Reliable adaptive recoding for batched network coding with burst-noise channels

    Jie Wang, Talha Bozkus, Yao Xie, and Urbashi Mitra. Reliable adaptive recoding for batched network coding with burst-noise channels. In 2023 57th Asilomar Conference on Signals, Systems, and Computers , pages 220–224. IEEE, 2023

    work page 2023

  4. [3]

    Markov decision processes with applications in wireless sensor networks: A survey

    Mohammad Abu Alsheikh, Dinh Thai Hoang, Dusit Niyato, Hwee-Pink Tan, and Shaowei Lin. Markov decision processes with applications in wireless sensor networks: A survey. IEEE Communications Surveys & Tutorials, 17(3):1239–1267, 2015. 13

    work page 2015

  5. [4]

    Q-learning algorithms: A comprehensive classification and appli- cations

    Beakcheol Jang, Myeonghwi Kim, Gaspard Harerimana, and Jong Wook Kim. Q-learning algorithms: A comprehensive classification and appli- cations. IEEE Access, 7:133653–133667, 2019

    work page 2019

  6. [5]

    Q-learning: Theory and applications

    Jesse Clifton and Eric Laber. Q-learning: Theory and applications. Annual Review of Statistics and Its Application , 7(1):279–301, 2020

    work page 2020

  7. [6]

    Double Q-learning

    Hado Hasselt. Double Q-learning. Advances in neural information processing systems, 23, 2010

    work page 2010

  8. [7]

    Ensemble bootstrapping for Q-learning

    Oren Peer, Chen Tessler, Nadav Merlis, and Ron Meir. Ensemble bootstrapping for Q-learning. In International Conference on Machine Learning, pages 8454–8463. PMLR, 2021

    work page 2021

Show all 44 references
  1. [8]

    Maxmin Q-learning: Controlling the estimation bias of Q-learning

    Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin Q-learning: Controlling the estimation bias of Q-learning. CoRR, abs/2002.06487, 2020

    2002 arXiv

  2. [9]

    Speedy Q-learning

    Mohammad Ghavamzadeh, Hilbert Kappen, Mohammad Azar, and Rémi Munos. Speedy Q-learning. Advances in neural information processing systems, 24, 2011

    2011

  3. [10]

    Pac model-free reinforcement learning

    Alexander L Strehl, Lihong Li, Eric Wiewiora, John Langford, and Michael L Littman. Pac model-free reinforcement learning. In Proceed- ings of the 23rd international conference on Machine learning , pages 881–888, 2006

    2006

  4. [11]

    Neural fitted Q iteration–first experiences with a data efficient neural reinforcement learning method

    Martin Riedmiller. Neural fitted Q iteration–first experiences with a data efficient neural reinforcement learning method. In Machine Learning: ECML 2005: 16th European Conference on Machine Learning, Porto, Portugal, October 3-7, 2005. Proceedings 16 , pages 317–328. Springer, 2005

    2005

  5. [12]

    Deep exploration via bootstrapped dqn

    Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016

    2016

  6. [13]

    Riedmiller

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013

    2013 arXiv

  7. [14]

    Q-learning with linear function approximation

    Francisco S Melo and M Isabel Ribeiro. Q-learning with linear function approximation. In Learning Theory: 20th Annual Conference on Learning Theory, COLT 2007, San Diego, CA, USA; June 13-15,

    2007

  8. [15]

    Sample complexity of reinforcement learning using linearly combined model ensembles

    Aditya Modi, Nan Jiang, Ambuj Tewari, and Satinder Singh. Sample complexity of reinforcement learning using linearly combined model ensembles. In International Conference on Artificial Intelligence and Statistics, pages 2010–2020. PMLR, 2020

    2010

  9. [16]

    Model-ensemble trust-region policy optimization

    Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802.10592, 2018

    2018 arXiv

  10. [17]

    Deep reinforcement learning in a handful of trials using prob- abilistic dynamics models

    Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using prob- abilistic dynamics models. Advances in neural information processing systems, 31, 2018

    2018

  11. [18]

    Asynchronous methods for deep reinforcement learning

    V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928–1937. PMLR, 2016

    1928

  12. [19]

    Ensemble link learning for large state space multiple access communications

    Talha Bozkus and Urbashi Mitra. Ensemble link learning for large state space multiple access communications. In 2022 30th European Signal Processing Conference (EUSIPCO), pages 747–751, 2022

    2022

  13. [20]

    Ensemble graph Q-learning for large scale networks

    Talha Bozkus and Urbashi Mitra. Ensemble graph Q-learning for large scale networks. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 2023

    2023

  14. [22]

    Leveraging digital cousins for ensemble q-learning in large-scale wireless networks

    Talha Bozkus and Urbashi Mitra. Leveraging digital cousins for ensemble q-learning in large-scale wireless networks. IEEE Transactions on Signal Processing , 72:1114–1129, 2024

    2024

  15. [23]

    A novel ensemble q-learning algorithm for policy optimization in large-scale networks

    Talha Bozkus and Urbashi Mitra. A novel ensemble q-learning algorithm for policy optimization in large-scale networks. In 2023 57th Asilomar Conference on Signals, Systems, and Computers , pages 1381–1386. IEEE, 2023

    2023

  16. [24]

    Link analysis for solving multiple- access mdps with large state spaces

    Talha Bozkus and Urbashi Mitra. Link analysis for solving multiple- access mdps with large state spaces. IEEE Transactions on Signal Processing, 71:947–962, 2023

    2023

  17. [25]

    Is q-learning provably efficient? Advances in neural information processing systems, 31, 2018

    Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? Advances in neural information processing systems, 31, 2018

    2018

  18. [26]

    Online robust reinforcement learning with model uncertainty

    Yue Wang and Shaofeng Zou. Online robust reinforcement learning with model uncertainty. Advances in Neural Information Processing Systems, 34:7193–7206, 2021

    2021

  19. [27]

    Conser- vative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conser- vative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:1179–1191, 2020

    2020

  20. [28]

    An optimistic perspective on offline reinforcement learning

    Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning , pages 104–114. PMLR, 2020

    2020

  21. [29]

    Leveraging offline data in online reinforcement learning

    Andrew Wagenmaker and Aldo Pacchiano. Leveraging offline data in online reinforcement learning. In International Conference on Machine Learning, pages 35300–35338. PMLR, 2023

    2023

  22. [30]

    Accelerating online reinforcement learning with offline datasets

    Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. CoRR, abs/2006.09359, 2020

    2006 arXiv

  23. [31]

    Policy finetuning: Bridging sample-efficient offline and online reinforce- ment learning

    Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforce- ment learning. Advances in neural information processing systems , 34:27395–27407, 2021

    2021

  24. [32]

    Hybrid rl: Using both offline and online data can make rl efficient

    Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.06718 , 2022

    2022 arXiv

  25. [33]

    Com- paring exploration strategies for q-learning in random stochastic mazes

    Arryon D Tijsma, Madalina M Drugan, and Marco A Wiering. Com- paring exploration strategies for q-learning in random stochastic mazes. In 2016 IEEE Symposium Series on Computational Intelligence (SSCI) , pages 1–8. IEEE, 2016

    2016

  26. [34]

    The role of coverage in online reinforcement learning

    Tengyang Xie, Dylan J Foster, Yu Bai, Nan Jiang, and Sham M Kakade. The role of coverage in online reinforcement learning. arXiv preprint arXiv:2210.04157, 2022

    2022 arXiv

  27. [35]

    Pessimistic model-based of- fline reinforcement learning under partial coverage

    Masatoshi Uehara and Wen Sun. Pessimistic model-based of- fline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226, 2021

    2021 arXiv

  28. [36]

    What can online reinforcement learning with function approximation benefit from general coverage conditions? In International Conference on Machine Learning, pages 22063–22091

    Fanghui Liu, Luca Viano, and V olkan Cevher. What can online reinforcement learning with function approximation benefit from general coverage conditions? In International Conference on Machine Learning, pages 22063–22091. PMLR, 2023

    2023

  29. [37]

    When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning

    Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, Xianyuan Zhan, et al. When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning. Advances in Neural Infor- mation Processing Systems , 35:36599–36612, 2022

    2022

  30. [38]

    Coverage analysis of multi- environment q-learning algorithms for wireless network optimization

    Talha Bozkus and Urbashi Mitra. Coverage analysis of multi- environment q-learning algorithms for wireless network optimization. In 2024 IEEE 25th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), pages 376–380, 2024

    2024

  31. [39]

    Convergence of Q-learning: A simple proof

    Francisco S Melo. Convergence of Q-learning: A simple proof. Institute Of Systems and Robotics, Tech. Rep , pages 1–4, 2001

    2001

  32. [40]

    Issues in using function ap- proximation for reinforcement learning

    Sebastian Thrun and Anton Schwartz. Issues in using function ap- proximation for reinforcement learning. In Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum, volume 6, pages 1–9, 1993

    1993

  33. [41]

    Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double Q-learning: Learning fast without a model. CoRR, abs/2101.05982, 2021

    2021 arXiv

  34. [42]

    The virtues of laziness in model-based rl: A unified objective and algorithms

    Anirudh Vemula, Yuda Song, Aarti Singh, Drew Bagnell, and Sanjiban Choudhury. The virtues of laziness in model-based rl: A unified objective and algorithms. In International Conference on Machine Learning , pages 34978–35005. PMLR, 2023

    2023

  35. [43]

    A multi-agent multi-environment mixed q-learning for partially decentralized wireless network optimiza- tion

    Talha Bozkus and Urbashi Mitra. A multi-agent multi-environment mixed q-learning for partially decentralized wireless network optimiza- tion. arXiv preprint arXiv:2409.16450 , 2024

    2024 arXiv

  36. [2007]

    Springer, 2007

    Proceedings 20 , pages 308–322. Springer, 2007

    2007

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.