Pith. sign in

REVIEW 2 major objections 5 minor 19 references

StaQ it! Growing neural networks for Policy Mirror Descent

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Finite-memory Policy Mirror Descent can retain the exact convergence guarantee of full PMD, provided it keeps enough Q-functions and re-weights the newest one.

desk verdict Solid theory for exact finite-memory PMD, but the deep RL version ships with an unmeasured Q-evaluation error that the paper's own bound amplifies enormously; referee-worthy with revisions. read the letter →

arxiv 2506.13862 v1 pith:YNO4DL6U submitted 2025-06-16 cs.LG cs.AI

Alena Shilova , Alex Davey , Brahim Driss , Riad Akrour This is my paper
classification cs.LGcs.AI
keywords PolicyMirrorDescententropyregularizationfinite-memoryupdatestackedneuralnetworksdeepreinforcementlearningQ-functionaveragingstability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's goal is to make Policy Mirror Descent (PMD) implementable in deep reinforcement learning without losing its convergence guarantee. In PMD, the entropy-regularized policy update is a weighted geometric sum of all past Q-functions, which is intractable for neural networks; the authors propose keeping only the last $M$ Q-functions and adding a weight-correction term. Their main theorem states that if $M$ is finite but large enough, this finite-memory scheme converges to the optimal entropy-regularized Q-function with no residual error, unlike a plain truncated sum. The resulting algorithm, StaQ, stores $M$ frozen Q-network snapshots, deletes the oldest when the stack overflows, and performs the policy update by a batched forward pass that is optimization-free. Experiments show StaQ is competitive with established deep RL baselines and exhibits markedly less performance oscillation, so the paper positions it as a stable testbed for separating policy-update error from policy-evaluation error.

What carries the argument

The load-bearing object is the logits update for finite-memory entropy-regularized Policy Mirror Descent. Exact EPMD sets $\xi^{k+1} = \beta \xi^k + \alpha Q^k_\tau$, which expands into an infinite geometric sum of Q-functions; StaQ replaces it with the weight-corrected finite-memory update $\xi^{k+1} = \beta \xi^k + \alpha Q^k_\tau + \frac{\alpha \beta^M}{1-\beta^M}(Q^k_\tau - Q^{k-M}_\tau)$, keeping only the last $M$ Q-functions. The correction term deletes the oldest Q-function while re-weighting the newest one so the retained weights sum to one. The argument is carried by the approximate policy improvement theorem, which bounds the loss in the Q-function at each step by a quantity proportional to the distance between the true previous policy and the policy actually used in the KL regularization; when the stack is long enough, that distance shrinks to zero as the Q-functions converge. The implementation packages this as a stacked neural network: $M$ frozen weight snapshots evaluated in one batched GPU forward pass, a policy update that is a single push onto the stack, and a FIFO deletion when the stack exceeds $M$.

What would settle it

Run the weight-corrected finite-memory EPMD update in a small tabular MDP with exact policy evaluation, choose $\gamma = 0.99$ and $\beta = 0.95$, and compare $M = 264$ with $M = 265$. Theorem 4.4 predicts $\|Q^*_\tau - Q^k_\tau\|_\infty$ decays to zero at $M = 265$ but plateaus at a positive value at $M = 264$.

Watch

Extended reading notes

Core claim

The central claim is Theorem 4.4: for the weight-corrected finite-memory EPMD update $\xi^{k+1} = \beta \xi^k + \alpha Q^k_\tau + \frac{\alpha \beta^M}{1-\beta^M}(Q^k_\tau - Q^{k-M}_\tau)$, if $M > \frac{\log\left(\frac{(1-\gamma)^2(1-\beta)}{\gamma^2(3+\beta)+1-\beta}\right)}{\log \beta}$, then $\|Q^*_\tau - Q^k_\tau\|_\infty$ is bounded by $(d_1 + d_2 d_3^{-1})^k \max\{\|Q^*_\tau\|_\infty/\gamma,\, \|Q^*_\tau\|_\infty + \|Q^0_\tau\|_\infty\}$, so the iterate converges to $Q^*_\tau$ without residual error. In plain terms, deleting the oldest Q-function is harmless once the stack is long enough, provided the newest Q-function is slightly overweighted so the geometric weights still sum to one. That re-weighting is the difference between a truncated sum, which leaves a permanent approximation error, and a convergent finite-memory algorithm.

Load-bearing premise

The convergence theory assumes the Q-function estimate used in the update is exact or has a bounded error $\epsilon_{\mathrm{eval}}$; the deep StaQ version learns off-policy with a replay buffer and never measures this error, so if Q-learning turns unstable the theorem's bound does not apply.

Editorial extensions

If this is right

  • A practitioner can run an exact closed-form PMD policy update with only a few hundred stored Q-networks: with $\gamma = 0.99$ and $\beta = 0.95$, the theorem requires $M = 265$.
  • Increasing memory beyond the threshold buys little asymptotically, since the finite-memory convergence rate approaches the exact EPMD rate $\beta + \gamma(1-\beta)$ as $M \to \infty$.
  • Training wall-clock time is nearly independent of $M$ on GPU, because all $M$ Q-networks are evaluated in parallel and the policy update is optimization-free.
  • Because the policy update introduces no approximation error, any remaining instability in StaQ can be attributed to policy evaluation, which makes the algorithm a clean testbed for improved Q-learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'keep the last $M$, re-weight the newest' correction is a parameter-isolation trick from continual learning; the same re-normalized sliding-window idea could be transferred to other online learners that maintain geometric averages over stale function snapshots.
  • The authors sketch a continuous-action extension along SAC lines, replacing the soft target $\exp(Q/\tau)$ with $\exp(\xi_k)$; a direct test would be whether averaging over many past Q-functions stabilizes the actor target without sacrificing the closed-form update.
  • Theorem 4.4's memory threshold should be read as pessimistic; the paper's own results use $M=300$, and measuring the actual evaluation error $\epsilon_{\mathrm{eval}}$ in the deep experiments would connect the practical stability to the error term in Theorem A.6.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes memory-limited variants of entropy-regularized Policy Mirror Descent (EPMD). After observing that the exact EPMD policy update requires summing all past Q-functions, the authors introduce two finite-memory updates: a vanilla truncation (Eq. 6), which retains a residual error of order β^M, and a weight-corrected update (Eq. 10), which renormalizes the weights so that, for M above a closed-form threshold, the iterates converge to the optimal entropy-regularized Q-function Q*_τ without residual error (Thm 4.4). The resulting algorithm, StaQ, stores the last M neural-network Q-functions as a stack of frozen weight snapshots, precomputes the logits ξ^k, and uses FQI with an ϵ-softmax behavior policy for policy evaluation. Experiments on discrete-action MuJoCo, classic control, and MinAtar environments compare StaQ with DQN, M-DQN, PQN, PPO, and TRPO, reporting competitive performance and reduced performance oscillation.

Significance. The exact finite-memory result is the main contribution. Theorem 4.4 gives an explicit, parameter-free sufficient condition on M, and the appendix develops the proofs in detail; if correct, it shows that a few hundred stored Q-functions suffice for convergence in the exact policy-evaluation setting. The stacked-network implementation is also a pragmatic engineering contribution, and the stability evidence is suggestive. However, the paper's broader claim that StaQ 'enjoys strong theoretical guarantees' is not fully supported: the deep RL version relies on approximate Q-functions whose error is never measured, and the experimental entropy-annealing schedule can push β into a regime where M=300 no longer satisfies the theorem's sufficient condition. The paper is therefore stronger as a theoretical contribution plus heuristic deep RL demonstration than as a fully certified deep RL algorithm.

major comments (2)
  1. [Sec. 4.2 (Thm 4.4) vs. Sec. 6 / Table 3] The experimental configuration may not satisfy the sufficient M threshold of Theorem 4.4. The theorem requires M > log(((1-γ)^2(1-β))/(γ^2(3+β)+1-β)) / log β. The paper illustrates the threshold with γ=0.99, β=0.95, giving M≈265, and then states that M=300 was used in all experiments. However, Table 3 shows that the scaled entropy coefficient τ̄ is annealed from 2.0 to 0.4, with η=10 for MuJoCo (and η=20 for classic control). Since β=η/(η+τ) and τ=τ̄/log|A|, β increases as τ decreases; for typical MuJoCo action-space sizes, the final β is roughly 0.98-0.99. At γ=0.99 and β=0.98 the threshold is approximately M≈717, and at β=0.99 it is approximately M≈1510. Thus M=300 may be below the sufficient threshold for the late-training β. The comment after Thm 4.4 that larger M was not observed to help empirically is not a formal justification, since the theorem gives only a sufficient condition. The authors should report the actual β range during training and either set M to satisfy the condition for the maximal β or explicitly state that the deep RL results are heuristic and not covered by Thm 4.4.
  2. [App. A (Thm A.6) and Sec. 5-7] The convergence guarantee for StaQ as run is not established because the policy evaluation error ε_eval is never measured. The deep RL implementation learns Q^k_τ by FQI from a replay buffer generated by an ϵ-softmax behavior policy (Sec. 5 and Eq. 176), and Assumption A.1 only posits a bounded error ε_eval without quantifying it. Theorem A.6 gives a residual term (1+γ^2)ε_eval / ((1-γ)(1-d1-d2)); at γ=0.99, β=0.95, M=300, d1+d2≈0.9996, so the multiplier is on the order of 5×10^5, making the bound vacuous unless ε_eval is extremely small. The paper's own App. B.4 (Fig. 10) documents large Q-value fluctuations between iterations, and Sec. 7 states that 'policy evaluation errors remain significant.' Therefore the no-residual-error guarantee of Thm 4.4 applies only to the exact policy-evaluation setting, and the claim that StaQ 'enjoys strong theoretical guarantees' is not justified for the algorithm as actually run. The authors should either measure or control ε_eval (or a value-error proxy) and verify the margin d1+d2<1, or explicitly separate the exact finite-memory guarantee from the heuristic deep RL evaluation.
minor comments (5)
  1. [Sec. 1] The phrase 'which we adress in this paper' contains a typo; it should read 'address'.
  2. [App. B.3 / B.4] Several passages, especially around Fig. 9 and Fig. 12, contain long garbled '/uni000000...' sequences that appear to result from a character-encoding failure; these should be repaired before publication.
  3. [Sec. 4.2] The statement that 'with higher values of β, we did not observe in practice better performance when using as large M as suggested by Thm 4.4' is informal and could be misread as contradicting the theorem; reporting the actual β values used and the associated threshold would be clearer.
  4. [Table 6] The repeated 'LayerNorm LayerNorm LayerNorm' and 'None None None' entries appear to be table-formatting artifacts; the table should be checked so that each column is aligned with its environment group.
  5. [General] No code availability statement is provided; given the complexity of the stacked-network implementation, releasing code would substantially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the finite-memory convergence proof is self-contained; self-citations are not load-bearing.

full rationale

No circular step is present. The central claim, Theorem 4.4, is derived from explicit stated assumptions (finite MDP, entropy-regularized EPMD update, exact policy evaluation in the main text, and a bounded ϵ_eval term in the appendix). The weight-corrected update in Eq. 10 is analyzed through the bounding sequence of Lemma 4.3, and the memory threshold M > log(((1-γ)^2(1-β))/(γ^2(3+β)+1-β)) / log β is an explicit sufficient condition computed from β and γ, not fitted to observed performance. The practical choice M = 300 is justified by the theorem's threshold of 265 and by computational considerations, and experimental returns are not presented as predictions of the theory. Citations to the authors' own prior work (e.g., Della Vecchia et al. 2022 on cascading networks) appear only as related work and are not load-bearing for the convergence argument; no uniqueness theorem or ansatz is imported from those citations. The appendix's Theorem A.6 explicitly carries the policy-evaluation error ϵ_eval, and the fact that ϵ_eval is not measured in the deep RL experiments is a limitation or assumption gap, not circularity, since ϵ_eval is not a fitted parameter renamed as a prediction. Accordingly, the score remains 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The theoretical convergence claims rest on standard MDP and entropy-regularization properties plus a bounded policy evaluation error. The practical algorithm adds hand-chosen hyperparameters (M, η, τ schedule, ϵ). No new physical or theoretical entities are postulated.

free parameters (5)
  • M (memory size) = 300 in all experiments
    The empirical algorithm sets M=300 across tasks, but Theorem 4.4 gives a sufficient lower bound M > log((1-γ)^2(1-β)/(γ^2(3+β)+1-β))/log β, which is not checked for the annealed entropy weight schedule.
  • KL weight η = 10 (MuJoCo), 20 (Classic, MinAtar)
    The decay factor β = η/(η+τ) enters the convergence rate and the M threshold; η is a hand-chosen hyperparameter.
  • Scaled entropy weight schedule = Anneal from 2.0 to 0.4 over 0.5M or 1M steps
    The entropy weight τ = scaled_weight/log|A| changes β over training; the theory assumes a fixed β, so the annealed β is not covered by the sufficient M condition.
  • Discount factor γ = 0.99
    The M threshold and convergence rate in Theorem 4.4 depend on γ; treated as a standard problem parameter.
  • Epsilon for ϵ-softmax behavior policy = 0.05
    The mix of softmax policy with a uniform policy stabilizes Q-learning in experiments; not part of the theoretical convergence proof.
assumptions (4)
  • domain assumption Finite state and action spaces for the MDP
    Sec. 3 defines the MDP with finite S and A; the softmax policy update and sampling require finite action spaces, as the authors state.
  • standard math Entropy-regularized Bellman operators are contractions and monotone, with fixed points Q^π_τ and Q*_τ
    Propositions A.1-A.5 in App. A.1, cited from Geist et al. (2019), are used throughout the proofs.
  • standard math Negative entropy is 1-strongly convex with respect to the L1 norm (Pinsker's inequality)
    Used in the proof of Corollary 4.1.1 (App. A.3.1, Eq. 52) to bound the L1 distance between policies.
  • domain assumption Approximate policy evaluation error is bounded by ϵ_eval
    Assumption A.1 in App. A.2; the deep RL implementation does not measure ϵ_eval, and the error bound in Thm A.6 scales with ϵ_eval / ((1-γ)(1-d1-d2)).

how reviews work

0 comments
Cite this review

Pith. "Pith review of StaQ it! Growing neural networks for Policy Mirror Descent." pith.science (2026). https://pith.science/paper/YNO4DL6U

@misc{pith2026250613862,
  author       = {Pith},
  title        = {Pith review of: StaQ it! Growing neural networks for Policy Mirror Descent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YNO4DL6U}},
  note         = {Machine review of arXiv:2506.13862}
}
abstract

In Reinforcement Learning (RL), regularization has emerged as a popular tool both in theory and practice, typically based either on an entropy bonus or a Kullback-Leibler divergence that constrains successive policies. In practice, these approaches have been shown to improve exploration, robustness and stability, giving rise to popular Deep RL algorithms such as SAC and TRPO. Policy Mirror Descent (PMD) is a theoretical framework that solves this general regularized policy optimization problem, however the closed-form solution involves the sum of all past Q-functions, which is intractable in practice. We propose and analyze PMD-like algorithms that only keep the last $M$ Q-functions in memory, and show that for finite and large enough $M$, a convergent algorithm can be derived, introducing no error in the policy update, unlike prior deep RL PMD implementations. StaQ, the resulting algorithm, enjoys strong theoretical guarantees and is competitive with deep RL baselines, while exhibiting less performance oscillation, paving the way for fully stable deep RL algorithms and providing a testbed for experimentation with Policy Mirror Descent.

Figures

Figures reproduced from arXiv: 2506.13862 by the authors.

Figure 1
Figure 1. Overview of StaQ, showing the continual training of a Q-function (left), from [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Policy return of StaQ and of deep RL baselines. Plots show mean and one standard [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Left: Deviation from mean policy returns for individual runs on Hopper, comparing StaQ and PQN, the best performing baseline on this environment. Returns are centered at every timestep by subtracting the mean across 10 seeds. In general, individual runs of StaQ have significantly lower variance across timesteps compared to baselines. For clarity, we plot the first three seeds, and one-sided tolerance intervals. See … view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Evolution of xk for two successive values of M, one being large enough for xk to converge. The plot additionally shows the sequence x ′ k introduced by Thm. 4.4 that closely follows the behavior of xk. See text for more details. Now to study the limit limM→∞ d1 + d2d −…
Figure 5
Figure 5. Figure 5: Policy performance across all environments. [PITH_FULL_IMAGE:figures/full_fig_p031_5.png]
Figure 6
Figure 6. Figure 6: Stability plots for Classic Control environments, plotting normalized performance [PITH_FULL_IMAGE:figures/full_fig_p032_6.png]
Figure 7
Figure 7. Figure 7: Stability plots for MuJoCo environments, plotting normalized performance of the [PITH_FULL_IMAGE:figures/full_fig_p033_7.png]
Figure 8
Figure 8. Figure 8: Stability plots for MinAtar environments, plotting normalized performance of the [PITH_FULL_IMAGE:figures/full_fig_p034_8.png]
Figure 9
Figure 9. Figure 9: Ablation study for different memory sizes [PITH_FULL_IMAGE:figures/full_fig_p035_9.png]
Figure 10
Figure 10. Figure 10: Q-values on four different states across 1000 iterations of StaQ, using an [PITH_FULL_IMAGE:figures/full_fig_p036_10.png]
Figure 11
Figure 11. Figure 11: The percentage of states (out of 100 states) in which from iteration [PITH_FULL_IMAGE:figures/full_fig_p036_11.png]
Figure 12
Figure 12. Figure 12: Left: Frequency of non-zero rewards of a uniform policy with sticky actions for different choice of Poisson rate λ on MountainCar over 5M timesteps. Middle: Entropy of learned policies under different behavior policies. Entropy of the uniform (Max entropy) policy plot…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 11 canonical work pages

  1. [1]

    single-task RL

    SAC is not directly used as a baseline because SAC is not compatible with discrete action spaces. However, M-DQN can be seen as an adaptation of SAC to discrete action spaces with an additional KL-divergence regularizer. Please see the discussion in Vieillard et al. (2020b) on page 3, between Eq. (1) and (2). Vieillard et al. (2020b) also describe Soft-DQ...

    work page 2020

  2. [2]

    We provide a discussion of the MountainCar environment and some of the challenges of exploration in an entropy- regularized setting in App. B.5. 0 1 2 3 4 5 2.5 3 3.5 4 4.5 5 5.5Return (x100) CartPole-v1 0 1 2 3 4 5 -0.8 -0.75 -0.7 -0.65 -0.6 Acrobot-v1 0 1 2 3 4 5 -2 -1 0 1 2 3 LunarLander-v2 0 1 2 3 4 5 -2 -1.8 -1.6 -1.4 -1.2 -1 MountainCar-v0 0 1 2 3 4...

    work page 2001

  3. [3]

    and a "sticky" behavior policy manages to find a good policy for MountainCar matching the best baseline. The final policy demonstrates much lower entropy compared toϵ-softmax policy that fails at learning for this environment, which confirms our statement that entropy maximization cannot be a universal tool when dealing with the exploration problems. C Co...

    work page 2018

  4. [7]

    and Tian, T

    Young, K. and Tian, T. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments.arXiv preprint arXiv:1903.03176,

    arXiv 1903

  5. [9]

    15 A Proofs This section includes proofs of the lemmas and theorems of the main paper. A.1 Properties of entropy regularized Bellman operators We first start with a reminder of some basic properties of the (entropy regularized) Bellman operators, as presented in (Geist et al., 2019). Within the MDP setting defined in Sec. 3, let T π τ be the operator defi...

    work page 2019

  6. [14]

    where the learner is presented with a sequence of MDPs and one evaluates whether the learner is able learn on the new MDPs while retaining the information of older ones (De Lange et al., 2021; Wang et al., 2024). Drawing a connection with RL is interesting because it opens up a plethora of CL methods that are not well researched in the deep RL context, bu...

    work page 2021

  7. [15]

    For PQN, we use the CleanRL implementation (Huang et al., 2022)

    For TRPO and PPO, we use the implementation provided instable-baselines4 (Raffin et al., 2021), while we used our in-house PyTorch implementation of (M)-DQN. For PQN, we use the CleanRL implementation (Huang et al., 2022). Across all environments, we enforce a time limit of 5000 steps. This is particularly useful for Seaquest-v1, since an agent can get st...

    work page 2019

  8. [16]

    40 Hyperparameter Classic MuJoCo MinAtar Discount factor (γ) 0.99 0.99 0.99 Horizon 2048 2048 1024 Num. epochs 10 10 3 Learning starts 5000 20000 20000 GAE parameter 0.95 0.95 0.95 VF coefficient 0.5 0.5 1 Entropy coefficient 0 0 0.01 Clipping parameter 0.2 0.20.1×α Optimizer Adam Adam Adam Architecture64×2 64×2 ∗ Conv(16, 3,

    work page

Show all 19 references
  1. [17]

    In the MinAtar environ- ments α is linearly annealed from 1 to 0 over the course of learning.∗Humanoid-v4 uses a hidden layer size of256

    + 128 MLP Activation function Tanh Tanh Tanh Learning rate3×10 −4 3×10 −4 2.5×10 −4 ×α Batch size 64 64 256 Table 4: PPO hyperparameters, based on (Schulman et al., 2017). In the MinAtar environ- ments α is linearly annealed from 1 to 0 over the course of learning.∗Humanoid-v4...

    2017

  2. [18]

    + 128 MLP Activation function Tanh Tanh Tanh Learning rate3×10 −4 3×10 −4 2.5×10 −4 Batch size 64 64 256 Table 5: TRPO hyperparameters, based on (Schulman et al., 2015).∗Humanoid-v4 uses a hidden layer size of256. 41 Hyperparameter Classic MuJoCo MinAtar Discount factor (γ) 0....

    2015

  3. [19]

    Classic and MinAtar hyperparameters are based on the original paper (Gallici et al., 2025), while MuJoCo hyperparameters were found by hyperparameter tuning

    + 128 MLP Normalization Type LayerNorm LayerNorm LayerNorm Input Normalization None None None Activation function ReLU ReLU ReLU Learning rate∗ 1×10 −4 1×10 −4 1×10 −4 Table 6: PQN hyperparameters. Classic and MinAtar hyperparameters are based on the original paper (Gallici et...

    2025

  4. [2003]

    doi: 10.1016/S0167-6377(02)00231-6

    ISSN 0167-6377. doi: 10.1016/S0167-6377(02)00231-6. Bhatt, A., Palenicek, D., Belousov, B., Argus, M., Amiranashvili, A., Brox, T., and Peters, J. CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity, March

    doi

  5. [2012]

    Mirror descent policy optimization

    Tomar, M., Shani, L., Efroni, Y., and Ghavamzadeh, M. Mirror descent policy optimization. arXiv preprint arXiv:2005.09814,

    2005 arXiv

  6. [2016]

    We can see in Fig

    and their off-policy nature might complicate learning (Kumar et al., 2020). We can see in Fig. 10 thatϵ = 0.05still exhibits sudden changes in the Q-function which might harm stability. While the averaging over past Q-functions of an EPMD policy can stabilize learning, we beli...

    2020

  7. [2019]

    S., Gower, R

    Yuan, R., Du, S. S., Gower, R. M., Lazaric, A., and Xiao, L. Linear convergence of natural policy gradient methods with log-linear policies.arXiv preprint arXiv:2210.01400,

    arXiv

  8. [2020]

    Homotopic policy mirror descent: Policy convergence, implicit regularization, and improved sample complexity.arXiv preprint arXiv:2201.09457,

    Li, Y., Lan, G., and Zhao, T. Homotopic policy mirror descent: Policy convergence, implicit regularization, and improved sample complexity.arXiv preprint arXiv:2201.09457,

    arXiv

  9. [2021]

    Randomized ensembled double q-learning: Learning fast without a model.arXiv preprint arXiv:2101.05982,

    Chen, X., Wang, C., Zhou, Z., and Ross, K. Randomized ensembled double q-learning: Learning fast without a model.arXiv preprint arXiv:2101.05982,

    arXiv

  10. [2022]

    Maxmin q-learning: Controlling the estimation bias of q-learning.arXiv preprint arXiv:2002.06487,

    Lan, Q., Pan, Y., Fyshe, A., and White, M. Maxmin q-learning: Controlling the estimation bias of q-learning.arXiv preprint arXiv:2002.06487,

    2002 arXiv

  11. [2023]

    van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J

    URLhttps: //zenodo.org/record/8127025. van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. Deep reinforcement learning and the deadly triad.arXiv,

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.