Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Exploration by Random Distribution Distillation

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Random Distribution Distillation claims that a prediction-error bonus can act as an unbiased estimator of the reciprocal visit count $1/n$, unifying curiosity-based and count-based exploration.

desk verdict RDD has a correct core estimator but the bridge from that estimator to the actual exploration bonus is broken, and the implementation does not match the theory. read the letter →

arxiv 2505.11044 v1 pith:EMGLXVFF submitted 2025-05-16 cs.LG

Zhirui Fang , Kai Yang , Jian Tao , Jiafei Lyu , Lusong Li , Li Shen , Xiu Li This is my paper · ORCID
classification cs.LG
keywords explorationintrinsicrewardrandomnetworkdistillationpseudo-countreinforcementlearningpredictionerrorsparse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Random Distribution Distillation (RDD) is an online reinforcement-learning exploration method that tries to combine the scalability of prediction-error curiosity with the principled decay of count-based bonuses. The intrinsic reward is the squared distance between a predictor network and the fixed random mean of a target network whose outputs are sampled from a normal distribution. The paper argues that in an idealized setting this reward estimates the reciprocal of the state's visit count, $1/n$, up to a convergence-discrepancy term, so the bonus shrinks as a state becomes familiar. If the argument holds, RND is the zero-variance special case of RDD, DRND is the finite-ensemble approximation, and RDD gives curiosity-driven exploration a pseudo-count mechanism. Experiments on sparse-reward Atari, Adroit, and Fetch tasks report faster convergence and higher returns than RND, DRND, and PPO.

What carries the argument

The load-bearing object is the statistic $z_n(s)=\|f^n_{\theta^*}(s)-\mu_{\bar\theta}(s)\|^2/\sigma_{\bar\phi}^2(s)$: the squared distance, normalized by target variance, between the empirical-risk-minimizing predictor after $n$ visits and the fixed random target mean. Because each visit draws an independent target output from $\mathcal{N}(\mu_{\bar\theta}(s),\sigma_{\bar\phi}^2(s))$, the average of $n$ samples has expected squared deviation $\sigma_{\bar\phi}^2(s)/n$ from the mean, making $z_n$ an unbiased estimator of $1/n$ with variance $2/n^2$. The same statistic grounds the bonus bound in Eq. 13, which decomposes the intrinsic reward into a pseudo-count term that decays with visits and a discrepancy term that shrinks as the predictor converges.

What would settle it

Repeatedly visit a single state while training the RDD predictor only on those visits, and record the bonus $b(s)=\frac{1}{d}\|f_\theta(s)-\mu_{\bar\theta}(s)\|^2$ as $n$ grows; if the bonus does not decrease like $\sigma^2/n$ plus a shrinking discrepancy term, and instead plateaus at a constant, the pseudo-count interpretation is refuted.

Watch

Extended reading notes

Core claim

The RDD bonus is $b(s)=\frac{1}{d}\|f_\theta(s)-\mu_{\bar\theta}(s)\|^2$, where $f_\theta$ is the predictor network and $\mu_{\bar\theta}(s)$ is the fixed mean of a random target distribution $\mathcal{N}(\mu_{\bar\theta}(s),\sigma_{\bar\phi}^2(s))$ from which fresh target outputs are sampled on each visit. The central theoretical claim is that the normalized statistic $z_n(s)=\|f^n_{\theta^*}(s)-\mu_{\bar\theta}(s)\|^2/\sigma_{\bar\phi}^2(s)$, evaluated at the optimal mean-squared-error predictor after $n$ visits, is an unbiased and consistent estimator of $1/n$ with variance $2/n^2$. The implemented bonus satisfies the bound $\|f_\theta(s)-\mu_{\bar\theta}(s)\|^2 \le \sigma_{\bar\phi}^2(s)/n + \mathbb{E}[\|f^n_{\theta^*}(s)-f_\theta(s)\|^2]$, splitting the reward into a pseudo-count term and a predictor-discrepancy term. From these results the paper concludes that RDD automatically couples exploration reward to both visit frequency and predictor convergence, that RND is the $\sigma_{\bar\phi}\to 0$ limit of RDD, and that DRND converges to RDD as the number of target networks grows.

Load-bearing premise

The count-decay claim holds only if the trained predictor at a state equals the exact running average of independent, state-specific target samples; when that fails, as the paper's own experiments partly concede by fixing the target mean to a scalar, the bonus need not decay like $1/n$.

Editorial extensions

If this is right

  • RDD gives a curiosity bonus a principled decay: rewards for frequently visited states shrink, so exploration shifts to unfamiliar states without needing a count table or density model.
  • RND is recovered as the limit where the target variance goes to zero, making RND's unnormalized prediction error a special case of a count-based statistic.
  • DRND's ensemble of target networks converges to RDD as the ensemble size grows, positioning RDD as the asymptotic limit of DRND.
  • Because the bonus bound includes a predictor-discrepancy term, the intrinsic reward is self-limiting: as the predictor converges, the bonus shrinks even for states that remain unfamiliar.
  • In the reported experiments, RDD converges faster and reaches higher final returns than RND, DRND, and PPO on sparse-reward Atari, Adroit, and Fetch tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test would be to measure the bonus decay curve on repeated identical states in a real environment; if $1/n$ tracking requires the predictor to be trained to convergence on those states, the practical gain may come from the discrepancy term rather than from pseudo-counting.
  • The ablation that fixes the target mean to the scalar $1$ suggests the implemented RDD is closer to a shaped novelty bonus than to the state-specific pseudo-count of the theory, so a state-dependent-mean variant could distinguish which design choice drives the reported gains.
  • Because output dimension $d$ reduces the variance of $z_n$, $d$ and $\sigma$ together act as a bias-variance knob for exploration intensity, suggesting a principled way to calibrate intrinsic-reward scale across environments.
  • If the predictor never fully fits repeated states, RDD may reduce to an RND-like unnormalized bonus; comparing RDD against RND with matched network capacity on identical visit sequences would reveal how much of the benefit is actually count-like.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Random Distribution Distillation (RDD), an exploration bonus for online RL. RDD samples the target network's output for a state from a normal distribution, trains a predictor with MSE loss, and sets the intrinsic reward to b(s) = (1/d)||f_theta(s) - mu_theta_bar(s)||^2, the squared distance between the predictor and the target distribution mean. The central theoretical claim is that the idealized statistic z_n = ||f^n_theta*(s) - mu_theta_bar(s)||^2 / sigma^2 is an unbiased and consistent estimator of 1/n with variance 2/n^2, so the bonus is bounded by a pseudo-count term sigma^2/n plus a discrepancy term that accounts for predictor convergence (Eq. 13). The paper further claims that RND is the sigma -> 0 special case and that DRND converges to RDD as the number of target networks grows. Experimental results are reported on Atari, Adroit, and Fetch tasks, along with ablations of the target mean, variance, and output dimension.

Significance. If the theoretical claims were correct and the implementation matched the model, RDD would be a useful unification of prediction-error and count-based exploration: it would provide a pseudo-count-like decay with a simple one-network bonus and with variance smaller than DRND's. The core computation in Appendix A, Var(z_n) = 2/n^2, is correct, and Figure 1 qualitatively supports the statistical variance-reduction story. However, the central bridge from the idealized statistic to the actual bonus, Eq. (13), is mathematically invalid as printed, the proof of consistency in Theorem 4.3 contains an error, and the implemented algorithm uses a constant target mean across states, whereas the theory requires a state-dependent target mean. These are load-bearing issues: they break the paper's main claim that the RDD bonus decays like a pseudo-count. The idea is salvageable, but the manuscript needs substantial revision before the claim can be accepted.

major comments (4)
  1. [§4.4, Eq. (13)] The decomposition in Eq. (13) does not follow mathematically. The left-hand side ||f_theta(s) - mu_theta_bar(s)||^2 is deterministic for fixed theta, so the first equality, which rewrites it as an expectation, is not meaningful in the form given. For the inequality, the squared-norm triangle inequality is ||a + b||^2 <= 2||a||^2 + 2||b||^2, so the correct bound would carry a factor of 2 on both the pseudo-count term and the discrepancy term. As printed, the claimed upper bound sigma^2/n + E[||f^n_theta*(s) - f_theta(s)||^2] is not established; the cross term needs to be handled explicitly and the coefficients corrected.
  2. [§4.2–§4.3, Eqs. (4) and (13)] The pseudo-count decay claim rests on identifying the trained predictor with f^n_theta*(s), the exact per-state empirical-risk minimizer of Eq. (4). In the implemented algorithm the predictor is a single network trained by SGD on all states jointly, and the paper gives no argument that it converges to the per-state empirical-risk minimizer for each state. If the discrepancy term E[||f^n_theta*(s) - f_theta(s)||^2] does not vanish with n, Eq. (13) does not imply that the bonus decays with the visit count n. This is the load-bearing step of the pseudo-count claim and needs either a convergence proof under the actual training procedure or a controlled experiment showing state-specific bonus decay that is independent of global training progress.
  3. [§5.4, Table 4] The implemented algorithm does not instantiate the model analyzed in Lemma 4.1. Table 4 and the ablation in Section 5.4 set the target mean mu to the scalar 1 (and sigma = 1.0) for every state, whereas Lemma 4.1 requires a state-dependent target mean mu_theta_bar(s). With a constant target distribution N(1, 1) shared across all states, the MSE-optimal predictor of Eq. (3) is the constant function f_theta(s) = 1 everywhere, so the bonus b(s) = ||f_theta(s) - 1||^2 decays as the network learns this constant function, with no dependence on how often each state is visited. The empirical results therefore do not test the pseudo-count interpretation. The paper must either modify the algorithm to use a state-dependent random target mean network, as the theory assumes, or explicitly state that the pseudo-count claim applies only to an idealized version of the algorithm and not to the evaluated method.
  4. [§4.2, Theorem 4.3] Theorem 4.3 contains an error in its statement and its proof does not establish consistency. The text says 'lim_{n->infinity} P(|z_n - 1/n| < epsilon) = 0', which is the opposite of consistency; the limit should be 1. In addition, the epsilon in the theorem statement is a function of n and delta whose second term tends to sqrt(ln(2/delta)/2), a positive constant, so the bound does not show that for a fixed epsilon the probability tends to zero. This is repairable: applying Chebyshev's inequality to Var(z_n) = 2/n^2 from Eq. (21) gives P(|z_n - 1/n| > epsilon) <= 2/(n^2 epsilon^2) -> 0, which is a clean consistency proof. As printed, both the theorem statement and the Appendix C derivation need correction.
minor comments (5)
  1. [§4.3, Eq. (8)] The notation mu_theta_bar(s) = mu_theta_bar(s) x 1_d is self-referential and confusing; the scalar mean should be named differently, for example mu, and the vectorized form written as mu * 1_d.
  2. [Algorithm 1, line 10] The pseudo-code says to compute 'the mean mu_theta(s_t) and second moment sigma_phi(s_t)', which suggests state-dependent target parameters, but Table 4 and Section 5.4 fix mu = 1 as a scalar constant. Please clarify whether mu is a learned state-dependent network or a constant, and make the pseudo-code consistent with the implementation.
  3. [§5.1, Figure 1] The quantities yapprox and zapprox are used in the caption and the plots but are not defined in the caption; please define them in the text or the caption so the figure is self-contained.
  4. [Table 1] Several rows in Table 1 appear to contain identical values across the RDD and DRND columns (for example, Montezuma and PrivateEye), which makes it difficult to assess the claimed improvements; please reformat the table so that the per-algorithm columns are clearly separated and the reported values are unambiguous.
  5. [§7, Limitations] The independence assumption stated in the Limitations section is precisely the assumption that the pseudo-count proof requires; the paragraph should state which theoretical conclusions are invalidated when the assumption fails, rather than only noting that similar states may have similar distributions.

Circularity Check

1 steps flagged · score 4.0 of 10

Central 1/n estimator analysis is self-contained, but the advertised variance comparison against DRND is imported from the authors' own prior ICML paper.

  1. self citation load bearing [Corollary 4.2 and Appendix B (proof of Corollary 4.2), Eqs. (22)-(23)]
    "As noted in Appendix A of DRND [12], when the target network fi tar(s)∼N(µ ¯θ(s),σ 2 ¯ϕ(s)), the mean and variance of DRND statistic yn(s) are given by: E[yn(s)] = E[zn(s)] = 1/n. Var[yn(s)] = ... = 2/n^2 + 4[µ ¯θ(s)]^2/(nσ 2 ¯ϕ(s)) ≥ Var[zn(s)]."

    The paper's claim that RDD's zn has strictly smaller variance than DRND's yn rests entirely on the DRND variance formula quoted from [12], a paper by the same authors (Yang, Tao, Lyu, Li). No derivation of Var[yn] is given in this manuscript; the advertised 'more effective' comparison therefore reduces to an unverified (in this paper) self-citation. The central 1/n unbiasedness of zn itself is derived from scratch in Appendix A, so the circularity is limited to the DRND-comparison claim.

full rationale

The main estimator analysis is not circular: Appendix A computes E[zn(s)] = 1/n and Var[zn(s)] = 2/n^2 directly from the i.i.d. Gaussian sampling assumption via moment identities, and Theorem 4.3 proves concentration from chi-square/Chernoff bounds; these do not presuppose the pseudo-count conclusion. The bonus bound in Eq. (13) is a decomposition rather than a tautology, though its triangle-inequality step is mathematically suspect (a squared-norm triangle inequality needs cross terms), which is a correctness issue, not circularity. Section 7 explicitly concedes that the independence-of-target-distributions assumption is violated in practice, weakening the pseudo-count interpretation but not making the derivation circular. The one genuinely circular element is the variance comparison to DRND: Corollary 4.2 and Appendix B import Var[yn(s)] from the authors' own ICML 2024 paper [12] and use it to claim RDD is 'more effective' than DRND; that specific advantage is not independently established here. Because the central 1/n decay claim has independent content, the overall circularity score is moderate rather than high.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or mathematical entity: it is a new estimator and reward design built from a Gaussian target distribution. The load-bearing ingredients are the i.i.d.-Gaussian-target assumption, the optimal-predictor idealization, and the tuned constants (sigma, mu, d, lambda), none of which have independent empirical support outside the paper's own ablations.

free parameters (4)
  • sigma (variance of the target distribution) = 1.0 (ablated: 0.1, 1.0, state-dependent sigma(s))
    Fixed variance of the Gaussian target. The ablation (Section 5.4) found sigma=1 best; sigma controls the interpolation between RND (sigma->0) and the pseudo-count regime. The theory holds for any sigma>0, so the value is an engineering choice, not a theoretical input.
  • mu (mean of the target distribution) = 1.0 (ablated: 0, 0.5, 1)
    The ablation chose mean 1 because unseen states then yield reward near 1 given the predictor's initialization; but the theory models mu_theta-bar(s) as a state-dependent random network, so the implemented constant scalar conflicts with the theoretical premise.
  • d (predictor/target output dimension) = 512 for image inputs, 64 for state vectors
    Larger d reduces the estimator variance by factor 1/d (Eq 10); the values are chosen per observation type and by the ablation in Section 5.4.
  • lambda (intrinsic reward scale) = 1.0
    The intrinsic and extrinsic advantages are summed with equal weight in Algorithm 1 (line 18); the paper does not ablate this coefficient.
assumptions (4)
  • domain assumption Target outputs f^i_tar(s) for successive visits to a state are i.i.d. samples from N(mu_theta-bar(s), sigma^2), with mu_theta-bar(s) a fixed random function
    Used in Lemma 4.1, Corollary 4.2, Theorem 4.3 (Eqs 4-7). The paper's own Section 7 concession that distributions across similar states are not independent weakens this premise in practice; the experiments use a constant mean 1, which contradicts the state-specificity.
  • domain assumption The MSE-trained predictor attains the empirical-risk minimizer f^n_theta*(s) = (1/n) sum_i f^i_tar(s) after n visits to s (Eq 4)
    Bridges the 1/n statistic to the algorithm's bonus. SGD-trained networks are not exact empirical-risk minimizers, and the discrepancy term in Eq 13 that is supposed to absorb this gap is never quantified.
  • standard math Standard MDP formalism, discounted returns, and actor-critic/Q-learning updates (Section 3)
    The RL framework the method is embedded in; standard background.
  • standard math Properties of chi-squared distributions, Gaussian moments, Chebyshev and Chernoff bounds, and the DRND variance formula from [12]'s appendix
    Used in Appendices A-C; the application in Theorem 4.3 is flawed (vacuous bound), and the DRND formula is quoted from the authors' own prior work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploration by Random Distribution Distillation." pith.science (2026). https://pith.science/paper/EMGLXVFF

@misc{pith2026250511044,
  author       = {Pith},
  title        = {Pith review of: Exploration by Random Distribution Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EMGLXVFF}},
  note         = {Machine review of arXiv:2505.11044}
}
read the original abstract

Exploration remains a critical challenge in online reinforcement learning, as an agent must effectively explore unknown environments to achieve high returns. Currently, the main exploration algorithms are primarily count-based methods and curiosity-based methods, with prediction-error methods being a prominent example. In this paper, we propose a novel method called \textbf{R}andom \textbf{D}istribution \textbf{D}istillation (RDD), which samples the output of a target network from a normal distribution. RDD facilitates a more extensive exploration by explicitly treating the difference between the prediction network and the target network as an intrinsic reward. Furthermore, by introducing randomness into the output of the target network for a given state and modeling it as a sample from a normal distribution, intrinsic rewards are bounded by two key components: a pseudo-count term ensuring proper exploration decay and a discrepancy term accounting for predictor convergence. We demonstrate that RDD effectively unifies both count-based and prediction-error approaches. It retains the advantages of prediction-error methods in high-dimensional spaces, while also implementing an intrinsic reward decay mode akin to the pseudo-count method. In the experimental section, RDD is compared with more advanced methods in a series of environments. Both theoretical analysis and experimental results confirm the effectiveness of our approach in improving online exploration for reinforcement learning tasks.

Figures

Figures reproduced from arXiv: 2505.11044 by the authors.

Figure 1
Figure 1. Numerical Example. We investigated the decay of bonus with each access to the same state when the [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Main experiment. We selected the currently most effective prediction-error method on 12 different [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Visualize the exploration process on the MountainCar. In the MountainCar environment, the position [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: We mainly analyzed the mean and variance of the target network, as well as the influence of the output [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Environments of experiments • FetchPush: The robot arm is trained to push an object to a target position. The target position is randomly generated on the table, and the reward is given if the object reaches the target position. Maximum episode length is 2.5 × 106 step…
Figure 6
Figure 6. Figure 6: Diagram of RDD G Compute Resources The experiments in this paper were conducted on a computing cluster, with the detailed hardware configurations listed in [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Exploration by Random Reward Perturbation

    cs.LG 2025-06 reject novelty 3.0 of 10

    Adding annealed Gaussian noise to rewards can help RL exploration, but this paper's proof of that claim is invalid and its SAC algorithm actually uses biased, non-zero-mean noise.

Reference graph

Works this paper leans on

45 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deep exploration via bootstrapped dqn,

    I. Osband, C. Blundell, A. Pritzel, and B. Van Roy, “Deep exploration via bootstrapped dqn,” Advances in neural information processing systems, vol. 29, 2016

    work page 2016

  2. [2]

    An analysis of model-based interval estimation for markov decision processes,

    A. L. Strehl and M. L. Littman, “An analysis of model-based interval estimation for markov decision processes,”Journal of Computer and System Sciences, vol. 74, no. 8, pp. 1309–1331, 2008

    2008

  3. [3]

    Minimax regret bounds for reinforcement learning,

    M. G. Azar, I. Osband, and R. Munos, “Minimax regret bounds for reinforcement learning,” in International conference on machine learning. PMLR, 2017, pp. 263–272

    work page 2017

  4. [4]

    The alberta plan for ai research,

    R. S. Sutton, M. Bowling, and P. M. Pilarski, “The alberta plan for ai research,”arXiv preprint arXiv:2208.11173, 2022

    arXiv 2022

  5. [5]

    Unifying count-based exploration and intrinsic motivation,

    M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos, “Unifying count-based exploration and intrinsic motivation,”Advances in neural information processing systems, vol. 29, 2016

    work page 2016

  6. [6]

    Flipping coins to estimate pseudocounts for exploration in reinforcement learning,

    S. Lobel, A. Bagaria, and G. Konidaris, “Flipping coins to estimate pseudocounts for exploration in reinforcement learning,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 22 594–22 613

    work page 2023

  7. [7]

    Count-based exploration with neural density models,

    G. Ostrovski, M. G. Bellemare, A. Oord, and R. Munos, “Count-based exploration with neural density models,” inInternational conference on machine learning. PMLR, 2017, pp. 2721– 2730

    work page 2017

  8. [8]

    Count-based exploration with the successor representation,

    M. C. Machado, M. G. Bellemare, and M. Bowling, “Count-based exploration with the successor representation,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 04, 2020, pp. 5125–5133

    work page 2020

Show all 45 references
  1. [9]

    Curiosity-driven exploration by self- supervised prediction,

    D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-driven exploration by self- supervised prediction,” inInternational conference on machine learning. PMLR, 2017, pp. 2778–2787

    2017

  2. [10]

    Exploration by random network distillation,

    Y . Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration by random network distillation,” arXiv preprint arXiv:1810.12894, 2018

    2018 arXiv

  3. [11]

    Is q-learning provably efficient?

    C. Jin, Z. Allen-Zhu, S. Bubeck, and M. I. Jordan, “Is q-learning provably efficient?”Advances in neural information processing systems, vol. 31, 2018

    2018

  4. [12]

    Exploration and anti-exploration with distributional random network distillation,

    K. Yang, J. Tao, J. Lyu, and X. Li, “Exploration and anti-exploration with distributional random network distillation,” inForty-first International Conference on Machine Learning, 2024

    2024

  5. [13]

    Q-learning,

    C. J. Watkins and P. Dayan, “Q-learning,”Machine learning, vol. 8, pp. 279–292, 1992

    1992

  6. [14]

    Human-level control through deep rein- forcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovskiet al., “Human-level control through deep rein- forcement learning,”nature, vol. 518, no. 7540, pp. 529–533, 2015

    2015

  7. [15]

    Rainbow: Combining improvements in deep reinforcement learning,

    M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: Combining improvements in deep reinforcement learning,” inProceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018

    2018

  8. [16]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” inInternational conference on machine learning. Pmlr, 2018, pp. 1861–1870. 10

    2018

  9. [17]

    Using confidence bounds for exploitation-exploration trade-offs,

    P. Auer, “Using confidence bounds for exploitation-exploration trade-offs,”Journal of Machine Learning Research, vol. 3, no. Nov, pp. 397–422, 2002

    2002

  10. [18]

    Improving generalization for temporal difference learning: The successor representa- tion,

    P. Dayan, “Improving generalization for temporal difference learning: The successor representa- tion,”Neural computation, vol. 5, no. 4, pp. 613–624, 1993

    1993

  11. [19]

    On bonus-based exploration methods in the arcade learning environment,

    A. A. Taiga, W. Fedus, M. C. Machado, A. Courville, and M. G. Bellemare, “On bonus-based exploration methods in the arcade learning environment,”arXiv preprint arXiv:2109.11052, 2021

    2021 arXiv

  12. [20]

    # exploration: A study of count-based exploration for deep reinforcement learning,

    H. Tang, R. Houthooft, D. Foote, A. Stooke, O. Xi Chen, Y . Duan, J. Schulman, F. DeTurck, and P. Abbeel, “# exploration: A study of count-based exploration for deep reinforcement learning,” Advances in neural information processing systems, vol. 30, 2017

    2017

  13. [21]

    Optimistic exploration even with apessimistic initialisation

    A. INITIALISATION, “Optimistic exploration even with apessimistic initialisation.”

  14. [22]

    First return, then explore,

    A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune, “First return, then explore,” Nature, vol. 590, no. 7847, pp. 580–586, 2021

    2021

  15. [23]

    Maximum entropy gain exploration for long horizon multi-goal reinforcement learning,

    S. Pitis, H. Chan, S. Zhao, B. Stadie, and J. Ba, “Maximum entropy gain exploration for long horizon multi-goal reinforcement learning,” inInternational Conference on Machine Learning. PMLR, 2020, pp. 7750–7761

    2020

  16. [24]

    Incentivizing exploration in reinforcement learning with deep predictive models,

    B. C. Stadie, S. Levine, and P. Abbeel, “Incentivizing exploration in reinforcement learning with deep predictive models,”arXiv preprint arXiv:1507.00814, 2015

    2015 arXiv

  17. [25]

    Vime: Variational information maximizing exploration,

    R. Houthooft, X. Chen, Y . Duan, J. Schulman, F. De Turck, and P. Abbeel, “Vime: Variational information maximizing exploration,”Advances in neural information processing systems, vol. 29, 2016

    2016

  18. [26]

    Latent world models for intrinsically motivated exploration,

    A. Ermolov and N. Sebe, “Latent world models for intrinsically motivated exploration,”Ad- vances in Neural Information Processing Systems, vol. 33, pp. 5565–5575, 2020

    2020

  19. [27]

    Byol-explore: Exploration by bootstrapped prediction,

    Z. Guo, S. Thakoor, M. Pîslar, B. Avila Pires, F. Altché, C. Tallec, A. Saade, D. Calandriello, J.-B. Grill, Y . Tanget al., “Byol-explore: Exploration by bootstrapped prediction,”Advances in neural information processing systems, vol. 35, pp. 31 855–31 870, 2022

    2022

  20. [28]

    Near-optimal reinforcement learning in polynomial time,

    M. Kearns and S. Singh, “Near-optimal reinforcement learning in polynomial time,”Machine learning, vol. 49, pp. 209–232, 2002

    2002

  21. [29]

    R-max-a general polynomial time algorithm for near- optimal reinforcement learning,

    R. I. Brafman and M. Tennenholtz, “R-max-a general polynomial time algorithm for near- optimal reinforcement learning,”Journal of Machine Learning Research, vol. 3, no. Oct, pp. 213–231, 2002

    2002

  22. [30]

    Exploration in metric state spaces,

    S. Kakade, M. J. Kearns, and J. Langford, “Exploration in metric state spaces,” inProceedings of the 20th International Conference on Machine Learning (ICML-03), 2003, pp. 306–312

    2003

  23. [31]

    R. S. Sutton, A. G. Bartoet al.,Introduction to reinforcement learning. MIT press Cambridge, 1998, vol. 135

    1998

  24. [32]

    Large-scale study of curiosity-driven learning,

    Y . Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros, “Large-scale study of curiosity-driven learning,”arXiv preprint arXiv:1808.04355, 2018

    2018 arXiv

  25. [33]

    Highly efficient self-adaptive reward shaping for reinforcement learning,

    H. Ma, Z. Luo, T. V . V o, K. Sima, and T.-Y . Leong, “Highly efficient self-adaptive reward shaping for reinforcement learning,”arXiv preprint arXiv:2408.03029, 2024

    2024 arXiv

  26. [34]

    Reward shaping for reinforcement learning with an assistant reward agent,

    H. Ma, K. Sima, T. V . V o, D. Fu, and T.-Y . Leong, “Reward shaping for reinforcement learning with an assistant reward agent,” inForty-first international conference on machine learning, 2024

    2024

  27. [35]

    Uncertainty-aware reward-free exploration with general function approximation,

    J. Zhang, W. Zhang, D. Zhou, and Q. Gu, “Uncertainty-aware reward-free exploration with general function approximation,”arXiv preprint arXiv:2406.16255, 2024

    2024 arXiv

  28. [36]

    Learning to shape rewards using a game of two partners,

    D. Mguni, T. Jafferjee, J. Wang, N. Perez-Nieves, W. Song, F. Tong, M. Taylor, T. Yang, Z. Dai, H. Chenet al., “Learning to shape rewards using a game of two partners,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 10, 2023, pp. 11 604–11 612

    2023

  29. [37]

    Exploration-guided reward shaping for reinforce- ment learning under sparse rewards,

    R. Devidze, P. Kamalaruban, and A. Singla, “Exploration-guided reward shaping for reinforce- ment learning under sparse rewards,”Advances in Neural Information Processing Systems, vol. 35, pp. 5829–5842, 2022

    2022

  30. [38]

    Addressing function approximation error in actor-critic methods,

    S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” inInternational conference on machine learning. PMLR, 2018, pp. 1587–1596. 11

    2018

  31. [39]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  32. [40]

    The arcade learning environment: An evaluation platform for general agents,

    M. G. Bellemare, Y . Naddaf, J. Veness, and M. Bowling, “The arcade learning environment: An evaluation platform for general agents,”Journal of artificial intelligence research, vol. 47, pp. 253–279, 2013

    2013

  33. [41]

    Learn- ing complex dexterous manipulation with deep reinforcement learning and demonstrations,

    A. Rajeswaran, V . Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine, “Learn- ing complex dexterous manipulation with deep reinforcement learning and demonstrations,” arXiv preprint arXiv:1709.10087, 2017

    2017 arXiv

  34. [42]

    Multi-goal reinforcement learning: Challenging robotics environments and request for research,

    M. Plappert, M. Andrychowicz, A. Ray, B. McGrew, B. Baker, G. Powell, J. Schneider, J. Tobin, M. Chociej, P. Welinderet al., “Multi-goal reinforcement learning: Challenging robotics environments and request for research,”arXiv preprint arXiv:1802.09464, 2018

    2018 arXiv

  35. [43]

    Gymnasium: A standard interface for reinforcement learning environments,

    M. Towers, A. Kwiatkowski, J. Terry, J. U. Balis, G. De Cola, T. Deleu, M. Goulao, A. Kallinteris, M. Krimmel, A. KGet al., “Gymnasium: A standard interface for reinforcement learning environments,”arXiv preprint arXiv:2407.17032, 2024

    2024 arXiv

  36. [44]

    Double check your state before trusting it: Confidence-aware bidirectional offline model-based imagination,

    J. Lyu, X. Li, and Z. Lu, “Double check your state before trusting it: Confidence-aware bidirectional offline model-based imagination,”Advances in Neural Information Processing Systems, vol. 35, pp. 38 218–38 231, 2022

    2022

  37. [45]

    Optimistic initialization for exploration in continuous control,

    S. Lobel, O. Gottesman, C. Allen, A. Bagaria, and G. Konidaris, “Optimistic initialization for exploration in continuous control,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 7, 2022, pp. 7612–7619. 12 APPENDIX A Proof of Lemma 4.1 First, we p...

    2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.