Uncertainty Prioritized Experience Replay

Claudia Clopath; Rodrigo Carrasco-Davis; Sebastian Lee; Will Dabney

REVIEW 2 major objections 5 minor 95 references

Uncertainty Prioritized Experience Replay

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes replacing TD-error priorities in experience replay with an information-gain criterion $p_i = \frac{1}{2}\log(1 + \hat E_\delta / \hat A)$, and reports that this raises median human-normalized Atari-57 scores.

desk verdict A genuinely useful idea for replay prioritization, but the Atari-57 headline rests on a tuning protocol that undermines it until fixed. read the letter →

arxiv 2506.09270 v1 pith:A44HYHPG submitted 2025-06-10 cs.LG

Rodrigo Carrasco-Davis , Sebastian Lee , Claudia Clopath , Will Dabney This is my paper

classification cs.LG

keywords experiencereplayprioritizedepistemicuncertaintyaleatoricinformationgaindistributionalreinforcementlearningquantileregressionAtari-57

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard prioritized experience replay, which samples transitions in proportion to their temporal-difference error, wastes updates on noisy transitions that cannot teach the agent anything. It proposes prioritizing by an information-gain criterion $p_i = \frac{1}{2}\log(1 + \hat E_\delta / \hat A)$, where $\hat E_\delta$ is a target-aware epistemic uncertainty and $\hat A$ is aleatoric (irreducible) noise. The authors show in a bandit, a gridworld, and the Atari-57 suite that this priority improves sample efficiency and final performance over TD-error prioritization, with the Atari gains coming from an ensemble of QR-DQN heads. If the claim holds, replay buffers can be steered toward learnable transitions rather than merely surprising ones.

What carries the argument

The load-bearing object is the target total uncertainty decomposition $\hat U_\delta = \mathbb{E}_{\tau,\psi}[(\Theta - \theta_\tau(\psi))^2] = \delta^2_\Theta + \hat E + \hat A$, obtained by adding and subtracting the ensemble-mean quantile value inside the squared error. $\delta^2_\Theta$ is the squared distance from the ensemble-mean estimate to the Q-learning target, $\hat E$ is the ensemble disagreement over quantiles, and $\hat A$ is the variance of the ensemble-average distribution. Summing the first two gives the target epistemic uncertainty $\hat E_\delta$, which fixes the blind spot of ensemble disagreement alone, since identically initialized heads give zero $\hat E$. The information-gain formula $\frac{1}{2}\log(1 + \hat E_\delta/\hat A)$ then calibrates epistemic uncertainty by aleatoric noise, so transitions whose surprise is mostly irreducible noise are down-weighted.

What would settle it

In an Atari game with artificially added reward noise, compare UPER against PER: if UPER's median human-normalized score does not remain above PER's as noise increases, the claim that information-gain prioritization avoids noisy-TV oversampling fails.

Watch

Extended reading notes

Core claim

The central claim is that the right quantity to replay is not how wrong the prediction was but how much a new update can reduce the estimator's uncertainty. Concretely, the paper defines target epistemic uncertainty as $\hat E_\delta(s,a) = \delta^2_\Theta(s,a) + \hat E(s,a)$, the squared distance from the ensemble-mean quantile estimate to the TD target plus the ensemble disagreement, and aleatoric uncertainty $\hat A(s,a)$ as the variance of the ensemble-averaged return distribution. Combining them as the information gain of a hypothetical Gaussian posterior yields the priority $p_i = \frac{1}{2}\log(1 + \hat E_\delta/\hat A)$. The paper's thesis is that this priority avoids the noisy-TV failure mode of TD-error prioritization and yields higher median human-normalized Atari-57 scores than PER, QR-DQN, QR-PER, and an ensemble QR-DQN with TD-error prioritization, while ablations show the gain is attributable to the priority variable itself.

Load-bearing premise

The claim rests on the assumption that the ensemble of QR-DQN heads estimates the same epistemic and aleatoric variances that appear in the Gaussian information-gain derivation; the paper validates this mapping with a single simulation, so if the estimates are miscalibrated in deep RL, the priority is not a true information gain.

Editorial extensions

If this is right

Replacing only the priority variable in PER, while keeping the replay algorithm and architecture unchanged, is enough to raise median human-normalized Atari-57 performance above TD-error prioritization.
Prioritization should use target epistemic uncertainty $\hat E_\delta$ rather than ensemble disagreement $\hat E$, because the distance-to-target term captures bias that disagreement misses.
The information-gain form should beat prioritization by epistemic uncertainty alone, since dividing by aleatoric uncertainty suppresses transitions whose surprise is mostly irreducible noise.
The benefit transfers to other distributional agents, as an ensemble C51 agent with UPER outperformed its PER counterpart on four of five tested Atari games.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

(Editorial inference) The same information-gain ratio could serve as an acquisition function for active learning or data selection outside RL, since it targets the same trade-off between reducible uncertainty and data noise.
(Editorial inference) The paper's supplementary observation that quantile regression underestimates distribution tails implies $\hat A$ may be biased low, which would inflate UPER priorities; a bias-corrected aleatoric estimate is a testable variant.
(Editorial inference) The bias-as-temperature analysis suggests the functional form of the priority matters most under model misspecification, so comparing UPER variants on out-of-distribution or noise-injected Atari levels would sharpen the design choice.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The idea is genuinely worth having, but the headline Atari-57 claim is not yet supported by the experimental protocol as written. Read this for the decomposition and the information-gain priority, not for the median curve.

What is new: the target epistemic uncertainty \hat E_\delta = \delta_\Theta^2 + \hat E, which adds the squared distance to the TD target to the ensemble disagreement of Clements et al., and the priority p_i = 1/2 log(1 + \hat E_\delta / \hat A). The algebra in Eq. 9 is correct — it is just a bias-variance decomposition of the squared error to the target over quantiles and ensemble. The bandit and gridworld toys are well chosen to show why TD-error over-samples noisy transitions, and the QR-ENS-PER ablation is the right control, because it isolates the priority variable from the ensemble architecture. The simulation in SM 3.2 does at least attempt to justify the mapping from ensemble variance to the Gaussian prior/data variances. The paper also honestly flags, in SM 3.4, that model misspecification acts as a temperature on the priority distribution. No glaring citation gaps that I can see.

Soft spots: the main empirical claim is selection-biased. The three UPER hyperparameters were chosen on five Atari games, and those games are inside Atari-57 and include exactly the games (Asterix, Chopper Command) used to showcase the gains. Baselines get no analogous sweep, so part of the median advantage could be tuned luck. Three seeds and no significance test make it worse. The QR-ENS-PER comparison is also not protected, since that baseline inherits UPER's hyperparameters rather than its own tuning budget. The mapping from \hat E_\delta and \hat A to Gaussian prior and data variances is plausible but only demonstrated on one synthetic regression; in deep RL it remains a heuristic. No code is released, which matters for an empirical paper.

None of this kills the core idea. The decomposition is correct, the mechanism is coherent, and the toy results support it. But the deep RL evidence is conditional: UPER could well be a real improvement, and the paper currently overstates how well the Atari results demonstrate that.

I would send this to peer review, with a request to fix the evaluation: hold out the tuning games (or tune baselines the same way), add seeds and significance testing, and release code. A reviewer should focus on the empirical protocol, not the math. I'd mention UPER in my own work, but not rely on the Atari median as established.

Referee Report

2 major / 5 minor

Summary. The paper proposes Uncertainty Prioritized Experience Replay (UPER), which replaces the TD-error priority in prioritized experience replay with an information-gain criterion built from estimates of epistemic and aleatoric uncertainty. The authors extend the Clements et al. (2020) uncertainty decomposition by adding a distance-to-target term, defining the target epistemic uncertainty in Eq. (9), and derive the priority variable p_i = (1/2) log(1 + \hat E_delta / \hat A) in Eq. (11). They demonstrate the method on a conal bandit and a noisy gridworld, then report Atari-57 median human-normalized scores comparing UPER against QR-DQN, PER, QR-PER, and QR-ENS-PER, with additional ablations and a C51 extension in the supplementary material.

Significance. If the Atari-57 results are robust, the paper makes a useful contribution: it identifies a concrete failure mode of TD-error prioritization in noisy environments and proposes a principled, uncertainty-based alternative with a sound algebraic decomposition. The derivation in Appendix A is correct, the toy experiments cleanly illustrate the noisy-TV intuition, and the ablation in SM 6.1 (QR-ENS-EPI, QR-ENS-UNI, QR-ENS-PER) is a well-designed attempt to isolate the priority variable from architectural changes. The authors also report computational costs transparently. The paper is weakened, however, by an evaluation protocol that tunes UPER on part of the same Atari-57 set used for the headline comparison, and by the lack of statistical testing. These issues concern the central empirical claim and need to be addressed before the paper can be accepted.

major comments (2)

[SM 6 / Fig. 2] The main empirical claim is compromised by selection bias in the hyperparameter tuning. SM 6 states that UPER's learning rate, Adam epsilon, and priority exponent were selected by average performance over 2 seeds on Chopper Command, Asterix, Gopher, Space Invaders, and Battlezone. These five games are part of the Atari-57 set whose median is reported in Fig. 2, and the paper's motivating examples of large gains (Asterix, Chopper Command) are exactly among the tuning games. Baselines, by contrast, use their original published hyperparameters without an analogous sweep. With only 3 seeds per game and no significance test or confidence interval for the median difference, the reported 'significantly higher' median could reflect favorable hyperparameter selection on the evaluation subset rather than a general benefit of information-gain prioritization. The QR-ENS-PER comparison is also affected, because QR-ENS-PER inherits the hyperparameters tuned for UPER rather than receiving its own tuning budget. I recommend either a formal held-out split (e.g., tune on a separate subset and evaluate on the remaining games) or a fixed, pre-specified hyperparameter setting for UPER, together with additional seeds and a significance test or interval estimate.
[SM 3.2 / Eq. (11)] The information-gain derivation substitutes \hat E_delta for the Gaussian prior variance and \hat A for the data variance in Eq. (11), but the mapping is validated only by the simple Gaussian regression simulation in SM 3.2. That simulation itself shows quantitative mismatches: the ensemble disagreement \hat E converges to zero at a different rate than the Bayesian posterior variance (Fig. 3d), and \hat A underestimates the true data variance (Fig. 3f). In the deep RL setting there is no evidence that these ensemble statistics correspond to the variances of a Gaussian posterior over the value estimate. If the mapping fails, p_i is not actually an information gain, and the observed advantage over plain epistemic prioritization could arise from a different mechanism. The authors should either validate the variance interpretation in a more realistic function-approximation setting, or soften the claim that UPER prioritizes by information gain and present Eq. (11) explicitly as a heuristic calibrated by the toy analysis.

minor comments (5)

[Abstract / Fig. 2] The abstract and Fig. 2 caption claim UPER is 'significantly' better than baselines, but no significance test is reported; the claim should be rephrased as a higher median or supported by a statistical test.
[SM 1.1] SM 1.1 ends with an incomplete sentence: 'This form of estimating aleatoric uncertainty does not require quantile regression, but' — the sentence should be completed or removed.
[SM 7 / Fig. 16] The caption of Fig. 16 says 'ensemble C51 agent with PER vs ensemble C51 agent with PER' for both arms; the second instance should presumably be 'UPER'.
[Fig. 2] The shaded region in Fig. 2 is described as 'two standard deviations', but it is unclear whether this is the standard deviation of the per-game means across seeds or of the median; please clarify the aggregation and whether the shading is on the mean or median curve.
[Eqs. (8)-(11)] The notation \hat E is used both for the ensemble disagreement in Eq. (8) and for the target epistemic uncertainty \hat E_delta in Eq. (9); using a distinct symbol, such as \hat E_ens, would reduce confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: UPER's priority variable is derived from a Gaussian information-gain model with explicitly stated variance estimates, and the Atari ablation is architecturally controlled.

full rationale

The derivation chain is self-contained and does not reduce to its own inputs. The target uncertainty decomposition in Eq. (9) and Appendix A is an algebraic identity obtained by expanding E[(Theta - theta)^2] about the ensemble mean; no term in that expansion is defined in terms of the final priority variable. The information-gain formulation in Eq. (11) and SM 3.1 is a standard Gaussian entropy calculation, and the identification sigma^2 = E_hat_delta, sigma_x^2 = A_hat is an explicitly stated modeling assumption rather than a hidden fit. SM 3.2 provides an independent sanity check against the Bayesian posterior on synthetic Gaussian data, not against the downstream RL performance objective, so it does not make the UPER claim true by construction. The central Atari ablation (SM 6.1) varies only the priority variable among identical QR-DQN ensembles, controlling for architecture and base algorithm. Citations to Dabney et al. (2017) and Bellemare et al. (2017) supply standard distributional-RL machinery and are not load-bearing for the UPER claim itself. The SM 6 hyperparameter sweep on five Atari games is a legitimate benchmark-selection concern for external validity, but it does not make the reported median improvement equivalent to the fitted hyperparameters by construction, so it falls outside the circularity criteria used here.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the assumption that ensemble-based variance estimates faithfully represent the epistemic and aleatoric components in the information gain model. This is a heuristic mapping, supported only by a simulation in SM 3.2. The free parameters are the hyperparameters of the deep RL agent, tuned for UPER on a subset of games.

free parameters (4)

learning rate (UPER) = not stated (swept 3e-5 to 5e-5)
Chosen by hyperparameter sweep on 5 Atari games; affects all Atari results.
Adam epsilon (UPER) = not stated (swept 6.1e-7 to 3.125e-4)
Chosen by sweep; part of optimizer configuration different from QR-DQN baseline.
prioritization exponent alpha = not stated (swept 0.6 to 1)
Controls the sampling distribution in PER; tuned for UPER only.
small constant in information gain denominator = not specified
SM 3.3 mentions adding small constants to avoid discontinuities at low noise; exact value not reported.

assumptions (4)

domain assumption Ensemble members are independent given masking and random initialization.
Sec. 5: masks and random init facilitate independence; the uncertainty decomposition assumes independence across ensemble heads to treat \hat E as posterior variance.
domain assumption Epistemic and aleatoric uncertainty can be estimated from the variance of ensemble quantiles (Clements et al. 2020).
Borrowed from Clements et al.; not proven in this paper, but standard in the field.
ad hoc to paper The Gaussian information gain formula applies when replacing prior variance with \hat E_delta and data variance with \hat A.
SM 3.2 provides a simulation but no derivation that the ensemble estimators behave as Gaussian posterior/data variances in RL.
domain assumption Excess risk equals epistemic uncertainty (DEUP framework).
Sec. 2.3.3, from Lahlou et al.; the paper adopts it and notes limitations in SM 3.4.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Uncertainty Prioritized Experience Replay." pith.science (2026). https://pith.science/paper/A44HYHPG

@misc{pith2026250609270,
  author       = {Pith},
  title        = {Pith review of: Uncertainty Prioritized Experience Replay},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A44HYHPG}},
  note         = {Machine review of arXiv:2506.09270}
}

read the original abstract

Prioritized experience replay, which improves sample efficiency by selecting relevant transitions to update parameter estimates, is a crucial component of contemporary value-based deep reinforcement learning models. Typically, transitions are prioritized based on their temporal difference error. However, this approach is prone to favoring noisy transitions, even when the value estimation closely approximates the target mean. This phenomenon resembles the noisy TV problem postulated in the exploration literature, in which exploration-guided agents get stuck by mistaking noise for novelty. To mitigate the disruptive effects of noise in value estimation, we propose using epistemic uncertainty estimation to guide the prioritization of transitions from the replay buffer. Epistemic uncertainty quantifies the uncertainty that can be reduced by learning, hence reducing transitions sampled from the buffer generated by unpredictable random processes. We first illustrate the benefits of epistemic uncertainty prioritized replay in two tabular toy models: a simple multi-arm bandit task, and a noisy gridworld. Subsequently, we evaluate our prioritization scheme on the Atari suite, outperforming quantile regression deep Q-learning benchmarks; thus forging a path for the use of uncertainty prioritized replay in reinforcement learning agents.

Figures

Figures reproduced from arXiv: 2506.09270 by the authors.

**Figure 1.** Conal Bandit. (a) multi-armed bandit task constructed such that each arm has identical mean payoff but increasing variance. (b) true MSE (average error across arms, between estimated reward and the true reward mean) over 200 iterations (each of 1000 steps) using different quantities to prioritise transitions from the replay buffer: absolute value of the TD error |δ| (PER), inverse counts (C being the number of visit… view at source ↗

**Figure 2.** (Left) Comparing Uncertainty Prioritized Experience Replay (UPER) with Prioritized Experience Replay (PER) and QR-DQN on the full Atari-57 benchmark. Median human normalized score for UPER is significantly higher than baselines throughout the learning trajectory. (Right) Example of per-game performance, with vastly superior performance on e.g. Asterix and Chopper Command; cases in which UPER is worse are far less ex… view at source ↗

**Figure 3.** Variances in the information gain can be approximated by epistemic and aleatoric uncertainty in the information gain: (a) and (b) Evolution during training of the posterior of the mean using an ensemble (gaussian fitted to members of the ensemble at each step) and an ideal Gaussian respectively, as described in Sec. 3.2. Training progresses from purple to yellow. (c): Fitted ensemble quantiles to true data distribu… view at source ↗

Figures from the paper (13 more)

**Figure 4.** Figure 4: Ratios can reduce entropy of distribution under bias. µδ ∗ σδ ∗ (a) ·10−11 ·10−11 0 1 2 3 ρη = U(0, 1) ρβ = U(0, 1) 0 2 4 6 8 10 0 2 4 6 (b) ·10−11 ·10−10 0 2 4 6 ρη = U(0, 1) ρβ = U(0, 3) 0 2 4 6 8 10 0 0.5 1 1.5 2 (c) ·10−11 ·10−11 0 1 2 3 ρη = U(0, 3) ρβ = U(0, 1) 0…

**Figure 5.** Figure 5: E 2/U closely approximates E for non-trivial bias. Interestingly however, for m = 2 there is very fast convergence of E 2/U and E as a function of C. So while m = 3 has a very stark trade-off, m = 2 is less extreme: For low C it may make you more wrong but generally yo…

**Figure 6.** Figure 6: Comparison of MSE for different prioritisation scheemes. Left panel, shows ratios and [PITH_FULL_IMAGE:figures/full_fig_p030_6.png]

**Figure 7.** Figure 7: Comparison of MSE for different prioritisation scheemes using [PITH_FULL_IMAGE:figures/full_fig_p031_7.png]

**Figure 8.** Figure 8: Comparison of MSE for different prioritisation scheemes using [PITH_FULL_IMAGE:figures/full_fig_p031_8.png]

**Figure 9.** Figure 9: Epistemic uncertainty Eˆ and target uncertainty δ 2 Θ decrease more rapidly for lower noise arm (first column), for UPER compared to other methods. The inclusion of aleatoric uncertainty in the prioritization variable, as utilized in the information gain formula, aims …

**Figure 10.** Figure 10: Comparison of ablated prioritization variables. Median Human Normalized Score for QR-DQN ensembles, where only the prioritization variable is changed. UPER, PER, EPI, and UNI use the information gain in Eq. 11, the TDerror, target epistemic uncertainty in Eq. 13, an…

**Figure 11.** Figure 11: Cumulated training improvement of UPER over PER defined as [PITH_FULL_IMAGE:figures/full_fig_p034_11.png]

**Figure 12.** Figure 12: Cumulated training improvement of UPER over QR-DQN defined as [PITH_FULL_IMAGE:figures/full_fig_p035_12.png]

**Figure 13.** Figure 13: Cumulated training improvement of UPER over QR-PER defined as [PITH_FULL_IMAGE:figures/full_fig_p035_13.png]

**Figure 14.** Figure 14: Cumulated training improvement of UPER over QR-ENS-PER defined as [PITH_FULL_IMAGE:figures/full_fig_p036_14.png]

**Figure 15.** Figure 15: Average performance and corresponding standard deviation for all games across 3 seeds. [PITH_FULL_IMAGE:figures/full_fig_p037_15.png]

**Figure 16.** Figure 16: Performance of an ensemble C51 agent with PER vs ensemble C51 agent with PER for [PITH_FULL_IMAGE:figures/full_fig_p038_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

95 extracted references · 32 canonical work pages

[1]

Query The Agent: Improving sample efficiency through epistemic uncertainty estimation

Julian Alverio, Boris Katz, and Andrei Barbu. Query The Agent : Improving sample efficiency through epistemic uncertainty estimation, October 2022. URL http://arxiv.org/abs/2210.02585. arXiv:2210.02585 [cs]

work page Pith review arXiv 2022
[2]

Hindsight Experience Replay

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight Experience Replay . In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/hash/453fadbd...

2017
[3]

Optimism and pessimism in optimised replay

Georgy Antonov, Christopher Gagne, Eran Eldar, and Peter Dayan. Optimism and pessimism in optimised replay. PLOS Computational Biology, 18 0 (1): 0 e1009634, January 2022. ISSN 1553-7358. doi:10.1371/journal.pcbi.1009634. URL https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1009634. Publisher: Public Library of Science

work page doi:10.1371/journal.pcbi.1009634 2022
[4]

Learning is planning: near Bayes-optimal reinforcement learning via Monte-Carlo tree search

John Asmuth and Michael L Littman. Learning is planning: near bayes-optimal reinforcement learning via monte-carlo tree search. arXiv preprint arXiv:1202.3699, 2012

work page Pith review arXiv 2012
[5]

Using confidence bounds for exploitation-exploration trade-offs

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3 0 (Nov): 0 397--422, 2002 a

2002
[6]

Using confidence bounds for exploitation-exploration trade-offs

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3, 2002 b . ISSN 1533-7928. URL https://www.jmlr.org/papers/v3/auer02a.html

2002
[7]

Never Give Up : Learning Directed Exploration Strategies , February 2020

Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, and Charles Blundell. Never Give Up : Learning Directed Exploration Strategies , February 2020. URL http://arxiv.org/abs/2002.06038. arXiv:2002.06038 [cs, stat]

arXiv 2020
[8]

Emergent Tool Use From Multi - Agent Autocurricula , February 2020

Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent Tool Use From Multi - Agent Autocurricula , February 2020. URL http://arxiv.org/abs/1909.07528. arXiv:1909.07528 [cs, stat]

arXiv 2020

Show all 95 references

[9]

The Effectiveness of Memory Replay in Large Scale Continual Learning , October 2020

Yogesh Balaji, Mehrdad Farajtabar, Dong Yin, Alex Mott, and Ang Li. The Effectiveness of Memory Replay in Large Scale Continual Learning , October 2020. URL http://arxiv.org/abs/2010.02418. arXiv:2010.02418 [cs]

2020 arXiv
[10]

Intrinsic motivation and reinforcement learning

Andrew G Barto. Intrinsic motivation and reinforcement learning. Intrinsically motivated learning in natural and artificial systems, pp.\ 17--47, 2013

2013
[11]

M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade Learning Environment : An Evaluation Platform for General Agents . Journal of Artificial Intelligence Research, 47: 0 253--279, June 2013. ISSN 1076-9757. doi:10.1613/jair.3912. URL https://jair.org/index.php/ja...

2013 doi
[12]

Unifying count-based exploration and intrinsic motivation

Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016 a

2016
[13]

Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos

Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying Count - Based Exploration and Intrinsic Motivation , November 2016 b . URL http://arxiv.org/abs/1606.01868. arXiv:1606.01868 [cs, stat]

2016 arXiv
[14]

A distributional perspective on reinforcement learning

Marc G Bellemare, Will Dabney, and R \'e mi Munos. A distributional perspective on reinforcement learning. In International conference on machine learning, pp.\ 449--458. PMLR, 2017

2017
[15]

Distributional Reinforcement Learning

Marc G Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning . MIT Press, 2023. URL http://www.distributional-rl.org

2023
[16]

Bickel and David A

Peter J. Bickel and David A. Freedman. Some Asymptotic Theory for the Bootstrap . The Annals of Statistics, 9 0 (6): 0 1196--1217, November 1981. ISSN 0090-5364, 2168-8966. doi:10.1214/aos/1176345637. URL https://projecteuclid.org/journals/annals-of-statistics/volume-9/issue-6...

1981
[17]

Wang, Will Dabney, Kevin J

Matthew Botvinick, Jane X. Wang, Will Dabney, Kevin J. Miller, and Zeb Kurth-Nelson. Deep Reinforcement Learning and Its Neuroscientific Implications . Neuron, 107 0 (4): 0 603--616, August 2020. ISSN 0896-6273. doi:10.1016/j.neuron.2020.06.014. URL https://www.sciencedirect.c...

2020 doi
[18]

Bayes-optimal reinforcement learning for discrete uncertainty domains

Emma Brunskill. Bayes-optimal reinforcement learning for discrete uncertainty domains. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 3, pp.\ 1385--1386, 2012

2012
[19]

Exploration by Random Network Distillation , October 2018

Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by Random Network Distillation , October 2018. URL http://arxiv.org/abs/1810.12894. arXiv:1810.12894 [cs, stat]

2018 arXiv
[20]

Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning , June 2022

Bertrand Charpentier, Ransalu Senanayake, Mykel Kochenderfer, and Stephan Günnemann. Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning , June 2022. URL http://arxiv.org/abs/2206.01558. arXiv:2206.01558 [cs]

2022 arXiv
[21]

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models , November 2018

Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models , November 2018. URL http://arxiv.org/abs/1805.12114. arXiv:1805.12114 [cs, stat]

2018 arXiv
[22]

Clements, Bastien Van Delft, Benoît-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth

William R. Clements, Bastien Van Delft, Benoît-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth. Estimating Risk and Uncertainty in Deep Reinforcement Learning , September 2020. URL http://arxiv.org/abs/1905.09638. arXiv:1905.09638 [cs, stat]

2020 arXiv
[23]

Active learning with statistical models

David A Cohn, Zoubin Ghahramani, and Michael I Jordan. Active learning with statistical models. Journal of artificial intelligence research, 4: 0 129--145, 1996

1996
[24]

Bellemare, and Rémi Munos

Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional Reinforcement Learning with Quantile Regression , October 2017. URL http://arxiv.org/abs/1710.10044. arXiv:1710.10044 [cs, stat]

2017 arXiv
[25]

Daw, Yael Niv, and Peter Dayan

Nathaniel D. Daw, Yael Niv, and Peter Dayan. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8 0 (12): 0 1704--1711, December 2005. ISSN 1546-1726. doi:10.1038/nn1560. URL https://www.nature.com/ar...

2005 doi
[26]

Magnetic control of tokamak plasmas through deep reinforcement learning

Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de las Casas, Craig Donner, Leslie Fritz, Cristian Galperti, Andrea Huber, James Keeling, Maria Tsimpoukelli, Jackie Kay, An...

2022
[27]

Revisiting fundamentals of experience replay

William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In Proceedings of the 37th International Conference on Machine Learning , ICML '20, pp.\ 3061--3071. JMLR.org, July 2020

2020
[28]

Foster and Matthew A

David J. Foster and Matthew A. Wilson. Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature, 440 0 (7084): 0 680--683, March 2006. ISSN 1476-4687. doi:10.1038/nature04587. URL https://www.nature.com/articles/nature04587. Number: 708...

2006 doi
[29]

Dropout as a bayesian approximation: Representing model uncertainty in deep learning

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp.\ 1050--1059. PMLR, 2016

2016
[30]

Dopamine, inference, and uncertainty

Samuel J Gershman. Dopamine, inference, and uncertainty. Neural Computation, 29 0 (12): 0 3311--3326, 2017

2017
[31]

Econometric analysis 4th edition

William H Greene. Econometric analysis 4th edition. International edition, New Jersey: Prentice Hall, pp.\ 201--215, 2000

2000
[32]

Grewe, and João Sacramento

Christian Henning, Maria Cervera, Francesco D' Angelo, Johannes von Oswald, Regina Traber, Benjamin Ehret, Seijin Kobayashi, Benjamin F. Grewe, and João Sacramento. Posterior Meta - Replay for Continual Learning . In Advances in Neural Information Processing Systems , volume 3...

2021
[33]

Rainbow: Combining Improvements in Deep Reinforcement Learning , October 2017

Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining Improvements in Deep Reinforcement Learning , October 2017. URL http://arxiv.org/abs/1710.02298. arXiv:1710.02298 [cs]

2017 arXiv
[34]

Meta reinforcement learning as task inference

Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. arXiv preprint arXiv:1905.06424, 2019

1905 arXiv
[35]

Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods

Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning, 110 0 (3): 0 457--506, March 2021. ISSN 1573-0565. doi:10.1007/s10994-021-05946-3. URL https://doi.org/10.1007/s10994-021-05946-3

2021 doi
[36]

Zico Kolter, and Roberta Raileanu

Yiding Jiang, J. Zico Kolter, and Roberta Raileanu. On the Importance of Exploration for Generalization in Reinforcement Learning , June 2023. URL http://arxiv.org/abs/2306.05483. arXiv:2306.05483 [cs]

2023 arXiv
[37]

Uncertainty- Aware Reinforcement Learning for Collision Avoidance , February 2017

Gregory Kahn, Adam Villaflor, Vitchyr Pong, Pieter Abbeel, and Sergey Levine. Uncertainty- Aware Reinforcement Learning for Collision Avoidance , February 2017. URL http://arxiv.org/abs/1702.01182. arXiv:1702.01182 [cs]

2017 arXiv
[38]

Continual Reinforcement Learning with Multi - Timescale Replay , April 2020

Christos Kaplanis, Claudia Clopath, and Murray Shanahan. Continual Reinforcement Learning with Multi - Timescale Replay , April 2020. URL http://arxiv.org/abs/2004.07530. arXiv:2004.07530 [cs, stat]

2020 arXiv
[39]

Towards Continual Reinforcement Learning : A Review and Perspectives , November 2022

Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards Continual Reinforcement Learning : A Review and Perspectives , November 2022. URL http://arxiv.org/abs/2012.13490. arXiv:2012.13490 [cs]

2022 arXiv
[40]

Roger Koenker and Kevin F. Hallock. Quantile Regression . Journal of Economic Perspectives, 15 0 (4): 0 143--156, December 2001. ISSN 0895-3309. doi:10.1257/jep.15.4.143. URL https://www.aeaweb.org/articles?id=10.1257/jep.15.4.143

2001 doi
[41]

DEUP : Direct Epistemic Uncertainty Prediction

Salem Lahlou, Moksh Jain, Hadi Nekoei, Victor Butoi, Paul Bertin, Jarrid Rector-Brooks, Maksym Korablyov, and Yoshua Bengio. DEUP : Direct Epistemic Uncertainty Prediction . Technical Report arXiv:2102.08501, arXiv, April 2022. URL http://arxiv.org/abs/2102.08501. arXiv:2102.0...

2022 arXiv
[42]

Bandit Algorithms

Tor Lattimore and Csaba Szepesvári. Bandit Algorithms . Cambridge University Press, Cambridge, 2020. ISBN 978-1-108-48682-8. doi:10.1017/9781108571401. URL https://www.cambridge.org/core/books/bandit-algorithms/8E39FD004E6CE036680F90DD0C6F09FC

2020 doi
[43]

Continual Learning Using Bayesian Neural Networks

Honglin Li, Payam Barnaghi, Shirin Enshaeifar, and Frieder Ganz. Continual Learning Using Bayesian Neural Networks . IEEE Transactions on Neural Networks and Learning Systems, 32 0 (9): 0 4243--4252, September 2021. ISSN 2162-2388. doi:10.1109/TNNLS.2020.3017292. URL https://i...

2021
[44]

Self-improving reactive agents based on reinforcement learning, planning and teaching

Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 8 0 (3): 0 293--321, May 1992. ISSN 1573-0565. doi:10.1007/BF00992699. URL https://doi.org/10.1007/BF00992699

1992 doi
[45]

Distributional reinforcement learning with epistemic and aleatoric uncertainty estimation

Qi Liu, Yanjie Li, Shiyu Chen, Ke Lin, Xiongtao Shi, and Yunjiang Lou. Distributional reinforcement learning with epistemic and aleatoric uncertainty estimation. Information Sciences, 644: 0 119217, October 2023. ISSN 0020-0255. doi:10.1016/j.ins.2023.119217. URL https://www.s...

2023
[46]

The Effects of Memory Replay in Reinforcement Learning , October 2017

Ruishan Liu and James Zou. The Effects of Memory Replay in Reinforcement Learning , October 2017. URL http://arxiv.org/abs/1710.06574. arXiv:1710.06574 [cs, stat]

2017 arXiv
[47]

Dolan, Zeb Kurth-Nelson, and Timothy E.J

Yunzhe Liu, Raymond J. Dolan, Zeb Kurth-Nelson, and Timothy E.J. Behrens. Human Replay Spontaneously Reorganizes Experience . Cell, 178 0 (3): 0 640--652.e14, July 2019. ISSN 00928674. doi:10.1016/j.cell.2019.06.012. URL https://linkinghub.elsevier.com/retrieve/pii/S0092867419306403

2019 doi
[48]

Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning , June 2023

Sam Lobel, Akhil Bagaria, and George Konidaris. Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning , June 2023. URL http://arxiv.org/abs/2306.03186. arXiv:2306.03186 [cs]

2023 arXiv
[49]

Björn Lütjens, Michael Everett, and Jonathan P. How. Safe Reinforcement Learning with Model Uncertainty Estimates , March 2019. URL http://arxiv.org/abs/1810.08700. arXiv:1810.08700 [cs]

2019 arXiv
[50]

Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation , May 2022

Vincent Mai, Kaustubh Mani, and Liam Paull. Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation , May 2022. URL http://arxiv.org/abs/2201.01666. arXiv:2201.01666 [cs]

2022 arXiv
[51]

Bayesian decision problems and markov chains

James John Martin. Bayesian decision problems and markov chains. (No Title), 1967

1967
[52]

Mattar and Nathaniel D

Marcelo G. Mattar and Nathaniel D. Daw. Prioritized memory access explains planning and hippocampal replay. Nature Neuroscience, 21 0 (11): 0 1609--1617, November 2018. ISSN 1546-1726. doi:10.1038/s41593-018-0232-z. URL https://www.nature.com/articles/s41593-018-0232-z. Number...

2018 doi
[53]

How to Stay Curious while avoiding Noisy TVs using Aleatoric Uncertainty Estimation

Augustine Mavor-Parker, Kimberly Young, Caswell Barry, and Lewis Griffin. How to Stay Curious while avoiding Noisy TVs using Aleatoric Uncertainty Estimation . In Proceedings of the 39th International Conference on Machine Learning , pp.\ 15220--15240. PMLR, June 2022. URL htt...

2022
[54]

McNamara, Álvaro Tejero-Cantero, Stéphanie Trouche, Natalia Campo-Urriza, and David Dupret

Colin G. McNamara, Álvaro Tejero-Cantero, Stéphanie Trouche, Natalia Campo-Urriza, and David Dupret. Dopaminergic neurons promote hippocampal reactivation and spatial memory persistence. Nature Neuroscience, 17 0 (12): 0 1658--1660, December 2014. ISSN 1546-1726. doi:10.1038/n...

2014 doi
[55]

Rusu, Joel Veness, Marc G

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, ...

2015 doi
[56]

Moore and Christopher G

Andrew W. Moore and Christopher G. Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine Learning, 13 0 (1): 0 103--130, October 1993. ISSN 1573-0565. doi:10.1007/BF00993104. URL https://doi.org/10.1007/BF00993104

1993 doi
[57]

Overcoming Exploration in Reinforcement Learning with Demonstrations , February 2018

Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming Exploration in Reinforcement Learning with Demonstrations , February 2018. URL http://arxiv.org/abs/1709.10089. arXiv:1709.10089 [cs]

2018 arXiv
[58]

Collision Probability Matching Loss for Disentangling Epistemic Uncertainty from Aleatoric Uncertainty

Hiromi Narimatsu, Mayuko Ozawa, and Shiro Kumano. Collision Probability Matching Loss for Disentangling Epistemic Uncertainty from Aleatoric Uncertainty . In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , pp.\ 11355--11370. PMLR, A...

2023
[59]

How to measure uncertainty in uncertainty sampling for active learning

Vu-Linh Nguyen, Mohammad Hossein Shaker, and Eyke Hüllermeier. How to measure uncertainty in uncertainty sampling for active learning. Machine Learning, 111 0 (1): 0 89--122, January 2022. ISSN 1573-0565. doi:10.1007/s10994-021-06003-9. URL https://doi.org/10.1007/s10994-021-06003-9

2022 doi
[60]

A review On reinforcement learning: Introduction and applications in industrial process control

Rui Nian, Jinfeng Liu, and Biao Huang. A review On reinforcement learning: Introduction and applications in industrial process control. Computers & Chemical Engineering, 139: 0 106886, August 2020. ISSN 0098-1354. doi:10.1016/j.compchemeng.2020.106886. URL https://www.scienced...

2020
[61]

Information- Directed Exploration for Deep Reinforcement Learning , March 2019

Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause. Information- Directed Exploration for Deep Reinforcement Learning , March 2019. URL http://arxiv.org/abs/1812.07544. arXiv:1812.07544 [cs, stat]

2019 arXiv
[62]

Efficient Exploration via Epistemic - Risk - Seeking Policy Optimization , June 2023

Brendan O'Donoghue. Efficient Exploration via Epistemic - Risk - Seeking Policy Optimization , June 2023. URL http://arxiv.org/abs/2302.09339. arXiv:2302.09339 [cs]

2023 arXiv
[63]

Dota 2 with Large Scale Deep Reinforcement Learning , December 2019

OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, and others . Dota 2 with Large Scale Deep Reinforcement Learning , December 2019. URL http://arxiv.org/abs/1912.06680. arXiv:1912.06680 [cs, stat]

2019 arXiv
[64]

Deep Exploration via Bootstrapped DQN , July 2016

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep Exploration via Bootstrapped DQN , July 2016. URL http://arxiv.org/abs/1602.04621. arXiv:1602.04621 [cs, stat]

2016 arXiv
[65]

Randomized prior functions for deep reinforcement learning

Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018

2018
[66]

Epistemic neural networks

Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Epistemic neural networks. arXiv preprint arXiv:2107.08924, 2021

2021 arXiv
[67]

Count-based exploration with neural density models

Georg Ostrovski, Marc G Bellemare, A \"a ron Oord, and R \'e mi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp.\ 2721--2730. PMLR, 2017 a

2017
[68]

Bellemare, Aaron van den Oord, and Remi Munos

Georg Ostrovski, Marc G. Bellemare, Aaron van den Oord, and Remi Munos. Count- Based Exploration with Neural Density Models , June 2017 b . URL http://arxiv.org/abs/1703.01310. arXiv:1703.01310 [cs]

2017 arXiv
[69]

What is intrinsic motivation? A typology of computational approaches

Pierre-Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? A typology of computational approaches. Frontiers in Neurorobotics, 1, 2007. ISSN 1662-5218. URL https://www.frontiersin.org/articles/10.3389/neuro.12.006.2007

2007 doi
[70]

Understanding and mitigating the limitations of prioritized experience replay

Yangchen Pan, Jincheng Mei, Amir-massoud Farahmand, Martha White, Hengshuai Yao, Mohsen Rohani, and Jun Luo. Understanding and mitigating the limitations of prioritized experience replay. In Proceedings of the Thirty - Eighth Conference on Uncertainty in Artificial Intelligenc...

2022
[71]

Efros, and Trevor Darrell

Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven Exploration by Self -supervised Prediction , May 2017. URL http://arxiv.org/abs/1705.05363. arXiv:1705.05363 [cs, stat]

2017 arXiv
[72]

L. A. Prashanth and Mohammad Ghavamzadeh. Variance-constrained actor-critic algorithms for discounted and average reward MDPs . Machine Learning, 105 0 (3): 0 367--417, December 2016. ISSN 1573-0565. doi:10.1007/s10994-016-5569-5. URL https://doi.org/10.1007/s10994-016-5569-5

2016 doi
[73]

Episodic Curiosity through Reachability , August 2019

Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, and Sylvain Gelly. Episodic Curiosity through Reachability , August 2019. URL http://arxiv.org/abs/1810.02274. arXiv:1810.02274 [cs, stat]

2019 arXiv
[74]

Prioritized Experience Replay , February 2016

Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Replay , February 2016. URL http://arxiv.org/abs/1511.05952. arXiv:1511.05952 [cs]

2016 arXiv
[75]

Comparing Direct and Indirect Temporal - Difference Methods for Estimating the Variance of the Return

Craig Sherstan, Dylan R Ashley, Brendan Bennett, Kenny Young, Adam White, Martha White, and Richard S Sutton. Comparing Direct and Indirect Temporal - Difference Methods for Estimating the Variance of the Return . 2018

2018
[76]

David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madele...

2016 doi
[77]

Matthew J. Sobel. The Variance of Discounted Markov Decision Processes . Journal of Applied Probability, 19 0 (4): 0 794--802, 1982. ISSN 0021-9002. doi:10.2307/3213832. URL https://www.jstor.org/stable/3213832. Publisher: Applied Probability Trust

1982
[78]

Stadie, Sergey Levine, and Pieter Abbeel

Bradly C. Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models , November 2015. URL http://arxiv.org/abs/1507.00814. arXiv:1507.00814 [cs, stat]

2015 arXiv
[79]

Reinforcement learning and its connections with neuroscience and psychology

Ajay Subramanian, Sharad Chitlangia, and Veeky Baths. Reinforcement learning and its connections with neuroscience and psychology. Neural Networks, 145: 0 271--287, January 2022. ISSN 0893-6080. doi:10.1016/j.neunet.2021.10.003. URL https://www.sciencedirect.com/science/articl...

2022 doi
[80]

Attentive Experience Replay

Peiquan Sun, Wengang Zhou, and Houqiang Li. Attentive Experience Replay . Proceedings of the AAAI Conference on Artificial Intelligence, 34 0 (04): 0 5900--5907, April 2020. ISSN 2374-3468. doi:10.1609/aaai.v34i04.6049. URL https://ojs.aaai.org/index.php/AAAI/article/view/6049...

2020 doi
[81]

Reinforcement learning: An Introduction

Richard Sutton and Andrew G Barto. Reinforcement learning: An Introduction . MIT Press, 2018

2018
[82]

Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2 0 (4): 0 160--163, July 1991. ISSN 0163-5719. doi:10.1145/122344.122377. URL https://dl.acm.org/doi/10.1145/122344.122377

1991
[83]

Policy gradients with variance related risk criteria

Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. In Proceedings of the 29th International Coference on International Conference on Machine Learning , ICML '12, pp.\ 1651--1658, Madison, WI, USA, June 2012. Omnipress. ISBN 978-...

2012
[84]

Learning the Variance of the Reward - To - Go

Aviv Tamar, Dotan Di Castro, and Shie Mannor. Learning the Variance of the Reward - To - Go . Journal of Machine Learning Research, 17 0 (13): 0 1--36, 2016. ISSN 1533-7928. URL http://jmlr.org/papers/v17/14-335.html

2016
[85]

\# exploration: A study of count-based exploration for deep reinforcement learning

Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. \# exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017

2017
[86]

Open- Ended Learning Leads to Generally Capable Agents , July 2021

Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, Nat McAleese, Nathalie Bradley-Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes-Fitt, Valentin...

2021 arXiv
[87]

Bradley Efron Tibshirani, R. J. An Introduction to the Bootstrap . Chapman and Hall/CRC, New York, May 1994. ISBN 978-0-429-24659-3. doi:10.1201/9780429246593

1994 doi
[88]

Q-learning

Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8: 0 279--292, 1992

1992
[89]

A Review of Reinforcement Learning for Controlling Building Energy Systems From a Computer Science Perspective

David Weinberg, Qian Wang, Thomas Ohlson Timoudas, and Carlo Fischione. A Review of Reinforcement Learning for Controlling Building Energy Systems From a Computer Science Perspective . Sustainable Cities and Society, 89: 0 104351, February 2023. ISSN 2210-6707. doi:10.1016/j.s...

2023
[90]

An introduction to the kalman filter

Greg Welch, Gary Bishop, et al. An introduction to the kalman filter. 1995

1995
[91]

A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning , October 2016

Martha White and Adam White. A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning , October 2016. URL http://arxiv.org/abs/1607.00446. arXiv:1607.00446 [cs, stat]

2016 arXiv
[92]

Minimum excess risk in bayesian learning

Aolin Xu and Maxim Raginsky. Minimum excess risk in bayesian learning. IEEE Transactions on Information Theory, 68 0 (12): 0 7935--7955, 2022

2022
[93]

Experience Replay Optimization

Daochen Zha, Kwei-Herng Lai, Kaixiong Zhou, and Xia Hu. Experience Replay Optimization . pp.\ 4243--4249, 2019. URL https://www.ijcai.org/proceedings/2019/589

2019
[94]

A survey on epistemic (model) uncertainty in supervised learning: Recent advances and applications

Xinlei Zhou, Han Liu, Farhad Pourpanah, Tieyong Zeng, and Xizhao Wang. A survey on epistemic (model) uncertainty in supervised learning: Recent advances and applications. Neurocomputing, 489: 0 449--465, June 2022. ISSN 0925-2312. doi:10.1016/j.neucom.2021.10.119. URL https://...

2022 doi
[95]

Varibad: A very good method for bayes-adaptive deep rl via meta-learning

Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. arXiv preprint arXiv:1910.08348, 2019

1910 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Query The Agent: Improving sample efficiency through epistemic uncertainty estimation

Julian Alverio, Boris Katz, and Andrei Barbu. Query The Agent : Improving sample efficiency through epistemic uncertainty estimation, October 2022. URL http://arxiv.org/abs/2210.02585. arXiv:2210.02585 [cs]

work page Pith review arXiv 2022

[2] [2]

Hindsight Experience Replay

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight Experience Replay . In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/hash/453fadbd...

2017

[3] [3]

Optimism and pessimism in optimised replay

Georgy Antonov, Christopher Gagne, Eran Eldar, and Peter Dayan. Optimism and pessimism in optimised replay. PLOS Computational Biology, 18 0 (1): 0 e1009634, January 2022. ISSN 1553-7358. doi:10.1371/journal.pcbi.1009634. URL https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1009634. Publisher: Public Library of Science

work page doi:10.1371/journal.pcbi.1009634 2022

[4] [4]

Learning is planning: near Bayes-optimal reinforcement learning via Monte-Carlo tree search

John Asmuth and Michael L Littman. Learning is planning: near bayes-optimal reinforcement learning via monte-carlo tree search. arXiv preprint arXiv:1202.3699, 2012

work page Pith review arXiv 2012

[5] [5]

Using confidence bounds for exploitation-exploration trade-offs

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3 0 (Nov): 0 397--422, 2002 a

2002

[6] [6]

Using confidence bounds for exploitation-exploration trade-offs

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3, 2002 b . ISSN 1533-7928. URL https://www.jmlr.org/papers/v3/auer02a.html

2002

[7] [7]

Never Give Up : Learning Directed Exploration Strategies , February 2020

Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, and Charles Blundell. Never Give Up : Learning Directed Exploration Strategies , February 2020. URL http://arxiv.org/abs/2002.06038. arXiv:2002.06038 [cs, stat]

arXiv 2020

[8] [8]

Emergent Tool Use From Multi - Agent Autocurricula , February 2020

Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent Tool Use From Multi - Agent Autocurricula , February 2020. URL http://arxiv.org/abs/1909.07528. arXiv:1909.07528 [cs, stat]

arXiv 2020