Pith. sign in

REVIEW 3 major objections 4 minor 67 references

Episodic Novelty Through Temporal Distance

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that episodic novelty is best measured by learned temporal distance between states, not by visitation counts or latent-space similarity, and that this replacement roughly doubles sample efficiency on the hardest…

desk verdict A solid empirical method paper that transplants successor distance into episodic intrinsic rewards; the experiments are extensive and the results look real, but the theory appendix has a fixable proof error that needs attention before the temporal-distance story can be taken at face value. read the letter →

arxiv 2501.15418 v1 pith:K5VDY32D submitted 2025-01-26 cs.LG cs.AI

Yuhua Jiang , Qihan Liu , Yiqin Yang , Xiaoteng Ma , Dianyu Zhong , Hao Hu , Jun Yang , Bin Liang
show 3 more authors
Bo Xu Chongjie Zhang Qianchuan Zhao
This is my paper · ORCID
classification cs.LGcs.AI
keywords episodicintrinsicmotivationcontextualMDPsparse-rewardexplorationtemporaldistancesuccessorcontrastivelearningquasimetricsampleefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

In contextual MDPs, where every episode is a new environment, exploration bonuses built from cross-episode experience are unreliable. The authors argue that the right novelty signal is temporal: how many on-policy environment steps separate a newly visited state from states already seen in the current episode. They propose ETD, which gives each state an intrinsic reward equal to the minimum learned temporal distance to any earlier state in the episode, and they claim this outperforms count-based bonuses (which collapse when all states are unique) and similarity-based bonuses (which lack a true metric). A sympathetic reader would care because sparse-reward procedural environments are a standing obstacle for reinforcement learning agents.

What carries the argument

The load-bearing object is the successor-distance quasimetric and its contrastive estimator. Successor distance $d^\pi_{\mathrm{SD}}(x,y)$ is the log-ratio of the discounted probability of reaching $y$ from $y$ versus from $x$, equivalently $-\log \mathbb{E}\left[\gamma^{H^\pi(x,y)}\right]$ for the hitting time $H$, which gives it the structure of a quasimetric: nonnegative, zero exactly at identity, and satisfying the triangle inequality. ETD estimates it with a symmetrized InfoNCE loss in which each positive pair is a state and its geometrically distributed future state, and the energy is $f_{\phi,\psi}(x,y) = c_\psi(y) - d_\phi(x,y)$; the potential $c_\psi$ absorbs state-marginal effects, so the remaining $d_\phi$ recovers the true distance. A metric residual network parameterizes $d_\phi$ to keep it asymmetric, the episodic memory stores CNN embeddings, and the intrinsic reward is the minimum over that memory at each step.

What would settle it

Run ETD in a tabular CMDP where exact on-policy successor distances are computable, and periodically compare the learned $d_\phi$ with the true $d^\pi_{SD}$; if the learned distance ceases to track the true value as the policy shifts while ETD's advantage over count-based baselines persists, then the temporal-distance story is not what is doing the work, whereas if performance drops whenever the estimate degrades, the claim is supported.

Watch

Extended reading notes

Core claim

ETD's central claim is that the right notion of novelty within an episode is temporal: how many environment steps, under the current policy, separate a newly visited state from the states already seen in that episode. The paper defines the bonus as $b_t = \min_{k \in [0,t)} d_\phi(s_k, s_t)$, with $d_\phi$ a learned estimate of the successor distance $d^\pi_{\mathrm{SD}}(x,y) = \log\left(p^\pi_\gamma(s_f=y\mid s_0=y) \big/ p^\pi_\gamma(s_f=y\mid s_0=x)\right)$, a quasimetric that satisfies positivity, identity, and the triangle inequality. The estimate comes from minimizing a symmetrized InfoNCE loss over pairs $(x_i, y_i)$ in which $y_i$ is a geometrically discounted future state of $x_i$, with the energy function parameterized as a potential network minus a quasimetric network so that the optimal $d_\phi$ equals the true successor distance. The paper claims this bonus outperforms count-based and existing similarity-based episodic bonuses on eight MiniGrid tasks, retains its usefulness under Gaussian state noise, and transfers to pixel-based Crafter and MiniWorld mazes.

Load-bearing premise

The method's advantage rests on the contrastively learned quasimetric $d_\phi$ being a faithful estimate of the true on-policy successor distance while the policy and observation distribution keep changing, and the paper validates that estimator only on a small maze toy.

Editorial extensions

If this is right

  • ETD reaches near-optimal performance on ObstructedMaze-Full within 20 million environment steps, roughly doubling the sample efficiency of NovelD, the strongest count-based baseline.
  • Under Gaussian state noise, count-based NovelD collapses while ETD keeps improving, because temporal distance remains meaningful when every state is unique.
  • Swapping the distance learner for inverse-dynamics or discriminator representations degrades performance on state-rich mazes, indicating that the metric itself, not just episodic memory, drives exploration.
  • On high-dimensional pixel-based Crafter and MiniWorld tasks, ETD matches or beats DEIR, NovelD, and PPO, showing the temporal-distance bonus scales beyond grid-state inputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test the paper does not run: on a tabular CMDP with exactly computable successor distances, compare ETD using learned $d_\phi$ against ETD using the true $d^\pi_{SD}$ and against count-based baselines, which would separate estimation error from bonus design.
  • Because the bonus is purely episodic, the paper itself notes it may be weaker in singleton MDPs; combining the same temporal-distance measure with a global entropy bonus is an explicit open direction that could extend the method's reach.
  • The minimum aggregation is aggressive by design; in stochastic or perceptually noisy environments, a soft-min or capped nearest-neighbor variant could be more stable, which is a testable extension rather than a paper claim.
  • In non-ergodic or terminating CMDPs, successor distances can be infinite, so an unstated consequence is that ETD would need a regularized or capped distance to remain usable in settings with absorbing states.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. ETD is an episodic intrinsic-reward algorithm for sparse-reward contextual MDPs. The paper defines state similarity via the on-policy successor distance d_SD(x,y) = log( p_gamma(sf=y|s0=y) / p_gamma(sf=y|s0=x) ), learns a quasimetric d_phi using symmetric InfoNCE with the energy f(x,y)=c_psi(y)-d_phi(x,y) and an MRN parametrization, and issues the intrinsic bonus b_t = min_{k<t} d_phi(s_k,s_t) within each episode. Experiments on MiniGrid, noisy MiniGrid, Crafter, and MiniWorld claim consistent gains over count-based, similarity-based, and global-bonus baselines; ablations compare representation learners, aggregate functions, symmetric/asymmetric distances, and energy/loss choices. The appendices contain theoretical properties of successor distance, continuous-control experiments, and an admitted partial failure to replicate the DEIR baseline in ObstructedMaze-Full.

Significance. The empirical payload is substantial: eight MiniGrid maps, two pixel-based domains, noise-robustness experiments, and several ablations, all with code released and hyperparameters reported. The noisy-state results and the ObstructedMaze-Full result are the most convincing evidence that a temporal-distance bonus can beat strong episodic baselines. If the theoretical bridge were fully proven, the paper would offer a clean explanation for these gains: temporal distance is invariant to irrelevant observation noise and behaves as a metric. The main gap is that Proposition 3, the only formal link between the learned d_phi and the successor distance, is not proved as written, and the DEIR comparison is weakened by the replication admission. These are local, fixable problems rather than indications that the empirical method is unsound.

major comments (3)
  1. [Appendix B, Proposition 3 and Eq. (11)] The proof of Proposition 3 is algebraically invalid as written. Substituting the two bracketed terms of Eq. (11) into Eq. (4) gives f*(x,y) = log(C p_sf(y) p_gamma(y|x) / p_gamma(y|y)^2), not log(p_gamma(y|x)/(C p_sf(y))). The subsequent line 'log(p_gamma(y|y)/p_gamma(y|x)) = f*(y,y) - f*(y,x)' is also incorrect, because the second argument of f is the future state: f*(y,y)-f*(y,x) = c(y)-d(y,y) - (c(x)-d(y,x)), which is not d(x,y) in general; the correct identity is d(x,y) = f*(y,y) - f*(x,y). The proof also does not address the redundancy in the decomposition f = c(y) - d(x,y), where a state-dependent shift added to both c and d leaves f unchanged, so the uniqueness claim is under-specified. Please replace the derivation with a correct proof or with an explicit citation to the existing proof in Myers et al. (2024).
  2. [Section 5.1 and Appendix E.2] The statement that the authors 'adopted their original hyperparameters but couldn't fully replicate their ObstructedMaze-Full performance' means that the DEIR curves in Figure 5 are from a potentially weaker reproduction, so the claim that ETD 'outperforms all other methods' in that environment is not directly supported by the figure as presented. Please report the original DEIR numbers alongside the reproduced ones, or quantify the gap and state explicitly which curve appears in Figure 5.
  3. [Section 4.1 and Appendix C.3] The only evidence that d_phi actually equals the on-policy successor distance is visual inspection of Figures 2 and 17. The text in Appendix C.3 says the learned distance 'remains very close to ground truth,' but no quantitative error or ranking metric is given, and the ground truth is shown for only one maze. Given that Proposition 3 is currently unproven, please add a quantitative comparison (e.g., mean absolute error or rank correlation against an exact d_SD computed on the small mazes, at least under the data-collection policy).
minor comments (4)
  1. [Appendix B, Proposition 2] The triangle-inequality proof contains two typos: the hitting-time triangle inequality requires H(z,y), not H(y,z), and the final expression should be d_SD(x,y) = f(H(x,y)), not f(H(x,z)).
  2. [Algorithm 1] The bonus b_{t+1} is defined as min over k in [0,t+1), but at that point s_{t+1} is not yet in episodic memory; the range should be k in [0,t] or the memory update should occur before the min operation.
  3. [Throughout] There are several typos: 'Minigird-DooKey' in Figure 1, 'ObsturctedMaze' in Tables 2-5, 'erengy' in the Figure 3 caption, 'availabe' in footnote 1, and 'the the expected accumulated reward' in Section 2.
  4. [Figures 5, 6, 10, 11] The learning curves show means over 5 seeds without variance or confidence bands; adding error bars or a statement that they are omitted for readability would make the comparisons easier to assess.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the learned temporal distance is validated against an independently defined successor distance and external benchmarks.

full rationale

The derivation chain is self-contained rather than circular: the successor distance d_SD is defined independently in Eq. 2 as log(pγ(y|y)/pγ(y|x)), the quasimetric properties are proved in Propositions 1-2, and Proposition 3 claims that the InfoNCE loss of Eq. 3 with the potential-minus-quasimetric parameterization of Eq. 4 has a unique minimizer whose quasimetric component equals d_SD. The learned dφ is therefore presented as an estimator of an independently defined quantity, not as a fit parameter renamed as a prediction. The episodic bonus b_t = min_k dφ(s_k,s_t) of Eq. 5 is then compared against external baselines (NovelD, DEIR, E3B, EC, NGU, RND, Count, PPO) on MiniGrid, Crafter, and MiniWorld, and the learned distance is checked against ground truth in the SpiralMaze toy (Fig. 2 and Appendix C.3). No load-bearing step depends on a self-citation: the successor-distance construction is cited to Myers et al. 2024, with no author overlap, and the InfoNCE uniqueness result is cited to Ma & Collins 2018 and Poole et al. 2019. The only self-citation is a related-work pointer (Jiang et al. 2024) and is not load-bearing. For completeness, I note that Appendix B's Proposition 3 (Eq. 11 and the following f*(y,y)-f*(y,x) identity) contains apparent sign/index errors, so the claimed equivalence between the learned quasimetric and d_SD is not established as written; however, this is a correctness and verification gap, not a circularity, because the claimed equivalence is not true by construction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests mainly on the contrastive learning objective recovering the successor distance (borrowed from CMD), a specific reward aggregation choice, and several untested assumptions about ergodicity and diagonal identifiability. No new physical or conceptual entities are invented.

free parameters (2)
  • intrinsic reward coefficient beta = 1e-2 for MiniGrid, 1e-2 for Crafter and MiniWorld, with per-task variation (Tables 2-11)
    The bonus weight beta is tuned per environment and directly controls the contribution of the learned temporal distance to the reward; the method's performance depends on it.
  • discount factor gamma = 0.99
    gamma defines the temporal distance via the discounted state occupancy measure and the geometric distribution for sampling positive pairs; it is set to a standard RL value but shapes the distance scale.
assumptions (5)
  • domain assumption Ergodicity of the CMDP: every state is reachable from every other state under the current policy, so successor distances are finite.
    The paper flags in Appendix A that the successor distance may be infinite in non-ergodic settings, and Proposition 1 relies on finite hitting times.
  • standard math The contrastive InfoNCE objective converges to the optimal energy function f*(x,y)=log(p_gamma(y|x)/(C p_sf(y))) when the batch size is large enough.
    Proposition 3 relies on the known consistency result for InfoNCE (Ma & Collins 2018; Poole et al. 2019), and the paper states 'If the batch size is large enough'.
  • domain assumption The MRN parameterization (potential network minus quasimetric network) can exactly represent the successor distance in the function class, and the learned solution satisfies d_phi(y,y)=0.
    The uniqueness claim in Proposition 3 requires the diagonal condition d(y,y)=0, which the proof does not establish; the network architecture is assumed capable of representing the target.
  • domain assumption The policy pi is fixed when defining the successor distance, but the algorithm updates pi during training; the distance is assumed to track the current policy's occupancy without destabilizing the reward signal.
    Algorithm 1 updates f and pi jointly, while Proposition 1 is a statement about a fixed pi. The paper does not analyze this non-stationarity.
  • standard math Hitting times satisfy the triangle inequality H(x,z) <= H(x,y)+H(y,z), and the log-moment generating function is subadditive.
    Proposition 2's triangle inequality proof uses Hunter (2005) Lemma 4.1 and a concavity argument; the proof as written is garbled, but the underlying statements are standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Episodic Novelty Through Temporal Distance." pith.science (2026). https://pith.science/paper/K5VDY32D

@misc{pith2026250115418,
  author       = {Pith},
  title        = {Pith review of: Episodic Novelty Through Temporal Distance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K5VDY32D}},
  note         = {Machine review of arXiv:2501.15418}
}
read the original abstract

Exploration in sparse reward environments remains a significant challenge in reinforcement learning, particularly in Contextual Markov Decision Processes (CMDPs), where environments differ across episodes. Existing episodic intrinsic motivation methods for CMDPs primarily rely on count-based approaches, which are ineffective in large state spaces, or on similarity-based methods that lack appropriate metrics for state comparison. To address these shortcomings, we propose Episodic Novelty Through Temporal Distance (ETD), a novel approach that introduces temporal distance as a robust metric for state similarity and intrinsic reward computation. By employing contrastive learning, ETD accurately estimates temporal distances and derives intrinsic rewards based on the novelty of states within the current episode. Extensive experiments on various benchmark tasks demonstrate that ETD significantly outperforms state-of-the-art methods, highlighting its effectiveness in enhancing exploration in sparse reward CMDPs.

Figures

Figures reproduced from arXiv: 2501.15418 by the authors.

Figure 2
Figure 2. Distance from to all other states in a 17x17 SpiralMaze. Darker colors indicate greater distance. (Left) Euclidean distance of embeddings trained by inverse dynamics. (Center) Likelihood estimation of easy transitions (EC). (Right) The learned temporal distance (Ours). (NovelD-noise). In contrast, our proposed method, ETD, maintained strong performance even with the added noise, demonstrating its robustness where No… view at source ↗
Figure 3
Figure 3. Overview of ETD. ETD encourages visits to temporally distant states from episodic memory. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Rendering of the environments used in this work. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (15 more)
Figure 5
Figure 5. Figure 5: Training performance of ETD and the baselines on 8 most challenging Minigrid environ [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Training performance on Minigrid with noise environments. The x-axis represents the [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Ablation of representation learning. Representation Learning To further illustrate the effectiveness of temporal distance as an intrinsic re￾ward, we compare the ETD with the Euclidean dis￾tance within both inverse dynamics and discriminator representation learning con…
Figure 8
Figure 8. Figure 8: Ablation of aggregate function 0.0 0.2 0.4 0.6 0.8 1.0 1e6 0.0 0.2 0.4 0.6 0.8 1.0 Average Episode Reward DoorKey-16x16 0.0 0.2 0.4 0.6 0.8 1.0 1e7 0.0 0.2 0.4 0.6 0.8 ObstructedMaze-1Q Symmetric Asymmetric [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 10
Figure 10. Figure 10: Training performance of ETD and the baselines on MiniWorld Maze with different sizes. [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 11
Figure 11. Figure 11: Evaluating ETD and the baselines on Crafter. [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: DMC Experiments. All results averaged over 5 seeds. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: MetaWorld Experiments. All results averaged over 5 seeds. [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: HalfcheetahVelSparse Experiments. All results averaged over 5 seeds. [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Ablations of Energy Function and Contrastive Loss in 17x17 SpiralMaze-Noisy. [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: Ablations of Energy Function and Contrastive Loss in MiniGrid-ObstructedMaze-1Q. [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: More Complex Examples 20 [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: Rendering of MiniGrid Environments used in this work. [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: Rendering of MiniWorld Environments in this work. [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]
Figure 20
Figure 20. Figure 20: MRN Network E.2 HYPERPARAMETERS We found that applying batch normalization to all non-RNN layers could significantly boost the learning speed, especially in environments with stable observations, a finding also noted in the DEIR paper. We use Adam optimizer with ϵ = 1…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 43 canonical work pages

  1. [1]

    An evaluation study of intrinsic motivation techniques applied to reinforcement learning over hard exploration environments

    Alain Andres, Esther Villar-Rodriguez, and Javier Del Ser. An evaluation study of intrinsic motivation techniques applied to reinforcement learning over hard exploration environments. In Andreas Holzinger, Peter Kieseberg, A. Min Tjoa, and Edgar Weippl (eds.), Machine Learning and Knowledge Extraction, pp.\ 201--220, Cham, 2022. Springer International Pub...

    work page 2022

  2. [2]

    Never give up: Learning directed exploration strategies

    Adri \`a Puigdom \`e nech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Mart \' n Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038, 2020

    arXiv 2002

  3. [3]

    TLDR : Unsupervised goal-conditioned RL via temporal distance-aware representations

    Junik Bae, Kwanyoung Park, and Youngwoon Lee. TLDR : Unsupervised goal-conditioned RL via temporal distance-aware representations. In 8th Annual Conference on Robot Learning, 2024. URL https://openreview.net/forum?id=deywgeWmL5

    work page 2024

  4. [4]

    M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 0 253–279, June 2013. ISSN 1076-9757. doi:10.1613/jair.3912. URL http://dx.doi.org/10.1613/jair.3912

    doi:10.1613/jair.3912 2013

  5. [5]

    Unifying count-based exploration and intrinsic motivation

    Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016

    work page 2016

  6. [6]

    Model-free episodic control

    Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint arXiv:1606.04460, 2016

    arXiv 2016

  7. [7]

    Accelerating goal-conditioned rl algorithms and research, 2024

    Michał Bortkiewicz, Władek Pałucki, Vivek Myers, Tadeusz Dziarmaga, Tomasz Arczewski, Łukasz Kuciński, and Benjamin Eysenbach. Accelerating goal-conditioned rl algorithms and research, 2024. URL https://arxiv.org/abs/2408.11052

    arXiv 2024

  8. [8]

    Exploration by random network distillation

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018

    arXiv 2018

Show all 67 references
  1. [9]

    Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks

    Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306.13831, 2023

    2023 arXiv

  2. [10]

    Leveraging procedural generation to benchmark reinforcement learning, 2020

    Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning, 2020. URL https://arxiv.org/abs/1912.01588

    2020 arXiv

  3. [11]

    Adversarial intrinsic motivation for reinforcement learning

    Ishan Durugkar, Mauricio Tec, Scott Niekum, and Peter Stone. Adversarial intrinsic motivation for reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 8622--8636, 2021

    2021

  4. [12]

    Contrastive learning as goal-conditioned reinforcement learning

    Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Ruslan Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL...

    2022

  5. [13]

    Adversarially guided actor-critic

    Yannis Flet-Berliac, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist. Adversarially guided actor-critic. arXiv preprint arXiv:2102.04376, 2021

    2021 arXiv

  6. [14]

    Go beyond imagination: maximizing episodic reachability with world models

    Yao Fu, Run Peng, and Honglak Lee. Go beyond imagination: maximizing episodic reachability with world models. In International Conference on Machine Learning, pp.\ 10405--10420. PMLR, 2023

    2023

  7. [15]

    Threedworld: A platform for interactive multi-modal physical simulation

    Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, et al. Threedworld: A platform for interactive multi-modal physical simulation. arXiv preprint arXiv:2007.04954, 2020

    2007 arXiv

  8. [16]

    Diego Gomez, Michael Bowling, and Marlos C. Machado. Proper laplacian representation learning, 2024. URL https://arxiv.org/abs/2310.10833

    2024 arXiv

  9. [17]

    Benchmarking the spectrum of agent capabilities

    Danijar Hafner. Benchmarking the spectrum of agent capabilities. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=1W0z96MFEoH

    2022

  10. [18]

    Dynamical distance learning for semi-supervised and unsupervised skill discovery

    Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. arXiv preprint arXiv:1907.08225, 2019

    1907 arXiv

  11. [19]

    Exploration via elliptical episodic bonuses

    Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rockt \"a schel. Exploration via elliptical episodic bonuses. Advances in Neural Information Processing Systems, 35: 0 37631--37646, 2022

    2022

  12. [20]

    A study of global and episodic bonuses for exploration in contextual mdps

    Mikael Henaff, Minqi Jiang, and Roberta Raileanu. A study of global and episodic bonuses for exploration in contextual mdps. In International Conference on Machine Learning, pp.\ 12972--12999. PMLR, 2023

    2023

  13. [21]

    Stationary distributions and mean first passage times of perturbed markov chains

    Jeffrey J Hunter. Stationary distributions and mean first passage times of perturbed markov chains. Linear Algebra and its Applications, 410: 0 217--243, 2005

    2005

  14. [22]

    Logarithmic triangle inequality

    ictibones. Logarithmic triangle inequality. Math Stack Exchange, 2017. URL https://math.stackexchange.com/questions/396529

    2017

  15. [23]

    Learning diverse risk preferences in population-based self-play

    Yuhua Jiang, Qihan Liu, Xiaoteng Ma, Chenghao Li, Yiqin Yang, Jun Yang, Bin Liang, and Qianchuan Zhao. Learning diverse risk preferences in population-based self-play. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 12910--12918, 2024

    2024

  16. [24]

    Learning to achieve goals

    Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, volume 2, pp.\ 1094--8. Citeseer, 1993

    1993

  17. [25]

    Martin Klissarov and Marlos C. Machado. Deep Laplacian-based Options for Temporally-Extended Exploration . In Proceedings of the 40th International Conference on Machine Learning , pp.\ 17198--17217. PMLR, July 2023

    2023

  18. [26]

    Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel

    Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment, 2020. URL https://arxiv.org/abs/2006.13760

    2020 arXiv

  19. [27]

    igibson 2.0: Object-centric simulation for robot learning of everyday household tasks

    Chengshu Li, Fei Xia, Roberto Mart \' n-Mart \' n, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, et al. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. arXiv preprint arXiv:2108.03...

    2021 arXiv

  20. [28]

    Metric residual network for sample efficient goal-conditioned reinforcement learning

    Bo Liu, Yihao Feng, Qiang Liu, and Peter Stone. Metric residual network for sample efficient goal-conditioned reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 8799--8806, 2023

    2023

  21. [29]

    Behavior from the void: Unsupervised active pre-training

    Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34: 0 18459--18473, 2021

    2021

  22. [30]

    Flipping coins to estimate pseudocounts for exploration in reinforcement learning

    Sam Lobel, Akhil Bagaria, and George Konidaris. Flipping coins to estimate pseudocounts for exploration in reinforcement learning. In International Conference on Machine Learning, pp.\ 22594--22613. PMLR, 2023

    2023

  23. [31]

    Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency

    Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. arXiv preprint arXiv:1809.01812, 2018

    2018 arXiv

  24. [32]

    Count-based exploration with the successor representation

    Marlos C Machado, Marc G Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 5125--5133, 2020

    2020

  25. [33]

    Count-based exploration in feature space for reinforcement learning

    Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Marcus Hutter. Count-based exploration in feature space for reinforcement learning. arXiv preprint arXiv:1706.08090, 2017

    2017 arXiv

  26. [34]

    Jean-Arcady Meyer and Stewart W. Wilson. A Possibility for Implementing Curiosity and Boredom in Model-Building Neural Controllers, pp.\ 222--227. 1991

    1991

  27. [35]

    Improving intrinsic exploration with language abstractions

    Jesse Mu, Victor Zhong, Roberta Raileanu, Minqi Jiang, Noah Goodman, Tim Rockt \"a schel, and Edward Grefenstette. Improving intrinsic exploration with language abstractions. Advances in Neural Information Processing Systems, 35: 0 33947--33960, 2022

    2022

  28. [36]

    Learning Temporal Distances : Contrastive Successor Features Can Provide a Metric Structure for Decision-Making

    Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, and Benjamin Eysenbach. Learning Temporal Distances : Contrastive Successor Features Can Provide a Metric Structure for Decision-Making . In International Conference on Machine Learning , 2024

    2024

  29. [37]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  30. [38]

    Count-based exploration with neural density models

    Georg Ostrovski, Marc G Bellemare, A \"a ron Oord, and R \'e mi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp.\ 2721--2730. PMLR, 2017

    2017

  31. [39]

    What is intrinsic motivation? a typology of computational approaches

    Pierre-Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics, 1: 0 108, 2007

    2007

  32. [40]

    Intrinsic motivation systems for autonomous mental development

    Pierre-Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for autonomous mental development. IEEE transactions on evolutionary computation, 11 0 (2): 0 265--286, 2007

    2007

  33. [41]

    Interesting object, curious agent: Learning task-agnostic exploration

    Simone Parisi, Victoria Dean, Deepak Pathak, and Abhinav Gupta. Interesting object, curious agent: Learning task-agnostic exploration. Advances in Neural Information Processing Systems, 34: 0 20516--20530, 2021

    2021

  34. [42]

    METRA : Scalable unsupervised RL with metric-aware abstraction

    Seohong Park, Oleh Rybkin, and Sergey Levine. METRA : Scalable unsupervised RL with metric-aware abstraction. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=c5pwL0Soay

    2024

  35. [43]

    Curiosity-driven exploration by self-supervised prediction

    Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp.\ 2778--2787. PMLR, 2017

    2017

  36. [44]

    On variational bounds of mutual information

    Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp.\ 5171--5180. PMLR, 2019

    2019

  37. [45]

    Neural episodic control

    Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In International conference on machine learning, pp.\ 2827--2836. PMLR, 2017

    2017

  38. [46]

    Ride: Rewarding impact-driven exploration for procedurally-generated environments

    Roberta Raileanu and Tim Rockt \"a schel. Ride: Rewarding impact-driven exploration for procedurally-generated environments. arXiv preprint arXiv:2002.12292, 2020

    2002 arXiv

  39. [47]

    Exploring through random curiosity with general value functions

    Aditya Ramesh, Louis Kirsch, Sjoerd van Steenkiste, and J \"u rgen Schmidhuber. Exploring through random curiosity with general value functions. Advances in Neural Information Processing Systems, 35: 0 18733--18748, 2022

    2022

  40. [48]

    Episodic curiosity through reachability

    Nikolay Savinov, Anton Raichuk, Rapha \"e l Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. arXiv preprint arXiv:1810.02274, 2018

    2018 arXiv

  41. [49]

    Habitat: A platform for embodied ai research, 2019

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A platform for embodied ai research, 2019. URL https://arxiv.org/abs/1904.01201

    2019 arXiv

  42. [50]

    State entropy maximization with random encoders for efficient exploration

    Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, pp.\ 9443--9454. PMLR, 2021

    2021

  43. [51]

    Time-contrastive networks: Self-supervised learning from video

    Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pp.\ 1134--1141. IEEE, 2018

    2018

  44. [52]

    Deepmind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  45. [53]

    Deir: efficient and robust exploration through discriminative-model-based episodic intrinsic rewards

    Shanchuan Wan, Yujin Tang, Yingtao Tian, and Tomoyuki Kaneko. Deir: efficient and robust exploration through discriminative-model-based episodic intrinsic rewards. arXiv preprint arXiv:2304.10770, 2023

    2023 arXiv

  46. [54]

    Towards better laplacian representation in reinforcement learning with generalized graph drawing

    Kaixin Wang, Kuangqi Zhou, Qixin Zhang, Jie Shao, Bryan Hooi, and Jiashi Feng. Towards better laplacian representation in reinforcement learning with generalized graph drawing. In International Conference on Machine Learning, pp.\ 11003--11012. PMLR, 2021

    2021

  47. [55]

    Reachability-aware laplacian representation in reinforcement learning

    Kaixin Wang, Kuangqi Zhou, Jiashi Feng, Bryan Hooi, and Xinchao Wang. Reachability-aware laplacian representation in reinforcement learning. arXiv preprint arXiv:2210.13153, 2022

    2022 arXiv

  48. [56]

    Optimal goal-reaching reinforcement learning via quasimetric learning

    Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learning, pp.\ 36411--36430. PMLR, 2023

    2023

  49. [57]

    The laplacian in RL : Learning representations with efficient approximations

    Yifan Wu, George Tucker, and Ofir Nachum. The laplacian in RL : Learning representations with efficient approximations. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=HJlNpoA5YQ

    2019

  50. [58]

    Sapien: A simulated part-based interactive environment

    Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 1109...

    2020

  51. [59]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp.\ 1094--1100. PMLR, 2020

    2020

  52. [60]

    Rank the episodes: A simple approach for exploration in procedurally-generated environments

    Daochen Zha, Wenye Ma, Lei Yuan, Xia Hu, and Ji Liu. Rank the episodes: A simple approach for exploration in procedurally-generated environments. arXiv preprint arXiv:2101.08152, 2021

    2021 arXiv

  53. [61]

    Made: Exploration via maximizing deviation from explored regions

    Tianjun Zhang, Paria Rashidinejad, Jiantao Jiao, Yuandong Tian, Joseph E Gonzalez, and Stuart Russell. Made: Exploration via maximizing deviation from explored regions. Advances in Neural Information Processing Systems, 34: 0 9663--9680, 2021 a

    2021

  54. [62]

    Noveld: A simple yet effective exploration criterion

    Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems, 34: 0 25217--25230, 2021 b

    2021

  55. [63]

    Explore to generalize in zero-shot RL

    Ev Zisselman, Itai Lavie, Daniel Soudry, and Aviv Tamar. Explore to generalize in zero-shot RL . In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=37cADkATD0

    2023

  56. [64]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  57. [65]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  58. [66]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  59. [67]

    Software

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.