Pith. sign in

REVIEW 4 major objections 5 minor 74 references

Efficient Skill Discovery via Regret-Aware Optimization

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Regret-aware skill discovery claims faster, more diverse exploration by proposing exactly the skills the agent has not yet mastered.

desk verdict RSD is a sensible extension of regret-based curriculum learning to skill discovery, but the 15% zero-shot claim is confounded by an unfair test-query protocol. read the letter →

arxiv 2506.21044 v1 pith:WTCL3SM7 submitted 2025-06-26 cs.LG cs.AI

He Zhang , Ming Zhou , Shaopeng Zhai , Ying Sun , Hui Xiong This is my paper
classification cs.LGcs.AI
keywords unsupervisedskilldiscoveryregret-awareoptimizationmin-maxadversariallearningtemporalrepresentationdiversityzero-shotgoalreachingreinforcementexplorationcurriculum
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Unsupervised skill discovery normally samples a fixed, uniform distribution of skills and tries to make all of them distinguishable, which wastes effort on behaviors the agent has already mastered or cannot yet attempt. This paper claims that sampling should instead be adversarial: a skill generator proposes skills whose value functions improved the most between consecutive training stages, so exploration is repeatedly pushed toward the frontier of what the agent can learn next. The result, the authors argue, is a method that reaches higher state coverage in fewer interaction steps, produces more diverse skills in asymmetric maze-like environments, and transfers better zero-shot to goal-reaching tasks. The headline numbers are a 0.507 versus 0.435 zero-shot success rate in Antmaze-large and a 15% claimed improvement over existing methods in high-dimensional settings.

What carries the argument

The load-bearing object is the regret proxy Reg_k(z) = $V^{{π^k_{θ1}}$}(s0|z) - $V^{{π^{k-1}}$_{θ1}}(s0|z), the difference in value between the latest and previous agent policy for a given skill; the skill generator maximizes it while the agent policy minimizes it, which is what makes the optimization min-max. Around that proxy, the method relies on a tanh-bounded temporal representation that lets non-unit skill vectors encode magnitude, a renormalized skill direction to keep inner-product scales stable, and a population of Gaussian skill generators whose worst member is periodically replaced. Two regularizers matter: a KL term that keeps the new generator distribution distinct from the current population, and a proximity term that keeps it near representations of states actually seen, counteracting the acknowledged bias of the regret estimate for unseen skills.

What would settle it

Rerun the Antmaze-large zero-shot evaluation with the regret term in the skill-generator objective replaced by a constant (or by random noise), keeping the population, regularizers, and bounded representation unchanged; if the 0.507 success rate does not fall toward the 0.435 METRA-d baseline, the regret mechanism is not doing the claimed work.

Watch

Extended reading notes

Core claim

The central claim is that skill discovery and policy learning form a min-max game, not a one-way diversity maximization. RSD defines the regret of a skill z at stage k as Reg_k(z)=$V^{{π^k_{θ1}}$}(s0|z)-$V^{{π^{k-1}}$_{θ1}}(s0|z), the improvement in the skill-conditioned value from the previous policy to the current one; the agent policy minimizes regret by mastering the skills that are proposed, while a trainable skill generator maximizes it by proposing skills that are still far from converged. To make this work in high dimension, RSD learns a tanh-bounded temporal representation so skill vectors can carry magnitude information, uses a non-unit updated skill direction to stabilize the inner-product reward scale, and keeps a fixed-size population of skill generators with KL and proximity regularizers so that new skills stay both distinct from old ones and close to the frontier of already seen states. The authors claim this combination improves learning efficiency and diversity over mutual-information baselines, with the largest gains in skill-asymmetric environments such as mazes.

Load-bearing premise

The load-bearing premise is that the improvement in the agent's measured value for a skill is a faithful sign of how much that skill still needs practice, and the paper concedes this measurement is biased for skills the agent has never seen.

Editorial extensions

If this is right

  • A fixed interaction budget should cover more of a complex map when skills are sampled by regret rather than uniformly, because the agent spends its effort on under-converged skills.
  • In skill-asymmetric environments, the learned skill set should be more diverse and more transferable to downstream goal-reaching, where the paper reports 0.507 versus 0.435 success on Antmaze-large.
  • Representing skills with non-unit vectors in a bounded representation space should let the agent distinguish similar-looking behaviors that stop at different depths behind obstacles.
  • Maintaining a population of skill generators rather than a single one should reduce skill forgetting and keep skill diversity stable across training stages.
  • The method should achieve these gains without extra intrinsic exploration rewards, since the regret signal itself redirects sampling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If regret accurately tracks under-convergence, the same sampling principle should transfer to hierarchical RL or goal-conditioned RL, where the value difference between successive policies is equally observable; the paper only hints at this direction.
  • A reader should expect the advantage to shrink in perfectly symmetric environments, since uniform sampling is already optimal there; the paper's own Ant results show this pattern, but the causal link is editorial inference.
  • One testable extension is to replace the value-difference regret with a bootstrap or ensemble uncertainty estimate; if the gains survive, the method does not depend on the specific proxy.
  • The mechanism also predicts that noisier value estimates, for example under sparse rewards or long horizons, will degrade the benefit unless the regret estimator is made more stable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Regret-aware Skill Discovery (RSD), an unsupervised skill discovery method that alternates between training an agent policy to master currently proposed skills and training a skill-generator policy to propose new skills that maximize a regret proxy, defined as the improvement in the agent's own value function between consecutive learning stages. The method maintains a population of skill generators to avoid forgetting, uses a bounded temporal representation space with non-unit skill vectors, and is evaluated on Ant, Maze2d-large, Antmaze-medium, and Antmaze-large, with additional Kitchen experiments in the appendix. The central claims are improved learning efficiency and skill diversity, and a roughly 15% zero-shot improvement over baselines in high-dimensional environments, with the headline result being Antmaze-large success rate 0.507 for RSD versus 0.435 for METRA-d.

Significance. If the stated claims hold, RSD would be a practical upgrade over METRA and related temporal-representation methods: the regret-as-progress signal is a conceptually appealing bridge between auto-curriculum learning and unsupervised skill discovery, and the population-of-generators mechanism addresses a real forgetting concern. The paper is also open-source, and the Kitchen experiments include multi-seed statistics, which is a strength. However, the significance is currently conditional: the zero-shot evaluation is confounded by asymmetric skill-vector construction, the central regret proxy is not validated, and the main experimental tables and figures lack multi-seed error bars. These issues must be resolved before the efficiency and zero-shot claims can be accepted.

major comments (4)
  1. [4.3, Table 1] The zero-shot evaluation is not a controlled comparison because the skill query is constructed differently for RSD and for the baselines. For the baselines, z_g is the unit vector of the difference between representation-space vectors of the goal and the start, while for RSD, z_g is the non-unit vector representation of s_g. Since RSD's intrinsic reward (Eq. 13) and its skill space both exploit the magnitude of z, the non-unit query carries goal-distance information that the unit-vector baseline query does not. The headline improvement (Antmaze-large AR 0.507 vs 0.435 for METRA-d, a 16.6% relative gain) may therefore reflect the richer test-time query rather than better learned skills. I ask the authors to evaluate all methods with an identical z-construction rule, or to justify the differing rules as the intended deployment protocol and provide results for both variants.
  2. [3.1, Eq. 7] The regret proxy is defined as the improvement in the agent's own value function between consecutive stages, Reg_k(z) = V^{pi_k_theta1}(s0 | z) - V^{pi_{k-1}_theta1}(s0 | z). This is not the 'discrepancy between the actual policy strength and the maximum strength' stated in the introduction, and it is not validated as a measure of underconvergence. The paper acknowledges in Section 3.3 that the estimate is biased for unseen z, but it provides no ground-truth regret comparison, no analysis of the bias, and no ablation that isolates the regret mechanism (for example, comparing against uniform skill sampling or against an alternative progress signal). Because Eq. 14 and Eq. 18 make this proxy the sole driver of skill generation, the missing validation is load-bearing for the efficiency and zero-shot claims.
  3. [Figure 3, Table 1] The main empirical claims are reported without error bars or seed variance. Figure 3 shows coverage curves as single lines and Table 1 reports single FD/AR numbers for each method, while only the Kitchen experiments in Appendix F report mean ± std over three seeds. Without multi-seed statistics, the observed improvements in CoverCoords and in the zero-shot AR values cannot be distinguished from run-to-run noise. Please report mean ± std across at least three to five seeds for Figure 3 and Table 1, with an appropriate significance statement.
  4. [3.2, Eq. 13, Appendix C] The method assumes that the agent policy 'eventually arrives at the vector z in the representation space' and that z can be any arbitrary vector in a bounded space, but the precise domain of z and the degree to which trajectories actually reach z are not specified or measured. Since Eq. 13 uses the distance ||z - phi(s_t)|| as a reward and the zero-shot evaluation feeds non-unit z_g, the authors should state the domain of z, explain how the Gaussian skill generator maps to that domain, and provide a quantitative check that skills realize their z (for example, the distribution of ||phi(s_T) - z|| at the end of skill-conditioned rollouts). Without this information, the interpretation of the skill space and of the zero-shot queries remains ambiguous.
minor comments (5)
  1. [3.1, Eq. 6] The value function in Eq. 6 includes the SAC entropy term -alpha log pi, while Eq. 1 defines V^pi as the expected sum of intrinsic rewards; please clarify how these two definitions relate and whether the entropy-adjusted value is the intended quantity in the regret definition of Eq. 7.
  2. [3.2, Eq. 11] The per-timestep constraint C_phi = min(epsilon, 1/T - ||phi(s_{t+1}) - phi(s_t)||) is optimized as a soft penalty via the dual variable lambda, so the statement that the total displacement bound in Eq. 11 'must hold' is stronger than what the procedure actually enforces; please describe the constraint handling more precisely.
  3. [Appendix C.1] The hyperparameter table is provided only for Maze2d-large; since the main results include Antmaze-medium and Antmaze-large, please provide the full hyperparameter settings for all environments or state explicitly which values are shared across environments.
  4. [Figure 3] The figure captions do not identify which curve corresponds to each baseline; please add a legend or an explicit description of the curve colors and line styles so that the reader can identify RSD relative to METRA, LSD, DIAYN, and DADS.
  5. [Abstract, Section 4.3] The abstract states '15% zero-shot improvement', but Table 1 shows a 16.6% relative improvement in Antmaze-large and a 3.7% relative improvement in Maze2d-large; please specify the environment or aggregation to which the 15% figure refers.

Circularity Check

1 steps flagged · score 4.0 of 10

Regret is defined as the agent's own value improvement, making the core convergence signal self-referential; the zero-shot tests remain independent downstream evidence.

  1. self definitional [Section 3.1 Eq. (7), Section 3.3 Eq. (14), Section 4.2 Fig. 7]
    "We assume that the regret at learning stage k can be calculated as follows: Regk = V k − V k−1 ... the larger the regret, the more under-converged strength the skill possesses. ... This trend also serves as evidence that our algorithm is capable of achieving convergence."

    Eq. (7) defines regret as the stage-to-stage increase of the agent's own value function under RSD's learned reward, and Eq. (14) trains the skill generator to maximize exactly this quantity. Therefore 'larger regret means more under-converged strength' is true by definition: a large V^k − V^{k−1} merely says the value was lower at the previous stage. The convergence evidence in Fig. 7 then reports that this same self-defined quantity decays to zero, which restates the training signal rather than independently validating convergence. The paper notes the estimate is biased for unseen z but gives no comparison to ground-truth regret or any external measure of policy-strength convergence.

full rationale

The central learning signal of RSD is self-referential: regret is not an external quantity but the improvement of the agent's own value function, and the skill generator maximizes that exact quantity. Consequently, the conceptual claim that high-regret skills are underconverged, and the convergence evidence in Fig. 7, are restatements of the definition of the training signal rather than independent validation. This is a genuine but partial circularity: the zero-shot navigation results in Table 1 are an independent downstream test, so the headline empirical claim is not wholly determined by the regret definition. The paper is otherwise self-contained against external baselines (METRA, LSD, DIAYN, DADS), and I found no load-bearing self-citation or imported uniqueness theorem. The zero-shot query asymmetry (RSD uses z_g = φ(s_g) while baselines use a unit vector) is a confounding protocol choice and a correctness risk, but it is not a circular derivation, so I do not count it as a circular step.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. The free parameters are standard hyperparameters tuned on the evaluation environments, and the axioms are domain-specific assumptions about skill learning and regret estimation that are not independently validated outside the paper.

free parameters (6)
  • alpha_1 (diversity regularizer weight) = 5
    Tuned in Appendix E; controls trade-off between regret maximization and diversity of the skill population.
  • alpha_2 (proximity regularizer weight) = 1
    Tuned in Appendix E; prevents the skill generator from straying too far from seen state representations.
  • l (maximum population size) = 15
    Tuned in Appendix E; larger values improve performance with diminishing returns, so 15 is chosen by hand.
  • epsilon (dual slack in C_phi) = 0.001
    Used in the per-timestep displacement constraint (Eq. 12); chosen by hand, no sensitivity analysis.
  • option dimension d = 2
    Latent skill vector dimension; set to 2 to match METRA baselines, but no analysis of higher dimensions in main experiments.
  • steps in each stage = 50
    Scheduling choice for alternating agent policy and skill generator updates; no analysis of sensitivity.
assumptions (5)
  • domain assumption Skills with weak strength should be explored more, and skills with converged strength less (Section 1).
    The core principle of the method; presented intuitively, not derived.
  • ad hoc to paper Reg_k(z) = V^{pi_k}(s0|z) - V^{pi_k-1}(s0|z) approximates the true regret for skill z (Eq. 7).
    No validation against ground-truth regret; the paper acknowledges bias for unseen skills (Section 3.3).
  • domain assumption Skills exhibit sequential dependencies, with basic skills acquired before advanced ones (Sections 1 and C.2).
    Motivates the curriculum-style sampling, but no empirical evidence for this ordering in the tested environments.
  • ad hoc to paper The agent policy eventually arrives at the vector z in the representation space (Section 3.2).
    Used to justify the reward r_phi = ||z - phi(s_t)|| - ||z - phi(s_{t+1})||; not proven or verified.
  • ad hoc to paper The per-timestep constraint 1/T - ||phi(s_{t+1}) - phi(s_t)|| >= 0 guarantees the total displacement bound (Eq. 11).
    A sufficient but not necessary condition for the cumulative Lipschitz constraint; introduces the epsilon hyperparameter.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Skill Discovery via Regret-Aware Optimization." pith.science (2026). https://pith.science/paper/WTCL3SM7

@misc{pith2026250621044,
  author       = {Pith},
  title        = {Pith review of: Efficient Skill Discovery via Regret-Aware Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WTCL3SM7}},
  note         = {Machine review of arXiv:2506.21044}
}
read the original abstract

Unsupervised skill discovery aims to learn diverse and distinguishable behaviors in open-ended reinforcement learning. For existing methods, they focus on improving diversity through pure exploration, mutual information optimization, and learning temporal representation. Despite that they perform well on exploration, they remain limited in terms of efficiency, especially for the high-dimensional situations. In this work, we frame skill discovery as a min-max game of skill generation and policy learning, proposing a regret-aware method on top of temporal representation learning that expands the discovered skill space along the direction of upgradable policy strength. The key insight behind the proposed method is that the skill discovery is adversarial to the policy learning, i.e., skills with weak strength should be further explored while less exploration for the skills with converged strength. As an implementation, we score the degree of strength convergence with regret, and guide the skill discovery with a learnable skill generator. To avoid degeneration, skill generation comes from an up-gradable population of skill generators. We conduct experiments on environments with varying complexities and dimension sizes. Empirical results show that our method outperforms baselines in both efficiency and diversity. Moreover, our method achieves a 15% zero shot improvement in high-dimensional environments, compared to existing methods.

Figures

Figures reproduced from arXiv: 2506.21044 by the authors.

Figure 1
Figure 1. Comparison between uniform skill discovery and regret￾aware skill discovery. Blue lines indicate skills with converged strength (low regret). Red lines indicate skills need further explore (high regret). By re-balancing exploration based on regret signals, the regret-aware method exhibits improved efficiency. learn a basic skill ‘picking up’ before the advanced skill ‘opening the grip’ (Liu et al., 2024). Intuitivel… view at source ↗
Figure 2
Figure 2. Overview of RSD. The leftmost illustrates how the state space is projected onto a bounded temporal representation space. The right side shows learning process of the skill generator: circles denotes skill distributions, dashed regions indicate coverage, the larger the more diverse, solid lines represent trajectories, and flags correspond to skills sampled from these distributions. Collectively, these circles form th… view at source ↗
Figure 3
Figure 3. Comparison between RSD and baselines, represented as red curves in different environments. The x-axis shows timesteps of interaction, while the y-axis represents Unique Coordinates, which measure state coverage achieved through sufficiently sampled skills. (a) RSD. (b) METRA [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Visualization of Dynamical skills learned in Ant envi￾ronment. Trajectories in State Space is shown on the left, and trajectories in Representation Space (Repr. Space) on the right. temporal dynamic skills. The Ant environment contains an ant robot moving in an open ma…
Figure 7
Figure 7. Figure 7: The curves of Regret and Entropy over timesteps in Maze2d-large. Regret is calculated using Eq. 7 after updating the πθ1 . And the Entropy denotes the entropy of the Pz after updating the πθ2 , estimated by a Monte Carlo method. of similar but different skills [PITH_F…
Figure 6
Figure 6. Figure 6: Filled contour plot of the population of skill generators in representation space across different training stages. Color-coded lines depict distinct skill trajectories sampled according to Pz. ous methods within the same number of timesteps. Second, in skill-symmetric…
Figure 8
Figure 8. Figure 8: Examples of the environments Our research encompasses four environments: Ant, maze2d-umaze-dense-v1, antmaze-medium-diverse-v0, and antmaze￾large-diverse-v0, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Goals for zero-shot evaluation [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Visualization of our zero-shot performance [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Hyperparameter Tuning of α1, α2 and maximum length l. We aim to analyze the impact of the hyperparameters α1, α2, and the maximum length l on the performance of our model. In [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Example of Kitchen Environment. F.3. Results at different Skill Dimensions Results at Skill Dimension = 24: Our method significantly outperformed METRA, particularly at later training stages. At 400k steps, our method achieved 5.08±0.45 completed tasks compared to MET…
Figure 13
Figure 13. Figure 13: Examples of the environments Overall, our method consistently demonstrated superior sample efficiency and final performance compared to METRA. Particularly at a skill dimension of 32, the performance gains were increasingly pronounced as training progressed (e.g., at …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

74 extracted references · 57 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    M., Crump, T., and Far, B

    Afsar, M. M., Crump, T., and Far, B. Reinforcement learning based recommender systems: A survey. ACM Computing Surveys, 55 0 (7): 0 1--38, 2022

    work page 2022

  3. [3]

    Hindsight experience replay

    Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay. CoRR, abs/1707.01495, 2017

    arXiv 2017

  4. [4]

    Tldr: Unsupervised goal-conditioned rl via temporal distance-aware representations

    Bae, J., Park, K., and Lee, Y. Tldr: Unsupervised goal-conditioned rl via temporal distance-aware representations. arXiv preprint arXiv:2407.08464, 2024

    arXiv 2024

  5. [5]

    K., and Konidaris, G

    Bagaria, A., Senthil, J. K., and Konidaris, G. Skill discovery for exploration and planning using deep skill graphs. In International Conference on Machine Learning, pp.\ 521--531. PMLR, 2021

    work page 2021

  6. [6]

    Constrained ensemble exploration for unsupervised skill discovery

    Bai, C., Yang, R., Zhang, Q., Xu, K., Chen, Y., Xiao, T., and Li, X. Constrained ensemble exploration for unsupervised skill discovery. In Forty-first International Conference on Machine Learning, 2024

    work page 2024

  7. [7]

    X., Tanner, J., Vuong, Q., Walling, A., Wang, H., and Zhilinsky, U

    Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., Levine, S., Li-Bell, A., Mothukuri, M., Nair, S., Pertsch, K., Shi, L. X., Tanner, J., Vuong, Q., Walling, A., Wang, H., and Zhilinsky, U. _0 : A vision-language-action flow model for general robot control, 2024

    work page 2024

  8. [8]

    Exploration by random network distillation

    Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Exploration by random network distillation. In International Conference on Learning Representations, 2019

    work page 2019

Show all 74 references
  1. [9]

    Explore, discover and learn: Unsupervised discovery of state-covering skills

    Campos, V., Trott, A., Xiong, C., Socher, R., Gir \'o -i Nieto, X., and Torres, J. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pp.\ 1317--1327. PMLR, 2020

    2020

  2. [10]

    Language as a cognitive tool to imagine goals in curiosity driven exploration

    Colas, C., Karch, T., Lair, N., Dussoux, J.-M., Moulin-Frier, C., Dominey, P., and Oudeyer, P.-Y. Language as a cognitive tool to imagine goals in curiosity driven exploration. Advances in Neural Information Processing Systems, 33: 0 3761--3774, 2020

    2020

  3. [11]

    Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: a short survey

    Colas, C., Karch, T., Sigaud, O., and Oudeyer, P.-Y. Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: a short survey. Journal of Artificial Intelligence Research, 74: 0 1159--1199, 2022

    2022

  4. [12]

    Goal-conditioned imitation learning

    Ding, Y., Florensa, C., Abbeel, P., and Phielipp, M. Goal-conditioned imitation learning. Advances in neural information processing systems, 32, 2019

    2019

  5. [13]

    Adversarial intrinsic motivation for reinforcement learning

    Durugkar, I., Tec, M., Niekum, S., and Stone, P. Adversarial intrinsic motivation for reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 8622--8636, 2021

    2021

  6. [14]

    Diversity is all you need: Learning skills without a reward function

    Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019

    2019

  7. [15]

    C-learning: Learning to achieve goals via recursive classification

    Eysenbach, B., Salakhutdinov, R., and Levine, S. C-learning: Learning to achieve goals via recursive classification. arXiv preprint arXiv:2011.08909, 2020

    2011 arXiv

  8. [16]

    Eysenbach, B., Zhang, T., Levine, S., and Salakhutdinov, R. R. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35: 0 35603--35620, 2022

    2022

  9. [17]

    Curriculum-guided hindsight experience replay

    Fang, M., Zhou, T., Du, Y., Han, L., and Zhang, Z. Curriculum-guided hindsight experience replay. Advances in neural information processing systems, 32, 2019

    2019

  10. [18]

    Automatic goal generation for reinforcement learning agents

    Florensa, C., Held, D., Geng, X., and Abbeel, P. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pp.\ 1515--1528. PMLR, 2018

    2018

  11. [19]

    Accuracy-based curriculum learning in deep reinforcement learning

    Fournier, P., Sigaud, O., Chetouani, M., and Oudeyer, P.-Y. Accuracy-based curriculum learning in deep reinforcement learning. arXiv preprint arXiv:1806.09614, 2018

    2018 arXiv

  12. [20]

    D4rl: Datasets for deep data-driven reinforcement learning, 2020

    Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning, 2020

    2020

  13. [21]

    Learning to reach goals via iterated supervised learning

    Ghosh, D., Gupta, A., Reddy, A., Fu, J., Devin, C., Eysenbach, B., and Levine, S. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088, 2019

    1912 arXiv

  14. [22]

    and Oudeyer, P.-Y

    Gottlieb, J. and Oudeyer, P.-Y. Towards a neuroscience of active sampling and curiosity. Nature Reviews Neuroscience, 19 0 (12): 0 758--770, 2018

    2018

  15. [23]

    J., and Wierstra, D

    Gregor, K., Rezende, D. J., and Wierstra, D. Variational intrinsic control, 2016

    2016

  16. [24]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.\ 1861--1870. PMLR, 2018

    2018

  17. [25]

    Haber, N., Mrowca, D., Fei-Fei, L., and Yamins, D. L. Emergence of structured behaviors from curiosity-based intrinsic motivation. arXiv preprint arXiv:1802.07461, 2018

    2018 arXiv

  18. [26]

    Exploration in deep reinforcement learning: From single-agent to multiagent domain

    Hao, J., Yang, T., Tang, H., Bai, C., Liu, J., Meng, Z., Liu, P., and Wang, Z. Exploration in deep reinforcement learning: From single-agent to multiagent domain. IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  19. [27]

    Open-endedness is essential for artificial superhuman intelligence

    Hughes, E., Dennis, M., Parker-Holder, J., Behbahani, F., Mavalankar, A., Shi, Y., Schaul, T., and Rocktaschel, T. Open-endedness is essential for artificial superhuman intelligence. arXiv preprint arXiv:2406.04268, 2024

    2024 arXiv

  20. [28]

    Unsupervised curricula for visual meta-reinforcement learning

    Jabri, A., Hsu, K., Gupta, A., Eysenbach, B., Levine, S., and Finn, C. Unsupervised curricula for visual meta-reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019

    2019

  21. [29]

    A comprehensive survey on self-interpretable neural networks

    Ji, Y., Sun, Y., Zhang, Y., Wang, Z., Zhuang, Y., Gong, Z., Shen, D., Qin, C., Zhu, H., and Xiong, H. A comprehensive survey on self-interpretable neural networks. arXiv preprint arXiv:2501.15638, 2025

    2025

  22. [30]

    N., Grefenstette, E., and Rockt \" a schel, T

    Jiang, M., Dennis, M., Parker - Holder, J., Foerster, J. N., Grefenstette, E., and Rockt \" a schel, T. Replay-guided adversarial environment design. CoRR, abs/2110.02439, 2021 a

    2021 arXiv

  23. [31]

    Prioritized level replay

    Jiang, M., Grefenstette, E., and Rockt \"a schel, T. Prioritized level replay. In International Conference on Machine Learning, pp.\ 4940--4950. PMLR, 2021 b

    2021

  24. [32]

    Kaelbling, L. P. Learning to achieve goals. In IJCAI, volume 2, pp.\ 1094--8. Citeseer, 1993

    1993

  25. [33]

    and Oudeyer, P.-Y

    Kaplan, F. and Oudeyer, P.-Y. In search of the neural circuits of intrinsic motivation. Frontiers in neuroscience, 1: 0 9, 2007

    2007

  26. [34]

    K., Lee, H., Hwang, D., Park, S., Min, K., and Choo, J

    Kim, H., Lee, B. K., Lee, H., Hwang, D., Park, S., Min, K., and Choo, J. Learning to discover skills through guidance. Advances in Neural Information Processing Systems, 36, 2024

    2024

  27. [35]

    Unsupervised skill discovery with bottleneck option learning, 2021

    Kim, J., Park, S., and Kim, G. Unsupervised skill discovery with bottleneck option learning, 2021

    2021

  28. [36]

    Active world model learning with progress curiosity

    Kim, K., Sano, M., De Freitas, J., Haber, N., and Yamins, D. Active world model learning with progress curiosity. In International conference on machine learning, pp.\ 5306--5315. PMLR, 2020

    2020

  29. [37]

    Exploration in deep reinforcement learning: A survey

    Ladosz, P., Weng, L., Kim, M., and Oh, H. Exploration in deep reinforcement learning: A survey. Information Fusion, 85: 0 1--22, 2022. ISSN 1566-2535

    2022

  30. [38]

    B., Yarats, D., Rajeswaran, A., and Abbeel, P

    Laskin, M., Liu, H., Peng, X. B., Yarats, D., Rajeswaran, A., and Abbeel, P. Cic: Contrastive intrinsic control for unsupervised skill discovery, 2022

    2022

  31. [39]

    and Seo, S.-W

    Lee, S.-H. and Seo, S.-W. Unsupervised skill discovery for learning shared structures across changing environments. In International Conference on Machine Learning, pp.\ 19185--19199. PMLR, 2023

    2023

  32. [40]

    Revisiting graph adversarial attack and defense from a data distribution perspective

    Li, K., Liu, Y., Ao, X., and He, Q. Revisiting graph adversarial attack and defense from a data distribution perspective. In The Eleventh International Conference on Learning Representations, 2023

    2023

  33. [41]

    Boosting the adversarial robustness of graph neural networks: An ood perspective

    Li, K., Chen, Y., Liu, Y., Wang, J., He, Q., Cheng, M., and Ao, X. Boosting the adversarial robustness of graph neural networks: An ood perspective. In The Twelfth International Conference on Learning Representations, 2024

    2024

  34. [42]

    A single goal is all you need: Skills and exploration emerge from contrastive rl without rewards, demonstrations, or subgoals

    Liu, G., Tang, M., and Eysenbach, B. A single goal is all you need: Skills and exploration emerge from contrastive rl without rewards, demonstrations, or subgoals. arXiv preprint arXiv:2408.05804, 2024

    2024 arXiv

  35. [43]

    Choreographer: Learning and adapting skills in imagination

    Mazzaglia, P., Verbelen, T., Dhoedt, B., Lacoste, A., and Rajeswar, S. Choreographer: Learning and adapting skills in imagination. In 3rd Offline RL Workshop: Offline RL as a''Launchpad'', 2022

    2022

  36. [44]

    Planning with goal-conditioned policies

    Nasiriany, S., Pong, V., Lin, S., and Levine, S. Planning with goal-conditioned policies. Advances in neural information processing systems, 32, 2019

    2019

  37. [45]

    Wasserstein dependency measure for representation learning

    Ozair, S., Lynch, C., Bengio, Y., Van den Oord, A., Levine, S., and Sermanet, P. Wasserstein dependency measure for representation learning. Advances in Neural Information Processing Systems, 32, 2019

    2019

  38. [46]

    Lipschitz-constrained unsupervised skill discovery

    Park, S., Choi, J., Kim, J., Lee, H., and Kim, G. Lipschitz-constrained unsupervised skill discovery. In International Conference on Learning Representations, 2022

    2022

  39. [47]

    Hiql: Offline goal-conditioned rl with latent states as actions

    Park, S., Ghosh, D., Eysenbach, B., and Levine, S. Hiql: Offline goal-conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 36, 2024 a

    2024

  40. [48]

    Foundation policies with hilbert representations

    Park, S., Kreiman, T., and Levine, S. Foundation policies with hilbert representations. In Forty-first International Conference on Machine Learning, 2024 b

    2024

  41. [49]

    METRA : Scalable unsupervised RL with metric-aware abstraction

    Park, S., Rybkin, O., and Levine, S. METRA : Scalable unsupervised RL with metric-aware abstraction. In The Twelfth International Conference on Learning Representations, 2024 c

    2024

  42. [50]

    Evolving curricula with regret-based environment design

    Parker-Holder, J., Jiang, M., Dennis, M., Samvelyan, M., Foerster, J., Grefenstette, E., and Rockt \"a schel, T. Evolving curricula with regret-based environment design. In International Conference on Machine Learning, pp.\ 17473--17498. PMLR, 2022

    2022

  43. [51]

    A., and Darrell, T

    Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised prediction. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, ...

    2017

  44. [52]

    H., Dalal, M., Lin, S., Nair, A., Bahl, S., and Levine, S

    Pong, V. H., Dalal, M., Lin, S., Nair, A., Bahl, S., and Levine, S. Skew-fit: state-covering self-supervised reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, pp.\ 7783--7792, 2020

    2020

  45. [53]

    Automatic curriculum learning for deep rl: A short survey

    Portelas, R., Colas, C., Weng, L., Hofmann, K., and Oudeyer, P.-Y. Automatic curriculum learning for deep rl: A short survey. arXiv preprint arXiv:2003.04664, 2020

    2003 arXiv

  46. [54]

    Rutherford, A., Beukman, M., Willi, T., Lacerda, B., Hawes, N., and Foerster, J. N. No regrets: Investigating and improving regret approximations for curriculum discovery. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  47. [55]

    Prioritized experience replay

    Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , 2016

    2016

  48. [56]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  49. [57]

    Dynamics-aware unsupervised discovery of skills

    Sharma, A., Gu, S., Levine, S., Kumar, V., and Hausman, K. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020

    2020

  50. [58]

    Deterministic policy gradient algorithms

    Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In International conference on machine learning, pp.\ 387--395. Pmlr, 2014

    2014

  51. [59]

    A general reinforcement learning algorithm that masters chess, shogi, and go through self-play

    Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T., Simonyan, K., and Hassabis, D. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science...

    2018

  52. [60]

    Intrinsic motivation and automatic curricula via asymmetric self-play

    Sukhbaatar, S., Lin, Z., Kostrikov, I., Synnaeve, G., Szlam, A., and Fergus, R. Intrinsic motivation and automatic curricula via asymmetric self-play. In 6th International Conference on Learning Representations, ICLR 2018, 2018

    2018

  53. [61]

    Policy continuation with hindsight inverse dynamics

    Sun, H., Li, Z., Liu, X., Zhou, B., and Lin, D. Policy continuation with hindsight inverse dynamics. Advances in Neural Information Processing Systems, 32, 2019

    2019

  54. [62]

    Hierarchical reinforcement learning for dynamic autonomous vehicle navigation at intelligent intersections

    Sun, Q., Zhang, L., Yu, H., Zhang, W., Mei, Y., and Xiong, H. Hierarchical reinforcement learning for dynamic autonomous vehicle navigation at intelligent intersections. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp.\ 4852--4861, 2023

    2023

  55. [63]

    Market-aware long-term job skill recommendation with explainable deep reinforcement learning

    Sun, Y., Ji, Y., Zhu, H., Zhuang, F., He, Q., and Xiong, H. Market-aware long-term job skill recommendation with explainable deep reinforcement learning. ACM Transactions on Information Systems, 43 0 (2): 0 1--35, 2025

    2025

  56. [64]

    Sutton, R. S. Reinforcement learning: An introduction. A Bradford Book, 2018

    2018

  57. [65]

    S., McAllester, D., Singh, S., and Mansour, Y

    Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999

    1999

  58. [66]

    dm\_control: Software and tasks for continuous control

    Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S., Bohez, S., Merel, J., Erez, T., Lillicrap, T., Heess, N., and Tassa, Y. dm\_control: Software and tasks for continuous control. Software Impacts, 6: 0 100022, 2020. ISSN 2665-9638

    2020

  59. [67]

    M., Mathieu, M., Dudzik, A., Chung, J., Choi, D

    Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575 0 (7782): 0 350--354, 2019

    2019

  60. [68]

    Optimal goal-reaching reinforcement learning via quasimetric learning

    Wang, T., Torralba, A., Isola, P., and Zhang, A. Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learning, pp.\ 36411--36430. PMLR, 2023

    2023

  61. [69]

    Ski LD : Unsupervised skill discovery guided by factor interactions

    Wang, Z., Hu, J., Chuck, C., Chen, S., Mart \' n-Mart \' n, R., Zhang, A., Niekum, S., and Stone, P. Ski LD : Unsupervised skill discovery guided by factor interactions. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 a

    2024

  62. [70]

    A comprehensive survey of forgetting in deep learning beyond continual learning

    Wang, Z., Yang, E., Shen, L., and Huang, H. A comprehensive survey of forgetting in deep learning beyond continual learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024 b

    2024

  63. [71]

    Neural program synthesis by self-learning

    Xu, Y., Dai, L., Singh, U., Zhang, K., and Tu, Z. Neural program synthesis by self-learning. arXiv preprint arXiv:1910.05865, 2019

    1910 arXiv

  64. [72]

    Behavior contrastive learning for unsupervised skill discovery

    Yang, R., Bai, C., Guo, H., Li, S., Zhao, B., Wang, Z., Liu, P., and Li, X. Behavior contrastive learning for unsupervised skill discovery. In International conference on machine learning, pp.\ 39183--39204. PMLR, 2023

    2023

  65. [73]

    Interactive interior design recommendation via coarse-to-fine multimodal reinforcement learning

    Zhang, H., Sun, Y., Guo, W., Liu, Y., Lu, H., Lin, X., and Xiong, H. Interactive interior design recommendation via coarse-to-fine multimodal reinforcement learning. In Proceedings of the 31st ACM International Conference on Multimedia, pp.\ 6472--6480, 2023

    2023

  66. [74]

    Generative learning plan recommendation for employees: A performance-aware reinforcement learning approach

    Zheng, Z., Sun, Y., Song, X., Zhu, H., and Xiong, H. Generative learning plan recommendation for employees: A performance-aware reinforcement learning approach. In Proceedings of the 17th ACM Conference on Recommender Systems, pp.\ 443--454, 2023

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.