Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Improving Environment Novelty Quantification for Effective Unsupervised Environment Design

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A curriculum teacher can score environment novelty from the student's own state-action coverage, and mixing that score with regret-based replay improves zero-shot generalization.

desk verdict A concrete, curriculum-aware novelty metric for UED with consistent empirical gains; the per-level noise question needs a proper analysis, but the paper merits review. read the letter →

arxiv 2502.05726 v1 pith:SA3UIUGV submitted 2025-02-08 cs.LG stat.ML

Jayden Teoh , Wenjun Li , Pradeep Varakantham This is my paper · ORCID
classification cs.LGstat.ML
keywords unsupervisedenvironmentdesignautocurriculanoveltyquantificationGaussianmixturemodelsstate-actioncoveragezero-shotgeneralizationcurriculumlearningreinforcement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a UED teacher can quantify a level's novelty by comparing the state-action pairs the student visits there against the state-action coverage built from recent curriculum levels. It proposes CENIE, which fits a Gaussian Mixture Model to a FIFO buffer of the student's recent state-action pairs and scores each candidate level by the negative mean log-likelihood of the state-action pairs it induces. Mixing this novelty score into the replay prioritization of PLR and ACCEL broadens the student's state-action coverage and improves zero-shot transfer on maze navigation, bipedal walking, and car racing benchmarks. The underlying point is that regret alone can leave the curriculum stuck in a comfort zone, and a curriculum-aware novelty objective is what pushes the agent into under-explored regions.

What carries the argument

The load-bearing mechanism is the Gaussian Mixture Model over the student's state-action coverage, fitted to a FIFO buffer of state-action pairs from the 32 most recent training levels. The model is refitted every rollout, with the number of Gaussian components $K$ selected from a fixed range by the silhouette score, so the density estimate tracks the student's evolving coverage. Candidate levels are ranked by negative mean log-likelihood under this model, and that rank is turned into a replay probability by the same prioritization formula used for regret; the two replay probabilities are linearly combined with coefficient $\alpha = 0.5$ to decide which levels enter the buffer and which are replayed for training.

What would settle it

Run ACCEL-CENIE on a BipedalWalker variant with action-independent broadcast noise added to observations; if the novelty score disproportionately promotes noisy levels that carry no learning potential and zero-shot returns drop relative to regret-only ACCEL, then coverage log-likelihood is not a reliable novelty proxy in the way the paper claims.

Watch

Extended reading notes

Core claim

CENIE's central claim is that environment novelty in UED is a property of the student-level interaction, not of environment parameters: a level is novel when it moves the student into state-action regions that recent curriculum experience has left underrepresented. The paper operationalizes this by fitting a Gaussian Mixture Model to a FIFO buffer $\Gamma$ of state-action pairs from the most recent training levels, then scoring a candidate level $l_\theta$ as $$ \mathrm{NOVELTY}(l_\$\theta$) = -\frac{1}{|X_\$\theta$|}\sum_{t=1}^{T} \log p(x_t \mid \lambda_\Gamma), $$ where $X_\theta$ is the set of state-action pairs on that level and $\lambda_\Gamma$ is the fitted mixture. When this score is converted into a replay probability by rank prioritization and mixed with the regret-based replay probability at equal weight, the resulting ACCEL-CENIE and PLR-CENIE achieve broader state-action coverage and better zero-shot transfer than the regret-only versions: close to 55% of optimal normalized performance in BipedalWalker, and best-or-matched returns on all 20 held-out Formula One tracks in CarRacing.

Load-bearing premise

The load-bearing assumption is that a Gaussian Mixture Model fitted to the 32 most recent training levels gives a faithful picture of the region of state-action space the student has already explored, so that negative log-likelihood reliably ranks genuine novelty and is not dominated by high-dimensional noise or irrelevant observation variation.

Editorial extensions

If this is right

  • Augmenting ACCEL with CENIE's novelty objective broadens the student's state-action coverage in BipedalWalker from 42.5% to 47.6%, and PLR-CENIE broadens it from 43.4% to 55.3%.
  • On the six BipedalWalker test environments, ACCEL-CENIE outperforms ACCEL in five and reaches close to 55% of optimal performance, the best among all benchmarks.
  • In Minigrid, PLR-CENIE nearly matches ACCEL's out-of-distribution solved rate on PerfectMazeLarge, despite PLR using a random level generator instead of ACCEL's mutation editor.
  • In CarRacing, PLR-CENIE outperforms or matches the best baseline on all 20 F1 tracks and achieves the best aggregate IQM and optimality gap, suggesting novelty-driven curricula find high-regret levels rather than sacrificing them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same GMM log-likelihood could be used to score candidate levels before rollout, guiding the generator or mutation operator directly; this would likely make curriculum search more sample-efficient than generating first and filtering by replay score.
  • Feeding the GMM a learned latent state representation, rather than raw observations or recurrent features, could address the noisy-TV limitation the paper concedes, provided the encoder discards action-irrelevant variation.
  • The fixed $\alpha = 0.5$ mixture weight is a free parameter; annealing it from novelty-heavy to regret-heavy over training is a testable extension, since sparse-reward domains may want exploration early and dense-reward domains may want exploitation later.
  • Because the estimator only needs a stream of the student's state-action pairs, it is portable to automatic curriculum learning with predefined task distributions, without requiring a population of agents as EPOET does.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper introduces CENIE, a method for quantifying environment novelty in Unsupervised Environment Design. CENIE fits a Gaussian Mixture Model to a FIFO buffer of the student's recent state-action pairs (Section 4.1, Eq. 2) and scores a candidate level by the negative mean log-likelihood of one trajectory under that GMM (Eq. 3). The novelty score is combined with regret-based replay probabilities through a fixed linear mixture (Eq. 5) inside PLR and ACCEL, yielding PLR-CENIE and ACCEL-CENIE. The paper evaluates these algorithms on Minigrid, BipedalWalker, and CarRacing, reporting improved IQM and optimality gap on held-out levels, ablation studies with novelty-only variants, a difficulty-composition analysis of replayed levels, and a t-SNE-based state-action coverage comparison (Table 1).

Significance. If the central claim holds, CENIE is a useful, domain-agnostic complement to regret that can be added to existing replay-based UED algorithms without pairwise distance computations. The paper's strengths include ablations that isolate the novelty component, zero-shot evaluations on genuinely out-of-distribution levels (PerfectMazeLarge, 20 F1 tracks), a difficulty analysis connecting novelty to harder replayed levels, and an unusually candid limitations discussion (Appendix C). The main risk is that the novelty signal itself is never validated statistically: the score in Eq. (3) is computed from a single stochastic rollout per level, and the paper does not show that between-level novelty differences dominate within-level rollout variance. This concern is load-bearing because the replay-prioritization mechanism is rank-based.

major comments (3)
  1. [Section 4.1, Eq. (3); Algorithms 1 and 2] The novelty score is computed from a single stochastic rollout per candidate level (Algorithm 1 lines 7-8 and 22; Algorithm 2 lines 7-8 and 23), with stop-gradient on the policy. Since the student is a stochastic PPO policy, X_theta is a random draw from pi on l_theta; in sparse-reward Minigrid with LSTM hidden states (Appendix D.3), different seeds can produce qualitatively different coverage. The paper never reports a variance decomposition or repeated-rollout analysis showing that the between-level spread of Eq. (3) exceeds the within-level rollout noise. Because Eq. (4) converts scores into rank-based replay probabilities, large within-level variance would make replay priorities largely noisy, and the observed IQM gains could then be attributed to extra stochastic exploration rather than to a reliable novelty-driven curriculum. Please provide repeated-trajectory estimates on a sample of levels, bootstrap confidence intervals for novelty ranks, or a signal-to-noise ratio analysis to establish that the ranking is stable.
  2. [Appendix C vs. Section 5] Appendix C explicitly concedes that GMM-based density estimation is vulnerable to the curse of dimensionality and the noisy TV problem, yet the main empirical sections rely entirely on the GMM negative log-likelihood as the novelty ground truth. There is no independent validation of the novelty scores (for example, correlation with known environment parameters, human-annotated difficulty, or a learned latent representation). Since the central claim is that CENIE measures environment novelty, the paper should include at least one sanity check that the scores track something other than rollout noise: for instance, compare CENIE rankings against rankings from repeated rollouts or from a domain-defined novelty/difficulty ordering, or test the latent-representation variant suggested in Appendix C.
  3. [Section 5 and Table 3] The main performance claims ('substantial margin', 'state-of-the-art') are based on only 5 independent runs per algorithm, and the paper shows IQM/optimality gap plots without confidence intervals. In Table 3, several F1 tracks have overlapping standard errors across methods (e.g., Australia, Germany, Russia), so the aggregate claims need stratified bootstrap confidence intervals on the IQM and optimality gap differences, especially since the paper is comparing against the original ACCEL and PLR results.
minor comments (6)
  1. [Algorithm 1, line 9] The line labeled 'Compute regret score for l'_theta' should refer to the newly generated candidate level l_theta, not the edited level l'_theta, which is computed later at line 23.
  2. [Appendix D.1] The EM update formulas contain indexing errors: the E-step denominator uses sigma_i where it should use the full covariance Sigma_i, and the M-step expression for mu_i is not written as a proper weighted average (the sum of responsibilities appears in both numerator and denominator without the correct normalization). Please correct these equations.
  3. [Section 4.1, Eqs. (2) and (3)] Equation (2) sums over J samples while Eq. (3) sums over T timesteps; the notation should be unified so that |X_theta| and |Gamma| are used consistently.
  4. [Appendix B] There are two typos: 'UPDOMPs' should be 'UPOMDPs' and 'emcompasses' should be 'encompasses'.
  5. [Figure 7 caption and Section 5.3] The caption says 'The middle image in Figure 6b shows...' but the referenced track visualization appears in Figure 7; please check the cross-references.
  6. [Section 1 and reference [52]] The paper cites its own AAMAS extended abstract as the origin of the framework but does not explicitly itemize what is new beyond that abstract (e.g., the CarRacing experiments, the difficulty composition analysis, the full ablation study). A short sentence clarifying the delta would help reviewers and readers.

Circularity Check

0 steps flagged · score 2.0 of 10

No substantive circularity: the CENIE novelty score is computed online from the student's own trajectories and is evaluated on held-out levels, with no fitting to the test target; the only self-citation is a preliminary-version note that is not load-bearing.

full rationale

The paper's central novelty metric, Eq. (3), is the negative mean log-likelihood of a candidate level's state-action pairs under a GMM fitted to Gamma, a FIFO buffer of recent state-action pairs from the student's own curriculum (Section 4.1, Algorithm 1 lines 16-19). This is an online, policy-dependent statistic rather than a parameter fitted to test performance. Replay probabilities in Eqs. (4)-(5) combine rank-normalized novelty and regret, and all CENIE hyperparameters (alpha = 0.5, window size W = 32, component range [6,15]) are fixed beforehand (Table 7); no held-out test return or solved rate is used to fit them. The empirical claims are evaluated on unseen Minigrid mazes, BipedalWalker variants, and F1 CarRacing tracks, so the state-of-the-art claim is not forced by construction. The feedback loop between the curriculum and the novelty estimate is inherent to UED and is not circular because the novelty model is recomputed from recent trajectories while the claimed outcome is zero-shot transfer on external benchmarks. Appendix C explicitly concedes limitations such as the fixed 0.5-0.5 weighting, the curse of dimensionality, and vulnerability to the noisy TV problem; these are robustness and correctness concerns, not circularity. The only self-citation is [52], a preliminary AAMAS abstract, and Appendix D states that the current work 'builds upon and significantly extends' it. That self-citation is not load-bearing: the GMM fitting procedure, both CENIE algorithms, and all benchmark details are specified in the present paper, with baselines and codebases drawn from external prior work. The paper also distinguishes CENIE from curiosity-driven RL and ALP-GMM rather than renaming them. Overall, no circular step reduces a prediction to its own inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

All free parameters are hyperparameters of the CENIE implementation and are not fitted to the benchmark targets. The axioms are mostly standard for the UED framework. The key unproven premise is that GMM density on a FIFO coverage buffer is a valid, curriculum-aware novelty measure, which is acknowledged as an empirical heuristic.

free parameters (4)
  • α (novelty weight) = 0.5
    Weight balancing novelty and regret replay probabilities in Eq. 5; fixed without tuning or sensitivity analysis across the three domains.
  • FIFO window size W = 32 levels
    Number of recent training levels whose state-action pairs populate Γ; chosen by hand, no ablation reported.
  • GMM component range [Kmin, Kmax] = [6, 15]
    Range of mixture components searched at each update and selected by silhouette score; chosen by hand, no ablation reported.
  • Covariance regularization = 1e-2 (Minigrid), 1e-6 (BipedalWalker), 1e-1 (CarRacing)
    Regularization added to GMM covariance matrices, set per domain to stabilize fitting; listed in Table 7.
assumptions (4)
  • domain assumption Regret (PVL) is a useful measure of learning potential in UED
    The paper builds on PLR⊥ and ACCEL without re-deriving their objectives; this is standard in the UED literature and inherited from prior work.
  • domain assumption State-action coverage is related to sample efficiency and generalization
    Motivates CENIE in Section 4, citing Xie et al. [59]; not proven for the UED setting.
  • standard math GMMs can approximate smooth density distributions with enough components
    Used in Section 4.1 to justify fitting multiple K and selecting by silhouette score.
  • ad hoc to paper The student's policy is sufficiently stationary within the FIFO window for the GMM to be meaningful
    The FIFO buffer (W=32) is meant to mitigate catastrophic forgetting, but the GMM is refit online while the policy changes; no formal justification is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Environment Novelty Quantification for Effective Unsupervised Environment Design." pith.science (2026). https://pith.science/paper/SA3UIUGV

@misc{pith2026250205726,
  author       = {Pith},
  title        = {Pith review of: Improving Environment Novelty Quantification for Effective Unsupervised Environment Design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SA3UIUGV}},
  note         = {Machine review of arXiv:2502.05726}
}
read the original abstract

Unsupervised Environment Design (UED) formalizes the problem of autocurricula through interactive training between a teacher agent and a student agent. The teacher generates new training environments with high learning potential, curating an adaptive curriculum that strengthens the student's ability to handle unseen scenarios. Existing UED methods mainly rely on regret, a metric that measures the difference between the agent's optimal and actual performance, to guide curriculum design. Regret-driven methods generate curricula that progressively increase environment complexity for the student but overlook environment novelty -- a critical element for enhancing an agent's generalizability. Measuring environment novelty is especially challenging due to the underspecified nature of environment parameters in UED, and existing approaches face significant limitations. To address this, this paper introduces the Coverage-based Evaluation of Novelty In Environment (CENIE) framework. CENIE proposes a scalable, domain-agnostic, and curriculum-aware approach to quantifying environment novelty by leveraging the student's state-action space coverage from previous curriculum experiences. We then propose an implementation of CENIE that models this coverage and measures environment novelty using Gaussian Mixture Models. By integrating both regret and novelty as complementary objectives for curriculum design, CENIE facilitates effective exploration across the state-action space while progressively increasing curriculum complexity. Empirical evaluations demonstrate that augmenting existing regret-based UED algorithms with CENIE achieves state-of-the-art performance across multiple benchmarks, underscoring the effectiveness of novelty-driven autocurricula for robust generalization.

Figures

Figures reproduced from arXiv: 2502.05726 by the authors.

Figure 1
Figure 1. An overview of the CENIE framework. The teacher will utilise environment regret and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Zero-shot transfer performance in eight human-designed test environments. The plots are [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. (a) Aggregate zero-shot transfer performance in Minigrid domain across 5 independent [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Student’s generalization performance on 6 BipedalWalker testing environments during [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Difficulty composition of levels replayed by ACCEL and ACCEL-CENIE during training. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: (a) BipedalWalker domain and (b) Aggregate zero-shot transfer performance in Bipedal [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: (a) CarRacing domain and (b) Aggregate zero-shot transfer performance in CarRacing. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Total regret in level replay buffer for PLR⊥ and PLR-CENIE over training in CarRacing [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Zero-shot transfer performances in Minigrid. The plots are based on the median and [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: IQM and Optimality Gap ablations in Minigrid domain. Results are measured across 5 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Levels in the level replay buffer of PLR-CENIE. X-axis: number of student PPO updates. [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Evolution of the state-action space coverage of ACCEL-CENIE, ACCEL, PLR-CENIE, [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Zero-shot transfer return ablations in BipedalWalker domain. The plot is based on mean [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: IQM and Optimality Gap ablations in BipedalWalker domain. Results are measured across [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Complete training process of each algorithm on four CarRacing test environments. Plots [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: IQM and Optimality Gap ablations on the full CarRacing benchmark (20 F1 tracks). [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: Examples of testing levels in BipedalWalker domain. (a) BipedalWalker, (b) Hardcore, (c) [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Simply Environment Scaling: Designing Effective Environment Distributions for Multimodal Agent Learning

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Selecting 30 diverse environments by LLM-annotated ability coverage and training with harness-weakening plus state-scale curriculum improves multimodal agent success over naive scaling.

Reference graph

Works this paper leans on

64 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    Agarwal, M

    R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304–29320, 2021

    work page 2021

  2. [2]

    Akkaya, M

    I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019

    arXiv 1910

  3. [3]

    O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The Interna- tional Journal of Robotics Research, 39(1):3–20, 2020

    work page 2020

  4. [4]

    Arthur, S

    D. Arthur, S. Vassilvitskii, et al. k-means++: The advantages of careful seeding. In Soda, volume 7, pages 1027–1035, 2007

    work page 2007

  5. [5]

    I. Assent. Clustering high dimensional data. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2(4):340–350, 2012

    work page 2012

  6. [6]

    P. Auer, N. Cesa-Bianchi, Y . Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing, 32(1):48–77, 2002. doi: 10.1137/S0097539701398375. URL https://doi.org/10.1137/S0097539701398375

    doi:10.1137/s0097539701398375 2002

  7. [7]

    A. S. Azad, I. Gur, J. Emhoff, N. Alexis, A. Faust, P. Abbeel, and I. Stoica. Clutr: Curriculum learning via unsupervised task representation learning. In International Conference on Machine Learning, pages 1361–1395. PMLR, 2023

    work page 2023

  8. [8]

    M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 1479–1487, Red Hook, NY , USA, 2016. Curran Associates Inc. ISBN 9781510838819

    work page 2016

Show all 64 references
  1. [9]

    Berner, G

    C. Berner, G. Brockman, B. Chan, V . Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019

    1912 arXiv

  2. [10]

    Beukman, S

    M. Beukman, S. Coward, M. Matthews, M. Fellows, M. Jiang, M. Dennis, and J. Foerster. Refining minimax regret for unsupervised environment design. In International Conference on Machine Learning. PMLR, 2024

    2024

  3. [11]

    Blei and M

    D. Blei and M. Jordan. Variational inference for dirichlet process mixtures. Bayesian Analysis, 1, 03 2006. doi: 10.1214/06-BA104

    2006 doi

  4. [12]

    Blömer and K

    J. Blömer and K. Bujna. Simple methods for initializing the em algorithm for gaussian mixture models. CoRR, 2013

    2013

  5. [13]

    Borchert

    O. Borchert. Pycave, 2022. URL https://github.com/borchero/pycave/

    2022

  6. [14]

    Bouveyron, S

    C. Bouveyron, S. Girard, and C. Schmid. High-dimensional data clustering. Computational statistics & data analysis, 52(1):502–519, 2007

    2007

  7. [15]

    Bozdogan

    H. Bozdogan. Model selection and Akaike’s Information Criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52(3):345–370, Sept. 1987. ISSN 1860-0980. doi: 10.1007/BF02294361

    1987 doi

  8. [16]

    Brockman, V

    G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016. 11

    2016 arXiv

  9. [17]

    Bruce, M

    J. Bruce, M. Dennis, A. Edwards, J. Parker-Holder, Y . Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, Y . Aytar, S. Bechtle, F. Behbahani, S. Chan, N. Heess, L. Gonzalez, S. Osindero, S. Ozair, S. Reed, J. Zhang, K. Zolna, J. Clune, N. de Freitas, S. Singh, an...

    2024 arXiv

  10. [18]

    Burda, H

    Y . Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355, 2018

    2018 arXiv

  11. [19]

    Burda, H

    Y . Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation,

  12. [20]

    Chevalier-Boisvert, B

    M. Chevalier-Boisvert, B. Dai, M. Towers, R. de Lazcano, L. Willems, S. Lahlou, S. Pal, P. S. Castro, and J. Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306.13831, 2023

    2023 arXiv

  13. [21]

    Cobbe, O

    K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning, pages 1282–1289. PMLR, 2019

    2019

  14. [22]

    A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological) , 39(1): 1–22, 1977

    1977

  15. [23]

    Dennis, N

    M. Dennis, N. Jaques, E. Vinitsky, A. Bayen, S. Russell, A. Critch, and S. Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33:13049–13061, 2020

    2020

  16. [24]

    M. A. Figueiredo. On gaussian radial basis function approximations: Interpretation, extensions, and learning strategies. In Proceedings 15th International Conference on Pattern Recognition. ICPR-2000, volume 2, pages 618–621. IEEE, 2000

    2000

  17. [25]

    K. P. F.R.S. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559–572, 1901. doi: 10.1080/14786440109462720

    1901 doi

  18. [26]

    Hu and J

    H. Hu and J. N. Foerster. Simplified action decoder for deep multi-agent reinforcement learning. arXiv preprint arXiv:1912.02288, 2019

    1912 arXiv

  19. [27]

    Jiang, M

    M. Jiang, M. Dennis, J. Parker-Holder, J. Foerster, E. Grefenstette, and T. Rocktäschel. Replay- guided adversarial environment design. Advances in Neural Information Processing Systems, 34:1884–1897, 2021

    2021

  20. [28]

    Jiang, E

    M. Jiang, E. Grefenstette, and T. Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pages 4940–4950. PMLR, 2021

    2021

  21. [29]

    D. P. Kingma and M. Welling. Auto-encoding variational bayes, 2022. URL https://arxiv. org/abs/1312.6114

    2022 arXiv

  22. [30]

    Lehman and K

    J. Lehman and K. Stanley. Exploiting open-endedness to solve problems through the search for novelty. Artificial Life - ALIFE, 01 2008

    2008

  23. [31]

    Lehman and K

    J. Lehman and K. O. Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary Computation, 19(2):189–223, 2011. doi: 10.1162/EVCO_a_00025

    2011 doi

  24. [32]

    J. Z. Leibo, E. Hughes, M. Lanctot, and T. Graepel. Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research, 2019. URL https://arxiv.org/abs/1903.00742

    2019 arXiv

  25. [33]

    W. Li, P. Varakantham, and D. Li. Effective diversity in unsupervised environment design. arXiv preprint arXiv:2301.08025, 2023

    2023 arXiv

  26. [34]

    Mediratta, M

    I. Mediratta, M. Jiang, J. Parker-Holder, M. Dennis, E. Vinitsky, and T. Rocktäschel. Stabilizing unsupervised environment design with a learned adversary, 2023. URL https://arxiv.org/ abs/2308.10797. 12

    2023 arXiv

  27. [35]

    Musgrave, S

    K. Musgrave, S. J. Belongie, and S. N. Lim. Pytorch adapt. ArXiv, abs/2211.15673, 2022

    2022 arXiv

  28. [36]

    Ostrovski, M

    G. Ostrovski, M. G. Bellemare, A. van den Oord, and R. Munos. Count-based exploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 2721–2730. JMLR.org, 2017

    2017

  29. [37]

    Parker-Holder, M

    J. Parker-Holder, M. Jiang, M. Dennis, M. Samvelyan, J. Foerster, E. Grefenstette, and T. Rocktäschel. Evolving curricula with regret-based environment design. arXiv preprint arXiv:2203.01302, 2022

    2022 arXiv

  30. [38]

    Pathak, P

    D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In International conference on machine learning , pages 2778–2787. PMLR, 2017

    2017

  31. [39]

    Pinto, J

    L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta. Robust adversarial reinforcement learning. In International conference on machine learning, pages 2817–2826. PMLR, 2017

    2017

  32. [40]

    Portelas, C

    R. Portelas, C. Colas, K. Hofmann, and P.-Y . Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments, 2019. URL https://arxiv. org/abs/1910.07224

    2019 arXiv

  33. [41]

    Portelas, C

    R. Portelas, C. Colas, K. Hofmann, and P.-Y . Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning, pages 835–853. PMLR, 2020

    2020

  34. [42]

    Portelas, C

    R. Portelas, C. Colas, L. Weng, K. Hofmann, and P.-Y . Oudeyer. Automatic curriculum learning for deep rl: a short survey. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI’20, 2021. ISBN 9780999241165

    2021

  35. [43]

    R. A. Redner and H. F. Walker. Mixture densities, maximum likelihood and the em algorithm. SIAM review, 26(2):195–239, 1984

    1984

  36. [44]

    D. J. Rezende and S. Mohamed. Variational inference with normalizing flows, 2016. URL https://arxiv.org/abs/1505.05770

    2016 arXiv

  37. [45]

    P. J. Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53–65, 1987

    1987

  38. [46]

    T. Schaul. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015

    2015 arXiv

  39. [47]

    Schmidhuber

    J. Schmidhuber. A Possibility for Implementing Curiosity and Boredom in Model-Building Neural Controllers. In From Animals to Animats: Proceedings of the First International Confer- ence on Simulation of Adaptive Behavior. The MIT Press, 02 1991. ISBN 9780262256674. doi: 10.75...

    1991 doi

  40. [48]

    Schulman, P

    J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015

    2015 arXiv

  41. [49]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  42. [50]

    Singh, R

    S. Singh, R. L. Lewis, A. G. Barto, and J. Sorg. Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2(2): 70–82, 2010. doi: 10.1109/TAMD.2010.2051031

    2010

  43. [51]

    K. O. Stanley and J. Lehman. Why Greatness Cannot Be Planned: The Myth of the Objective. Springer Publishing Company, Incorporated, 2015. ISBN 3319155237

    2015

  44. [52]

    Teoh Jing Xiang, W

    J. Teoh Jing Xiang, W. Li, and P. Varakantham. Unifying regret and state-action space coverage for effective unsupervised environment design. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’24, page 2507–2509, Richland, S...

    2024

  45. [53]

    Tobin, R

    J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23–30. IEEE, 2017

    2017

  46. [54]

    Van der Maaten and G

    L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008

    2008

  47. [55]

    Viroli and G

    C. Viroli and G. J. McLachlan. Deep gaussian mixture models, 2017. URL https://arxiv. org/abs/1711.06929

    2017 arXiv

  48. [56]

    R. Wang, J. Lehman, J. Clune, and K. O. Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01753, 2019

    1901 arXiv

  49. [57]

    R. Wang, J. Lehman, A. Rawal, J. Zhi, Y . Li, J. Clune, and K. Stanley. Enhanced poet: Open- ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International Conference on Machine Learning, pages 9940–9951. PMLR, 2020

    2020

  50. [58]

    Wattenberg, F

    M. Wattenberg, F. Viégas, and I. Johnson. How to use t-sne effectively.Distill, 1(10):e2, 2016

    2016

  51. [59]

    T. Xie, D. J. Foster, Y . Bai, N. Jiang, and S. M. Kakade. The role of coverage in online reinforcement learning. arXiv preprint arXiv:2210.04157, 2022

    2022 arXiv

  52. [60]

    Zhang, J

    J. Zhang, J. Lehman, K. Stanley, and J. Clune. Omni: Open-endedness via models of human notions of interestingness, 2024. URL https://arxiv.org/abs/2306.01711

    2024 arXiv

  53. [61]

    Zhao and V

    R. Zhao and V . Tresp. Curiosity-driven experience prioritization via density estimation, 2020. URL https://arxiv.org/abs/1902.08039

    2020 arXiv

  54. [62]

    R. Zhao, X. Sun, and V . Tresp. Maximum entropy-regularized multi-goal reinforcement learning. In International Conference on Machine Learning, pages 7553–7562, 2019

    2019

  55. [63]

    curiosity

    K. Zhou, Z. Liu, Y . Qiao, T. Xiang, and C. C. Loy. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. 14 A Extended Experiment Details and Ablation Studies In this section, we present extended experiment details regarding th...

    2022

  56. [2018]

    URL https://arxiv.org/abs/1810.12894

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.