{"id":"6d62e617-7430-4d22-8dec-d540637257ec","arxiv_id":"2501.12627","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A systematic empirical study showing that cycle-based fusion of two or three intrinsic rewards improves exploration and unsupervised RL performance more than single rewards.","lead":"An RL paper tests four simple ways of combining multiple intrinsic exploration rewards, including two new strategies that pick one reward at a time. On grid-world, procedurally generated, and Atari benchmarks, cycling through rewards was the most robust, and mixing two or three rewards usually beat any single one.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NGU baseline in Eq. (4) deviates from the original NGU algorithm, and since NGU appears in most top candidates, the cycle-strategy result may be an artifact of this nonstandard baseline.","rationale":"The reader correctly identified baseline implementation fidelity as the weakest assumption. I agree with that general concern, but I have located a specific, checkable deviation: the NGU formulation in Eq. (4). The original NGU uses the temporal difference of the RND error, a key component of its directed exploration. The paper's formula uses the raw RND error, which is closer to RND clipped and divided by episodic counts. This is not a minor implementation detail; it changes the intrinsic reward signal itself. Since NGU is the most frequently contributing reward in the best candidates (Q3) and is part of the recommended pair, any change in NGU can alter the relative performance of fusion strategies. For example, if the raw-error version produces a smoother or more persistent signal, cycle may appear more robust than summation because it gives each reward equal time, masking scale issues. With canonical NGU, summation might perform comparably or better. This concern is directly testable by swapping in an independent NGU implementation and rerunning a focused set of experiments. The proposed test covers the environments where cycle was strongest and uses the same PPO backbone and fusion code, so it isolates the effect of NGU's definition. Secondary issues, such as the unreported κ decay rate and per-benchmark β0 tuning, are also present, but they are less likely to reverse the central claim on their own. The paper is otherwise a broad empirical study with honest limitations; the NGU deviation is the one place where a specific, plausible bug could invalidate the headline conclusion. The reader's CONDITIONAL verdict remains appropriate: the paper should not be fully accepted until the NGU baseline is verified and the test above is run. I do not see a need to move to reject or accept without these checks.","tokens_in":31166,"tokens_out":10294,"duration_ms":101829,"concrete_test":"Replace the toolkit's NGU implementation with canonical NGU from the original authors (e.g., the NGU agent in dopamine or RLlib, which uses (α_{t+1}−α_t)_+). Keep all HIRE fusion code, PPO hyperparameters, and β0/κ settings unchanged. Rerun a representative subset: MiniGrid KeyCorridorS8R5 and MultiRoom-N7-S8, and Procgen Heist and Plunder, for the (NGU, RE3) and (NGU, RE3, ICM) combinations under both cycle and summation strategies, with five seeds each. If the strategy-level IQM ranking changes on any environment (e.g., summation beats cycle), the paper's robustness claim is an artifact of the nonstandard NGU.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing point is the fidelity of the NGU baseline. Appendix A, Eq. (4) defines NGU's intrinsic reward as I_t = min{max{α_t}, C}/√N_ep(s_t), where α_t is the RND error. The original NGU (Badia et al., 2020) uses the positive temporal difference of the RND error, (α_{t+1} − α_t)_+, to measure learning progress, scaled and clipped, not the raw error. The paper's version is essentially a clipped RND bonus divided by episodic pseudo-counts, which is a different algorithm. The paper states α_t is computed following RND, but raw RND error is not NGU's exploration signal. This matters because Q3 reports NGU appears in 12 of the 16 best reward candidates, and the recommended (NGU, RE3) pair depends on NGU. If the toolkit's NGU is the Eq. (4) version, the comparisons are against a nonstandard baseline, and cycle's superiority could stem from how this alternative reward interacts with normalization and fusion rather than from properties of genuine NGU. If the implementation is actually the original NGU, then Eq. (4) is an incorrect description, a reproducibility failure. Either way, the central claim about cycle robustness is not yet anchored to the published NGU algorithm.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HIRE, a framework for combining multiple intrinsic reward signals through four fusion strategies (summation, product, cycle, and maximum), and evaluates the resulting hybrid rewards on eight MiniGrid tasks, eight Procgen tasks, and five ALE games in an unsupervised pretraining/fine-tuning setup. Using PPO as a fixed backbone and the authors' RLLTE/RLEXPLORE implementations of ICM, NGU, RE3, and E3B, the authors report over four thousand runs with five seeds, aggregate results with IQM and bootstrap confidence intervals, and rank all strategy/candidate combinations. The main empirical findings are that the cycle fusion strategy is the most robust and best-performing on most tasks, that NGU and RE3 are the most frequent contributors to the best candidates, that the (NGU, RE3) combination is recommended, and that hybrid rewards improve fine-tuned ALE-5 returns relative to single intrinsic rewards.","tokens_in":31382,"tokens_out":11725,"duration_ms":124432,"significance":"If the empirical claims hold, the paper offers a simple, algorithm-agnostic recipe — time-multiplexing intrinsic rewards — that could be a reliable default for hard-exploration RL, and it provides one of the more systematic comparisons of fusion strategies and reward combinations to date. The strengths are the breadth of the study (16 environments plus ALE-5, 45 reward candidates per fusion strategy), the use of IQM and bootstrap confidence intervals, the fixed PPO hyperparameters across conditions, and the modularity of the framework. The main weaknesses are that the central claims about exploration diversity and skill acquisition are not measured directly, and that the results depend on the fidelity of the authors' own implementations of the four intrinsic reward baselines. With those caveats addressed, this would be a useful systematic empirical contribution to intrinsically motivated RL.","major_comments":[{"comment":"The abstract and conclusion claim that HIRE 'significantly enhance[s] exploration efficiency and diversity, as well as skill acquisition', but no direct measure of exploration diversity or skill acquisition is reported anywhere in the paper. All evidence is in the form of episode returns on MiniGrid/Procgen and fine-tuned returns on ALE-5. Please either add direct metrics (for example, state coverage, state entropy, or number/coverage of learned skills) or revise the claims to refer to return/performance rather than diversity. This matters because diversity is one of the two advertised benefits and is used to explain why the cycle strategy is superior.","section":"Abstract; §5.2 Q1–Q5"},{"comment":"The claim that 'the cycle strategy demonstrates superior robustness and achieves the best performance on most tasks' is based on top-1 counts (75% on MiniGrid, 50% on Procgen) and aggregate IQM, but no paired significance test across environments or seeds is provided. With five seeds per environment, the top-1 proportions may be within sampling noise, and overlapping bootstrap intervals in Figure 3 do not by themselves establish a significant difference. Please report paired bootstrap confidence intervals (for example, the probability of improvement) or a nonparametric paired test across the 16 environments, and calibrate the abstract and §5.2 wording accordingly.","section":"§5.2 Q1; Tables 5–6"},{"comment":"The ALE-5 evaluation is internally inconsistent: the text says that only (NGU, RE3) and (NGU, RE3, ICM) were selected as HIRE-2 and HIRE-3 candidates, yet Table 8 lists C(NGU, E3B, RE3, ICM) as the best candidate for BattleZone and Figure 12 includes HIRE-4 candidates in its rankings. Please clarify the exact candidate set used in the ALE-5 experiments. In addition, because the ALE-5 candidates were selected as the best performers on MiniGrid/Procgen, the URL comparison is a transfer/selection study rather than an unbiased evaluation of the framework; this limitation should be stated explicitly, or all hybrid combinations should be evaluated in the URL setting.","section":"§5.2 Q5; Table 8; Figure 12"},{"comment":"The fidelity of the NGU baseline is load-bearing: NGU appears in 12 of the 16 best reward candidates and in the recommended (NGU, RE3) combination, yet the implementation is taken from the authors' own RLLTE/RLEXPLORE toolkits rather than an independent reference implementation. The compressed formula in Eq. (4) is incomplete as written — 'min{max{α_t}, C}' lacks the lower clipping value — and the computation and normalization of α_t are not specified. Please report the exact α_t definition, the value of the decay rate κ in Eq. (2), the summation weights w_i in Table 1, and provide a sanity check by reproducing published single-reward baseline results with these implementations, so that the strategy rankings cannot be attributed to toolkit-specific artifacts.","section":"Appendix A Eq. (4); Appendix B.1"},{"comment":"The paper does not compare HIRE against established hybrid intrinsic reward methods such as RIDE, NovelD, AGAC, or even the original NGU product combination, which is itself a hybrid of episodic and lifelong novelty. Since the paper's contribution is specifically about combining intrinsic rewards, the lack of any existing hybrid baseline makes it difficult to judge whether the proposed fusion strategies offer practical advantages over prior hybrid designs. A comparison with at least one or two such baselines on the same environments would substantially strengthen the empirical claims.","section":"§2.2; §5"}],"minor_comments":[{"comment":"The notation 'min{max{α_t}, C}' is malformed; it should be written as, for example, min{max{α_t, 1}, C}, and the definition of α_t should be given in the same appendix rather than referring only to 'the RND method'.","section":"Appendix A, Eq. (4)"},{"comment":"The exploration coefficient β_t depends on a decay rate κ, but the value of κ is never reported in Table 4 or Appendix B. Please report κ for each benchmark, since it is a free parameter that affects the entire training curve.","section":"Eq. (2); Appendix B"},{"comment":"The statement that cycle and maximum 'can be viewed as special cases of the summation method' is inaccurate for maximum: maximum selects the largest realized reward and cannot be expressed as a fixed non-zero weight vector of the summation strategy.","section":"Table 1"},{"comment":"Figure 6 uses the legend HIRE-1 through HIRE-4, but the text says only HIRE-2 and HIRE-3 were the selected reward candidates; please reconcile the figure, the text, and Table 8 so that the ALE-5 candidate set is unambiguous.","section":"§5.2 Q5; Figure 6"},{"comment":"The description of the ALE-5 protocol says the fine-tuning phase uses 'extrinsic rewards only', but it is not stated whether β_t is set to zero or whether the intrinsic reward module is disabled; please clarify the exact protocol.","section":"§5.2 Q5"},{"comment":"The phrase 'significantly enhance exploration efficiency and diversity' is stronger than the evidence reported in the paper; consider replacing 'diversity' with 'downstream task performance' or adding the missing diversity metrics.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is closely tied to the authors' own RLLTE and RLEXPLORE toolkits, and all intrinsic reward baselines come from those toolkits. Given that NGU dominates the reported rankings, the editor may wish to ensure that the toolkit implementations match the original published algorithms and that the code is released. The ALE-5 'unsupervised RL' setup is also nonstandard relative to URLB; this is not a fatal issue but should be framed carefully in the final version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the genuinely useful part of this paper is empirical: a systematic comparison of four fusion strategies for combining intrinsic rewards on MiniGrid, Procgen, and ALE-5, analyzed with IQM and bootstrap confidence intervals. The takeaway that a simple cycle fusion (time-multiplexing between rewards) is robust across tasks is plausible and worth knowing. Second, there is a load-bearing fidelity problem in the NGU baseline. Eq. (4) defines NGU's intrinsic reward as a clipped RND error divided by episodic pseudo-counts, but the published NGU uses the positive temporal difference of the RND error, scaled and clipped. That is a different exploration signal. The authors use implementations from their own RLLTE and RLEXPLORE toolkits; if the toolkit's NGU is the Eq. (4) version, the comparisons are against a nonstandard baseline, and since NGU appears in twelve of the sixteen best reward candidates, the cycle result could be an artifact of that implementation rather than a property of genuine NGU. If the implementation actually matches the original NGU, then Eq. (4) is an incorrect description, which is a separate reproducibility failure. Either way, the central claim is not yet anchored to the published algorithm.\n\nWhat the paper does well: it is honest about limitations, keeps PPO hyperparameters fixed, reports aggregate statistics properly, and explicitly notes that cycle and maximum are special cases of summation with a single nonzero weight. The quantity-level analysis is a nice contribution; the finding that two or three rewards usually beat four in dynamic environments is a practical heuristic.\n\nSoft spots, in decreasing order of concern. The NGU baseline issue above is the main one. Second, beta0 is tuned per benchmark with a grid search and then fixed; that is a free parameter and should be reported as a sensitivity analysis, not a universal recommendation. Third, the claim of improved exploration diversity is inferred from returns only, not measured directly. Fourth, the ALE-5 candidates are selected post hoc from the authors' own MiniGrid/Procgen results, which is acceptable for exploratory analysis but should not be framed as confirmatory. The summation weights are also not reported.\n\nNone of these issues are fatal in themselves, but together they mean the paper needs revision and reproducibility artifacts before the cycle-robustness claim is solid. Who this is for: practitioners in hard-exploration RL who want usable heuristics for combining intrinsic rewards, and researchers working on intrinsic reward baselines. I would send it to a serious referee, conditional on the authors addressing the NGU description/implementation mismatch and releasing code or verification of the baseline implementations.","headline":"Useful empirical comparison of fusion strategies, but the NGU baseline in Eq. (4) is not the original NGU, and since NGU drives most top candidates, the cycle result needs to be verified against a faithful implementation.","tokens_in":31929,"tokens_out":2243,"would_cite":false,"duration_ms":23662,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that cycling through intrinsic rewards one at a time is the most robust way to combine exploration bonuses, beating sum, product, and max fusion on most hard-exploration tasks, and beating single bonuses in unsupervised…","keywords":["hybrid intrinsic rewards","intrinsic motivation","exploration","reward fusion strategies","unsupervised reinforcement learning","deep reinforcement learning","MiniGrid","Procgen"],"falsifier":"Re-run the same sixteen MiniGrid and Procgen tasks with independently reproduced implementations of ICM, NGU, RE3, and E3B from the original papers' released code, using the same PPO backbone, seeds, environment steps, and the same four fusion rules; if the cycle strategy does not stay at or near the top on most tasks, or if a single intrinsic reward already matches the best hybrids, the central claim fails. A cheaper probe is to hold one normalization scheme fixed across all four fusion rules and check whether the cycle advantage survives, since scale mismatch is the paper's own stated reason summation and product struggle.","tokens_in":30934,"feed_emoji":"🧭","tokens_out":11686,"duration_ms":104665,"temperature":0.7,"pith_summary":"Intrinsic reward shaping adds a built-in signal — curiosity, novelty, or state entropy — to push a reinforcement-learning agent toward unexplored behavior when the environment's own rewards are sparse or absent. This paper argues that one such signal is rarely enough, and that how you combine several matters as much as which ones you pick. It introduces HIRE (Hybrid Intrinsic REward), a modular framework that fuses any number of intrinsic rewards through four strategies — summation, product, cycle, and maximum — and sweeps all combinations of four representative rewards (ICM, NGU, RE3, E3B) across sixteen hard-exploration tasks in MiniGrid and Procgen, plus unsupervised pre-training on Atari. The central finding is that the newly proposed cycle rule, which activates one intrinsic reward at a time on a rotating schedule, is the most robust fusion strategy on most tasks, and that hybrid rewards outperform every single-reward baseline in the unsupervised setting. If the paper is right, practitioners get a simple, low-tuning default for sparse-reward problems: combine two or three complementary bonuses and let them take turns.","feed_headline":"Cycling intrinsic rewards wins most hard-exploration tasks","feed_subtitle":"A hybrid-reward study shows rotating two or three bonus signals beats any single one on sparse-reward games.","key_machinery":"The carrying object is the time-multiplexing fusion rule itself. HIRE defines four ways to collapse a set of intrinsic reward functions $\\{I^1_t,\\dots,I^n_t\\}$ into one scalar: summation $I_t = \\sum_{i=1}^n w^i_t I^i_t$, product $I_t = \\prod_{i=1}^n I^i_t$, cycle $I_t = I^i_t$ with $i = (t \\bmod n)$, and maximum $I_t = \\max_i I^i_t$. The cycle rule, newly proposed here, does the explanatory work: by attending to one motivation at a time, it sidesteps the scale-mismatch and conflicting-signal problems the authors attribute to summation and product, and the greedy over-exploration they attribute to maximum. The four rewards being fused span three families — ICM (prediction error of a learned dynamics model), NGU (lifelong RND novelty modulated by episodic pseudo-counts), RE3 (k-nearest-neighbor estimate of state-entropy), and E3B (ellipsoidal episodic bonus in a learned embedding) — and the whole module sits in front of a fixed PPO loop with all PPO hyperparameters held constant, so observed differences are attributed to the rewards and their fusion.","core_discovery":"The discovery is that the fusion rule is a first-class design choice in intrinsically motivated RL. Across more than four thousand runs, the cycle strategy $I_t = I^i_t$ with $i = (t \\bmod n)$ — using one intrinsic reward at a time and rotating through them — achieved the best performance on most of the sixteen MiniGrid and Procgen tasks and showed the greatest robustness, capturing 75% of top-1 candidates on MiniGrid and 50% on Procgen, while the product rule was the least stable on sequential navigation tasks like KeyCorridor. The reward NGU, itself a hybrid of lifelong and episodic novelty, appeared in twelve of the sixteen best candidates, and the authors single out the two-reward pair (NGU, RE3) as the best balance of exploration coverage and computational cost. In the ALE-5 unsupervised RL setting, hybrid intrinsic rewards produced significantly better fine-tuned scores than any single-reward pre-training. The quantity effect saturates: three rewards helped on navigation-heavy MiniGrid levels, two were generally best on procedurally generated Procgen levels, and frames-per-second falls steadily as rewards are added, so up to three rewards marks the practical efficiency frontier.","pith_inferences":["Cycle fusion may work as a form of temporal ensembling that averages out noise and scale quirks of individual reward channels; a direct test is to compare the fixed rotating schedule against random or adversarially chosen orders to see whether periodicity itself is what helps.","The results suggest reward fusion is a hyperparameter space of its own, and learned arbitration — letting the agent choose which bonus is active based on state or recent progress — is a natural successor to the fixed cycle rule that this paper implicitly benchmarks.","Because only the two best-performing combinations (NGU, RE3) and (NGU, RE3, ICM) were carried into the ALE-5 study, the claim that hybrids beat singles in unsupervised RL rests on a narrow slice; extending to more combinations and more games would test whether the advantage belongs to hybrids generally or to NGU's presence.","The recommendation of (NGU, RE3) is partly a compute argument, since RE3 needs no learned representation, so the same four fusion rules could be re-tested with representation-learning-free or skill-discovery rewards to see whether the cycle advantage generalizes beyond the curiosity and novelty families."],"forward_implications":["A default recipe for sparse-reward tasks: take two complementary bonuses such as NGU and RE3, cycle between them, and expect stronger and more robust scores than either bonus alone, with no fusion-weight tuning.","The quantity effect is finite — adding a third reward helps on sequential navigation tasks in MiniGrid, but on dynamically generated Procgen levels more rewards tend to hurt, so practitioners should test two and three rather than assume more is better.","Hybrid intrinsic rewards transfer to unsupervised RL: pre-training with combined bonuses improves downstream fine-tuning on ALE-5 compared with pre-training on any single bonus.","The product rule is the least reliable general choice for sequential tasks, although it can win in dynamic environments such as Chaser and Miner where satisfying several motivations simultaneously matters.","Computational cost grows with reward count, so two or three rewards define a practical efficiency frontier for the framework."],"supporting_citations":[{"why":"Supplies ICM, the curiosity-driven intrinsic reward that is one of the four fusion candidates.","marker":"[Pathak et al., 2017]"},{"why":"Supplies NGU, the reward present in twelve of the sixteen best hybrid candidates, and the canonical example of combining lifelong with episodic novelty.","marker":"[Badia et al., 2020]"},{"why":"Supplies RE3, the state-entropy reward in the recommended (NGU, RE3) pairing.","marker":"[Seo et al., 2021]"},{"why":"Supplies E3B, the elliptical episodic bonus that is the fourth fusion candidate.","marker":"[Henaff et al., 2022]"},{"why":"Provide the implementations and recommended configurations of all four baseline intrinsic rewards, which the strategy comparison rests on.","marker":"[Yuan et al., 2025; Yuan et al., 2024]"},{"why":"Supplies the interquartile-mean KPI and stratified-bootstrap confidence procedure used for all aggregated performance comparisons.","marker":"[Agarwal et al., 2021]"},{"why":"The MiniGrid benchmark, one of the two main testbeds for the fusion-strategy and quantity comparisons.","marker":"[Chevalier-Boisvert et al., 2023]"},{"why":"The Procgen benchmark, the other main testbed, whose procedurally generated levels test the robustness of each fusion strategy.","marker":"[Cobbe et al., 2020]"},{"why":"Justifies ALE-5 as a representative five-game subset of Atari for the unsupervised RL experiment.","marker":"[Aitchison et al., 2023]"}],"fun_headline_variants":["Rotating reward signals beats any single intrinsic bonus","Two or three reward signals beat one in sparse-reward RL","Switching intrinsic rewards each step boosts exploration efficiency","Hybrid rewards: cycle two or three exploratory signals for best RL","Rotating intrinsic rewards: 75% top-1 on MiniGrid, 50% on Procgen"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison assumes that the four intrinsic-reward implementations (ICM, NGU, RE3, E3B), taken from the authors' own toolkits with fixed recommended hyperparameters (Appendix B.1), faithfully represent how those algorithms perform; if those implementations differ from canonical versions or carry hidden tuning that happens to suit the cycle rule, the strategy ranking could be a property of the toolkit rather than of hybrid fusion.","fun_headline_variants_meta":{"raw":{"variants":["Rotating reward signals beats any single intrinsic bonus","Two or three reward signals beat one in sparse-reward RL","Switching intrinsic rewards each step boosts exploration efficiency","Hybrid rewards: cycle two or three exploratory signals for best RL","Rotating intrinsic rewards: 75% top-1 on MiniGrid, 50% on Procgen"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000461,"raw_usage":{"total_tokens":2288,"prompt_tokens":906,"completion_tokens":1382,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":1292}},"tokens_in":522,"tokens_out":1382,"duration_ms":10997,"temperature":1.0,"reasoning_tokens":1292,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:57:57.080752+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same sixteen MiniGrid and Procgen tasks with independently reproduced implementations of ICM, NGU, RE3, and E3B from the original papers' released code, using the same PPO backbone, seeds, environment steps, and the same four fusion rules; if the cycle strategy does not stay at or near the top on most tasks, or if a single intrinsic reward already matches the best hybrids, the central claim fails. A cheaper probe is to hold one normalization scheme fixed across all four fusion rules and check whether the cycle advantage survives, since scale mismatch is the paper's own stated reason summation and product struggle.","supporting_citations":[{"cited_title":"Never give up: Learning directed exploration strategies","cited_arxiv_id":null,"evidence_quote":"Supplies NGU, the reward present in twelve of the sixteen best hybrid candidates, and the canonical example of combining lifelong with episodic novelty."},{"cited_title":"State entropy maximization with random encoders for efficient exploration","cited_arxiv_id":null,"evidence_quote":"Supplies RE3, the state-entropy reward in the recommended (NGU, RE3) pairing."},{"cited_title":"Exploration via elliptical episodic bonuses","cited_arxiv_id":null,"evidence_quote":"Supplies E3B, the elliptical episodic bonus that is the fourth fusion candidate."},{"cited_title":"Rllte: Long-term evolution project of reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Provide the implementations and recommended configurations of all four baseline intrinsic rewards, which the strategy comparison rests on."},{"cited_title":"Deep reinforcement learning at the edge of the statistical precipice","cited_arxiv_id":null,"evidence_quote":"Supplies the interquartile-mean KPI and stratified-bootstrap confidence procedure used for all aggregated performance comparisons."},{"cited_title":"Leveraging procedural generation to benchmark reinforcement learning","cited_arxiv_id":null,"evidence_quote":"The Procgen benchmark, the other main testbed, whose procedurally generated levels test the robustness of each fusion strategy."},{"cited_title":"Atari-5: Distilling the arcade learning environment down to five games","cited_arxiv_id":null,"evidence_quote":"Justifies ALE-5 as a representative five-game subset of Atari for the unsupervised RL experiment."}],"review_version":1}