Pith. sign in

REVIEW 3 major objections 4 minor 79 references

Understanding Behavioral Metric Learning: A Large-Scale Study on Distracting Reinforcement Learning Environments

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A large-scale study of five behavioral-metric methods in reinforcement learning finds that the standalone benefit of the metric loss for filtering task-irrelevant noise is marginal; self-prediction and normalization carry the gains.

desk verdict A careful, unusually transparent large-scale benchmark whose central negative result on metric losses is plausible and mostly supported, but method rankings need caveats because several implementations were modified and one uses a near-disabled metric loss. read the letter →

arxiv 2506.00563 v2 pith:4IVL4FSO submitted 2025-05-31 cs.LG cs.AI

Ziyan Luo , Tianwei Ni , Pierre-Luc Bacon , Doina Precup , Xujie Si This is my paper
classification cs.LGcs.AI
keywords behavioralmetricsbisimulationreinforcementlearningrepresentationdenoisingfactorself-predictionlossfeaturenormalizationisometricembedding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Behavioral metric learning—approximating bisimulation or MICo distances and embedding them into a representation space—is widely credited with making deep reinforcement learning robust to task-irrelevant noise. This paper tests that credit at scale: it unifies five recent metric-based methods as isometric embeddings, runs them with baselines across 370 task configurations in state- and pixel-based control, and measures both return and a new denoising factor. Its central finding is that in an isolated setting where the encoder is trained by the metric loss alone, the explicit metric loss provides only marginal denoising benefit over self-prediction or no metric objective at all. The paper concludes that the robustness gains previously attributed to metric learning largely come from auxiliary self-prediction losses and feature normalization. If correct, this redirects attention from metric objectives to these other components.

What carries the argument

The paper's organizing device is the isometric embedding definition (Eq. 2), which frames each method as an encoder that tries to make representation distances reproduce a target behavioral distance approximated from rewards and transitions. Its evaluation machinery consists of the denoising factor (DF, Eq. 9), which measures an encoder's ability to pull together observations that share a task-relevant state (positive pairs) while separating random pairs, and the isolated metric estimation setting (Sec. 4.4), in which a separate encoder is trained solely by the metric loss so that the metric's contribution can be measured without interference from RL, reward-prediction, and self-prediction losses.

What would settle it

Re-run the isolated metric estimation experiment with metric targets computed from on-policy rollout data, or with an off-policy correction, on the same six state-based tasks; if encoders trained with the metric loss then achieve clearly higher denoising factors than encoders trained with self-prediction alone, the paper's marginality conclusion would be an artifact of target misspecification rather than a property of metric learning.

Watch

Extended reading notes

Core claim

The paper claims that when the encoder is influenced solely by the metric loss, the standalone benefit of learning a metric by an explicit metric loss becomes marginal (Sec. 5.3). In the full benchmarks, SimSR—which combines a metric loss with self-prediction and L2 normalization—outperforms all methods on state-based tasks, and ablations show its success is largely driven by the self-prediction loss and feature normalization; DeepMDP with LayerNorm performs comparably to SimSR, while DBC, the most commonly used metric-learning baseline, consistently performs worst. The paper also reports that SAC and DeepMDP are strong baselines, and that the widely used grayscale-video distraction is not harder than a clean background for several methods, while IID Gaussian noise with random projection is the challenging setting.

Load-bearing premise

The benchmarked methods compute on-policy metric targets using rewards and transitions sampled from an off-policy replay buffer, and the comparison's validity depends on this off-policy data standing in for the on-policy distances the theory requires.

Editorial extensions

If this is right

  • If the claims hold, previously reported robustness gains of bisimulation-metric methods should be re-attributed to self-prediction losses and normalization rather than to the metric objective itself.
  • The denoising factor gives the field a quantitative measure of distraction filtering that can be reported alongside returns in future benchmarks.
  • Adding LayerNorm to the representation space improves both reward and denoising for most methods, including DeepMDP, making normalization a first-class design choice in noisy environments.
  • Simple baselines SAC and DeepMDP remain competitive on both state- and pixel-based distracting tasks, raising the bar for what a metric-based method must demonstrate.
  • The commonly used grayscale-video OOD benchmark is not harder than a clean background for several methods; IID Gaussian noise with random projection is a more discriminating test.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: if the on-policy/off-policy gap in target estimation is closed by computing metric targets from on-policy rollouts, the marginal benefit of the metric loss might reappear, which would localize the cause of the observed marginality to target misspecification rather than to metric learning in principle.
  • The marginality result suggests that future metric-based methods should be designed to complement self-prediction and normalization, or to show value on settings like random projection where simple baselines struggle.
  • The denoising factor could be borrowed as an auditing tool for representation learning beyond behavioral metrics, for example to compare data augmentation and self-supervised losses on distraction filtering.
  • One could extend the isolated setting to discrete control and other base algorithms to check whether the marginality is specific to SAC-based continuous control.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents a large-scale empirical study of behavioral metric learning methods in distracting reinforcement learning environments. It unifies five recent metric-based representation learning methods (DBC, DBC-normed, MICo, RAP, SimSR) under an isometric-embedding framework, introduces a denoising factor (DF) to quantify an encoder's ability to filter task-irrelevant noise, and proposes an isolated metric estimation setting to evaluate the metric loss alone. The authors benchmark seven methods (including SAC and DeepMDP baselines) across 20 state-based and 14 pixel-based DeepMind Control tasks under multiple noise settings, with both ID and OOD generalization evaluations. Main findings are that SimSR performs best in state-based tasks, RAP performs best in pixel-based tasks, SAC and DeepMDP are strong baselines, DBC performs worst, and in the isolated setting, an explicit metric loss offers only marginal denoising benefit compared with self-prediction (ZP) loss. The paper also includes theoretical analyses of denoising properties of bisimulation metrics and releases an open-source codebase.

Significance. If the findings hold, they substantially qualify the robustness gains previously attributed to bisimulation metrics in deep RL, suggesting that self-prediction and feature normalization are the main drivers. The study is unusually comprehensive: 12 seeds for state-based and 5 seeds for pixel-based tasks, 95% confidence intervals, per-task tables accompanying every aggregated figure, and a modular open-source implementation. The introduction of the denoising factor and the isolated estimation setting are useful evaluation tools for future work. The theoretical claims (denoising properties of BSM and PBSM, transition-distance preservation under isometry) are supported by proofs in Appendix D. The benchmarking itself is internally consistent, and the paper is transparent about many of its design choices. However, the central negative result in the isolated setting is conditional on an acknowledged but uncontrolled theory-practice gap: the on-policy behavioral metric targets are estimated from off-policy replay data, which may handicap metric losses relative to policy-independent baselines.

major comments (3)
  1. [Sec. 5.3, Eq. (15), Eq. (24); Sec. 3.4] The paper's strongest claim—that explicit metric losses provide only marginal standalone denoising benefit—is not yet fully supported because the evaluated metric methods estimate on-policy PBSM/MICo targets from off-policy replay data. The authors explicitly flag this gap in Sec. 3.4 ('both PBSM and MICo are on-policy metrics, but the sampled rewards used in d_R are from a replay buffer'), and Eq. (15) and Eq. (24) indeed use rewards and next-state samples from a replay buffer that contains a mixture of past behavior policies. The comparison baselines in the isolated setting (ZP, RP) are policy-independent, so this off-policy mismatch biases only the metric methods. As a result, the observed 'marginal benefit of an explicit metric loss' could reflect a misspecified target rather than an inherent property of metric learning. I recommend adding a control experiment in which the metric targets are computed using data collected by a fixed policy that matches the target policy (e.g., a frozen SAC policy), or at least an analysis that quantifies the degree of off-policyness and its effect on DF. Absent such a control, the central conclusion should be tempered to state that current off-policy implementations of metric losses provide marginal benefit.
  2. [Sec. 3.3, Sec. 5.1] The benchmarking results in Sec. 5.1, including the finding that 'DBC consistently performs the worst,' are based on modified versions of the published methods. As stated in Sec. 3.3, DBC-normed is stripped of its intrinsic-reward and inverse-dynamics components, DBC's observation reconstruction is removed, and DBC-normed and SimSR use a single probabilistic transition model instead of their original components. These modifications are reasonable for isolating the metric loss, but the paper's headline claims are then not directly about the original methods. This is a serious attribution issue: readers may conclude that the original DBC algorithm is inferior, when in fact the study evaluates a reduced variant. The authors should either clearly rephrase the findings as applying only to the stripped-down versions, or add a brief experiment showing the original versions perform similarly.
  3. [Sec. 4.3, Sec. 5.1] The denoising factor is computed using the Euclidean (L2) distance dΨ = ||·||_2, but the encoders being evaluated have different output scales: SimSR applies L2 normalization, DBC-normed applies max normalization, other pixel-based encoders use LayerNorm, and state-based encoders may use no normalization. While the ratio form of DF provides some scale invariance, the comparability of DF across methods with differently scaled representation spaces is not fully established. The paper should clarify whether the encoder outputs are re-normalized before computing DF and discuss the potential impact of output scale on the DF comparisons in Fig. 5 and Fig. 12.
minor comments (4)
  1. [Sec. 3.1, Eq. (2)] The definition of isometric embedding (Eq. 2) assumes an equality between the target metric d_X and the representation metric d_Ψ, but the target metric is a pseudometric or diffuse metric in the methods studied (PBSM, MICo). This is a minor conceptual inconsistency; the paper might mention that the embedding is into a pseudometric space.
  2. [Sec. 5.2, Fig. 3] The ablation in Fig. 3 compares 'SimSR (Basic)' with and without ZP loss, but the caption does not specify which hyperparameters (e.g., loss coefficients) are used for the 'No ZP' variant. I assume the ZP loss coefficient is set to zero; this should be stated explicitly.
  3. [Appendix F.1] In the hyperparameter exceptions, the authors note that for MICo they use a hidden unit size of 256 rather than the original 1024 and a reward scale of 1 rather than 0.1. This is a significant deviation from the original method; a sentence explaining the potential effect on MICo's performance and why this choice is necessary for fairness would be useful.
  4. [Sec. 5.4, Fig. 6] The reward-difference plots in Fig. 6 aggregate over 14 pixel-based tasks, but the y-axis label says 'Reward difference' without units; please add a caption that explains the difference is OOD reward minus ID reward (or vice versa), as the color direction is not immediately clear.

Circularity Check

1 steps flagged · score 3.0 of 10

The isolated-setting denoising comparison is partially self-referential because the denoising factor (DF) is defined through the same representation distance that the metric losses directly optimize; the broader empirical benchmark is otherwise self-contained.

  1. self definitional [Sec. 4.3, Definition 4 (Eq. 9); related to Sec. 3.2, Eq. 6]
    "DFπdΨ(ϕ) := NegπdΨ(ϕ) − PosπdΨ(ϕ) / NegπdΨ(ϕ) + PosπdΨ(ϕ) ∈ [−1,1]. ... It also provides insight into how the behavioral metrics are approximated, given that exact behavioral metrics are nearly inaccessible via fixed-point iteration in high-dimensional state or action spaces."

    DF is defined as a normalized difference of Pos (Eq. 7) and Neg (Eq. 8), both expectations of the representation distance dΨ(ϕ(x),ϕ(·)). The metric loss JM in Eq. 6 is ℓ(dΨ(ϕ(x1),ϕ(x2)) − d̂X(x1,x2)), i.e., it drives the very same dΨ quantities to match a behavioral-metric target. Since positive pairs (same task-relevant state) have near-zero target distance and negative pairs have large target distance, an encoder that satisfactorily minimizes JM will by construction produce small positive scores and large negative scores, hence a high DF. Thus the DF measured in the isolated setting is partly a measure of whether the metric loss was satisfied, not an independent measure of denoising.

full rationale

The paper's headline findings are empirical benchmarks over many tasks and seeds, not derivations that collapse into fitted constants. No parameter is fitted to a subset and then reported as a prediction. Self-citations such as Ni et al. (2024) for the ZP/DeepMDP baseline are not load-bearing: they supply an algorithm and implementation detail, not an unverified uniqueness theorem, and the comparison against external SAC and environment benchmarks gives independent content. The one genuine circularity concern is the denoising factor: DF (Eq. 9) is built from the same representation distance dΨ that metric losses (Eq. 6) optimize, so isolated-setting DF comparisons are partially self-referential, as the paper itself acknowledges in Sec. 4.3. The paper also explicitly flags the on-policy/off-policy gap in Sec. 3.4 ('both PBSM and MICo are on-policy metrics, but the sampled rewards used in d̂R are from a replay buffer'); this is a real confound for the metric methods but is a correctness/validity concern, not circularity. Overall, the central empirical contribution remains self-contained against external benchmarks, so the score is low.

Assumptions & free parameters 4 free parameters · 6 assumptions · 2 invented entities

The ledger is light because the paper is an empirical study: its free parameters are hyperparameters inherited from the original method implementations, not fitted to produce the conclusions. The load-bearing premises are the EX-BMDP oracle structure needed for DF, the isometric embedding frame, and the off-policy approximation of on-policy metric targets, which the paper itself flags as the main theory-practice gap. No new physical entities are postulated; the DF and the isolated encoder are operational measurement tools with externally checkable outputs.

free parameters (4)
  • beta (angular distance coefficient) = 0.1 for MICo; 1e-6 for RAP
    Controls the angular term in the MICo-style representational distance (Eq. 25); the RAP value nearly disables the metric loss, a fact the authors disclose in Appendix F.1.
  • metric coefficients c_R and c_T = c_R = 1, c_T = 0.99
    Balance reward vs. transition distances in the target metric (Eq. 3, Table 9); adopted from prior implementations rather than fitted to this benchmark.
  • metric loss coefficient lambda_M = 0.5
    Weights the metric loss in the joint objective (Table 9); chosen from prior implementations.
  • DF sample counts = 16 positive and 16 negative examples per anchor
    The denoising factor (Eq. 9) is estimated with 16 positive/negative samples per anchor (Sec. 5); the estimator variance from this choice is not characterized.
assumptions (6)
  • domain assumption EX-BMDP factorization: latent state z = (s, xi) with independent noise evolution and reward depending only on s
    Underlies all denoising analysis and the construction of positive pairs for DF (Sec. 2.1, Sec. 4.3).
  • domain assumption Block structure of BMDP and existence of oracle encoders q^-1 and phi*
    Required to know which observation pairs share a task-relevant state when computing DF (Sec. 2.1, Def. 2-4).
  • standard math Existence and uniqueness of fixed points for BSM, PBSM, and MICo distances
    Invoked for Prop. 1 and Prop. 2; results from Ferns et al. 2004/2011, Castro 2020, Castro et al. 2021.
  • ad hoc to paper Isometric embedding assumption (Eq. 2) and transition distance preservation (Eq. 5)
    Def. 1 is the paper's unifying frame; Lemma 1 shows transition distances are preserved under isometry, which the paper says is 'implicitly assumed in prior work' (Sec. 3.1).
  • domain assumption Off-policy replay samples adequately approximate the on-policy metric targets
    All benchmarked methods use replay buffer samples in the metric loss (Eq. 15); the paper explicitly flags this as a theory-practice gap in Sec. 3.4.
  • standard math Convergence of the iterative bisimulation operator
    Used in the proof of Prop. 1 (Appendix D.2), citing Ferns et al. 2004/2011.
invented entities (2)
  • Denoising factor (DF) independent evidence
    purpose: Quantifies an encoder's ability to collapse noise-equivalent observations while separating task-different observations (Eq. 9).
    DF is operationally defined from the encoder's representation distances and the EX-BMDP oracle; any implementation can compute it, and its values are reported across all benchmark settings, so it is externally checkable.
  • Isolated metric encoder phi-tilde independent evidence
    purpose: An encoder trained only by the metric loss (no RL signal) to measure the standalone contribution of metric learning (Sec. 4.4).
    The construct is fully specified by the objective and training data; the paper reports DF and reward curves under it, which any re-implementation can verify.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Behavioral Metric Learning: A Large-Scale Study on Distracting Reinforcement Learning Environments." pith.science (2026). https://pith.science/paper/4IVL4FSO

@misc{pith2026250600563,
  author       = {Pith},
  title        = {Pith review of: Understanding Behavioral Metric Learning: A Large-Scale Study on Distracting Reinforcement Learning Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4IVL4FSO}},
  note         = {Machine review of arXiv:2506.00563}
}
read the original abstract

A key approach to state abstraction is approximating behavioral metrics (notably, bisimulation metrics) in the observation space and embedding these learned distances in the representation space. While promising for robustness to task-irrelevant noise, as shown in prior work, accurately estimating these metrics remains challenging, requiring various design choices that create gaps between theory and practice. Prior evaluations focus mainly on final returns, leaving the quality of learned metrics and the source of performance gains unclear. To systematically assess how metric learning works in deep reinforcement learning (RL), we evaluate five recent approaches, unified conceptually as isometric embeddings with varying design choices. We benchmark them with baselines across 20 state-based and 14 pixel-based tasks, spanning 370 task configurations with diverse noise settings. Beyond final returns, we introduce the evaluation of a denoising factor to quantify the encoder's ability to filter distractions. To further isolate the effect of metric learning, we propose and evaluate an isolated metric estimation setting, in which the encoder is influenced solely by the metric loss. Finally, we release an open-source, modular codebase to improve reproducibility and support future research on metric learning in deep RL.

Figures

Figures reproduced from arXiv: 2506.00563 by the authors.

Figure 1
Figure 1. Examples of background noise settings in pixel-based domains. In image settings, the back￾ground is fixed; in video settings, it varies slightly; IID Gaussian noise is independently sampled each timestep. 1 Introduction Real-world environments often present high-dimensional, noisy observations, posing challenges for RL. For instance, in image-based settings, task-irrelevant variations in background, lighting, and vi… view at source ↗
Figure 2
Figure 2. Benchmarking results: performance of seven methods across diverse noise settings, aggregating episodic rewards from 20 state-based (first two rows) and 14 pixel-based tasks (last row). “Noise std” denotes the IID Gaussian noise’s standard deviation σ, while “noise dim” denotes its dimension m. Bars show 95% CI. task-relevant state, varying only the noise type, which justifies this choice. Following DBC-normed, we us… view at source ↗
Figure 3
Figure 3. Ablation study on ZP loss on SimSR. “SimSR” is the agent benchmarked in Sec. 5.1, where ZP is integral to the metric estimation. Therefore, we resort to “SimSR (Basic)” setting (Theorem 2, Zang et al. (2022)), where ZP is independent of the metric estimation, and “SimSR (Basic, No ZP)” is the setting that ZP is detached from SimSR (Basic). “SimSR (with RP)” adds RP loss to original SimSR. This ablation highlights th… view at source ↗
Figures from the paper (43 more)
Figure 4
Figure 4. Figure 4: Aggregated reward (top row) and DF (bottom row) of seven agents on IID Gaussian with random projection settings, varying noise standard deviation, in the 6 selected state-based tasks. ing on (ii), additionally applying ZP loss to ϕ˜ for all metric learning methods. All…
Figure 6
Figure 6. Figure 6: Reward difference between ID and OOD evaluation in the grayscale video setting (left) and colored video setting (right), aggregated over 14 pixel-based tasks in [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 5
Figure 5. Figure 5: DF for isolated encoder ϕ˜ – top row: without LayerNorm; middle row: with LayerNorm; bottom row: with LayerNorm, ϕ˜ co-trained with metric and ZP losses. See [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 7
Figure 7. Figure 7: Aggregated reward (top row) and DF (bottom row) of 7 agents on various noise settings in 14 pixel￾based tasks in [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: SAC architecture used in our experiments. Encoder Transition Model Reward Model Critic ZP Loss RP Loss Q loss Actor Actor loss Data [PITH_FULL_IMAGE:figures/full_fig_p032_8.png]
Figure 10
Figure 10. Figure 10: General architecture of metric learn￾ing methods, summarizing the architecture used in our benchmarked metric learning methods ( [PITH_FULL_IMAGE:figures/full_fig_p032_10.png]
Figure 12
Figure 12. Figure 12: Benchmarking results: reward (left) and denoising factor (right) of seven methods to IID Gaussian noise dimension (Noise Dim) and standard deviation (Noise Std). Each point is aggregated by 20 state-based tasks in [PITH_FULL_IMAGE:figures/full_fig_p035_12.png]
Figure 13
Figure 13. Figure 13: Performance on individual state-based tasks. [PITH_FULL_IMAGE:figures/full_fig_p042_13.png]
Figure 14
Figure 14. Figure 14: Performance on individual state-based tasks. [PITH_FULL_IMAGE:figures/full_fig_p042_14.png]
Figure 15
Figure 15. Figure 15: Performance on individual state-based tasks. [PITH_FULL_IMAGE:figures/full_fig_p043_15.png]
Figure 16
Figure 16. Figure 16: Performance on individual state-based tasks. [PITH_FULL_IMAGE:figures/full_fig_p043_16.png]
Figure 17
Figure 17. Figure 17: Performance on individual state-based tasks. [PITH_FULL_IMAGE:figures/full_fig_p044_17.png]
Figure 18
Figure 18. Figure 18: Performance on individual state-based tasks. [PITH_FULL_IMAGE:figures/full_fig_p044_18.png]
Figure 19
Figure 19. Figure 19: Performance on individual state-based tasks. [PITH_FULL_IMAGE:figures/full_fig_p045_19.png]
Figure 20
Figure 20. Figure 20: Performance on individual state-based tasks. [PITH_FULL_IMAGE:figures/full_fig_p045_20.png]
Figure 21
Figure 21. Figure 21: Performance on individual state-based tasks. [PITH_FULL_IMAGE:figures/full_fig_p046_21.png]
Figure 22
Figure 22. Figure 22: Performance on individual pixel-based tasks. [PITH_FULL_IMAGE:figures/full_fig_p046_22.png]
Figure 23
Figure 23. Figure 23: Performance on individual pixel-based tasks. [PITH_FULL_IMAGE:figures/full_fig_p047_23.png]
Figure 24
Figure 24. Figure 24: Performance on individual pixel-based tasks. [PITH_FULL_IMAGE:figures/full_fig_p047_24.png]
Figure 25
Figure 25. Figure 25: Performance on individual pixel-based tasks. [PITH_FULL_IMAGE:figures/full_fig_p048_25.png]
Figure 26
Figure 26. Figure 26: Performance on individual pixel-based tasks. [PITH_FULL_IMAGE:figures/full_fig_p048_26.png]
Figure 27
Figure 27. Figure 27: Performance on individual pixel-based tasks. [PITH_FULL_IMAGE:figures/full_fig_p049_27.png]
Figure 28
Figure 28. Figure 28: Case study on 6 state-based DMC tasks on DBC-normed with LayerNorm and its variants: applying the target trick (DBC-normed (Target)), using Huber loss (DBC-normed (Huber)) instead of MSE as the metric loss, and applying both (DBC-normed (Huber & Target)). X-axis is th…
Figure 29
Figure 29. Figure 29: Case study on six DMC state-based tasks examining the effects of including LayerNorm (the first and third vs. second and fourth rows). X-axis stands for the environmental step. See [PITH_FULL_IMAGE:figures/full_fig_p050_29.png]
Figure 30
Figure 30. Figure 30: Aggregated reward (left) and denoising factor (right) of methods on IID Gaussian noise with random projection setting, varying noise standard deviation, in the 6 selected state-based tasks. 50 [PITH_FULL_IMAGE:figures/full_fig_p050_30.png]
Figure 31
Figure 31. Figure 31: Performance on individual state-based tasks under the IID Gaussian noise with random [PITH_FULL_IMAGE:figures/full_fig_p051_31.png]
Figure 32
Figure 32. Figure 32: Performance on individual state-based tasks under the IID Gaussian noise with random [PITH_FULL_IMAGE:figures/full_fig_p051_32.png]
Figure 33
Figure 33. Figure 33: Performance on individual state-based tasks under the IID Gaussian noise with random [PITH_FULL_IMAGE:figures/full_fig_p051_33.png]
Figure 34
Figure 34. Figure 34: Performance on individual state-based tasks under the IID Gaussian noise with random [PITH_FULL_IMAGE:figures/full_fig_p052_34.png]
Figure 35
Figure 35. Figure 35: Performance on individual state-based tasks under the IID Gaussian noise with random [PITH_FULL_IMAGE:figures/full_fig_p052_35.png]
Figure 36
Figure 36. Figure 36: Reward curves for the isolated metric estimation setting, using the SAC agent with [PITH_FULL_IMAGE:figures/full_fig_p052_36.png]
Figure 37
Figure 37. Figure 37: DF for the agent encoder ϕ (co-trained with RL in Sec. 5.1) without LayerNorm. 52 [PITH_FULL_IMAGE:figures/full_fig_p052_37.png]
Figure 38
Figure 38. Figure 38: Denoising factor curves on six pixel-based DMC tasks under the [PITH_FULL_IMAGE:figures/full_fig_p053_38.png]
Figure 39
Figure 39. Figure 39: Denoising factor curves on six pixel-based DMC tasks under the [PITH_FULL_IMAGE:figures/full_fig_p053_39.png]
Figure 40
Figure 40. Figure 40: Denoising factor curves on six pixel-based DMC tasks under the [PITH_FULL_IMAGE:figures/full_fig_p053_40.png]
Figure 41
Figure 41. Figure 41: Denoising factor curves on six pixel-based DMC tasks under the [PITH_FULL_IMAGE:figures/full_fig_p053_41.png]
Figure 42
Figure 42. Figure 42: Denoising factor curves on six pixel-based DMC tasks under the [PITH_FULL_IMAGE:figures/full_fig_p053_42.png]
Figure 43
Figure 43. Figure 43: Per-task OOD generalization reward curves for pixel-based DMC tasks. [PITH_FULL_IMAGE:figures/full_fig_p054_43.png]
Figure 44
Figure 44. Figure 44: Per-task OOD generalization reward curves for pixel-based DMC tasks. [PITH_FULL_IMAGE:figures/full_fig_p054_44.png]
Figure 45
Figure 45. Figure 45: Per-task OOD generalization reward curves for pixel-based DMC tasks. [PITH_FULL_IMAGE:figures/full_fig_p055_45.png]
Figure 46
Figure 46. Figure 46: Per-task OOD generalization reward curves for pixel-based DMC tasks. [PITH_FULL_IMAGE:figures/full_fig_p055_46.png]
Figure 47
Figure 47. Figure 47: Pixel-based DMC per-task generalization reward difference curves. 0 250 500 750 1000 cartpole/balance cartpole/balance_sparse walker/stand finger/spin cartpole/swingup ball_in_cup/catch 0 250 500 750 1000 walker/walk point_mass/easy cartpole/swingup_sparse reacher/eas…
Figure 48
Figure 48. Figure 48: Pixel-based DMC per-task generalization reward difference curves. 56 [PITH_FULL_IMAGE:figures/full_fig_p056_48.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 47 canonical work pages

  1. [1]

    Contrastive behavioral similarity embeddings for generalization in reinforcement learning

    Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Marc G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. arXiv preprint arXiv:2101.05265, 2021 a

    arXiv 2021

  2. [2]

    Deep reinforcement learning at the edge of the statistical precipice

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34: 0 29304--29320, 2021 b

    work page 2021

  3. [3]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  4. [4]

    Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity

    Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. arXiv preprint arXiv:1902.05605, 2019

    arXiv 1902

  5. [5]

    Online abstraction with mdp homomorphisms for deep learning

    Ondrej Biza and Robert Platt. Online abstraction with mdp homomorphisms for deep learning. arXiv preprint arXiv:1811.12929, 2018

    arXiv 2018

  6. [6]

    Scalable methods for computing state similarity in deterministic markov decision processes

    Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 10069--10076, 2020

    2020

  7. [7]

    Mico: Improved representations via sampling-based state similarity for markov decision processes

    Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. Mico: Improved representations via sampling-based state similarity for markov decision processes. Advances in Neural Information Processing Systems, 34: 0 30113--30126, 2021

    work page 2021

  8. [8]

    A Kernel Perspective on Behavioural Metrics for Markov Decision Processes

    Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. A kernel perspective on behavioural metrics for markov decision processes. arXiv preprint arXiv:2310.19804, 2023

    work page Pith review arXiv 2023

Show all 79 references
  1. [9]

    Learning representations via a robust behavioral metric for deep reinforcement learning

    Jianda Chen and Sinno Pan. Learning representations via a robust behavioral metric for deep reinforcement learning. Advances in Neural Information Processing Systems, 35: 0 36654--36666, 2022

    2022

  2. [10]

    State chrono representation for enhancing generalization in reinforcement learning

    Jianda Chen, Zichen Chen, Sinno Pan, Tianwei Zhang, et al. State chrono representation for enhancing generalization in reinforcement learning. Advances in Neural Information Processing Systems, 37: 0 73309--73336, 2024

    2024

  3. [11]

    Offline reinforcement learning with pseudometric learning

    Robert Dadashi, Shideh Rezaeifar, Nino Vieillard, L \'e onard Hussenot, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning with pseudometric learning. In International Conference on Machine Learning, pp.\ 2307--2318. PMLR, 2021

    2021

  4. [12]

    Bisimulation for labelled markov processes

    Jos \'e e Desharnais, Abbas Edalat, and Prakash Panangaden. Bisimulation for labelled markov processes. Information and Computation, 179 0 (2): 0 163--193, 2002

    2002

  5. [13]

    Provably efficient rl with rich observations via latent state decoding

    Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning, pp.\ 1665--1674. PMLR, 2019

    2019

  6. [14]

    Differential privacy

    Cynthia Dwork. Differential privacy. In International colloquium on automata, languages, and programming, pp.\ 1--12. Springer, 2006

    2006

  7. [15]

    Provable rl with exogenous distractors via multistep inverse dynamics

    Yonathan Efroni, Dipendra Misra, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Provable rl with exogenous distractors via multistep inverse dynamics. arXiv preprint arXiv:2110.08847, 2021

    2021 arXiv

  8. [16]

    Metrics for finite markov decision processes

    Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In Proceedings of the 20th conference on Uncertainty in artificial intelligence, pp.\ 162--169, 2004

    2004

  9. [17]

    Bisimulation metrics for continuous markov decision processes

    Norm Ferns, Prakash Panangaden, and Doina Precup. Bisimulation metrics for continuous markov decision processes. SIAM Journal on Computing, 40 0 (6): 0 1662--1714, 2011

    2011

  10. [18]

    For sale: State-action representation learning for deep reinforcement learning

    Scott Fujimoto, Wei-Di Chang, Edward Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforcement learning. Advances in neural information processing systems, 36: 0 61573--61624, 2023

    2023

  11. [19]

    Deepmdp: Learning continuous latent space models for representation learning

    Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International conference on machine learning, pp.\ 2170--2179. PMLR, 2019

    2019

  12. [20]

    Fully homomorphic encryption using ideal lattices

    Craig Gentry. Fully homomorphic encryption using ideal lattices. In Proceedings of the forty-first annual ACM symposium on Theory of computing, pp.\ 169--178, 2009

    2009

  13. [21]

    Equivalence notions and model minimization in markov decision processes

    Robert Givan, Thomas Dean, and Matthew Greig. Equivalence notions and model minimization in markov decision processes. Artificial intelligence, 147 0 (1-2): 0 163--223, 2003

    2003

  14. [22]

    Measuring visual generalization in continuous control from pixels, 2020

    Jake Grigsby and Yanjun Qi. Measuring visual generalization in continuous control from pixels, 2020

    2020

  15. [23]

    Bootstrap your own latent-a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altch \'e , Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neur...

    2020

  16. [24]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.\ 1861--1870. Pmlr, 2018

    2018

  17. [25]

    Generalization in reinforcement learning by soft data augmentation

    Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmentation. In International Conference on Robotics and Automation, 2021

    2021

  18. [26]

    Td-mpc2: Scalable, robust world models for continuous control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828, 2023

    2023 arXiv

  19. [27]

    Bisimulation makes analogies in goal-conditioned reinforcement learning

    Philippe Hansen-Estruch, Amy Zhang, Ashvin Nair, Patrick Yin, and Sergey Levine. Bisimulation makes analogies in goal-conditioned reinforcement learning. In International Conference on Machine Learning, pp.\ 8407--8426. PMLR, 2022

    2022

  20. [28]

    Dropout q-functions for doubly efficient reinforcement learning

    Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning. arXiv preprint arXiv:2110.02034, 2021

    2021 arXiv

  21. [29]

    Offline rl with observation histories: Analyzing and improving sample complexity

    Joey Hong, Anca Dragan, and Sergey Levine. Offline rl with observation histories: Analyzing and improving sample complexity. arXiv preprint arXiv:2310.20663, 2023

    2023 arXiv

  22. [30]

    Robust estimation of a location parameter

    Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics: Methodology and distribution, pp.\ 492--518. Springer, 1992

    1992

  23. [31]

    Dissecting deep rl with high update ratios: Combatting value divergence

    Marcel Hussing, Claas Voelcker, Igor Gilitschenski, Amir-massoud Farahmand, and Eric Eaton. Dissecting deep rl with high update ratios: Combatting value divergence. arXiv preprint arXiv:2403.05996, 2024

    2024 arXiv

  24. [32]

    Agent-controller representations: Principled offline rl with rich exogenous information

    Riashat Islam, Manan Tomar, Alex Lamb, Yonathan Efroni, Hongyu Zang, Aniket Didolkar, Dipendra Misra, Xin Li, Harm Van Seijen, Remi Tachet des Combes, et al. Agent-controller representations: Principled offline rl with rich exogenous information. arXiv preprint arXiv:2211.00164, 2022

    2022 arXiv

  25. [33]

    Notes on state abstractions, 2018

    Nan Jiang. Notes on state abstractions, 2018

    2018

  26. [34]

    The kinetics human action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017

    2017 arXiv

  27. [35]

    Towards robust bisimulation metric learning

    Mete Kemertas and Tristan Aumentado-Armstrong. Towards robust bisimulation metric learning. Advances in Neural Information Processing Systems, 34: 0 4764--4777, 2021

    2021

  28. [36]

    Actor-critic algorithms

    Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999

    1999

  29. [37]

    On the necessity of abstraction

    George Konidaris. On the necessity of abstraction. Current opinion in behavioral sciences, 29: 0 1--7, 2019

    2019

  30. [38]

    Towards a unified theory of state abstraction for mdps

    Lihong Li, Thomas J Walsh, and Michael L Littman. Towards a unified theory of state abstraction for mdps. AI&M, 1 0 (2): 0 3, 2006

    2006

  31. [39]

    Normalization enhances generalization in visual reinforcement learning

    Lu Li, Jiafei Lyu, Guozheng Ma, Zilin Wang, Zhenjie Yang, Xiu Li, and Zhiheng Li. Normalization enhances generalization in visual reinforcement learning. arXiv preprint arXiv:2306.00656, 2023

    2023 arXiv

  32. [40]

    Does self-supervised learning really improve reinforcement learning from pixels? Advances in Neural Information Processing Systems, 35: 0 30865--30881, 2022

    Xiang Li, Jinghuan Shang, Srijan Das, and Michael Ryoo. Does self-supervised learning really improve reinforcement learning from pixels? Advances in Neural Information Processing Systems, 35: 0 30865--30881, 2022

    2022

  33. [41]

    Policy-independent behavioral metric-based representation for deep reinforcement learning

    Weijian Liao, Zongzhang Zhang, and Yang Yu. Policy-independent behavioral metric-based representation for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 8746--8754, 2023

    2023

  34. [42]

    Robust representation learning by clustering with bisimulation metrics for visual reinforcement learning with distractions

    Qiyuan Liu, Qi Zhou, Rui Yang, and Jie Wang. Robust representation learning by clustering with bisimulation metrics for visual reinforcement learning with distractions. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp.\ 8843--8851, 2023

    2023

  35. [43]

    A calculus of communicating systems

    Robin Milner. A calculus of communicating systems. Springer, 1980

    1980

  36. [44]

    Learning temporal distances: Contrastive successor features can provide a metric structure for decision-making

    Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, and Benjamin Eysenbach. Learning temporal distances: Contrastive successor features can provide a metric structure for decision-making. arXiv preprint arXiv:2406.17098, 2024

    2024 arXiv

  37. [45]

    Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning

    Michal Nauman, Micha Bortkiewicz, Piotr Mi o \'s , Tomasz Trzci \'n ski, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. arXiv preprint arXiv:2403.00514, 2024

    2024 arXiv

  38. [46]

    Bridging state and history representations: Understanding self-predictive rl

    Tianwei Ni, Benjamin Eysenbach, Erfan Seyedsalehi, Michel Ma, Clement Gehring, Aditya Mahajan, and Pierre-Luc Bacon. Bridging state and history representations: Understanding self-predictive rl. arXiv preprint arXiv:2401.08898, 2024

    2024 arXiv

  39. [47]

    Control-oriented model-based reinforcement learning with implicit differentiation

    Evgenii Nikishin, Romina Abachi, Rishabh Agarwal, and Pierre-Luc Bacon. Control-oriented model-based reinforcement learning with implicit differentiation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 7886--7894, 2022

    2022

  40. [48]

    Labelled Markov Processes

    Prakash Panangaden. Labelled Markov Processes. Imperial College Press, 2009

    2009

  41. [49]

    Policy gradient methods in the presence of symmetries and state abstractions

    Prakash Panangaden, Sahand Rezaei-Shoshtari, Rosie Zhao, David Meger, and Doina Precup. Policy gradient methods in the presence of symmetries and state abstractions. Journal of Machine Learning Research, 25 0 (71): 0 1--57, 2024

    2024

  42. [50]

    Concurrency and automata on infinite sequences

    David Park. Concurrency and automata on infinite sequences. In Theoretical Computer Science: 5th GI-Conference Karlsruhe, March 23-25, 1981, volume 104, pp.\ 167. Springer, 1981

    1981

  43. [51]

    State-action similarity-based representations for off-policy evaluation

    Brahma Pavse and Josiah Hanna. State-action similarity-based representations for off-policy evaluation. Advances in Neural Information Processing Systems, 36: 0 42298--42329, 2023

    2023

  44. [52]

    An algebraic approach to abstraction in reinforcement learning

    Balaraman Ravindran. An algebraic approach to abstraction in reinforcement learning. University of Massachusetts Amherst, 2004

    2004

  45. [53]

    Model minimization in hierarchical reinforcement learning

    Balaraman Ravindran and Andrew G Barto. Model minimization in hierarchical reinforcement learning. In Abstraction, Reformulation, and Approximation: 5th International Symposium, SARA 2002 Kananaskis, Alberta, Canada August 2--4, 2002 Proceedings 5, pp.\ 196--211. Springer, 2002

    2002

  46. [54]

    Continuous mdp homomorphisms and homomorphic policy gradient

    Sahand Rezaei-Shoshtari, Rosie Zhao, Prakash Panangaden, David Meger, and Doina Precup. Continuous mdp homomorphisms and homomorphic policy gradient. Advances in Neural Information Processing Systems, 35: 0 20189--20204, 2022

    2022

  47. [55]

    Learning action-based representations using invariance

    Max Rudolph, Caleb Chuck, Kevin Black, Misha Lvovsky, Scott Niekum, and Amy Zhang. Learning action-based representations using invariance. arXiv preprint arXiv:2403.16369, 2024

    2024 arXiv

  48. [56]

    Facenet: A unified embedding for face recognition and clustering

    Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 815--823, 2015

    2015

  49. [57]

    Data-efficient reinforcement learning with self-predictive representations

    Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929, 2020

    2007 arXiv

  50. [58]

    Bisimulation metric for model predictive control

    Yutaka Shimizu and Masayoshi Tomizuka. Bisimulation metric for model predictive control. arXiv preprint arXiv:2410.04553, 2024

    2024 arXiv

  51. [59]

    Reinforcement learning with soft state aggregation

    Satinder Singh, Tommi Jaakkola, and Michael Jordan. Reinforcement learning with soft state aggregation. Advances in neural information processing systems, 7, 1994

    1994

  52. [60]

    A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning

    Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.07860, 2022

    2022 arXiv

  53. [61]

    The distracting control suite--a challenging benchmark for reinforcement learning from pixels

    Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting control suite--a challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02722, 2021

    2021 arXiv

  54. [62]

    Approximate information state for approximate planning and reinforcement learning in partially observed systems

    Jayakumar Subramanian, Amit Sinha, Raihan Seraj, and Aditya Mahajan. Approximate information state for approximate planning and reinforcement learning in partially observed systems. Journal of Machine Learning Research, 23 0 (12): 0 1--83, 2022

    2022

  55. [63]

    Deepmind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  56. [64]

    Lax probabilistic bisimulation

    Jonathan Taylor. Lax probabilistic bisimulation. 2008

    2008

  57. [65]

    Learning representations for pixel-based control: What matters and why? arXiv preprint arXiv:2111.07775, 2021

    Manan Tomar, Utkarsh A Mishra, Amy Zhang, and Matthew E Taylor. Learning representations for pixel-based control: What matters and why? arXiv preprint arXiv:2111.07775, 2021

    2021 arXiv

  58. [66]

    dm\_control: Software and tasks for continuous control

    Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm\_control: Software and tasks for continuous control. Software Impacts, 6: 0 100022, 2020

    2020

  59. [67]

    Plannable approximations to mdp homomorphisms: Equivariance under actions

    Elise Van der Pol, Thomas Kipf, Frans A Oliehoek, and Max Welling. Plannable approximations to mdp homomorphisms: Equivariance under actions. arXiv preprint arXiv:2002.11963, 2020 a

    2002 arXiv

  60. [68]

    Mdp homomorphic networks: Group symmetries in reinforcement learning

    Elise Van der Pol, Daniel Worrall, Herke van Hoof, Frans Oliehoek, and Max Welling. Mdp homomorphic networks: Group symmetries in reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 4199--4210, 2020 b

    2020

  61. [69]

    When does self-prediction help? understanding auxiliary tasks in reinforcement learning

    Claas Voelcker, Tyler Kastner, Igor Gilitschenski, and Amir-massoud Farahmand. When does self-prediction help? understanding auxiliary tasks in reinforcement learning. arXiv preprint arXiv:2406.17718, 2024

    2024 arXiv

  62. [70]

    Efficient potential-based exploration in reinforcement learning using inverse dynamic bisimulation metric

    Yiming Wang, Ming Yang, Renzhi Dong, Binbin Sun, Furui Liu, et al. Efficient potential-based exploration in reinforcement learning using inverse dynamic bisimulation metric. Advances in Neural Information Processing Systems, 36: 0 38786--38797, 2023

    2023

  63. [71]

    Rethinking exploration in reinforcement learning with effective metric-based exploration bonus

    Yiming Wang, Kaiyan Zhao, Furui Liu, et al. Rethinking exploration in reinforcement learning with effective metric-based exploration bonus. Advances in Neural Information Processing Systems, 37: 0 57765--57792, 2024

    2024

  64. [72]

    Mastering visual continuous control: Improved data-augmented reinforcement learning

    Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021 a

    2021 arXiv

  65. [73]

    Improving sample efficiency in model-free reinforcement learning from images

    Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the aaai conference on artificial intelligence, volume 35, pp.\ 10674--10681, 2021 b

    2021

  66. [74]

    Rl-vigen: A reinforcement learning benchmark for visual generalization

    Zhecheng Yuan, Sizhe Yang, Pu Hua, Can Chang, Kaizhe Hu, and Huazhe Xu. Rl-vigen: A reinforcement learning benchmark for visual generalization. Advances in Neural Information Processing Systems, 36: 0 6720--6747, 2023

    2023

  67. [75]

    Simsr: Simple distance-based state representations for deep reinforcement learning

    Hongyu Zang, Xin Li, and Mingzhong Wang. Simsr: Simple distance-based state representations for deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp.\ 8997--9005, 2022

    2022

  68. [76]

    Understanding and addressing the pitfalls of bisimulation-based representations in offline reinforcement learning

    Hongyu Zang, Xin Li, Leiji Zhang, Yang Liu, Baigui Sun, Riashat Islam, Remi Tachet des Combes, and Romain Laroche. Understanding and addressing the pitfalls of bisimulation-based representations in offline reinforcement learning. Advances in Neural Information Processing Syste...

    2023

  69. [77]

    Natural environment benchmarks for reinforcement learning

    Amy Zhang, Yuxin Wu, and Joelle Pineau. Natural environment benchmarks for reinforcement learning. arXiv preprint arXiv:1811.06032, 2018

    2018 arXiv

  70. [78]

    Learning invariant representations for reinforcement learning without reconstruction

    Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.10742, 2020

    2006 arXiv

  71. [79]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.