Policy-shaped prediction: avoiding distractions in model-based reinforcement learning

Isaac Kauvar; Miles Hutson; Nick Haber

REVIEW 5 major objections 6 minor 17 references

Policy-shaped prediction: avoiding distractions in model-based reinforcement learning

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Policy-shaped world models shrug off learnable distractions

desk verdict Worth reading for the Reafferent benchmark and the segmentation-weighted saliency idea, but the paper overclaims attribution: the adversarial head alone reproduces most of the robustness gain on their main result, and a value-gradient variant beats the full method. read the letter →

arxiv 2412.05766 v1 pith:O3Z3M4MB submitted 2024-12-08 cs.LG cs.AI

Miles Hutson , Isaac Kauvar , Nick Haber This is my paper · ORCID

classification cs.LGcs.AI

keywords model-basedreinforcementlearningworldmodelsdistractionrobustnesspolicy-gradientsaliencyimagesegmentationadversarialactionpredictionDreamerV3ReafferentDeepmindControl

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Policy-Shaped Prediction (PSP) argues that a reconstruction-based world model wastes capacity when it devotes pixels to content that is predictable but useless for choosing actions, and that this waste can be fixed by telling the model which pixels its policy actually looks at. The paper shows that weighting the DreamerV3 image-reconstruction loss by object-aggregated policy gradients, plus an adversarial head that stops the encoder from re-encoding the previous action, lets an agent keep a faithful model of the task while ignoring learnable background distractions. On the new Reafferent Deepmind Control Suite, PSP scores 383.1 versus 158.4 for DreamerV3 on Cheetah Run and obtains a nonzero Hopper Stand mean score where every model-based baseline is near zero, though that Hopper mean comes from one of three seeds. If correct, the method gives a recipe for pointing a world model's finite capacity at what matters, without changing the agent at inference time.

What carries the argument

Three components carry PSP. First, policy-gradient loss weighting: the scalar action sum $a = \sum_j a_j$ is differentiated with respect to input pixels to form $\partial a/\partial x_i$, which scales each pixel's reconstruction error (Eq. 1). Second, segmentation aggregation: a pretrained segmenter (SAM, or SAM2 variants) masks the image into objects, and each pixel weight becomes the mean absolute gradient inside its object, with a 99th-percentile clip and a 0.9/0.1 interpolation toward a uniform weight so the model still learns background early (Eqs. 2-3). Third, adversarial action prediction head: an MLP predicts the previous action from the encoder's embedding, and its scaled gradient ($\epsilon = 10^3$) is subtracted from the world-model update, so the encoder has no incentive to store information about its own past action (Eqs. 4-5). These modifications apply only during training, leaving the inference-time DreamerV3 policy unchanged.

What would settle it

Train PSP on Reafferent Cheetah Run with per-segment loss weights replaced by random permutations of the true weights, keeping segmentation and the adversarial head intact: if the score stays near 383, the policy-gradient signal is not the cause of the gain, while a drop toward DreamerV3's 158 would confirm the salience weighting is doing the work.

Watch

Extended reading notes

Core claim

The central discovery is that the gradient of the policy with respect to input pixels, once averaged inside segmentation masks, is a stable enough salience signal to reshape a world model's learning objective. PSP multiplies the per-pixel reconstruction loss of DreamerV3 by this policy-shaped weight, so the model must reproduce pixels the policy depends on even if the agent is small and uncertain, while the adversarial action prediction head removes the redundant reconstruction of self-generated visual changes. The paper reports that this combination achieves roughly twice DreamerV3's score on Reafferent Cheetah Run (383.1 vs 158.4), is the only model-based method tested with a nonzero mean on Reafferent Hopper Stand (128.5 vs at most 4.6 for the model-based baselines), stays within the normal range on unmodified Deepmind Control, and outperforms DreamerV3 on Distracting Control Hopper Stand (417.7 vs 173.7).

Load-bearing premise

The whole method rests on the assumption that the policy's pixel gradients, averaged within segmented objects, mark the parts of the scene the world model should spend its capacity on, and that this marking stays trustworthy while the policy and world model are being learned together.

Editorial extensions

If this is right

PSP roughly doubles DreamerV3's score on Reafferent Cheetah Run (383.1 vs 158.4) and is the only tested model-based method with a nonzero mean on Reafferent Hopper Stand.
On unmodified Deepmind Control, PSP (Cheetah 712.3, Hopper 865.6) stays within the range of DreamerV3 and the leading baselines, so distraction suppression does not come at the cost of clean-environment performance.
On Distracting Control Suite, PSP outperforms DreamerV3 on both Cheetah Run (408.6 vs 243.8) and Hopper Stand (417.7 vs 173.7).
The method's modifications are training-only, so a PSP-trained agent runs the standard DreamerV3 inference pipeline with no extra segmentation or salience computation.
Segmentation quality matters most on hard tasks: switching from SAM to SAM2-tiny preserves Cheetah performance but loses ground on Hopper, and SAM2-large recovers most of the lost Distracting-Control score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the salience signal is the load-bearing part, a natural test is to shuffle the per-segment weights: random weights should severely degrade Reafferent Cheetah Run, while keeping the rest of PSP intact would show whether the policy-gradient signal is doing the work.
The same recipe could transfer to other reconstruction-based world models and to model-free agents that learn a state encoder from pixels, since nothing in the weighting or adversarial head depends on DreamerV3's RSSM specifics.
The adversarial head's fixed scale ($\epsilon = 10^3$) is a hyperparameter the paper does not sweep; on tasks with higher-dimensional action spaces the gradient subtraction may need to be larger or learned, and sweeping it would show whether the efference-copy mechanism is robust.
Because segmentation is computed once per collected image and stored in the replay buffer, the method's cost can be shifted off the training loop entirely; using video segmentation models for temporal aggregation is a likely next step that could handle dynamic object boundaries.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Thanks for the report. I read the paper carefully. My take: it is a solid submission with a genuinely useful new benchmark, but the central claim—that policy-shaped prediction is what drives the robustness—does not hold up under the ablations. The full method works, but the attribution is shaky.

What's actually new and good: the Reafferent DeepMind Control benchmark is a nice stress test. Distractors are learnable, deterministic functions of action and time, so they are perfectly predictable but useless to the policy. That directly targets the failure mode where reconstruction-based world models waste capacity. The method itself is a reasonable combination: policy-gradient saliency weighted reconstruction, SAM-based object aggregation to stabilize the noisy gradients, and an adversarial action-prediction head. The segmentation aggregation is a sensible fix, and I give credit for testing SAM2 as an alternative. On Reafferent Cheetah Run, full PSP clearly beats DreamerV3 (383 vs 158) and most baselines. On Hopper Stand, PSP gets a nonzero mean where all model-based baselines are near zero, though only one of three seeds succeeds and the variance is huge.

The soft spots are real. The abstract claims PSP "outperforms a variety of other approaches" and a 2x improvement, but Table 1 shows TIA and DreamerPro beat PSP on individual Distracting Control tasks. More importantly, the ablations in Table 2 undermine the attribution of the robustness result. The adversarial-head-only variant (None, no segmentation, no policy gradient) scores 324.4 on Reafferent Cheetah, already 2x DreamerV3, and the value-gradient-only variant scores 445.7, higher than full PSP's 383.1. The paper acknowledges the trade-off but never tests these simpler variants on Hopper or Distracting Control, so we don't know whether the full method is needed there. The missing code link (appendix says "this GitHub Repository" with no URL) is a reproducibility issue.

The paper's own limitations section is honest, and the authors clearly know the method has an object-centric bias and compute costs. But the overclaim in the abstract and the untested ablations need fixing.

Who this is for: anyone working on MBRL robustness, representation learning, or visual distraction. It deserves a serious referee because the benchmark and the method are worth engaging with, but the authors should qualify the claims, run the key ablations on all environments, and provide a working code link.

Recommendation: send to peer review with a request for major revision.

Referee Report

5 major / 6 minor

Summary. The paper proposes Policy-Shaped Prediction (PSP), a modification of DreamerV3 that reweights the image reconstruction loss with segmentation-aggregated gradients of the policy with respect to input pixels, and adds an adversarial action-prediction head intended to prevent the image encoder from wasting capacity on self-generated action information. It introduces Reafferent DeepMind Control, a new benchmark whose distracting backgrounds are deterministic functions of the previous action and time, and reports that PSP outperforms DreamerV3, DreamerPro, Task Informed Abstractions, Denoised MDPs, and DrQv2 on this benchmark while maintaining comparable performance on unmodified and standard Distracting Control tasks. The paper includes ablations on Cheetah Run, experiments with SAM and SAM2 segmentation backbones, and an appendix documenting computational overhead and resource usage.

Significance. If the attribution claim held, PSP would be a useful contribution to distraction-robust model-based reinforcement learning, and the Reafferent DMC benchmark is a well-motivated testbed for learnable distractors. The paper makes a good-faith effort to report compute costs, includes a code repository, and tests multiple segmentation backbones, which are strengths. However, the headline 2x robustness gain is not cleanly attributable to the policy-shaped prediction mechanism: the ablations in Table 2 show that an adversarial-head-only variant reproduces most of the gain on the main benchmark, and a value-gradient-only variant scores even higher. The Hopper Stand evidence is also statistically thin, with only one of three seeds succeeding. These issues need to be addressed before the central claims can be accepted.

major comments (5)

[Section 3.4, Table 2] The claim that the PSP combination yields the 2x robustness improvement is not supported by the ablations on the main benchmark. The adversarial-head-only variant ('None ✗ ✓') scores 324.4 ± 2.3 on Reafferent Cheetah Run, already 2.05x DreamerV3's 158.4 ± 45.7, and the value-gradient-only variant ('Value ✗ ✗') scores 445.7 ± 126.9, which is higher than full PSP's 383.1 ± 23.8. The narrative that segmentation-based aggregation is 'critical' is also not evident from these rows, since the policy-gradient-only variant without segmentation or adversarial head ('Policy ✗ ✗') scores 379.0, close to full PSP's 383.1. The abstract and contribution list should be rewritten to attribute the gain to the adversarial head and/or value-gradient weighting, or the full method must be shown to beat these components on the relevant environments.
[Section 3.4] The ablation study is limited to Cheetah Run, so the paper never establishes that the full PSP configuration is necessary for the Hopper Stand or Distracting Control results. Given the Cheetah ablations, the Hopper Stand outcome could be driven by a component other than policy-shaped prediction. Table 1's 'unmatched performance' claim is therefore not yet supported for those environments; at minimum, the missing ablation rows should be run or the claim should be restricted to the environments where the full method is actually compared against its components.
[Table 1, Reafferent Hopper Stand] The Reafferent Hopper Stand result is statistically weak: PSP's mean is 128.5 with a standard deviation of 215.7, and the text and Table 3 indicate that only one of three seeds succeeds. With n=3, this is not a reliable demonstration of 'scores beyond the reach of any of the baselines.' The paper should report individual seeds, provide additional runs, or explicitly qualify the claim as based on a single successful seed.
[Section 2.2-2.3] The method depends on several constants that receive no sensitivity analysis: the interpolation factor alpha = 0.9 in Eq. (3), the 99th-percentile clipping of the raw salience map, and the adversarial gradient scale epsilon = 1e3. The paper explicitly invokes alpha to avoid a chicken-and-egg failure where neither the policy nor the world model is useful, so it is load-bearing to know whether PSP is robust to reasonable variations of these values. A small sensitivity study or at least a discussion of the chosen values would strengthen the paper.
[Appendix C] The comparison between PSP and the baselines involves a substantial compute disparity: PSP trials use 4 A40 GPUs for training plus 4 A40 GPUs for segmentation, while baselines run on a single A40 GPU or a desktop 2070 SUPER. Since the baselines use 'default hyperparameters' and no tuning budget is reported, the resource asymmetry could confound the headline improvement. The paper should discuss whether the baselines were given comparable compute or tuning, or should temper the comparison accordingly.

minor comments (6)

[Section 2.1, Eq. (1)] The symbol 'a' is overloaded: it denotes the action vector in 'a = E(pi(s))' and the scalar sum 'a = sum_j a_j'. Please use distinct notation for the vector and the scalar.
[Algorithm 1, line 12] The update 'phi <- Adam(gradient(L) - epsilon * derivative(L_adv_head)/derivative(rho), phi)' is confusing because the subtracted term is written as a derivative with respect to rho while the parameter being updated is phi. Please clarify that the adversarial gradient is applied only through the encoder parameters and specify exactly which parameters receive the subtracted gradient.
[Tables 1 and 3] The SAM condition on Reafferent Hopper Stand is reported as 128.5 ± 215.7 in Table 1 and 130.3 ± 214.1 in Table 3. Please reconcile the discrepancy or explain why the numbers differ.
[Section 3.4] The sentence 'segmentation-based aggregation is critical to improving our model's performance amid distractors' is not clearly supported by Table 2, where the policy-gradient-only variant without segmentation achieves 379.0 on Reafferent Cheetah Run. Please soften this claim or provide the missing comparison that demonstrates the critical role of segmentation.
[Abstract and Section 1] The unqualified statement that PSP 'outperforms a variety of other approaches' is too strong given the high variance on Hopper Stand and the ablation results in Table 2. Please qualify the claim to reflect the environments and seeds for which the advantage is consistent.
[Throughout] There are minor typographical issues, including 'paramaterized' in Algorithm 1 and inconsistent spelling of 'VaGraM'/'VaGram' in Section 4. Please proofread.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central robustness claims are empirical comparisons against external baselines, and no prediction reduces to a fitted input or to a self-citation.

full rationale

PSP's central claims are empirical: on Reafferent Cheetah Run PSP scores 383.1 ± 23.8 versus DreamerV3's 158.4 ± 45.7, with results reported alongside external baselines (DreamerV3, DreamerPro, TIA, Denoised MDP, DrQv2) on standard DMC and Distracting Control Suite. The method's components—policy-gradient weighting, SAM-based aggregation, and the adversarial action head—are implemented as explicit training modifications (Eqs. 1–5); no parameter is fitted to the reported benchmark and then renamed a prediction. The ablation table (Table 2) even reports variants that outperform full PSP on the reafferent task, which cuts against any claim that the headline result is forced by construction. There are no load-bearing self-citations: the reference list contains no works by the present authors. The only mildly self-referential element is that the salience-map visualizations in Fig. 5 display exactly the weights defined in Eq. 2 and used to re-weight the reconstruction loss, so they do not independently confirm task-relevance; this is a limitation of the interpretability evidence, not a circular derivation of any performance claim.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central empirical claim depends on two hand-tuned scalars (alpha, epsilon), a clipping choice, and several domain assumptions about salience, segmentation quality, and the nature of the benchmark. No new physical entities are introduced.

free parameters (3)

alpha (loss interpolation factor) = 0.9
In Eq. 3, the salience weighting is linearly interpolated with a uniform weighting using alpha = 0.9 for all experiments; no sensitivity sweep is reported.
epsilon (adversarial gradient scale) = 1e3
In Section 2.3, the gradient of the action-prediction loss is subtracted from the world model gradient scaled by epsilon = 1e3; no ablation is reported.
Salience clipping percentile = 99th percentile
Section 2.2 clips the raw salience map to the 99th percentile before aggregation; chosen by hand.

assumptions (4)

domain assumption Policy gradient w.r.t. input pixels is a valid salience signal for task relevance
Invoked in Section 2.1 to justify Eq. 1; the authors rely on the gradient of scalarized action a w.r.t. image x to identify useful pixels.
domain assumption SAM segmentation provides object boundaries aligned with task-relevant entities
Section 2.2 uses SAM masks to average gradient weights; the method assumes the agent and important objects are segmented correctly.
domain assumption Subtracting the action-prediction head gradient removes duplicate action information without harming other encoder features
Section 2.3 uses a fixed epsilon=1e3; the paper does not demonstrate that useful information is preserved.
domain assumption Reafferent backgrounds are entirely predictable from (time, previous action) and useless for policy
Section 3.2 constructs the benchmark with 2500 backgrounds mapped from 625 time bins and 4 action bins; this is a design assumption about what counts as a distraction.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Policy-shaped prediction: avoiding distractions in model-based reinforcement learning." pith.science (2026). https://pith.science/paper/O3Z3M4MB

@misc{pith2026241205766,
  author       = {Pith},
  title        = {Pith review of: Policy-shaped prediction: avoiding distractions in model-based reinforcement learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O3Z3M4MB}},
  note         = {Machine review of arXiv:2412.05766}
}

read the original abstract

Model-based reinforcement learning (MBRL) is a promising route to sample-efficient policy optimization. However, a known vulnerability of reconstruction-based MBRL consists of scenarios in which detailed aspects of the world are highly predictable, but irrelevant to learning a good policy. Such scenarios can lead the model to exhaust its capacity on meaningless content, at the cost of neglecting important environment dynamics. While existing approaches attempt to solve this problem, we highlight its continuing impact on leading MBRL methods -- including DreamerV3 and DreamerPro -- with a novel environment where background distractions are intricate, predictable, and useless for planning future actions. To address this challenge we develop a method for focusing the capacity of the world model through synergy of a pretrained segmentation model, a task-aware reconstruction loss, and adversarial learning. Our method outperforms a variety of other approaches designed to reduce the impact of distractors, and is an advance towards robust model-based reinforcement learning.

Figures

Figures reproduced from arXiv: 2412.05766 by the authors.

**Figure 2.** Schematic of the Reafferent Deepmind Control environment. The distracting background is entirely predictable based on the agent’s previous action and the elapsed time in the episode. In the natural world, distractions can be highly complex, but in many cases are also highly predictable. For instance, the creaking sound a rusty bicycle makes as you pedal, or the movement of your own shadow as you dance outside. We w… view at source ↗

**Figure 3.** Training curve comparisons on Reafferent Deepmind Control. Mean [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Reconstructed image comparison, PSP vs. DreamerV3 on Reafferent Cheetah Run, same episode and time point. True, reconstructed, difference (true - recon.). DreamerV3 accurately reproduces the background but not the cheetah. We find that none of the baseline MBRL meth…

**Figure 5.** Figure 5: Example salience maps (policyshaped loss weights) highlight the agent. Notably, the model-free DrQv2 agent shows a reduction in performance from its previous performance on the unmodified environment, but overall demonstrates quite robust performance ( [PITH_FULL_IMA…

**Figure 6.** Figure 6: Training curve comparison on Distracting Control. Mean [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 2 canonical work pages

[4]

Hafner, J

D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104,

arXiv
[10]

N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714,

arXiv
[11]

Simonyan, A

K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034,

arXiv
[13]

URL http://arxiv.org/abs/1706.03825. A. Stone, O. Ramirez, K. Konolige, and R. Jonschkowski. The distracting control suite–a challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02722,

arXiv
[14]

Tassa, Y

Y . Tassa, Y . Doron, A. Muldal, T. Erez, Y . Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690,

arXiv
[16]

T. Wang, S. S. Du, A. Torralba, P. Isola, A. Zhang, and Y . Tian. Denoised mdps: Learning world models better than the world itself. arXiv preprint arXiv:2206.15477,

arXiv
[17]

Z. Wang, Y . Ze, Y . Sun, Z. Yuan, and H. Xu. Generalizable visual reinforcement learning with segment anything model. arXiv preprint arXiv:2312.17116,

arXiv
[18]

Yarats, R

12 D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021a. D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Reinforcement learning with prototypical rep- resentations. In International Conference on Machine Learning , pages 11920–11931. PMLR...

arXiv

Show all 17 references

[300]

We believe this level of resource consumption could be easily reduced

Baseline trials could be run on only a single A40 GPU or a desktop NVIDIA 2070 SUPER, usually in less than a day, and accounted for a comparably negligible level of resources. We believe this level of resource consumption could be easily reduced. The modifications to the Dream...
[2017]

URL http://arxiv.org/abs/1705.06950. A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, et al. Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026,

arXiv
[2018]

Hafner, T

D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603,

1912 arXiv
[2019]

Hafner, T

D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193,

2010 arXiv
[2020]

Grooten, G

B. Grooten, G. Sokar, S. Dohare, E. Mocanu, M. E. Taylor, M. Pechenizkiy, and D. C. Mocanu. Automatic noise filtering with dynamic sparse training in deep reinforcement learning. arXiv preprint arXiv:2302.06548, 2023a. B. Grooten, T. Tomilin, G. Vasan, M. E. Taylor, A. R. Mahm...

arXiv
[2021]

Kaiser, M

L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv:1903.00374,

1903 arXiv
[2022]

Lambert, B

N. Lambert, B. Amos, O. Yadan, and R. Calandra. Objective mismatch in model-based reinforcement learning. arXiv preprint arXiv:2002.04523,

2002 arXiv
[2023]

11 A. Lamb, R. Islam, Y . Efroni, A. Didolkar, D. Misra, D. Foster, L. Molu, R. Chari, A. Krishnamurthy, and J. Langford. Guaranteed discovery of control-endogenous latent states with multi-step inverse models. arXiv preprint arXiv:2207.08229,

arXiv
[2024]

V oelcker, V

C. V oelcker, V . Liao, A. Garg, and A.-m. Farahmand. Value gradient weighted model-based rein- forcement learning. arXiv preprint arXiv:2204.01464,

arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [4]

Hafner, J

D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104,

arXiv

[2] [10]

N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714,

arXiv

[3] [11]

Simonyan, A

K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034,

arXiv

[4] [13]

URL http://arxiv.org/abs/1706.03825. A. Stone, O. Ramirez, K. Konolige, and R. Jonschkowski. The distracting control suite–a challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02722,

arXiv

[5] [14]

Tassa, Y

Y . Tassa, Y . Doron, A. Muldal, T. Erez, Y . Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690,

arXiv

[6] [16]

T. Wang, S. S. Du, A. Torralba, P. Isola, A. Zhang, and Y . Tian. Denoised mdps: Learning world models better than the world itself. arXiv preprint arXiv:2206.15477,

arXiv

[7] [17]

Z. Wang, Y . Ze, Y . Sun, Z. Yuan, and H. Xu. Generalizable visual reinforcement learning with segment anything model. arXiv preprint arXiv:2312.17116,

arXiv

[8] [18]

Yarats, R

12 D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021a. D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Reinforcement learning with prototypical rep- resentations. In International Conference on Machine Learning , pages 11920–11931. PMLR...

arXiv