Pith. sign in

REVIEW 12 cited by

Data-Efficient Reinforcement Learning with Self-Predictive Representations

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2007.05929 v4 pith:GWL3RHIK submitted 2020-07-12 cs.LG stat.ML

Max Schwarzer , Ankesh Anand , Rishab Goel , R Devon Hjelm , Aaron Courville , Philip Bachman This is my paper
classification cs.LGstat.ML
keywords futurerepresentationsagentdatainteractionenvironmentlearninglimited
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

While deep reinforcement learning excels at solving tasks where large amounts of data can be collected through virtually unlimited interaction with the environment, learning from limited interaction remains a key challenge. We posit that an agent can learn more efficiently if we augment reward maximization with self-supervised objectives based on structure in its visual input and sequential interaction with the environment. Our method, Self-Predictive Representations(SPR), trains an agent to predict its own latent state representations multiple steps into the future. We compute target representations for future states using an encoder which is an exponential moving average of the agent's parameters and we make predictions using a learned transition model. On its own, this future prediction objective outperforms prior methods for sample-efficient deep RL from pixels. We further improve performance by adding data augmentation to the future prediction loss, which forces the agent's representations to be consistent across multiple views of an observation. Our full self-supervised objective, which combines future prediction and data augmentation, achieves a median human-normalized score of 0.415 on Atari in a setting limited to 100k steps of environment interaction, which represents a 55% relative improvement over the previous state-of-the-art. Notably, even in this limited data regime, SPR exceeds expert human scores on 7 out of 26 games. The code associated with this work is available at https://github.com/mila-iqia/spr

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Self-Predictive Dynamics for Generalization of Vision-based Reinforcement Learning

    cs.CV 2025-06 conditional novelty 7.0 of 10

    A self-supervised auxiliary loss combining weak and strong augmentations, an adversarial discriminator, and inverse-then-forward latent dynamics improves both data efficiency and zero-shot generalization in vision-based RL.

  2. Understanding Behavioral Metric Learning: A Large-Scale Study on Distracting Reinforcement Learning Environments

    cs.LG 2025-05 conditional novelty 7.0 of 10

    Across noisy DeepMind Control tasks, explicit bisimulation-metric losses add little denoising benefit beyond plain self-prediction and feature normalization, which dominate performance.

  3. PAMD: Structured Adaptive Distances for Bisimulation Representations in Visual Reinforcement Learning

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A pairwise-adaptive Mahalanobis distance, parameterized as a positive-definite quadratic form, improves representation learning and control performance when plugged into bisimulation-based visual RL algorithms.

  4. Human-Like Goalkeeping in a Realistic Football Simulation: a Sample-Efficient Reinforcement Learning Approach

    cs.AI 2025-10 conditional novelty 6.0 of 10

    A sample-efficient SAC-based method with replay-ratio resets, offline data bootstrapping, and expert-driven fine-tuning produces a goalkeeper that outperforms the built-in AI in EA SPORTS FC 25.

  5. Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A constrained rational activation with denominator degree one larger than numerator and no constant term stabilizes high-UTD continuous control, while trading off long-term plasticity.

  6. Universal Value-Function Uncertainties

    cs.LG 2025-05 conditional novelty 6.0 of 10

    UVU measures value-function uncertainty by the TD-trained prediction error between an online network and a fixed random target, and in the infinite-width NTK limit this error exactly matches ensemble variance.

  7. Hadamax Encoding: Elevating Performance in Model-Free Atari

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Hadamax, a Hadamard-product and max-pooling encoder, improves PQN's median human-normalized Atari-57 score by about 80% with no algorithmic changes.

  8. TAPO: Transition-Aware Policy Optimization for LLM Agents

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Interleaving action-conditioned next-observation supervision with group RL on a shared LLM backbone consistently lifts long-horizon agent success over pure policy optimization.

  9. Sample-Efficient Reinforcement Learning Controller for Deep Brain Stimulation in Parkinson's Disease

    cs.LG 2025-07 reject novelty 5.0 of 10

    A DDPG-based adaptive DBS controller with a predictive reward model and Gumbel-Softmax exploration suppresses beta power faster than standard DDPG in simulation and survives FP16 quantization.

  10. rQdia: Regularizing Q-Value Distributions With Image Augmentation

    cs.LG 2025-06 reject novelty 5.0 of 10

    rQdia regularizes Q-value distributions across augmented images, reporting improvements on continuous control and Atari benchmarks over DrQ, SAC, and Data-Efficient Rainbow.

  11. Self-Supervised Evolution Operator Learning for High-Dimensional Dynamical Systems

    cs.LG 2025-05 conditional novelty 5.0 of 10

    A contrastive self-supervised loss is shown to be equivalent to learning the evolution operator's spectral decomposition, recovering slow modes in proteins, ligand binding, and ENSO climate data.

  12. A Survey of State Representation Learning for Deep Reinforcement Learning

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A six-class taxonomy of state representation learning methods for model-free online deep reinforcement learning, with selection guidelines, evaluation metrics, and future directions.

Pith tools