Pith. sign in

REVIEW 4 major objections 5 minor 93 references

Zero-Shot Reinforcement Learning Under Partial Observability

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Memory models restore zero-shot RL when states are only partially observed.

desk verdict Solid empirical first step on memory-augmented zero-shot RL under partial observability, but the trajectory-level FB theory is assumed rather than proven and the Cheetah results undercut the abstract's overclaim. read the letter →

arxiv 2506.15446 v1 pith:SVXXI4RI submitted 2025-06-18 cs.LG cs.AI

Scott Jeen , Tom Bewley , Jonathan M. Cullen This is my paper · ORCID
classification cs.LGcs.AI
keywords zero-shotreinforcementlearningpartialobservabilityforward-backwardrepresentationssuccessormeasuresmemorymodelsPOMDPGRUdynamicsgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks what happens to zero-shot reinforcement learning when the agent no longer sees the full Markov state, and shows that standard behavior foundation models degrade sharply through two failure routes: they misidentify the state when the forward model and policy receive partial observations, and they misidentify the task when the backward model receives partial observations. To fix this, the paper adds memory models—gated recurrent units (GRUs) that compress a length-L history of observations and actions—to each of the forward model, backward model, and policy, and trains the forward-backward objective over trajectories instead of states. If the paper is right, the memory-augmented version, FB-M, recovers most of the lost zero-shot performance: near-oracle behavior on Quadruped, better-than-baseline scores on most partially observed Walker tasks, and gains over memory-free methods when dynamics change at test time. This matters because partial observability is the rule in real-world deployment, not the exception.

What carries the argument

The central object is the memory-augmented forward-backward representation, in which three gated recurrent units (GRUs) f_F, f_B, and f_π turn length-L observation-action histories into hidden states, and the successor measure over trajectories is factorized as F(f_F(τ_0^L), z)^T B(f_B(τ^L)), with the policy choosing actions by maximizing F(f_π(τ^L), z)^T z. This dot product is what lets the agent both predict long-run occupancy (through F) and infer the task (through B), and the paper's empirical claim is that this factorization survives the move from states to memory hidden states, with the GRU hidden state acting as a learned approximate Markov state.

What would settle it

In a small partially observed gridworld where the true trajectory occupancy can be computed exactly, train FB-M and compare its predicted successor measure F(f_F(τ_0^L), z)^T B(f_B(τ^L)) against the Monte Carlo occupancy of the learned policy; if the two diverge substantially while returns stay high, the memory-based extension is helping for reasons other than the claimed mechanism.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the failure of zero-shot reinforcement learning under partial observability can be traced to two identifiable mechanisms—state misidentification, where the forward model and policy receive observations that do not reveal the underlying Markov state, and task misidentification, where the backward model cannot recover the task embedding from partial observations. The paper's contribution is to replace the state arguments of the forward model F, backward model B, and policy π with the hidden states of separate memory models f_F, f_B, and f_π that compress length-L trajectories of observations and actions, and to assert that the forward-backward training objective survives this replacement: the trajectory-level successor measure $M^{{π_z}}$(τ_0^L, X) is approximated by ∫_X F(f_F(τ_0^L), z)^T B(f_B(τ^L)) ρ(dτ^L), and the policy maximizes F(f_π(τ^L), z)^T z. By evaluating against memory-free FB, the stacked-observation FB-stack, and HILP on ExORL benchmarks amended with noisy states, flickering states, and changed dynamics, the paper reports that FB-M improves over all baselines in most settings, approaches oracle performance on Quadruped, and slightly exceeds the MDP-trained oracle in the extrapolation-dynamics case on Cheetah and Quadruped.

Load-bearing premise

The paper assumes, without proof, that the forward-backward guarantee still holds when the successor measure is computed from GRU hidden states rather than true Markov states—that is, that the dot product of the memory-compressed forward and backward embeddings accurately approximates the trajectory-level occupancy, and that the Q function this dot product defines is a valid critic for the policy.

Editorial extensions

If this is right

  • FB-M outperforms memory-free FB and the stacking baseline on almost all partially observed state settings, reaching close to oracle performance on Quadruped.
  • On dynamics changes at test time, FB-M matches or beats all baselines in every environment, including the extrapolation regime where it can slightly exceed the MDP-trained oracle.
  • The proposed memory-based formulation applies equally to USF-based behavior foundation models, not just FB.
  • GRU memory models outperform transformer and S4d memory models in this setting, and using non-GRU memory for both forward and backward models causes training collapse.
  • Stacking recent observations (FB-stack) is an inconsistent remedy: it helps on Walker and Quadruped partially observed states but hurts on Cheetah and in most dynamics-change settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the trajectory-level FB guarantee holds, then in environments where a bounded observation history is a sufficient statistic of the hidden state, a suitable context length L should recover full-observability performance asymptotically, suggesting a practical rule: set L to the mixing time of the observation process.
  • The training collapse when both F and B use non-GRU memory hints that the factorization M ≈ F(f_F)^T B(f_B) is especially fragile to poorly conditioned hidden-state representations; probing the condition number or spectral properties of the hidden-state embeddings during training could predict such collapses before they happen.
  • A natural testable extension is to vary the context lengths of the forward and backward memories independently for tasks whose rewards depend on shorter histories than their dynamics, which the paper notes but does not evaluate.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper extends behaviour foundation models (BFMs), specifically the forward-backward (FB) representation, to partially observable RL by inserting GRU memory models into the forward model, backward model, and policy, producing FB-M (and a USF variant in Appendix B). It identifies state misidentification and task misidentification as the two failure mechanisms, proposes trajectory-level successor-measure equations (Eq. 5) and a z-inference rule (Eq. 6), and evaluates FB-M against FB, HILP, FB-stack, and an MDP oracle on noisy/flickering ExORL (Walker, Cheetah, Quadruped) and on dynamics interpolation/extrapolation. The main empirical finding is that FB-M improves over memory-free baselines in several settings, especially Quadruped and dynamics extrapolation, while the GRU compares favorably to transformer/S4 memory models; the paper also reports instability when both F and B use non-GRU memories.

Significance. If the empirical findings hold, this is a useful and timely contribution: it is, to my knowledge, the first systematic study of zero-shot RL under partial observability, it repurposes a standard single-task remedy (memory models) in a sensible way, and it ships open-source code. The evaluation is generally careful: 5 seeds, IQM aggregation, bootstrap confidence intervals, per-task tables, and an MDP oracle are all present. However, the theoretical grounding of the trajectory-level FB equations is assumed rather than proven, the reported benefits are not uniform across domains (Cheetah is a clear counterexample to the abstract's blanket wording), and the checkpoint-selection protocol may inflate absolute performance. The paper's value is therefore mostly empirical, and it would be strengthened by a more cautious framing and by additional validation of the successor-measure approximation.

major comments (4)
  1. [Section 4.3, Eq. (5), and Section 6.1] The paper asserts that the FB guarantee of Touati and Ollivier transfers from Markov states to GRU hidden states, but no proof or formal condition is given. This is load-bearing because Eq. (5) defines the method. Furthermore, Section 6.1 reports that when both F and B use non-GRU memory models, training collapses and states that 'the combined representation M(...) ≈ F(f_F(τ))^T B(f_B(τ+)) is degenerate.' That observation is in direct tension with the assumed validity of Eq. (5) for GRU memory models. Please either provide conditions under which Eq. (5) holds for memory-based embeddings, or explicitly reframe the contribution as an empirical adaptation and soften the theoretical framing. A concrete diagnostic would be to compare F(f_F(τ0),z)^T B(f_B(τ)) against Monte Carlo estimates of the trajectory-level successor measure on held-out trajectories.
  2. [Abstract and Table 3] The abstract states that the proposed methods 'show improved performance over memory-free baselines.' Table 3 contradicts this for Cheetah: in the flickering setting FB-M scores 173±51 vs 182±25 for memory-free FB, and in the noisy setting 150±59 vs 213±53. The paper's own Section 5.2 correctly avoids claiming an FB-M advantage over FB on Cheetah, but the abstract and conclusion do not. Please qualify the claim by domain or provide an aggregate statistical test (e.g., a paired bootstrap over tasks and seeds) that supports the blanket statement.
  3. [Section 5.1, Evaluation protocol] The protocol selects, for each method, the checkpoint at which the all-task IQM aggregated across seeds is maximized. This is a form of test-time model selection and can inflate absolute scores; it also makes the 'near-oracle' claims in Section 5.2 (e.g., Quadruped flickering, 673±19 vs oracle 729±6) hard to interpret. Since all methods use the same rule, relative rankings may be roughly preserved, but I ask for a sensitivity check with a fixed training budget (e.g., the final checkpoint) to confirm the main conclusions, and a statement of how much the selection rule affects reported numbers.
  4. [Equation (6)] The task-embedding inference rule z_test ≈ E[R(s) B(f_B(τ))] is stated without justification. In standard FB, Eq. (3) is justified by the assumption that the reward lies in the span of the backward embeddings B(s). For trajectories, the reward depends on the terminal Markov state s_L, and it is not shown that R(s_L) is well approximated as a linear function of B(f_B(τ)) across the trajectory distribution. Please provide a derivation or at least an empirical check (e.g., compare z inferred via Eq. (6) with the oracle z used at training time, or verify that reward prediction error is small on held-out labelled trajectories).
minor comments (5)
  1. [Abstract and Section 5.1] There are several typos: 'partially observability' should be 'partial observability', 'MuJoCu' should be 'MuJoCo', 'comparision' should be 'comparison', and 'evaled' should be 'evaluated'.
  2. [Section 5.2] The sentence 'Our proposed approach (FB-M) outperforms this baseline in all settings except Walker where it performs similarly' is ambiguous because the previous sentence mentions both FB-stack and memory-free FB; 'this baseline' should be explicitly identified as FB-stack.
  3. [Section A.1] The text says 'We consider 3 environments (three locomotion and one goal-directed)' but Table 1 lists only Walker, Quadruped, and Cheetah, all classified as locomotion. Please correct the mismatch.
  4. [Section C.3] The caption of Figure 8 says 'standard FP architecture' and the text says 'a standard FP architecture'; these should be 'FB'.
  5. [Section 6.1] The phrase 'a fixed hidden state size of 32^2 = 1024 dimensions' is an unusual notation; simply saying 1024 dimensions would be clearer.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the memory-augmented FB equations are an asserted extension, not a derivation from their own conclusion, and the empirical claims are evaluated against external memory-free baselines and an MDP oracle.

full rationale

The paper's central object, FB-M, is obtained by amending the standard FB formulation to condition F, B, and pi on GRU hidden states of observation-action trajectories (Equation 5). This is an extension of Touati and Ollivier's FB guarantee to a trajectory-level successor measure, but the paper does not claim to derive this extension from the original guarantee; it asserts the approximation and then tests it empirically. No fitted parameter is renamed as a prediction: the task embeddings in Equation 6 are inferred from reward-labelled trajectories using the same linear-in-B rule as standard FB, and the zero-shot tasks are held out from pre-training. The reported hyperparameter choices (sigma, p_flick, context length L) were selected on Walker variants, but the main comparative claims are about relative performance across all tested environments, not about constants that are then re-predicted. The citation to the authors' prior work (Jeen et al. 2024) is used for low-quality-data extensions and batch-size choices, and it is not load-bearing for the central claim that memory models mitigate state and task misidentification. Section 6.1's report that the combined F/B representation degenerates with non-GRU memory models is a genuine limitation on the generality of the claim, but it is a validity concern, not a circularity: the paper does not use that degeneracy as an input to define or force its main result. The claim that the FB guarantee survives replacement of states by GRU hidden states is unproven, but an unproven assumption is different from a conclusion that reduces by definition or by fitted input to its own premises. On the evidence quoted in the manuscript, the derivations are self-contained and the empirical evaluation is externally anchored, so the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No invented physical entities are introduced. The main ledger entries are the unproven trajectory-level FB objective in Eq.5, the coverage assumption on RND data, and three experimental hyperparameters fixed by small sweeps. No free parameter is fit to the held-out test tasks, so the zero-shot evaluation itself is not circular.

free parameters (3)
  • noise variance sigma_noise = 0.2
    Severity of state noise in noisy-state experiments; chosen via a Walker-only sweep in Appendix A.2 and applied to all domains.
  • flicker probability pflick = 0.2
    Probability of zeroing a state in flickering experiments; chosen via a Walker-only sweep in Appendix A.2.
  • context length L = 32
    History length for memory models; selected by sweeping L in {2, 4, 8, 16, 32} in Appendix C.2 and used for both forward and backward models.
assumptions (4)
  • ad hoc to paper The FB successor-measure training objective and z-inference rule remain valid when states are replaced by GRU hidden states.
    Eq.5 is written as an approximation claim, but the paper gives no proof that the trajectory-level FB objective inherits the convergence guarantees of Touati and Ollivier. This is the central unproved premise of the method.
  • domain assumption RND-collected offline datasets provide enough coverage for zero-shot generalization on all tasks.
    Section 5.1 trains on ExORL RND datasets or self-collected RND data; the authors acknowledge in Section 6.2 that poor coverage would break inference and generalization.
  • domain assumption A GRU memory model with context length 32 can compress observation-action trajectories into hidden states that adequately approximate the Markov state.
    Section 4.2 and Appendix C.2 motivate this with prior literature and Walker-only sweeps, but the approximation quality is not measured directly.
  • domain assumption The MuJoCo/DeepMind Control Suite locomotion domains with dense rewards are representative of the partially observable zero-shot RL setting.
    The benchmark choice in Section 5.1 limits the claim to simulated continuous-control locomotion; extrapolation to visual or real-world POMDPs is untested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zero-Shot Reinforcement Learning Under Partial Observability." pith.science (2026). https://pith.science/paper/SVXXI4RI

@misc{pith2026250615446,
  author       = {Pith},
  title        = {Pith review of: Zero-Shot Reinforcement Learning Under Partial Observability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SVXXI4RI}},
  note         = {Machine review of arXiv:2506.15446}
}
read the original abstract

Recent work has shown that, under certain assumptions, zero-shot reinforcement learning (RL) methods can generalise to any unseen task in an environment after reward-free pre-training. Access to Markov states is one such assumption, yet, in many real-world applications, the Markov state is only partially observable. Here, we explore how the performance of standard zero-shot RL methods degrades when subjected to partially observability, and show that, as in single-task RL, memory-based architectures are an effective remedy. We evaluate our memory-based zero-shot RL methods in domains where the states, rewards and a change in dynamics are partially observed, and show improved performance over memory-free baselines. Our code is open-sourced via: https://enjeeneer.io/projects/bfms-with-memory/.

Figures

Figures reproduced from arXiv: 2506.15446 by the authors.

Figure 1
Figure 1. BFMs with memory. In the case of FB, the forward model F and backward model B condition on the output of memory models that compress trajectories of observations and actions. According to standard FB theory, their dot product predicts Mπz (τ L t , τ L + ), the successor measure from L-length trajectory τ L t to L-length future trajectory τ L + , from which a Q function can be derived [PITH_FULL_IMAGE:figures/full_f… view at source ↗
Figure 2
Figure 2. The failure modes of BFMs under partial observability. FB’s average (IQM) all-task return on Walker when observations are passed to its respective components. Observations are created by adding Gaussian noise to the underlying states. (Left) Observations are passed as input to B causing FB to misidentify the task. (Middle) Observations are passed as input to F and π causing FB to misidentify the state. (Right) Obser… view at source ↗
Figure 3
Figure 3. Aggregate zero-shot task performance on ExORL with partially observed states. IQM of task scores across all tasks on noisy and flickering variants of Walker, Cheetah and Quadruped, normalised against the performance of FB in the fully observed environment. 5 random seeds. Memory model We use a GRU as our memory model (Cho, 2014). GRUs are the most performant memory model on POPGym (Morad et al., 2023) which tests pa… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Aggregate zero-shot task performance on ExORL with changed dynamics at test time. IQM of task scores across all tasks when trained on dynamics where mass and damping coefficients are scaled to {0.5×, 1.5×} their usual values and evaled on {1.0×, 2.0×} their usual value…
Figure 5
Figure 5. Figure 5: Aggregate zero-shot task performance of FB-M with different memory models. IQM of task scores across all tasks on Walker flickering. (Left) Observations are passed only to a memory-based back￾ward model; the forward model and policy are memory-free. (Middle) Observatio…
Figure 6
Figure 6. Figure 6: POMDP hyperparameter sweep. We evaluate the performance of standard FB on Walker when the states are noised according to σ ∈ {0.05, 0.1, 0.2} and dropped according to pf ∈ {0.05, 0.1, 0.2}. B Universal Successor Features with Memory USFs require access to a feature map…
Figure 7
Figure 7. Figure 7: Hyperparameter sweep over context length L. We evaluate the performance of FB-M with GRU memory model on Walker noisy ((a) and (c) and Walker flickering ((b) and (d)). When we sweep over the forward model’s context length, we pass states to the backward model and keep …
Figure 8
Figure 8. Figure 8: BFMs without memory. FB is optimised in a standard actor critic setup (Konda & Tsitsiklis, 1999). The policy π selects an action at conditioned on the current observation ot, and the task vector z. The Q function formed by the USF ψ evaluates the action at given the cu…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

93 extracted references · 28 canonical work pages

  1. [1]

    Deep reinforcement learning at the edge of the statistical precipice

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34: 0 29304--29320, 2021

    2021

  2. [2]

    Hindsight experience replay

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017

    2017

  3. [3]

    Rudder: Return decomposition for delayed rewards

    Jose A Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. Advances in Neural Information Processing Systems, 32, 2019

    2019

  4. [4]

    Optimal control of markov processes with incomplete state information

    Karl Johan str \"o m. Optimal control of markov processes with incomplete state information. Journal of mathematical analysis and applications, 10 0 (1): 0 174--205, 1965

    1965

  5. [5]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  6. [6]

    Reinforcement learning with long short-term memory

    Bram Bakker. Reinforcement learning with long short-term memory. Advances in neural information processing systems, 14, 2001

    2001

  7. [7]

    Augmented world models facilitate zero-shot dynamics generalization from a single offline environment

    Philip J Ball, Cong Lu, Jack Parker-Holder, and Stephen Roberts. Augmented world models facilitate zero-shot dynamics generalization from a single offline environment. In International Conference on Machine Learning, pp.\ 619--629. PMLR, 2021

    2021

  8. [8]

    Successor features for transfer in reinforcement learning

    Andr \'e Barreto, Will Dabney, R \'e mi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017

    2017

Show all 93 references
  1. [9]

    Learning successor states and goal-dependent values: A mathematical viewpoint

    L \'e onard Blier, Corentin Tallec, and Yann Ollivier. Learning successor states and goal-dependent values: A mathematical viewpoint. arXiv preprint arXiv:2101.07123, 2021

    2021 arXiv

  2. [10]

    Universal successor features approximators

    Diana Borsa, Andr \'e Barreto, John Quan, Daniel Mankowitz, R \'e mi Munos, Hado Van Hasselt, David Silver, and Tom Schaul. Universal successor features approximators. arXiv preprint arXiv:1812.07626, 2018

    2018 arXiv

  3. [11]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  4. [12]

    Acting optimally in partially observable stochastic domains

    Anthony R Cassandra, Leslie Pack Kaelbling, and Michael L Littman. Acting optimally in partially observable stochastic domains. In AAAI, volume 94, pp.\ 1023--1028, 1994

    1994

  5. [13]

    Learning phrase representations using rnn encoder-decoder for statistical machine translation

    Kyunghyun Cho. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014

    2014 arXiv

  6. [14]

    Quantifying generalization in reinforcement learning

    Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International conference on machine learning, pp.\ 1282--1289. PMLR, 2019

    2019

  7. [15]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 0 16344--16359, 2022

    2022

  8. [16]

    Improving generalization for temporal difference learning: The successor representation

    Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural computation, 5 0 (4): 0 613--624, 1993

    1993

  9. [17]

    Facing off world model backbones: Rnns, transformers, and s4

    Fei Deng, Junyeong Park, and Sungjin Ahn. Facing off world model backbones: Rnns, transformers, and s4. Advances in Neural Information Processing Systems, 36, 2023

    2023

  10. [18]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  11. [19]

    Finding structure in time

    Jeffrey L Elman. Finding structure in time. Cognitive science, 14 0 (2): 0 179--211, 1990

    1990

  12. [20]

    Contrastive learning as goal-conditioned reinforcement learning

    Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35: 0 35603--35620, 2022

    2022

  13. [21]

    Generalization and regularization in dqn

    Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv:1810.00123, 2018

    2018 arXiv

  14. [22]

    Hyperbolic discounting and learning over multiple horizons

    William Fedus, Carles Gelada, Yoshua Bengio, Marc G Bellemare, and Hugo Larochelle. Hyperbolic discounting and learning over multiple horizons. arXiv preprint arXiv:1902.06865, 2019

    1902 arXiv

  15. [23]

    A minimalist approach to offline reinforcement learning

    Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34: 0 20132--20145, 2021

    2021

  16. [24]

    Off-policy deep reinforcement learning without exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp.\ 2052--2062. PMLR, 2019

    2019

  17. [25]

    Amago: Scalable in-context reinforcement learning for adaptive agents

    Jake Grigsby, Linxi Fan, and Yuke Zhu. Amago: Scalable in-context reinforcement learning for adaptive agents. International Conference on Learning Representations, 2023

    2023

  18. [26]

    Amago-2: Breaking the multi-task barrier in meta-reinforcement learning with transformers

    Jake Grigsby, Justin Sasek, Samyak Parajuli, Daniel Adebi, Amy Zhang, and Yuke Zhu. Amago-2: Breaking the multi-task barrier in meta-reinforcement learning with transformers. Advances in Neural Information Processing Systems, 2024

    2024

  19. [27]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R \'e . Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021

    2021 arXiv

  20. [28]

    On the parameterization and initialization of diagonal state space models

    Albert Gu, Karan Goel, Ankit Gupta, and Christopher R \'e . On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 35: 0 35971--35983, 2022

    2022

  21. [29]

    World models

    David Ha and J \"u rgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018

    2018 arXiv

  22. [30]

    Dream to control: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019 a

    1912 arXiv

  23. [31]

    Learning latent dynamics for planning from pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pp.\ 2555--2565. PMLR, 2019 b

    2019

  24. [32]

    Mastering atari with discrete world models

    Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020

    2010 arXiv

  25. [33]

    Mastering diverse domains through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023

    2023 arXiv

  26. [34]

    Contextual markov decision processes, 2015

    Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes, 2015

    2015

  27. [35]

    Array programming with numpy

    Charles R Harris, K Jarrod Millman, St \'e fan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. Array programming with numpy. Nature, 585 0 (7825): 0 357--362, 2020

    2020

  28. [36]

    Deep recurrent q-learning for partially observable mdps

    Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series, 2015

    2015

  29. [37]

    Memory-based control with recurrent neural networks

    Nicolas Heess, Jonathan J Hunt, Timothy P Lillicrap, and David Silver. Memory-based control with recurrent neural networks. arXiv preprint arXiv:1512.04455, 2015

    2015 arXiv

  30. [38]

    Long short-term memory

    Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural computation, 9 0 (8): 0 1735--1780, 1997

    1997

  31. [39]

    Matplotlib: A 2d graphics environment

    John D Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9 0 (03): 0 90--95, 2007

    2007

  32. [40]

    Scott Jeen, Tom Bewley, and Jonathan M. Cullen. Zero-shot reinforcement learning from low quality data. Advances in Neural Information Processing Systems 38, 2024

    2024

  33. [41]

    Monotonic robust policy optimization with model discrepancy

    Yuankun Jiang, Chenglin Li, Wenrui Dai, Junni Zou, and Hongkai Xiong. Monotonic robust policy optimization with model discrepancy. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine ...

    2021

  34. [42]

    Planning and acting in partially observable stochastic domains

    Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101 0 (1-2): 0 99--134, 1998

    1998

  35. [43]

    Morel: Model-based offline reinforcement learning

    Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33: 0 21810--21823, 2020

    2020

  36. [44]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  37. [45]

    Actor-critic algorithms

    Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999

    1999

  38. [46]

    Stabilizing off-policy q-learning via bootstrapping error reduction

    Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019 a

    2019

  39. [47]

    Stabilizing off-policy q-learning via bootstrapping error reduction

    Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019 b

    2019

  40. [48]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779, 2020

    2006 arXiv

  41. [49]

    Batch reinforcement learning

    Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-art, pp.\ 45--73. Springer, 2012

    2012

  42. [50]

    Context-aware dynamics model for generalization in model-based reinforcement learning

    Kimin Lee, Younggyo Seo, Seunghyun Lee, Honglak Lee, and Jinwoo Shin. Context-aware dynamics model for generalization in model-based reinforcement learning. In International Conference on Machine Learning, pp.\ 5757--5766. PMLR, 2020

    2020

  43. [51]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  44. [52]

    Off-policy policy gradient with state distribution correction

    Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. arXiv preprint arXiv:1904.08473, 2019

    1904 arXiv

  45. [53]

    Structured state space models for in-context reinforcement learning

    Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. Structured state space models for in-context reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  46. [54]

    How far i'll go: Offline goal-conditioned reinforcement learning via f -advantage regression

    Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i'll go: Offline goal-conditioned reinforcement learning via f -advantage regression. arXiv preprint arXiv:2206.03023, 2022

    2022 arXiv

  47. [55]

    Robust reinforcement learning for continuous control with model misspecification

    Daniel J Mankowitz, Nir Levine, Rae Jeong, Yuanyuan Shi, Jackie Kay, Abbas Abdolmaleki, Jost Tobias Springenberg, Timothy Mann, Todd Hester, and Martin Riedmiller. Robust reinforcement learning for continuous control with model misspecification. arXiv preprint arXiv:1906.07516, 2019

    1906 arXiv

  48. [56]

    pandas: a foundational python library for data analysis and statistics

    Wes McKinney et al. pandas: a foundational python library for data analysis and statistics. Python for high performance and scientific computing, 14 0 (9): 0 1--9, 2011

    2011

  49. [57]

    Memory-based deep reinforcement learning for pomdps

    Lingheng Meng, Rob Gorbet, and Dana Kuli \'c . Memory-based deep reinforcement learning for pomdps. In 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp.\ 5619--5626. IEEE, 2021

    2021

  50. [58]

    Steps toward artificial intelligence

    Marvin Minsky. Steps toward artificial intelligence. Proceedings of the IRE, 49 0 (1): 0 8--30, 1961

    1961

  51. [59]

    Human-level control through deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518 0 (7540): 0 529--533, 2015

    2015

  52. [60]

    Popgym: Benchmarking partially observable reinforcement learning

    Steven Morad, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Prorok. Popgym: Benchmarking partially observable reinforcement learning. arXiv preprint arXiv:2303.01859, 2023

    2023 arXiv

  53. [61]

    Robust reinforcement learning

    Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17 0 (2): 0 335--359, 2005

    2005

  54. [62]

    Recurrent model-free rl can be a strong baseline for many pomdps

    Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free rl can be a strong baseline for many pomdps. arXiv preprint arXiv:2110.05038, 2021

    2021 arXiv

  55. [63]

    Robust control of markov decision processes with uncertain transition matrices

    Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53 0 (5): 0 780--798, 2005

    2005

  56. [64]

    a henb \

    Charles Packer, Katelyn Gao, Jernej Kos, Philipp Kr \"a henb \"u hl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv:1810.12282, 2018

    2018 arXiv

  57. [65]

    Stabilizing transformers for reinforcement learning

    Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International conference on machine learning, pp.\ 7487--...

    2020

  58. [66]

    Hiql: Offline goal-conditioned rl with latent states as actions

    Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal-conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 36, 2024 a

    2024

  59. [67]

    Foundation policies with hilbert representations

    Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. International Conference on Machine Learning, 2024 b

    2024

  60. [68]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017

    2017

  61. [69]

    Fast imitation via behavior foundation models

    Matteo Pirotta, Andrea Tirinzoni, Ahmed Touati, Alessandro Lazaric, and Yann Ollivier. Fast imitation via behavior foundation models. In International Conference on Learning Representations, 2024

    2024

  62. [70]

    Automatic data augmentation for generalization in deep reinforcement learning

    Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006.12862, 2020

    2006 arXiv

  63. [71]

    Epopt: Learning robust neural network policies using model ensembles

    Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv:1610.01283, 2016

    2016 arXiv

  64. [72]

    Synthetic returns for long-term credit assignment

    David Raposo, Sam Ritter, Adam Santoro, Greg Wayne, Theophane Weber, Matt Botvinick, Hado van Hasselt, and Francis Song. Synthetic returns for long-term credit assignment. arXiv preprint arXiv:2102.12425, 2021

    2021 arXiv

  65. [73]

    Reward-free curricula for training robust world models

    Marc Rigter, Minqi Jiang, and Ingmar Posner. Reward-free curricula for training robust world models. arXiv preprint arXiv:2306.09205, 2023

    2023 arXiv

  66. [74]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10684--10695, 2022

    2022

  67. [75]

    Python: a programming language for software integration and development

    Michel F Sanner et al. Python: a programming language for software integration and development. J Mol Graph Model, 17 0 (1): 0 57--61, 1999

    1999

  68. [76]

    Universal value function approximators

    Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pp.\ 1312--1320. PMLR, 2015

    2015

  69. [77]

    Reinforcement learning upside down: Don't predict rewards--just map them to actions

    Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards--just map them to actions. arXiv preprint arXiv:1912.02875, 2019

    1912 arXiv

  70. [78]

    Reinforcement learning in markovian and non-markovian environments

    J \"u rgen Schmidhuber. Reinforcement learning in markovian and non-markovian environments. Advances in neural information processing systems, 3, 1990

    1990

  71. [79]

    Trajectory-wise multiple choice learning for dynamics generalization in reinforcement learning

    Younggyo Seo, Kimin Lee, Ignasi Clavera Gilaberte, Thanard Kurutach, Jinwoo Shin, and Pieter Abbeel. Trajectory-wise multiple choice learning for dynamics generalization in reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 12968--12979, 2020

    2020

  72. [80]

    Temporal credit assignment in reinforcement learning

    Richard Stuart Sutton. Temporal credit assignment in reinforcement learning. University of Massachusetts Amherst, 1984

    1984

  73. [81]

    Deepmind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  74. [82]

    Learning robot soccer from egocentric vision with deep reinforcement learning

    Dhruva Tirumala, Markus Wulfmeier, Ben Moran, Sandy Huang, Jan Humplik, Guy Lever, Tuomas Haarnoja, Leonard Hasenclever, Arunkumar Byravan, Nathan Batchelor, et al. Learning robot soccer from egocentric vision with deep reinforcement learning. arXiv preprint arXiv:2405.02425, 2024

    2024 arXiv

  75. [83]

    Domain randomization for transferring deep neural networks from simulation to the real world

    Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp.\ 23--3...

    2017

  76. [84]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 5026--5033. IEEE, 2012

    2012

  77. [85]

    Learning one representation to optimize all rewards

    Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34: 0 13--23, 2021

    2021

  78. [86]

    Does zero-shot reinforcement learning exist? In The Eleventh International Conference on Learning Representations, 2023

    Ahmed Touati, J \'e r \'e my Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In The Eleventh International Conference on Learning Representations, 2023

    2023

  79. [87]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  80. [88]

    Active perception and reinforcement learning

    Steven D Whitehead and Dana H Ballard. Active perception and reinforcement learning. In Machine Learning Proceedings 1990, pp.\ 179--188. Elsevier, 1990

    1990

  81. [89]

    Policy gradient critics

    Daan Wierstra and J \"u rgen Schmidhuber. Policy gradient critics. In European Conference on Machine Learning, pp.\ 466--477. Springer, 2007

    2007

  82. [90]

    Meta-gradient reinforcement learning with an objective discovered online

    Zhongwen Xu, Hado P van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and David Silver. Meta-gradient reinforcement learning with an objective discovered online. Advances in Neural Information Processing Systems, 33: 0 15254--15264, 2020

    2020

  83. [91]

    Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning

    Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201.13425, 2022

    2022 arXiv

  84. [92]

    Learning deep neural network policies with continuous memory states

    Marvin Zhang, Zoe McCarthy, Chelsea Finn, Sergey Levine, and Pieter Abbeel. Learning deep neural network policies with continuous memory states. In 2016 IEEE international conference on robotics and automation (ICRA), pp.\ 520--527. IEEE, 2016

    2016

  85. [93]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.