Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Slow large models can still act in realtime if inference runs staggered across parallel processes.

desk verdict Worth a serious referee: the regret decomposition and staggering algorithms are a real step for realtime RL, but the delay-regret term in Theorem 1 does not match the paper's primary Algorithm 1. read the letter →

arxiv 2412.14355 v1 pith:AL3YEW2E submitted 2024-12-18 cs.LG cs.AI

Matthew Riemer , Gopeshh Subbaraj , Glen Berseth , Irina Rish This is my paper · ORCID
classification cs.LGcs.AI
keywords realtimereinforcementlearningasynchronousinferencestaggeredprocessesdelayedsemi-MDPregretdecompositioninteractionfrequencylarge-scaleRLinaction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the usual way of deploying reinforcement learning agents—infer an action, send it, wait for the next state—makes large, slow models structurally unable to control realtime environments. Its central claim is that regret from simply not acting while a model computes grows with model inference time and never decays as time passes, while regret from acting on stale information is governed only by how unpredictable the environment is over the inference horizon. The paper proves this by decomposing realtime regret into learning, inaction, and delay terms, and it introduces two staggered asynchronous inference algorithms that trade parallel compute for interaction frequency. Empirically, the approach keeps a billion-parameter policy competitive in realtime Game Boy games where sequential interaction with the same model collapses to random behavior.

What carries the argument

The load-bearing object is the induced delayed semi-MDP $\tilde{\mathcal{M}}_{\mathrm{delay}}$ (Definition 1), which recasts any choice of environment step time $T_M$, interaction time $T_I$, and policy inference time $T_\theta$ as a semi-MDP whose actions are delayed by $\lceil \tau_\theta/\tau_M\rceil$ steps, during which the default policy $\beta$ governs. The regret decomposition of Theorem 1 then separates the unavoidable cost of learning, the cost of falling back to $\beta$ during inference, and the cost of acting on stale states. The two staggering algorithms are the constructive part: they space $N_I$ inference processes evenly (Algorithm 1 by the maximum observed inference time, Algorithm 2 by the running mean) so that the expected gap between actions shrinks linearly in $N_I$ until it reaches the environment step time.

What would settle it

Run a policy in a realtime environment while logging per-state inference times and the reward the default policy earns during each inference; if inference takes longest precisely on states where the default policy is worst, the accumulated inaction regret should exceed the independence-based formula from Theorem 1, contradicting its predictions.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is Theorem 1: for an asynchronous MDP with a default behavior policy $\beta$, total realtime regret decomposes into $\Delta_{\mathrm{learn}}(\tau)+\Delta_{\mathrm{inaction}}(\tau)+\Delta_{\mathrm{delay}}(\tau)$. In the worst case, sequential interaction leaves $\Delta_{\mathrm{inaction}}(\tau) \in \Omega((\tau/\bar{\tau}_\theta)(\bar{\tau}_\theta-\bar{\tau}_M)/\bar{\tau}_M)$, a per-unit-time regret that never vanishes as $\tau\to\infty$, whereas the delay term is $\Omega((\tau/\bar{\tau}_I)\mathbb{E}[1-(p_{\mathrm{minimax}})^{\lceil\tau_\theta/\tau_M\rceil}])$, so it disappears in deterministic environments and grows only with environment stochasticity. Algorithms 1 and 2 stagger $N_I$ inference processes so that the effective interaction time becomes $\bar{\tau}_I \le \min(\tau_{\max}^{\theta}/N_I,\bar{\tau}_M)$ or $\bar{\tau}_I=\min(\bar{\tau}_\theta/N_I,\bar{\tau}_M)$; once $N_I$ reaches $\lceil \tau_{\max}^{\theta}/\bar{\tau}_M\rceil$ or $\lceil\bar{\tau}_\theta/\bar{\tau}_M\rceil$, inaction regret disappears and only delay regret remains. This is what makes models orders of magnitude larger than the environment's frame time usable, provided the environment is not too stochastic over the inference horizon.

Load-bearing premise

The proof treats environment step time, interaction time, and inference time as independent random variables, so if a state's difficulty makes inference take longer (or if inference time depends on the action sampled), the expected counts behind the regret bounds no longer factor as written and the guarantees can fail.

Editorial extensions

If this is right

  • A model with arbitrarily long inference time can maintain full interaction frequency in a realtime environment, as long as enough staggered processes are available and the environment is sufficiently deterministic over the inference horizon.
  • The compute requirement to eliminate inaction grows only linearly with mean (or maximum) inference time, so doubling model size roughly doubles the processes needed rather than making realtime control impossible.
  • Asynchronous learning with round-robin parameter updates can also keep learning from every transition without blocking action inference, separating learning throughput from model size.
  • Sequential interaction is not just slow but structurally lossy for large models: its per-unit-time inaction regret has a positive limit that no amount of additional training time removes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This suggests a practical deployment rule: measure the environment's stochasticity over the expected inference horizon, and if it is small, spend processes rather than shrinking the model to achieve realtime control.
  • The independence assumption behind the regret bounds points to a stress test: policies with adaptive computation times, where harder states take longer to process, could concentrate inaction on the very states where the default policy is worst, and would need a version of the theory that couples $T_\theta$ to the state.
  • The same staggering logic could transfer to other latency sources in the action loop, such as communication or sensor preprocessing, giving a linear-scaling recipe for any pipeline stage whose delay is independent of the state encountered.
  • Designing a safe default behavior $\beta$ becomes a first-class design choice rather than an innocent fallback, since inaction regret is measured against exactly what $\beta$ does while the model computes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies realtime reinforcement learning in which the environment does not pause while the agent computes actions. It introduces an induced delayed semi-MDP formalism, proposes a three-term regret decomposition into learning, inaction, and delay terms, and argues that sequential interaction incurs worst-case inaction regret that does not vanish as time grows when inference is slower than the environment step. The paper then proposes two staggered asynchronous inference algorithms and claims that with enough processes the inaction term can be eliminated, leaving only a delay term that depends on the environment's stochasticity over the inference horizon. The empirical section reports large-scale experiments on simulated Game Boy and Atari environments, showing that staggered asynchronous inference with 1B-parameter networks substantially outperforms sequential interaction baselines.

Significance. The paper addresses a timely and practically important problem: whether large, slow neural policies can be deployed in realtime RL. The inaction-regret observation and the two staggering algorithms are valuable, and the experimental study is extensive, with released code and a deliberate attempt to separate delay and inaction effects from model capacity. The linear scaling law N_I^* is largely a direct consequence of the algorithm design rather than an empirical discovery, but measuring it independently across model sizes is a useful contribution. However, the delay-regret component of Theorem 1 is not correctly connected to Algorithm 1 and, more seriously, the proof of Eq. (4) uses a definition of pminimax that does not support the stated lower bound. These issues need to be repaired before the theoretical claims can stand; the empirical results are not enough to compensate for an incorrect central theorem.

major comments (3)
  1. [Eq. (4) and Appendix B.2] The delay-regret lower bound ∆delay(τ) ∈ Ω((τ/τ̄I) E[1 − pminimax^{⌈τθ/τM⌉}]) is not valid with pminimax defined as min_{s,a} max_{s′} p(s′|s,a). In the proof's n-state cycle, the self-transition probability is set to pminimax, but for that construction max_{s′} p(s′|s,a) = max(pminimax, 1−pminimax), which equals pminimax only when pminimax ≥ 1/2. More fundamentally, the probability that the state after k steps is the same as the state at decision time is not generally (pminimax)^k. In a uniformly random n-state environment, pminimax = 1/n and the probability of being in the original state after k ≥ 1 steps is 1/n, not (1/n)^k; in that environment an optimal policy with k-step delay earns at least 1/n per step by randomizing, so the per-step regret is at most 1 − 1/n, which can be far smaller than the quantity 1 − (1/n)^k used in Eq. (4). The theorem therefore needs a corrected measure of 'effective stochasticity over the inference horizon' and a rederived lower-bound argument.
  2. [Section 3.2, Algorithm 1, and Remark 2] The delay-regret term in Theorem 1 is not the one actually incurred by Algorithm 1. In Algorithm 1, after sampling an action with inference time τθ, every process sleeps until the current global maximum τ̂_max^θ before registering the action, so the action is applied ⌈τ̂_max^θ/τM⌉ environment steps after the state was read, not ⌈τθ/τM⌉ steps. The correct delay term for Algorithm 1 is therefore E[1 − pminimax^{⌈τ̂_max^θ/τM⌉}], which converges in the long run to a term involving τ_max^θ, not E[1 − pminimax^{⌈τθ/τM⌉}]. For heavy-tailed or bimodal Tθ, τ_max^θ can be much larger than τ̄θ or can diverge, so N_I^* = ⌈τ_max^θ/τ̄M⌉ may be arbitrarily large and the residual 'only delay regret remains' claim is much weaker than advertised. The paper should either restate the delay analysis for Algorithm 1 explicitly, use Algorithm 2 for the headline claims, or impose and justify a bounded-support assumption on Tθ.
  3. [Appendix B, Assumptions 1–3] The proofs of Eqs. (3) and (4) factor expectations, for example writing the expected number of default actions as (τ/τ̄I)(τ̄I − τ̄M)/τ̄M, which requires TM and TI to be independent, and the delay bound requires Tθ to be independent of the state-action process. If inference time depends on the state or on the sampled action, or if Algorithm 1's interaction time depends on Tθ through the running maximum, these factorizations are not guaranteed to hold and the bounds in Theorem 1 can fail. The assumptions should be stated as explicit conditions in Theorem 1, or the proof should use a weaker coupling argument that permits dependence. A simple empirical check would be to measure the correlation between per-state inference time and the observed inaction or delay regret in one of the Game Boy environments.
minor comments (5)
  1. [Abstract and Remark 1] The phrase 'minimizing long-term regret is generally impossible' overstates the worst-case lower bound; the result is that sequential interaction cannot guarantee vanishing regret in the worst case. Please qualify the claim accordingly.
  2. [Figure 7] For Algorithm 1, N_I^* is determined by τ_max^θ rather than τ̄θ, so plotting N_I^* against τ̄θ conflates two different quantities; if the shape of the Tθ distribution changes with model size, linear scaling in τ̄θ is not a direct consequence of Remark 2.
  3. [References] References [63] and [64] appear to be the same paper, and the same work is also listed as [17] in related work; please deduplicate the bibliography.
  4. [Definition 1] The notation at+⌈τθ/τM⌉ uses t both as a ground-step index and as a wall-clock quantity; please clarify the indexing convention.
  5. [Algorithm 1] The pseudocode samples 'a, τθ ∼ πθ(st)', but τθ is not a standard output of a policy; please clarify whether τθ is measured wall-clock time or a random variable associated with the policy evaluation, since Algorithm 1's sleep logic depends on this quantity.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the regret bounds and scaling guarantees are derived from explicit assumptions and independently verified; the only self-citation is non-load-bearing.

full rationale

The derivation chain is self-contained. Theorem 1 decomposes regret into separately defined learning, inaction, and delay terms; Equation 3 counts expected default actions under the stated independence of TM, TI, and Tθ, and Equation 4 is proved against an explicit worst-case cycle environment in Appendix B. Neither bound is obtained by fitting a parameter or by assuming the conclusion. The scaling guarantees in Remark 2 (Appendix B.4) are direct consequences of the sleep-and-stagger schedules in Algorithms 1 and 2, so the linear N_I* scaling is partly by construction; however, Figure 7 measures N_I* empirically as an independent check, so this is not a fitted-input-called-prediction. The only self-citation bearing on the proof is the appendix note that issues in the workshop version [85] were corrected here; it is not load-bearing, and the proof is provided in the present paper rather than imported. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. A genuine correctness concern exists but is not circular: Eq. (4) uses ⌈τθ/τM⌉, whereas Algorithm 1 registers actions after the sleep-adjusted ⌈τ̂_max^θ/τM⌉, so the delay-regret term may be misspecified for the paper's primary algorithm; this affects correctness, not the circularity score. Accordingly, the score of 2 reflects only a minor non-load-bearing self-citation, not a circular derivation.

Assumptions & free parameters 0 free parameters · 6 assumptions · 2 invented entities

No free parameters are fitted to make the theory work; the empirical scaling laws are measured, not fit. The main modeling assumptions are the independence of the three time random variables and the existence of a fixed default policy β. The induced delayed semi-MDP is a formal construct, not an empirical entity.

assumptions (6)
  • domain assumption The environment evolves in continuous time and cannot be paused; when the agent has no action available, the environment follows a default behavior policy β.
    Section 2, definition of asynchronous MDP Masync; this is the core modeling choice that distinguishes realtime from discrete-time RL.
  • domain assumption Environment step time TM, interaction time TI, and inference time Tθ are independent random variables.
    Appendix B, assumptions 1-3; used to compute expected counts in Equations 2-4. If inference times correlate with state, the decomposition may fail.
  • domain assumption Asynchronous learning can learn from every interaction with the induced delayed semi-MDP.
    Appendix B, assumption 4; used to connect ∆learn to τ/τ̄I.
  • standard math The underlying MDP is Markovian and has bounded rewards |r| ≤ rmax.
    Section 2 background; needed for standard regret bounds and for the semi-MDP construct.
  • standard math Known regret lower bounds for learning (Jaksch et al. [31], Ortner [68]) hold and are used as external benchmarks.
    Equation 2 extends the known Ω(√T) lower bound to continuous time.
  • ad hoc to paper The worst-case lower bound constructions (two-state environment for Eq 3, n-state cycle for Eq 4) are valid representations of the worst case.
    Appendix B, proofs of Equations 3 and 4; these environments are purpose-built to realize the bounds.
invented entities (2)
  • Induced Delayed Semi-MDP (Mdelay)
    purpose: Formal object connecting an asynchronous ground MDP to a delayed semi-MDP that the agent actually solves; used to define ∆realtime.
    Definition 1 and Figure 2. It is a mathematical construction that frames the decision problem; it does not make falsifiable predictions outside the paper.
  • p_minimax (environment stochasticity measure)
    purpose: A lower-bound measure of how likely the environment is to stay in the current state, used in Equation 4 to bound delay regret.
    Section 2, Equation 4 and Appendix B. It is a definition, not an independent empirical quantity; the paper also offers a tighter variant via π*-irrelevance abstraction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference." pith.science (2026). https://pith.science/paper/AL3YEW2E

@misc{pith2026241214355,
  author       = {Pith},
  title        = {Pith review of: Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AL3YEW2E}},
  note         = {Machine review of arXiv:2412.14355}
}
read the original abstract

Realtime environments change even as agents perform action inference and learning, thus requiring high interaction frequencies to effectively minimize regret. However, recent advances in machine learning involve larger neural networks with longer inference times, raising questions about their applicability in realtime systems where reaction time is crucial. We present an analysis of lower bounds on regret in realtime reinforcement learning (RL) environments to show that minimizing long-term regret is generally impossible within the typical sequential interaction and learning paradigm, but often becomes possible when sufficient asynchronous compute is available. We propose novel algorithms for staggering asynchronous inference processes to ensure that actions are taken at consistent time intervals, and demonstrate that use of models with high action inference times is only constrained by the environment's effective stochasticity over the inference horizon, and not by action frequency. Our analysis shows that the number of inference processes needed scales linearly with increasing inference times while enabling use of models that are multiple orders of magnitude larger than existing approaches when learning from a realtime simulation of Game Boy games such as Pok\'emon and Tetris.

Figures

Figures reproduced from arXiv: 2412.14355 by the authors.

Figure 1
Figure 1. Frameworks for Environment Interaction in RL. a) The typical sequential interaction paradigm where both learning and action inference block the environment from moving forward. b) The more realistic setting considered in this work where the environment, the agent’s inference process, and agent’s learning process all proceed at their own rate and interact asynchronously. Multiple self-loops are depicted to denote mul… view at source ↗
Figure 2
Figure 2. Induced Delayed Semi-MDP. We il￾lustrate the semi-MDP described in Definition 1 following the style of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Realtime Interaction Frequency. We illustrate the comparative interaction frequency of methods that sequence learning and inference and those that maintain multiple staggered asynchronous processes. Even when inference times are greater than the environment step time, it is possible to use asynchronous compute to eliminate inaction and learn from every step. Round-Robin Asynchronous Learning: Langford et al. [50] la… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Realtime Pokémon Performance. a) Battles won in Pokémon Blue over time for |θ| = 100M. b) Wild Pokémon caught in Pokémon Blue over time for |θ| = 100M. The parallel learning baseline considers an effective batch size that is 33 times larger with 33 times fewer updates.…
Figure 5
Figure 5. Figure 5: Realtime Tetris Performance vs. |θ|. The average episodic return over 2,000 episodes of learning. We compare models with a single inference process to those that perform staggered asynchronous inference following Algorithm 1 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: We also provide results for three Atari games with agents trained for 2,000 episodes in the same setting (see [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: a) We plot the scaling behavior of the inference compute requirement N∗ I as the expected action inference time τ¯θ increases for ResNet polices across CPUs in the Game Boy environment. b) We plot the scaling behavior of N∗ I instead as a function of the model size |θ|…
Figure 8
Figure 8. Figure 8: a) We plot the scaling behavior of the learning compute requirement N∗ L ≥ 1 as the expected transition learning time τ¯L increases for ResNet polices across CPUs in the Game Boy environment. b) We plot the scaling behavior of N∗ L instead as a function of the model si…
Figure 9
Figure 9. Figure 9: a) A frame from the final battle of Pokémon Blue when the agent is deciding on the next move. b) A frame from the final catching encounter of Pokémon Blue when the agent has just successfully caught Mewtwo. C) A frame from Tetris right before the agent completes its fi…
Figure 10
Figure 10. Figure 10: Realtime Atari Environments. Frames from the Atari environments a) Boxing, b) Krull, and c) Name This Game. These particular games were selected because DQN learning is known to achieve comparable performance to humans within 2,000 episodes when trained without delay.…
Figure 11
Figure 11. Figure 11: Realtime Pokémon Performance for Staggered Asynchronous Interaction & Learning. a) Battles won in Pokémon Blue as a function of time for |θ| = 1M. b) Wild Pokémon caught in Pokémon Blue as a function of time for |θ| = 1M. A.1 POKÉMON RESULTS WITH SMALLER MODEL SIZES D…
Figure 12
Figure 12. Figure 12: Realtime Pokémon Performance for Staggered Asynchronous Interaction & Learning. a) Battles won in Pokémon Blue as a function of time for |θ| = 10M. b) Wild Pokémon caught in Pokémon Blue as a function of time for |θ| = 10M. (a) 100M: Pokémon Battles Won vs. Number of …
Figure 13
Figure 13. Figure 13: Realtime Pokémon Performance for Staggered Asynchronous Interaction & Learning. a) Battles won in Pokémon Blue as a function of non-noop actions taken in the environment for |θ| = 100M. b) Wild Pokémon caught in Pokémon Blue as a function of non-noop actions taken in …
Figure 14
Figure 14. Figure 14: Realtime Atari Rainbow Performance vs. |θ|. The average episodic return over 2,000 simulated learning episodes. We compare Rainbow models with a single inference process to staggered asynchronous inference following Algorithm 1 in a) Boxing, b) Krull, and c) Name This…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RT-HCP: Dealing with Inference Delays and Sample Efficiency to Learn Directly on Robotic Platforms

    cs.LG 2025-09 conditional novelty 6.0 of 10

    RT-HCP combines a physics-informed model, multi-step planning, and an actor-critic policy to learn a swing-up controller on a real Furuta pendulum under strict time and sample limits.

  2. Position: Theory of Mind Benchmarks are Broken for Large Language Models

    cs.AI 2024-12 conditional novelty 6.0 of 10

    The paper proposes that LLM theory-of-mind evaluation should measure functional adaptation to partners, not just literal prediction of their behavior, and shows the two can diverge sharply in simple games.

Reference graph

Works this paper leans on

112 extracted references · 44 canonical work pages · cited by 2 Pith papers

  1. [1]

    Context-specific representation abstraction for deep option learning

    Marwa Abdulhai, Dong-Ki Kim, Matthew Riemer, Miao Liu, Gerald Tesauro, and Jonathan P How. Context-specific representation abstraction for deep option learning. arXiv preprint arXiv:2109.09876, 2021

    arXiv 2021

  2. [2]

    Blind decision making: Reinforcement learning with delayed observations

    Mridul Agarwal and Vaneet Aggarwal. Blind decision making: Reinforcement learning with delayed observations. Pattern Recognition Letters, 150:176–182, 2021

    2021

  3. [3]

    Efficient black-box planning using macro-actions with focused effects

    Cameron Allen, Michael Katz, Tim Klinger, George Konidaris, Matthew Riemer, and Gerald Tesauro. Efficient black-box planning using macro-actions with focused effects. arXiv preprint arXiv:2004.13242, 2020

    arXiv 2004

  4. [4]

    The option-critic architecture

    Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017

    2017

  5. [5]

    Reinforcement learning and its relationship to supervised learning

    Andrew G Barto and Thomas G Dietterich. Reinforcement learning and its relationship to supervised learning. Handbook of Learning and Approximate Dynamic Programming , pp. 45–63, 2004

    2004

  6. [6]

    Continual learning with self-organizing maps

    Pouya Bashivan, Martin Schrimpf, Robert Ajemian, Irina Rish, Matthew Riemer, and Yuhai Tu. Continual learning with self-organizing maps. arXiv preprint arXiv:1904.09330, 2019

    arXiv 1904

  7. [7]

    The arcade learning environment: An evaluation platform for general agents

    Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013

    2013

  8. [8]

    Dynamic programming and optimal control

    Dimitri P Bertsekas. Dynamic programming and optimal control. Journal of the Operational Research Society, 47(6):833–833, 1996

    1996

Show all 112 references
  1. [9]

    Reinforcement learning with random delays

    Yann Bouteiller, Simon Ramstedt, Giovanni Beltrame, Christopher Pal, and Jonathan Bi- nas. Reinforcement learning with random delays. In International conference on learning representations, 2020

    2020

  2. [10]

    Openai gym

    G Brockman. Openai gym. arXiv preprint arXiv:1606.01540, 2016. 11

    2016 arXiv

  3. [11]

    Recursive routing networks: Learning to compose modules for language understanding

    Ignacio Cases, Clemens Rosenbaum, Matthew Riemer, Atticus Geiger, Tim Klinger, Alex Tamkin, Olivia Li, Sandhini Agarwal, Joshua D Greene, Dan Jurafsky, et al. Recursive routing networks: Learning to compose modules for language understanding. In Proceedings of the 2019 Confere...

    2019

  4. [12]

    Automatically composing representation transformations as a means for generalization

    Michael B Chang, Abhishek Gupta, Sergey Levine, and Thomas L Griffiths. Automatically composing representation transformations as a means for generalization. 2019

    2019

  5. [13]

    Leveraging procedural generation to benchmark reinforcement learning

    Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pp. 2048–2056. PMLR, 2020

    2020

  6. [14]

    Learning how to interact with a complex interface using hierarchical reinforcement learning

    Gheorghe Comanici, Amelia Glaese, Anita Gergely, Daniel Toyama, Zafarali Ahmed, Tyler Jackson, Philippe Hamel, and Doina Precup. Learning how to interact with a complex interface using hierarchical reinforcement learning. arXiv preprint arXiv:2204.10374, 2022

    2022 arXiv

  7. [15]

    Pepcvae: Semi-supervised targeted design of antimicrobial peptide sequences

    Payel Das, Kahini Wadhawan, Oscar Chang, Tom Sercu, Cicero Dos Santos, Matthew Riemer, Vijil Chenthamarakshan, Inkit Padhi, and Aleksandra Mojsilovic. Pepcvae: Semi-supervised targeted design of antimicrobial peptide sequences. arXiv preprint arXiv:1810.07743, 2018

    arXiv

  8. [16]

    Acting in delayed environments with non- stationary markov policies

    Esther Derman, Gal Dalal, and Shie Mannor. Acting in delayed environments with non- stationary markov policies. In International Conference on Learning Representations, 2021

    2021

  9. [17]

    Acting in delayed environments with non- stationary markov policies

    Esther Derman, Gal Dalal, and Shie Mannor. Acting in delayed environments with non- stationary markov policies. arXiv preprint arXiv:2101.11992, 2021

    2021 arXiv

  10. [18]

    Contextual moral value alignment through context-based aggregation

    Pierre Dognin, Jesus Rios, Ronny Luss, Inkit Padhi, Matthew D Riemer, Miao Liu, Prasanna Sattigeri, Manish Nagireddy, Kush R Varshney, and Djallel Bouneffouf. Contextual moral value alignment through context-based aggregation. arXiv preprint arXiv:2403.12805, 2024

    2024 arXiv

  11. [19]

    Challenges of real-world rein- forcement learning

    Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world rein- forcement learning. arXiv preprint arXiv:1904.12901, 2019

    1904 arXiv

  12. [20]

    Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures

    Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning,...

    2018

  13. [21]

    Reducing the cost of cycle-time tuning for real-world policy optimization

    Homayoon Farrahi and A Rupam Mahmood. Reducing the cost of cycle-time tuning for real-world policy optimization. In 2023 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE, 2023

    2023

  14. [22]

    Learning with opponent-learning awareness

    Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 122–130. International Foundat...

    2018

  15. [23]

    Dice: The infinitely differentiable monte carlo estimator

    Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. Dice: The infinitely differentiable monte carlo estimator. In International Conference on Machine Learning, pp. 1529–1538. PMLR, 2018

    2018

  16. [24]

    Harris, K

    Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fe...

    2020

  17. [25]

    Alexandria: Extensible framework for rapid exploration of social media

    Fenno F Heath, Richard Hull, Elham Khabiri, Matthew Riemer, Noi Sukaviriya, and Roman Vaculín. Alexandria: Extensible framework for rapid exploration of social media. In 2015 IEEE International Congress on Big Data, pp. 483–490. IEEE, 2015. 12

    2015

  18. [26]

    Rainbow: Combining improvements in deep reinforcement learning

    Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence,...

    2018

  19. [27]

    Texplore: real-time sample-efficient reinforcement learning for robots

    Todd Hester and Peter Stone. Texplore: real-time sample-efficient reinforcement learning for robots. Machine learning, 90:385–429, 2013

    2013

  20. [28]

    A real-time model-based reinforcement learning architecture for robot control

    Todd Hester, Michael Quinlan, and Peter Stone. A real-time model-based reinforcement learning architecture for robot control. arXiv preprint arXiv:1105.1749, 2011

    2011 arXiv

  21. [29]

    Rtmba: A real-time model-based reinforce- ment learning architecture for robot control

    Todd Hester, Michael Quinlan, and Peter Stone. Rtmba: A real-time model-based reinforce- ment learning architecture for robot control. In 2012 IEEE International Conference on Robotics and Automation, pp. 85–90. IEEE, 2012

    2012

  22. [30]

    Why can a machine beat mario but not pokemon?, 2018

    Shayaan Jagtap. Why can a machine beat mario but not pokemon?, 2018. URL https: //towardsdatascience.com/why-can-a-machine-beat-mario-but-not-pokemon-ff61313187e1

    2018

  23. [31]

    Near-optimal regret bounds for reinforcement learning

    Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 11(4), 2010

    2010

  24. [32]

    Is q-learning provably efficient? Advances in neural information processing systems, 31, 2018

    Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? Advances in neural information processing systems, 31, 2018

    2018

  25. [33]

    Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, pp

    Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, pp. 5084–5096. PMLR, 2021

    2021

  26. [34]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  27. [35]

    Reinforcement learning from delayed observations via world models

    Armin Karamzade, Kyungmin Kim, Montek Kalsi, and Roy Fox. Reinforcement learning from delayed observations via world models. arXiv preprint arXiv:2403.12309, 2024

    2024 arXiv

  28. [36]

    Dynamic decision frequency with continuous options

    Amirmohammad Karimi, Jun Jin, Jun Luo, A Rupam Mahmood, Martin Jagersand, and Samuele Tosatto. Dynamic decision frequency with continuous options. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7545–7552. IEEE, 2023

    2023

  29. [37]

    Markov decision processes with delays and asynchronous cost collection

    Konstantinos V Katsikopoulos and Sascha E Engelbrecht. Markov decision processes with delays and asynchronous cost collection. IEEE transactions on automatic control , 48(4): 568–574, 2003

    2003

  30. [38]

    Domain scoping for subject matter experts

    Elham Khabiri, Matthew Riemer, Fenno F Heath III, and Richard Hull. Domain scoping for subject matter experts. In 2015 AAAI Fall Symposium Series, 2015

    2015

  31. [39]

    Towards continual reinforcement learning: A review and perspectives

    Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. arXiv preprint arXiv:2012.13490, 2020

    2012 arXiv

  32. [40]

    Learning hierarchical teaching policies for cooperative agents

    Dong-Ki Kim, Miao Liu, Shayegan Omidshafiei, Sebastian Lopez-Cot, Matthew Riemer, Gol- naz Habibi, Gerald Tesauro, Sami Mourad, Murray Campbell, and Jonathan P How. Learning hierarchical teaching policies for cooperative agents. arXiv preprint arXiv:1903.03216, 2019

    1903 arXiv

  33. [41]

    Hetero- geneous knowledge transfer via hierarchical teaching in cooperative multiagent reinforcement learning

    Dong-Ki Kim, Miao Liu, Shayegan Omidshafiei, Sebastian Lopez-Cot, Matthew Riemer, Gerald Tesauro, Murray Campbell, Sami Mourad, Golnaz Habibi, and Jonathan P How. Hetero- geneous knowledge transfer via hierarchical teaching in cooperative multiagent reinforcement learning. 2019

    2019

  34. [42]

    A policy gradient algorithm for learning to learn in multiagent reinforcement learning

    Dong Ki Kim, Miao Liu, Matthew D Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cot, Gerald Tesauro, and Jonathan How. A policy gradient algorithm for learning to learn in multiagent reinforcement learning. In International Conference on Machine Learn...

    2021

  35. [43]

    Influencing long-term behavior in multiagent reinforcement learning

    Dong-Ki Kim, Matthew Riemer, Miao Liu, Jakob Foerster, Michael Everett, Chuangchuang Sun, Gerald Tesauro, and Jonathan P How. Influencing long-term behavior in multiagent reinforcement learning. Advances in Neural Information Processing Systems, 35:18808–18821, 2022

    2022

  36. [44]

    Game-theoretical perspectives on active equilibria: A preferred solution concept over nash equilibria

    Dong-Ki Kim, Matthew Riemer, Miao Liu, Jakob N Foerster, Gerald Tesauro, and Jonathan P How. Game-theoretical perspectives on active equilibria: A preferred solution concept over nash equilibria. arXiv preprint arXiv:2210.16175, 2022

    arXiv

  37. [45]

    Overcoming catastrophic forgetting in neural networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of scien...

    2017

  38. [46]

    A study of compositional generalization in neural models

    Tim Klinger, Dhaval Adjodah, Vincent Marois, Josh Joseph, Matthew Riemer, Alex’Sandy’ Pentland, and Murray Campbell. A study of compositional generalization in neural models. arXiv preprint arXiv:2006.09437, 2020

    2006 arXiv

  39. [47]

    Iterated reasoning with mutual infor- mation in cooperative and byzantine decentralized teaming

    Sachin Konan, Esmaeil Seraj, and Matthew Gombolay. Iterated reasoning with mutual infor- mation in cooperative and byzantine decentralized teaming. arXiv preprint arXiv:2201.08484, 2022

    2022 arXiv

  40. [48]

    Asynchronous coagent networks

    James Kostas, Chris Nota, and Philip Thomas. Asynchronous coagent networks. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5426–5435. PMLR, 13–18 Jul 202...

    2020

  41. [49]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179–1191, 2020

    2020

  42. [50]

    Slow learners are fast

    John Langford, Alexander J Smola, and Martin Zinkevich. Slow learners are fast. In Pro- ceedings of the 22nd International Conference on Neural Information Processing Systems, pp. 2331–2339, 2009

    2009

  43. [51]

    Emergent multi-agent communication in the deep learning era

    Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv preprint arXiv:2006.02419, 2020

    2006 arXiv

  44. [52]

    Towards a unified theory of state abstraction for mdps

    Lihong Li, Thomas J Walsh, and Michael L Littman. Towards a unified theory of state abstraction for mdps

  45. [53]

    Learning without forgetting

    Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017

    2017

  46. [54]

    Multi-agent actor-critic for mixed cooperative-competitive environments

    Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30, 2017

    2017

  47. [55]

    Consolidation via policy information regularization in deep rl for multi-agent games

    Tyler Malloy, Tim Klinger, Miao Liu, Matthew Riemer, Gerald Tesauro, and Chris R Sims. Consolidation via policy information regularization in deep rl for multi-agent games. arXiv preprint arXiv:2011.11517, 2020

    2011 arXiv

  48. [56]

    Deep rl with information constrained policies: Generalization in continuous control

    Tyler Malloy, Chris R Sims, Tim Klinger, Miao Liu, Matthew Riemer, and Gerald Tesauro. Deep rl with information constrained policies: Generalization in continuous control. arXiv preprint arXiv:2010.04646, 2020

    2010 arXiv

  49. [57]

    Rl generalization in a theory of mind game through a sleep metaphor (student abstract)

    Tyler Malloy, Tim Klinger, Miao Liu, Gerald Tesauro, Matthew Riemer, and Chris R Sims. Rl generalization in a theory of mind game through a sleep metaphor (student abstract). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 15841–15842, 2021

    2021

  50. [58]

    Capacity-limited decentralized actor-critic for multi-agent games

    Tyler Malloy, Chris R Sims, Tim Klinger, Miao Liu, Matthew Riemer, and Gerald Tesauro. Capacity-limited decentralized actor-critic for multi-agent games. In 2021 IEEE Conference on Games (CoG), pp. 1–8. IEEE, 2021. 14

    2021

  51. [59]

    Learning in factored domains with information-constrained visual representations

    Tyler Malloy, Miao Liu, Matthew D Riemer, Tim Klinger, Gerald Tesauro, and Chris R Sims. Learning in factored domains with information-constrained visual representations. arXiv preprint arXiv:2303.17508, 2023

    2023 arXiv

  52. [60]

    Summarizing societies: Agent abstraction in multi-agent reinforcement learning

    Amin Memarian, Maximilian Puelma Touzel, Matthew Riemer, Rupali Bhati, and Irina Rish. Summarizing societies: Agent abstraction in multi-agent reinforcement learning. In From Cells to Societies: Collective Learning across Scales, 2022

    2022

  53. [61]

    Human-level control through deep reinforcement learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015

    2015

  54. [62]

    Asynchronous methods for deep rein- forcement learning

    V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep rein- forcement learning. In International conference on machine learning, pp. 1928–1937. PMLR, 2016

    1928

  55. [64]

    Revisiting state augmentation methods for reinforcement learning with stochastic delays

    Somjit Nath, Mayank Baranwal, and Harshad Khadilkar. Revisiting state augmentation methods for reinforcement learning with stochastic delays. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 1346–1355, 2021

    2021

  56. [65]

    Gotta learn fast: A new benchmark for generalization in rl

    Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprint arXiv:1804.03720, 2018

    2018 arXiv

  57. [66]

    Sequoia: A software framework to unify continual learning research

    Fabrice Normandin, Florian Golemo, Oleksiy Ostapenko, Pau Rodriguez, Matthew D Riemer, Julio Hurtado, Khimya Khetarpal, Ryan Lindeborg, Lucas Cecchi, Timothée Lesort, et al. Sequoia: A software framework to unify continual learning research. arXiv preprint arXiv:2108.01005, 2021

    2021 arXiv

  58. [67]

    Learning to teach in cooperative multiagent reinforcement learning

    Shayegan Omidshafiei, Dong-Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christo- pher Amato, Murray Campbell, and Jonathan P How. Learning to teach in cooperative multiagent reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 3...

    2019

  59. [68]

    Regret bounds for reinforcement learning via markov chain concentration

    Ronald Ortner. Regret bounds for reinforcement learning via markov chain concentration. Journal of Artificial Intelligence Research, 67:115–128, 2020

    2020

  60. [69]

    Comvas: Contextual moral values alignment system

    Inkit Padhi, Pierre Dognin, Jesus Rios, Ronny Luss, Swapnaja Achintalwar, Matthew Riemer, Miao Liu, Prasanna Sattigeri, Manish Nagireddy, Kush R Varshney, et al. Comvas: Contextual moral values alignment system. Proceedings of the International Joint Conference on Artificial I...

    2024

  61. [70]

    Pytorch: An imperative style, high- performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  62. [71]

    Markov decision processes

    ML Puterman. Markov decision processes. 1994. Jhon Wiley & Sons, New Jersey, 1994

    1994

  63. [72]

    Machine theory of mind

    Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International conference on machine learning, pp. 4218–4227. PMLR, 2018

    2018

  64. [73]

    Real-time reinforcement learning

    Simon Ramstedt and Chris Pal. Real-time reinforcement learning. Advances in neural information processing systems, 32, 2019. 15

    2019

  65. [74]

    Distributed computing in social media analytics

    Matthew Riemer. Distributed computing in social media analytics. Distributed Computing in Big Data Analytics: Concepts, Technologies and Applications, pp. 121–135, 2017

    2017

  66. [75]

    A deep learning and knowledge transfer based architecture for social media user characteristic determination

    Matthew Riemer, Sophia Krasikov, and Harini Srinivasan. A deep learning and knowledge transfer based architecture for social media user characteristic determination. In Proceedings of the third International Workshop on Natural Language Processing for Social Media, pp. 39–47, 2015

    2015

  67. [76]

    Correcting forecasts with multifactor neural attention

    Matthew Riemer, Aditya Vempaty, Flavio Calmon, Fenno Heath, Richard Hull, and Elham Khabiri. Correcting forecasts with multifactor neural attention. In International Conference on Machine Learning, pp. 3010–3019. PMLR, 2016

    2016

  68. [77]

    Generative knowledge distillation for general purpose function compression

    Matthew Riemer, Michele Franceschini, Djallel Bouneffouf, and Tim Klinger. Generative knowledge distillation for general purpose function compression. NIPS 2017 Workshop on Teaching Machines, Robots, and Humans, 5:30, 2017

    2017

  69. [78]

    Representation stability as a regular- izer for improved text analytics transfer learning

    Matthew Riemer, Elham Khabiri, and Richard Goodwin. Representation stability as a regular- izer for improved text analytics transfer learning. arXiv preprint arXiv:1704.03617, 2017

    2017 arXiv

  70. [79]

    Learning to learn without forgetting by maximizing transfer and minimizing interference

    Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910, 2018

    arXiv

  71. [80]

    Learning abstract options

    Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. Advances in neural information processing systems, 31, 2018

    2018

  72. [81]

    Scalable recollections for continual lifelong learning

    Matthew Riemer, Tim Klinger, Djallel Bouneffouf, and Michele Franceschini. Scalable recollections for continual lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1352–1359, 2019

    2019

  73. [82]

    On the role of weight sharing during deep option learning

    Matthew Riemer, Ignacio Cases, Clemens Rosenbaum, Miao Liu, and Gerald Tesauro. On the role of weight sharing during deep option learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5519–5526, 2020

    2020

  74. [83]

    Continual learning in environments with polynomial mixing times

    Matthew Riemer, Sharath Chandra Raparthy, Ignacio Cases, Gopeshh Subbaraj, Maximil- ian Puelma Touzel, and Irina Rish. Continual learning in environments with polynomial mixing times. Advances in Neural Information Processing Systems, 2022

    2022

  75. [84]

    Balancing context length and mixing times for reinforcement learning at scale

    Matthew Riemer, Khimya Khetarpal, Janarthanan Rajendran, and Sarath Chandar. Balancing context length and mixing times for reinforcement learning at scale. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  76. [85]

    Realtime reinforcement learning: Towards rapid asynchronous deployment of large models

    Matthew Riemer, Gopeshh Subbaraj, Glen Berseth, and Irina Rish. Realtime reinforcement learning: Towards rapid asynchronous deployment of large models. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024

    2024

  77. [86]

    Routing networks: Adaptive selection of non-linear functions for multi-task learning

    Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In International Conference on Learning Representations. International Conference on Learning Representations, ICLR, 2018

    2018

  78. [87]

    Dispatched routing networks

    Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, Atticus Geiger, Lauri Karttunen, Joshua D Greene, Dan Jurafsky, and Christopher Potts. Dispatched routing networks. Technical report, 2019

    2019

  79. [88]

    Routing networks and the challenges of modular and compositional computation

    Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation. arXiv preprint arXiv:1904.12774, 2019

    1904 arXiv

  80. [89]

    Control delay in reinforcement learning for real-time dynamic systems: A memoryless approach

    Erik Schuitema, Lucian Bu¸ soniu, Robert Babuška, and Pieter Jonker. Control delay in reinforcement learning for real-time dynamic systems: A memoryless approach. In 2010 IEEE/RSJ international conference on intelligent robots and systems, pp. 3226–3231. IEEE, 2010. 16

    2010

  81. [90]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  82. [91]

    Bigger, better, faster: Human-level atari with human-level efficiency

    Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Marc G Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pp. 30365–30380. PMLR, 2023

    2023

  83. [92]

    Pokémon red/blue/rng manipulation faq, 2020

    speedrun.com. Pokémon red/blue/rng manipulation faq, 2020. URL http://http: //wiki.pokemonspeedruns.com/index.php?title=Pok%C3%A9mon_Red/Blue/RNG_ Manipulation_FAQ

    2020

  84. [93]

    Pokemon red/blue leaderboard, 2024

    speedrun.com. Pokemon red/blue leaderboard, 2024. URL https://www.speedrun.com/ pkmnredblue

    2024

  85. [94]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018

    2018

  86. [95]

    Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning

    Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2): 181–211, 1999

    1999

  87. [96]

    Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction

    Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agent...

    2011

  88. [97]

    A deep dive into the trade-offs of parameter-efficient preference alignment techniques

    Megh Thakkar, Quentin Fournier, Matthew D Riemer, Pin-Yu Chen, Amal Zouaq, Payel Das, and Sarath Chandar. A deep dive into the trade-offs of parameter-efficient preference alignment techniques. arXiv preprint arXiv:2406.04879, 2024

    2024 arXiv

  89. [98]

    Combining domain and alignment vectors to achieve better knowledge-safety trade-offs in llms

    Megh Thakkar, Yash More, Quentin Fournier, Matthew Riemer, Pin-Yu Chen, Amal Zouaq, Payel Das, and Sarath Chandar. Combining domain and alignment vectors to achieve better knowledge-safety trade-offs in llms. arXiv preprint arXiv:2411.06824, 2024

    2024 arXiv

  90. [99]

    Philip S. Thomas. Policy gradient coagent networks. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.),Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL https://proceedings.neurips.cc/paper_files/ pa...

    2011

  91. [100]

    Scalable approaches for a theory of many minds

    Maximilian Puelma Touzel, Amin Memarian, Matthew Riemer, Andrei Mircea, Andrew Robert Williams, Elin Ahlstrand, Lucas Lehnert, Rupali Bhati, Guillaume Dumas, and Irina Rish. Scalable approaches for a theory of many minds. In Agentic Markets Workshop at ICML 2024, 2024

    2024

  92. [101]

    Gymnasium: A standard interface for reinforcement learning environments

    Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024

    2024 arXiv

  93. [102]

    Reactive reinforcement learning in asynchronous environments

    Jaden B Travnik, Kory W Mathewson, Richard S Sutton, and Patrick M Pilarski. Reactive reinforcement learning in asynchronous environments. Frontiers in Robotics and AI, 5:79, 2018

    2018

  94. [103]

    Learning and planning in environments with delayed feedback

    Thomas J Walsh, Ali Nouri, Lihong Li, and Michael L Littman. Learning and planning in environments with delayed feedback. Autonomous Agents and Multi-Agent Systems, 18: 83–105, 2009

    2009

  95. [104]

    Game Boy — Wikipedia, the free encyclopedia

    Wikipedia. Game Boy — Wikipedia, the free encyclopedia. http://en.wikipedia.org/w/index. php?title=Game%20Boy&oldid=1224667625, 2024. [Online; accessed 20-May-2024]

    2024

  96. [105]

    Learning latent representations to influence multi-agent interaction

    Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representations to influence multi-agent interaction. In Conference on robot learning , pp. 575–588. PMLR, 2021. 17 (a) Pokémon Battling (b) Pokémon Catching (c) Tetris Figure 9: a) A frame fr...

    2021

  97. [106]

    Asynchronous reinforcement learning for real-time control of physical robots

    Yufeng Yuan and A Rupam Mahmood. Asynchronous reinforcement learning for real-time control of physical robots. In 2022 International Conference on Robotics and Automation (ICRA), pp. 5546–5552. IEEE, 2022

    2022

  98. [107]

    Asynchronous methods for model-based reinforcement learning

    Yunzhi Zhang, Ignasi Clavera, Boren Tsai, and Pieter Abbeel. Asynchronous methods for model-based reinforcement learning. arXiv preprint arXiv:1910.12453, 2019

    1910 arXiv

  99. [108]

    NoFrameSkip-v4

    Modjtaba Shokrian Zini, Mohammad Pedramfar, Matthew Riemer, Ahmadreza Moradipari, and Miao Liu. Coagent networks revisited. arXiv preprint arXiv:2001.10474, 2020. A F URTHER DETAILS SUPPORTING THE MAIN TEXT Software Libraries: Our experiments leverage Numpy [24], which is publ...

    2001 arXiv

  100. [109]

    The environment step time can be treated as an independent random variable TM with sampled values τM ∼ TM and expected value ¯τM := E[TM]

  101. [110]

    The environment interaction time can be treated as an independent random variable TI with sampled values τI ∼ TI and expected value ¯τI := E[TI]

  102. [111]

    The action inference time of the policy can be treated as an independent random variable Tθ with sampled values τθ ∼ Tθ and expected value ¯τθ := E[Tθ]

  103. [112]

    Issues with the proof in an earlier workshop version of the paper [85] have been corrected here

    Asynchronous learning can learn from every interaction with ˜Mdelay. Issues with the proof in an earlier workshop version of the paper [85] have been corrected here. B.1 D EFINITION 1 Most of Definition 1 just recaps the dynamics of how the agent interacts with an asynchronous...

  104. [113]

    To show it is a semi-MDP as well, we consider the same proof style of Theorem 1 in Sutton et al

    where the action delay distribution is defined by the random variable ⌈τθ/τM⌉. To show it is a semi-MDP as well, we consider the same proof style of Theorem 1 in Sutton et al. [95]: A semi-MDP consists of (1) a set of states, (2) a set of actions, (3) for each pair of state an...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.