Pith. sign in

REVIEW 5 major objections 4 minor 34 references

ParMod: A Parallel and Modular Framework for Learning Non-Markovian Tasks

T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ParMod claims that a non-Markovian task written in LTL_f can be decomposed by DFA-state ranks into phases, trained in parallel by separate agents, and still converge to an optimal policy of the product MDP.

desk verdict A genuinely useful modular RL framework with strong empirical results, but the convergence-to-optimality proof doesn't hold as written; worth a serious referee but not acceptance as-is. read the letter →

arxiv 2412.12700 v1 pith:Q4BITHB6 submitted 2024-12-17 cs.LG cs.AI

Ruixuan Miao , Xu Lu , Cong Tian , Bin Yu , Zhenhua Duan This is my paper · ORCID
classification cs.LGcs.AI
keywords ReinforcementLearningTemporalLogicNon-MarkovianrewardParalleltrainingshapingTask-phaseclassificationProductMDPLTL_f
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is about learning tasks whose reward depends on history, not just the current state. ParMod takes a formal specification in LTL_f, converts it into a deterministic automaton, and splits the automaton states into ranked 'task phases' by their average distance to acceptance. Each phase is trained by its own agent in parallel, with an initial-state buffer that hands the next phase a starting point whenever a phase boundary is crossed, and with rewards shaped by a potential that depends only on the automaton rank. The paper's central claim is that this does not sacrifice optimality: with tabular Q-learning, ParMod is said to converge to an optimal policy of the product MDP under infinite visitation. The value of the claim is practical: if it holds, sparse non-Markovian tasks that defeat monolithic or merely distributed RL become a set of denser, parallelizable sub-problems.

What carries the argument

The central object is the rank, a task-phase index assigned to each DFA state by the Task Phase Classification (TPC) algorithm: the average graph distance from the state to accepting states is normalized into $N$ ranks. The machinery is the product MDP $M^\otimes$ whose states are pairs $(s,q)$ of environment state and DFA state; ParMod trains one agent per rank, and its critic TD target for an experience ending in rank $k'$ uses the critic of network $k'$ rather than the current network. Initial-state buffers store product states at phase boundaries to seed later phases, and a potential function $\rho(q)=C/(N-\mathrm{rank}(q))$ reshapes rewards as $\gamma\rho(q_{t+1})-\rho(q_t)$ so that every phase transition produces a dense signal. These pieces carry the argument: the partition creates short-horizon sub-problems, the buffers eliminate the exploration bottleneck of late phases, and the shaping is designed to preserve the optimal policy of the product MDP.

What would settle it

Run ParMod's tabular update from Algorithm 2 on a tiny product MDP (for example, two environment states and a two-state DFA whose rank-0 phase transitions into rank-1) and compare the learned values of each modular Q-table against a standard single Q-learning run on the same product MDP; any difference at a shared state-action pair would falsify the transferred convergence argument.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the memory required by a non-Markovian task can be read off an automaton and turned into a partition of the task into phases: each non-accepting, non-error DFA state gets a rank between 0 and N-1 by normalizing the average distance from that state to the accepting states, and states with the same rank belong to the same phase. ParMod trains one agent per phase, and each agent's critic update uses the next phase's critic rather than its own when the automaton rank changes, so the value signal propagates backward across phase boundaries. Initial-state buffers collected during training let later phases start from states that a single agent would rarely reach, and a potential-based reward shaping term $\gamma\rho(q_{t+1})-\rho(q_t)$ with $\rho(q)=C/(N-\mathrm{rank}(q))$ makes the reward dense. The theoretical claim is Theorem 4: ParMod with tabular Q-learning converges to an optimal policy of the product MDP whenever every state-action pair is visited infinitely often. The empirical claim is that on nine benchmarks from Waterworld, Racecar, and Halfcheetah, ParMod reaches a 100% success rate with shorter policies and faster convergence than the flat baseline, the modular DDPG-style baseline, reward-machine Q-learning, and the distributed IMPALA and DPPO baselines.

Load-bearing premise

The whole optimality argument rests on assuming that the value assigned by each phase's own Q-table to a state-action pair is exactly the value a single global Q-table would assign; the paper asserts this equality but does not prove it.

Editorial extensions

If this is right

  • For any NMT expressible in LTL_f, the same pipeline works: formula to DFA, TPC ranks, parallel agents, and shaped rewards; the paper demonstrates it with both SAC and PPO as underlying algorithms.
  • Because the shaping potential depends only on the DFA state, it should preserve the optimal policy of the product MDP, so the parallel speedup does not change the task being solved.
  • Tasks with much larger automata (up to 290 DFA states and 1091 transitions in the paper) still show steadily rising reward curves, indicating the decomposition scales beyond the small benchmarks.
  • The initial-state buffers directly attack the sparse-reward bottleneck: later phases get explored even when a single agent would almost never reach them, which the paper identifies as the reason IMPALA and DPPO fail.
  • If Theorem 4 is accepted, ParMod acts as a general wrapper: any convergent base RL algorithm can be plugged into the phases and inherit the optimality guarantee.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of Theorem 4 would be to run the tabular update on a small hand-built two-phase product MDP and compare each resulting $Q_k$ with a single global Q-learning run; the paper assumes these coincide but does not verify it.
  • The rank metric is purely structural; replacing the graph distance with a distance weighted by environment transition probabilities could produce phases that are better matched to actual learning difficulty, and could be tested on the same benchmarks.
  • The cross-phase critic bootstrap resembles asynchronous value iteration with changing targets; one might prove convergence under a contraction condition on the phase partition, or construct adversarial partitions that break convergence.
  • Because initial-state buffers resample from product states discovered by previous trajectories, the training distribution is off-policy in a way that standard RL theory does not cover; the framework might benefit from off-policy correction or importance weighting as the number of phases grows.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes ParMod, a parallel and modular RL framework for non-Markovian tasks (NMTs) specified in LTL_f. The method converts an LTL_f specification into a DFA, partitions the DFA states into ranks (task phases) via Algorithm 1, trains one agent per phase with separate initial-state buffers and experience buffers, and reshapes rewards using a potential function defined over DFA states. The paper claims that ParMod with tabular Q-learning converges to an optimal policy of the product MDP (Theorem 4), and empirically reports faster convergence, higher success rates, and better policy quality than Base, Mod, QRM, IMPALA, and DPPO on continuous benchmarks including Waterworld, Racecar, and Halfcheetah.

Significance. If the theoretical claims were valid, ParMod would be a useful contribution connecting temporal-logic specifications with scalable RL through parallel modular training. The empirical work is extensive: open-source code, 13 tasks, multiple ablations (TPC vs. random classification, SPECTRL reward comparison, distributed version), and comparisons against distributed RL baselines. These engineering and empirical contributions are real and should be credited. However, the theoretical core of the paper is not sound: the convergence proof is circular, the algorithm's update rule is undefined on key transitions, and the reward shaping changes the underlying objective. The paper's central guarantee of optimality is therefore not established, and the theoretical section contains concrete mathematical errors.

major comments (5)
  1. [Section 5, Theorem 4] The proof of Theorem 4 is circular. It states that "Q(s⊗,a) actually represents the global Q-function, whose value is equal to Q_k(s⊗,a) for each state-action pair" and then concludes that Q-learning convergence transfers to the modular Q-functions. That equality is precisely what the cross-rank bootstrap update (Algorithm 2, lines 21-24) must establish; it is assumed without proof. The coupled update equations across ranks define a different operator from standard Q-learning, and no argument shows that its fixed point coincides with the global Q-function.
  2. [Algorithm 2, line 22; Algorithm 1] The update rule in Algorithm 2 is undefined for transitions into accepting or error states. Algorithm 1 assigns ranks only to states in Q \ F \ E, yet line 22 computes k' = rank(q_{i+1}) for every stored experience, including those with q_{i+1} in F or E. The pseudocode therefore does not specify the bootstrap target for exactly the transitions that deliver completion or failure reward. Any convention that sets Q_terminal = 0 is not stated in the algorithm or used in the proof of Theorem 4.
  3. [Section 4.2, reward shaping] The potential function assigns nonzero potentials to terminal states: rho(F) = C and rho(E) = C/(N+1). Potential-based reward shaping preserves the optimal policy only when terminal-state potentials are zero. With nonzero terminal potentials, the reshaped return of a trajectory ending at time L contains the extra term gamma^{L+1} rho(q_L) - rho(q_0), which depends on the completion time L and on whether the terminal state is accepting or error. Consequently, the tabular Q-learning in ParMod converges to the optimum of the reshaped, phase-truncated MDP, not to the optimum of the original product MDP asserted in Theorem 4.
  4. [Section 5, Theorem 3, Eq. (4)] Equation (4) incorrectly replaces E[gamma^L] with P(s0) gamma^L. The expression "P(s0) gamma^L" mixes a probability with a random variable; the correct factor is E[gamma^L 1_{success}] (or r E[gamma^L] after conditioning on success). For gamma < 1, maximizing E[gamma^L] is not equivalent to minimizing the expected number of steps, and for gamma = 1 the argument only recovers the success probability if L is independent of the success event, which is not shown. This invalidates the stated characterization of the product MDP's optimal policy.
  5. [Section 5, Theorem 4; Algorithm 2, line 19] The convergence proof ignores the phase truncation that is built into Algorithm 2. Line 19 terminates an episode whenever the DFA state reaches a different rank or enters F/E, so each modular Q-function is trained on a truncated sub-MDP. The theorem asserts convergence to the optimal policy of the full product MDP without analyzing how the composition of phase-local value functions, connected by the initial-state buffers and cross-rank bootstrapping, reconstructs the global optimal value. This is a load-bearing gap in the proof.
minor comments (4)
  1. [Section 4.2] There are several typos: "algorhtims" should be "algorithms", "nuder" in the proof of Lemma 1 should be "under", and "is is" in the proof of Theorem 4 should be "is".
  2. [Section 4] The text describing Figure 2 mentions "the accepting state q5", but the DFA in Figure 3 (used in the running Example 1) has q4 as the accepting state. Please clarify whether Figure 2 uses a different DFA or correct the caption/text.
  3. [Section 6.1] In the description of the distributed experiment, "CUP cores" should be "CPU cores".
  4. [Algorithm 2] The initial-state buffer notation is inconsistent: the text uses "B_i" in Section 4.2 but the algorithm listing uses "B_k" without a formal definition of B_k. Please define the buffer symbols consistently.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 4's optimality proof assumes the modular critics equal the global Q-function—the very invariant the cross-rank bootstrap must establish—so the convergence claim reduces to a definitional assertion.

  1. self definitional [Section 5, Theorem 4 proof (after the modular Q-learning update and standard Q-learning update)]
    "Note that 𝑄(𝑠⊗,𝑎) actually represents the global Q-function, whose value is equal to 𝑄𝑘(𝑠⊗,𝑎) for each state-action pair. As Q-learning converges to the optimal Q-function, then the optimal Q-value function for 𝑄𝑟𝑎𝑛𝑘(𝑞) is equivalent to the optimal Q-function for 𝑄(⟨𝑠,𝑞⟩,⋅) in 𝑀⊗."

    The proof transfers the standard Q-learning convergence theorem to ParMod by declaring that the modular critics Q_k jointly constitute the global Q-function of the product MDP. But Algorithm 2 (lines 21-24) updates Q_k using a bootstrap target from a different critic Q_{k'} whenever the next DFA state has a different rank; whether that update preserves the declared equality with the global Q-function is exactly what must be shown to conclude optimality. The assertion also cannot hold as stated because rank is undefined on F∪E (Algorithm 1 assigns ranks only to Q\F\E), so terminal transitions are not covered, and the reshaped reward with nonzero terminal potentials changes the fixed point.

full rationale

The paper's Theorems 1-3 give a standard equivalence between an NMRDP and its product MDP, and the experimental comparison against external baselines (Base, Mod, QRM, IMPALA, DPPO, SPECTRL) is empirically self-contained. The circularity is localized to Theorem 4, the central optimality claim: its proof reduces to the sentence "Q(s⊗,a) actually represents the global Q-function, whose value is equal to Q_k(s⊗,a)", which is the invariant the cross-rank bootstrap update must establish. If that equality is taken as a definition, the update is formally standard Q-learning, but then the learned fixed point is that of the reshaped, phase-truncated process, not necessarily of M⊗; the paper provides no argument that the reshaping and phase termination preserve the product MDP's Q*. This makes the theoretical optimality proof circular/unsupported. No load-bearing self-citations are involved (Melo's Q-learning convergence is an external, standard result), so the score is moderate rather than severe.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

ParMod introduces no new physical or logical entities; its components are product-MDP states, task phases, and modular networks. The main hand-chosen quantities are the number of phases N and the potential constant C. The central composition assumption of phase-wise optimality is treated as an axiom in the convergence proof.

free parameters (2)
  • N (number of task phases) = 6 for Waterworld Tasks 1-3, 10 for Racecar Tasks 4-6, 3 for Halfcheetah Tasks 7-9
    Chosen per task by the authors; Section 6.2 reports that the optimal N cannot be determined theoretically and requires parameter tuning.
  • C (potential constant) = 100
    Scale constant in rho(q)=C/(N-rank(q)); set to 100 for all algorithms, including baselines.
assumptions (3)
  • standard math Every LTL_f formula can be converted into an equivalent DFA.
    Invoked in Section 3.1 and used to construct the product MDP; this is a known result the paper relies on without proving.
  • standard math Potential-based reward shaping F(s,a,s') = gamma*rho(s') - rho(s) preserves the optimal policy.
    Used in Algorithm 2 line 17 and Section 4.2; the paper does not state the theorem or its episodic/terminal-state conditions.
  • ad hoc to paper Rank-based phase decomposition with per-phase agents and initial-state handoff preserves the optimal policy of the global product MDP.
    This is the core premise of the framework; Theorem 4 assumes it rather than deriving it, so it functions as an axiom for ParMod.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ParMod: A Parallel and Modular Framework for Learning Non-Markovian Tasks." pith.science (2026). https://pith.science/paper/Q4BITHB6

@misc{pith2026241212700,
  author       = {Pith},
  title        = {Pith review of: ParMod: A Parallel and Modular Framework for Learning Non-Markovian Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q4BITHB6}},
  note         = {Machine review of arXiv:2412.12700}
}
read the original abstract

The commonly used Reinforcement Learning (RL) model, MDPs (Markov Decision Processes), has a basic premise that rewards depend on the current state and action only. However, many real-world tasks are non-Markovian, which has long-term memory and dependency. The reward sparseness problem is further amplified in non-Markovian scenarios. Hence learning a non-Markovian task (NMT) is inherently more difficult than learning a Markovian one. In this paper, we propose a novel \textbf{Par}allel and \textbf{Mod}ular RL framework, ParMod, specifically for learning NMTs specified by temporal logic. With the aid of formal techniques, the NMT is modulaized into a series of sub-tasks based on the automaton structure (equivalent to its temporal logic counterpart). On this basis, sub-tasks will be trained by a group of agents in a parallel fashion, with one agent handling one sub-task. Besides parallel training, the core of ParMod lies in: a flexible classification method for modularizing the NMT, and an effective reward shaping method for improving the sample efficiency. A comprehensive evaluation is conducted on several challenging benchmark problems with respect to various metrics. The experimental results show that ParMod achieves superior performance over other relevant studies. Our work thus provides a good synergy among RL, NMT and temporal logic.

Figures

Figures reproduced from arXiv: 2412.12700 by the authors.

Figure 1
Figure 1. An instance of the Waterworld problem Example 1 (Waterworld problem - detailed). The en￾vironment comprises a two-dimensional container with balls of diverse colors inside. Each ball travels in one direction at a constant speed and rebounds when it hits the boundary. The agent, depicted as a white ball, is capable of accelerating or decelerating in any direction. As shown in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the framework ParMod NMT into sub-tasks, multiple agents are simultaneously cre￾ated to learn sub-tasks, with each agent focusing on one sub-task. A comprehensive policy will be synthesised from sub-policies ultimately [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. DFA of 𝜑 = (¬𝑟 ∧ ¬𝑔) U((𝑟 ∧ ¬𝑔) ∧ ○((¬𝑟 ∧ ¬𝑔)U(𝑔 ∧ ¬𝑟))). ¬𝑔 ∧ ¬𝑟, 𝑔 ∧ ¬𝑟, 𝑟 ∧ ¬𝑔 are logical representations of {𝑔, 𝑟}, {𝑔}, {𝑟} respectively. 𝑔 represents {𝑔, 𝑟} or {𝑔}, simi￾lar to 𝑟. Example 2 (rank of DFA states). Consider the DFA in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Relationship of task phases in ParMod thread 𝑇𝑘 . Line 16 checks whether the task phase is changed when interacting with the environment. If so, the next prod￾uct state 𝑠 ⊗ 𝑡+1 = ⟨𝑠𝑡+1, 𝑞𝑡+1⟩ will be saved in the initial state buffer with the same rank index of 𝑞𝑡+1, i…
Figure 6
Figure 6. Figure 6: Depiction of the Halfcheetah problem states [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Results on the training speed fail to do so. The curves of ParMod and Base for Task1 are basically identical. The reason is that Task1 is relatively sim￾ple for SAC itself. Moreover, Base exhibits superior perfor￾mance to Mod except Task5, Task6 and Task 8. We can in￾f…
Figure 8
Figure 8. Figure 8: Comparison between TPC and random task phase classification ident that ParMod achieves the perfect success rate of 100% in all tasks. ParMod is also the winner in policy quality and convergence rate. Here “—" denotes no success policy has been found. It is worth to men…
Figure 9
Figure 9. Figure 9: Comparison in the training speed: ParMod vs. SPECTRL (122% over Base, 281% over Mod and 158% for QRM). For the more complex Task3, the average convergence speed in￾creases to 334%, with improvements of 368% over Mod and 300% over Base, while QRM fails to find any succe…
Figure 12
Figure 12. Figure 12: The impact of the number of categories 𝑁 on training speed which is equivalent to using the baseline RL algorithm. Con￾versely, refining an NMT into too many categories is also not a good choice. Since an excessive categories necessitates more networks, thereby increa…
Figure 13
Figure 13. Figure 13: Results on the training speed of ParMod: Dis￾tributed (64 CPU cores) vs. Standalone (24 CPU cores) that, excessive threads will lead to contention for computa￾tional resources in standalone ParMod, resulting in mutual blocking among threads. Distributed ParMod can mit…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 24 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize "" * " " * ...

  2. [2]

    author Bozkurt, A. K. , author Wang, Y. , author Zavlanos, M. M. , & author Pajic, M. ( year 2020 ). title Control synthesis from linear temporal logic specifications using model-free reinforcement learning . In booktitle 2020 IEEE International Conference on Robotics and Automation (ICRA) \/ (pp. pages 10349--10355 ). organization IEEE

    work page 2020

  3. [3]

    , author Chen, O

    author Camacho, A. , author Chen, O. , author Sanner, S. , & author McIlraith, S. A. ( year 2018 ). title Non-markovian rewards expressed in ltl: Guiding search via reward shaping (extended version) . In booktitle GoalsRL, a workshop collocated with ICML/IJCAI/AAMAS \/

    work page 2018

  4. [4]

    author Cohen, M. H. , author Serlin, Z. , author Leahy, K. , & author Belta, C. ( year 2023 ). title Temporal logic guided safe model-based reinforcement learning: A hybrid systems approach . journal Nonlinear Analysis: Hybrid Systems \/ , volume 47 \/ , pages 101295 . https://www.sciencedirect.com/science/article/pii/S1751570X22000905. :https://doi.org/1...

    arXiv 2023

  5. [5]

    , & author Favorito, M

    author De Giacomo, G. , & author Favorito, M. ( year 2021 ). title Compositional approach to translate ltlf/ldlf into deterministic finite automata . In booktitle Proceedings of the International Conference on Automated Planning and Scheduling \/ (pp. pages 122--130 ). volume volume 31

    work page 2021

  6. [6]

    , & author Vardi, M

    author De Giacomo, G. , & author Vardi, M. Y. ( year 2013 ). title Linear temporal logic and linear dynamic logic on finite traces . In booktitle IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence \/ (pp. pages 854--860 ). organization Association for Computing Machinery

    work page 2013

  7. [7]

    , author Soyer, H

    author Espeholt, L. , author Soyer, H. , author Munos, R. , author Simonyan, K. , author Mnih, V. , author Ward, T. , author Doron, Y. , author Firoiu, V. , author Harley, T. , author Dunning, I. et al. ( year 2018 ). title Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures . In booktitle International conference on ...

    work page 2018

  8. [8]

    , author Zhou, A

    author Haarnoja, T. , author Zhou, A. , author Abbeel, P. , & author Levine, S. ( year 2018 ). title Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor . In editor J. G. Dy , & editor A. Krause (Eds.), booktitle Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ...

    work page 2018

Show all 34 references
  1. [9]

    , author Kroening, D

    author Hasanbeig, M. , author Kroening, D. , & author Abate, A. ( year 2020 ). title Deep reinforcement learning with temporal logics . In booktitle Formal Modeling and Analysis of Timed Systems: 18th International Conference, FORMATS 2020, Vienna, Austria, September 1--3, 202...

    2020

  2. [10]

    , author Guez, A

    author van Hasselt, H. , author Guez, A. , & author Silver, D. ( year 2016 ). title Deep reinforcement learning with double q-learning . In editor D. Schuurmans , & editor M. P. Wellman (Eds.), booktitle Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, ...

    2016 doi

  3. [11]

    , author Tb, D

    author Heess, N. , author Tb, D. , author Sriram, S. , author Lemmon, J. , author Merel, J. , author Wayne, G. , author Tassa, Y. , author Erez, T. , author Wang, Z. , author Eslami, S. et al. ( year 2017 ). title Emergence of locomotion behaviours in rich environments . journ...

    2017 arXiv

  4. [12]

    , author Quan, J

    author Horgan, D. , author Quan, J. , author Budden, D. , author Barth-Maron, G. , author Hessel, M. , author Van Hasselt, H. , & author Silver, D. ( year 2018 ). title Distributed prioritized experience replay . journal arXiv preprint arXiv:1803.00933 \/ ,

    2018 arXiv

  5. [13]

    author Icarte, R. T. , author Klassen, T. Q. , author Valenzano, R. , & author McIlraith, S. A. ( year 2022 ). title Reward machines: Exploiting reward function structure in reinforcement learning . journal Journal of Artificial Intelligence Research \/ , volume 73 \/ , pages 173--208

    2022

  6. [14]

    , author Alur, R

    author Jothimurugan, K. , author Alur, R. , & author Bastani, O. ( year 2019 ). title A composable specification language for reinforcement learning tasks . journal Advances in Neural Information Processing Systems \/ , volume 32 \/

    2019

  7. [15]

    , author Irpan, A

    author Kalashnikov, D. , author Irpan, A. , author Pastor, P. , author Ibarz, J. , author Herzog, A. , author Jang, E. , author Quillen, D. , author Holly, E. , author Kalakrishnan, M. , author Vanhoucke, V. et al. ( year 2018 ). title Qt-opt: Scalable deep reinforcement learn...

    2018 arXiv

  8. [16]

    , author Ostrovski, G

    author Kapturowski, S. , author Ostrovski, G. , author Quan, J. , author Munos, R. , & author Dabney, W. ( year 2018 ). title Recurrent experience replay in distributed reinforcement learning . In booktitle International conference on learning representations \/

    2018

  9. [17]

    , & author Soudjani, S

    author Kazemi, M. , & author Soudjani, S. ( year 2020 ). title Formal policy synthesis for continuous-state systems via reinforcement learning . In booktitle Integrated Formal Methods: 16th International Conference, IFM 2020, Lugano, Switzerland, November 16--20, 2020, Proceed...

    2020

  10. [18]

    author Konda, V. R. , & author Tsitsiklis, J. N. ( year 1999 ). title Actor-critic algorithms . In editor S. A. Solla , editor T. K. Leen , & editor K. M \" u ller (Eds.), booktitle Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, ...

    1999

  11. [19]

    , author Chen, T

    author Li, Z. , author Chen, T. , author Hong, Z.-W. , author Ajay, A. , & author Agrawal, P. ( year 2023 ). title Parallel q -learning: Scaling off-policy reinforcement learning under massively parallel simulation . In booktitle International Conference on Machine Learning \/...

    2023

  12. [20]

    , author Fu, W

    author Mei, Z. , author Fu, W. , author Gao, J. , author Wang, G. , author Zhang, H. , & author Wu, Y. ( year 2023 ). title Srl: Scaling distributed reinforcement learning to over ten thousand cores . journal arXiv preprint arXiv:2306.16688 \/ ,

    2023 arXiv

  13. [21]

    author Melo, F. S. ( year 2001 ). title Convergence of q-learning: A simple proof . journal Institute Of Systems and Robotics, Tech. Rep \/ , (pp. pages 1--4 )

    2001

  14. [22]

    , author Badia, A

    author Mnih, V. , author Badia, A. P. , author Mirza, M. , author Graves, A. , author Lillicrap, T. , author Harley, T. , author Silver, D. , & author Kavukcuoglu, K. ( year 2016 ). title Asynchronous methods for deep reinforcement learning . In booktitle International confere...

    2016

  15. [23]

    , author Kavukcuoglu, K

    author Mnih, V. , author Kavukcuoglu, K. , author Silver, D. , author Graves, A. , author Antonoglou, I. , author Wierstra, D. , & author Riedmiller, M. ( year 2013 ). title Playing atari with deep reinforcement learning . journal arXiv preprint arXiv:1312.5602 \/ ,

    2013 arXiv

  16. [24]

    , author Nishihara, R

    author Moritz, P. , author Nishihara, R. , author Wang, S. , author Tumanov, A. , author Liaw, R. , author Liang, E. , author Elibol, M. , author Yang, Z. , author Paul, W. , author Jordan, M. I. et al. ( year 2018 ). title Ray: A distributed framework for emerging \ AI \ appl...

    2018

  17. [25]

    , author Srinivasan, P

    author Nair, A. , author Srinivasan, P. , author Blackwell, S. , author Alcicek, C. , author Fearon, R. , author De Maria, A. , author Panneershelvam, V. , author Suleyman, M. , author Beattie, C. , author Petersen, S. et al. ( year 2015 ). title Massively parallel methods for...

    2015 arXiv

  18. [26]

    , author Sakakibara, A

    author Oura, R. , author Sakakibara, A. , & author Ushio, T. ( year 2020 ). title Reinforcement learning of control policy for linear temporal logic specifications using limit-deterministic generalized b \"u chi automata . journal IEEE Control Systems Letters \/ , volume 4 \/ ...

    2020

  19. [27]

    , author Hill, A

    author Raffin, A. , author Hill, A. , author Gleave, A. , author Kanervisto, A. , author Ernestus, M. , & author Dormann, N. ( year 2021 ). title Stable-baselines3: Reliable reinforcement learning implementations . journal The Journal of Machine Learning Research \/ , volume 2...

    2021

  20. [28]

    , author Wolski, F

    author Schulman, J. , author Wolski, F. , author Dhariwal, P. , author Radford, A. , & author Klimov, O. ( year 2017 ). title Proximal policy optimization algorithms . journal arXiv preprint arXiv:1707.06347 \/ ,

    2017 arXiv

  21. [29]

    , author Huang, A

    author Silver, D. , author Huang, A. , author Maddison, C. J. , author Guez, A. , author Sifre, L. , author Van Den Driessche, G. , author Schrittwieser, J. , author Antonoglou, I. , author Panneershelvam, V. , author Lanctot, M. et al. ( year 2016 ). title Mastering the game ...

    2016

  22. [30]

    , author Lever, G

    author Silver, D. , author Lever, G. , author Heess, N. , author Degris, T. , author Wierstra, D. , & author Riedmiller, M. A. ( year 2014 ). title Deterministic policy gradient algorithms . In booktitle Proceedings of the 31th International Conference on Machine Learning, ICM...

    2014

  23. [31]

    author Sutton, R. S. , & author Barto, A. G. ( year 2018 ). title Reinforcement learning: An introduction \/ . publisher MIT press

    2018

  24. [32]

    , author Gretton, C

    author Thi \'e baux, S. , author Gretton, C. , author Slaney, J. , author Price, D. , & author Kabanza, F. ( year 2006 ). title Decision-theoretic planning with non-markovian rewards . journal Journal of Artificial Intelligence Research \/ , volume 25 \/ , pages 17--74

    2006

  25. [33]

    , author Le, H

    author Voloshin, C. , author Le, H. , author Chaudhuri, S. , & author Yue, Y. ( year 2022 ). title Policy optimization with linear temporal logic constraints . journal Advances in Neural Information Processing Systems \/ , volume 35 \/ , pages 17690--17702

    2022

  26. [34]

    ( year 2009 )

    author Wawrzy \'n ski, P. ( year 2009 ). title A cat-like robot real-time learning to run . In booktitle Adaptive and Natural Computing Algorithms: 9th International Conference, ICANNGA 2009, Kuopio, Finland, April 23-25, 2009, Revised Selected Papers 9 \/ (pp. pages 380--390 ...

    2009

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.