Pith. sign in

REVIEW 3 major objections 5 minor 44 references

Automata-Conditioned Cooperative Multi-Agent Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By feeding each agent the minimized automaton remaining for every task, the paper turns a non-Markovian cooperative planning problem into a Markovian game and proves the reformulation is optimal in the infinite-horizon limit.

desk verdict Solid empirical framework for automata-conditioned MARL, but the optimality proof for reward shaping has a real gap. read the letter →

arxiv 2511.02304 v2 pith:JVV5I46Q submitted 2025-11-04 cs.MA cs.AIcs.CLcs.FLcs.LG

Beyazit Yalcinkaya , Marcell Vazquez-Chanlatte , Ameesh Shah , Hanna Krasowski , Sanjit A. Seshia This is my paper · ORCID
classification cs.MAcs.AIcs.CLcs.FLcs.LG
keywords multi-agentreinforcementlearningdeterministicfiniteautomataautomata-conditionedpoliciespotential-basedrewardshapingtask-conditionedcentralizedtrainingdecentralizedexecutiontemporaltaskgeneralizationRADembeddings
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that cooperative multi-agent reinforcement learning for temporally extended tasks can be made practical when tasks are given as deterministic finite automata (DFAs). Its core move is to show that the non-Markovian problem of tracking progress through a trace collapses to a Markovian game: at each step, agents observe the minimal DFA remaining after reading the latest labels, and the paper proves (Theorem 1) that maximizing discounted rewards in this reformulated game yields the same optimal policy as the original team-success objective in the limit γ→1. To make learning feasible, it adds potential-based reward shaping per agent, pretrained automata embeddings to avoid learning task representations from scratch, and shows that learned value functions can rank task assignments so the best permutation can be selected at test time. A sympathetic reader would care because this is a path from symbolic temporal specifications to decentralized team policies that scale from two to four agents and generalize across task classes.

What carries the argument

The load-bearing construction is the product DFA space D^n with the progression operation A/σ, which maps a vector of assigned DFAs to a vector of minimized residual DFAs after reading the latest labels. This turns the non-Markovian history-tracking problem into an MDP over (state, DFA-vector) pairs. Carrying the argument are: (i) the potential-based shaped reward R_D^n + γΦ_i(T)−Φ_i for each agent, where Φ_i=1 exactly when agent i's residual DFA is trivially accepting; (ii) RAD embeddings, a pretrained encoder Ψ that is injective up to DFA minimization, so the MDP can be compressed to a latent product space Z^n without changing rewards or transitions; and (iii) the induced value function V(

What would settle it

Take a two-agent grid world where one agent's sub-task is reachable by a shortcut that permanently blocks the other agent's path; train with the shaped reward and with the sparse team reward, and check whether the shaped policy converges to the same success probability as the sparse policy as γ→1. Alternatively, evaluate the term γ^T Φ(s_T) − Φ(s_0) under two policies with identical team-success probability but different completion times; if the shaped return differs, the shaping is not policy-invariant and Theorem 1 does not extend to the shaped reward.

Watch

Extended reading notes

Core claim

The central claim is Theorem 1: for a Markov game with n agents and n DFAs drawn from a prior, the joint policy that maximizes the discounted expected reward of the Markovian product game, where each agent observes its current minimized DFA progression, is also optimal for the original problem of maximizing the probability that all agents' traces satisfy their assigned DFAs, in the limit as the discount factor approaches 1. The paper further claims that this remains true after applying potential-based reward shaping with per-agent potential functions that fire when an individual DFA becomes accepting, and that using pretrained, provably correct RAD embeddings in place of the DFAs themselves

Load-bearing premise

The argument that potential-based reward shaping preserves optimality in this multi-agent episodic setting is inherited from the single-agent, zero-terminal-potential theorem in reference [5]; here the terminal potential is nonzero—an agent completing its own DFA receives 1—so the telescoping term γ^T Φ(s_T) − Φ(s_0) is policy-dependent, and the paper provides no proof that the shaped objective has the same optima as the original team-success objective.

Editorial extensions

If this is right

  • In the γ→1 limit, training on the shaped Markovian objective is equivalent to training on the sparse team-success objective, so the framework inherits optimality with respect to the ACC-MARL problem.
  • Because RAD embeddings uniquely represent minimized DFAs, policies trained on them can be reused across task classes; experiments show generalization to Reach and ReachAvoid tasks and to out-of-distribution DFAs with up to 10 states.
  • Learned value functions rank task assignments, so at test time the team can enumerate permutations and pick a Pareto-optimal assignment, improving success probability in asymmetric environments like Rooms.
  • The ablation results show that either removing reward shaping or learning embeddings from scratch degrades performance, particularly in four-agent settings, identifying the two ingredients as necessary for feasibility.
  • The framework produces emergent cooperative behaviors—button-pressing to unlock doors, door-holding, and task short-circuiting—that are not explicitly rewarded beyond the shaped sub-task signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The progression-based Markovian trick is not limited to DFAs; it should transfer to reward machines, LTL, or any automaton with a progression operation, potentially giving a unified treatment for symbolic multi-agent RL.
  • A natural testable extension is to use the same value functions to reassign tasks mid-episode, not just at the start; the paper leaves this unexplored, but the framework's state already contains all residual DFAs.
  • The optimal-assignment argument relies on full observability and enumeration; for larger teams, the RAD embedding space could be searched directly, which the paper itself notes as future work.
  • If the shaping preservation fails in the multi-agent case (see weakest assumption), the practical gains might still hold but the optimality guarantee would break; a direct test would be to compute shaped versus unshaped returns in a small grid world.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ACC-MARL, a framework for multi-task cooperative multi-agent reinforcement learning with tasks specified as deterministic finite automata (DFAs). The authors identify three challenges — history dependency, sparse team reward, and a representation bottleneck — and propose three corresponding remedies: (i) a Markovian reformulation in which agents observe the current minimal DFA of every agent's task; (ii) per-agent potential-based reward shaping designed to densify the team reward; and (iii) the use of pretrained RAD embeddings for DFA representations. The paper gives a formal problem statement (Problem 1), proves a Theorem 1 that the Markovian reformulation is asymptotically optimal as the discount factor tends to 1, asserts that the reward shaping preserves optimality, asserts that RAD embeddings carry over to a modified encoder, and proposes using learned value functions for optimal task assignment at test time. Experiments on Button and Room environments with two and four agents are reported, including ablations and a qualitative analysis of coordinated behaviors.

Significance. If the theoretical claims were correct, the paper would make a useful contribution to multi-agent RL with symbolic temporal specifications: it would provide a principled way to convert a non-Markovian team task into a Markovian one, a dense reward that provably does not change the optimal policy, and a scalable latent representation for DFA-conditioned policies. The empirical results, particularly the four-agent experiments and the qualitative demonstrations of door-holding and short-circuiting, are suggestive and the code release (DFAx, ACC-MARL) is a practical asset. The main source of value is the combination of a formal framework with a working implementation. However, the formal claims are currently under-supported, and one of them — the optimality of the shaped reward — is undermined by a standard and well-known failure mode of potential-based shaping with nonzero terminal potentials.

major comments (3)
  1. [Section 3.4, Eq. (7) and Appendix B.4] The claim that the per-agent shaped reward preserves optimality is not correct for the given potential. Summing the shaping term over an episode gives γ^T Φ_i(A_T) − Φ_i(A_0), and since terminal states of the product DFA space include all-trivial vectors, an agent whose task is satisfied has Φ_i(A_T)=1. Thus the shaped return differs from the original return by a policy-dependent quantity that rewards both faster completion and the agent's own success rather than team success. For example, a helper agent whose DFA is initially A_top receives −(1−γ) at every nonterminal step, penalizing longer episodes even when they are necessary for team success. The cited result [5] is for a common potential and, in the general case, assumes zero (or asymptotically vanishing) terminal potential. This is a load-bearing error: the proof that ACC-MARL solves Problem 1 relies on this step. The authors must
  2. [Section 3.5] The paper asserts that the RAD encoder Ψ satisfies the injectivity condition (7), and then uses this condition to claim that the latent-space reformulation over Z^n is equivalent to the DFA-space reformulation. However, the actual encoder is a modified GATv2 with a different featurization: DFA transitions are treated as edges rather than nodes, and the update rule (Appendix B.4) differs from the architecture in the cited works. No proof or reference is provided that this modified encoder preserves injectivity up to DFA minimization. Since the optimality of the π'' policies depends on this equivalence, the assertion is load-bearing. The authors should either supply a theorem for the new architecture, reduce it to the earlier results with explicit conditions, or state the injectivity as an assumption and verify it empirically.
  3. [Appendix B.1] The statement that the value-based assignment A★ maximizes the team success probability is not established. The functions V_i are the value functions of policies trained on the shaped objective, which (even ignoring the first concern) are not guaranteed to be optimal for Problem 1. Maximizing a sum of per-agent value functions over assignments does not imply maximizing the joint success probability; a Pareto-optimal point with respect to per-agent values is not necessarily one that maximizes J(π). The formal inequality asserted for all A′ in perm(A) needs a proof. As stated, this is an unsupported theoretical claim, although the experimental results in Rooms-2/Rooms-4 provide some empirical support.
minor comments (5)
  1. [Appendix B.1] The proof of Theorem 1 interchanges the limit γ→1− with the maximization over policies. Since episodes can in principle be unbounded and the objective is a discounted sum, the authors should justify this interchange (e.g., by bounded rewards and a properness condition) or state the result under an explicit finite-horizon/absorbing assumption.
  2. [Section 3.3] The paper uses 'potential-based reward shaping' and cites [5] without noting that the standard policy-invariance result applies to the sum of shaping terms over infinite horizons or with zero terminal potential. The text should be explicit about the terminal potential issue, even if a fix is adopted.
  3. [Section 4.2] Helper agents (trivially accepting DFAs) are sampled with positive probability. With the current shaping potential, these agents receive a negative reward at each step before termination. This artifact is not mentioned and can affect learning. Rewriting the potential as suggested would also eliminate this artifact.
  4. [Section 3.2] In Eq. (5), the terminal condition includes A_{t+1}∈D_T^n, but the reward R_D^n only fires when the vector equals A_top. It may be useful to distinguish all-trivial states with some rejecting entries from the all-accepting state; the current notation is slightly ambiguous.
  5. [Table 1] The table formatting is dense; the distinction between random and optimal assignment results is not as clear as it could be. Consider using separate rows or a clearer header.

Circularity Check

0 steps flagged · score 2.0 of 10

No by-construction circularity: the central Markovian reformulation theorem is proved directly; the main caveats are unproven imported optimality claims, not input-output equivalences.

full rationale

The paper's central derivation, Theorem 1, is an explicit equivalence proof (App. B.1) that constructs a history-to-state mapping and verifies matching rewards and transition probabilities; it does not assume the conclusion. The PBRS claim in Sec. 3.3 is imported from [5] and is a potential correctness gap because the per-agent potentials can be nonzero at terminal states, so the usual policy-invariance telescoping argument does not automatically apply; but this is an unsupported theorem transfer, not a reduction of the output to the input. Similarly, Sec. 3.4 imports Eq. (7) from the authors' prior [36,37] and applies it to a modified edge-featurized GATv2 encoder described in App. B.4 without re-proof; this is a self-citation with a missing verification step, but the central reformulation and the empirical ablations stand independently. The task-assignment section uses learned value functions as a heuristic proxy; it is underproved but not definitionally circular. Overall, no step in the paper equates its conclusion with its assumptions by construction, so circularity is minor at most.

Assumptions & free parameters 4 free parameters · 7 assumptions · 0 invented entities

The central claim rests on five domain assumptions and two ad hoc assumptions. The most fragile are the PBRS optimality assertion with nonzero terminal potentials and the injectivity of the modified RAD encoder; both are load-bearing for the 'optimality' claims but are not proved in this paper.

free parameters (4)
  • Discount factor γ = 0.99
    Chosen as a standard IPPO hyperparameter; the theory requires γ→1. It interacts with PBRS terminal-potential bias, so the trained policies may favor speed/partial success over pure team-success probability.
  • Max states in training DFA prior = 5 states (10 for OOD and pretraining)
    RAD DFAs are sampled with at most 5 states during policy training; generalization to 10-state OOD tasks is measured against this distributional choice.
  • Entropy coefficient = 0.02 (2-agent), 0.05 (4-agent)
    Hand-tuned per environment class; higher exploration in 4-agent layouts. Success probabilities in Table 1 depend on this choice.
  • Shaping potential Φ_i = 1 if A[i]=A_top else 0
    Defined by hand in Section 3.3. Its nonzero value at successful terminal states is the load-bearing assumption for the PBRS optimality claim.
assumptions (7)
  • domain assumption All tasks are plan DFAs, i.e., accepting states are sink states.
    Section 2: this ensures that once a task is satisfied it remains satisfied, so the Markovian reward at first acceptance matches the terminal reward in Problem 1.
  • domain assumption The product game M|_L M_D^n is Markovian and its optimal policy coincides with Problem 1 as γ→1.
    Central theorem, proved in Appendix B.1. The proof relies on the plan-DFA sink property and on the episode terminating when all DFAs are trivial.
  • ad hoc to paper PBRS with per-agent potentials preserves team-optimality in episodic multi-agent games with nonzero terminal potentials.
    Invoked at Eq. (6) with a citation to [5]. The standard theorem's conditions are not checked; terminal states have Φ_i=1 for successful agents, so the telescoping term is policy-dependent.
  • ad hoc to paper RAD embeddings Ψ satisfy Eq. (7): minimized DFAs are equal iff embeddings are equal, for all DFAs in the induced space, with the modified edge-featurized GATv2.
    Asserted in Section 3.4 borrowing from [37]; no proof is given for the modified architecture in this paper. If distinct tasks share an embedding, the latent reformulation is not equivalent.
  • domain assumption Labeling functions L_i are known, static, and partition each agent's observation space into alphabet symbols.
    Assumed in Problem 1 and used to update DFAs; the paper lists this as a limitation in Appendix A.
  • domain assumption IPPO in the independent-learning setting converges to a near-optimal joint policy.
    All empirical results are obtained with IPPO; no convergence guarantee for cooperative MARL is provided. The paper relies on the empirical success curves.
  • standard math DFA minimization returns a unique canonical representative (minimize(A)).
    Standard automata theory (Hopcroft). Used to define progression and Eq. (7).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automata-Conditioned Cooperative Multi-Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/JVV5I46Q

@misc{pith2026251102304,
  author       = {Pith},
  title        = {Pith review of: Automata-Conditioned Cooperative Multi-Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JVV5I46Q}},
  note         = {Machine review of arXiv:2511.02304}
}
read the original abstract

We study learning multi-task, multi-agent policies for cooperative, temporal objectives, under centralized training, decentralized execution. In this setting, using automata to represent tasks assigned to agents enables breaking down a team-level objective into simpler, smaller sub-tasks. However, existing approaches remain sample-inefficient and are limited to the single-task case, requiring retraining policies for each new task. In this work, we present Automata-Conditioned Cooperative Multi-Agent Reinforcement Learning (ACC-MARL), a framework for learning task-conditioned, decentralized team policies. We identify challenges to the feasibility of ACC-MARL, propose solutions, and prove that our approach is optimal. We further show that learned value functions can be used to assign tasks optimally at test time. Experiments demonstrate emergent task-aware, multi-step coordination among agents, such as pressing a button to unlock a door, holding the door, and short-circuiting tasks.

Figures

Figures reproduced from arXiv: 2511.02304 by the authors.

Figure 1
Figure 1. Motivating example Buttons-2– details are below. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of ACC-MARL with Rooms-2 environment given on the right. During training, we sample tasks from a [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Considered four-agent variations of TokenEnv are given in (a) and (b), and (c) presents a sample RAD DFA. Let 𝑉𝑖 : 𝑆 ∗ 𝑖 × 𝐷 𝑛 → R denote the optimal value function of the decentralized policy 𝜋𝑖 : 𝑆 ∗ 𝑖 × 𝐷 𝑛 maximizing 𝐽 (𝝅) as in Problem 1, and let A ∈ 𝐷 𝑛 be a DFA task assignment. Define 𝑉 : 𝐷 𝑛 → R as: 𝑉 (A) ≜ E𝜏∼M,𝝅,A "∑︁𝑛 𝑖=1 𝑉𝑖(𝜏𝑖 , A) # . We can use this as a proxy value function over task assignments and e… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Success probabilities of learned policies throughout training, reported over 5 random seeds – shaded regions indicate [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Policy architecture of an agent 𝑖. 4.3 Decentralized policies We learn a single policy deployed for each agent independently. The policy architecture for an agent 𝑖 is given in [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: A qualitative analysis of learned policies. See Figures 11 to 14 in the Appendix for longer traces. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Different types of DFA tasks sampled from the task distributions considered in the paper. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Success probabilities of learned policies throughout training, reported over 5 random seeds – shaded regions indicate [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Discounted returns of learned policies throughout training, reported over 5 random seeds – shaded regions indicate [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Discounted returns of learned policies throughout training, reported over 5 random seeds – shaded regions indicate [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Agent 1 in the helper agent role, assisting Agent 2 throughout the episode. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Agents holding the door for each other and completing both tasks. [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Agent 2 in the helper agent role, assisting Agent 1 throughout the episode. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Agents picking the shortest path in Agent 1’s DFA and completing both tasks. [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 8 linked inside Pith

  1. [5]

    Sam Devlin and Daniel Kudenko. 2011. Theoretical considerations of potential- based reward shaping for multi-agent systems. InInternational Conference on Autonomous Agents and Multiagent Systems. 225–232

    2011

  2. [1]

    Adrian K Agogino and Kagan Tumer. 2004. Unifying temporal and structural credit assignment problems. InInternational Conference on Autonomous Agents and Multiagent Systems

    2004

  3. [2]

    2018.JAX: composable transformations of Python+NumPy programs

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018.JAX: composable transformations of Python+NumPy programs. http://github.com/jax-ml/jax

    2018

  4. [3]

    Shaked Brody, Uri Alon, and Eran Yahav. 2021. How attentive are graph attention networks?arXiv preprint arXiv:2105.14491(2021)

    arXiv 2021

  5. [4]

    Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviy- chuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. 2020. Is independent learning all you need in the starcraft multi-agent challenge?arXiv preprint arXiv:2011.09533(2020)

    arXiv 2020

  6. [6]

    Zijian Guo, İlker Işık, HM Ahmad, and Wenchao Li. 2025. One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learning.arXiv preprint arXiv:2508.01561(2025)

    arXiv 2025

  7. [7]

    Mohammadhosein Hasanbeig, Daniel Kroening, and Alessandro Abate. 2020. Deep reinforcement learning with temporal logics. InInternational Conference on Formal Modeling and Analysis of Timed Systems. Springer, 1–22

    2020

  8. [8]

    John Hopcroft. 1971. An n log n algorithm for minimizing states in a finite automaton. InTheory of machines and computations. Elsevier, 189–196

    1971

Show all 44 references
  1. [9]

    Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. 2021. Updet: Universal multi-agent reinforcement learning via policy decoupling with transformers. arXiv preprint arXiv:2101.08001(2021)

    2021 arXiv

  2. [10]

    Shariq Iqbal, Robby Costales, and Fei Sha. 2022. ALMA: Hierarchical Learning for Composite Multi-Agent Tasks. InAdvances in Neural Information Processing Systems, Vol. 35. 7155–7166. https://proceedings.neurips.cc/paper_files/paper/ 2022/file/2f27964513a28d034530bfdd117ea31d-P...

    2022

  3. [11]

    Shariq Iqbal, Christian A Schroeder De Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. 2021. Randomized entity-wise factorization for multi- agent reinforcement learning. InInternational Conference on Machine Learning. 4596–4606

    2021

  4. [12]

    Mathias Jackermeier and Alessandro Abate. 2024. DeepLTL: Learning to Effi- ciently Satisfy Complex LTL Specifications for Multi-Task RL.arXiv preprint arXiv:2410.04631(2024)

    2024 arXiv

  5. [13]

    Kishor Jothimurugan, Rajeev Alur, and Osbert Bastani. 2019. A composable specification language for reinforcement learning tasks.Advances in Neural Information Processing Systems32 (2019)

    2019

  6. [14]

    Kishor Jothimurugan, Suguman Bansal, Osbert Bastani, and Rajeev Alur. 2021. Compositional reinforcement learning from logical specifications.Advances in Neural Information Processing Systems34 (2021), 10026–10039

    2021

  7. [15]

    Mohammad Karimadini, Hai Lin, and Ali Karimoddini. 2016. Cooperative tasking for deterministic specification automata.Asian Journal of Control18, 6 (2016), 2078–2087

    2016

  8. [16]

    Xianglong Li, Yuan Li, Jieyuan Zhang, Xinhai Xu, and Donghong Liu. 2024. A hierarchical multi-agent allocation-action learning framework for multi-subtask games.Complex & Intelligent Systems10, 2 (2024), 1985–1995

    2024

  9. [17]

    Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. 2022. Discovered policy optimisation.Advances in Neural Information Processing Systems35 (2022), 16455–16468

    2022

  10. [18]

    Yuxiang Mai, Yifan Zang, Qiyue Yin, Wancheng Ni, and Kaiqi Huang. 2023. Deep Multitask Multiagent Reinforcement Learning With Knowledge Transfer.IEEE Transactions on Games16, 3 (2023), 566–576

    2023

  11. [19]

    Jan Martens and Anton Wijs. 2024. An Evaluation of Massively Parallel Algo- rithms for DFA Minimization.arXiv preprint arXiv:2410.22764(2024)

    2024 arXiv

  12. [20]

    Kanefumi Matsuyama, Kefan Su, Jiangxing Wang, Deheng Ye, and Zongqing Lu. 2025. CORD: Generalizable Cooperation via Role Diversity.arXiv preprint arXiv:2501.02221(2025)

    2025 arXiv

  13. [21]

    Cyrus Neary, Zhe Xu, Bo Wu, and Ufuk Topcu. 2020. Reward machines for cooperative multi-agent reinforcement learning.arXiv preprint arXiv:2007.01962 (2020)

    2020 arXiv

  14. [22]

    Andrew Y Ng, Daishi Harada, and Stuart Russell. 1999. Policy Invariance Un- der Reward Transformations: Theory and Application to Reward Shaping. In International Conference on Machine Learning. 278—-287

    1999

  15. [23]

    Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P How, and John Vian. 2017. Deep decentralized multi-task multi-agent reinforcement learn- ing under partial observability. InInternational Conference on Machine Learning. PMLR, 2681–2690

    2017

  16. [24]

    Wenjie Qiu, Wensen Mao, and He Zhu. 2023. Instructing goal-conditioned reinforcement learning agents with temporal logic objectives.Advances in Neural Information Processing Systems36 (2023), 39147–39175

    2023

  17. [25]

    Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Garðar Ingvarsson Juto, Timon Willi, Ravi Hammond, Akbir Khan, Chris- tian Schroeder de Witt, et al. 2024. Jaxmarl: Multi-agent rl environments and algorithms in jax.Advances in Neural Informatio...

    2024

  18. [26]

    Philipp Schillinger, Mathias Bürger, and Dimos V Dimarogonas. 2018. Decompo- sition of finite LTL specifications for efficient multi-agent planning. InDistributed Autonomous Robotic Systems: The 13th International Symposium. Springer, 253– 267

    2018

  19. [27]

    Philipp Schillinger, Mathias Bürger, and Dimos V Dimarogonas. 2018. Simulta- neous task allocation and planning for temporal logic goals in heterogeneous multi-robot systems.The international journal of robotics research37, 7 (2018), 818–838

    2018

  20. [28]

    Ameesh Shah, Niklas Lauffer, Thomas Chen, Nikhil Pitta, and Sanjit A Seshia

  21. [29]

    Ameesh Shah, Cameron Voloshin, Chenxi Yang, Abhinav Verma, Swarat Chaud- huri, and Sanjit A. Seshia. 2025. LTL-Constrained Policy Optimization with Cycle Experience Replay.Transactions on Machine Learning Research(2025)

    2025

  22. [30]

    Sophia Smith, Cyrus Neary, and Ufuk Topcu. 2023. Automatic Decomposition of Reward Machines for Decentralized Multiagent Reinforcement Learning. In IEEE Conference on Decision and Control (CDC). 5423–5430

    2023

  23. [31]

    Klassen, Richard Valenzano, and Sheila A

    Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Valenzano, and Sheila A. McIlraith

  24. [32]

    Pashootan Vaezipoor, Andrew C Li, Rodrigo A Toro Icarte, and Sheila A Mcil- raith. 2021. LTL2Action: Generalizing LTL instructions for multi-task RL. In International Conference on Machine Learning. 10497–10508

    2021

  25. [33]

    Cameron Voloshin, Abhinav Verma, and Yisong Yue. 2023. Eventual Dis- counting Temporal Logic Counterfactual Experience Replay.arXiv preprint arXiv:2303.02135(2023)

    2023 arXiv

  26. [34]

    Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. 2021. {RODE}: Learning Roles to Decompose Multi-Agent Tasks. InInternational Conference on Learning Representations. https://openreview.net/ forum?id=TTUVg6vkNjK

    2021

  27. [35]

    Beyazit Yalcinkaya, Niklas Lauffer, Marcell Vazquez-Chanlatte, and Sanjit Seshia

  28. [36]

    Beyazit Yalcinkaya, Niklas Lauffer, Marcell Vazquez-Chanlatte, and Sanjit A Seshia. 2024. Compositional Automata Embeddings for Goal-Conditioned Rein- forcement Learning. InNeural Information Processing Systems, Vol. 38

    2024

  29. [37]

    Beyazit Yalcinkaya, Niklas Lauffer, Marcell Vazquez-Chanlatte, and Sanjit A Seshia. 2025. Provably Correct Automata Embeddings for Optimal Automata- Conditioned Reinforcement Learning. InProceedings of the International Confer- ence on Neuro-symbolic Systems. 661–675

    2025

  30. [38]

    Yang Yu, Qiyue Yin, Junge Zhang, and Kaiqi Huang. 2023. Prioritized tasks mining for multi-task cooperative multi-agent reinforcement learning. InInternational Conference on Autonomous Agents and Multiagent Systems. 1615–1623

    2023

  31. [39]

    Fuxiang Zhang, Chengxing Jia, Yi-Chen Li, Lei Yuan, Yang Yu, and Zongzhang Zhang. 2022. Discovering generalizable multi-agent coordination skills from multi-task offline data. InThe Eleventh International Conference on Learning Representations

    2022

  32. [40]

    Guobin Zhu, Rui Zhou, Wenkang Ji, Hongyin Zhang, Donglin Wang, and Shiyu Zhao. 2025. Multi-Task Multi-Agent Reinforcement Learning via Skill Graphs. IEEE Robotics and Automation Letters(2025)

    2025

  33. [41]

    𝑛Û 𝑖=1 𝜏|= 𝐿𝑖 A0[𝑖] # =E 𝑠0∼𝜄 A0∼𝜄𝑛 𝐷

    Xiaofei Zhu, Jiazhong Xu, Jianghua Ge, Yaping Wang, and Zhiqiang Xie. 2023. Multi-task multi-agent reinforcement learning for real-time scheduling of a dual-resource flexible job shop with robots.Processes11, 1 (2023), 267. B APPENDIX B.1 Proof of Theorem 1 Theorem 1.Maximizin...

    2023

  34. [2022]

    Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning.J. Artif. Int. Res.73 (2022). https://doi.org/10.1613/jair.1.12440

    2022 doi

  35. [2023]

    In NeurIPS 2023 Workshop on Goal-Conditioned Reinforcement Learning

    Automata conditioned reinforcement learning with experience replay. In NeurIPS 2023 Workshop on Goal-Conditioned Reinforcement Learning

    2023

  36. [2025]

    InInter- national Conference on Autonomous Agents and Multiagent Systems

    Learning Symbolic Task Decompositions for Multi-Agent Teams. InInter- national Conference on Autonomous Agents and Multiagent Systems. 1904––1913

    1904

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.