Pith. sign in

REVIEW 3 major objections 3 minor 2 cited by

Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus

T0 review · 3 major / 3 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A consensus vector turns cooperative multi-agent control into a single-agent PPO problem.

desk verdict CMAT is a genuinely new architecture for centralized MARL, but the 'at least as good as MAT' theorem is false for the deterministic consensus actually implemented. read the letter →

arxiv 2604.13472 v2 pith:NCHXKQIP submitted 2026-04-15 cs.LG cs.AIcs.MA

Zijian Zhao , Jing Gao , Sen Li This is my paper · ORCID
classification cs.LGcs.AIcs.MA
keywords multi-agentreinforcementlearningconsensushierarchicalsingle-agentRLtransformerPPOorder-independentdecisionmakingcooperativeMARLlatentcoordination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that cooperative MARL with a fully observable global state can be recast as hierarchical single-agent reinforcement learning: first generate one latent consensus vector from all agents' observations, then sample every agent's action simultaneously conditioned on that consensus. The joint policy factorizes as π(c|O)∏π_i(a_i|O,c), and because the consensus generator is deterministic, the PPO ratio collapses to a product of per-agent action probabilities, so single-agent PPO can optimize the joint policy directly. If correct, this removes the action-order sensitivity and credit-assignment distortions of sequential multi-agent transformers, and replaces their Nash-equilibrium convergence caveat with a single-agent optimization perspective. The paper reports consistent wins over strong baselines on StarCraft II, Multi-Agent MuJoCo, and Google Research Football. The central promise is that a learned consensus is a sufficient coordination device: once agents share it, independent simultaneous action sampling is enough.

What carries the argument

The consensus vector c=µ(O), a deterministic function of the global observation produced by the decoder's m iterations and the actor-compressor. It acts as the coordination signal: conditioned on c, each agent's action distribution becomes independent of the other agents' actions. This factorization is what lets the joint policy be trained with single-agent PPO, because the importance ratio in Eq. (11) becomes a product of per-agent conditional-action probabilities. The iteration count m=n is the main new hyperparameter, chosen to mirror MAT's n-step decoding process.

What would settle it

Run CMAT on the two-action one-step cooperative game shown in Fig. 2 with consensus iterations m=n. If CMAT converges to the Pareto-suboptimal joint action (A,A) or to (B,A), the claim that consensus avoids MAT's convergence trap fails. For the policy-class claim, enumerate the tabular game and check whether a MAT policy π1(a1)π2(a2|a1) can be expressed as π_1(a1|O,c)π_2(a2|O,c) for some c=µ(O); a counterexample would disprove the strict inclusion.

Watch

Extended reading notes

Core claim

CMAT replaces the autoregressive, order-dependent generation of agent actions in MAT with an iterative latent consensus. A Transformer encoder processes the joint observation, a critic-compressor reduces it to an initial consensus vector, a decoder iterates that vector m times, and an actor-compressor mixes all iterations into the final consensus c=µ(O). Conditioned on c, all agents sample their actions simultaneously and order-independently. The paper claims this makes the joint policy a hierarchical SARL policy optimized by single-agent PPO (Eqs. 3, 10, 11), eliminating order bias and the actor-critic inconsistency that besets sequential MAT variants. It further claims, in Appendix E, that

Load-bearing premise

The load-bearing premise is that a single deterministic consensus vector c=µ(O), generated before any agent acts, can encode enough coordination information to make the agents' actions conditionally independent without losing expressive power; the appendix's proof that the policy class strictly contains MAT's additionally assumes c can encode the actions of earlier agents in a sequential ordering, which c cannot do because those actions do not yet exist when c is generated.

Editorial extensions

If this is right

  • Joint actions are sampled simultaneously and order-independently, so CMAT avoids the order-sensitivity and credit-assignment distortions of sequential MAT variants without expanding the search space to n! action orders.
  • The joint policy is trainable with single-agent PPO; no multi-agent policy-gradient or sequential trust-region machinery is required beyond the consensus factorization.
  • If the consensus is informative, the action head cannot simply ignore it without sacrificing reward, so empirical wins over a consensus-free simultaneous baseline (Triple-BERT-style) are evidence that the consensus carries coordination information.
  • Fine-tuning either the consensus generator or the action heads while freezing the other yields similar improvements, supporting the paper's interpretation of the two-phase training as block-coordinate ascent toward a Stackelberg equilibrium.
  • Ablations show that mixing all iteration outputs beats using only the last consensus, and that m=n outperforms m=0, n/2, and 2n, indicating the iteration count matters for consensus quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The appendix's argument that CMAT's policy class strictly contains MAT's requires c to encode the prefix actions of a sequential policy, but c is generated before any action exists; unless the encoder or decoder sees past actions, that inclusion is not established. This is an editorial observation about a proof gap, not a claim the paper makes explicitly.
  • The practical ceiling of the method is the information capacity of a single consensus vector. For tasks where coordination requires agent-specific or pairwise messages, one shared vector may become a bottleneck; comparing CMAT against communication-based methods on such tasks would clarify this.
  • The 'single-agent PPO' framing is a loss-function equivalence, not a formal reduction of the multi-agent problem; the centralized critic and shared reward remain. A sharper statement of when the consensus factorization strictly enlarges the representable policy class would put the bridge to SARL on firmer ground.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes CMAT, a centralized cooperative MARL algorithm that replaces MAT's sequential action generation with an iterative latent-consensus mechanism. A Transformer encoder processes joint observations; a decoder iterates a consensus vector for m steps; all agents then emit actions simultaneously conditioned on this consensus. The joint policy is written as π(c|O)∏π_i(a_i|O,c) and trained with single-agent PPO. Experiments on SMAC, Multi-Agent MuJoCo, and Google Research Football claim superior performance over MAT, PMAT, HAPPO, MAPPO, and Triple-BERT, with ablations on the consensus-compression design and the number of consensus iterations.

Significance. If the empirical results hold, CMAT offers a practically useful architecture that removes MAT's sensitivity to action-generation order and demonstrates that a hierarchical consensus mechanism can improve MARL. The paper provides code, evaluates across three benchmark families, and includes targeted ablations. However, the theoretical claim that CMAT's policy class strictly contains MAT's, and hence that CMAT is at least as good as MAT, is inconsistent with the deterministic consensus actually implemented. The empirical section also lacks final aggregate tables, making the performance claims harder to verify. The conceptual idea of casting MARL as hierarchical SARL is promising, but the manuscript in its current form overstates its theoretical support.

major comments (3)
  1. [Appendix E.6 / Eq. (11) / Algorithm 1] Appendix E.6 defines ΠCMAT with a stochastic consensus distribution πc and claims ΠMAT ⊊ ΠCMAT. The actual method, however, uses a deterministic consensus generator c=μθ(O) (Eq. 11; Algorithm 1, lines 11–14). Under deterministic consensus the marginal joint policy is π(A|O)=∏_{i=1}^n π_i(a_i|O, μ(O)), a product distribution over agents. This class is a strict subset of MAT's autoregressive class, not a superset. Therefore the conclusion that 'the optimal policy within CMAT's policy class is at least as good as that within MAT's policy class' does not apply to the implemented algorithm. The authors should either implement a stochastic consensus (sampling c) or substantially qualify/remove the theoretical superiority claim.
  2. [Eq. (11)] The derivation of the PPO importance ratio is formally incorrect as written. The expression πθ(A|O)/πθ−(A|O) = πcθ(c|O)∏π_i(a_i|O,c) / [πcθ−(c−|O)∏π_i(a_i|O,c−)], with c=μθ(O) and c−=μθ−(O), is not the ratio of marginal probabilities of the sampled joint action, because the numerator and denominator evaluate the joint distribution at different consensus values. The final expression ∏π_iθ(a_i|O,μθ(O))/∏π_iθ−(a_i|O,μθ−(O)) is the correct importance ratio for the marginal deterministic-consensus policy, but the intermediate cancellation argument is unsound. Please rewrite the derivation starting from the marginal π(A|O)=∫πc(c|O)∏π_i(a_i|O,c)dc.
  3. [Section 4.2 / Figure 4] The paper states that CMAT 'achieves superior performance in most scenarios' and after fine-tuning 'the best performance across all scenarios', but no aggregate final-return table is provided. Training curves with 5 seeds are insufficient to assess statistical significance, especially where curves overlap. Please report final mean and standard deviation (or win rate) for every method and scenario, and state whether any significance test was performed. Also discuss the potential impact of the reduced rollout threads (Appendix D) on the baseline methods' reported performance.
minor comments (3)
  1. [Section 4.3] The intuitive claim that 'in the worst-case scenario, CMAT can degrade to MAT' when m=n is unsupported: the CMAT decoder iterates a hidden consensus, not a sequence of agent actions, so it cannot recover MAT's autoregressive action distribution. Please clarify what this statement means or remove it.
  2. [Eq. (10)] The ratio R_i^t(θ) in the actor loss is identical for all agents i; the subscript i is misleading. Consider using R^t(θ).
  3. [Appendix E.6] The definition of ΠCMAT as {π(A|O)=πc(c|O)∏π_i(ai|O,c)} is ambiguous: the left side is a distribution over A, while the right side is a joint distribution over (c,A). Please define ΠCMAT as the set of marginals ∫πc(c|O)∏π_i(ai|O,c)dc.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Eq. (3) is a proposed factorization and Eq. (11) is the corresponding importance ratio; empirical claims are tested against external baselines, so no prediction reduces to a fitted input or self-citation.

full rationale

The claimed derivation chain is not circular. Eq. (3) defines a joint policy as a latent-conditioned factorization; Eq. (11) is exactly the importance ratio for that factorization under single-agent PPO, so the PPO training claim follows from the model definition rather than from the experimental results. The consensus c is learned, not fit to a target quantity that is later 'predicted'. The experiments compare CMAT against external baselines (MAT, PMAT, Triple-BERT, HAPPO, MAPPO) on standard benchmarks, so the main empirical claims are self-contained. The paper does cite its own prior work (Triple-BERT [12], CSI-BERT2 [23], stage lighting [41]) and the compressor design is attributed to [23;24], but these citations are architectural/related-work precedence, not the load-bearing argument: the central expressive-advantage claim in Appendix E.6 is derived from the paper's own policy-class comparison, not from a self-cited theorem. The most serious issue is a validity gap, not circularity: Appendix E.6 defines ΠCMAT with a stochastic πc and claims 'any sequential policy can be recovered by letting c encode the prefix actions', while the implemented Algorithm 1 and Eq. (15) use deterministic c=µ(O) generated before all actions; with that policy class ΠCMAT is a product-distribution class and need not contain ΠMAT. Appendix E.8 itself concedes 'Proving convergence for CMAT with neural function approximation remains open.' These are correctness concerns about the theory, but they do not make the derivation equivalent to its inputs. Similarly, the fine-tuning ablation compares CMAT variants against each other, which is an experimental interpretation, not a fitted-input-called-prediction. Therefore no circular step is present.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The method's central new ingredient is a deterministic consensus bottleneck. The paper introduces one hand-set hyperparameter (iteration count m), relies on standard tabular convergence assumptions, and makes two ad hoc moves: treating deterministic consensus density as 1 in the PPO ratio, and claiming policy-class inclusion over MAT.

free parameters (1)
  • consensus iteration times m = n (number of agents)
    Chosen by hand as default; sensitivity studied in §4.3 (m=0, ⌊n/2⌋, 2n) with no formal criterion. The paper calls m the only newly introduced hyperparameter.
assumptions (4)
  • domain assumption The underlying environment is a fully observable cooperative Markov game with shared reward and centralized execution.
    Section 2.1: 'fully cooperative and fully observable setting ... equivalent to a centralized controller with complete information.'
  • standard math Tabular softmax policy parameterization, Robbins–Monro step sizes, ergodicity, and finiteness of state/action/consensus sets.
    Appendix E.1 lists these as required for the local convergence arguments; the authors acknowledge they do not hold in the deep network implementation.
  • ad hoc to paper For a deterministic consensus generator, πc(c|O)=1 and πc(c−|O)=1, allowing cancellation in the PPO importance ratio.
    Section 3.3, Eq. (11): the ratio is reduced by dropping πc terms; this is only meaningful if the joint density is understood as marginalized over the deterministic c=µ(O).
  • ad hoc to paper ΠMAT⊊ΠCMAT; any sequential MAT policy can be recovered by encoding prefix actions in c.
    Appendix E.6: stated as immediate, but c=µ(O) is generated before any action, so it cannot encode prefix actions; the inclusion claim is false.
invented entities (1)
  • latent consensus vector c
    purpose: High-level action / coordination signal shared by all agents, generated by iterative decoder compression before per-agent actions.
    An internal latent produced by decoder iterations and compressor; the paper offers no external falsifiable handle for it beyond the overall method's performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus." pith.science (2026). https://pith.science/paper/NCHXKQIP

@misc{pith2026260413472,
  author       = {Pith},
  title        = {Pith review of: Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NCHXKQIP}},
  note         = {Machine review of arXiv:2604.13472}
}
read the original abstract

Cooperative multi-agent reinforcement learning (MARL) is widely used to address large joint observation and action spaces by decomposing a centralized control problem into multiple interacting agents. However, such decomposition often introduces additional challenges, including non-stationarity, unstable training, weak coordination, and limited theoretical guarantees. In this paper, we propose the Consensus Multi-Agent Transformer (CMAT), a centralized framework that bridges cooperative MARL to a hierarchical single-agent reinforcement learning (SARL) formulation. CMAT treats all agents as a unified entity and employs a Transformer encoder to process the large joint observation space. To handle the extensive joint action space, we introduce a hierarchical decision-making mechanism in which a Transformer decoder autoregressively generates a high-level consensus vector, simulating the process by which agents reach agreement on their strategies in latent space. Conditioned on this consensus, all agents generate their actions simultaneously, enabling order-independent joint decision making and avoiding the sensitivity to action-generation order in conventional Multi-Agent Transformers (MAT). This factorization allows the joint policy to be optimized using single-agent PPO while preserving expressive coordination through the latent consensus. To evaluate the proposed method, we conduct experiments on benchmark tasks from StarCraft II, Multi-Agent MuJoCo, and Google Research Football. The results show that CMAT achieves superior performance over recent centralized solutions, sequential MARL methods, and conventional MARL baselines. The code for this paper is available at:https://github.com/RS2002/CMAT .

Figures

Figures reproduced from arXiv: 2604.13472 by the authors.

Figure 1
Figure 1. Comparison between CMAT and Conventional Decentralized MARL Methods. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Payoff Matrix of Co￾operative Game: The joint ac￾tions (A, A) and (B, B) are both NE, while (B, B) is the global optimum. Agent 1 A B Agent 2 A 1 -100 B 0 100 MAT [8] formulates the cooperative Markov game as a sequential model based on the Transformer architecture [9], with the goal of capturing action dependencies among agents. Specifically, it first employs an encoder to extract observation features for all agent… view at source ↗
Figure 3
Figure 3. Network Architecture: The Transformer encoder first extracts the features and relationships [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Training Curves under 5 Random Seeds: The shadow parts represent the standard deviation. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Ablation Study under 5 Random Seeds 4.3 Ablation Study and Sensitivity Analysis To further illustrate the efficacy of each proposed module, we conduct a series of ablation studies and sensitivity analyses of hyper-parameters, including: • Consensus Mixture Versus Last …

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Aggregate in the Advantage, Not the Ratio: A Canonical-Form Analysis of Cooperative Multi-Agent Policy Optimization

    cs.MA 2026-07 conditional novelty 7.0 of 10

    For cooperative PPO, the expected gradient at the on-policy point depends on advantage and ratio aggregation supports only through their matrix product, and the variance-optimal design keeps the ratio per-agent and ag...

  2. Action-Factored Multi-Agent Reinforcement Learning for Scalable Quantum Device Tuning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Online action-space factorization via Kalman-refined cross-capacitance lets shared multi-agent policies zero-shot tune larger quantum-dot arrays with near-constant steps.

Reference graph

Works this paper leans on

72 extracted references · 11 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Multi-agent reinforcement learning for resources allocation optimization: a survey,

    M. A. Hady, S. Hu, M. Pratama, Z. Cao, and R. Kowalczyk, “Multi-agent reinforcement learning for resources allocation optimization: a survey,”Artificial Intelligence Review, vol. 58, no. 11, p. 354, 2025

    2025

  2. [2]

    A comprehensive survey on multi- agent cooperative decision-making: Scenarios, approaches, challenges and perspectives,

    W. Jin, H. Du, B. Zhao, X. Tian, B. Shi, and G. Yang, “A comprehensive survey on multi- agent cooperative decision-making: Scenarios, approaches, challenges and perspectives,”arXiv preprint arXiv:2503.13415, 2025

    arXiv 2025

  3. [3]

    Monotonic value function factorisation for deep multi-agent reinforcement learning,

    T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic value function factorisation for deep multi-agent reinforcement learning,”Journal of Machine Learning Research, vol. 21, no. 178, pp. 1–51, 2020

    2020

  4. [4]

    Mean field multi-agent reinforcement learning,

    Y . Yang, R. Luo, M. Li, M. Zhou, W. Zhang, and J. Wang, “Mean field multi-agent reinforcement learning,” inInternational conference on machine learning, pp. 5571–5580, PMLR, 2018

    2018

  5. [5]

    Counterfactual multi-agent policy gradients,

    J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” inProceedings of the AAAI conference on artificial intelligence, vol. 32, 2018

    2018

  6. [6]

    A review of cooperative multi-agent deep reinforcement learning,

    A. Oroojlooy and D. Hajinezhad, “A review of cooperative multi-agent deep reinforcement learning,”Applied Intelligence, vol. 53, no. 11, pp. 13677–13722, 2023

    2023

  7. [7]

    Exponential topology-enabled scalable communication in multi-agent reinforcement learning,

    X. Li, X. Wang, C. Bai, and J. Zhang, “Exponential topology-enabled scalable communication in multi-agent reinforcement learning,” inThe Thirteenth International Conference on Learning Representations, 2025

    2025

  8. [8]

    Multi-agent reinforcement learning is a sequence modeling problem,

    M. Wen, J. Kuba, R. Lin, W. Zhang, Y . Wen, J. Wang, and Y . Yang, “Multi-agent reinforcement learning is a sequence modeling problem,”Advances in Neural Information Processing Systems, vol. 35, pp. 16509–16521, 2022

    2022

Show all 72 references
  1. [9]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  2. [10]

    Pmat: Optimizing action generation order in multi-agent reinforcement learning,

    K. Hu, M. Wen, X. Wang, S. Zhang, Y . Shi, M. Li, M. Li, and Y . Wen, “Pmat: Optimizing action generation order in multi-agent reinforcement learning,” inProceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, pp. 997–1005, 2025

    2025

  3. [11]

    Aoad-mat: Transformer-based multi-agent deep reinforcement learning model considering agents’ order of action decisions,

    S. Takayama and K. Fujita, “Aoad-mat: Transformer-based multi-agent deep reinforcement learning model considering agents’ order of action decisions,” inInternational Conference on Principles and Practice of Multi-Agent Systems, pp. 303–310, Springer, 2025. 10

    2025

  4. [12]

    Triple-bert: Do we really need marl for order dispatch on ride-sharing platforms?,

    Z. Zhao and S. Li, “Triple-bert: Do we really need marl for order dispatch on ride-sharing platforms?,” inThe Fourteenth International Conference on Learning Representations, 2026

    2026

  5. [13]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  6. [14]

    The starcraft multi-agent challenge,

    S. Whiteson, M. Samvelyan, T. Rashid, C. De Witt, G. Farquhar, N. Nardelli, T. Rudner, C. Hung, P. Torr, and J. Foerster, “The starcraft multi-agent challenge,” inProceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, pp. 2186–2188, 2019

    2019

  7. [15]

    Deep multi- agent reinforcement learning for decentralized continuous cooperative control,

    C. S. de Witt, B. Peng, P.-A. Kamienny, P. Torr, W. Böhmer, and S. Whiteson, “Deep multi- agent reinforcement learning for decentralized continuous cooperative control,”arXiv preprint arXiv:2003.06709, vol. 19, 2020

    2003 arXiv

  8. [16]

    Google research football: A novel reinforcement learning environment,

    K. Kurach, A. Raichuk, P. Sta´nczyk, M. Zaj ˛ ac, O. Bachem, L. Espeholt, C. Riquelme, D. Vincent, M. Michalski, O. Bousquet,et al., “Google research football: A novel reinforcement learning environment,” inProceedings of the AAAI conference on artificial intelligence, vol. 34...

    2020

  9. [17]

    Markov games as a framework for multi-agent reinforcement learning,

    M. L. Littman, “Markov games as a framework for multi-agent reinforcement learning,” in Machine learning proceedings 1994, pp. 157–163, Elsevier, 1994

    1994

  10. [18]

    High-dimensional continuous control using generalized advantage estimation,

    J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,”arXiv preprint arXiv:1506.02438, 2015

    2015 arXiv

  11. [19]

    Trust region policy optimisa- tion in multi-agent reinforcement learning,

    J. Kuba, R. Chen, M. Wen, Y . Wen, F. Sun, J. Wang, and Y . Yang, “Trust region policy optimisa- tion in multi-agent reinforcement learning,” inICLR 2022-10th International Conference on Learning Representations, p. 1046, The International Conference on Learning Representation...

    2022

  12. [20]

    Heterogeneous-agent reinforcement learning,

    Y . Zhong, J. G. Kuba, X. Feng, S. Hu, J. Ji, and Y . Yang, “Heterogeneous-agent reinforcement learning,”Journal of Machine Learning Research, vol. 25, no. 32, pp. 1–67, 2024

    2024

  13. [21]

    Maximum entropy heterogeneous-agent reinforcement learning,

    J. Liu, Y . Zhong, S. Hu, H. Fu, Q. FU, X. Chang, and Y . Yang, “Maximum entropy heterogeneous-agent reinforcement learning,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  14. [22]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo- g...

    2019

  15. [23]

    Csi-bert2: A bert-inspired framework for efficient csi prediction and classification in wireless communication and sensing,

    Z. Zhao, F. Meng, Z. Lyu, H. Li, X. Li, and G. Zhu, “Csi-bert2: A bert-inspired framework for efficient csi prediction and classification in wireless communication and sensing,”IEEE Transactions on Mobile Computing, 2025

    2025

  16. [24]

    Midibert-piano: Large-scale pre-training for symbolic music classification tasks,

    Y .-H. Chou, I.-C. Chen, J. Ching, C.-J. Chang, and Y .-H. Yang, “Midibert-piano: Large-scale pre-training for symbolic music classification tasks,”Journal of Creative Music Systems, vol. 8, no. 1, 2024

    2024

  17. [25]

    R. D. Luceet al.,Individual choice behavior, vol. 4. Wiley New York, 1959

    1959

  18. [26]

    The analysis of permutations,

    R. L. Plackett, “The analysis of permutations,”Journal of the Royal Statistical Society Series C: Applied Statistics, vol. 24, no. 2, pp. 193–202, 1975

    1975

  19. [27]

    The surprising effectiveness of ppo in cooperative multi-agent games,

    C. Yu, A. Velu, E. Vinitsky, J. Gao, Y . Wang, A. Bayen, and Y . Wu, “The surprising effectiveness of ppo in cooperative multi-agent games,”Advances in neural information processing systems, vol. 35, pp. 24611–24624, 2022

    2022

  20. [28]

    A survey of progress on cooperative multi-agent reinforcement learning in open environment,

    L. Yuan, Z. Zhang, L. Li, C. Guan, and Y . Yu, “A survey of progress on cooperative multi-agent reinforcement learning in open environment,”arXiv preprint arXiv:2312.01058, 2023. 11

    2023 arXiv

  21. [29]

    Hysteretic q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams,

    L. Matignon, G. J. Laurent, and N. Le Fort-Piat, “Hysteretic q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams,” in2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 64–69, IEEE, 2007

    2007

  22. [30]

    Modelling the dynamic joint policy of teammates with attention multi-agent ddpg,

    H. Mao, Z. Zhang, Z. Xiao, and Z. Gong, “Modelling the dynamic joint policy of teammates with attention multi-agent ddpg,”arXiv preprint arXiv:1811.07029, 2018

    2018 arXiv

  23. [31]

    Shapley counterfactual credits for multi-agent reinforcement learning,

    J. Li, K. Kuang, B. Wang, F. Liu, L. Chen, F. Wu, and J. Xiao, “Shapley counterfactual credits for multi-agent reinforcement learning,” inProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 934–942, 2021

    2021

  24. [32]

    Value-decomposition networks for cooperative multi- agent learning based on team reward,

    P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V . Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls,et al., “Value-decomposition networks for cooperative multi- agent learning based on team reward,” inProceedings of the 17th International Conference o...

  25. [33]

    Value-decomposition multi-agent actor-critics,

    J. Su, S. Adams, and P. Beling, “Value-decomposition multi-agent actor-critics,” inProceedings of the AAAI conference on artificial intelligence, vol. 35, pp. 11352–11360, 2021

    2021

  26. [34]

    Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning,

    K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y . Yi, “Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning,” inInternational conference on machine learning, pp. 5887–5896, PMLR, 2019

    2019

  27. [35]

    Hierarchical value decomposition for effective on-demand ride- pooling,

    J. Hao and P. Varakantham, “Hierarchical value decomposition for effective on-demand ride- pooling,” inProceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pp. 580–587, 2022

    2022

  28. [36]

    Efficient distributed reinforcement learning through agreement,

    P. Varshavskaya, L. P. Kaelbling, and D. Rus, “Efficient distributed reinforcement learning through agreement,” inDistributed Autonomous Robotic Systems 8, pp. 367–378, Springer, 2009

    2009

  29. [37]

    Learning multiagent communication with backpropagation,

    S. Sukhbaatar, R. Fergus,et al., “Learning multiagent communication with backpropagation,” Advances in neural information processing systems, vol. 29, 2016

    2016

  30. [38]

    Tarmac: Targeted multi-agent communication,

    A. Das, T. Gervet, J. Romoff, D. Batra, D. Parikh, M. Rabbat, and J. Pineau, “Tarmac: Targeted multi-agent communication,” inInternational Conference on machine learning, pp. 1538–1546, PMLR, 2019

    2019

  31. [39]

    Context-aware communication for multi-agent reinforcement learning,

    X. Li and J. Zhang, “Context-aware communication for multi-agent reinforcement learning,” inProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 1156–1164, 2024

    2024

  32. [40]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” inInternational Conference on Learning ...

    2021

  33. [41]

    Automatic stage lighting control: Is it a rule- driven process or generative task?,

    Z. Zhao, D. Jin, Z. Zhou, and X. Zhang, “Automatic stage lighting control: Is it a rule- driven process or generative task?,” inThe Fourteenth International Conference on Learning Representations, 2026

    2026

  34. [42]

    On transforming reinforcement learning with transformers: The development trajectory,

    S. Hu, L. Shen, Y . Zhang, Y . Chen, and D. Tao, “On transforming reinforcement learning with transformers: The development trajectory,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 8580–8599, 2024

    2024

  35. [43]

    Deep recurrent q-learning for partially observable mdps.,

    M. J. Hausknecht and P. Stone, “Deep recurrent q-learning for partially observable mdps.,” in AAAI fall symposia, vol. 45, p. 141, 2015

    2015

  36. [44]

    Stabilizing transformers for reinforcement learning,

    E. Parisotto, F. Song, J. Rae, R. Pascanu, C. Gulcehre, S. Jayakumar, M. Jaderberg, R. L. Kaufman, A. Clark, S. Noury,et al., “Stabilizing transformers for reinforcement learning,” in International conference on machine learning, pp. 7487–7498, PMLR, 2020

    2020

  37. [45]

    Transdreamer: Reinforcement learning with trans- former world models,

    C. Chen, J. Yoon, Y .-F. Wu, and S. Ahn, “Transdreamer: Reinforcement learning with trans- former world models,” inDeep RL Workshop NeurIPS 2021, 2021. 12

    2021

  38. [46]

    Dream to control: Learning behaviors by latent imagination,

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, “Dream to control: Learning behaviors by latent imagination,” inInternational Conference on Learning Representations, 2020

    2020

  39. [47]

    Offline reinforcement learning as one big sequence modeling problem,

    M. Janner, Q. Li, and S. Levine, “Offline reinforcement learning as one big sequence modeling problem,”Advances in neural information processing systems, vol. 34, pp. 1273–1286, 2021

    2021

  40. [48]

    Decision transformer: Reinforcement learning via sequence modeling,

    L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch, “Decision transformer: Reinforcement learning via sequence modeling,”Advances in neural information processing systems, vol. 34, pp. 15084–15097, 2021

    2021

  41. [49]

    Online decision transformer,

    Q. Zheng, A. Zhang, and A. Grover, “Online decision transformer,” ininternational conference on machine learning, pp. 27042–27059, PMLR, 2022

    2022

  42. [50]

    Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl,

    T. Yamagata, A. Khalil, and R. Santos-Rodriguez, “Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl,” inInternational Conference on Machine Learning, pp. 38989–39007, PMLR, 2023

    2023

  43. [51]

    A survey on transfer learning,

    S. J. Pan and Q. Yang, “A survey on transfer learning,”IEEE Transactions on knowledge and data engineering, vol. 22, no. 10, pp. 1345–1359, 2009

    2009

  44. [52]

    Can wikipedia help offline reinforcement learning?,

    M. Reid, Y . Yamada, and S. S. Gu, “Can wikipedia help offline reinforcement learning?,”arXiv preprint arXiv:2201.12122, 2022

    2022 arXiv

  45. [53]

    Pre-trained language models for interactive decision-making,

    S. Li, X. Puig, C. Paxton, Y . Du, C. Wang, L. Fan, T. Chen, D.-A. Huang, E. Akyürek, A. Anandkumar,et al., “Pre-trained language models for interactive decision-making,”Advances in Neural Information Processing Systems, vol. 35, pp. 31199–31212, 2022

    2022

  46. [54]

    Masked autoencoding for scalable and generalizable decision making,

    F. Liu, H. Liu, A. Grover, and P. Abbeel, “Masked autoencoding for scalable and generalizable decision making,”Advances in Neural Information Processing Systems, vol. 35, pp. 12608– 12618, 2022

    2022

  47. [55]

    Masked trajectory models for prediction, representation, and control,

    P. Wu, A. Majumdar, K. Stone, Y . Lin, I. Mordatch, P. Abbeel, and A. Rajeswaran, “Masked trajectory models for prediction, representation, and control,” inInternational Conference on Machine Learning, pp. 37607–37623, PMLR, 2023

    2023

  48. [56]

    Prompting decision transformer for few-shot policy generalization,

    M. Xu, Y . Shen, S. Zhang, Y . Lu, D. Zhao, J. Tenenbaum, and C. Gan, “Prompting decision transformer for few-shot policy generalization,” ininternational conference on machine learning, pp. 24631–24645, PMLR, 2022

    2022

  49. [57]

    Con- textual transformer for offline meta reinforcement learning,

    R. Lin, Y . Li, X. Feng, Z. Zhang, X. H. W. Fung, H. Zhang, J. Wang, Y . Du, and Y . Yang, “Con- textual transformer for offline meta reinforcement learning,”arXiv preprint arXiv:2211.08016, 2022

    2022 arXiv

  50. [58]

    Multi-game decision transformers,

    K.-H. Lee, O. Nachum, M. S. Yang, L. Lee, D. Freeman, S. Guadarrama, I. Fischer, W. Xu, E. Jang, H. Michalewski,et al., “Multi-game decision transformers,”Advances in neural information processing systems, vol. 35, pp. 27921–27936, 2022

    2022

  51. [59]

    A generalist agent,

    S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y . Sulsky, J. Kay, J. T. Springenberg,et al., “A generalist agent,”arXiv preprint arXiv:2205.06175, 2022

    2022 arXiv

  52. [60]

    Learning multi-agent communication from graph modeling perspective,

    S. Hu, L. Shen, Y . Zhang, and D. Tao, “Learning multi-agent communication from graph modeling perspective,” 2024

    2024

  53. [61]

    MaskMA: Towards zero-shot multi-agent decision making with mask-based collaborative learning,

    J. Liu, Y . Zhang, C. Li, Z. You, Z. Zhou, C. Yang, Y . Yang, Y . Liu, and W. Ouyang, “MaskMA: Towards zero-shot multi-agent decision making with mask-based collaborative learning,”Trans- actions on Machine Learning Research, 2024

    2024

  54. [62]

    Updet: Universal multi-agent reinforcement learning via policy decoupling with transformers,

    S. Hu, F. Zhu, X. Chang, and X. Liang, “Updet: Universal multi-agent reinforcement learning via policy decoupling with transformers,”arXiv preprint arXiv:2101.08001, 2021

    2021 arXiv

  55. [63]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga,et al., “Pytorch: An imperative style, high-performance deep learning library,”Advances in Neural Information Processing Systems, vol. 32, 2019. 13

    2019

  56. [64]

    On the theory of policy gradient methods: Optimality, approximation, and distribution shift,

    A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan, “On the theory of policy gradient methods: Optimality, approximation, and distribution shift,”Journal of Machine Learning Research, vol. 22, no. 98, pp. 1–76, 2021

    2021

  57. [65]

    On the global convergence rates of softmax policy gradient methods,

    J. Mei, C. Xiao, C. Szepesvari, and D. Schuurmans, “On the global convergence rates of softmax policy gradient methods,” inInternational conference on machine learning, pp. 6820–6829, PMLR, 2020

    2020

  58. [66]

    Convergence rates of bayesian network policy gradient for cooperative multi-agent reinforcement learning,

    D. Chen, Z. Zhang, X. Kuang, X. Shen, O. Ozer, and Q. Zhang, “Convergence rates of bayesian network policy gradient for cooperative multi-agent reinforcement learning,” inNeurIPS 2024 Workshop on Bayesian Decision-making and Uncertainty, 2024

    2024

  59. [67]

    A reinforcement learning algorithm for obtaining the nash equilib- rium of multi-player matrix games,

    V . Nanduri and T. K. Das, “A reinforcement learning algorithm for obtaining the nash equilib- rium of multi-player matrix games,”IIE Transactions, vol. 41, no. 2, pp. 158–167, 2009

    2009

  60. [68]

    R. S. Sutton, A. G. Barto,et al.,Reinforcement learning: An introduction, vol. 1. MIT press Cambridge, 1998

    1998

  61. [69]

    A generalized alternating method for bilevel learning under the polyak-{\L}ojasiewicz condition,

    Q. Xiao, S. Lu, and T. Chen, “A generalized alternating method for bilevel learning under the polyak-{\L}ojasiewicz condition,”arXiv preprint arXiv:2306.02422, 2023

    2023 arXiv

  62. [70]

    Convergence proof for actor-critic methods applied to ppo and rudder,

    M. Holzleitner, L. Gruber, J. Arjona-Medina, J. Brandstetter, and S. Hochreiter, “Convergence proof for actor-critic methods applied to ppo and rudder,” inTransactions on large-scale data- and knowledge-centered systems XLVIII: special issue in memory of univ. prof. dr. roland...

    2021

  63. [71]

    Ppo-clip attains global optimality: To- wards deeper understandings of clipping,

    N.-C. Huang, P.-C. Hsieh, K.-H. Ho, and I.-C. Wu, “Ppo-clip attains global optimality: To- wards deeper understandings of clipping,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 12600–12607, 2024

    2024

  64. [72]

    Implicit learning dynamics in stackelberg games: Equilibria characterization, convergence analysis, and empirical study,

    T. Fiez, B. Chasnov, and L. Ratliff, “Implicit learning dynamics in stackelberg games: Equilibria characterization, convergence analysis, and empirical study,” inInternational conference on machine learning, pp. 3133–3144, PMLR, 2020. 14 Appendix Contents A Related Work 16 A.1...

    2020

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.