Pith. sign in

REVIEW 4 major objections 5 minor 49 references

Tackling Uncertainties in Multi-Agent Reinforcement Learning through Integration of Agent Termination Dynamics

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding a barrier-function loss built from agent deaths improves convergence and task completion in cooperative multi-agent reinforcement learning.

desk verdict A sensible but modest safety-signal idea for distributional MARL, with an overreaching theory section that should be cut or corrected. read the letter →

arxiv 2501.12061 v1 pith:ULWF5RXV submitted 2025-01-21 cs.LG cs.MA

Somnath Hazra , Pallab Dasgupta , Soumyajit Dey This is my paper · ORCID
classification cs.LGcs.MA
keywords multi-agentreinforcementlearningdistributionalcontrolbarrierfunctionagentterminationdynamicscooperativeMARLStarCraftIImicromanagementquantileregressiongradientsurgery
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that cooperative multi-agent reinforcement learning can be made faster and safer by learning from an intrinsic safety signal the environment already provides: the number of agents that die. It adds a control-barrier-function loss, built from discounted agent terminations, to a distributional value-learning objective, and combines the two gradients with projection-based gradient surgery. The authors report that this hybrid loss improves convergence, reduces unsafe exploration in early training, and outperforms state-of-the-art distributional MARL baselines on StarCraft II micromanagement and MetaDrive benchmarks. The paper also gives a tabular convergence-rate bound and a probabilistic safety-verification statement for the resulting policy. If the central claim holds, it means a cheap, hand-computed safety signal can stabilize multi-agent learning without changing how agents execute their policies.

What carries the argument

The central machinery is the control barrier function (CBF) certificate, a function $B^{\pi}: \mathcal{S} \to \mathbb{R}$ with the invariant property $B^{\pi}(s') - B^{\pi}(s) \leq -\lambda_B B^{\pi}(s)$; here $B^{\pi}(s) = \text{deaths}(s) + \gamma_B B^{\pi}(s')$, so the agent-death count plays the role of vulnerability. The network is trained by minimizing Eq. (5), the positive part of the invariant violation, over on-policy transitions, alongside a distributional Huber-quantile loss from IQN. The two losses are merged by gradient surgery, a PCGrad-style projection used when their angle exceeds 90 degrees, rather than by scalar weighting alone. Distributional value factorization follows the mean-shape decomposition satisfying the Distributional IGM principle, with a QPLEX-style dueling architecture; a hypernetwork layer in each local policy generates input weights from the predicted return distribution. The theoretical claim relies on the performance-difference lemma and a CRPO-style tabular analysis, with convergence of the underlying TD and value-decomposition updates assumed.

What would settle it

Take the trained policy, run many held-out StarCraft episodes, and compute the empirical frequency with which $B^{\pi}(s') > (1-\lambda_B)B^{\pi}(s)$ on consecutive on-policy states; if the invariant is violated on a substantial fraction of transitions while the win rate stays high, the barrier interpretation is unsupported and the loss is functioning as reward shaping. A complementary check is to replace the death count in Eq. (4) with a random or state-independent signal and retrain: if win rates do not drop, the death-count mechanism is not the cause of the reported gains.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that integrating agent termination dynamics into distributional MARL through a barrier-function loss improves convergence and task completion. The barrier is defined as $B^{\pi}(s) = \text{agents dead at } s + \gamma_B B^{\pi}(s')$, and the method minimizes the amount by which consecutive on-policy states violate the barrier-decrease condition $B^{\pi}(s') - B^{\pi}(s) \leq -\lambda_B B^{\pi}(s)$. This safety loss is combined with the Huber quantile (IQN) loss using gradient manipulation, so that reward and safety objectives either agree or are projected out of conflict. The paper reports higher fractions of test battles won above 0.6, 0.8, and 0.9 win-rate thresholds on StarCraft hard and super-hard scenarios, and higher average returns on MetaDrive, compared with distributional MARL baselines; integrating the same loss into QMIX also improves easy-scenario performance. The theoretical section gives a suboptimality bound of $O\left(\sqrt{|\mathcal{S}||\mathcal{U}| / ((1-\gamma)^3 T)}\right)$ for tabular natural policy gradient with this loss, and formulates safety verification as a chance-constrained program solved by sampling and discarding.

Load-bearing premise

The method assumes that the number of agents that die, discounted forward in time, behaves like a true safety barrier: on the states the policy visits, the barrier value can be made to decrease by at least $\lambda_B B^{\pi}(s)$ at every step, and pushing the loss in Eq. (5) toward zero is what lowers future deaths; the Section 5 proof assumes, rather than proves, a parallel convergence analysis for the constraint. If the death count is not a valid barrier certificate, the added loss is just reward shaping.

Editorial extensions

If this is right

  • On hard and super-hard SMAC scenarios, the death-count barrier loss reports a higher fraction of runs with test win rates above 0.6, 0.8, and 0.9 than the distributional MARL baselines RMIX, DMIX, QDIST, CBF, RESZ, and RISKQ.
  • The same barrier loss can be attached to a non-distributional factorization: adding it to QMIX improves easy-scenario win rates relative to QMIX, VDN, and QTRAN, suggesting the safety signal is independent of distributional value learning.
  • The tabular analysis claims a suboptimality gap of $O\left(\sqrt{|\mathcal{S}||\mathcal{U}| / ((1-\gamma)^3 T)}\right)$ with confidence $1-\delta$, placing the method in the same sample-complexity class as standard policy-gradient results.
  • Safety can be certified probabilistically: with confidence at least $1-\beta$, the probability that a test trajectory exceeds the allowed termination count is at most $\varepsilon$, via scenario optimization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is real, its benefit should be largest in the regime the paper targets: early training, when return distributions are least reliable and death counts are most informative; one testable prediction is that removing the barrier loss after convergence should barely change final performance.
  • The Eq. (5) loss behaves like a barrier certificate only if the invariant $B^{\pi}(s') \leq (1-\lambda_B)B^{\pi}(s)$ is actually satisfiable on the visited states; when it is not, the term functions as reward shaping, so inspecting empirical violation rates on held-out episodes would separate the two explanations.
  • The construction is not tied to deaths: it could be applied to other intrinsic fault metrics, such as cumulative health loss, collision counts, or distance to road boundaries in driving, whenever the metric decreases along successful trajectories.
  • Because the barrier loss is computed on on-policy samples, the method assumes the sampling distribution covers the states where safety matters; training with off-policy replay or a stale barrier is a natural stress test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes augmenting distributional multi-agent reinforcement learning with a barrier-function loss derived from agent termination counts. The barrier signal B(s) = deaths(s) + gamma_B B(s') is turned into a hinge loss (Eq. 5), combined with a Huber quantile TD loss through PCGrad-style gradient manipulation (Eqs. 7-8), and integrated into a QPLEX/DMIX-style value factorization with a hypernetwork for local policy input layers. The theoretical section claims a convergence rate for the barrier-augmented policy (Theorem 5.1) and a chance-constrained safety verification bound (Theorem 5.2). The empirical section reports win rates on StarCraft II micromanagement and MetaDrive scenarios, comparing DBF/QBF against RMIX, DMIX, QDIST, CBF, RESQ, RISKQ, VDN, QMIX, and QTRAN, plus ablations on gamma_B and gradient weights.

Significance. The core idea of using agent termination dynamics as an environment-derived safety signal is attractive and could be valuable if the barrier invariant is actually enforced, as it would give a simple way to inject safety into distributional MARL without a separate safety critic. I credit the paper for evaluating against a wide set of baselines, for including ablations of gamma_B and the gradient weights, and for making the code available. However, the significance is currently conditional: the main convergence theorem is not proved in the appendix, and the empirical claims lack error bars, seed counts, and direct casualty metrics, so the evidence for the central 'improved convergence and safety' claim is weaker than the text suggests.

major comments (4)
  1. [Theorem 5.1 / Appendix C] Theorem 5.1 is not established by the supplied proof. The only bridge from the reward-only analysis to the barrier-augmented loss is the sentence in Appendix C: 'we assume a parallel convergence analysis for the constraint holds true, with our constraints scaling with the rewards to obey the limit.' This is an explicit assumption, not a derivation, and neither L_B from Eq. (5) nor the barrier invariant from Definition 1 appears in Lemmas C.2 and C.3. Moreover, the appendix's final displayed bound is of order |S||U|/((1-gamma)^1.5 sqrt(T)), while the theorem states Theta(sqrt(|S||U|/((1-gamma)^3 T))); these differ by a factor of sqrt(|S||U|), so the algebra in the appendix does not produce the stated theorem. Either give a complete proof that handles the barrier term and the PCGrad coupling, or explicitly weaken Theorem 5.1 to a conjecture with numerical support.
  2. [Sections 4.1 and Algorithm 1] The paper never shows that B(s) = deaths(s) + gamma_B B(s') satisfies the barrier certificate invariant in Definition 1, namely B(s') - B(s) <= -lambda_B B(s), on on-policy trajectories, nor that minimizing Eq. (5) under the conditional update in Algorithm 1 (lines 7-10, where L_B is applied only when V_B > omega) maintains that invariant. If the recursively computed death count does not behave like a valid barrier certificate, the barrier loss is essentially a reward-shaping term, and the claims of improved safety and convergence are not supported. The authors should either prove the invariant under the update rule or report empirical barrier-violation rates alongside the win-rate curves.
  3. [Section 6, Figures 5-9] The empirical comparison is under-specified: there are no error bars, no number of seeds, no inter-quartile ranges, and no casualty statistics, despite the paper's safety-focused claims. In addition, the hyperparameter selection in Section 6.3 (gamma_B on 5m_vs_6m and 3s_vs_5z; beta_Q/beta_B on 3s_vs_5z and so_many_baneling) uses scenarios that also appear in the main comparisons, creating a selection-on-test risk. Please report mean and variance over multiple seeds for all tasks and, for the safety claim, report the empirical distribution of the termination metric V_B^pi(tau).
  4. [Section 5.1, Eq. (9) and Theorem 5.2] The safety verification result is quoted from the scenario-optimization literature and assumes N i.i.d. test constraints V_B^pi(tau_i) <= omega. The paper does not show that the policy produced by Algorithm 1 has any specified relation to the solution of the chance-constrained program, particularly because the training objective uses a soft PCGrad-combined loss and the barrier loss is applied only conditionally. The distribution shift between on-policy training samples and the final test trajectories is not addressed, so as stated Theorem 5.2 does not certify the trained policy; it only restates a generic scenario-optimization bound.
minor comments (5)
  1. [Eq. (5)] In Eq. (5), |S| is used to denote the number of visited states in a trajectory, which conflicts with the standard use of |S| for the cardinality of the state space; please use a different symbol such as M or T.
  2. [Lemma C.2] In Lemma C.2, the notation V_t(s,u) and V_t(s) are used inconsistently; the action argument appears in V_t in the lemma statement and in the proof, although V is defined as a state-dependent value function. Please clean up the notation so that each symbol has one meaning.
  3. [Section 6.1] The sentence 'Results 7 indicate DBF achieves a higher fraction of test wins' should read 'Figure 7 indicates...'.
  4. [Theorems 5.1 and 5.2] The symbol m is used both for the number of policy parameters in Theorem 5.2 and for the policy parameter vector in Delta(U)^{|S|} in Theorem 5.1; these two uses should be disambiguated.
  5. [Section 6.2] The sentence 'Episodes were terminated when more than half the agents are eliminated during training or when any agent reached termination during evaluation' is ambiguous about whether the training-time and evaluation-time termination criteria are different; please clarify.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 5.1's proof assumes, rather than derives, convergence of the barrier-constraint term, making the strongest formal claim partially circular.

  1. other [Appendix C, paragraph after Lemma C.3 (Proof of Theorem 5.1)]
    "The lemmas C.2 and C.3 proved above also hold true for the barrier function that we had used for applying the loss due to the constraint. Owing to better convergence (proved by the authors in [24]) we have used the loss function based on the barrier function. For time complexity analysis, we assume a parallel convergence analysis for the constraint holds true, with our constraints scaling with the rewards to obey the limit."

    Theorem 5.1 promises a convergence bound for the policy trained with the barrier-function loss. The proof's Lemmas C.2 and C.3 bound only the reward component of the update. The paper then asserts, without derivation, that the same lemmas hold for the barrier function and that 'a parallel convergence analysis for the constraint holds true.' This is the missing step: the convergence of the barrier-augmented objective is not a consequence of the preceding lemmas; it is assumed. Since that convergence is precisely what Theorem 5.1 is advertised to prove, the formal claim reduces to an unproved assumption.

full rationale

The empirical core of the paper is not circular: the barrier loss is computed from observable agent-death counts, an independent signal from the environment, and the win-rate comparisons with baselines are external benchmarks. The overlap between survival and winning, and the tuning of gamma_B and beta_Q/beta_B on two target scenarios, are methodological weaknesses but not construction-level circularity. The load-bearing formal claim, however, is partially circular: Theorem 5.1's proof transitions from the reward-only bound to the barrier-augmented bound by assuming 'a parallel convergence analysis for the constraint holds true,' i.e., by assuming the convergence that the theorem is meant to establish. The final bound in Appendix C also does not match the theorem's stated rate, which further severs the derivation. Score 6 reflects a central theoretical claim whose proof reduces to an assumption, while the empirical contribution retains independent content.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central method rests on two empirical hyperparameters (gamma_B, beta_Q/beta_B) and one unspecified hyperparameter (lambda_B), plus the untested assumption that a hand-computed death count behaves like a barrier certificate in a partially observable MARL setting. The theoretical guarantee in Theorem 5.1 additionally assumes the underlying TD update converges and that the CRPO-style analysis in the cited literature transfers to the barrier-augmented loss. No new physical or conceptual entities are postulated.

free parameters (3)
  • gamma_B = 0.5
    Discount factor for the barrier function; tuned via ablation on 5m_vs_6m and 3s_vs_5z (Fig. 10), chosen to maximize win rates.
  • lambda_B = not reported
    Convergence rate hyperparameter in Eq. 5; no value is given anywhere in the text, yet it controls the strictness of the barrier invariant and hence the loss magnitude.
  • beta_Q, beta_B = 0.5, 0.5
    Gradient weights for reward vs barrier loss; chosen from ablation on 3s_vs_5z and so_many_baneling (Fig. 11), then fixed for all other experiments.
assumptions (3)
  • domain assumption The barrier certificate invariant (Definition 1, condition 3) transfers from continuous control to discrete, partially observable MARL, and minimizing Eq. 5 enforces it.
    Section 4.1 defines B^pi from death counts but does not prove that the computed B satisfies the invariant; the loss only penalizes violations.
  • domain assumption Convergence of the underlying TD update algorithm is assumed, and the CRPO-style analysis in the cited literature applies to the barrier-augmented NPG update.
    Theorem 5.1 preamble states 'assuming convergence of the underlying TD update algorithm'; Appendix C says 'we assume a parallel convergence analysis for the constraint holds true' and removes gradient weights as constants.
  • ad hoc to paper On-policy barrier-loss samples can be combined with off-policy TD loss via PCGrad without introducing bias.
    Algorithm 1 uses replay buffer samples for TD error and on-policy samples for the barrier loss; no analysis of this mixing is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tackling Uncertainties in Multi-Agent Reinforcement Learning through Integration of Agent Termination Dynamics." pith.science (2026). https://pith.science/paper/ULWF5RXV

@misc{pith2026250112061,
  author       = {Pith},
  title        = {Pith review of: Tackling Uncertainties in Multi-Agent Reinforcement Learning through Integration of Agent Termination Dynamics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ULWF5RXV}},
  note         = {Machine review of arXiv:2501.12061}
}
read the original abstract

Multi-Agent Reinforcement Learning (MARL) has gained significant traction for solving complex real-world tasks, but the inherent stochasticity and uncertainty in these environments pose substantial challenges to efficient and robust policy learning. While Distributional Reinforcement Learning has been successfully applied in single-agent settings to address risk and uncertainty, its application in MARL is substantially limited. In this work, we propose a novel approach that integrates distributional learning with a safety-focused loss function to improve convergence in cooperative MARL tasks. Specifically, we introduce a Barrier Function based loss that leverages safety metrics, identified from inherent faults in the system, into the policy learning process. This additional loss term helps mitigate risks and encourages safer exploration during the early stages of training. We evaluate our method in the StarCraft II micromanagement benchmark, where our approach demonstrates improved convergence and outperforms state-of-the-art baselines in terms of both safety and task completion. Our results suggest that incorporating safety considerations can significantly enhance learning performance in complex, multi-agent environments.

Figures

Figures reproduced from arXiv: 2501.12061 by the authors.

Figure 1
Figure 1. Integration of the safety constraints along with loss [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall framework illustrating the training objective. The calculated errors are combined using gradient ma [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Analysis of gradient manipulation. Owing to optimization on two tasks, there may occasionally occur a conflict among the gradients 𝑔𝑄 and 𝑔𝐵, i.e. 𝜃 > 90◦ . In those cases, it is essential to project the gradient vector onto the normal plane of the other vector for effective updates to the network [46]. For cases when 𝜃 ≤ 90◦ , i.e., the gradients do not conflict, the gradient manipulation is not required. The overa… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The input layer weights of 𝜋𝑖 generated using a hyper-network frm previous step’s return distribution. Now we focus on the architecture of the local policy, 𝜋𝑖 . As illus￾trated in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Comparison with the state-of-the-art algorithms using the StarCraft hard and super-hard scenarios. Our method is [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Comparison with the baselines using the StarCraft easy scenarios. Our method is marked as QBF (Q-function evaluation [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Summary of evaluations on StarCraft easy scenarios. We also compared our algorithm with traditional MRL ap￾proaches: • VDN [38] which uses sum￾mation of local re￾turns • QMIX [26] uses monotonic fac￾torization of value function • QTRAN [32] uses transform￾ers for facto…
Figure 11
Figure 11. Figure 11: Ablation Study on different values of 𝛽𝐵 and 𝛽𝑄 on StarCraft environments. 7 CONCLUSION In this paper, we proposed a novel approach to distributional multi￾agent reinforcement learning (MARL) that integrates fault tolera￾tion based on an agent safety boundary (based o…
Figure 10
Figure 10. Figure 10: Ablation Study on 𝛾𝐵 tested on StarCraft. In our experiments, for the discount factor of barrier function, we had used 𝛾𝐵 = 0.5. We had tested various values for the hyper￾parameter on two StarCraft scenarios before using them in the rest of our evaluations. The hyper…
Figure 12
Figure 12. Figure 12: Analysis of win rate on the StarCraft hard and super-hard scenarios. [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Analysis of win rate on the StarCraft easy scenarios. [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 38 canonical work pages

  1. [1]

    Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. 2021. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research 22, 98 (2021), 1–76

    work page 2021

  2. [2]

    Albrecht, Filippos Christianos, and Lukas Schäfer

    Stefano V. Albrecht, Filippos Christianos, and Lukas Schäfer. 2024. Multi-Agent Reinforcement Learning: Foundations and Modern Approaches . MIT Press. https: //www.marl-book.com

    work page 2024

  3. [3]

    Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. 2018. Safe reinforcement learning via shielding. In Proceedings of the AAAI conference on artificial intelligence , Vol. 32

    work page 2018

  4. [4]

    Osbert Bastani. 2021. Safe reinforcement learning with nonlinear dynamics via model predictive shielding. In 2021 American control conference (ACC). IEEE, 3488–3494

    work page 2021

  5. [5]

    Marc G Bellemare, Will Dabney, and Rémi Munos. 2017. A distributional perspec- tive on reinforcement learning. In International conference on machine learning . PMLR, 449–458

    work page 2017

  6. [6]

    Suda Bharadwaj, Roderik Bloem, Rayna Dimitrova, Bettina Konighofer, and Ufuk Topcu. 2019. Synthesis of minimum-cost shields for multi-agent systems. In 2019 American Control Conference (ACC). IEEE, 1048–1055

    work page 2019

  7. [7]

    Marco C Campi and Simone Garatti. 2011. A sampling-and-discarding approach to chance-constrained optimization: feasibility and optimality. Journal of opti- mization theory and applications 148, 2 (2011), 257–280

    work page 2011

  8. [8]

    Yu-Han Chang, Tracey Ho, and Leslie Kaelbling. 2003. All learning is local: Multi-agent learning in global reward games. Advances in neural information processing systems 16 (2003)

    work page 2003

Show all 49 references
  1. [9]

    Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. 2018. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning . PMLR, 1096–1105

    2018

  2. [10]

    Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. 2018. Distri- butional reinforcement learning with quantile regression. In Proceedings of the AAAI conference on artificial intelligence , Vol. 32

    2018

  3. [11]

    Ingy ElSayed-Aly, Suda Bharadwaj, Christopher Amato, Rüdiger Ehlers, Ufuk Topcu, and Lu Feng. 2021. Safe multi-agent reinforcement learning via shielding. arXiv preprint arXiv:2101.11196 (2021)

    2021 arXiv

  4. [12]

    Matthew Hausknecht and Peter Stone. 2015. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series

    2015

  5. [13]

    Wan Ju Kang David Earl Hostallero, Kyunghwan Son, Daewoo Kim, and Yung Yi Qtran. 2019. Learning to factorize with transformation for cooperative multi- agent reinforcement learning. In Proceedings of the 31st International Conference on Machine Learning, Proceedings of Machine...

    2019

  6. [14]

    Jifeng Hu, Yanchao Sun, Hechang Chen, Sili Huang, Yi Chang, Lichao Sun, et al

  7. [15]

    Yiding Jiang, J Zico Kolter, and Roberta Raileanu. 2024. On the importance of exploration for generalization in reinforcement learning. Advances in Neural Information Processing Systems 36 (2024)

    2024

  8. [16]

    Sham Kakade and John Langford. 2002. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning. 267–274

    2002

  9. [17]

    Quanyi Li, Zhenghao Peng, Zhenghai Xue, Qihang Zhang, and Bolei Zhou. 2021. MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforce- ment Learning. arXiv preprint arXiv:2109.12674 (2021)

    2021 arXiv

  10. [18]

    Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems 30 (2017)

    2017

  11. [19]

    Clare Lyle, Marc G Bellemare, and Pablo Samuel Castro. 2019. A comparative analysis of expected and distributional reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 33. 4504–4511

    2019

  12. [20]

    Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. 2019. Distributional reinforcement learning for efficient exploration. In International conference on machine learning . PMLR, 4424–4434

    2019

  13. [21]

    Jihwan Oh, Joonkee Kim, Minchan Jeong, and Se-Young Yun. 2023. Toward risk- based optimistic exploration for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2303.01768 (2023)

    2023 arXiv

  14. [22]

    Jihwan Oh, Joonkee Kim, and Se-Young Yun. 2022. Risk perspective exploration in distributional reinforcement learning. arXiv preprint arXiv:2206.14170 (2022)

    2022 arXiv

  15. [23]

    Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. 2008. Optimal and approximate Q-value functions for decentralized POMDPs. Journal of Artificial Intelligence Research 32 (2008), 289–353

    2008

  16. [24]

    Zengyi Qin, Kaiqing Zhang, Yuxiao Chen, Jingkai Chen, and Chuchu Fan. 2021. Learning safe multi-agent control with decentralized neural barrier certificates. arXiv preprint arXiv:2101.05436 (2021)

    2021 arXiv

  17. [25]

    Wei Qiu, Xinrun Wang, Runsheng Yu, Xu He, Rundong Wang, Bo An, Svetlana Obraztsova, and Zinovi Rabinovich. 2020. RMIX: Risk-sensitive multi-agent reinforcement learning. (2020)

    2020

  18. [26]

    Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Far- quhar, Jakob Foerster, and Shimon Whiteson. 2020. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research 21, 178 (2020), 1–51

    2020

  19. [27]

    Mark Rowland, Robert Dadashi, Saurabh Kumar, Rémi Munos, Marc G Bellemare, and Will Dabney. 2019. Statistics and samples in distributional reinforcement learning. In International Conference on Machine Learning . PMLR, 5528–5536

    2019

  20. [28]

    Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Far- quhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. 2019. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043 (2019)

    2019 arXiv

  21. [29]

    Siqi Shen, Chennan Ma, Chao Li, Weiquan Liu, Yongquan Fu, Songzhu Mei, Xinwang Liu, and Cheng Wang. 2024. RiskQ: risk-sensitive multi-agent reinforce- ment learning value factorization. Advances in Neural Information Processing Systems 36 (2024)

    2024

  22. [30]

    Siqi Shen, Mengwei Qiu, Jun Liu, Weiquan Liu, Yongquan Fu, Xinwang Liu, and Cheng Wang. 2022. Resq: A residual q function-based approach for multi- agent reinforcement learning value factorization. Advances in Neural Information Processing Systems 35 (2022), 5471–5483

    2022

  23. [31]

    Arambam James Singh and Arvind Easwaran. 2024. PAS: Probably Approximate Safety Verification of Reinforcement Learning Policy Using Scenario Optimiza- tion. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems. 1745–1753

    2024

  24. [32]

    Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. 2019. Qtran: Learning to factorize with transformation for cooperative multi- agent reinforcement learning. In International conference on machine learning . PMLR, 5887–5896

    2019

  25. [33]

    Kyunghwan Son, Junsu Kim, Sungsoo Ahn, Roben D Delos Reyes, Yung Yi, and Jinwoo Shin. 2022. Disentangling sources of risk for distributional multi-agent reinforcement learning. In International Conference on Machine Learning . PMLR, 20347–20368

    2022

  26. [34]

    Kyunghwan Son, Junsu Kim, Yung Yi, and Jinwoo Shin. 2021. Disentangling sources of risk for distributional multi-agent reinforcement learning. (2021)

    2021

  27. [35]

    Wei-Fang Sun, Cheng-Kuang Lee, and Chun-Yi Lee. 2021. DFAC framework: Factorizing the value function via quantile mixture for multi-agent distributional q-learning. In International Conference on Machine Learning . PMLR, 9945–9954

    2021

  28. [36]

    Wei-Fang Sun, Cheng-Kuang Lee, and Chun-Yi Lee. 2021. A Distributional Per- spective on Value Function Factorization Methods for Multi-Agent Reinforcement Learning.. In AAMAS. 1671–1673

    2021

  29. [37]

    Wei-Fang Sun, Cheng-Kuang Lee, Simon See, and Chun-Yi Lee. 2023. A uni- fied framework for factorizing distributional value functions for multi-agent reinforcement learning. Journal of Machine Learning Research 24, 220 (2023), 1–32

    2023

  30. [38]

    Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vini- cius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. 2017. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296 (2017)

    2017 arXiv

  31. [39]

    Jianhao Wang, Zhizhou Ren, Beining Han, Jianing Ye, and Chongjie Zhang

  32. [40]

    Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. 2020. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062 (2020)

    2020 arXiv

  33. [41]

    Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. 2016. Dueling network architectures for deep reinforcement learning. In International conference on machine learning . PMLR, 1995–2003

    2016

  34. [42]

    Tengyu Xu, Yingbin Liang, and Guanghui Lan. 2021. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning . PMLR, 11480–11491

    2021

  35. [43]

    Tengyu Xu, Zhe Wang, and Yingbin Liang. 2020. Improving sample complexity bounds for (natural) actor-critic algorithms. Advances in Neural Information Processing Systems 33 (2020), 4358–4369

    2020

  36. [44]

    Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. 2020. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03939 (2020)

    2020 arXiv

  37. [45]

    Yujie Yang, Yuxuan Jiang, Yichen Liu, Jianyu Chen, and Shengbo Eben Li. 2023. Model-free safe reinforcement learning through neural barrier certificate. IEEE Robotics and Automation Letters 8, 3 (2023), 1295–1302

    2023

  38. [46]

    Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems 33 (2020), 5824–5836

    2020

  39. [47]

    Wenbo Zhang, Osbert Bastani, and Vijay Kumar. 2019. Mamps: Safe multi- agent reinforcement learning via model predictive shielding. arXiv preprint arXiv:1910.12639 (2019). A ALGORITHM In Algorithm 1 below, we outline the steps taken for our policy up- date. Please note that th...

    2019 arXiv

  40. [2021]

    Advances in Neural Information Processing Systems 34 (2021), 29142– 29155

    Towards understanding cooperative multi-agent q-learning with value factorization. Advances in Neural Information Processing Systems 34 (2021), 29142– 29155

    2021

  41. [2022]

    Advances in Neural Information Processing Systems 35 (2022), 12619–12632

    Distributional reward estimation for effective multi-agent deep reinforce- ment learning. Advances in Neural Information Processing Systems 35 (2022), 12619–12632

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.