Pith. sign in

REVIEW 4 major objections 4 minor 34 references

Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read InSPO claims offline multi-agent RL can be made coordinated and OOD-safe by updating agents sequentially on dataset actions with an entropy bonus, monotonically improving and converging to a quantal response equilibrium.

desk verdict A useful offline MARL method with a real proof gap in its central convergence theorem, but worth serious review. read the letter →

arxiv 2412.07639 v2 pith:NLFBT3HI submitted 2024-12-10 cs.AI cs.LG

Zongkai Liu , Qian Lin , Chao Yu , Xiawei Wu , Yile Liang , Donghui Li , Xuetao Ding This is my paper · ORCID
classification cs.AIcs.LG
keywords offlinemulti-agentreinforcementlearningin-samplesequentialpolicyoptimizationquantalresponseequilibriumout-of-distributionjointactionsmaximum-entropyregularizationbehavior-regularizedMarkovgameStarCraftIImicromanagementcooperativeMARL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes InSPO, an offline multi-agent reinforcement learning algorithm that updates agents one at a time using only actions found in the pre-collected dataset. Its central claim is that this in-sample sequential scheme avoids out-of-distribution joint actions, the failure mode where each agent's individually reasonable choice forms a joint action never seen during training, while still coordinating with teammates' updated policies. Adding a policy-entropy bonus lets the method reweight low-probability actions in the behavior policy, which the authors argue prevents premature convergence to sub-optimal equilibria. The authors prove that each update monotonically improves the joint policy and that the algorithm converges to a quantal response equilibrium, and they report state-of-the-art or competitive results on matrix games, Bridge, and StarCraft II micromanagement.

What carries the argument

The central object is the in-sample sequential policy improvement step under the Maximum-Entropy Behavior-Regularized Markov Game (MEBR-MG) objective. For each agent in a random order, the new policy minimizes a KL divergence to the closed-form Boltzmann policy $\pi^i_{\text{new}} \propto \mu^i \cdot \exp\big((Q - \beta \log \mu^i)/(\alpha+\beta)\big)$, using only state-action pairs from the dataset; the per-agent Q-value is evaluated with previously updated agents and later agents still at the old policy. The reverse-KL regularizer makes the joint regularization decompose as a sum over agents, and the entropy term $\beta H(\pi)$ provides the perturbation that lets the method escape local optima. In the practical implementation, local Q-functions are trained with importance resampling and a CQL-style penalty rather than an explicit joint Q-function.

What would settle it

In a tabular two-agent game, delete the globally optimal joint action from the offline dataset (give it zero probability) while retaining all other actions; the paper's Eq. (12) predicts InSPO assigns it zero probability. Observing positive probability on that action would refute the support-preservation claim, while observing zero would confirm that the entropy bonus only reweights actions already present.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that two failure modes of offline MARL—out-of-distribution joint actions and convergence to local optima—can be addressed by the same mechanism: a sequential, in-sample policy update in a maximum-entropy behavior-regularized Markov game. When the regularizer is reverse KL divergence, the global regularization decomposes over agents, so each agent's optimal update has the closed form $\pi^i_{\text{new}} \propto \mu^i \cdot \exp\big((Q - \beta \log \mu^i)/(\alpha+\beta)\big)$, and the resulting objective can be optimized over dataset actions only. Updating agents in sequence lets later agents condition on teammates' updated policies, avoiding conflicting update directions, while the entropy term ensures that actions with low probability under the behavior policy are still explored. The theoretical result is that this alternating evaluation-improvement procedure converges to quantal response equilibrium (Theorem 5), with monotonic improvement at every step (Proposition 4).

Load-bearing premise

The behavior policy must have placed at least some probability on the globally optimal joint action, because InSPO's in-sample updates can only reweight actions already present in the dataset; if the optimal action is entirely absent, the method cannot discover it.

Editorial extensions

If this is right

  • If InSPO's convergence claim is right, offline MARL can be built on a stochastic equilibrium concept (QRE) rather than deterministic Nash equilibria, giving a principled account of bounded-rational cooperative policies.
  • Because the learned policy shares the support of the behavior policy, in-sample sequential updates rule out out-of-distribution joint actions by construction.
  • The method should escape sub-optimal equilibria whenever the globally optimal joint action appears in the dataset with even small probability, as the imbalanced M-NE game shows where only InSPO reaches the (C,C) optimum.
  • Competitive StarCraft II results indicate the sequential scheme scales beyond matrix games and can match value-decomposition methods in near-IGM environments.
  • Monotonic improvement per iteration implies stable training within each evaluation-improvement cycle, at least in the tabular setting where Theorem 5 is proven.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the learned policy's support is contained in the behavior policy's support, InSPO's global-optimum performance is bounded by dataset coverage: on datasets where the optimal joint action never appears, the entropy bonus can only reweight existing actions, not invent new ones.
  • The same reverse-KL decomposition could be combined with support-expanding mechanisms, such as model-based rollouts or diffusion-generated transitions, to relax the coverage assumption; that combination goes beyond the paper.
  • Because the limiting policy is a QRE whose shape depends on the temperatures $\alpha$ and $\beta$, hyperparameter tuning is effectively equilibrium selection, which is consistent with the variation seen in the paper's $\alpha$ ablation.
  • The sequential update scheme's insensitivity to update order in the paper's ablation suggests robustness, but it also leaves adaptive ordering as an open lever for further coordination gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes InSPO, an offline multi-agent reinforcement learning algorithm that sequentially updates policies using in-sample learning under a maximum-entropy behavior-regularized Markov game (MEBR-MG). The authors derive a closed-form policy update via reverse KL regularization and policy entropy, claim monotonic policy improvement and convergence to quantal response equilibrium (QRE), and provide a practical implementation with local Q-functions, importance resampling, and a CQL penalty. Experiments on matrix games, Bridge, and StarCraft II micromanagement compare against BC, OMAR, AlberDICE, CFCQL, and OMIGA, with InSPO achieving competitive or state-of-the-art results on several tasks.

Significance. The paper addresses a relevant problem—out-of-distribution (OOD) joint actions in offline MARL—and proposes an interesting combination of sequential updates and entropy regularization. The MEBR-MG framework and the QRE characterization are novel, and the empirical results, especially on the M-NE game and StarCraft II, are strong. The authors also release code, which supports reproducibility. However, the theoretical contributions are not fully substantiated: the monotonic improvement proof contains a technical gap, the QRE convergence proof is too abbreviated, and the practical algorithm is not covered by the theoretical guarantees. If these issues are resolved, the paper would be a valuable contribution to offline MARL.

major comments (4)
  1. [Appendix A.3 (proof of Proposition 4)] The summation of inequality (19) is invalid. For the n-th term, the expectation is over a^{i1:n-1} ∼ π_new and a^{i_n} ∼ π̄^{i_n}, not over the joint policy π̄. The right-hand side of the summed inequality is therefore not E_{a∼π̄}[A_{π_old}(s,a) − α log (π̄/μ) − β log π̄]. Consequently, the contradiction with L_{π_old}(π̄) > L_{π_old}(π_new) is not established. A corrected proof of monotonic improvement is needed, since this result underpins Theorem 5.
  2. [Appendix A.4 (Theorem 10)] The proof asserts that monotone convergence of Q^{π_k} and boundedness imply that the policies converge to a limit point, and that this limit point satisfies the QRE condition. Monotone Q-function convergence does not by itself imply policy convergence (the softmax policy may cycle), and the argument that the limit policy satisfies the inequality for all i and π^i is not derived. The theorem should either be proved with a complete fixed-point argument or be restricted to the exact tabular update with an explicit statement of all assumptions.
  3. [The Practical Implementation of InSPO (Eqs. 14–15, 20, 23–24)] The evaluated algorithm is not the same as the one for which the theory is proven. The practical version replaces the global Q-function with per-agent local Q-functions trained on an importance-resampled dataset, uses a geometric-mean importance ratio in Eq. (23), adds a CQL penalty in Eq. (24) that changes the Bellman fixed point, and uses a local advantage in Eq. (20) instead of the sequential advantage in Eq. (13). No theorem shows that these approximations preserve monotonic improvement or QRE convergence. The abstract and conclusion claim these guarantees for InSPO without qualification, so the central theoretical claim is not established for the algorithm evaluated in Tables 1–4.
  4. [Appendix A.2 (Proposition 3)] The KKT derivation treats Qπ as independent of the optimizing agent's own policy π^i. As written, the derivation gives a best-response formula for a fixed Q function, and only by requiring self-consistency does it characterize a QRE. The proof should explicitly state this fixed-point structure; otherwise the derivation appears circular. The resulting formula is a standard logit QRE, so this is a correctness-risk issue rather than a fatal flaw, but it needs clarification.
minor comments (4)
  1. [References] The reference list contains two entries labeled Xu et al. 2023a and 2023b with identical titles and venues; the in-text citation to Xu et al. 2023b for StarCraft II datasets does not match the cited paper. Please correct the citation and reference list.
  2. [Equation (7)] The notation A^{i1:n}_{π_old}(s,a^{i_n}) is used before it is clearly defined; the definition is placed inside Eq. (7) after the minimizer, and the dependency on a^{i1:n-1} is not made explicit. Please make the notation consistent and unambiguous.
  3. [The Proposed Method] The statement that 'the optimal solution of the perturbed game aligns with the solution concept of quantal response equilibrium' is imprecise because QRE is traditionally defined for normal-form games; the Markov-game extension in Definition 1 should be more carefully motivated.
  4. [Limitations] The entropy-based exploration only re-weights actions that already appear in the offline dataset; Proposition 3 implies the QRE policy has the same support as the behavior policy. The paper should state this data-coverage assumption explicitly, e.g., that the globally optimal joint action must have positive probability under the behavior policy for the M-NE game result to hold.

Circularity Check

0 steps flagged · score 0.0 of 10

No meaningful circularity: InSPO's monotonic-improvement and QRE-convergence claims are self-contained proofs about a clearly defined entropy-regularized game, not fitted quantities or load-bearing self-citations.

full rationale

The central theoretical claims are self-contained. Algorithm 1 is a tabular alternation between policy evaluation under the MEBR-MG Bellman operator (Eq. 10) and the in-sample sequential improvement step (Eq. 13). Proposition 3 (Appendix A.2) derives the softmax form of a QRE from the KKT conditions of the entropy- and KL-regularized objective; it does not assume that form as an input. Proposition 4 (Appendix A.3) is a monotone policy-improvement argument using the multi-agent advantage decomposition of Kuba et al., which is an external and independently derivable lemma rather than a self-citation. Theorem 5 (Appendix A.4) uses monotonicity plus boundedness to obtain a limit point and verifies the QRE inequality (8) at that limit point. This is a genuine fixed-point proof rather than a tautology: the algorithm is not defined as 'the thing that converges to QRE'; the QRE condition is derived from the game definition and checked at the limit. No fitted parameter is relabeled as a prediction, and no load-bearing premise is imported from the authors' own prior work. The paper's QRE is defined for its own maximum-entropy behavior-regularized game, which is a modeling choice; the convergence theorem is about that modified game rather than the original unregularized game, but this is a scope limitation, not circular reasoning. Implementation gaps — local Q-functions, importance resampling with a geometric-mean ratio, and the CQL penalty in Eqs. 14-24 — mean that Theorem 5 is proven only for the tabular Algorithm 1 rather than for the full deep implementation; this is a correctness or approximation issue orthogonal to circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

All theoretical results are in a tabular, exact setting and rely on an accurate behavior-policy estimate and on dataset support containing useful actions. The empirical method adds per-dataset tuning of α, β, and Dbar and a CQL penalty. No new physical entities are introduced.

free parameters (4)
  • α (KL regularization temperature) = tuned per task from {0.1, 0.5, 1, 3, 5}, or auto-tuned with target Dbar
    Controls conservatism in Eqs. (5), (9), (13); if too large the policy collapses toward the behavior policy, if too small distribution shift occurs; tuned for each benchmark.
  • β (entropy coefficient) = tuned from {0, 5, 10} with exponential decay
    Entropy bonus is the mechanism credited for escaping local optima (Fig. 5a); without it InSPO fails on M-NE imbalanced; value is dataset-dependent.
  • Dbar (target KL divergence) = 0.18 default, tuned from {0.08, 0.18, 0.3}
    Target value in auto-tuned α loss (Eq. 25); chosen by validation performance.
  • α_CQL = 0.1 default
    CQL penalty weight in practical Q-learning objective (Eq. 24); present in implementation but absent from the theory.
assumptions (5)
  • domain assumption The behavior policy μ is accurately estimated by behavior cloning from the offline dataset.
    Algorithm 1 line 1, Eq. (21)-(22); all KL regularizers and importance ratios ρ depend on μ. If μ is wrong, support and density ratios used by InSPO are incorrect.
  • domain assumption The offline dataset contains every action that could be part of a good solution with positive probability.
    Eq. (12) forces the learned policy to share the support of μ; M-NE imbalanced dataset places 1% mass on (C,C), which is what lets entropy exploration find it. If a good joint action has zero support, the method cannot discover it.
  • domain assumption The convergence guarantees hold only in a tabular setting with exact Q-evaluation and exact policy updates.
    Appendix A.4 starts 'In the tabular setting'; Proposition 8 and Theorem 10 assume exact Eq. (10) and Eq. (13), while the deployed algorithm uses neural approximations and importance resampling.
  • standard math Multi-agent advantage decomposition (Lemma 9) from Kuba et al. is valid.
    Used in the proof of Proposition 8 to convert per-agent update inequalities into a joint-policy inequality; sourced from prior work.
  • ad hoc to paper In the QRE derivation, Qπ is differentiated as if it were independent of the optimizing agent's own policy πi.
    Appendix A.2 Lagrangian treats E_{a-i}[Qπ(s,ai,a-i)] as fixed when taking ∂L/∂πi(ai|s), although Qπ is defined through joint policy π. This is a fixed-point characterization, not a complete best-response derivation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization." pith.science (2026). https://pith.science/paper/NLFBT3HI

@misc{pith2026241207639,
  author       = {Pith},
  title        = {Pith review of: Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NLFBT3HI}},
  note         = {Machine review of arXiv:2412.07639}
}
read the original abstract

Offline Multi-Agent Reinforcement Learning (MARL) is an emerging field that aims to learn optimal multi-agent policies from pre-collected datasets. Compared to single-agent case, multi-agent setting involves a large joint state-action space and coupled behaviors of multiple agents, which bring extra complexity to offline policy optimization. In this work, we revisit the existing offline MARL methods and show that in certain scenarios they can be problematic, leading to uncoordinated behaviors and out-of-distribution (OOD) joint actions. To address these issues, we propose a new offline MARL algorithm, named In-Sample Sequential Policy Optimization (InSPO). InSPO sequentially updates each agent's policy in an in-sample manner, which not only avoids selecting OOD joint actions but also carefully considers teammates' updated policies to enhance coordination. Additionally, by thoroughly exploring low-probability actions in the behavior policy, InSPO can well address the issue of premature convergence to sub-optimal solutions. Theoretically, we prove InSPO guarantees monotonic policy improvement and converges to quantal response equilibrium (QRE). Experimental results demonstrate the effectiveness of our method compared to current state-of-the-art offline MARL methods.

Figures

Figures reproduced from arXiv: 2412.07639 by the authors.

Figure 1
Figure 1. XOR game. (a) is the reward matrix of joint ac [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. M-NE game. (a) is the reward matrix of joint ac [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Final joint policy on XOR game for dataset (b). [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Ablation on entropy and sequential update scheme. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 27 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Barde, P.; Foerster, J.; Nowrouzezahrai, D.; and Zhang, A. 2024. A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem. In International Conference on Autonomous Agents and Multiagent Systems. International Foundation for Autonomous Agents and Multiagent Systems / ACM

    work page 2024

  4. [4]

    Ding, Z.; Su, K.; Hong, W.; Zhu, L.; Huang, T.; and Lu, Z. 2022. Multi-Agent Sequential Decision-Making via Communication. arXiv:2209.12713

    work page Pith review arXiv 2022

  5. [5]

    Figueiredo Prudencio, R.; Maximo, M. R. O. A.; and Colombini, E. L. 2024. A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems. IEEE Transactions on Neural Networks and Learning Systems, 35(8): 10237–10257

    work page 2024

  6. [6]

    Fu, W.; Yu, C.; Xu, Z.; Yang, J.; and Wu, Y. 2022. Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning. In International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, 6863--6877. PMLR

    work page 2022

  7. [7]

    Haarnoja, T.; Zhou, A.; Abbeel, P.; and Levine, S. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, 1861--1870. PMLR

    2018

  8. [8]

    Haarnoja, T.; Zhou, A.; Hartikainen, K.; Tucker, G.; Ha, S.; Tan, J.; Kumar, V.; Zhu, H.; Gupta, A.; Abbeel, P.; and Levine, S. 2019. Soft Actor-Critic Algorithms and Applications. arXiv:1812.05905

    arXiv 2019

Show all 34 references
  1. [9]

    Kostrikov, I.; Nair, A.; and Levine, S. 2022. Offline Reinforcement Learning with Implicit Q-Learning. In International Conference on Learning Representations. OpenReview.net

    2022

  2. [10]

    G.; Chen, R.; Wen, M.; Wen, Y.; Sun, F.; Wang, J.; and Yang, Y

    Kuba, J. G.; Chen, R.; Wen, M.; Wen, Y.; Sun, F.; Wang, J.; and Yang, Y. 2022. Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning. In The Tenth International Conference on Learning Representations. OpenReview.net

    2022

  3. [11]

    Kumar, A.; Zhou, A.; Tucker, G.; and Levine, S. 2020. Conservative Q-Learning for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems

    2020

  4. [12]

    Li, Z.; Pan, L.; and Huang, L. 2023. Beyond Conservatism: Diffusion Policies in Offline Multi-agent Reinforcement Learning. arXiv:2307.01472

    2023 arXiv

  5. [13]

    Li, Z.; Zhao, W.; Wu, L.; and Pajarinen, J. 2024. Backpropagation Through Agents. In Annual AAAI Conference on Artificial Intelligence. AAAI Press

    2024

  6. [14]

    Littman, M. L. 1994. Markov Games as a Framework for Multi-Agent Reinforcement Learning. In Machine Learning, Proceedings of the Eleventh International Conference, 157--163. Morgan Kaufmann

    1994

  7. [15]

    Liu, J.; Zhong, Y.; Hu, S.; Fu, H.; Fu, Q.; Chang, X.; and Yang, Y. 2024. Maximum Entropy Heterogeneous-Agent Reinforcement Learning. In International Conference on Learning Representations. OpenReview.net

    2024

  8. [16]

    E.; Lee, J.; Yoon, J.; Leonardos, S.; Abbeel, P.; and Kim, K

    Matsunaga, D. E.; Lee, J.; Yoon, J.; Leonardos, S.; Abbeel, P.; and Kim, K. 2023. AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation. In Advances in Neural Information Processing Systems

    2023

  9. [17]

    D.; and Palfrey, T

    McKelvey, R. D.; and Palfrey, T. R. 1995. Quantal response equilibria for normal form games. Games and economic behavior, 10(1): 6--38

    1995

  10. [18]

    Pan, L.; Huang, L.; Ma, T.; and Xu, H. 2022. Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification. In International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, 17221--17237. PMLR

    2022

  11. [19]

    S.; Farquhar, G.; Foerster, J

    Rashid, T.; Samvelyan, M.; de Witt, C. S.; Farquhar, G.; Foerster, J. N.; and Whiteson, S. 2018. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. In International Conference on Machine Learning, volume 80 of Proceedings of Machine Learn...

    2018

  12. [20]

    Schlegel, M.; Chung, W.; Graves, D.; Qian, J.; and White, M. 2019. Importance Resampling for Off-policy Prediction. In Advances in Neural Information Processing Systems, 1797--1807

    2019

  13. [21]

    Shao, J.; Qu, Y.; Chen, C.; Zhang, H.; and Ji, X. 2023. Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning. In Advances in Neural Information Processing Systems

    2023

  14. [22]

    J.; Hostallero, D.; and Yi, Y

    Son, K.; Kim, D.; Kang, W. J.; Hostallero, D.; and Yi, Y. 2019. QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning. In International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, 5887--5896. PMLR

    2019

  15. [23]

    Sutton, R. S. 2018. Reinforcement Learning: An Introduction

    2018

  16. [24]

    Wang, J.; Ren, Z.; Liu, T.; Yu, Y.; and Zhang, C. 2021. QPLEX: Duplex Dueling Multi-Agent Q-Learning. In International Conference on Learning Representations. OpenReview.net

    2021

  17. [25]

    Wang, J.; Ye, D.; and Lu, Z. 2023. More Centralized Training, Still Decentralized Execution: Multi-Agent Conditional Policy Factorization. In International Conference on Learning Representations. OpenReview.net

    2023

  18. [26]

    Wang, X.; Tian, Z.; Wan, Z.; Wen, Y.; Wang, J.; and Zhang, W. 2023 a . Order Matters: Agent-by-agent Policy Optimization. In International Conference on Learning Representations. OpenReview.net

    2023

  19. [27]

    Wang, X.; Xu, H.; Zheng, Y.; and Zhan, X. 2023 b . Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization. In Advances in Neural Information Processing Systems

    2023

  20. [28]

    Wu, Z.; Yu, C.; Ye, D.; Zhang, J.; Piao, H.; and Zhuo, H. H. 2021. Coordinated Proximal Policy Optimization. In Advances in Neural Information Processing Systems, 26437--26448

    2021

  21. [29]

    Xu, H.; Jiang, L.; Li, J.; Yang, Z.; Wang, Z.; Chan, W. K. V.; and Zhan, X. 2023 a . Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization. In International Conference on Learning Representations. OpenReview.net

    2023

  22. [30]

    Xu, H.; Jiang, L.; Li, J.; Yang, Z.; Wang, Z.; Chan, W. K. V.; and Zhan, X. 2023 b . Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization. In International Conference on Learning Representations. OpenReview.net

    2023

  23. [31]

    Yang, Y.; Ma, X.; Li, C.; Zheng, Z.; Zhang, Q.; Huang, G.; Yang, J.; and Zhao, Q. 2021. Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning. In Advances in Neural Information Processing Systems

    2021

  24. [32]

    M.; and Wu, Y

    Yu, C.; Velu, A.; Vinitsky, E.; Gao, J.; Wang, Y.; Bayen, A. M.; and Wu, Y. 2022. The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games. In Advances in Neural Information Processing Systems

    2022

  25. [33]

    Zhang, K.; Yang, Z.; and Başar, T. 2021. Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms. arXiv:1911.10635

    2021 arXiv

  26. [34]

    Zhu, Z.; Liu, M.; Mao, L.; Kang, B.; Xu, M.; Yu, Y.; Ermon, S.; and Zhang, W. 2023. Madiff: Offline multi-agent learning with diffusion models. arXiv:2305.17330

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.