Pith. sign in

REVIEW 3 major objections 7 minor 43 references

Fixing Incomplete Value Function Decomposition for Multi-Agent Reinforcement Learning

T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A weighted-advantage fixing layer is sufficient and necessary for the full class of IGM value decompositions, and the resulting QFIX family matches or outperforms QPLEX with smaller mixers.

desk verdict Solid theory, honest experiments, but the IGM-completeness claim is not actually what the Q+FIX runs test—worth a serious referee once the state-only caveat is moved from the appendix into the main text. read the letter →

arxiv 2505.10484 v1 pith:7RV2VVYH submitted 2025-05-15 cs.LG

Andrea Baisero , Rupali Bhati , Shuo Liu , Aathira Pillai , Christopher Amato This is my paper · ORCID
classification cs.LG
keywords multi-agentreinforcementlearningvaluefunctiondecompositionindividual-globalmaxIGM-completeQFIXcooperativeMARLdecentralizedPOMDPcentralizedtrainingexecution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Cooperative multi-agent reinforcement learning methods that decompose a joint value into per-agent utilities need the individual-global max (IGM) property: the actions that maximize each agent's utility must match the actions that maximize the joint value. The paper argues that the whole class of values satisfying IGM has a simple characterization: take any base decomposition that already satisfies IGM (such as VDN or QMIX), compute its joint advantage, multiply that advantage by a positive weight and add a history-dependent offset. This single "fixing" layer, called QFIX, is both sufficient and necessary for IGM, and its additive reparameterization Q+FIX gives a family of mixers that are simpler and smaller than QPLEX while matching or beating it on SMACv2 and Overcooked. If correct, teams can upgrade existing incomplete mixers to full representational power with a thin wrapper instead of redesigning their value-decomposition architecture.

What carries the argument

The central object is the fixing layer, a thin network implementing $\hat Q_{FIX}(h,a)=w(h,a)\hat A_{fixee}(h,a)+b(h)$. It is built on a simplified necessary-and-sufficient form of the IGM constraints (Proposition 2): the joint advantage is zero exactly when all individual advantages are zero, and negative otherwise. The layer reads the non-positive joint advantage of any IGM-satisfying "fixee" model (the additive sum of VDN or the monotonic mixer of QMIX), multiplies it by a positive weight $w>0$ that may depend on joint history and action, and adds a history bias $b$; this turns an IGM-satisfying but incomplete base model into an IGM-complete one. The additive Q+FIX reparameterization, with $w>-1$, keeps the fixee term in the sum and enables the practical tricks of advantage detaching and intervention annealing.

What would settle it

Train the same agents with state-only QFIX and history-state QFIX on a Dec-POMDP where the optimal joint value depends on unobservable histories so that the state-only form provably cannot represent the target; if state-only QFIX matches history-state QFIX everywhere, IGM-completeness is not the operative factor in the reported gains, while a clear gap would show that the full theory matters.

Watch

Extended reading notes

Core claim

The paper's central claim is that the full class of (measurable) IGM values can be parameterized as $$\hat Q_{FIX}(h,a)=w(h,a)\hat A_{fixee}(h,a)+b(h),$$ where $\hat A_{fixee}$ is the non-positive joint advantage of any IGM-satisfying base model, $w>0$ is an arbitrary positive function, and $b$ is an arbitrary function of joint history. Proposition 4 states that this model satisfies IGM and that its function class is exactly the class of measurable IGM values; because VDN and QMIX are valid fixees, QFIX-sum and QFIX-mono immediately cover the full class, and QFIX-lin does so with per-agent weights. The additive reparameterization $\hat Q_{+FIX}=\hat Q_{fixee}+w\hat A_{fixee}+b$ preserves these properties and, with detached advantages and annealed intervention, is what the paper evaluates. The paper further claims this formulation exposes QPLEX's core mechanism while discarding its extra transformations, and that Q+FIX variants match or beat QPLEX on SMACv2 and Overcooked with smaller mixers and more stable convergence.

Load-bearing premise

The empirical evaluation uses QFIX variants whose fixing network sees only the current state and joint action, and the paper itself proves those state-only variants are not IGM-complete, so the performance claims rest on the assumption that this restriction keeps the practical benefits of the complete theory intact.

Editorial extensions

If this is right

  • VDN and QMIX, the two most common value-decomposition baselines, become IGM-complete by adding a small fixing network rather than switching to a new architecture.
  • QFIX-sum and QFIX-lin use substantially smaller mixers than QPLEX while matching or exceeding its performance on SMACv2.
  • The theory covers partially observable decentralized control, so QFIX does not inherit the restriction of prior methods that only apply to fully observable settings.
  • Because QFIX can recover its own fixee by setting $w=1$ and $b=\hat V_{fixee}$, well-tuned base models can be preserved and extended rather than replaced.
  • The model-size ablation shows the performance gains come from the fixing structure rather than from extra parameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the fixing-layer template should transfer to any future IGM-satisfying mixer; the only requirement is that the base model's joint advantage vanish exactly on the individually maximal joint actions, so newer, richer base models could be promoted to completeness with the same thin wrapper.
  • Editorial inference: because Appendix E.2 proves state-only fixing networks are not IGM-complete, the SMACv2 and Overcooked results do not directly test the completeness theorem; a comparison of state-only against history-state QFIX on a task whose optimal joint value depends on private histories would isolate whether the theoretical class is what drives the reported gains.
  • Editorial inference: the stop-gradient and intervention-annealing techniques are heuristic; varying the fixee's strength and the annealing schedule could test whether the fixing layer's gradient interference is the mechanism behind QFIX's stability advantage.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper studies value function decomposition in cooperative multi-agent reinforcement learning. It proposes a new characterization of the IGM property: joint advantages are negative exactly when at least one individual advantage is negative (Proposition 2). This leads to a simple family QIGM(h,a) = w(h,a) f(u_1,...,u_N) + b(h), which the paper proves is IGM-complete for arbitrary positive w, b, and a suitable f (Proposition 3). Building on this, the authors introduce QFIX, which applies a fixing network to the joint advantage of an existing IGM-satisfying 'fixee' model such as VDN or QMIX, and a reparameterized additive variant Q+FIX, with three instances (sum, mono, lin). They also discuss stateful variants and their theoretical consequences. The empirical section evaluates Q+FIX variants on SMACv2 and Overcooked, reporting improved or comparable performance relative to VDN, QMIX, and QPLEX, with smaller mixing models and a model-size ablation. The paper concludes that QFIX expands the representation capabilities of prior methods while being simpler than QPLEX.

Significance. If the theoretical result holds, the paper makes a valuable contribution: it identifies a single weighted transformation of a fixee's joint advantage as the core mechanism underlying IGM-complete factorization, thereby simplifying QPLEX's architecture. The completeness construction in Proposition 3 is elegant, self-contained, and non-circular: the proof constructs w and b from an arbitrary IGM target function rather than fitting them to data. The paper also engages seriously with measurability and universal-approximation issues in Appendix A, and it releases code, reports bootstrap confidence intervals, and includes a model-size ablation. However, the empirical evaluation only instantiates state-only Q+FIX variants, which the paper itself proves are not IGM-complete; therefore the experiments do not directly test the central theoretical claim that IGM-completeness is what makes QFIX effective. In addition, the formal statements of Propositions 4 and 5 claim equality with the class of measurable IGM values, while the proofs establish only density (approximation) via universal approximation.

major comments (3)
  1. [Section 5, Appendix F.1, Appendix E.2] The empirical evaluation in Section 5 and Appendix F.1 instantiates Q+FIX with state-only fixing models w(s,a) and b(s) (the form of Eq. (49) in Appendix E.2), a configuration that the paper itself proves satisfies stateful-IGM but is not IGM-complete. Consequently, the performance gains over VDN/QMIX and the comparison with QPLEX in Figures 2 and 3 do not provide evidence for the paper's central theoretical claim that IGM-completeness is what makes QFIX effective. The model-size ablation controls for parameter count, but not for the gap between history-state QFIX (which is IGM-complete) and the evaluated state-only QFIX. Please either add experiments with history-state QFIX (w(h,s,a), b(h,s)) on at least a subset of tasks, or substantially reframe the abstract and introduction to separate the theoretical completeness result from the empirical evaluation of the state-only variants.
  2. [Proposition 4 / Appendix B.2] Proposition 4 states that 'the function class of QFIX is that of (measurable) IGM values,' but the proof in Appendix B.2 only shows that the constructed w and b can be approximated by neural networks via universal approximation; that is, the function class is dense in, not equal to, the class of measurable IGM values. This is exactly the standard the paper applies to QPLEX in Appendix A, where it concludes that QPLEX realizes 'technically ... measurable IGM values' only in an approximation sense. The same precision should be used for QFIX; otherwise the claim is stronger than what is proven. Please state explicitly that Propositions 4 and 5 are density results and specify the approximation topology (e.g., convergence in measure or in Lp).
  3. [Appendix B.3] The proof of Proposition 5 is too terse. It asserts that QFIX-lin 'is a monotonic function of individual advantages and therefore satisfies IGM' without showing the max-set identity: for a joint action composed of componentwise maximizing actions the weighted advantage sum is zero, while any other joint action has at least one strictly negative individual advantage and hence a negative sum. The completeness claim is also stated without explicitly proving that every QFIX-sum function is a QFIX-lin function with equal weights. Given that QFIX-lin is a central variant in the evaluation, the proof should be expanded to a full argument.
minor comments (7)
  1. [Section 5 / Figure 3] Figure 3 includes QPLEX in the Overcooked results, but Section 5 states that JaxMARL provides baselines for IQL, VDN, and QMIX 'but not QPLEX.' Please clarify whether QPLEX was implemented and run for Overcooked, and if so, describe the implementation and hyperparameters used.
  2. [Section 6 / Limitations] The Limitations section is very brief and does not mention that the evaluated state-only Q+FIX variants are not IGM-complete. Please add a sentence acknowledging this gap and its implications for interpreting the empirical results.
  3. [Appendix C.6] The first equality in the derivation of Q+FIX-mono incorrectly writes Q_VDN and A_VDN; the fixee should be Q_MIX and A_MIX. Please correct this typo.
  4. [Appendix G.1] The description of the weight model's output constraint, 'lambda w: |w+1|-1+10e-8', is confusing. Please explain how this parameterizes w > -1 (e.g., via an absolute-value or softplus reparameterization) and why the epsilon term is needed.
  5. [Appendix B.2] The proof of Proposition 4 should explicitly invoke the measurability and approximation assumptions needed for Hornik's theorems, rather than referring to Appendix A only implicitly, so that the statement of the proposition is self-contained.
  6. [Table 1] The header 'Protoss Terran, Zerg' is ambiguous; it should be formatted as 'Protoss / Terran / Zerg' or similar.
  7. [Main text / Section 5] The paper says it implements 'three variants' of QFIX, but the experiments evaluate only the additive Q+FIX variants. Please say 'three variants of additive QFIX' to avoid ambiguity.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: QFIX's IGM-completeness is proven by explicit construction; the state-only experimental variants are an acknowledged scope gap, not a circular step.

full rationale

The paper's central theoretical claim is self-contained. Proposition 3 proves that QIGM(h,a)=w(h,a)f(u_1,...,u_N)+b(h) is IGM-complete by an explicit existence argument: for any target IGM value Q, the proof sets b(h)=V(h) and w(h,a)=A(h,a)/f(u_1,...,u_N), then verifies equality QIGM=Q pointwise. This is a construction from the target function, not a fit to data, and no predicted quantity is defined in terms of a fitted constant. Proposition 4 then applies the same argument to QFIX by observing that A_fixee(h,a) has exactly the required properties of f when the fixee satisfies IGM. The completeness step is an instance of the already-proved representation, and the only external ingredient is the standard Hornik universal-approximation theorem, used to turn measurability of the constructed w and b into approximability. The paper's use of the same-group reference [9] (Marchesini et al.) is limited to the stateful-IGM definition and to contextual statements about state-only QPLEX; the paper proves its own stateful QFIX claims directly in Appendices E.1 and E.2, so the self-citation is not load-bearing. The most notable gap is not circularity: all SMACv2 experiments use state-only Q+FIX with weights w(s,a) and bias b(s) (Appendix F.1), and Appendix E.2 explicitly states that this state-only variant is not IGM-complete. This means the empirical evaluation does not directly test the IGM-completeness benefit, and the performance gains may come from other components (state conditioning, advantage detaching, added capacity). But that is a scope/evidence limitation, not a reduction of a prediction to its own inputs. No derivation step in the paper merely renames an input or imports a uniqueness claim from the authors' prior work. Accordingly, the paper receives a low score reflecting only the minor non-load-bearing self-citation, with no circular step identified.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central theory is parameter-free: w and b are arbitrary functions in the representation theorem, so no constants are fitted to make the derivation work. The listed free parameters are implementation choices for the empirical study. The axioms are the standard MARL/IGM framework, the UAT density step, measurability, and the extra premise that state-only QFIX retains the practical advantages despite losing completeness.

free parameters (4)
  • w(s,a) and b(s) hidden layer size = 64 units for SMACv2; conv features for Overcooked
    Hand-chosen capacity for the fixing networks; affects expressiveness and training stability (Appendix G).
  • Intervention annealing schedule = lambda from 1.0 to 0 over first 500k timesteps in Overcooked; not used in SMACv2
    Hand-chosen to stabilize early learning (Section 4.3, Appendix F.2).
  • Advantage detaching = on for all Q+FIX variants
    Binary design choice adopted from QPLEX; improves optimization stability but is not theoretically required (Section 4.3, Appendix D).
  • w positivity epsilon = 10e-8
    Numerical safeguard in the constraint lambda w: |w+1|-1+10e-8 (Appendix G).
assumptions (6)
  • standard math The Dec-POMDP formulation and the IGM definition (Definition 1) as the consistency criterion between individual and joint action selection.
    Background framework from the cited MARL literature (Sections 3.1 and 3.2); not proved in this paper.
  • domain assumption Fixee models VDN and QMIX satisfy IGM, so their joint advantages are zero exactly when all individual advantages are zero.
    Used in Section 4.2 to ensure A_fixee plays the role of f in Eq. (8). VDN follows from additivity; QMIX requires monotonicity, and in practice strict monotonicity of the mixing network.
  • standard math MLPs with a single hidden layer can approximate measurable functions in measure (Hornik's universal approximation theorem).
    Invoked in the proofs of Propositions 4 and 5 (Appendix B.2 and B.3) to claim the function class of QFIX is the measurable IGM class; also discussed in Appendix A.
  • domain assumption Target IGM values are measurable.
    Explicitly assumed so the UAT construction is valid (Appendix A, Appendix B.2). Excludes pathological non-measurable values.
  • domain assumption Stateful-IGM (Definition 3) is the right extension of IGM when joint values depend on centralized state.
    Taken from Marchesini et al. [9], cited in Section 3.2 and used in Section 4.4 and Appendix E.
  • ad hoc to paper The state-only QFIX implementation w(s,a) and b(s) retains the practical benefits of the IGM-complete framework even though it is not IGM-complete.
    Section 4.4 and Appendix E.2 prove state-only QFIX is not complete, yet the main empirical results use this variant (Appendix F.1). The premise that this is the right comparison is load-bearing for the empirical claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fixing Incomplete Value Function Decomposition for Multi-Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/7RV2VVYH

@misc{pith2026250510484,
  author       = {Pith},
  title        = {Pith review of: Fixing Incomplete Value Function Decomposition for Multi-Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7RV2VVYH}},
  note         = {Machine review of arXiv:2505.10484}
}
read the original abstract

Value function decomposition methods for cooperative multi-agent reinforcement learning compose joint values from individual per-agent utilities, and train them using a joint objective. To ensure that the action selection process between individual utilities and joint values remains consistent, it is imperative for the composition to satisfy the individual-global max (IGM) property. Although satisfying IGM itself is straightforward, most existing methods (e.g., VDN, QMIX) have limited representation capabilities and are unable to represent the full class of IGM values, and the one exception that has no such limitation (QPLEX) is unnecessarily complex. In this work, we present a simple formulation of the full class of IGM values that naturally leads to the derivation of QFIX, a novel family of value function decomposition models that expand the representation capabilities of prior models by means of a thin "fixing" layer. We derive multiple variants of QFIX, and implement three variants in two well-known multi-agent frameworks. We perform an empirical evaluation on multiple SMACv2 and Overcooked environments, which confirms that QFIX (i) succeeds in enhancing the performance of prior methods, (ii) learns more stably and performs better than its main competitor QPLEX, and (iii) achieves this while employing the simplest and smallest mixing models.

Figures

Figures reproduced from arXiv: 2505.10484 by the authors.

Figure 1
Figure 1. Diagrams for QFIX (left) and Q+FIX (right). [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. SMACv2 results, bootstrapped 95% CI. Aggregate returns are normalized per-task via G˜ i .= Gi−mink Gk maxk Gk−mink Gk , where {Gi}i is the total set of returns logged by all models in a given task. Annealing the intervention Another implementation detail we found to be occasionally useful to stabilize learning has been to introduce the fixing intervention smoothly during the early stages of training (≈ 5% of total t… view at source ↗
Figure 3
Figure 3. Overcooked return mean, bootstrapped 95% CI (20 seeds). based on 9 scenarios obtained by combining the 3 races (Protoss, Terran, and Zerg) with 3 team sizes (5vs5, 10vs10, and 20vs20). We use shorthand labels, e.g., P5, T10, Z20. Pymarl2 provides base implementations for VDN, QMIX, and QPLEX, and we implemented Q+FIX-{sum,mono,lin}. Fig. 2a contains the evaluation results based on mean performance, with 5 independen… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Specialized diagrams for Q+FIX-sum, Q+FIX-mono, and Q+FIX-lin. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: SMACv2 winrate results, bootstrapped 95% CI. F Evaluation details and additional results F.1 SMACv2 Implementation details We note that Pymarl2 provides stateful implementations of QMIX and QPLEX. For QPLEX in particular, this means that state-only weights wi(s) and λi…
Figure 6
Figure 6. Figure 6: SMACv2 ablation results, bootstrapped 95%. Aggregation computed as in [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: Aggregate probability of improvement (POI), bootstrapped 95% CI. [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: Overcooked return mean, bootstrapped 95% CI (20 seeds). F.2 Overcooked Observability Overcooked is a fully observable environment, with each agent receiving observa￾tions whose information content is equivalent to the state. Therefore, the challenge of these tasks is p…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 33 canonical work pages

  1. [1]

    Deep Reinforcement Learning at the Edge of the Statistical Precipice

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Belle- mare. Deep Reinforcement Learning at the Edge of the Statistical Precipice. In Advances in Neural Information Processing Systems, volume 34, pages 29304–29320. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/hash/ f514cec81cb148...

    work page 2021

  2. [2]

    Ho, Thomas L

    Micah Carroll, Rohin Shah, Mark K. Ho, Thomas L. Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anca Dragan. On the Utility of Learning about Humans for Human-AI Coordination, January 2020. URL http://arxiv.org/abs/1910.05789. arXiv:1910.05789 [cs]

    arXiv 2020

  3. [3]

    G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2(4):303–314, December 1989. ISSN 1435-568X. doi: 10.1007/ BF02551274. URL https://doi.org/10.1007/BF02551274

    doi:10.1007/bf02551274 1989

  4. [4]

    Foerster, and Shimon Whiteson

    Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N. Foerster, and Shimon Whiteson. SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning, October 2023. URL http://arxiv.org/ abs/2212.07489. arXiv:2212.07489

    arXiv 2023

  5. [5]

    Approximation capabilities of multilayer feedforward networks

    Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):251–257, January 1991. ISSN 0893-6080. doi: 10.1016/0893-6080(91)90009-T. URL https://www.sciencedirect.com/science/article/pii/089360809190009T

    arXiv 1991

  6. [6]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January

  7. [7]

    On the Variance of the Adaptive Learning Rate and Beyond

    Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the Variance of the Adaptive Learning Rate and Beyond. In International Conference on Learning Representations, September 2019. URL https://openreview.net/ forum?id=rkgz2aEKDr

    work page 2019

  8. [8]

    Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

    Ryan Lowe, YI WU, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mor- datch. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. In Advances in Neural Information Processing Systems , volume 30. Curran Asso- ciates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ 68a9750337a418a86fe06c1991a1d64c-Abstract.html

    work page 2017

Show all 43 references
  1. [9]

    On Stateful Value Factorization in Multi-Agent Reinforcement Learning, September 2024

    Enrico Marchesini, Andrea Baisero, Rupali Bhati, and Christopher Amato. On Stateful Value Factorization in Multi-Agent Reinforcement Learning, September 2024. URL http://arxiv. org/abs/2408.15381. arXiv:2408.15381 [cs]. 10

    2024 arXiv

  2. [10]

    Oliehoek and Christopher Amato

    Frans A. Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs. Springer, 2016

    2016

  3. [11]

    Approximation theory of the MLP model in neural networks

    Allan Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica, 8: 143–195, January 1999. ISSN 1474-0508, 0962-4929. doi: 10.1017/S0962492900002919. URL https://www.cambridge.org/core/journals/acta-numerica/article/ abs/approximation-theory-of-the-mlp-mo...

    1999 doi

  4. [12]

    Weighted QMIX: Expand- ing Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

    Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: Expand- ing Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 33, pages 10199–10210. Curran Associates,...

  5. [13]

    Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

    Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. Journal of Machine Learning Research, 21(178):1–51, 2020. ISSN 1533-7928. U...

    2020

  6. [14]

    JaxMARL: Multi-Agent RL Environments and Algorithms in JAX

    Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Garðar Ingvarsson, Timon Willi, Ravi Hammond, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, Saptarashmi Bandyopadhyay, Mikayel Samvelyan, Minqi Jiang, Robert Tjarko Lange, Shimon Whit...

    2024

  7. [15]

    QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learn- ing

    Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learn- ing. In Proceedings of the 36th International Conference on Machine Learning, pages 5887–

  8. [16]

    Leibo, Karl Tuyls, and Thore Graepel

    Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-Decomposition Networks For Cooperative Multi-Agent Learning, June 2017. URL http://arxiv...

    2017 arXiv

  9. [17]

    QPLEX: Duplex Dueling Multi-Agent Q-Learning

    Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: Duplex Dueling Multi-Agent Q-Learning. In International Conference on Learning Representations, October 2020. URL https://openreview.net/forum?id=Rcmk0xxIQV

    2020

  10. [18]

    Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization

    Jianhao Wang, Zhizhou Ren, Beining Han, Jianing Ye, and Chongjie Zhang. Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization. In Ad- vances in Neural Information Processing Systems , volume 34, pages 29142–29155. Cur- ran Associates, Inc., 2021. UR...

    2021

  11. [19]

    strong” forms of UAT not formally applicable, and come to the primary conclusions that (i) only “weak

    Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling Network Architectures for Deep Reinforcement Learning. In Proceedings of The 33rd International Conference on Machine Learning, pages 1995–2003. PMLR, June 2016. URL https://proceeding...

    1995

  12. [22]

    Q∈FC (QIGM) =⇒ Q∈FC IGM, i.e.,QIGM satisfies IGM (already proven above),

  13. [23]

    Step 1 was already proven earlier

    Q∈FC IGM =⇒ Q∈FC (QIGM), i.e., any IGM function is representable byQIGM. Step 1 was already proven earlier. Next, we prove step 2. LetQi(hi,ai) andQ(h,a) denote an arbitrary set of individual and joint values that satisfy IGM, i.e.,Q∈FC IGM. Let us denote the usual correspondi...

  14. [24]

    a∗ = argmaxaQIGM(h,s,a), i.e., the individual maximal actions also maximize the joint history-state values

  15. [25]

    a∗ = argmaxa Es|h [QIGM(h,s,a)], i.e., the individual maximal actions also maximize the marginalized joint history-state values. Step 1. The advantage utilities corresponding toa∗ are zero∀i(ui = 0) by definition, and QIGM(h,s,a∗) =w(h,s,a∗)f(u1,...,u N)| {z } =0 +b(h,s ) =b(h...

  16. [26]

    a∗ = argmaxaQIGM(h,s,a), i.e., the individual maximal actions also maximize the state-only values

  17. [27]

    X” over VDN. 0 2 M 4 M 6 M 8 M 10 M Timesteps 0% 25% 50% 75% 100%Pr(X > Q+FIX-sum) Model Q+FIX-sum Q+FIX-mono Q+FIX-lin QPLEX QMIX VDN (b) POI of model “X

    a∗ = argmaxa Es|h [QIGM(h,s,a)], i.e., the individual maximal actions also maximize the marginalized joint state-only values. Step 1. The advantage utilities corresponding toa∗ are zero∀i(ui = 0) by definition, and QIGM(h,s,a∗) =w(s,a∗)f(u1,...,u N)| {z } =0 +b(s) =b(s). (50) ...

  18. [28]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The primary contributions (derivation of a simple IGM value decomposition framework, of the QFIX family, and the corre...

  19. [29]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Limitations are mentioned in the conclusions section. Guidelines: • The answer NA means that the paper has no limitation while the answer No means th...

  20. [30]

    Proof sketches were omitted in the main document due to the proofs being strictly technical, and space limitations

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 28 Answer: [Yes] Justification: Every novel proposition/theorem is clearly indicated as such, with clearly stated assumpt...

  21. [31]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  22. [32]

    Instructions for the Pymarl2 imple- mentation are provided in the readme

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Anonymized code is provided...

  23. [33]

    Any additional component (e.g., the architectures of Q+FIX) is both described in the appendix, provided as supplementary material, and will be linked in the camera ready)

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: As the experiments are based ...

  24. [34]

    30 Guidelines: • The answer NA means that the paper does not include experiments

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: All results are shown with clearly-stated bootstra...

  25. [35]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: The appendix contains a...

  26. [36]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: The work does not violate any of the guidelines outlines in the Code of Ethics....

  27. [37]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 31 Answer: [NA] Justification: The work develops general-purpose foundational methods not intrinsically tied to particular application...

  28. [38]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses...

  29. [39]

    Our own implementations continue to use the same license

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The paper’s expe...

  30. [40]

    These will be provided as forks from the corresponding repositories

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The paper provides implementations of Q+FIX based on two well-known multi-agent RL frameworks: Pymarl2 and JaxMARL. ...

  31. [41]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  32. [42]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  33. [43]

    Answer: [NA] Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

  34. [2017]

    arXiv:1412.6980 [cs]

    URL http://arxiv.org/abs/1412.6980. arXiv:1412.6980 [cs]

    arXiv

  35. [5896]

    URL https://proceedings.mlr.press/v97/son19a.html

    PMLR, May 2019. URL https://proceedings.mlr.press/v97/son19a.html. ISSN: 2640-3498

    2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.