{"id":"4397e24b-4ac1-469b-b5de-328c373a525d","arxiv_id":"2412.07639","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"InSPO updates agents sequentially with in-sample objectives and entropy regularization, avoiding out-of-distribution joint actions and converging to a quantal response equilibrium in offline MARL.","lead":"InSPO is a new offline multi-agent reinforcement learning algorithm that updates agents one at a time using only actions that occur in the logged dataset, while adding an entropy bonus to keep exploring rare actions. It claims monotonic improvement, convergence to a quantal response equilibrium, and strong results on coordination games and StarCraft II.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"InSPO's QRE convergence guarantee is proven only for the exact tabular update in Algorithm 1 (Eqs. 10/13), not for the implemented algorithm with local Q-functions, importance resampling, and CQL; the paper does not show these approximations preserve the theorem.","rationale":"Reading in good faith, the paper is a well-executed empirical contribution: the matrix-game results are clean, the ablations support the roles of entropy and sequential updates, and the SMAC comparison is competitive. The reader's weakest_assumption (data coverage) is real but inherent to any offline algorithm with reverse-KL regularization: the QRE has the same support as the behavior policy, so the method cannot discover actions absent from the dataset. The paper only claims to explore low-probability actions within the behavior policy, and the M-NE datasets include the optimal action with positive probability, so this is a limitation, not a defect in the argument. The more load-bearing concern is the gap between the theoretical guarantee and the implemented algorithm. The theorem is explicitly tabular and uses a global Q-function, while the practical algorithm adds local Q-functions, importance resampling, and a CQL penalty. Because the CQL term changes the objective being optimized, convergence of the exact procedure does not imply convergence of the practical procedure to the same QRE. Without a proof or an ablation showing the practical components preserve the fixed point, the central claim as stated in the abstract is not established for the method evaluated. I also note a concrete flaw in the proof of Proposition 4: the summed inequality does not produce E_{a~\\bar\\pi}[A] because earlier agents are drawn from \\pi_new on both sides; a standard coordinate-ascent argument may salvage the result, but the written proof needs correction. These issues warrant conditional acceptance, with the requested revisions being a scoped statement of the theorem and either an extension of the guarantee to the practical algorithm or an explicit limitation that the guarantee applies only to the tabular variant.","tokens_in":18880,"tokens_out":17485,"duration_ms":158697,"concrete_test":"On the M-NE imbalanced matrix game, run (i) the exact tabular Algorithm 1 (global Q via Eq. 10, policy update via Eq. 13) and (ii) the practical InSPO with alpha_CQL set to 0 in Eq. (24). If both converge to the same QRE from Eq. (12), the extra components are not load-bearing; if the practical version fails to reach (C,C) while the exact version succeeds, the reported result depends on the unproven CQL/local-Q modifications and the theorem does not transfer. Additionally, re-derive the summed inequality in Appendix A.3 using \\bar\\pi's own marginals on the right-hand side; if the equality fails, Proposition 4's proof needs repair.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims InSPO 'guarantees monotonic policy improvement and converges to QRE.' Appendix A.4 proves this only for the tabular Algorithm 1 that uses a global Q-function (Eq. 10) and the exact sequential update (Eq. 13). The evaluated implementation replaces the global Q with per-agent local Q-functions optimized on an importance-resampled dataset (Eqs. 14-15), uses a geometric-mean importance ratio (Eq. 23), adds a CQL penalty (Eq. 24), and replaces the sequential advantage in Eq. (13) with a local advantage Q_phi_i - E_pi_theta_i[Q_phi_i] in Eq. (20). The CQL term is a state-dependent regularizer not present in Eq. (10); it changes the Bellman fixed point, so a policy optimal for the modified objective need not be a QRE of the MEBR-MG. No theorem links these practical components to Proposition 4 or Theorem 5. Thus the central theoretical claim is not established for the algorithm whose results appear in Tables 1-4. Separately, the proof of Proposition 4 contains an invalid substitution when summing inequality (19): the right-hand side is written as E_{a~\\bar\\pi}[A], but each term draws earlier agents from \\pi_new, not \\bar\\pi; a corrected proof is needed even for the tabular case.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InSPO, an offline multi-agent reinforcement learning algorithm that sequentially updates policies using in-sample learning under a maximum-entropy behavior-regularized Markov game (MEBR-MG). The authors derive a closed-form policy update via reverse KL regularization and policy entropy, claim monotonic policy improvement and convergence to quantal response equilibrium (QRE), and provide a practical implementation with local Q-functions, importance resampling, and a CQL penalty. Experiments on matrix games, Bridge, and StarCraft II micromanagement compare against BC, OMAR, AlberDICE, CFCQL, and OMIGA, with InSPO achieving competitive or state-of-the-art results on several tasks.","tokens_in":19180,"tokens_out":7047,"duration_ms":59674,"significance":"The paper addresses a relevant problem—out-of-distribution (OOD) joint actions in offline MARL—and proposes an interesting combination of sequential updates and entropy regularization. The MEBR-MG framework and the QRE characterization are novel, and the empirical results, especially on the M-NE game and StarCraft II, are strong. The authors also release code, which supports reproducibility. However, the theoretical contributions are not fully substantiated: the monotonic improvement proof contains a technical gap, the QRE convergence proof is too abbreviated, and the practical algorithm is not covered by the theoretical guarantees. If these issues are resolved, the paper would be a valuable contribution to offline MARL.","major_comments":[{"comment":"The summation of inequality (19) is invalid. For the n-th term, the expectation is over a^{i1:n-1} ∼ π_new and a^{i_n} ∼ π̄^{i_n}, not over the joint policy π̄. The right-hand side of the summed inequality is therefore not E_{a∼π̄}[A_{π_old}(s,a) − α log (π̄/μ) − β log π̄]. Consequently, the contradiction with L_{π_old}(π̄) > L_{π_old}(π_new) is not established. A corrected proof of monotonic improvement is needed, since this result underpins Theorem 5.","section":"Appendix A.3 (proof of Proposition 4)"},{"comment":"The proof asserts that monotone convergence of Q^{π_k} and boundedness imply that the policies converge to a limit point, and that this limit point satisfies the QRE condition. Monotone Q-function convergence does not by itself imply policy convergence (the softmax policy may cycle), and the argument that the limit policy satisfies the inequality for all i and π^i is not derived. The theorem should either be proved with a complete fixed-point argument or be restricted to the exact tabular update with an explicit statement of all assumptions.","section":"Appendix A.4 (Theorem 10)"},{"comment":"The evaluated algorithm is not the same as the one for which the theory is proven. The practical version replaces the global Q-function with per-agent local Q-functions trained on an importance-resampled dataset, uses a geometric-mean importance ratio in Eq. (23), adds a CQL penalty in Eq. (24) that changes the Bellman fixed point, and uses a local advantage in Eq. (20) instead of the sequential advantage in Eq. (13). No theorem shows that these approximations preserve monotonic improvement or QRE convergence. The abstract and conclusion claim these guarantees for InSPO without qualification, so the central theoretical claim is not established for the algorithm evaluated in Tables 1–4.","section":"The Practical Implementation of InSPO (Eqs. 14–15, 20, 23–24)"},{"comment":"The KKT derivation treats Qπ as independent of the optimizing agent's own policy π^i. As written, the derivation gives a best-response formula for a fixed Q function, and only by requiring self-consistency does it characterize a QRE. The proof should explicitly state this fixed-point structure; otherwise the derivation appears circular. The resulting formula is a standard logit QRE, so this is a correctness-risk issue rather than a fatal flaw, but it needs clarification.","section":"Appendix A.2 (Proposition 3)"}],"minor_comments":[{"comment":"The reference list contains two entries labeled Xu et al. 2023a and 2023b with identical titles and venues; the in-text citation to Xu et al. 2023b for StarCraft II datasets does not match the cited paper. Please correct the citation and reference list.","section":"References"},{"comment":"The notation A^{i1:n}_{π_old}(s,a^{i_n}) is used before it is clearly defined; the definition is placed inside Eq. (7) after the minimizer, and the dependency on a^{i1:n-1} is not made explicit. Please make the notation consistent and unambiguous.","section":"Equation (7)"},{"comment":"The statement that 'the optimal solution of the perturbed game aligns with the solution concept of quantal response equilibrium' is imprecise because QRE is traditionally defined for normal-form games; the Markov-game extension in Definition 1 should be more carefully motivated.","section":"The Proposed Method"},{"comment":"The entropy-based exploration only re-weights actions that already appear in the offline dataset; Proposition 3 implies the QRE policy has the same support as the behavior policy. The paper should state this data-coverage assumption explicitly, e.g., that the globally optimal joint action must have positive probability under the behavior policy for the M-NE game result to hold.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The gap between the tabular theory and the practical algorithm is substantial; the authors should be encouraged either to provide a theorem covering the practical components or to clearly frame the theory as a stylized motivation. The proof error in Appendix A.3 requires a genuine fix, not just a clarification, and the convergence proof in Appendix A.4 needs to be made rigorous. The empirical results are promising, and the paper is likely publishable after these theoretical issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth your attention: InSPO combines in-sample sequential policy updates with an entropy bonus, giving a clean fix for OOD joint actions and a genuine mechanism for escaping local optima. The XOR and M-NE game examples are well chosen, and the empirical results on those games and Bridge are convincing. The code is public, and the paper is generally well written.\n\nThe main problem is in the theory. The proof of Proposition 8 (monotonic improvement) contains an invalid substitution when summing inequality (19): the right-hand side is written as an expectation under \\barπ, but each term in the sum draws earlier agents from π_new, not \\barπ. Without a corrected argument, the claim that π_new = arg max L_{π_old}(π) is not established, so Theorem 5 (convergence to QRE) does not follow from the proof as written. This is not a minor typo; it is the load-bearing step.\n\nThere is also a gap between the tabular theory and the evaluated implementation. The theorem holds for Algorithm 1 with a global Q-function and the exact sequential update. The practical version uses local Q-functions, importance resampling, a geometric-mean ratio, and a CQL penalty. No theorem connects these to the QRE guarantee. The abstract's unconditional \"converges to QRE\" therefore overstates what is proven.\n\nOn the empirical side, the SMAC table is a mixed bag. Some baseline numbers are taken directly from Shao et al., and on a few maps InSPO is clearly worse than CFCQL (e.g., 3s vs 5z medium). The \"outperforms state-of-the-art\" claim in the abstract should be softened to \"competitive with\" or similar.\n\nNone of this makes the paper a reject. The core idea is sound and the experiments show it works well on the tasks that motivated it. But the theoretical results need a genuine repair, not just a footnote, and the claims need to be scaled back to what the evidence supports.\n\nWho should read it: anyone working on offline MARL or on in-sample policy optimization. It deserves a serious referee and could become a solid contribution after a careful revision. I would engage with it, but I would not cite the convergence theorem until the proof is fixed.","headline":"A useful offline MARL method with a real proof gap in its central convergence theorem, but worth serious review.","tokens_in":19727,"tokens_out":3469,"would_cite":true,"duration_ms":33371,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"InSPO claims offline multi-agent RL can be made coordinated and OOD-safe by updating agents sequentially on dataset actions with an entropy bonus, monotonically improving and converging to a quantal response equilibrium.","keywords":["offline multi-agent reinforcement learning","in-sample sequential policy optimization","quantal response equilibrium","out-of-distribution joint actions","maximum-entropy regularization","behavior-regularized Markov game","StarCraft II micromanagement","cooperative MARL"],"falsifier":"In a tabular two-agent game, delete the globally optimal joint action from the offline dataset (give it zero probability) while retaining all other actions; the paper's Eq. (12) predicts InSPO assigns it zero probability. Observing positive probability on that action would refute the support-preservation claim, while observing zero would confirm that the entropy bonus only reweights actions already present.","tokens_in":18652,"feed_emoji":"🤖","tokens_out":6787,"duration_ms":60900,"temperature":0.7,"pith_summary":"The paper proposes InSPO, an offline multi-agent reinforcement learning algorithm that updates agents one at a time using only actions found in the pre-collected dataset. Its central claim is that this in-sample sequential scheme avoids out-of-distribution joint actions, the failure mode where each agent's individually reasonable choice forms a joint action never seen during training, while still coordinating with teammates' updated policies. Adding a policy-entropy bonus lets the method reweight low-probability actions in the behavior policy, which the authors argue prevents premature convergence to sub-optimal equilibria. The authors prove that each update monotonically improves the joint policy and that the algorithm converges to a quantal response equilibrium, and they report state-of-the-art or competitive results on matrix games, Bridge, and StarCraft II micromanagement.","feed_headline":"InSPO: sequential in-sample updates give offline MARL a QRE guarantee","feed_subtitle":"Updating agents one by one on dataset actions avoids out-of-distribution joint actions and escapes local optima.","key_machinery":"The central object is the in-sample sequential policy improvement step under the Maximum-Entropy Behavior-Regularized Markov Game (MEBR-MG) objective. For each agent in a random order, the new policy minimizes a KL divergence to the closed-form Boltzmann policy $\\pi^i_{\\text{new}} \\propto \\mu^i \\cdot \\exp\\big((Q - \\beta \\log \\mu^i)/(\\alpha+\\beta)\\big)$, using only state-action pairs from the dataset; the per-agent Q-value is evaluated with previously updated agents and later agents still at the old policy. The reverse-KL regularizer makes the joint regularization decompose as a sum over agents, and the entropy term $\\beta H(\\pi)$ provides the perturbation that lets the method escape local optima. In the practical implementation, local Q-functions are trained with importance resampling and a CQL-style penalty rather than an explicit joint Q-function.","core_discovery":"On the paper's own terms, the discovery is that two failure modes of offline MARL—out-of-distribution joint actions and convergence to local optima—can be addressed by the same mechanism: a sequential, in-sample policy update in a maximum-entropy behavior-regularized Markov game. When the regularizer is reverse KL divergence, the global regularization decomposes over agents, so each agent's optimal update has the closed form $\\pi^i_{\\text{new}} \\propto \\mu^i \\cdot \\exp\\big((Q - \\beta \\log \\mu^i)/(\\alpha+\\beta)\\big)$, and the resulting objective can be optimized over dataset actions only. Updating agents in sequence lets later agents condition on teammates' updated policies, avoiding conflicting update directions, while the entropy term ensures that actions with low probability under the behavior policy are still explored. The theoretical result is that this alternating evaluation-improvement procedure converges to quantal response equilibrium (Theorem 5), with monotonic improvement at every step (Proposition 4).","pith_inferences":["Because the learned policy's support is contained in the behavior policy's support, InSPO's global-optimum performance is bounded by dataset coverage: on datasets where the optimal joint action never appears, the entropy bonus can only reweight existing actions, not invent new ones.","The same reverse-KL decomposition could be combined with support-expanding mechanisms, such as model-based rollouts or diffusion-generated transitions, to relax the coverage assumption; that combination goes beyond the paper.","Because the limiting policy is a QRE whose shape depends on the temperatures $\\alpha$ and $\\beta$, hyperparameter tuning is effectively equilibrium selection, which is consistent with the variation seen in the paper's $\\alpha$ ablation.","The sequential update scheme's insensitivity to update order in the paper's ablation suggests robustness, but it also leaves adaptive ordering as an open lever for further coordination gains."],"forward_implications":["If InSPO's convergence claim is right, offline MARL can be built on a stochastic equilibrium concept (QRE) rather than deterministic Nash equilibria, giving a principled account of bounded-rational cooperative policies.","Because the learned policy shares the support of the behavior policy, in-sample sequential updates rule out out-of-distribution joint actions by construction.","The method should escape sub-optimal equilibria whenever the globally optimal joint action appears in the dataset with even small probability, as the imbalanced M-NE game shows where only InSPO reaches the (C,C) optimum.","Competitive StarCraft II results indicate the sequential scheme scales beyond matrix games and can match value-decomposition methods in near-IGM environments.","Monotonic improvement per iteration implies stable training within each evaluation-improvement cycle, at least in the tabular setting where Theorem 5 is proven."],"supporting_citations":[{"why":"Defines quantal response equilibrium, the solution concept the paper proves InSPO converges to.","marker":"(McKelvey and Palfrey 1995)"},{"why":"AlberDICE is the sequential-update offline MARL baseline whose Nash-convergence limitation InSPO addresses, and it supplies the Bridge datasets and comparative results.","marker":"(Matsunaga et al. 2023)"},{"why":"HAPPO provides the sequential trust-region update and the multi-agent advantage decomposition lemma used in the monotonic-improvement proof.","marker":"(Kuba et al. 2022)"},{"why":"Supplies the in-sample learning principle and the critique of out-of-sample policy extraction that motivates InSPO's dataset-only updates.","marker":"(Xu et al. 2023a)"},{"why":"OMIGA frames the behavior-regularized Markov game and in-sample value regularization that InSPO builds on and compares against.","marker":"(Wang et al. 2023b)"},{"why":"Shows value-decomposition failure in multi-modal reward landscapes, motivating the sequential approach, and provides the Bridge environment.","marker":"(Fu et al. 2022)"},{"why":"Importance resampling is used in InSPO's practical local Q-function training to reduce variance.","marker":"(Schlegel et al. 2019)"},{"why":"CQL contributes the conservative Q-learning penalty used to regularize InSPO's local Q-functions in implementation.","marker":"(Kumar et al. 2020)"}],"fun_headline_variants":["InSPO: sequential in-sample updates avoid OOD joints","InSPO: safe offline MARL via QRE-guaranteed sequential updates","Offline MARL: sequential in-sample updates dodge OOD and reach QRE","Sequential in-sample updates: offline MARL's OOD cure and QRE path","No OOD joints: InSPO updates agents sequentially in-sample"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The behavior policy must have placed at least some probability on the globally optimal joint action, because InSPO's in-sample updates can only reweight actions already present in the dataset; if the optimal action is entirely absent, the method cannot discover it.","fun_headline_variants_meta":{"raw":{"variants":["InSPO: sequential in-sample updates avoid OOD joints","InSPO: safe offline MARL via QRE-guaranteed sequential updates","Offline MARL: sequential in-sample updates dodge OOD and reach QRE","Sequential in-sample updates: offline MARL's OOD cure and QRE path","No OOD joints: InSPO updates agents sequentially in-sample"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000933,"raw_usage":{"total_tokens":4002,"prompt_tokens":964,"completion_tokens":3038,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":2937}},"tokens_in":580,"tokens_out":3038,"duration_ms":20768,"temperature":1.0,"reasoning_tokens":2937,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:39:03.986328+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a tabular two-agent game, delete the globally optimal joint action from the offline dataset (give it zero probability) while retaining all other actions; the paper's Eq. (12) predicts InSPO assigns it zero probability. Observing positive probability on that action would refute the support-preservation claim, while observing zero would confirm that the entropy bonus only reweights actions already present.","supporting_citations":[{"cited_title":"D.; and Palfrey, T","cited_arxiv_id":null,"evidence_quote":"Defines quantal response equilibrium, the solution concept the paper proves InSPO converges to."},{"cited_title":"E.; Lee, J.; Yoon, J.; Leonardos, S.; Abbeel, P.; and Kim, K","cited_arxiv_id":null,"evidence_quote":"AlberDICE is the sequential-update offline MARL baseline whose Nash-convergence limitation InSPO addresses, and it supplies the Bridge datasets and comparative results."},{"cited_title":"G.; Chen, R.; Wen, M.; Wen, Y.; Sun, F.; Wang, J.; and Yang, Y","cited_arxiv_id":null,"evidence_quote":"HAPPO provides the sequential trust-region update and the multi-agent advantage decomposition lemma used in the monotonic-improvement proof."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Importance resampling is used in InSPO's practical local Q-function training to reduce variance."}],"review_version":1}