{"id":"b4e14481-2253-4b80-8b8b-dab308b35887","arxiv_id":"2505.05968","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"OMSD uses diffusion models to estimate per-agent conditional score functions of the joint behavior policy, replacing the standard product-factorization assumption and improving offline cooperative MARL performance on multimodal datasets.","lead":"This paper presents a new algorithm, OMSD, for offline multi-agent reinforcement learning that models how teams of agents behave using a chain of conditional probability distributions, one agent at a time. It reports large gains over previous methods on several multi-agent robot and particle benchmarks, especially when the training data contains multiple competing ways to cooperate.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 8 omits the suffix coupling terms Σ_{j>i} ∇_{a_i} log μ_j(a_j|s,a_{<j}) that the chain-rule factorization requires, so the sequential score update is not the stated KL gradient and the 'unbiased' claim is unsupported.","rationale":"The reader identified the same load-bearing weakness: Eq. 8 treats ∇_{a_i} log μ_i(a_i|s,a_{<i}) as the complete behavior regularization, dropping the suffix terms Σ_{j>i} ∇_{a_i} log μ_j(a_j|s,a_{<j}) that the exact chain-rule score contains. This is the single most load-bearing concern because the paper's central claim is not only empirical; it asserts an 'unbiased' sequential decomposition that protects against OOD joint actions. The dropped terms are precisely the mathematical expression of inter-agent coordination, so their silent omission undermines the theoretical guarantee, not a minor implementation detail. The empirical contribution may survive: the bandit, MPE, and MaMuJoCo results are internally consistent, two underperforming tasks are honestly reported, and OMSD may still function as an effective heuristic regularizer. But the supplied equations do not support the 'unbiased' and 'strongly guarantees' language. An explicit corrected derivation, a bound on the suffix terms, or a redefinition of the objective as a coordinate-wise surrogate would repair the gap. Because this is addressable and the empirical core is credible, the reader's CONDITIONAL verdict remains appropriate; no verdict change is needed.","tokens_in":25015,"tokens_out":6940,"duration_ms":72892,"concrete_test":"Re-run the two-agent bandit of Sec. 5.1: fit the sequential diffusion models μ_1(a_1|s) and μ_2(a_2|s,a_1) from the dataset, and along the published OMSD trajectory compute the norms of the retained term R = ∇_{a1} log μ_1(a_1|s) and the dropped suffix term S = ∇_{a1} log μ_2(a_2|s,a_1) at sampled policy actions. Also run an OMSD variant whose agent-1 update includes S in Eq. 8. If ‖S‖/‖R‖ is non-negligible (e.g., >0.1) anywhere, or if adding S materially changes the policy trajectory or final bandit reward, then the dropped coupling terms are material and Eq. 8 is a biased approximation of the joint score in the paper's own motivating example. If ‖S‖ is negligible, the concern is defused for the bandit and should be re-tested on MaMuJoCo, where inter-agent coupling is stronger.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that OMSD provides 'unbiased, coordinated' policy updates by sequentially decomposing the joint behavior policy (Sec. 4.2). This requires Eq. 8 to be the gradient of the KL objective in Eq. 6. Under the motivating factorization μ(a|s)=∏_{j=1}^n μ_j(a_j|s,a_{1:j-1}), the exact per-agent score of the joint log-policy is ∇_{a_i} log μ(a|s) = ∇_{a_i} log μ_i(a_i|s,a_{<i}) + Σ_{j>i} ∇_{a_i} log μ_j(a_j|s,a_{<j}). Eq. 8 keeps only the first term. The dropped suffix terms are not zero in general: for n=2, agent 1's update omits ∇_{a1} log μ_2(a_2|s,a_1), which measures exactly how strongly agent 2's behavior depends on agent 1's action, i.e., the coordination structure the method claims to preserve. Thus the update in Eq. 8 does not descend D_KL[π_θ || μ] (nor the surrogate in Eq. 21), and the labels 'unbiased' (Sec. 4.2) and 'strongly guarantees' (Appendix G.3) are unsupported. No bound, stationarity condition, or corrected derivation for the dropped terms is provided. The method may still work as a heuristic coordinate-style regularizer, but the load-bearing guarantee about avoiding OOD joint actions is not established by the paper's equations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OMSD, an offline cooperative MARL algorithm. It models the joint behavior policy with a chain-rule factorization and learns each per-agent conditional score with a diffusion model; these scores are used as behavior regularizers in per-agent policy updates guided by a centralized IQL critic. The authors argue that this sequential score decomposition avoids the multimodality and coordination failures of IGO-style product factorization, and they report large improvements over prior offline MARL methods on MPE and MaMuJoCo benchmarks, with ablations and t-SNE visualizations supporting the in-distribution claim. The central theoretical assertion is that the resulting per-agent update is an 'unbiased' sequential decomposition of the joint-policy KL constraint.","tokens_in":25224,"tokens_out":13219,"duration_ms":136742,"significance":"If the theoretical claim were correct, the paper would make a meaningful contribution: it identifies a concrete failure mode of product-factorized behavior regularization under multimodal offline data and provides a diffusion-based estimator that acts on per-agent scores while preserving decentralized execution. Strengths of the empirical work are the externally evaluated rollouts on standard benchmarks, five-seed reporting, ablations over the regularization temperature and score-decomposition variants, and honest acknowledgment of the two tasks where OMSD underperforms. However, the central 'unbiased' derivation is not correct as written, so the paper's main methodological guarantee is not established; the empirical results are nevertheless substantial and the gap appears fixable by revising the objective or the claims.","major_comments":[{"comment":"The derivation of the per-agent update is not the gradient of the stated KL objective. Under the chain-rule factorization μ(a|s)=∏_{j=1}^n μ_j(a_j|s,a_{<j}), the score of the joint behavior policy with respect to a_i is ∇_{a_i} log μ(a|s) = ∇_{a_i} log μ_i(a_i|s,a_{<i}) + Σ_{j>i} ∇_{a_i} log μ_j(a_j|s,a_{<j}). Eq. (8) keeps only the first term. In the n=2 case, the dropped term ∇_{a_1} log μ_2(a_2|s,a_1) measures exactly how strongly agent 2's behavior depends on agent 1's action, which is the coordination structure the method claims to preserve. Eq. (6) also writes the regularizer as DKL(π_{θ_i}π_{θ_{-i}} || μ_i μ_{-i}) without defining μ_{-i}; if μ_{-i}=∏_{j≠i} μ_j, then the gradient of that KL contains the suffix terms, and if μ_{-i} is meant to absorb them, the equality to the joint KL is lost. Thus Eqs. (7)–(8) do not descend the stated objective or the surrogate in Eq. (21); the labels 'unbiased' (Sec. 4.2) and 'strongly guarantees' (Appendix G.3) are unsupported. Please provide a corrected derivation, include the suffix terms via a joint score estimator, or explicitly reformulate the objective and remove the guarantee language.","section":"§4.2, Eqs. (6)–(8)"},{"comment":"The conditional diffusion models are trained on prefixes a_{i-} drawn from the dataset, but at policy-update time Eq. (8) evaluates ∇_{a_i} log μ_i(a_i|s,a_{i-}) at prefixes sampled from the updated policies π_{θ_{i-}}(s). No correction is provided for this covariate shift, and the claim in Sec. 4.2 that the conditional structure 'ensures that the current agent is updated with respect to a meaningful, in-distribution context' is not established by the equations. This is a second, distinct distribution-shift source that is central to the paper's OOD-safety claim; it should be analyzed, or the claim should be restricted to the empirical t-SNE evidence.","section":"§4.3 and Appendix G.3, Eq. (20)"}],"minor_comments":[{"comment":"The algorithm refers to Eq. (7) for the policy update, but the practical update implemented in the paper is Eq. (8).","section":"Algorithm 1, line 12"},{"comment":"The phrase 'episodic infinite horizon problem' is contradictory; the paper should specify either an episodic finite-horizon or an infinite-horizon discounted setting.","section":"§3.1"},{"comment":"Reference [16] is cited for classifier-free guidance, but [16] is IDQL, not a classifier-free-guidance diffusion paper; the citation should be corrected.","section":"Appendix G.3"},{"comment":"The claim to be 'the first work to explicitly address the distributional gap between offline and online MARL' is too strong given earlier works on OOD joint actions, such as AlberDICE, COMADICE, and SIPO; the novelty claim should be softened.","section":"Abstract and Conclusion"},{"comment":"The running-time statements are inconsistent: the text says OMSD took 10 hours for HalfCheetah and 1 hour for MPE, then lists separate 10-hour pretraining and 12-hour policy-training times; the total should be clarified.","section":"Appendix H"},{"comment":"The caption states that shaded columns represent the proposed method, but no shading is visible in the manuscript; use boldface, an asterisk, or a printed shading to mark OMSD.","section":"Table 2"},{"comment":"Minor typographical issues should be fixed: 'infactorization' in the §4.1 heading, 'classsifier' in Appendix G.3, 'comparation' in §5.3, and inconsistent spelling of 'Dilac/DiLac'.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The concern about Eq. (8) raised by the reader is valid and is the main technical obstruction to acceptance. The empirical study is solid and the method may work as a heuristic, so I do not recommend rejection; the authors should either correct the derivation or explicitly downgrade the theoretical claims from 'unbiased' to a heuristic per-agent regularization."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful empirical paper with an overstated theoretical wrapper. The core idea—replace IGO product factorization with sequential conditional score regularization, distilled via diffusion models—is new enough and works well on MPE and MaMuJoCo. But Eq. (8) does not follow from Eq. (6), and the 'unbiased' language should not survive review.\n\nWhat's good: the multimodal joint behavior problem is real and clearly motivated; Proposition 4.1 cleanly shows how an IGO factorized approximation can blow up support; the bandit experiment makes the failure mode concrete. Results are extensive (5 seeds, normalized scores, ablations, t-SNE), and the two tasks where OMSD underperforms are acknowledged. The empirical core looks internally consistent, and I did not find contradictions in the benchmark tables. This is the strongest part of the paper.\n\nThe soft spot is the derivation. Under their chain-rule factorization, the exact per-agent score of the joint log-policy includes suffix terms Σ_{j>i} ∇_{a_i} log μ_j(a_j|s,a_{<j}). Eq. (8) drops them. So the update is not the gradient of the KL objective in Eq. (6), and 'unbiased' and 'strongly guarantees' are unsupported. It may still work as coordinate-wise conditional regularization—the bandit and benchmark results suggest it does—but the paper needs either a corrected derivation, an explicit statement that this is a surrogate objective, or a bound on the dropped terms. This is addressable, not fatal.\n\nSmaller issues: the SOTA numbers come with per-task/dataset sweeps of β and IQL critic hyperparameters; the paper says 'open-source code' but no link appears; 'first to address the distributional gap' ignores AlberDICE and MOMA-PPO, which are cited. None of these sink the empirical contribution.\n\nBottom line: worth a serious referee. I would send it out, with the derivation issue as the main request. If the authors reframe the claim honestly, it is a solid paper for the offline MARL crowd.","headline":"Useful empirical paper with an overstated theoretical wrapper; the sequential score-decomposition idea is worth taking seriously, but Eq. (8) does not match Eq. (6).","tokens_in":25883,"tokens_out":5086,"would_cite":true,"duration_ms":51895,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OMSD claims that offline cooperative multi-agent reinforcement learning can avoid out-of-distribution joint actions by decomposing the joint behavior policy into a chain of per-agent conditional scores, estimated by diffusion models…","keywords":["offline multi-agent reinforcement learning","sequential score decomposition","multimodal behavior policies","distribution shift","diffusion models","decentralized execution","cooperative MARL","behavior regularization"],"falsifier":"On a two- or three-agent bandit with two optimal coordinated modes and strongly correlated agents, compare the OMSD update against a full chain-rule update that includes the neglected suffix terms $\\sum_{j>i}\\nabla_{a_i}\\log\\mu_j(a_j|s,a_{1:j-1})$; if the truncated update selects a different mode, or lands on out-of-distribution joint actions with materially higher probability, the paper's 'unbiased' claim fails. A direct quantitative check is to compute the norm of the dropped terms on the actual MPE and MaMuJoCo datasets and show whether ignoring them changes the learned policy's support.","tokens_in":24674,"feed_emoji":"🤖","tokens_out":9536,"duration_ms":86571,"temperature":0.7,"pith_summary":"This paper argues that the central difficulty in offline cooperative multi-agent reinforcement learning is not the size of the joint action space but the multimodality of what the data actually contains: offline datasets mix several equally good coordination patterns, so a joint behavior policy that is factorized into independent per-agent policies spreads probability mass onto exponentially many incoherent joint actions. The proposed method, OMSD, instead decomposes the joint behavior policy along a chain of conditional distributions, $\\mu(a|s) = \\prod_{i=1}^n \\mu_i(a_i|s, a_{1:i-1})$, and uses diffusion models to estimate each agent's conditional score, the gradient of the log conditional density, as behavior regularization inside a joint-action critic. The paper's claim is that this sequential decomposition yields coordinated mode selection under decentralized execution and protects policy updates from drifting out of the dataset distribution. The payoff, if correct, is that offline MARL can learn from heterogeneous, mixed-quality demonstrations without the distribution shift that undermines existing policy-based methods.","feed_headline":"Sequential score chains stop offline MARL distribution shift","feed_subtitle":"Per-agent conditional scores keep coordinated policies inside the offline data and beat baselines on multimodal benchmarks.","key_machinery":"The load-bearing object is the sequential decomposition of the joint behavior policy into per-agent conditional distributions, together with the score functions of those conditionals. For each agent $i$, a diffusion model is trained to estimate the conditional score $\\nabla_{a_i}\\log\\mu_i(a_i|s,a_{1:i-1})$ through its noise prediction, conditioning on the state and on the actions of earlier agents; during policy update, agent $i$ conditions on the freshly updated policies of its prefix agents, which is what converts a single global distributional constraint into coordinated, decentralized, in-sample updates. A shared joint-action critic trained with expectile regression supplies the top-down reward signal, and the $\\beta$-weighted conditional score supplies the bottom-up in-distribution constraint, and the paper claims this combination avoids both uncoordinated Q-gradients and out-of-distribution joint actions.","core_discovery":"The central claim is that policy-based offline MARL fails structurally because the joint behavior policy in offline data is multimodal and cannot be written as a product of independent marginals, yet standard methods regularize each agent against exactly such a product. OMSD replaces the product with a chain of conditionals, $\\mu(a|s) = \\prod_{i=1}^n \\mu_i(a_i|s, a_{1:i-1})$, and derives each agent's policy gradient as a joint-action Q-gradient plus the conditional behavior score: $\\nabla_{\\theta_i} L = E[\\nabla_{a_i} Q_\\phi(s,a) + \\frac{1}{\\beta}\\nabla_{a_i}\\log\\mu_i(a_i|s,a_{i-})]\\nabla_{\\theta_i}\\pi_{\\theta_i}(s)$, where the score is read out of a diffusion model conditioned on the prefix agents' actions. The paper proves that the independent factorization of a two-mode optimal policy yields $2^n$ spurious modes with total variation distance approaching 1 as the number of agents grows, and the empirical sections show OMSD outperforming the compared baselines on 11 of 13 MPE and MaMuJoCo tasks, with the largest gains on the most multimodal datasets.","pith_inferences":["The agent ordering is a free parameter the paper does not tune: since the analysis shows the dropped suffix terms in the exact chain-rule expansion carry the approximation error, permuting the order should measurably change performance, and an order chosen to minimize that error could be learned from the dataset.","The same chain-of-conditionals idea transfers to partially observable settings by conditioning scores on observation-action histories rather than joint states, and to heterogeneous teams by conditioning on groups of agents rather than single agents.","Proposition 4.1 implies a sharp, testable contrast: distribution shift under the factorized baseline grows exponentially in the number of agents, while the chain decomposition should degrade only with the number of conditioning steps, and measuring that gap on a controlled task would isolate the mechanism the paper claims.","A natural correction to the truncation concern would be an update that keeps the omitted suffix terms by back-propagating through later agents' conditional score models during training; the paper's own BRPO-IGO versus OMSD ablation already brackets how much the decomposition choice matters."],"forward_implications":["Offline cooperative MARL can be built on a joint-action critic plus per-agent conditional behavior scores, bypassing value decomposition and the coverage biases that come with it.","Multimodal datasets, from multiple Nash equilibria, anonymized agent identities, or mixed-quality demonstrations, become usable instead of imposing conflicting regularization on independent policies.","Because the diffusion model is used only to read out score estimates and never to sample actions, training is coordinated while execution stays lightweight and fully decentralized.","The method's largest reported margins come on medium and random datasets, where multimodality is most pronounced, suggesting the gain comes specifically from preserving modes rather than from better single-mode imitation.","Sequential conditioning acts as a safeguard: when early agents produce slightly out-of-distribution actions, later agents still receive in-distribution conditional context and pull the joint policy back toward the data support."],"supporting_citations":[{"why":"Supplies the score-regularized policy optimization framework and the diffusion score-distillation trick that OMSD extends from single-agent to multi-agent learning.","marker":"[7]"},{"why":"Provides the behavior-regularized policy optimization equivalence (Lemma 3.1) from which the per-agent KL objectives and gradient forms are derived.","marker":"[55]"},{"why":"Supplies the implicit Q-learning expectile regression used to pre-train the centralized joint-action critic.","marker":"[21]"},{"why":"Provides the denoising diffusion training objective used to learn the per-agent conditional score models.","marker":"[17]"},{"why":"Supplies the OMAR benchmark datasets, baseline scores, and the normalized-score evaluation protocol used for MPE and HalfCheetah.","marker":"[36]"},{"why":"Provides the OG-MARL dataset suite used for the MaMuJoCo evaluations.","marker":"[11]"},{"why":"The alternating stationary-distribution-correction baseline that frames the out-of-distribution joint-action problem OMSD addresses.","marker":"[33]"},{"why":"Provides the MADiff diffusion baseline and its standardized scores used in the comparison tables.","marker":"[61]"}],"fun_headline_variants":["Sequential score chains tame offline MARL","Decompose scores sequentially to fix offline MARL","Score chain conditionals beat offline MARL distribution","Conditional score chains keep offline MARL on track","Sequential scores beat offline MARL baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper treats each agent's conditional behavior score as the complete regularization signal, ignoring the extra terms that the exact chain-rule expansion of the joint policy would add from later agents' conditional distributions; if those neglected terms are large, the gradient no longer minimizes the stated objective and the promised safety against out-of-distribution joint actions is not guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["Sequential score chains tame offline MARL","Decompose scores sequentially to fix offline MARL","Score chain conditionals beat offline MARL distribution","Conditional score chains keep offline MARL on track","Sequential scores beat offline MARL baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001379,"raw_usage":{"total_tokens":5620,"prompt_tokens":1014,"completion_tokens":4606,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":4534}},"tokens_in":630,"tokens_out":4606,"duration_ms":29635,"temperature":1.0,"reasoning_tokens":4534,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:52:52.345185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a two- or three-agent bandit with two optimal coordinated modes and strongly correlated agents, compare the OMSD update against a full chain-rule update that includes the neglected suffix terms $\\sum_{j>i}\\nabla_{a_i}\\log\\mu_j(a_j|s,a_{1:j-1})$; if the truncated update selects a different mode, or lands on out-of-distribution joint actions with materially higher probability, the paper's 'unbiased' claim fails. A direct quantitative check is to compute the norm of the dropped terms on the actual MPE and MaMuJoCo datasets and show whether ignoring them changes the learned policy's support.","supporting_citations":[{"cited_title":"Score regularized policy optimization through diffusion behavior","cited_arxiv_id":null,"evidence_quote":"Supplies the score-regularized policy optimization framework and the diffusion score-distillation trick that OMSD extends from single-agent to multi-agent learning."},{"cited_title":"Off-the-grid marl: Datasets and baselines for offline multi-agent reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Provides the OG-MARL dataset suite used for the MaMuJoCo evaluations."},{"cited_title":"AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation","cited_arxiv_id":"2311.02194","evidence_quote":"The alternating stationary-distribution-correction baseline that frames the out-of-distribution joint-action problem OMSD addresses."},{"cited_title":"guide-then-select","cited_arxiv_id":null,"evidence_quote":"Provides the MADiff diffusion baseline and its standardized scores used in the comparison tables."}],"review_version":1}