{"id":"cd7e5ee2-a200-4a32-8b5e-550b82de0b21","arxiv_id":"2505.05262","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SMPE2 augments the MAA2C multi-agent actor-critic with per-agent variational state-belief inference, learnable observation filters, and count-based intrinsic exploration, and reports improved performance on MPE, LBF, and RWARE benchmarks.","lead":"The paper proposes SMPE2, a multi-agent reinforcement learning method in which each agent learns a compressed belief about the unseen global state from its own observations, then uses that belief both to improve its policy and to drive exploration. The method reports better average rewards than several state-of-the-art baselines on cooperative particle, foraging, and warehouse tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed adversarial exploration is not implemented: intrinsic reward is per-agent novelty on z_i with no coupling to other agents' reconstruction losses, so the central mechanism-label is unsupported without a direct test.","rationale":"The reader's weakest-assumption analysis focused on whether an agent's own observation o_i carries enough evidence to reconstruct useful features of o_-i; that is a real concern, but the AM filters explicitly downweight features that cannot be inferred through z_i, partly mitigating it. The more load-bearing issue is the paper's central mechanism claim: the 'adversarial exploration' is presented as a distinctive contribution, yet the actual objective in Section 3.2 contains no dependence on other agents' reconstruction losses. The intrinsic reward is purely a count-based novelty bonus on z_i, which is a function of o_i; any benefit to other agents' models is only an indirect and unmeasured side effect. This matters because the abstract, introduction, and Section 3.2 all credit the gains to adversarial exploration, and the ablation in Figure 5 compares against raw-observation SimHash without reporting the correlation that would validate the adversarial interpretation. The empirical results themselves are plausible, the code is available, and the ablations support the importance of state modelling and exploration components, so conditional acceptance remains appropriate; the concern should be settled by the proposed instrumentation before the adversarial claim is taken at face value.","tokens_in":173,"tokens_out":10600,"duration_ms":129888,"concrete_test":"Instrument SMPE2 to log, for every transition stored for agent i, the intrinsic reward r_hat_i and the reconstruction loss L_rec^j of each other agent j evaluated on that transition (with encoder-decoder and target-filter parameters frozen). Compute the Spearman correlation between r_hat_i and the corresponding L_rec^j across all six seeds and training steps. If the correlation is not significantly positive, the claimed adversarial coupling does not materialize and the exploration mechanism reduces to ordinary per-agent novelty on a learned embedding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline contribution includes 'adversarial type of exploration policies which encourages agents to discover novel, high-value states while improving the discriminative abilities of others.' However, Section 3.2 defines only per-agent count-based intrinsic rewards on the agent's own belief: r_hat_i = 1/sqrt(n(SH(z_i))) with z_i ~ q(zi|oi). No term in the reward, actor loss, or encoding loss depends on other agents' reconstruction losses, and no cross-agent gradient or credit is assigned for making other agents' ED targets harder. The text asserts that novel o_i 'aim to adversarially increase the losses of other agents' reconstruction models,' but this is a causal claim about the training dynamics, not an objective that SMPE2 optimizes. The algorithm is well-defined, but the adversarial mechanism is not actually present in the loss; the reported gains could come entirely from standard count-based exploration plus state-conditioned policies. This is not an internal inconsistency, but it means the paper's stated mechanism is unverified and the central novelty claim is overstated until the coupling is demonstrated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SMPE2, a cooperative multi-agent reinforcement learning method for partially observable, communication-free settings. Each agent learns a variational encoder-decoder that maps its own observation o_i to a latent belief z_i and reconstructs filtered observations of other agents, with learned per-agent weight vectors w_i intended to suppress non-informative features. The belief z_i is fed into the agent's policy, and a count-based intrinsic reward is computed by applying SimHash to z_i to encourage exploration in belief space. SMPE2 is built on the MAA2C algorithm and is evaluated on MPE, LBF, and RWARE benchmarks against several baselines, reporting improved episodic rewards. Ablations examine the contribution of the filters, the KL regularization, the norm loss, the second critic, and the intrinsic reward, and the appendix documents hyperparameters, running times, and an analysis of the learned filters.","tokens_in":23571,"tokens_out":8266,"duration_ms":83851,"significance":"If the reported empirical results are reliable, the paper offers a practically useful and simple recipe for improving actor-critic MARL under partial observability: augment the policy with learned latent beliefs and use the same beliefs for count-based exploration. The method does not require communication or prior knowledge of observation semantics, and the provided code and running-time comparisons are useful. However, the advertised 'adversarial exploration' mechanism, which is central to the title and abstract, is not actually implemented in the objective: the intrinsic reward for an agent depends only on its own belief, with no coupling to other agents' reconstruction losses. The empirical gains are likely real, but the mechanistic claim is unsupported and the theoretical framing (Proposition 2.1) is a trivial equivalence.","major_comments":[{"comment":"The paper's headline claim of an 'adversarial type of exploration' is not reflected in the algorithm. The intrinsic reward r_hat_i = 1/sqrt(n(SH(z_i))) depends only on agent i's own belief z_i, which is a deterministic function of its observation o_i. No term in the total loss L_SMPE (Section 3.1) involves another agent's reconstruction loss L_rec^j, and Algorithm 1 (line 16) computes the intrinsic reward per agent without any cross-agent term. The statement that novel observations 'adversarially increase the losses of other agents' reconstruction models' is a causal claim about training dynamics, but SMPE2 optimizes no objective that rewards such increases. The ablation in Figure 5 comparing belief-space hashing ('intr') with observation-space hashing ('obs rew') tests the hash domain, not adversarialness. To support the advertised mechanism, the authors should either (a) introduce an explicit coupling, e.g., an intrinsic reward that depends on other agents' reconstruction errors, or (b) provide direct evidence, e.g., measure that states discovered by one agent cause measurable increases in other agents' L_rec, and show that this correlates with performance, or (c) rewrite the abstract and Section 3.2 to describe the contribution as belief-based count-based exploration without the 'adversarial' label. As it stands, the central novelty claim is overstated, and the reported improvements are consistent with standard count-based exploration plus belief-conditioned policies.","section":"Section 3.2 and Algorithm 1"},{"comment":"Proposition 2.1 is a trivial equivalence: adding a latent variable z that the policy class is free to ignore cannot shrink the set of achievable value functions, and the proof correctly observes that any Dec-POMDP policy can be represented by a policy that ignores z. The subsequent interpretation, that this 'allows us to explore how agents can form meaningful beliefs' that 'enhance their own policies', is a non sequitur. Optimality of the unconstrained objective says nothing about whether the implemented heuristic—adding λ·ELBO (Eq. 2) with a fixed λ and training w_i through a separate critic loss (Eq. 6)—yields policies that are near-optimal for V_SM. As written, the connection from the theoretical objective to the loss L_encodings (Eq. 7) is heuristic. The authors should either weaken the theoretical framing (e.g., present Proposition 2.1 only as an equivalence observation and describe the ELBO as a practical auxiliary objective) or provide a formal statement with assumptions under which the auxiliary objective preserves value optimality.","section":"Section 2.2, Proposition 2.1"}],"minor_comments":[{"comment":"The heading 'Missing Proof' should be renamed to 'Proof of Proposition 2.1' or simply 'Proposition 2.1 Proof'.","section":"Appendix D"},{"comment":"Table 1 reports the accuracy for 'zi (without wi)' as 80.8%, while the text in the same subsection says 'or high accuracy (80.3%)'; these numbers should be reconciled.","section":"Appendix E.4.8"},{"comment":"The notation e_w_i (the target for the filter weights) is used but never defined in the main text; it is only implicitly introduced in Algorithm 1 as the target weight network (phi^w_i)'. Please define it near Eq. (3).","section":"Equation (3)"},{"comment":"Typographical issues: 'By doing do' should be 'By doing so', and in Section 4.2 'The only exception on this is' should be 'The only exception to this is'.","section":"Section 3.2"},{"comment":"The method is referred to as both 'SMPE' and 'SMPE2'; please use 'SMPE2' consistently as the algorithm name, and reserve 'SMPE' for the no-intrinsic-reward ablation variant.","section":"Abstract and Section 1"}],"recommendation":"major_revision","confidential_remarks":"The experimental evidence appears credible, the code is provided, and the paper is generally clear. The main concern for publication is the mismatch between the advertised 'adversarial exploration' and the actual algorithm; this is a fixable issue either by adding a cross-agent coupling or by revising the claims. I am not questioning the empirical results, which seem solid, but the paper as submitted overstates its central novelty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth your time, but read it with a grain of salt around the word \"adversarial.\" What's actually new and good: a per-agent variational encoder-decoder that reconstructs filtered other-agent observations, feeding the belief into the policy, plus count-based intrinsic rewards in belief space. The combination—with learnable filters tied to a value critic—is not in the cited prior work, and the empirical story is solid. Six seeds, confidence intervals, ablations, code, and comparisons against strong baselines (MAPPO, COMA, ATM, EOI, EMC, MASER, LIAM, SIDE) on MPE, LBF, and RWARE. They even solve a couple of LBF tasks previously flagged as open challenges. The running-time table and the t-SNE/logistic-regression analysis of belief cohesiveness are nice touches. Credit where due: this is a serious, reproducible empirical contribution.\n\nNow the soft spots. The stress-test note is right: the \"adversarial exploration\" mechanism is not actually implemented. Section 3.2 defines a per-agent SimHash bonus on z_i, and no term in the reward, actor loss, or encoding loss depends on other agents' reconstruction losses. The text says novel o_i \"aim to adversarially increase the losses of other agents' reconstruction models,\" but that's a story about emergent training dynamics, not a loss the algorithm optimizes. The gains could plausibly come entirely from standard count-based exploration in a good representation space plus belief-conditioned policies. This is a framing problem, not a fatal one, but the authors should either add a direct test (e.g., track other agents' ED losses when one agent visits novel z_i) or drop the \"adversarial\" language.\n\nProposition 2.1 is a trivial equivalence—the Dec-POMDP objective is recovered by letting the policy ignore z—so it doesn't carry any weight. The second critic's connection to the encoder is also indirect; the ablations show it helps, but the exact mechanism (why training w_i to predict a filtered-state value improves the belief encoder) is under-explained. And the reconstruction target presumes o_i is informative enough to infer useful features of o_-i; that's plausible in these benchmarks but may not transfer everywhere. These are minor-to-moderate concerns.\n\nOverall: the central empirical claim is well supported, and the paper deserves a real peer-review process. I'd recommend acceptance with a request to fix the adversarial-exploration claim and clarify the second critic's role. The paper is a good candidate for a reading group discussion about the gap between mechanism labels and implemented losses.\n\nFor peer review: accept. For my own work: I'd probably cite the belief-space exploration and the filter visualization.","headline":"SMPE2 is a solid empirical MARL paper whose 'adversarial exploration' label overstates what the losses actually implement; the method still deserves a serious referee and likely acceptance after the mechanism claim is reined in.","tokens_in":24147,"tokens_out":2510,"would_cite":true,"duration_ms":27980,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Belief-modelling agents beat state-of-the-art MARL.","keywords":["cooperative multi-agent reinforcement learning","partial observability","state modelling","variational inference","agent modelling","count-based intrinsic exploration","sparse-reward exploration","actor-critic"],"falsifier":"Run SMPE2 on a cooperative task engineered so that each agent's observation is independent random noise carrying no information about the other agents' observations or the underlying state; if the method still improves over the MAA2C backbone, the reconstruction loss must be learning something other than state information, while if it collapses to backbone performance the state-modelling assumption is confirmed as load-bearing. A quantitative complement is to measure the filtered reconstruction error of o_-i against a chance-level predictor under such constructed noise.","tokens_in":1623,"feed_emoji":"🤖","tokens_out":7634,"duration_ms":92198,"temperature":0.7,"pith_summary":"The paper tries to establish that cooperative multi-agent reinforcement learning under partial observability can be substantially improved if each agent learns a compressed, probabilistic belief about the non-observable joint state, filters that belief to keep only information relevant to its own value, and uses it both as an extra input to its policy and as the space for intrinsic exploration. The authors claim that combining variational inference over filtered reconstruction targets with SimHash novelty rewards in belief space lets distributed agents coordinate in sparse-reward tasks without communication and without prior knowledge of what observation features mean. If true, any actor-critic MARL learner could be upgraded with the same state-modelling and exploration mechanism, with consequences for robotic swarms, warehouse logistics, and search-and-rescue.","feed_headline":"Belief-modelling agents beat state-of-the-art MARL","feed_subtitle":"Agents reconstruct hidden teammate observations via filtered beliefs, then explore novel beliefs to cooperate.","key_machinery":"The central object is the per-agent variational encoder-decoder with agent-modelling filters w_i: each agent learns a latent belief z_i ~ q_omega_i(z_i | o_i) that is trained to reconstruct w_i·o_-i, with w_i produced by a sigmoid MLP so that non-informative features are down-weighted. A second critic V_k(hat-s) on the filtered state ties the belief to policy optimization, and the adversarial exploration uses SimHash counts of z_i to give intrinsic reward for novel belief states. Together these components make the belief informative, policy-relevant, and explorative, and they are what carries the claimed performance gains.","core_discovery":"Each agent i runs a variational encoder-decoder that maps its own observation o_i to a latent belief z_i, and is trained to reconstruct only the informative features of the other agents' observations o_-i, where per-feature importance weights w_i are learned alongside. A second critic conditioned on the filtered state makes w_i and z_i optimize the value function rather than reconstruction accuracy alone, and a SimHash counter on z_i supplies intrinsic rewards that push the agent to discover novel observations, which in turn become novel training targets for the other agents' decoders. The paper reports that SMPE2 outperforms MAA2C, COMA, MAPPO, ATM, EOI, EMC, and MASER on MPE, LBF, and RWARE tasks, including LBF tasks previously flagged as open challenges.","pith_inferences":["The interpretability of the learned filters w_i suggests that the method could double as a debugging tool, revealing which local features each agent actually relies on for coordination.","The adversarial-exploration argument points toward a cooperative game-theoretic reading in which each agent's exploration deliberately creates harder training targets for the others; formalizing that as a min-max or influence objective is a natural extension the paper does not develop.","Because z_i is conditioned only on the current observation, the intrinsic reward emphasizes observation novelty over trajectory novelty; in tasks where high-value states are reached only through long action sequences this may under-reward gradual progress, which is a testable limitation.","The good performance in most experiments is obtained with shared policy parameters, so whether SMPE2's gains survive in fully heterogeneous teams with separate networks remains an open empirical question that the paper's backbone-flexibility result only partially answers."],"forward_implications":["SMPE2 outperforms state-of-the-art MARL algorithms on complex fully cooperative tasks in the MPE, LBF, and RWARE benchmarks.","The ablation studies show that removing the agent-modelling filters, the KL term, the L2 normalization on w_i, or the second critic degrades performance, so each component of the method is load-bearing.","When the KL regularization is enabled, agents' belief embeddings become cohesive across agents, as shown by t-SNE visualization and by a logistic-regression separation accuracy that drops from 99.3% to 57.5%.","SMPE2 is backbone-flexible: using MAPPO as the actor-critic backbone in place of MAA2C still yields large gains over plain MAPPO.","SMPE2 manages to solve sparse-reward LBF tasks that prior work highlighted as open challenges, where other intrinsic-motivation methods fail."],"supporting_citations":[{"why":"Supplies the amortized variational inference machinery, including the ELBO and the reparameterization trick, on which the belief encoder-decoder is built.","marker":"(Kingma & Welling, 2013)"},{"why":"Provides the MAA2C backbone algorithm, the RWARE benchmark, and the benchmarking setup used throughout the experiments.","marker":"(Papoudakis et al., 2020)"},{"why":"Provides the SimHash count-based exploration mechanism that SMPE2 adapts to operate in the belief space z_i.","marker":"(Tang et al., 2017)"},{"why":"Establishes that redundant state information can harm performance, which motivates the agent-modelling filters w_i.","marker":"(Guan et al., 2022b)"},{"why":"Supplies LIAM, the agent-modelling baseline that the paper extends to MLIAM and compares against.","marker":"(Papoudakis et al., 2021)"},{"why":"Supplies SIDE, a state-inference baseline that does not use the inferred belief during execution and is used for comparison.","marker":"(Xu et al., 2022)"},{"why":"Supplies the MAPPO baseline and its hyperparameter configuration, both for direct comparison and as a backbone for SMPE2.","marker":"(Yu et al., 2022)"},{"why":"Supplies the COMA algorithm used as a baseline in the MPE benchmark experiments.","marker":"(Foerster et al., 2018b)"},{"why":"Provides the LBF benchmark and the agent-modelling formulation that the state-modelling framework builds on.","marker":"(Albrecht & Stone, 2017)"},{"why":"Provides QMIX, whose poor LBF performance limits the EMC and MASER baselines in the comparison.","marker":"(Rashid et al., 2020)"}],"fun_headline_variants":["Belief-state modeling plus adversarial exploration lifts MARL agents","SMPE: Inferring hidden state beliefs beats top MARL baselines","Filtered belief representations boost cooperative multi-agent RL","Agents discover novel beliefs to outperform state-of-the-art MARL","Adversarial belief exploration improves multi-agent cooperation"],"cache_read_input_tokens":26368,"weakest_assumption_plain":"The whole construction assumes that an agent's own observation contains enough statistical information to reconstruct, after filtering, the features of other agents' observations that matter for coordination; if individual observations are too weak or ambiguous, the belief z_i cannot encode useful state information and both the policy enhancement and belief-space exploration collapse.","fun_headline_variants_meta":{"raw":{"variants":["Belief-state modeling plus adversarial exploration lifts MARL agents","SMPE: Inferring hidden state beliefs beats top MARL baselines","Filtered belief representations boost cooperative multi-agent RL","Agents discover novel beliefs to outperform state-of-the-art MARL","Adversarial belief exploration improves multi-agent cooperation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1320,"prompt_tokens":888,"completion_tokens":432,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":349}},"tokens_in":504,"tokens_out":432,"duration_ms":4846,"temperature":1.0,"reasoning_tokens":349,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:08:30.858878+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SMPE2 on a cooperative task engineered so that each agent's observation is independent random noise carrying no information about the other agents' observations or the underlying state; if the method still improves over the MAA2C backbone, the reconstruction loss must be learning something other than state information, while if it collapses to backbone performance the state-modelling assumption is confirmed as load-bearing. A quantitative complement is to measure the filtered reconstruction error of o_-i against a chance-level predictor under such constructed noise.","supporting_citations":[{"cited_title":"and Albrecht, S","cited_arxiv_id":null,"evidence_quote":"Provides the MAA2C backbone algorithm, the RWARE benchmark, and the benchmarking setup used throughout the experiments."},{"cited_title":"\\# exploration: A study of count-based exploration for deep reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Provides the SimHash count-based exploration mechanism that SMPE2 adapts to operate in the belief space z_i."},{"cited_title":"Agent modelling under partial observability for deep reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Supplies LIAM, the agent-modelling baseline that the paper extends to MLIAM and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LBF benchmark and the agent-modelling formulation that the state-modelling framework builds on."},{"cited_title":"S., Farquhar, G., Foerster, J., and Whiteson, S","cited_arxiv_id":null,"evidence_quote":"Provides QMIX, whose poor LBF performance limits the EMC and MASER baselines in the comparison."}],"review_version":1}