{"id":"f994d561-a63f-453b-b00a-601015a87d75","arxiv_id":"2501.08778","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DNA-MARL lets agents with only local observations cooperate by running consensus on value targets and parameters, outperforming prior networked-agent methods and approaching centralized-training performance.","lead":"The paper introduces DNA-MARL, a decentralized multi-agent reinforcement learning method where partially observable agents average their local value estimates over a switching communication network to approximate a shared team value. The method beats earlier decentralized networked-agent baselines on cooperative benchmarks and sometimes matches agents trained with a central critic.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (15) claims consensus on local TD targets approximates a central critic, but even with perfect consensus the average of local targets is not the Bellman target unless the team value is additive over agents' observations; the off-policy ablation already shows the mechanism can hurt.","rationale":"The paper is honestly written, with open-sourced code, fair-looking baselines, and explicit ablations. The strongest claim is that DNA-MARL can emulate a central critic under partial observability via double networked averaging. The reader correctly identified Eq. (15) as the weakest link. I sharpen this: the issue is not merely a missing error bound; the asserted approximation is not a Bellman identity. With perfect consensus, the target converges to the average of local targets, which equals the central-critic target only if the team value function is additive across agents' observations. This is an unstated and untested structural assumption, similar to VDN's additivity but without the centralized training that would validate it. The empirical record supports the concern: in the off-policy LBF environments, the team-Q consensus component degrades performance relative to parameter consensus alone (Fig. 7), indicating that the mechanism does not generally produce a useful team critic. In on-policy tasks it sometimes helps, so the paper's contribution may still be valuable, but the central claim of emulating a central critic is not established. The proposed concrete test directly measures whether the consensus target approximates the central TD target; this would settle the concern empirically. The verdict remains CONDITIONAL: the authors should either derive conditions under which Eq. (15) holds, demonstrate on representative tasks that the consensus target is close to the central target, or temper the claim to settings where the additive factorization is known to hold.","tokens_in":24399,"tokens_out":7594,"duration_ms":80233,"concrete_test":"On the LBF Hard and MPE Tag tasks, train a centralized critic (as in MAA2C) and DNAA2C with identical seeds. During evaluation, record for the same trajectories both the central target r_team^t + γV(o^{t+1}) and the consensus target \\bar{y}^t produced by Eq. (15). Report the normalized mean absolute error between the two target sequences. If the error exceeds, say, 10% of the typical return scale, then Eq. (15) is not approximating the central critic, and the 'emulates a central critic' claim is unsupported. Conversely, if the error is small, the approximation survives despite the missing derivation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is the 'team-V consensus' target in Eq. (15). After K consensus rounds, the update target is \\bar{y}_i^t ≈ (1/N)∑_j (r_j^{t+1} + γV(o_j^{t+1}; ω_j^-)). Even in the limit K→∞, this equals the arithmetic mean of the initial local targets. For this to be the TD target of a central critic, r_team + γV(o'), two conditions are required: (i) the local critics factorize the team value as V(o) ≈ (1/N)∑_j V(o_j; ω_j), i.e., an additive decomposition, and (ii) all agents use a common parameter vector. Neither condition is stated or checked, and (ii) is violated because parameter consensus is run for only K=5 or K=1 rounds with C=1 edge per round, so the critics remain non-identical. The approximation is therefore not a consequence of consensus; it assumes the very additivity that a central critic would not have. The paper's own off-policy ablation (Fig. 7) is consistent with this failure: team-Q consensus degrades performance relative to parameter consensus alone in LBF. Thus the claim that DNA-MARL 'can indeed emulate a central critic' is not supported by the mechanism as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DNA-MARL, a decentralized training and decentralized execution (DTDE) multi-agent reinforcement learning method for cooperative agents under partial observability. Agents communicate over a time-varying network to run consensus not only on actor and critic parameters, as in prior networked-agent work, but also on local value targets (team-V for A2C, team-Q for DQN). The paper introduces the ND-POMG framework, derives a factorized team policy gradient, and evaluates DNAA2C and DNAQL on Level-Based Foraging and Multi-Agent Particle Environment benchmarks against CTDE, DTDE, and independent-learner baselines. The central claims are that DNA-MARL outperforms previous networked-agent methods and that, despite randomized communication with limited rounds, it can emulate a central critic that uses system-wide observations.","tokens_in":24650,"tokens_out":3848,"duration_ms":40143,"significance":"If the central mechanism were sound, the paper would make a useful contribution to privacy-preserving and communication-limited cooperative MARL: it relaxes full observability assumptions in networked-agent methods and provides an open-source implementation with a systematic benchmark comparison using bootstrap confidence intervals and ablations. The formalization of ND-POMG is clear, and the experiments cover both on-policy and off-policy instantiations. However, the load-bearing step is the team-value consensus approximation in Eq. (15), which is asserted heuristically rather than derived or bounded. The empirical evidence is mixed: team-V consensus helps in some on-policy tasks, but the off-policy ablation shows team-Q consensus hurting performance in LBF. The contribution is therefore better characterized as an empirical heuristic with promising results than as a theoretically grounded networked MARL method.","major_comments":[{"comment":"The central approximation is asserted without proof or error bound. Even in the limit K→∞ with perfect consensus, the consensus target is (1/N)∑_j (r_j + γV(o_j'; ω_j^-)). For this to equal the central TD target r + γV(o'; ω^-), one must assume an additive decomposition V(o; ω) ≈ (1/N)∑_j V(o_j; ω_j) and a common parameter vector ω. Neither condition is stated, checked, or derived, and the finite parameter consensus used in the experiments (K=5 or K=1, C=1, Table 5) does not provide identical critics. The paper's own wording, 'Ideally, the following approximation will hold,' confirms that this is a heuristic, and no experiment verifies the approximation error directly. Because every actor and critic update in Eqs. (iii) and (iv) uses this quantity, the validity of the method rests on an unexamined assumption.","section":"§4.1.3, Eq. (15)"},{"comment":"The team policy gradient is derived for the centralized case in Eq. (10) and then localized by replacing the global state s with the local observation o_i and the team advantage A_θ with A_i. This replacement is described as 'standard practice,' but the paper provides no analysis of the resulting bias. The system dynamics still depend on the joint policy and the joint state, so the local gradient is not the true team policy gradient. If the approximation in Eq. (11) is poor, the actor updates are biased even when the critic is perfectly accurate. The paper should either provide a formal bias bound, a reference that establishes such a bound in this setting, or an ablation that measures the gap between Eq. (10) and Eq. (11) in a tractable environment.","section":"§4.1.2, Eq. (11)"},{"comment":"The off-policy ablation directly contradicts the general claim that team-value consensus emulates a central critic. In all three LBF instances, the DNA group (team-Q consensus plus parameter consensus) performs worse than the DQ group (parameter consensus only), and the paper's own hypothesis is that 'the best Q policy has the agents acting independently.' This is evidence that the team-Q consensus mechanism can inject a systematic bias rather than improve cooperation. The paper should either restrict the emulation claim to the on-policy setting, explain why team-Q consensus fails while team-V consensus succeeds, or provide additional evidence that the failure is an artifact of the specific Q-learning instantiation.","section":"§6.1 and Appendix D, Fig. 7"},{"comment":"The hyperparameters K, I, and C were selected by grid search on three seeds and then evaluated on the same tasks with ten seeds. Because the selection and evaluation use the same environments, the reported maximum average episodic returns may reflect selection bias toward configurations that happened to perform well in the small validation sample. The paper should discuss this limitation and, ideally, validate the selected hyperparameters on held-out tasks or report the variance across the grid search. This concern does not invalidate the empirical results, but it weakens the strength of the 'superior performance' claim.","section":"Appendix C, Table 5"}],"minor_comments":[{"comment":"The summation index in Eq. (15) is inconsistent: the left-hand side is written as a sum over i of (1/N) terms, but the right-hand side should sum over the N agents; using the same symbol i on both sides is confusing. Please rewrite with distinct indices.","section":"§4.1.3, Eq. (15)"},{"comment":"The pseudocode computes y_i^t = r_i^{t+1} + γV(o_{t+1}; ω_i), but the text in Eq. (13) uses the target network parameters ω_i^-; the pseudocode should be updated to match the target-network notation.","section":"Algorithm 1, line 13"},{"comment":"The caption says 'For six tasks the component that improves performance is consensus on the parameters and not team-Q consensus,' but only five panels are shown in the figure; please check the panel count and caption wording.","section":"Appendix D, Fig. 7 caption"},{"comment":"DVA2C is described as 'a model-based algorithm, whereby it has a neural network that estimates the discounted team return.' This phrasing is unclear, since the implementation follows Zhang et al. [38], which is a model-free networked actor-critic method; please clarify what 'model-based' means here.","section":"§5.2, DVA2C baseline"},{"comment":"The abstract claims 'superior performance of DNA-MARL over previous methods,' but in Table 1 DNAQL in MPE Tag (15.77) is far below VDN (23.27) and statistically indistinguishable from INDQL (15.54). The text should qualify the superiority claim in this setting.","section":"Table 1, off-policy MPE Tag"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical study of a heuristic extension of networked actor-critic methods, and the open-source code and benchmark methodology are commendable. The main risk is that the central mechanism (Eq. 15) is not theoretically grounded, and the off-policy ablation already shows a case where the mechanism hurts. I would encourage the authors to either provide a formal justification under explicit assumptions or to reframe the paper as an empirical study of a heuristic, with claims appropriately restricted. The hyperparameter selection procedure is another concern that should be disclosed more prominently."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2501.08778. First, the new mechanism is real: consensus on local TD targets (Eq. 14-15), on top of parameter consensus, is not in the cited Zhang or Chen work, and it does move networked MARL into the partial observability setting. Second, the central approximation is asserted, not derived. The paper says \"Ideally, the following approximation will hold\" and then uses it as the backbone of every update.\n\nWhat is good: the empirical work is honest and fairly complete. The authors build on the Papoudakis benchmark, compare against sensible baselines (DVA2C, PIC, independent learners, CTDE upper bounds), run bootstrap hypothesis tests, report ablations, and release code. In the on-policy A2C variant, DNA-MARL is close to MAA2C in most scenarios and beats DVA2C. That is a credible result. The ablation methodology (DV vs TV vs DNA) is the right way to isolate the contribution's effect.\n\nWhere it is soft: the theory. Eq. (15) claims that averaging local targets yields an approximation of a central critic. Even with perfect consensus, the average of local targets equals (1/N) times the sum of (r_i + gamma V(o_i')). That is only the Bellman target of a central critic if the value function factorizes additively over agents' observations and agents share a common parameter vector. Neither condition is stated or checked. The finite K used (K=5 in A2C, K=1 in Q-learning, with C=1 edge per round) is far from consensus convergence, so agents' critics remain non-identical. The paper's own off-policy ablation (Fig. 7) is consistent with the mechanism failing: team-Q consensus degrades performance relative to parameter consensus alone in LBF. The authors attribute this to the best Q-policy being independent, but it undercuts the general claim that DNA-MARL \"can indeed emulate a central critic.\"\n\nAlso, the hyperparameters K, I, and C are tuned per benchmark via grid search. That is reported transparently, but it means the headline results carry a few free parameters. Not fatal, but worth keeping in mind.\n\nOverall: this is a plausible empirical contribution for researchers building practical decentralized MARL systems, and it deserves a serious referee. It is not yet a well-grounded method. The authors should either provide conditions or error bounds for the team-value approximation, or temper the central claim and present the mechanism as a heuristic that helps in some settings. Given the reproducibility and honest ablation, I would send it to review with a request for theory or a softened claim.","headline":"Useful empirical extension of networked MARL to partial observability, but the team-value consensus mechanism is asserted without theory and the ablation data already shows it can hurt.","tokens_in":25213,"tokens_out":1748,"would_cite":true,"duration_ms":17816,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that decentralized agents under partial observability can approximate a central critic by running consensus on both local value targets and model parameters, and that this outperforms previous networked-agent methods.","keywords":["multi-agent reinforcement learning","partial observability","consensus","decentralized training","actor-critic","deep Q-network","team value function","networked agents"],"falsifier":"On a small task where the true team value can be computed exactly, measure the mean squared error between the consensus target $\\bar{y}_t^i$ from (15) and the true central-critic target as $K$ increases; if DNA-MARL already matches the central critic while this error stays large, then the consensus approximation is not the source of the improvement, and if the error fails to shrink with $K$ while performance degrades toward independent learning, the central claim is refuted.","tokens_in":24132,"feed_emoji":"🤝","tokens_out":7778,"duration_ms":76605,"temperature":0.7,"pith_summary":"The paper sets out to show that cooperative reinforcement learning does not require a central trainer or shared observations: agents that only see their own observations and collect their own rewards can still learn a shared team objective. To that end it introduces the networked dynamic partially observable Markov game and DNA-MARL, a class of algorithms that runs consensus steps on local value targets in addition to consensus on actor and critic parameters. The central claim is that averaging these local targets approximates the team value function, so each agent's policy gradient points toward the team advantage rather than only its own reward. If the claim holds, distributed systems in privacy-sensitive or lossy communication settings can approach the performance of centralized training without ever pooling observations. The paper's experiments on level-based foraging and multi-agent particle tasks show DNA-MARL matching or closely approaching the centralized baselines and outperforming earlier networked-agent methods.","feed_headline":"Consensus on value guesses lets hidden-state agents cooperate","feed_subtitle":"By averaging their own value targets over a switching network, decentralized agents approximate a central critic in benchmark tasks.","key_machinery":"The load-bearing object is the team-value consensus step: agent $i$ turns its local TD target $y_t^i$ into a consensus variable and applies $K$ rounds of distributed averaging with Metropolis weights, which are local weights that guarantee asymptotic average consensus over a switching graph. This is the 'double' in DNA-MARL, because the same consensus operator is also applied to critic parameters and actor parameters, following earlier networked-agent work. The paper's idealized identity (15) asserts that the averaged target equals the team value, which is what turns local actor updates into updates in the direction of the team advantage. The empirical section then treats $K$ as a cooperation dial between independent learning ($K=0$) and full cooperative mode.","core_discovery":"The paper claims that DNA-MARL lets decentralized agents under partial observability approximate a central critic that sees the full system. The mechanism is a second consensus layer: after each episode agents compute their own TD target $y_t^i = r_{t+1}^i + \\gamma V(o_{t+1}^i; \\omega_-^i)$, average these scalars over a randomly switching communication graph for $K$ rounds, and use the averaged target $\\bar{y}_t^i$ in both critic and actor updates. The authors assert, via equation (15), that the consensus average of local targets can stand in for the true team value $V(o;\\omega)$, and they supplement it with periodic consensus on the critic and actor parameters from earlier work. Across six on-policy and six off-policy benchmark tasks, DNA-A2C and DNA-Q-learning generally track the centralized CTDE baselines more closely than prior decentralized methods do, and the ablation study attributes the gain to the team-value consensus. The intended conclusion is that a distributed, private, partially observable learning system can emulate centralized training.","pith_inferences":["A testable consequence the paper does not examine: the bias in (15) should grow with reward variance and network diameter, so the number of consensus rounds needed to match a central critic should scale with both; current results only probe small networks with $K \\in \\{1,5,10\\}$.","The team-target consensus can also be viewed as a variance-reduction mechanism that might help even when agents have nothing to coordinate; comparing DNA-MARL against independent learners that simply average their rewards would separate the cooperative contribution from the noise-reduction contribution.","The failure in the off-policy Tag setting, where DNAQL only matches independent learning, suggests the max operator inside Q-learning targets may amplify consensus bias; swapping team-$V$ consensus for team-$Q$ consensus on identical tasks would localize whether the fragile part is the target type or the consensus step."],"forward_implications":["In privacy-sensitive domains, agents can cooperate without sharing observations or rewards: only scalar value targets and averaged parameters leave the node.","The $K$ consensus rounds give a direct dial between independent learning ($K=0$) and fully cooperative behavior, so a system can tune how much communication budget buys how much cooperation.","Because the updates are agnostic to the base RL algorithm, the same double-averaging recipe transfers to on-policy methods such as TRPO or PPO and to off-policy Q-learning, as demonstrated with A2C and DQN.","If DNA-MARL closes most of the gap to centralized critics in the tested benchmarks, decentralized training with random packet dropout no longer requires a fully observable state or agent-chosen communication, covering applications such as power dispatch, packet routing, and sensor networks."],"supporting_citations":[{"why":"Supplies the critic-parameter consensus baseline (distributed-V) that DNA-MARL extends with target consensus, and the comparison method in experiments.","marker":"[38]"},{"why":"Supplies the actor/policy consensus approach and the homogeneous Markov game setting whose observability assumptions DNA-MARL removes.","marker":"[4]"},{"why":"Supplies the distributed averaging consensus update and Metropolis weights that implement the team-value consensus step.","marker":"[36]"},{"why":"Supplies the benchmark tasks, evaluation protocol, and baseline hyperparameters used in all experiments.","marker":"[24]"},{"why":"Supplies VDN, the centralized value-decomposition baseline that DNAQL is compared against in the off-policy setting.","marker":"[31]"},{"why":"Supplies PIC, the centralized critic used as the performance upper bound representing previous networked-agent methods in off-policy comparisons.","marker":"[17]"},{"why":"Supplies DQN, the single-agent algorithm that the double networked averaging Q-learner extends.","marker":"[19]"},{"why":"Supplies the multi-agent particle environments adapted to partial observability and individual rewards.","marker":"[18]"}],"fun_headline_variants":["Hidden-state agents share value guesses to cooperate","Decentralized agents learn team values via consensus","No central view needed: consensus value learning works","Partial observability? Consensus on value targets helps","Team value consensus drives hidden-state cooperation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method works only if the consensus average of local reward-plus-next-value estimates really approximates the team value function; the paper states this as an ideal approximation without an error bound, and the experiments use only 1 or 5 consensus rounds, so a biased average would feed systematic bias into every actor and critic update.","fun_headline_variants_meta":{"raw":{"variants":["Hidden-state agents share value guesses to cooperate","Decentralized agents learn team values via consensus","No central view needed: consensus value learning works","Partial observability? Consensus on value targets helps","Team value consensus drives hidden-state cooperation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1240,"prompt_tokens":912,"completion_tokens":328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":260}},"tokens_in":528,"tokens_out":328,"duration_ms":3491,"temperature":1.0,"reasoning_tokens":260,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:18:24.319588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a small task where the true team value can be computed exactly, measure the mean squared error between the consensus target $\\bar{y}_t^i$ from (15) and the true central-critic target as $K$ increases; if DNA-MARL already matches the central critic while this error stays large, then the consensus approximation is not the source of the improvement, and if the error fails to shrink with $K$ while performance degrades toward independent learning, the central claim is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the critic-parameter consensus baseline (distributed-V) that DNA-MARL extends with target consensus, and the comparison method in experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the actor/policy consensus approach and the homogeneous Markov game setting whose observability assumptions DNA-MARL removes."},{"cited_title":"Leibo, Karl Tuyls, and Thore Graepel","cited_arxiv_id":null,"evidence_quote":"Supplies VDN, the centralized value-decomposition baseline that DNAQL is compared against in the off-policy setting."},{"cited_title":"Yeh, and Alexander G","cited_arxiv_id":null,"evidence_quote":"Supplies PIC, the centralized critic used as the performance upper bound representing previous networked-agent methods in off-policy comparisons."}],"review_version":1}