{"id":"6cc8f718-45f1-40c8-a34b-5c368c3b00fb","arxiv_id":"2508.14131","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Scaling team members' rewards when multiple agents get positive rewards improves MADDPG's empirical team and individual reward in one MPE environment.","lead":"A new multi-agent reinforcement learning algorithm extends MADDPG by multiplying rewards when several teammates receive positive rewards at the same step. In a single PettingZoo environment, the authors report higher team and individual rewards compared to MADDPG.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The defining formula for φ_i (Eq. 3) is missing, making the core algorithm underspecified and the central claim untestable as submitted.","rationale":"The reader's REJECT verdict is well-supported, but their 'weakest_assumption' targeted the reward-to-cooperation mapping. I agree that assumption is fragile, but the more fundamental defect is that the core update is not actually defined: Eq. (3) is referenced in both the text and Algorithm 1 yet the displayed formula is empty. This makes the algorithm non-reproducible and the experimental comparison uninterpretable. The reader did list the missing equation among red flags, so there is partial agreement. I would not change the verdict—rejection is correct—but the concrete test focuses on recovering the formula, because no meaningful evaluation can occur before that is resolved. Even after recovery, the single-run, no-seed experimental design would need substantial strengthening. The paper may contain a plausible heuristic, but as submitted it does not demonstrate its central claim.","tokens_in":4510,"tokens_out":3608,"duration_ms":40124,"concrete_test":"Contact the authors or inspect any supplementary code/repository to obtain the exact definition of Eq. (3), including the condition for φ_i > 1, the value of φ_i otherwise, and the treatment of boundary cases such as k = L. If no complete formula can be produced, the central claim is untestable and the manuscript should remain rejected. If the formula is supplied, implement the algorithm with those precise rules in the same PettingZoo MPE environment, run at least 10 independent seeds with identical hyperparameters (L=1, φ=2), and report mean ± standard deviation of final episode reward for each team and agent. The central claim holds only if the improvement over a MADDPG baseline is consistent across seeds and exceeds the inter-run variance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the algorithm's defining formula for φ_i is absent. In Section 3, after Eq. (2), the text states 'Then, we can calculate φ_i as follows:' and then displays '((3)' with no equation body; Algorithm 1 line 12 instructs 'Calculate ϕ_i according to the formula (3)' but no such formula exists anywhere in the manuscript. The central claim—that this algorithm yields better team and individual rewards—depends entirely on this multiplier, including its boundary conditions (k > L vs. k ≥ L), its default value when the condition fails (1? 0? φ?), and the sign convention for 'positive rewards.' Without it, the method is underspecified and the PettingZoo experiments cannot be reproduced or even formally interpreted. The empirical comparison further rests on a single unseeded run with no error bars and no identification of the exact MPE scenario, so even the intended algorithm lacks statistical support. This is not a stylistic issue: every reported improvement is uninterpretable until Eq. (3) is supplied.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an extension of MADDPG for cooperative/competitive multi-agent environments. The algorithm introduces a per-agent multiplier phi_i in the critic's TD target: y = phi_i r_i + gamma Q'_i, where phi_i is meant to amplify the reward when a sufficient number of teammates simultaneously receive positive rewards. The authors claim that this reward shaping encourages cooperation and yields higher team and individual rewards than MADDPG in a PettingZoo Multi-Particle Environment with two teams. The manuscript includes one experiment comparing the proposed method to MADDPG over 25,000 episodes and reports qualitatively that the proposed algorithm achieves higher red-team and individual rewards.","tokens_in":4717,"tokens_out":2919,"duration_ms":33606,"significance":"If the method were fully specified and the empirical claims were rigorously supported, the contribution would be a simple, potentially useful heuristic for reward shaping in cooperative MARL. The paper does not provide a theoretical derivation, machine-checked proofs, or reproducible code; its value would rest entirely on empirical demonstration. As submitted, however, the core algorithm is underspecified and the experimental evidence is anecdotal, so the significance cannot be assessed.","major_comments":[{"comment":"The defining formula for phi_i is missing. The text states 'Then, we can calculate φi as follows:' followed by an empty display '((3)'. Algorithm 1 line 12 also instructs the reader to 'Calculate ϕ_i according to the formula (3)', but no such formula appears anywhere. This is a load-bearing omission: the method is undefined without knowing the default value of phi_i when the cooperation condition fails, whether the trigger is k > L or k >= L, and how 'positive rewards' are determined. The reported improvement cannot be reproduced or even formally interpreted.","section":"Section 3, Eq. (3)"},{"comment":"The empirical comparison is statistically unsupported. The paper does not report the number of random seeds, error bars, confidence intervals, or any statistical significance test. Only one environment is used, and its exact PettingZoo scenario is not identified. Figures 2-4 appear to be placeholders or are not legible in the submission, and no numerical reward values are given. Consequently, the central claim that the algorithm yields higher team and individual rewards is not established.","section":"Section 4, Experiments"},{"comment":"No ablation or sensitivity analysis is provided for the two introduced hyperparameters L and phi. The paper fixes L=1 and phi=2 without justification. Since the entire method reduces to the rule 'if at least L teammates have positive rewards, multiply the reward by phi,' the behavior of the algorithm under other settings is unknown. This is particularly important because the paper's motivation—that positive teammate rewards identify cooperative episodes—is asserted rather than demonstrated.","section":"Section 4, hyperparameters L and phi"}],"minor_comments":[{"comment":"The manuscript contains numerous typos and grammatical errors, e.g., 'higer' instead of 'higher', 'an cooperative behavior' instead of 'a cooperative behavior', and 'teach agent' instead of 'each agent'. These should be corrected.","section":"Abstract and throughout"},{"comment":"Several equations are missing or garbled. For example, the Markov Game tuple is incomplete, and the Q-function definition is not fully written out. The notation phi_i versus φ_i is inconsistent.","section":"Section 2, notation"},{"comment":"The text says 'We also compare each team's reward in Fig 1,' but Figure 1 shows the MPE environment; the comparison plots appear to be Figures 2 and 3. The figure numbering should be corrected.","section":"Section 4, figure references"},{"comment":"The citation to MADDPG is inconsistent: the background cites [5] for MADDPG, but [5] is a paper on peer-to-peer energy trading, while [7] is the original Lowe et al. paper. Several references are also incomplete or incorrectly formatted.","section":"References"},{"comment":"The introduction mentions that PPO is employed, but PPO is never used in the methodology or experiments. This is misleading and should be removed or clarified.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is in a rough draft state: the central equation is missing, figures are not properly embedded, and the empirical evaluation is far below the standard for the journal. The proposed idea may have merit, but as submitted it is not verifiable. If the authors can supply the missing formula and a rigorous experimental study with multiple seeds and ablations, a resubmission could be reconsidered; otherwise rejection is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea is a small, reasonable twist on reward shaping for MADDPG—scale the reward by a constant phi when more than L teammates have positive rewards. That is the whole contribution, and it is modest. What the paper does well: the motivation is stated plainly, the algorithm sketch is simple to grasp, and the authors are honest that this is an incremental improvement over MADDPG.\n\nThe problems are unfortunately load-bearing. The defining formula for phi_i, cited as (3) throughout Section 3 and Algorithm 1, is simply absent from the text. Your reader caught this: the text says “we can calculate φi as follows:” and then shows “((3)” with no body. So the method cannot be implemented or verified. The prose around it suggests a piecewise definition—phi > 1 when k > L, presumably 1 otherwise—but “presumably” is not a definition. That alone is enough to reject.\n\nThe experiments do not rescue it. One PettingZoo MPE environment, one run, no error bars, no seeds, no ablations over the two hyperparameters, no comparison with any existing reward-shaping baseline, and the specific environment variant is not precisely identified. The figures are hard to parse and the claimed improvement may be within run-to-run variance. There are also citation oddities: MADDPG is attributed to an energy-trading application paper rather than Lowe et al., and PPO is mentioned but not used. These don’t directly sink the argument but they signal carelessness.\n\nOn the positive side, the basic intuition—cooperation is more likely when several teammates get positive rewards simultaneously—is reasonable, and the modification is simple enough that it might work in some settings. But as submitted, there is no valid demonstration. The paper is not ready for peer review. The fix is straightforward: supply the missing equation, run multiple seeds with proper statistics, add ablations, and compare to standard reward-shaping baselines. If those come out positive, it could be a marginal but honest contribution. Right now it’s a promising sketch, not a paper.\n\nFor your decision: I wouldn’t send this to reviewers as is. The missing equation makes the central claim unverifiable, and the empirical case is statistically void. If the authors resubmit with the method fully specified and the experiments redone, it would be worth a look.","headline":"A simple reward-scaling heuristic on top of MADDPG, but the paper is missing the equation that defines the scale factor, so the method cannot be reproduced and the single unseeded experiment cannot carry the claim.","tokens_in":5198,"tokens_out":2773,"would_cite":false,"duration_ms":28248,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Amplifying rewards when several teammates score positive at once improves both team and individual returns over the standard MADDPG baseline.","keywords":["multi-agent reinforcement learning","cooperation","MADDPG","reward weighting","critic target modification","team reward","mixed cooperative-competitive environments","multi-particle environments"],"falsifier":"Run the same algorithm on a two-agent cooperative task where one agent must sacrifice its own reward to give its teammate a positive reward. Under the paper's rule with $L=1$, the sacrificing agent's transition has $k=1$ (only the teammate's reward is positive), so the boosting applies to the teammate but not to the sacrificer; if the modified algorithm still beats MADDPG, the positive-reward count is not the active mechanism, and if it fails, the reward-coincidence mechanism is confirmed as essential.","tokens_in":4386,"feed_emoji":"🤝","tokens_out":13291,"duration_ms":129864,"temperature":0.7,"pith_summary":"The paper tries to show that a small change to MADDPG—a standard multi-agent actor-critic algorithm—makes agents cooperate better in a mixed cooperative–competitive game. In the critic target $y = \\varphi_i r_i + \\gamma Q'_i$, the paper replaces the recorded reward $r_i$ with an amplified version whenever enough agents on agent $i$'s team receive positive rewards at the same time; in the experiments the multiplier is set to $\\varphi_i = 2$ when at least $L=1$ teammate reward is positive. The authors' rationale is that positive rewards on most cooperating agents are the signature of cooperative behavior, so boosting those transitions reinforces cooperation. On a two-team multi-particle environment with four chasers, two evaders, and three obstacles, the modified algorithm reports higher team reward for the chasing team and higher individual rewards for each chaser than the unmodified baseline, with the evaders' reward nearly unchanged.","feed_headline":"Twofold reward boost lifts team and agent scores","feed_subtitle":"Amplifying rewards when several teammates score positive at once trains a better cooperative policy than the baseline.","key_machinery":"The central mechanism is the cooperation-reward multiplier $\\varphi_i$ inserted into the TD target of the MADDPG critic. It is a step function: $\\varphi_i = \\phi$ (with $\\phi = 2$) when the count $k$ of positive rewards among the $m$ agents on agent $i$'s team exceeds $L$ (with $L=1$), and $\\varphi_i = 1$ otherwise. This single parameter does the work of the whole proposal—identifying cooperative transitions by reward coincidence and strengthening their learning signal by reward amplification—while leaving MADDPG's actor-critic structure, experience replay, target networks, and decentralized execution untouched.","core_discovery":"The central claim is that cooperative behavior in multi-agent reinforcement learning can be detected by counting positive rewards within a team and encouraged by scaling the reward that enters the TD target. For agent $i$, the loss is $L(\\theta_i) = \\mathbb{E}_{x,a,r,x'}\\left[(Q^{\\mu}_i(x,a_1,\\ldots,a_N)-y)^2\\right]$ with $y = \\varphi_i r_i + \\gamma Q^{\\mu'}_i(x',a'_1,\\ldots,a'_N)$. The multiplier $\\varphi_i$ is determined by the number $k$ of positive rewards among the $m$ agents on $i$'s team: when $k$ exceeds the threshold $L$, $\\varphi_i$ becomes the hyperparameter $\\phi$ (chosen as 2), and otherwise it stays 1. The authors state this motivation explicitly: they assume that when agents e","pith_inferences":["A natural extension is to replace the binary threshold $k>L$ with a graded multiplier such as scaling with the fraction of positive teammate rewards, which would test whether the discontinuity is necessary for the reported gain or merely an artifact of the chosen hyperparameters.","The same positive-reward-count rule could be layered onto other centralized-critic MARL algorithms, not only MADDPG, whenever the critic target exposes the per-agent reward; that generalization is not tested in the paper.","The paper's own motivation implies a boundary condition: if cooperation often gives the acting agent zero or delayed reward while rewarding only the teammate, the count $k$ will not fire on the right transitions, so the method should be stress-tested in such asymmetric-payoff tasks.","The observed convergence of per-agent reward curves suggests behavioral convergence, and measuring policy entropy or action agreement across agents would tell whether the agents truly specialize or become interchangeable—something the paper does not examine."],"forward_implications":["For the chasing team, the modified algorithm reaches a higher total reward than MADDPG over 25,000 episodes, while the evading team's total reward is essentially the same.","Each red agent's individual reward is higher under the modified algorithm, and the per-agent reward curves become similar late in training, pointing to a shared cooperative policy.","The change is a one-line modification to the critic target, so the same default MADDPG settings and hyperparameters can be kept except for the new pair $(L, \\phi)$.","Higher individual reward accompanies higher team reward in this environment, so the encouragement does not appear to force agents to sacrifice personal return.","The result supports the paper's claim that amplifying rewards in episodes identified as cooperative improves both team-level and individual-level outcomes."],"supporting_citations":[{"why":"Supplies the MADDPG algorithm and notation for the critic and actor updates; the modified TD target is written in this notation.","marker":"[7]"},{"why":"Cited as a MADDPG algorithm for multi-agent control, the starting point for the proposed improvement.","marker":"[5]"},{"why":"Establishes the independent-versus-cooperative agents framing that motivates identifying and rewarding cooperative behavior.","marker":"[1]"}],"fun_headline_variants":["Team rewards scaled up when agents cooperate","Cooperation-triggered reward boost beats MADDPG","Multiplier on team success lifts agent scores","Rewarding joint positive moves improves agents","Detect teamwork, double rewards for better policies"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that cooperative behavior reliably shows up as simultaneous positive rewards for most agents on a team, so counting positive rewards is a dependable way to identify the transitions worth amplifying; if a helpful action usually rewards only one teammate or rewards arrive late, the multiplier will fire on the wrong transitions and the reported gain would not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Team rewards scaled up when agents cooperate","Cooperation-triggered reward boost beats MADDPG","Multiplier on team success lifts agent scores","Rewarding joint positive moves improves agents","Detect teamwork, double rewards for better policies"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000134,"raw_usage":{"total_tokens":928,"prompt_tokens":650,"completion_tokens":278,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":394,"completion_tokens_details":{"reasoning_tokens":219}},"tokens_in":394,"tokens_out":278,"duration_ms":3889,"temperature":1.0,"reasoning_tokens":219,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:56:44.124742+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same algorithm on a two-agent cooperative task where one agent must sacrifice its own reward to give its teammate a positive reward. Under the paper's rule with $L=1$, the sacrificing agent's transition has $k=1$ (only the teammate's reward is positive), so the boosting applies to the teammate but not to the sacrificer; if the modified algorithm still beats MADDPG, the positive-reward count is not the active mechanism, and if it fails, the reward-coincidence mechanism is confirmed as essential.","supporting_citations":[{"cited_title":"Multi-agent actor-critic for mixed cooperative- competitive environments[J]","cited_arxiv_id":null,"evidence_quote":"Supplies the MADDPG algorithm and notation for the critic and actor updates; the modified TD target is written in this notation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited as a MADDPG algorithm for multi-agent control, the starting point for the proposed improvement."},{"cited_title":"cooperative agents","cited_arxiv_id":null,"evidence_quote":"Establishes the independent-versus-cooperative agents framing that motivates identifying and rewarding cooperative behavior."}],"review_version":1}