{"id":"a7e4646c-db41-4e0f-888f-261948d6b7af","arxiv_id":"2412.14779","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"TAR2 redistributes sparse multi-agent rewards both across time and across agents, but its optimal-policy-preservation proof depends on a trajectory-dependent 'potential' and is not valid.","lead":"A reward redistribution method for multi-agent reinforcement learning, TAR2, densifies sparse team rewards by splitting them across time steps and agents, and claims this reshaping preserves the optimal policy. The paper's proof of that guarantee does not hold up because the shaping potential it builds depends on the final episode return, so the results rest on an invalid theoretical step.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2's potential is trajectory-dependent, so the central optimal-policy-preservation claim does not follow; the paper's proofs also misuse policy-gradient direction claims.","rationale":"The reader's verdict identifies the same core defect: Theorem 2 constructs the potential from the episode return, making it trajectory-dependent, and relies on γ=1. My independent reading of §4.2–4.4 confirms this. The proof's algebra is also confused in the transition to the policy-gradient claim: it conflates scalar weights with direction of gradient vectors and does not properly handle discounted returns. Since the paper's headline contribution is the theoretical guarantee of optimal policy preservation, and that guarantee is not established, the verdict should remain REJECT. The empirical results cover only one SMACLite scenario despite the abstract promising several environments, which further weakens the paper, but the theoretical flaw is the decisive issue.","tokens_in":14618,"tokens_out":1571,"duration_ms":12778,"concrete_test":"Fix a minimal MDP where the episodic reward is nonzero only on a specific trajectory, and define TAR2 weights w_t, w'_{t,i} that depend on the final outcome as in §4.2/§4.3. Check numerically whether the reshaped reward Rω,κ = Rζ + r_global,episodic(τ)·(w'_{t,i} w_t) equals Rζ + Φ(s_{t+1}) − Φ(s_t) for any state-only potential Φ. A direct computation of the potential difference identity on a two-state, two-timestep example (with γ=1 and γ=0.99) should resolve the proof's validity; if the identity fails, the optimal-policy invariance theorem cannot be invoked.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim of the paper is that TAR2 preserves the optimal policy because its reshaped reward is potential-based (Theorem 2, §4.3). The proof fails at the point where it defines the potential. The paper claims Rω,κ = Rζ + γϕ(s_{t+1}) − ϕ(s_t) and asserts this holds with γ=1 for ϕ(s_t) = r_global,episodic(τ)·(Σ_{t'=0}^{t} w'_{t',i} w_{t'}). But this ϕ depends on the entire trajectory through r_global,episodic(τ), including future actions and the episode outcome. Potential-based shaping (Ng 1999, Devlin & Kudenko 2011) requires Φ to be a function of the current state only; otherwise the optimal-policy invariance theorem does not apply. Even if one reinterprets ϕ as depending on the full trajectory, the shaping term r_global,episodic(τ)·(w'_{t,i} w_t) is not a state-dependent potential difference when weights depend on future information. In discounted settings (γ<1), the paper only handles γ=1; the proof explicitly says 'assuming γ=1' and never extends to the discounted case stated in the abstract. Additionally, Proposition 1 (§4.4) claims gradient updates share 'the same direction' but only shows ∇E[r_global,episodic] = (1/(1−ΣM_t))·∇E[Σ r_k,t], where the scalar factor is trajectory-dependent and not per-update; the direction claim is not established, and there are algebraic errors in the derivation (e.g., the step rewriting r_global,episodic(τ) in the second term of the gradient equation). Thus the central theoretical guarantee—optimal policy preservation—is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Temporal-Agent Reward Redistribution (TAR2), a reward redistribution method for cooperative multi-agent reinforcement learning with sparse, episodic global rewards. TAR2 learns temporal weights w_t and agent weights w'_{t,i} to decompose the episode return r_global,episodic(τ) into per-agent, per-timestep rewards, and trains independent single-agent RL algorithms on the redistributed reward. The paper's main advertised contributions are Theorem 2 (§4.3), claiming that the reshaped reward is equivalent to potential-based reward shaping and therefore preserves the optimal policy, and Proposition 1 (§4.4), claiming that policy gradient updates under the redistributed reward have the same direction as under the original reward. Experiments compare TAR2 against episodic, dense, AREL, and IRCR reward baselines in the SMACLite 5m_vs_6m scenario using IPPO and MAPPO.","tokens_in":14972,"tokens_out":10617,"duration_ms":71393,"significance":"If the optimal-policy-preservation claim were valid, TAR2 would offer a valuable practical bridge between single-agent and multi-agent RL: it would densify sparse team rewards without altering the optimal team behavior, and it would let practitioners reuse standard single-agent algorithms. The paper's architecture, which alternates temporal and agent attention over a trajectory, is a reasonable extension of AREL, and the baseline comparison includes natural alternatives. The paper does not ship code or machine-checked proofs, and its central theoretical guarantee is not established: the proof of Theorem 2 defines the shaping potential as a function of the full episode return, which is outside the state-potential class required by the cited invariance theorems, and the argument only treats γ=1. Proposition 1's 'same direction' claim is also unsupported because the derived scaling factor is trajectory-dependent inside the expectation. Because these issues attack the paper's primary claim, the contribution as presented is not sound; the empirical results, limited to a single environment scenario, cannot compensate.","major_comments":[{"comment":"The proof defines ϕ(s_t) = r_global,episodic(τ)(Σ_{t′=0}^{t} w′_{t′,i} w_{t′}), which depends on the entire trajectory through the episode return and through weights evaluated on the full trajectory, including future actions and the final outcome. The potential-based shaping theorem invoked in §3.3 requires a potential Φ: S → R that is a function of the current state only. Because the constructed ϕ is not a state potential, the equality R_{ω,κ} = R_ζ + γϕ(s′) − ϕ(s) is not a potential-based shaping relation, and constructing ϕ after the fact from the same episodic return is circular. The optimal-policy invariance therefore does not follow.","section":"§4.3, Theorem 2 proof"},{"comment":"The proof explicitly sets γ=1 (\"assuming γ = 1\") and never extends the argument to γ<1. Theorem 2 and the abstract claim the result for the general Dec-POMDP with discount factor γ∈[0,1] defined in §3.1. For γ<1 the required shaping term γϕ(s′) − ϕ(s) has no demonstrated correspondence to the redistributed credit w′_{t,i}w_t r_global,episodic(τ), so the theorem is not proven in the discounted setting it states.","section":"§4.3, Theorem 2"},{"comment":"The proof derives ∇_θ E[r_global,episodic] = ∇_θ E[Σ_t r_{k,t}] + ∇_θ E[(Σ_t M_t) r_global,episodic] and then sets δ(τ) = 1 − Σ_t M_t with 0 ≤ δ(τ) ≤ 1. Even if this algebra is correct, ∇_θ E[δ(τ) r_global,episodic] need not point in the same direction as ∇_θ E[r_global,episodic] because δ(τ) is a trajectory-dependent scalar inside the expectation; direction preservation would require δ(τ) to be constant or independent of θ. The paper also obtains Σ_i w′_{t,i}=1 and Σ_t w_t=1 by selecting one particular solution of the conservation equation rather than deriving it from Assumption 2, and the claimed smaller-magnitude property of the update is not shown.","section":"§4.4, Proposition 1"},{"comment":"The experimental section reports only the SMACLite 5m_vs_6m scenario, despite the abstract and the contribution list claiming validation on \"various battle environments\" and \"various environment configurations of Google football.\" The warm-up period of 2000 episodes used for TAR2 (described in §6) is an additional hyperparameter that is not analyzed, and no code or reproducibility details are provided. These omissions make the empirical contribution considerably weaker than the text implies.","section":"§5–§6, Figure 1"}],"minor_comments":[{"comment":"The definition of ϕ(s_t) in the proof of Theorem 2 appears to contain a typo: it writes w′_{t′,i} w′_t where the product should be w′_{t′,i} w_{t′}.","section":"§4.3"},{"comment":"The step from Σ_t (Σ_i w′_{t,i}) w_t = 1 to equations (2) and (3) assumes a specific factorization; there are infinitely many solutions, so this should be presented as a design choice or an additional assumption rather than as a derivation.","section":"§4.2"},{"comment":"The notation F(s_t,a_t,s_{t+1}) in the proof of Theorem 2 is inconsistent with the state-only potential F_i(s,s′) in Theorem 1; the presence of the action argument obscures the requirement that a potential-based shaping reward depend only on states.","section":"§4.3"},{"comment":"The variance bound in §3.4 is derived for a single sample and does not by itself establish that variance grows linearly with the number of agents; the conclusion is stated more strongly than the derivation supports.","section":"§3.4"},{"comment":"The legend in Figure 1 labels the proposed method \"ATRR-agent\" while the text refers to \"TAR2-agent\"; these should be aligned.","section":"Figure 1"},{"comment":"The method is called \"TAR2\" in the body but \"TAR^2\" in the abstract; please standardize the notation.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The central theoretical claim of the paper is not supported by the proof: the potential in Theorem 2 is trajectory-dependent, the argument covers only γ=1, and Proposition 1's gradient-direction conclusion does not follow from the derived trajectory-dependent scaling. These are load-bearing issues, not presentation problems. The empirical evaluation is also far narrower than the abstract claims. If the authors were to remove the invariance and direction-preservation claims and reposition the work purely as an empirical reward-redistribution heuristic, a future submission could be considered, but that would be a substantially different paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the central claim of TAR2—that it preserves the optimal policy because its reward reshaping is potential-based—does not go through. The proof in §4.3 constructs the potential using the full episode return r_global,episodic(τ), making it depend on the entire trajectory, including future actions and the final outcome. That is not a state potential in the sense Ng (1999) and Devlin & Kudenko (2011) require. Without a state-only potential, the invariance guarantee simply does not follow. The proof also only handles γ=1 and never extends to the discounted case advertised in the abstract.\n\nThat said, the paper is not empty. TAR2 makes an explicit design choice that earlier AREL left implicit: it separates temporal and agent attention stages and pins down normalization constraints (∑_i w'_{t,i}=1 and ∑_t w_t=1). That gives a clean, concrete recipe for turning one sparse episodic reward into per-agent per-step rewards. The empirical comparison on the single SMACLite scenario shows TAR2 accelerates IPPO and MAPPO relative to episodic or uniform temporal redistribution, which is a useful data point.\n\nThe other soft spots are proportionate to the theorem. Proposition 1 claims policy gradients \"share the same direction\" but only derives a trajectory-dependent scalar factor in front of the gradient; that does not establish per-update direction preservation, and there are algebraic slips in the derivation. The experiments also cover only one environment (5m_vs_6m), despite the abstract promising several, and no code or data are provided.\n\nWho should read this? Researchers interested in reward redistribution for MARL will find the architectural idea and the normalization trick worth a look, but they should not rely on the optimality guarantee. I would not cite the theorem as proven. This version should be rejected, but the underlying idea is salvageable: if the authors can derive a proper state-based potential—or at least clearly state the restrictive Markovian conditions under which their shaping is valid—there is a solid empirical paper here.\n\nMy recommendation: reject the current version, but give the authors a concrete path to a revised submission.","headline":"TAR2's core optimal-policy-preservation proof fails: the hypothesized potential depends on the whole trajectory, so the paper's main theoretical claim is unsupported, though the empirical recipe is plausible.","tokens_in":15569,"tokens_out":2014,"would_cite":false,"duration_ms":15284,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TAR2 redistributes sparse, episode-end team rewards into per-agent, per-timestep rewards, and the authors prove this leaves the optimal policy unchanged because the reshaping is potential-based.","keywords":["multi-agent reinforcement learning","temporal credit assignment","agent credit assignment","reward redistribution","potential-based reward shaping","sparse rewards","policy invariance"],"falsifier":"Take any environment in which the same state, or the same agent history, can be followed by different episode-end returns; then the formula $\\phi(s_t)=r_{\\text{global,episodic}}(\\tau)\\sum_{t'=0}^t w'_{t',i}w_{t'}$ assigns different values to the same state across episodes, so $\\gamma\\phi(s_{t+1})-\\phi(s_t)$ is not a well-defined state-based potential. A brute-force policy check on such an environment, for instance a two-timestep cooperative task with $\\gamma<1$, would determine whether TAR2's reshaping preserves the optimal policy.","tokens_in":14384,"feed_emoji":"🤖","tokens_out":11739,"duration_ms":89423,"temperature":0.7,"pith_summary":"This paper introduces Temporal-Agent Reward Redistribution (TAR2), a learned reward scheme for cooperative multi-agent systems that only get one team reward at the end of an episode. TAR2 splits that episode return across time steps and across agents, using two stacked attention modules. The authors prove, as their central claim, that the reshaped reward is equivalent to potential-based reward shaping, so any optimal policy under the reshaped reward is also optimal under the original environment reward. If correct, the method would let sparse, long-horizon multi-agent tasks be solved with ordinary single-agent reinforcement learning algorithms without changing what the team should do. Experiments on a StarCraft-like battle scenario report faster and more stable learning than episodic rewards or existing temporal-only redistribution.","feed_headline":"Sparse multi-agent rewards get dense without changing optimal policy","feed_subtitle":"A learned decomposition splits the team reward across time and agents, letting single-agent RL solve cooperative tasks.","key_machinery":"The load-bearing object is the reward redistribution function $r_{i,t}=w'_{t,i} w_t r_{\\text{global,episodic}}(\\tau)$, with temporal weights $w_t$ produced by a temporal attention module and agent weights $w'_{t,i}$ produced by an agent attention module; the normalizations $\\sum_t w_t=1$ and $\\sum_i w'_{t,i}=1$ guarantee the per-agent, per-timestep rewards sum back to the episode's total return. The argument then tries to place this additive term inside the potential-based shaping identity $F(s_t,s_{t+1})=\\gamma\\phi(s_{t+1})-\\phi(s_t)$, with the cumulative weighted return acting as the potential, so that the known policy-invariance theorem for potential-based shaping can be invoked. The same weights also define the $\\delta(\\tau)$ factor in Proposition 1, which is what aligns the policy-gradient directions.","core_discovery":"On the paper's own terms, the discovery is that the two-step decomposition $r_{i,t} = w'_{t,i} w_t r_{\\text{global,episodic}}(\\tau)$, where the temporal weights $w_t$ and agent weights $w'_{t,i}$ each sum to one, produces a reward function $R_{\\omega,\\kappa}(s_t,a_t,s_{t+1}) = R_\\zeta(s_t,a_t,s_{t+1}) + w'_{t,i} w_t r_{\\text{global,episodic}}(\\tau)$. Theorem 2 claims that, for $\\gamma=1$, this additive term equals $\\gamma\\phi(s_{t+1})-\\phi(s_t)$ with $\\phi(s_t)=r_{\\text{global,episodic}}(\\tau)\\sum_{t'=0}^{t} w'_{t',i} w_{t'}$, putting the redistributed reward in the potential-based shaping form whose policy-invariance property is proved in the paper's Section 3.3. A second result, Proposition 1, claims that the policy-gradient update for any agent under TAR2 has the same direction as under the original reward, scaled by a nonnegative factor $\\delta(\\tau)\\in[0,1]$, so the sequence of policy updates stays aligned. Together these results are the paper's reason for claiming that TAR2 can densify sparse global rewards without sacrificing optimality.","pith_inferences":["Because TAR2's weights are learned from the completed trajectory, the method is inherently a hindsight credit-assignment scheme; a natural extension would be to train the redistribution function to be causal, using only information available up to time $t$, and measure how much of the speed-up is lost.","The normalization $\\sum_i w'_{t,i}=1$ gives each timestep a soft partition of credit across agents, which invites comparing the learned attention weights to Shapley-value or difference-reward attributions in the same environments.","The same temporal-agent decomposition could be applied to other trajectory-feedback problems, such as single-agent sparse-reward control or settings with several sparse outcome signals, by treating each signal as a separate episodic return."],"forward_implications":["Sparse, episode-end rewards in cooperative MARL can be replaced by dense per-agent, per-timestep rewards without changing the optimal team policy, if the theorem holds.","Off-the-shelf single-agent algorithms such as IQL, IAC, and IPPO can be applied directly to multi-agent tasks, because the credit-assignment burden is moved into the reward model rather than the learning algorithm.","The claimed direction-preservation result implies that policy updates under TAR2 and under the original reward move toward the same optimum, so reward redistribution should not conflict with convergence.","The empirical comparison in the paper indicates that TAR2-trained agents learn faster and more stably than episodic, IRCR, or AREL temporal reward baselines in the 5m_vs_6m SMACLite scenario."],"supporting_citations":[{"why":"Supplies the potential-based reward shaping theorem that the proof of Theorem 2 invokes.","marker":"Ng (1999)"},{"why":"Extends potential-based shaping to multi-agent systems, forming the invariance condition stated in the paper's Theorem 1.","marker":"Devlin & Kudenko (2011)"},{"why":"Also cited for multi-agent Nash equilibrium invariance under potential-based reward transformations.","marker":"Xiaosong Lu (2011)"},{"why":"Provides the AREL temporal attention reward-redistribution architecture that TAR2 extends with agent-level attention.","marker":"Xiao et al. (2022)"},{"why":"Supplies the sum-decomposable return structure that Assumption 1 borrows for the redistribution model.","marker":"Ren et al. (2021)"},{"why":"Provides the SMACLite benchmark used for the empirical evaluation.","marker":"Michalski et al. (2023)"}],"fun_headline_variants":["Sparse rewards turn dense without shifting optimal policy","Credit assignment that densifies rewards and proves policy invariance","TAR2: Dense rewards from sparse global signals, optimality preserved","Reward redistribution across time and agents keeps optimal policy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof's load-bearing premise is that the shaping term built from the episode's final total reward can be treated as a potential that depends only on the current state, and that the discount factor may be set to one; if either fails, the optimal-policy guarantee does not follow.","fun_headline_variants_meta":{"raw":{"variants":["Sparse rewards turn dense without shifting optimal policy","Credit assignment that densifies rewards and proves policy invariance","TAR2: Dense rewards from sparse global signals, optimality preserved","Reward redistribution across time and agents keeps optimal policy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1655,"prompt_tokens":999,"completion_tokens":656,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":589}},"tokens_in":615,"tokens_out":656,"duration_ms":5951,"temperature":1.0,"reasoning_tokens":589,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:55:59.328892+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any environment in which the same state, or the same agent history, can be followed by different episode-end returns; then the formula $\\phi(s_t)=r_{\\text{global,episodic}}(\\tau)\\sum_{t'=0}^t w'_{t',i}w_{t'}$ assigns different values to the same state across episodes, so $\\gamma\\phi(s_{t+1})-\\phi(s_t)$ is not a well-defined state-based potential. A brute-force policy check on such an environment, for instance a two-timestep cooperative task with $\\gamma<1$, would determine whether TAR2's reshaping preserves the optimal policy.","supporting_citations":[{"cited_title":"Theoretical considerations of potential-based reward shaping for multi-agent systems","cited_arxiv_id":null,"evidence_quote":"Extends potential-based shaping to multi-agent systems, forming the invariance condition stated in the paper's Theorem 1."}],"review_version":1}