Pith. sign in

REVIEW 4 major objections 6 minor 53 references

Agent-Temporal Credit Assignment for Optimal Policy Preservation in Sparse Multi-Agent Reinforcement Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TAR2 redistributes sparse, episode-end team rewards into per-agent, per-timestep rewards, and the authors prove this leaves the optimal policy unchanged because the reshaping is potential-based.

desk verdict TAR2's core optimal-policy-preservation proof fails: the hypothesized potential depends on the whole trajectory, so the paper's main theoretical claim is unsupported, though the empirical recipe is plausible. read the letter →

arxiv 2412.14779 v1 pith:6RHDQHGY submitted 2024-12-19 cs.MA cs.AIcs.GTcs.LGcs.RO

Aditya Kapoor , Sushant Swamy , Kale-ab Tessera , Mayank Baranwal , Mingfei Sun , Harshad Khadilkar , Stefano V. Albrecht This is my paper · ORCID
classification cs.MAcs.AIcs.GTcs.LGcs.RO
keywords multi-agentreinforcementlearningtemporalcreditassignmentagentrewardredistributionpotential-basedshapingsparserewardspolicyinvariance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Temporal-Agent Reward Redistribution (TAR2), a learned reward scheme for cooperative multi-agent systems that only get one team reward at the end of an episode. TAR2 splits that episode return across time steps and across agents, using two stacked attention modules. The authors prove, as their central claim, that the reshaped reward is equivalent to potential-based reward shaping, so any optimal policy under the reshaped reward is also optimal under the original environment reward. If correct, the method would let sparse, long-horizon multi-agent tasks be solved with ordinary single-agent reinforcement learning algorithms without changing what the team should do. Experiments on a StarCraft-like battle scenario report faster and more stable learning than episodic rewards or existing temporal-only redistribution.

What carries the argument

The load-bearing object is the reward redistribution function $r_{i,t}=w'_{t,i} w_t r_{\text{global,episodic}}(\tau)$, with temporal weights $w_t$ produced by a temporal attention module and agent weights $w'_{t,i}$ produced by an agent attention module; the normalizations $\sum_t w_t=1$ and $\sum_i w'_{t,i}=1$ guarantee the per-agent, per-timestep rewards sum back to the episode's total return. The argument then tries to place this additive term inside the potential-based shaping identity $F(s_t,s_{t+1})=\gamma\phi(s_{t+1})-\phi(s_t)$, with the cumulative weighted return acting as the potential, so that the known policy-invariance theorem for potential-based shaping can be invoked. The same weights also define the $\delta(\tau)$ factor in Proposition 1, which is what aligns the policy-gradient directions.

What would settle it

Take any environment in which the same state, or the same agent history, can be followed by different episode-end returns; then the formula $\phi(s_t)=r_{\text{global,episodic}}(\tau)\sum_{t'=0}^t w'_{t',i}w_{t'}$ assigns different values to the same state across episodes, so $\gamma\phi(s_{t+1})-\phi(s_t)$ is not a well-defined state-based potential. A brute-force policy check on such an environment, for instance a two-timestep cooperative task with $\gamma<1$, would determine whether TAR2's reshaping preserves the optimal policy.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the two-step decomposition $r_{i,t} = w'_{t,i} w_t r_{\text{global,episodic}}(\tau)$, where the temporal weights $w_t$ and agent weights $w'_{t,i}$ each sum to one, produces a reward function $R_{\omega,\kappa}(s_t,a_t,s_{t+1}) = R_\zeta(s_t,a_t,s_{t+1}) + w'_{t,i} w_t r_{\text{global,episodic}}(\tau)$. Theorem 2 claims that, for $\gamma=1$, this additive term equals $\gamma\phi(s_{t+1})-\phi(s_t)$ with $\phi(s_t)=r_{\text{global,episodic}}(\tau)\sum_{t'=0}^{t} w'_{t',i} w_{t'}$, putting the redistributed reward in the potential-based shaping form whose policy-invariance property is proved in the paper's Section 3.3. A second result, Proposition 1, claims that the policy-gradient update for any agent under TAR2 has the same direction as under the original reward, scaled by a nonnegative factor $\delta(\tau)\in[0,1]$, so the sequence of policy updates stays aligned. Together these results are the paper's reason for claiming that TAR2 can densify sparse global rewards without sacrificing optimality.

Load-bearing premise

The proof's load-bearing premise is that the shaping term built from the episode's final total reward can be treated as a potential that depends only on the current state, and that the discount factor may be set to one; if either fails, the optimal-policy guarantee does not follow.

Editorial extensions

If this is right

  • Sparse, episode-end rewards in cooperative MARL can be replaced by dense per-agent, per-timestep rewards without changing the optimal team policy, if the theorem holds.
  • Off-the-shelf single-agent algorithms such as IQL, IAC, and IPPO can be applied directly to multi-agent tasks, because the credit-assignment burden is moved into the reward model rather than the learning algorithm.
  • The claimed direction-preservation result implies that policy updates under TAR2 and under the original reward move toward the same optimum, so reward redistribution should not conflict with convergence.
  • The empirical comparison in the paper indicates that TAR2-trained agents learn faster and more stably than episodic, IRCR, or AREL temporal reward baselines in the 5m_vs_6m SMACLite scenario.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because TAR2's weights are learned from the completed trajectory, the method is inherently a hindsight credit-assignment scheme; a natural extension would be to train the redistribution function to be causal, using only information available up to time $t$, and measure how much of the speed-up is lost.
  • The normalization $\sum_i w'_{t,i}=1$ gives each timestep a soft partition of credit across agents, which invites comparing the learned attention weights to Shapley-value or difference-reward attributions in the same environments.
  • The same temporal-agent decomposition could be applied to other trajectory-feedback problems, such as single-agent sparse-reward control or settings with several sparse outcome signals, by treating each signal as a separate episodic return.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Temporal-Agent Reward Redistribution (TAR2), a reward redistribution method for cooperative multi-agent reinforcement learning with sparse, episodic global rewards. TAR2 learns temporal weights w_t and agent weights w'_{t,i} to decompose the episode return r_global,episodic(τ) into per-agent, per-timestep rewards, and trains independent single-agent RL algorithms on the redistributed reward. The paper's main advertised contributions are Theorem 2 (§4.3), claiming that the reshaped reward is equivalent to potential-based reward shaping and therefore preserves the optimal policy, and Proposition 1 (§4.4), claiming that policy gradient updates under the redistributed reward have the same direction as under the original reward. Experiments compare TAR2 against episodic, dense, AREL, and IRCR reward baselines in the SMACLite 5m_vs_6m scenario using IPPO and MAPPO.

Significance. If the optimal-policy-preservation claim were valid, TAR2 would offer a valuable practical bridge between single-agent and multi-agent RL: it would densify sparse team rewards without altering the optimal team behavior, and it would let practitioners reuse standard single-agent algorithms. The paper's architecture, which alternates temporal and agent attention over a trajectory, is a reasonable extension of AREL, and the baseline comparison includes natural alternatives. The paper does not ship code or machine-checked proofs, and its central theoretical guarantee is not established: the proof of Theorem 2 defines the shaping potential as a function of the full episode return, which is outside the state-potential class required by the cited invariance theorems, and the argument only treats γ=1. Proposition 1's 'same direction' claim is also unsupported because the derived scaling factor is trajectory-dependent inside the expectation. Because these issues attack the paper's primary claim, the contribution as presented is not sound; the empirical results, limited to a single environment scenario, cannot compensate.

major comments (4)
  1. [§4.3, Theorem 2 proof] The proof defines ϕ(s_t) = r_global,episodic(τ)(Σ_{t′=0}^{t} w′_{t′,i} w_{t′}), which depends on the entire trajectory through the episode return and through weights evaluated on the full trajectory, including future actions and the final outcome. The potential-based shaping theorem invoked in §3.3 requires a potential Φ: S → R that is a function of the current state only. Because the constructed ϕ is not a state potential, the equality R_{ω,κ} = R_ζ + γϕ(s′) − ϕ(s) is not a potential-based shaping relation, and constructing ϕ after the fact from the same episodic return is circular. The optimal-policy invariance therefore does not follow.
  2. [§4.3, Theorem 2] The proof explicitly sets γ=1 ("assuming γ = 1") and never extends the argument to γ<1. Theorem 2 and the abstract claim the result for the general Dec-POMDP with discount factor γ∈[0,1] defined in §3.1. For γ<1 the required shaping term γϕ(s′) − ϕ(s) has no demonstrated correspondence to the redistributed credit w′_{t,i}w_t r_global,episodic(τ), so the theorem is not proven in the discounted setting it states.
  3. [§4.4, Proposition 1] The proof derives ∇_θ E[r_global,episodic] = ∇_θ E[Σ_t r_{k,t}] + ∇_θ E[(Σ_t M_t) r_global,episodic] and then sets δ(τ) = 1 − Σ_t M_t with 0 ≤ δ(τ) ≤ 1. Even if this algebra is correct, ∇_θ E[δ(τ) r_global,episodic] need not point in the same direction as ∇_θ E[r_global,episodic] because δ(τ) is a trajectory-dependent scalar inside the expectation; direction preservation would require δ(τ) to be constant or independent of θ. The paper also obtains Σ_i w′_{t,i}=1 and Σ_t w_t=1 by selecting one particular solution of the conservation equation rather than deriving it from Assumption 2, and the claimed smaller-magnitude property of the update is not shown.
  4. [§5–§6, Figure 1] The experimental section reports only the SMACLite 5m_vs_6m scenario, despite the abstract and the contribution list claiming validation on "various battle environments" and "various environment configurations of Google football." The warm-up period of 2000 episodes used for TAR2 (described in §6) is an additional hyperparameter that is not analyzed, and no code or reproducibility details are provided. These omissions make the empirical contribution considerably weaker than the text implies.
minor comments (6)
  1. [§4.3] The definition of ϕ(s_t) in the proof of Theorem 2 appears to contain a typo: it writes w′_{t′,i} w′_t where the product should be w′_{t′,i} w_{t′}.
  2. [§4.2] The step from Σ_t (Σ_i w′_{t,i}) w_t = 1 to equations (2) and (3) assumes a specific factorization; there are infinitely many solutions, so this should be presented as a design choice or an additional assumption rather than as a derivation.
  3. [§4.3] The notation F(s_t,a_t,s_{t+1}) in the proof of Theorem 2 is inconsistent with the state-only potential F_i(s,s′) in Theorem 1; the presence of the action argument obscures the requirement that a potential-based shaping reward depend only on states.
  4. [§3.4] The variance bound in §3.4 is derived for a single sample and does not by itself establish that variance grows linearly with the number of agents; the conclusion is stated more strongly than the derivation supports.
  5. [Figure 1] The legend in Figure 1 labels the proposed method "ATRR-agent" while the text refers to "TAR2-agent"; these should be aligned.
  6. [Throughout] The method is called "TAR2" in the body but "TAR^2" in the abstract; please standardize the notation.

Circularity Check

1 steps flagged · score 7.0 of 10

Theorem 2's potential is defined from the episodic return itself, so the optimal-policy-preservation proof holds only by construction; the paper also leaves γ=1 unextended and Proposition 1 treats a trajectory-dependent scalar as a constant multiplier.

  1. self definitional [Section 4.3, proof of Theorem 2 (after Eq. (4))]
    "It is therefore sufficient to show that the equation (4) takes the form Rω,κ(st,at,st+1) = Rζ(st,at,st+1) + γϕ(st+1)− ϕ(st). Comparing this format to equation (4), assuming γ = 1 we arrive at ϕ(st+1)− ϕ(st) = w′t,iwtrglobal,episodic(τ). This relation holds for ϕ(st) = rglobal,episodic(τ)(∑t t′=0 w′t′,iw′t)"

    The candidate potential ϕ(st) is defined in terms of rglobal,episodic(τ), the full-trajectory episodic return that includes future actions and the final outcome. The potential-shaping invariance theorem invoked in Section 3.3 (Ng 1999; Devlin & Kudenko 2011) requires a genuine state potential Φ:S→R, not a quantity depending on the whole trajectory. The displayed identity is manufactured by setting ϕ to a cumulative sum of the same episodic return times the redistribution weights; with that definition, every trajectory-dependent redistribution can be written as a 'potential difference'.

full rationale

The central circularity is in Theorem 2: the proof exhibits a potential only by defining it from the very episodic return that the redistribution is supposed to preserve, and the resulting ϕ is not a state-only function, so the invoked invariance theorem does not apply. This makes the headline theoretical claim true only by construction. The paper also restricts the proof to γ=1 ('assuming γ=1') without extending to the discounted Dec-POMDP setting; separately, Proposition 1's 'same direction' claim treats the trajectory-dependent scalar δ(τ) as if it were a constant positive multiplier of the gradient, which is an algebraic gap rather than a further circularity. No load-bearing self-citation chain appears: citations to AREL and SMACLite are architectural/environmental, and the Freed et al. self-citation is not used to justify the central claim. The empirical comparison is self-contained against baselines, but the theoretical guarantee that anchors the paper's contribution is circular as written, so the score is 7.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper introduces a learned redistribution function with two levels of weights and constrains their sums to 1; the central theoretical proof additionally assumes a future-dependent potential and γ=1. The only hand-tuned numeric hyperparameter reported is the 2000-episode warm-up.

free parameters (1)
  • Warm-up period = 2000 episodes
    Chosen by hand after 'empirically discovering' it performed better (§6); not justified by theory and affects the reported empirical results.
assumptions (5)
  • domain assumption The final joint history and action (h_{|τ|}, a_{|τ|}) are a good proxy for predicting the episodic global reward (Assumption 1).
    Introduced in §4.1; used to justify the form of r_{i,t}. Not validated empirically.
  • domain assumption The episodic return can be decomposed as a sum over agents and time steps, r_global,episodic(τ) = Σ_i Σ_t r_{i,t} (Assumptions 1 and 2).
    §4.1-4.2; requires the two-stage decomposition and the normalization constraints Σ_t w_t = 1 and Σ_i w'_{t,i} = 1.
  • ad hoc to paper The proof of optimality preservation assumes γ = 1 and does not extend to discounted returns.
    §4.3 'assuming γ = 1'; the standard potential shaping result is stated for general γ, but the constructed potential is only checked for the undiscounted case.
  • ad hoc to paper A shaping potential may depend on the episode return r_global,episodic(τ) and still guarantee policy invariance.
    Implicit in Theorem 2's proof, where ϕ(s_t) is defined in terms of r_global,episodic(τ); this violates the Markovian state-potential requirement of the cited potential-based shaping theorem.
  • domain assumption The weight functions w_t and w'_{t,i} are non-negative and bounded in [0,1], used to bound δ(τ) in Proposition 1.
    §4.4, used to claim δ(τ) ∈ [0,1]; not stated formally as a constraint in the model definition.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Agent-Temporal Credit Assignment for Optimal Policy Preservation in Sparse Multi-Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/6RHDQHGY

@misc{pith2026241214779,
  author       = {Pith},
  title        = {Pith review of: Agent-Temporal Credit Assignment for Optimal Policy Preservation in Sparse Multi-Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6RHDQHGY}},
  note         = {Machine review of arXiv:2412.14779}
}
abstract

In multi-agent environments, agents often struggle to learn optimal policies due to sparse or delayed global rewards, particularly in long-horizon tasks where it is challenging to evaluate actions at intermediate time steps. We introduce Temporal-Agent Reward Redistribution (TAR$^2$), a novel approach designed to address the agent-temporal credit assignment problem by redistributing sparse rewards both temporally and across agents. TAR$^2$ decomposes sparse global rewards into time-step-specific rewards and calculates agent-specific contributions to these rewards. We theoretically prove that TAR$^2$ is equivalent to potential-based reward shaping, ensuring that the optimal policy remains unchanged. Empirical results demonstrate that TAR$^2$ stabilizes and accelerates the learning process. Additionally, we show that when TAR$^2$ is integrated with single-agent reinforcement learning algorithms, it performs as well as or better than traditional multi-agent reinforcement learning methods.

Figures

Figures reproduced from arXiv: 2412.14779 by the authors.

Figure 1
Figure 1. Average agent episodic rewards with standard deviation for task 5m_vs_6m. [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 27 canonical work pages

  1. [1]

    Rtaw: An attention inspired reinforcement learning method for multi-robot task allocation in warehouse environments

    Aakriti Agrawal, Amrit Singh Bedi, and Dinesh Manocha. Rtaw: An attention inspired reinforcement learning method for multi-robot task allocation in warehouse environments. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp.\ 1393--1399, 2023. doi:10.1109/ICRA48891.2023.10161310

    arXiv 2023

  2. [2]

    Albrecht, Filippos Christianos, and Lukas Sch\"afer

    Stefano V. Albrecht, Filippos Christianos, and Lukas Sch\"afer. Multi-Agent Reinforcement Learning: Foundations and Modern Approaches. MIT Press, 2024. URL https://www.marl-book.com

    2024

  3. [3]

    (a partial survey of) decentralized, cooperative multi-agent reinforcement learning

    Christopher Amato. (a partial survey of) decentralized, cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2405.06161, 2024

    arXiv 2024

  4. [4]

    Rudder: Return decomposition for delayed rewards

    Jose A Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. Advances in Neural Information Processing Systems, 32, 2019

    2019

  5. [5]

    Multi-agent reinforcement learning for job shop scheduling in flexible manufacturing systems

    Schirin Baer, Jupiter Bakakeu, Richard Meyes, and Tobias Meisen. Multi-agent reinforcement learning for job shop scheduling in flexible manufacturing systems. In 2019 Second International Conference on Artificial Intelligence for Industries (AI4I), pp.\ 22--25, 2019. doi:10.1109/AI4I46381.2019.00014

    arXiv 2019

  6. [6]

    Dota 2 with large scale deep reinforcement learning

    Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemys aw D e biak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019

    arXiv 1912

  7. [7]

    A comprehensive survey of multiagent reinforcement learning

    Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38 0 (2): 0 156--172, 2008. doi:10.1109/TSMCC.2007.913919

    arXiv 2008

  8. [8]

    Primal \_2 : Pathfinding via reinforcement and imitation multi-agent learning-lifelong

    Mehul Damani, Zhiyao Luo, Emerson Wenzel, and Guillaume Sartoretti. Primal \_2 : Pathfinding via reinforcement and imitation multi-agent learning-lifelong. IEEE Robotics and Automation Letters, 6 0 (2): 0 2666--2673, 2021

    work page 2021

Show all 53 references
  1. [9]

    Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533, 2020

    Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533, 2020

    2011 arXiv

  2. [10]

    Theoretical considerations of potential-based reward shaping for multi-agent systems

    Sam Devlin and Daniel Kudenko. Theoretical considerations of potential-based reward shaping for multi-agent systems. In Adaptive Agents and Multi-Agent Systems, 2011. URL https://api.semanticscholar.org/CorpusID:1116773

    2011

  3. [11]

    Potential-based difference rewards for multiagent reinforcement learning

    Sam Devlin, Logan Yliniemi, Daniel Kudenko, and Kagan Tumer. Potential-based difference rewards for multiagent reinforcement learning. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pp.\ 165--172, 2014

    2014

  4. [12]

    Reinforcement learning with trajectory feedback

    Yonathan Efroni, Nadav Merlis, and Shie Mannor. Reinforcement learning with trajectory feedback. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp.\ 7288--7295, 2021

    2021

  5. [13]

    Counterfactual multi-agent policy gradients

    Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  6. [14]

    Learning cooperative multi-agent policies with partial reward decoupling

    Benjamin Freed, Aditya Kapoor, Ian Abraham, Jeff Schneider, and Howie Choset. Learning cooperative multi-agent policies with partial reward decoupling. IEEE Robotics and Automation Letters, 7 0 (2): 0 890--897, 2021

    2021

  7. [15]

    Learning guidance rewards with trajectory-space smoothing

    Tanmay Gangwani, Yuan Zhou, and Jian Peng. Learning guidance rewards with trajectory-space smoothing. Advances in Neural Information Processing Systems, 33: 0 822--832, 2020

    2020

  8. [16]

    Off-policy reinforcement learning with delayed rewards

    Beining Han, Zhizhou Ren, Zuofan Wu, Yuan Zhou, and Jian Peng. Off-policy reinforcement learning with delayed rewards. In International Conference on Machine Learning, pp.\ 8280--8303. PMLR, 2022

    2022

  9. [17]

    Hindsight credit assignment

    Anna Harutyunyan, Will Dabney, Thomas Mesnard, Mohammad Gheshlaghi Azar, Bilal Piot, Nicolas Heess, Hado P van Hasselt, Gregory Wayne, Satinder Singh, Doina Precup, et al. Hindsight credit assignment. Advances in neural information processing systems, 32, 2019

    2019

  10. [18]

    Human-level performance in 3d multiplayer games with population-based reinforcement learning

    Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science...

    2019

  11. [19]

    a fer, Andrew Wing Keung To, Kuan-Ho Lao, Murat Cubuktepe, Matthew Haley, Peter B \

    Aleksandar Krnjaic, Raul D Steleac, Jonathan D Thomas, Georgios Papoudakis, Lukas Sch \"a fer, Andrew Wing Keung To, Kuan-Ho Lao, Murat Cubuktepe, Matthew Haley, Peter B \"o rsting, et al. Scalable multi-agent reinforcement learning for warehouse logistics with robotic and hum...

    2022 arXiv

  12. [20]

    Google research football: A novel reinforcement learning environment

    Karol Kurach, Anton Raichuk, Piotr Sta \'n czyk, Micha Zaj a c, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI conferen...

    2020

  13. [21]

    Sequence modeling of temporal credit assignment for episodic reinforcement learning

    Yang Liu, Yunan Luo, Yuanyi Zhong, Xi Chen, Qiang Liu, and Jian Peng. Sequence modeling of temporal credit assignment for episodic reinforcement learning. arXiv preprint arXiv:1905.13420, 2019

    1905 arXiv

  14. [22]

    Multi-agent actor-critic for mixed cooperative-competitive environments

    Ryan Lowe, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30, 2017

    2017

  15. [23]

    Smaclite: A lightweight environment for multi-agent reinforcement learning

    Adam Michalski, Filippos Christianos, and Stefano V Albrecht. Smaclite: A lightweight environment for multi-agent reinforcement learning. arXiv preprint arXiv:2305.05566, 2023

    2023 arXiv

  16. [24]

    Policy invariance under reward transformations: Theory and application to reward shaping

    AY Ng. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the 16th International Conference on Machine Learning, pp.\ 278, 1999

    1999

  17. [25]

    Oliehoek and Chris Amato

    Frans A. Oliehoek and Chris Amato. A concise introduction to decentralized pomdps. In SpringerBriefs in Intelligent Systems, 2016. URL https://api.semanticscholar.org/CorpusID:3263887

    2016

  18. [26]

    Albrecht

    Georgios Papoudakis, Filippos Christianos, Lukas Sch \"a fer, and Stefano V. Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In NeurIPS Datasets and Benchmarks, 2020. URL https://api.semanticscholar.org/CorpusID:235417602

    2020

  19. [27]

    Align-rudder: Learning from few demonstrations by reward redistribution

    Vihang P Patil, Markus Hofmarcher, Marius-Constantin Dinu, Matthias Dorfer, Patrick M Blies, Johannes Brandstetter, Jose A Arjona-Medina, and Sepp Hochreiter. Align-rudder: Learning from few demonstrations by reward redistribution. arXiv preprint arXiv:2009.14108, 2020

    2009 arXiv

  20. [28]

    Monotonic value function factorisation for deep multi-agent reinforcement learning

    Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21 0 (178): 0 1--51, 2020

    2020

  21. [29]

    Learning long-term reward redistribution via randomized return decomposition

    Zhizhou Ren, Ruihan Guo, Yuan Zhou, and Jian Peng. Learning long-term reward redistribution via randomized return decomposition. arXiv preprint arXiv:2111.13485, 2021

    2021 arXiv

  22. [30]

    The starcraft multi-agent challenge

    Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043, 2019

    1902 arXiv

  23. [31]

    Primal: Pathfinding via reinforcement and imitation multi-agent learning

    Guillaume Sartoretti, Justin Kerr, Yunfei Shi, Glenn Wagner, TK Satish Kumar, Sven Koenig, and Howie Choset. Primal: Pathfinding via reinforcement and imitation multi-agent learning. IEEE Robotics and Automation Letters, 4 0 (3): 0 2378--2385, 2019

    2019

  24. [32]

    High-dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015

    2015 arXiv

  25. [33]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  26. [34]

    Agent-time attention for sparse rewards multi-agent reinforcement learning

    Jennifer She, Jayesh K Gupta, and Mykel J Kochenderfer. Agent-time attention for sparse rewards multi-agent reinforcement learning. arXiv preprint arXiv:2210.17540, 2022

    2022 arXiv

  27. [35]

    Multi-agent learning of efficient fulfilment and routing strategies in e-commerce

    Omkar Shelke, Pranavi Pathakota, Anandsingh Chauhan, Harshad Khadilkar, Hardik Meisheri, and Balaraman Ravindran. Multi-agent learning of efficient fulfilment and routing strategies in e-commerce. arXiv preprint arXiv:2311.16171, 2023

    2023 arXiv

  28. [36]

    Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning

    Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pp.\ 5887--5896. PMLR, 2019

    2019

  29. [37]

    Value-decomposition networks for cooperative multi-agent learning

    Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296, 2017

    2017 arXiv

  30. [38]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: A n Introduction . The MIT Press, Cambridge, MA, 1998

    1998

  31. [39]

    Multi-agent reinforcement learning: Independent versus cooperative agents

    Ming Tan. Multi-agent reinforcement learning: Independent versus cooperative agents. In International Conference on Machine Learning, 1997. URL https://api.semanticscholar.org/CorpusID:268857333

    1997

  32. [40]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  33. [41]

    Optimizing mixed autonomy traffic flow with decentralized autonomous vehicles and multi-agent rl

    Eugene Vinitsky, Nathan Lichtle, Kanaad Parvate, and Alexandre Bayen. Optimizing mixed autonomy traffic flow with decentralized autonomous vehicles and multi-agent rl. arXiv preprint arXiv:2011.00120, 2020

    2011 arXiv

  34. [42]

    Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Micha \"e l Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, L. Sifre, Trevor Cai, John P. Agapiou, Max Jader...

    2019

  35. [43]

    Shapley q-value: A local reward approach to solve global reward games

    Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu. Shapley q-value: A local reward approach to solve global reward games. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 7285--7292, 2020

    2020

  36. [44]

    Agent-temporal attention for reward redistribution in episodic multi-agent reinforcement learning

    Baicen Xiao, Bhaskar Ramasubramanian, and Radha Poovendran. Agent-temporal attention for reward redistribution in episodic multi-agent reinforcement learning. arXiv preprint arXiv:2201.04612, 2022

    2022 arXiv

  37. [45]

    Givigi Jr Xiaosong Lu, Howard M

    Sidney N. Givigi Jr Xiaosong Lu, Howard M. Schwartz. Policy invariance under reward transformations for general-sum stochastic games. Journal of Artificial Intelligence Research, 41: 0 397--406, 2011

    2011

  38. [46]

    The surprising effectiveness of ppo in cooperative multi-agent games

    Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35: 0 24611--24624, 2022

    2022

  39. [47]

    Fully decentralized multi-agent reinforcement learning with networked agents

    Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learning, pp.\ 5872--5881. PMLR, 2018

    2018

  40. [48]

    Multi-agent reinforcement learning: A selective overview of theories and algorithms

    Kaiqing Zhang, Zhuoran Yang, and Tamer Ba s ar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pp.\ 321--384, 2021

    2021

  41. [49]

    Grd: A generative approach for interpretable reward redistribution in reinforcement learning

    Yudi Zhang, Yali Du, Biwei Huang, Ziyan Wang, Jun Wang, Meng Fang, and Mykola Pechenizkiy. Grd: A generative approach for interpretable reward redistribution in reinforcement learning. arXiv preprint arXiv:2305.18427, 2023 a

    2023 arXiv

  42. [50]

    Learning a robust multiagent driving policy for traffic congestion reduction

    Yulin Zhang, William Macke, Jiaxun Cui, Sharon Hornstein, Daniel Urieli, and Peter Stone. Learning a robust multiagent driving policy for traffic congestion reduction. Neural Computing and Applications, pp.\ 1--14, 2023 b

    2023

  43. [51]

    Learning implicit credit assignment for cooperative multi-agent reinforcement learning

    Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, and Yuk Ying Chung. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. Advances in neural information processing systems, 33: 0 11853--11864, 2020

    2020

  44. [52]

    Towards long-delayed sparsity: learning a better transformer through reward redistribution

    Tianchen Zhu, Yue Qiu, Haoyi Zhou, and Jianxin Li. Towards long-delayed sparsity: learning a better transformer through reward redistribution. IJCAI '23, 2023. ISBN 978-1-956792-03-4. doi:10.24963/ijcai.2023/522. URL https://doi.org/10.24963/ijcai.2023/522

    2023 doi

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.