Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning

Mehdi Dastani; Shihan Wang; Shuai Han

REVIEW 4 major objections 5 minor 30 references

Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Per-agent influence scopes, measured by mutual information, drive credit assignment and exploration in sparse-reward MARL.

desk verdict Genuinely novel credit-assignment idea with solid ablations and open code; the unverified stability of one-shot influence scopes is the main caveat, but the paper merits a serious referee. read the letter →

arxiv 2505.08630 v2 pith:YMOTNQOS submitted 2025-05-13 cs.LG

Shuai Han , Mehdi Dastani , Shihan Wang This is my paper · ORCID

classification cs.LG

keywords multi-agentreinforcementlearningsparserewardscreditassignmentexplorationmutualinformationinfluencescopegoal-conditioned

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that in cooperative multi-agent reinforcement learning with sparse rewards, agents should be given individual goals defined by their 'influence scope' — the set of state dimensions whose changes depend on their action, measured by conditional mutual information. The authors argue that this scope solves two problems simultaneously: credit assignment, because an agent is rewarded only for progress on state segments its action can influence, with jointly influenced segments handled through a common/special split; and exploration, because count-based bonuses count only the state projections inside the scope instead of the full high-dimensional state. The algorithm, ISA, computes the scopes once from a few thousand random transitions, then trains exploration policies and goal-conditioned policies. If the claim is right, sparse-reward tasks can be solved without dense reward shaping and with an interpretable per-agent decomposition of global goals.

What carries the argument

The central object is the influence scope, an index set of state dimensions defined by $D(a_i)=\{k \mid I(\Delta s_k; a_i \mid a_{-i})>\delta\}$, computed from transitions collected under random policies. The dependence measure is estimated by equal-width binning of the state change $\Delta s_k$, binarization of the action $a_i$, and averaging the conditional mutual information over sampled values of the other agents' actions. This scope is the load-bearing device: it defines each agent's individual goal as the projection of a global goal onto $D_i$ (Definition 3), splits that goal into a common segment $D_c=\cap_i D_i$ and a special segment $D_i\setminus D_c$ (Definition 4), and then gates both the goal-conditioned reward in Eq. (7) and the exploration bonus in Eq. (10) on whether the current action can influence the common segment ($D(a_i)\cap D_c\neq\varnothing$).

What would settle it

Run ISA on a sparse-reward MPE task where one agent's action only affects a state dimension after another agent has reached a prerequisite landmark, and re-estimate the influence scopes midway through training. If the newly computed scopes differ from the initial scopes in a way that changes which dimensions are credited or explored, then the central assumption that scopes can be fixed before training is false; equivalently, a plot of $D_i$ over training that grows to include previously excluded dimensions would falsify the once-only computation.

Watch

Extended reading notes

Core claim

The central discovery is that a thresholded conditional mutual information, $D(a_i)= \{k \mid I(\Delta s_k; a_i \mid a_{-i}) > \delta\}$, defines an agent's influence scope and can drive both credit assignment and exploration in sparse-reward MARL. The paper shows how to estimate this quantity by discretizing state changes, binarizing actions, and averaging over other agents' actions, and then uses the agent-level scope $D_i=\cup_{a_i}D(a_i)$ to decompose a discovered global goal into per-agent sub-goals (Definition 3). The reward rule of Eq. (7) gives an agent credit from the common segment only when its current action can influence it, and otherwise only from its special segment, which prevents other agents' actions from corrupting its feedback. The exploration rule of Eq. (10) applies the same gating to count-based novelty bonuses restricted to the scoped projections (Eqs. (8)-(9)). In experiments on super-sparse SMAC and MPE tasks, the paper reports that ISA outperforms IPPO, QMIX, COMA, MASER, CMAE, FoX and HMASD in sample efficiency and final performance, and ablations show that removing the scope-based credit assignment or exploration degrades learning.

Load-bearing premise

The load-bearing premise is that the influence measure computed once from a few thousand random transitions, cut off at a hand-tuned threshold, correctly names the state components each agent's action can ever affect, and that these names stay correct throughout training even though they are never updated.

Editorial extensions

If this is right

Sparse-reward cooperative tasks become learnable without handcrafted dense reward shaping, because intrinsic rewards are derived from state dimensions each agent can actually influence.
The common/special segment split prevents unstable feedback: an agent is rewarded for the jointly influenced segment only when its current action affects it, so other agents' actions cannot corrupt its credit signal.
Exploration is more efficient because count-based bonuses count only scoped projections of the state, avoiding the curse of dimensionality in high-dimensional multi-agent state spaces.
Credit assignment becomes interpretable: the if-else rule in Eq. (7) yields a clear explanation of which actions are credited for which state changes, as demonstrated in the 8m example where agent 7 receives less reward while not contributing to the common segment.
The overhead of computing scopes is small because they are estimated from 2,000 to 10,000 transitions, and the method is applicable across SMAC and MPE benchmarks with a single threshold $\delta=0.3$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because Algorithm 1 computes influence scopes once before training, a natural extension is periodic re-estimation; if scopes drift as policies change, the fixed-scope version could miss newly relevant state dimensions.
The mutual information criterion captures immediate influence only; the paper's manual 2-step adjustment for 2s vs 1sc hints that a general multi-step influence measure would extend ISA to tasks with long action delays.
The threshold $\delta$ is hand-tuned; an adaptive threshold or a significance test on the mutual information estimate would remove a domain-dependent hyperparameter and make the method easier to apply.
The hard gating in Eq. (7) could be replaced by a soft weighting proportional to $I(\Delta s_k; a_i\mid a_{-i})$, which might smooth the reward signal when influence estimates are noisy near the threshold.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Han, Dastani, and Wang have written a paper that deserves a real referee, not a desk rejection. The novelty is genuine: influence scope, defined as the set of state dimensions whose change is conditionally mutually informative about an agent's action, is a clean way to delimit individual goals in cooperative MARL. The credit-assignment rule in Eq. 7, which gates common-segment reward on whether the action touches the common scope, is a simple and interpretable idea. The counting-based exploration restricted to the scope (Eq. 10) is a natural extension. The ablations are well-designed: they show that removing the scope, the segmentation, or the gating rule each hurts, which supports the components. The code is open, and the comparison includes strong baselines (MASER, CMAE, FoX, HMASD). The heat maps in Fig. 3 are a nice sanity check for why conditioning on other agents' actions matters.

The main soft spot is exactly what the stress-test flags: scopes are computed once from a few thousand random-policy transitions (Line 4 of Algorithm 1) and then used for the entire training run. Mutual information is a property of the joint distribution, not just the environment. The paper's Remark 1 claims the scope is an 'inherent property of the environment,' but that needs evidence. The deferred Appendix C.3 reportedly contains stability and significance tests, but they are not in the main text of this version, and the core stability claim is too load-bearing to leave in an appendix. The ablation 'w/o influence scope' shows that restricting the goal/count space helps in these tasks, but it does not show that the fixed scopes are the correct ones under the trained policies. That is a fixable issue: the authors could re-estimate scopes periodically, or at least plot the scopes' overlap across training, or show that results are insensitive to the choice of delta within the stated range.

The hyperparameter story is honest but mixed. Delta is set to 0.3 with a claimed workable range, but lambda, alpha1/beta1, and alpha2/beta2 are tuned per task (lambda is 0, 10, or 50 across environments). That is more tuning than I would like, but not disqualifying for a methods paper. The reported five-run learning curves show large margins, but significance testing is deferred to an appendix that is not present in this submission.

Overall: the central idea is well-motivated, the implementation is careful, and the empirical story is plausible. The stability of the influence scopes is a genuine question that the authors must address in the main text before the claim of 'significantly outperforms' is fully credible. I would send this to peer review with a request to move the stability analysis forward and to make the appendix available. It is a useful paper for anyone working on sparse-reward MARL, and I would cite it once the stability question is settled.

Referee Report

4 major / 5 minor

Summary. The paper proposes ISA (Influence Scope of Agents), a method for sparse-reward cooperative MARL that combines credit assignment and exploration. ISA defines the influence scope of an action as the set of state dimensions k for which the conditional mutual information I(Δs_k; a_i | a_{-i}) exceeds a threshold δ (Definition 1, Eq. 4), and the influence scope of an agent as the union of the scopes of its actions (Definition 2). These scopes are used to decompose a global goal into per-agent individual goals (Definition 3) and to split each individual goal into a common segment (jointly influenced by all agents) and a special segment (Definition 4). The credit-assignment rule in Eq. 7 gives an agent credit for progress on the common segment only if its action's scope intersects the common scope, while the exploration bonus in Eq. 10 gates counting-based novelty bonuses in the same way. Algorithm 1 computes the scopes once from random transitions (N = 2,000 or 10,000) and never updates them. Experiments on SMAC and MPE, with ablations and an interpretability example, are used to claim significant improvements over state-of-the-art baselines.

Significance. If the proposed approach is correct, it addresses a real and important problem in sparse-reward MARL: automatically decomposing global goals into interpretable, per-agent goals and assigning credit without dense reward shaping. The information-theoretic definition of influence scope is a conceptually clean way to connect actions to state dimensions, and the idea of restricting exploration to scopes is plausible and potentially useful. The paper also releases open-source code, which supports reproducibility. However, the central claims currently rest on empirical evidence whose key validity conditions are neither proved nor adequately tested: the stability of scopes computed from random transitions under later trained policies, the reliability of the MI estimates from limited samples, and the statistical significance of the reported improvements. The ablations do support the usefulness of the scope-based restriction and the two credit-assignment mechanisms, but they are reported on a single task. Overall, the paper presents a promising framework that needs additional evidence and analysis before the central claims can be accepted.

major comments (4)

[Section 4, Definition 1/Eq. (4); Algorithm 1, Line 4] The influence scopes D(a_i) and D_i are computed once from transitions collected under random or early exploration policies and are never updated during training. Mutual information is a property of the joint distribution of (Δs_k, a_i, a_{-i}), not an environment constant; as policies change, the relevant joint distribution shifts. Since Eq. (7) and Eq. (10) use the scopes as hard gates, an incorrect scope can either withhold credit for genuinely influenced dimensions or grant credit for dimensions the agent does not influence. The paper provides no argument or experiment showing that scopes estimated from a few thousand random transitions remain valid for the goal-conditioned policies being trained. I recommend adding an empirical stability check (e.g., recomputing scopes at several points during training and measuring agreement) or an intervention-based validation on at least one benchmark, and discussing under which conditions the random-policy scopes are provably equal to the scopes under the training distribution.
[Section 6, Hyperparameters] The threshold δ is fixed at 0.3 based on a reported 'workable range' [0.15, 0.45], but the other introduced hyperparameters (λ, α1 = β1, α2 = β2, N) are tuned per task, and in the SMAC domain α1 = β1 = 0, which disables the special-segment credit in Eqs. (7) and (10). The main text does not report sensitivity analysis; the reader is referred to Appendix C.3, which is not included in the manuscript under review. Because the central benefit of ISA depends on the quality of the influence scopes and on the balance between common and special segment rewards, the paper should present sensitivity results in the main text (or at least in an accessible appendix) and justify that the per-task tuning does not inadvertently favor ISA over baselines.
[Section 6, Results, Fig. 1] The main text claims that ISA 'significantly outperforms' baselines based on learning curves with 5 seeds and without reporting any statistical test. Significance testing is deferred to Appendix C.3, which is not available for review. Since the paper's central assertion is empirical, the main text should include basic statistics for the key environments (e.g., mean ± standard deviation over seeds and a paired test at a fixed environment step), at least for the main comparison curves.
[Section 5.1, Eq. (3)] The estimator of the conditional mutual information is described as randomly sampling combinations of a_{-i} and averaging the per-combination MI values. With N = 2,000 transitions and multi-agent action spaces, many a_{-i} combinations will have very few or no samples, so the estimated I(Δs_k; a_i | a_{-i}) may have high variance or bias. This directly affects the correctness of Definition 1. The paper should provide a bias/variance analysis or validate the estimator on a synthetic setting with known conditional independence, and should report the number of distinct a_{-i} combinations actually observed in the experiments.

minor comments (5)

[Section 3, Preliminaries] 'maximize the excepted accumulated reward' should read 'maximize the expected accumulated reward'.
[Section 6, Interpretability] The text says 'Fig .3(b) illustrates this scenario. At the time step of this screenshot...', but Fig. 3 shows heat maps, not a screenshot; the screenshot appears in Fig. 4(b). Please correct the figure reference.
[Section 5.1, Eq. (10) and following text] The final combination line 'ri + β2 r' is ambiguous because r_i is not explicitly defined as R_i+(s,a_i,s'); please define r_i+ consistently.
[Section 4, Definition 3 and Remark 3] The phrase 'can can influence' appears in the goal-decomposition paragraph; please fix the duplicate.
[References] References [Liu et al., 2021a] and [Liu et al., 2021b] have the same title and page numbers, which suggests a typo or unintended duplication. Please verify and correct.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: influence scopes are empirically estimated inputs used for reward shaping, and the reported performance is not forced by construction.

full rationale

The paper's derivation chain is not circular. ISA first estimates the influence scope D(ai) from mutual information between state changes and actions conditioned on other agents' actions (Definition 1, Eq. 4), computed from collected environment transitions. These scopes are then used as inputs to the credit-assignment reward (Eq. 7) and the exploration bonus (Eq. 10). The central empirical claim is that this reward-shaping scheme improves sample efficiency and final performance against baselines; this claim is not equivalent to the scopes themselves. The scopes do not encode the benchmark success criteria or the baseline results, so the measured performance is not determined by construction from the fitted quantities. The threshold delta = 0.3 is a hyperparameter fine-tuned over a stated stable range (Section 6), not a parameter fitted to reproduce the target outcome. The policy learning itself is delegated to an external algorithm (IPPO), and the empirical comparison is against independently defined baselines. There is no load-bearing self-citation: the cited prior work is external, and the paper does not invoke any uniqueness theorem or ansatz from the authors' own earlier work to force the chosen design. The concern that influence scopes are computed once from early transitions and not updated during training is a stability and robustness question, not a circularity in the derivation. Therefore, no step reduces to its own inputs by definition, and the paper warrants a circularity score of 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces the conceptual construct of influence scope but no new physical entity. The main load-bearing choices are the threshold delta and per-task scaling factors, which are tuned by hand, and the unproved assumption that MI-based influence identification remains valid during training.

free parameters (5)

delta (influence threshold) = 0.3
Threshold in Definition 1 that decides which state dimensions count as influenced; the paper fine-tunes delta and reports 0.3 works across the tested tasks.
lambda (Hamming distance weight) = varies: 10 in 8m, 0 in Navigation/Unlock, 50 elsewhere
Weighting of Hamming distance in the distance metric d(v1,v2) used in intrinsic reward Eqs. (5)-(6); tuned per environment.
alpha1 = beta1 (special segment scale) = 0 for SMAC, 0.2 for MPE
Scale factors for special segment rewards in Eqs. (7) and (10); set differently per environment family.
alpha2 = beta2 (environment reward scale) = 10 in 8m, 0 in 3m and 2s vs 1sc, 1 in MPE
Scaling of environmental reward in policy updates; tuned per task.
N (transitions for MI estimation) = 10,000 in 8m, 2,000 elsewhere
Number of transitions used to compute influence scopes in Step 1; chosen per task.

assumptions (4)

domain assumption In a Dec-POMDP, an action always affects certain dimensions of the environmental state more significantly than other dimensions.
Stated at the start of Section 4 as the grounding for influence scope.
domain assumption Mutual information I(Delta s_k; a_i | a_{-i}) > delta correctly identifies the dimensions genuinely influenced by action a_i.
Definition 1 treats this MI threshold as the definition of influence without proof of causal or statistical soundness.
domain assumption Influence scopes computed once from random transitions remain valid throughout training.
Algorithm 1 computes D(a_i), D_i, D^c, D^(i-c) in Line 4 before training and never updates them; policies change the state-action distribution later.
domain assumption The reward is determined by some subset of state dimensions D', and each state dimension is observable, numeric, and discretizable.
Required for goal projection and MI estimation via equal-width binning; implicit in Definitions 3-4 and Section 4.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/YMOTNQOS

@misc{pith2026250508630,
  author       = {Pith},
  title        = {Pith review of: Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YMOTNQOS}},
  note         = {Machine review of arXiv:2505.08630}
}

read the original abstract

Training cooperative agents in sparse-reward scenarios poses significant challenges for multi-agent reinforcement learning (MARL). Without clear feedback on actions at each step in sparse-reward setting, previous methods struggle with precise credit assignment among agents and effective exploration. In this paper, we introduce a novel method to deal with both credit assignment and exploration problems in reward-sparse domains. Accordingly, we propose an algorithm that calculates the Influence Scope of Agents (ISA) on states by taking specific value of the dimensions/attributes of states that can be influenced by individual agents. The mutual dependence between agents' actions and state attributes are then used to calculate the credit assignment and to delimit the exploration space for each individual agent. We then evaluate ISA in a variety of sparse-reward multi-agent scenarios. The results show that our method significantly outperforms the state-of-art baselines.

Figures

Figures reproduced from arXiv: 2505.08630 by the authors.

**Figure 2.** An illustration on decomposing individual goals from a global goal. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 4.** Ablations and interpretability for ISA Interpretability. Our credit assignment based on the influence scope offers good interpretability. Based on the if-else rule in Equ. (7), we can interpret whether a specific action ai of agent i has influence on the common segment Dc. For instance, when D(ai) ∩ Dc = ∅, the current action ai has no influence on Dc, and as a result, no reward from common segment shall be assign… view at source ↗

Figures from the paper (1 more)

**Figure 3.** Figure 3: Heat maps of mutual information values. Ablations. We perform ablations on the mutual information calculation conditioned on a−i and not conditioned on a−i in Equ. (3). The results are shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 27 canonical work pages

[1]

Hindsight experience replay

[Andrychowicz et al., 2017] Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Ad- vances in Neural Information Processing Systems 30: An- nual Conference on Neural Information Processing Sys- tems, pages 5048–5058,

work page 2017
[8]

MASER: multi-agent rein- forcement learning with subgoals generated from experi- ence replay buffer

[Jeon et al., 2022] Jeewon Jeon, Woojun Kim, Whiyoung Jung, and Youngchul Sung. MASER: multi-agent rein- forcement learning with subgoals generated from experi- ence replay buffer. In International Conference on Ma- chine Learning, volume 162, pages 10041–10052,

work page 2022
[9]

Fox: Formation-aware explo- ration in multi-agent reinforcement learning

[Jo et al., 2024] Yonghyeon Jo, Sunwoo Lee, Junghyuk Yeom, and Seungyul Han. Fox: Formation-aware explo- ration in multi-agent reinforcement learning. In Thirty- Eighth AAAI Conference on Artificial Intelligence , pages 12985–12994. AAAI Press,

work page 2024
[11]

Estimating mu- tual information

[Kraskov et al., 2004] Alexander Kraskov, Harald St¨ogbauer, and Peter Grassberger. Estimating mu- tual information. Physical review E , 69(6):066138,

work page 2004
[13]

Multi-agent actor-critic for mixed cooperative-competitive environ- ments

[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environ- ments. In Advances in Neural Information Processing Sys- tems, pages 6379–6390,

work page 2017
[14]

Oliehoek and Christo- pher Amato

[Oliehoek and Amato, 2016] Frans A. Oliehoek and Christo- pher Amato. A Concise Introduction to Decentral- ized POMDPs . Springer Briefs in Intelligent Systems. Springer,

2016
[18]

Foerster, and Shimon Whiteson

[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schr ¨oder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of the 35th International Confer- ence on Machine Learning, volume 80, pages 4292–4301. PMLR,

work page 2018
[19]

Changing the environment based on em- powerment as intrinsic motivation

[Salge et al., 2014] Christoph Salge, Cornelius Glackin, and Daniel Polani. Changing the environment based on em- powerment as intrinsic motivation. Entropy, 16(5):2789– 2819,

work page 2014

Show all 30 references

[24]

Strehl and Michael L

[Strehl and Littman, 2008] Alexander L. Strehl and Michael L. Littman. An analysis of model-based in- terval estimation for markov decision processes. J. Comput. Syst. Sci., 74(8):1309–1331,

2008
[25]

Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M

[Sutton et al., 2011] Richard S. Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M. Pilarski, Adam White, and Doina Precup. Horde: a scalable real-time ar- chitecture for learning knowledge from unsupervised sen- sorimotor interaction. In 10th International Confer...

2011
[26]

#exploration: A study of count-based exploration for deep reinforcement learning

[Tang et al., 2017] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schul- man, Filip De Turck, and Pieter Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Syst...

2017
[27]

DOP: off-policy multi-agent decomposed policy gradients

[Wang et al., 2021] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. DOP: off-policy multi-agent decomposed policy gradients. In 9th Interna- tional Conference on Learning Representations,

2021
[29]

Hierarchical multi- agent skill discovery

[Yang et al., 2024] Mingyu Yang, Yaodong Yang, Zhenbo Lu, Wengang Zhou, and Houqiang Li. Hierarchical multi- agent skill discovery. In Advances in Neural Information Processing Systems 36,

2024
[30]

To- ward socially friendly autonomous driving using multi- agent deep reinforcement learning

[Yeh and Soo, 2024] Jhih-Ching Yeh and V on-Wun Soo. To- ward socially friendly autonomous driving using multi- agent deep reinforcement learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 2573–2575, 2024

2024
[1959]

QTRAN: learn- ing to factorize with transformation for cooperative multi- agent reinforcement learning

[Son et al., 2019] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Hostallero, and Yung Yi. QTRAN: learn- ing to factorize with transformation for cooperative multi- agent reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, vol- ume 9...

2019
[2003]

Autotelic agents with intrinsically motivated goal-conditioned reinforce- ment learning: A short survey

[Colas et al., 2022] C´edric Colas, Tristan Karch, Olivier Sigaud, and Pierre-Yves Oudeyer. Autotelic agents with intrinsically motivated goal-conditioned reinforce- ment learning: A short survey. J. Artif. Intell. Res. , 74:1159–1199,

2022
[2004]

Taylor, Wenyuan Tao, and Zhen Wang

[Li et al., 2022] Pengyi Li, Hongyao Tang, Tianpei Yang, Xiaotian Hao, Tong Sang, Yan Zheng, Jianye Hao, Matthew E. Taylor, Wenyuan Tao, and Zhen Wang. PMIC: improving multi-agent reinforcement learning with pro- gressive mutual information collaboration. In Interna- tional Co...

2022
[2008]

Efficient planning for factored infinite-horizon dec-pomdps

[Pajarinen and Peltonen, 2011] Joni Pajarinen and Jaakko Peltonen. Efficient planning for factored infinite-horizon dec-pomdps. In IJCAI Proceedings-International Joint Conference on Artificial Intelligence , volume 22, page 325,

2011
[2011]

A survey of temporal credit assignment in deep reinforcement learning

[Pignatelli et al., 2023] Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, and Laura Toni. A survey of temporal credit assignment in deep reinforcement learning. CoRR, abs/2312.01072,

2023 arXiv
[2014]

[Samvelyan et al., 2019] Mikayel Samvelyan, Tabish Rashid, Christian Schr ¨oder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon White- son. The starcraft multi-agent challenge. In Proceedings of the ...

2019
[2015]

Coding theorems for a discrete source with a fidelity criterion.IRE Nat

[Shannon and others, 1959] Claude E Shannon et al. Coding theorems for a discrete source with a fidelity criterion.IRE Nat. Conv. Rec, 4(142-163):1,

1959
[2016]

Exploit- ing locality of interaction in factored dec-pomdps

[Oliehoek et al., 2008] Frans A Oliehoek, Matthijs TJ Spaan, Nikos Vlassis, and Shimon Whiteson. Exploit- ing locality of interaction in factored dec-pomdps. In Int. Joint Conf. on Autonomous Agents and Multi-Agent Sys- tems,

2008
[2017]

Parsing reward

[Berridge and Robinson, 2003] Kent C Berridge and Terry E Robinson. Parsing reward. Trends in neurosciences , 26(9):507–513,

2003
[2018]

ALMA: hierarchical learning for composite multi- agent tasks

[Iqbal et al., 2022] Shariq Iqbal, Robby Costales, and Fei Sha. ALMA: hierarchical learning for composite multi- agent tasks. In Advances in Neural Information Processing Systems,

2022
[2019]

Universal value function ap- proximators

[Schaul et al., 2015] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. In International conference on machine learning, pages 1312–1320. PMLR,

2015
[2020]

Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson

[Foerster et al., 2018] Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the Thirty-Second AAAI Conference on Ar- tificial Intelligence, pages 2974–2982,

2018
[2021]

CM3: coop- erative multi-goal multi-stage multi-agent reinforcement learning

[Yang et al., 2020] Jiachen Yang, Alireza Nakhaei, David Isele, Kikuo Fujimura, and Hongyuan Zha. CM3: coop- erative multi-goal multi-stage multi-agent reinforcement learning. In 8th International Conference on Learning Representations,

2020
[2022]

An empowerment-based solution to robotic manipulation tasks with sparse rewards

[Dai et al., 2023] Siyu Dai, Wei Xu, Andreas Hofmann, and Brian Williams. An empowerment-based solution to robotic manipulation tasks with sparse rewards. Au- tonomous Robots, 47(5):617–633,

2023
[2023]

Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533,

[De Witt et al., 2020] Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533,

2020 arXiv
[2024]

Multi- target pursuit by a decentralized heterogeneous UA V swarm using deep multi-agent reinforcement learning

[Kouzehgar et al., 2023] Maryam Kouzehgar, Youngbin Song, Malika Meghjani, and Roland Bouffanais. Multi- target pursuit by a decentralized heterogeneous UA V swarm using deep multi-agent reinforcement learning. In International Conference on Robotics and Automation , pages 3289–3295,

2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Hindsight experience replay

[Andrychowicz et al., 2017] Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Ad- vances in Neural Information Processing Systems 30: An- nual Conference on Neural Information Processing Sys- tems, pages 5048–5058,

work page 2017

[2] [8]

MASER: multi-agent rein- forcement learning with subgoals generated from experi- ence replay buffer

[Jeon et al., 2022] Jeewon Jeon, Woojun Kim, Whiyoung Jung, and Youngchul Sung. MASER: multi-agent rein- forcement learning with subgoals generated from experi- ence replay buffer. In International Conference on Ma- chine Learning, volume 162, pages 10041–10052,

work page 2022

[3] [9]

Fox: Formation-aware explo- ration in multi-agent reinforcement learning

[Jo et al., 2024] Yonghyeon Jo, Sunwoo Lee, Junghyuk Yeom, and Seungyul Han. Fox: Formation-aware explo- ration in multi-agent reinforcement learning. In Thirty- Eighth AAAI Conference on Artificial Intelligence , pages 12985–12994. AAAI Press,

work page 2024

[4] [11]

Estimating mu- tual information

[Kraskov et al., 2004] Alexander Kraskov, Harald St¨ogbauer, and Peter Grassberger. Estimating mu- tual information. Physical review E , 69(6):066138,

work page 2004

[5] [13]

Multi-agent actor-critic for mixed cooperative-competitive environ- ments

[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environ- ments. In Advances in Neural Information Processing Sys- tems, pages 6379–6390,

work page 2017

[6] [14]

Oliehoek and Christo- pher Amato

[Oliehoek and Amato, 2016] Frans A. Oliehoek and Christo- pher Amato. A Concise Introduction to Decentral- ized POMDPs . Springer Briefs in Intelligent Systems. Springer,

2016

[7] [18]

Foerster, and Shimon Whiteson

[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schr ¨oder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of the 35th International Confer- ence on Machine Learning, volume 80, pages 4292–4301. PMLR,

work page 2018

[8] [19]

Changing the environment based on em- powerment as intrinsic motivation

[Salge et al., 2014] Christoph Salge, Cornelius Glackin, and Daniel Polani. Changing the environment based on em- powerment as intrinsic motivation. Entropy, 16(5):2789– 2819,

work page 2014