Pith. sign in

REVIEW 3 major objections 4 minor 55 references

Online Security Learning in Cooperative Multi-Agent Systems under Hidden Byzantine Attacks

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proves that security regret under hidden Byzantine attacks splits exactly into return regret plus a nonnegative response gap, that this gap makes linear regret unavoidable for any learner facing unrestricted realized attacks…

desk verdict Solid theoretical contribution to Byzantine-robust MARL, but the main security-regret guarantee is conditional on an unproven oracle assumption that the paper itself acknowledges. read the letter →

arxiv 2608.06520 v1 pith:XVXP4BYI submitted 2026-08-06 cs.LG

Ximing Sun , Yue Wang This is my paper · ORCID
classification cs.LG
keywords Byzantineattacksmulti-agentreinforcementlearningrobustMarkovdecisionprocesssecurityregretresponsegaponlineestimation-to-decisionsrectangularity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies a finite-horizon cooperative team whose unknown, fixed subset of agents is Byzantine: after seeing the team's planned joint action, the compromised agents can overwrite their own coordinates before execution, and the learner sees only planned actions, public rewards, and public states. The paper's central claim is that this security problem is exactly a rectangular robust Markov decision process, and that the security regret, the gap between the optimal worst-case return and the deployed policy's worst-case return, splits into the return regret against the response that actually generated the data plus a nonnegative response gap $D_K$ measuring how far the realized attack is from worst case. Two indistinguishable one-horizon instances force $\Omega(K)$ expected security regret even when return regret is zero, so no public-feedback algorithm can certify sublinear security regret against unrestricted realized attacks. On the positive side, a stage-tied estimation-to-decisions learner achieves expected security regret $\widetilde O(H^2 S \sqrt{AK}) + \mathbb{E}[D_K]$ under a standard approximate decision-oracle assumption, improving the state dependence by $\sqrt{S}$ over the direct statewise construction. This matters because it separates the statistical cost of learning from the information-theoretic cost of certifying security, and it shows exactly what public feedback can and cannot certify about hidden attacks.

What carries the argument

The load-bearing object is the exact public-law reduction: for a fixed Byzantine identity $B^\star$, the set of attainable conditional public-outcome laws at each $(h,s,a)$ is the compact convex hull $\Gamma_{h,s,a}^{B^\star} = \mathrm{conv}\{P^{\mathrm{phys}}_h(\cdot \mid s, a \oplus_{B^\star} u) : u \in A^{B^\star}\}$, making the induced model $(s,a)$-rectangular. This reduction turns the operational worst-case value into ordinary rectangular robust dynamic programming while preserving exactness against full-history randomized overwrites. The second carrying mechanism is the stage-tied estimator: instead of maintaining a separate calibration process for every predecessor state, it shares one continuation vector per stage and pays the certificate entropy and calibration charge once per layer, which reduces the cumulative estimation budget from $\widetilde O(HS^2)$ to $\widetilde O(HS)$. Feeding this estimator into the robust estimation-to-decisions (E2D) framework yields the $\sqrt{S}$ improvement and converts the estimation and optimism bounds into the stated security-regret bound.

What would settle it

Run Algorithm 1 with an exhaustive-search implementation of the decision oracle on a small tabular instance whose responder is exactly worst case (so $D_K=0$); if the average security regret does not decrease at the claimed $\widetilde O(H^2 S \sqrt{A/K})$ rate as $K$ grows, the bound is contradicted. Alternatively, exhibit any learner that on the two horizon-one instances of Theorem 4.1 achieves $o(K)$ expected security regret on both, which would refute the lower-bound construction.

Watch

Extended reading notes

Core claim

The central discovery is that Byzantine action overwrites with hidden identity and hidden execution induce an exact $(s,a)$-rectangular robust MDP: each state-action row is the convex hull of the public-outcome laws produced by all feasible overwrites, so deterministic nonstationary Markov team policies and deterministic Markov worst responses suffice. The paper further proves the exact identity $\mathrm{Reg}^{\mathrm{sec}}_K = \mathrm{Reg}_K + D_K$, where $\mathrm{Reg}_K$ is the return regret evaluated against the response that actually generated the data and $D_K \ge 0$ is a cumulative response gap. Because the learner cannot observe overwrites or executed actions, $D_K$ is unidentifiable; the paper constructs two horizon-one instances with identical public transcripts in which return regret is zero but security regret is at least $K/2$ for one of them, making the $D_K$ dependence information-theoretically unavoidable. The algorithmic contribution is a stage-tied robust estimation-to-decisions learner that, under the framework's approximate decision oracle, attains expected security regret $\mathbb{E}[\mathrm{Reg}^{\mathrm{sec}}_K] \le \min\{HK,\, C_K + \mathbb{E}[D_K]\}$ with $C_K = \widetilde O(H^2 S\sqrt{AK})$, using one continuation certificate and one occupancy-calibration witness per stage to cut the estimation budget from $\widetilde O(HS^2)$ to $\widetilde O(HS)$. The learner never identifies the compromised subset and never estimates the overwrites or executed actions.

Load-bearing premise

The main regret guarantee rests on Assumption 5.2, which assumes an oracle that returns a $K^{-2}$-approximate policy distribution at every episode; the paper does not construct this oracle or prove that the underlying decision problem can be solved in polynomial time.

Editorial extensions

If this is right

  • If the realized response is an exact worst response against each deployed policy, $D_K=0$ and the security regret is bounded by $\widetilde O(H^2 S\sqrt{AK})$.
  • The lower bound means that an attacker who plays benignly but could have played worse is just as damaging to a security certificate as an attacker who actually plays worst case: public feedback alone cannot distinguish them.
  • The $\sqrt{S}$ improvement shows that sharing a continuation vector across states in the estimator is strictly better than statewise certificate indexing, so state-dimension dependence in robust online reinforcement learning can be reduced without changing the decision class.
  • A guarantee of the form $C_K + \mathbb{E}[D_K]$ gives a concrete research target: any exploration strategy that shrinks the response gap translates directly into a tighter security certificate.
  • Under the stage-tied learner, the Byzantine identity never has to be identified; the algorithm only needs the public-outcome laws relevant to robust decisions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the $K^{-2}$-approximate decision oracle of Assumption 5.2 can be implemented in polynomial time for the stage-tied class, the bound would become algorithmic rather than conditional; the paper leaves this open.
  • The response-gap decomposition suggests a dual objective for practice: rather than maximizing factual return alone, a learner could deliberately query policies that force the attacker to reveal worst-case behavior, potentially trading a small return-regret increase for a large $D_K$ decrease.
  • The blind-attacker $s$-rectangular geometry, where randomized Markov policies can be essential, indicates that the information available to the attacker, not just the corruption, changes the learning problem; a separate stage-tied analysis for that model is a natural next step.
  • The two-instance lower bound exploits unrestricted realized responses; restricting attention to worst-case or $\epsilon$-worst responders eliminates the obstruction, so the practical regime where $D_K$ is small is precisely where security learning is feasible.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies online learning in a finite-horizon cooperative multi-agent system where an unknown, fixed subset of agents can overwrite their coordinates of the team's planned joint action after observing the plan. The learner observes planned actions, public rewards, and states, but not the overwrite or the executed action. The authors establish an exact reduction: if the attacker observes the planned action, the resulting public-outcome model is an (s,a)-rectangular robust MDP; if the attacker is blind, the model is s-rectangular. They introduce security regret, which compares each deployed policy's return to its worst-case overwrite value, and prove that security regret decomposes pathwise into return regret against the realized response plus a nonnegative cumulative response gap D_K. A lower bound (Theorem 4.1) constructs two horizon-one instances with identical public transcripts and zero return regret, yet security regret is linear in at least one, showing D_K is information-theoretically necessary. The main positive result (Theorem 5.4) is a stage-tied robust estimation-to-decisions (E2D) learner that, under an approximate decision oracle (Assumption 5.2), attains expected security regret O~(H^2 S sqrt(A K)) + E[D_K], with the estimation budget improved by a factor of sqrt(S) over a direct statewise approach. The paper also compares its benchmark and rates with uninformed Markov game results from prior work.

Significance. The decomposition in Proposition 2.1 and the lower bound in Theorem 4.1 are simple, self-contained, and correct; they cleanly identify the fundamental impediment to security learning from public feedback: the realized response's gap to worst-case behavior is unobservable and can be linear. The exact public-law reduction (Proposition 3.1) and the Markov sufficiency theorem (Theorem 3.2) are valuable structural contributions that clarify how the attacker's information geometry dictates the robust MDP class. The stage-tied estimator, if its proof is correct, provides a modest but concrete improvement over the direct statewise adaptation of Appel–Kosoy's E2D framework. However, the central positive result is conditional on Assumption 5.2, an oracle that is neither constructed nor shown to exist with the required measurability; the paper explicitly acknowledges in Section 5.4 that polynomial-time DEC minimization is not established. Since D_K can be linear, the positive bound without a feasible oracle does not yield a certification of security, making the oracle issue load-bearing rather than cosmetic.

major comments (3)
  1. [Section 5.4, Assumption 5.2, and Theorem 5.4] The main positive guarantee (Theorem 5.4) is conditional on an approximate E2D decision oracle whose existence is merely assumed, not established. The authors state in Section 5.4 that they do not establish polynomial-time DEC minimization, and Assumption 5.2 only postulates the existence of an F_{k-1}-measurable pk satisfying (18). Because the local class H_k in (15) lives in a continuous certificate space (v,c ∈ [0,H]^S), the DEC in (16) is an infinite-dimensional optimization; Lemma F.12 proves compactness and Borel measurability of Q_k but does not prove that an approximate minimizer with the required measurability exists, let alone that it can be computed. Without this oracle, the bound E[Reg_sec] ≤ C_K + E[D_K] is not a constructive guarantee, and given that D_K can be linear (Theorem 4.1), the result reduces to a return-regret bound that does not certify security. I ask the authors to either (i) provide a proof of existence of a measurable approximate oracle, e.g., via standard measurable selection theorems after completion of the filtration, (ii) give a concrete oracle for a tractable subclass, or (iii) explicitly reframe the contribution as an oracle-based reduction and temper the claim of 'algorithmic foundations.'
  2. [Appendix F.4, Lemmas F.8 and F.10] The claimed sqrt(S) improvement rests on the stage-tied calibration argument, but the proof is far too compressed for verification. Lemma F.8 (supernormalization) and Lemma F.10 (one-layer calibration transport) are the critical steps that establish the layer-level rather than state-level charge, yet the derivations are only sketched, relying on a terse decomposition of bettor multipliers into outcome and action increments. I recommend that the authors expand the proofs of these two lemmas, isolating the exact telescoping identities and the convexity/inequality steps, so that the 'one certificate per layer' claim can be checked by an independent reader. At minimum, a high-level proof sketch describing where the sqrt(S) savings enters would substantially improve confidence in Theorem 5.3.
  3. [Section 5.4 and Conclusion] The paper's abstract and conclusion describe the work as providing 'comprehensive theoretical and algorithmic foundations,' but the only algorithmic component that is actually implemented and shown to run in finite time is the market-based estimator; the decision module is an assumed oracle that may not be realizable. I suggest that the authors calibrate the claims to distinguish between (a) the statistical estimation bound, which is their novel algorithmic contribution, and (b) the decision oracle, which is an imported abstraction. This will make the paper's actual scope transparent and avoid overstatement.
minor comments (4)
  1. [Theorem 5.4 statement] The definition of C_K is written as `C_K := 114H^2S√(AKΛK(K−2))` with unbalanced parentheses; this should be clarified, e.g., `C_K := 114 H^2 S sqrt(A K Λ_K(K^{-2}))`.
  2. [Abstract and Section 2.1] The abstract uses 'Byzantine comprised'; this appears to be a typo for 'Byzantine compromised'.
  3. [Section 5.1, Eq. (14)] The notation `N◦` for the analysis-only surrogate is not introduced before its first use in Proposition 5.1; a brief explanation of the superscript circle would improve readability.
  4. [Appendix B.3] The notation `q emp,⋆` and the definition of `Cpost` are used without an explicit formal statement of the empirical comparator class; moving the definition of `Ωemp_K(B⋆)` earlier in that subsection would help the reader follow the comparison.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central decomposition is an algebraic identity, the lower bound is a direct construction, and the E2D oracle is an explicit, unproven assumption rather than a disguised input.

full rationale

The paper's security-regret guarantee rests on Proposition 2.1, which states Reg_sec = Reg_K + D_K pathwise, with D_K := sum_k(m_k - W(pi_k)). This is an exact algebraic identity obtained by expanding definitions, not a fitted quantity later renamed as a prediction. The lower bound (Theorem 4.1) is self-contained: it constructs two horizon-one instances with identical public transcripts and computes their security values directly, yielding E[Reg_sec] >= K/2 for one of them while return regret is zero; no external benchmark or hidden parameter is imported. The positive result (Theorem 5.4) is explicitly conditional on Assumption 5.2, an approximate E2D decision oracle whose existence and implementability the paper does not claim to establish; Section 5.4 states that polynomial-time DEC minimization is not established. This is a stated assumption, not a result derived from the conclusion, so it is a correctness/complexity caveat rather than circularity. The E2D machinery is imported from the non-self-cited works of Appel and Kosoy [18] and Foster et al. [17], and the stage-tied market is a new construction with its own estimation theorem (Theorem 5.3). The only self-citation is Wang & Zou [39], used in a related-work contrast about prescribed uncertainty sets; it is not load-bearing for any theorem. Accordingly, no step reduces to its own input by construction, and the derivation chain is not circular; the minor self-citation justifies only a low non-zero score.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted constants or invented physical entities. It relies on standard robust-MDP dynamic programming, measurability results, and an explicit oracle assumption (Assumption 5.2). The fixed-identity/fixed-kernel assumption is a domain assumption of the threat model.

assumptions (4)
  • domain assumption The Byzantine identity B* and the physical public-outcome kernel are fixed and episode-invariant.
    Central to the reduction and the analysis; used in Section 2.1 and Lemma F.3. If the identity or kernel varied, the robust MDP rows would change and the regret analysis would break.
  • domain assumption There exists a K^{-2}-approximate E2D decision oracle returning F_{k-1}-measurable policy distributions (Assumption 5.2).
    The main regret bound depends on this oracle. The paper does not construct it or prove it exists; it is imported from the E2D framework and stated as an assumption.
  • standard math Standard measurable-selection and probability results (Borel right inverses, Ionescu-Tulcea, Azuma-Hoeffding).
    Used in the public-law reduction and the market analysis; standard results assumed without proof.
  • standard math Rectangular robust MDP dynamic programming lemmas from Iyengar and Nilim/El Ghaoui.
    Used to compute robust values and show Markov policy sufficiency; cited results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Online Security Learning in Cooperative Multi-Agent Systems under Hidden Byzantine Attacks." pith.science (2026). https://pith.science/paper/XVXP4BYI

@misc{pith2026260806520,
  author       = {Pith},
  title        = {Pith review of: Online Security Learning in Cooperative Multi-Agent Systems under Hidden Byzantine Attacks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XVXP4BYI}},
  note         = {Machine review of arXiv:2608.06520}
}
abstract

We study online cooperative control of a multi-agent system under Byzantine attacks. Namely, an unknown, fixed subset of agents are Byzantine comprised and can stealthily overwrite its own coordinates of the team's planned joint action after observing that plan. The learner observes planned actions, public rewards, and public states, but neither the overwrite nor the executed joint action. Our objective is security: to optimize the team performance against the worst overwrites and achieve the optimal security value. We first show that the attacker's information determines the geometry. An attacker that observes the planned action induces an exact $(s,a)$-rectangular robust Markov decision process (MDP) whose rows are convex hulls of overwrite-induced public-outcome laws, whereas a blind attacker induces an $s$-rectangular model. We then identify the information-theoretic limit of security learning, showing that the security regret decomposes exactly into return regret against the response generating the data and a cumulative response gap $D_K$. Two indistinguishable horizon-one instances force $\Omega(K)$ expected security regret while return regret is zero, showing that dependence on $D_K$ is unavoidable. Finally, we develop a stage-tied robust estimation-to-decisions learner and prove a regret bound of $\widetilde{\mathcal O}\!\left(H^2S\sqrt{AK}\right)+\mathbb E[D_K]$. Our studies thus provide comprehensive theoretical and algorithmic foundations of reliable multi-agent systems under Byzantine attacks.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 41 canonical work pages

  1. [1]

    A comprehensive survey on multi-agent reinforcement learning for connected and automated vehicles.Sensors, 23(10):4710, 2023

    Pamul Yadav, Ashutosh Mishra, and Shiho Kim. A comprehensive survey on multi-agent reinforcement learning for connected and automated vehicles.Sensors, 23(10):4710, 2023

    work page 2023

  2. [2]

    Min Hua, Xinda Qi, Dong Chen, Kun Jiang, Zemin Eitan Liu, Hongyu Sun, Quan Zhou, and Hongming Xu. Multi-agent reinforcement learning for connected and automated vehicles control: Recent advancements and future prospects.IEEE Transactions on Automation Science and Engineering, 2025

    work page 2025

  3. [3]

    Coordinated Multi-Agent Reinforcement Learning for Unmanned Aerial Vehicle Swarms in Autonomous Mobile Access Applications

    Chanyoung Park, Haemin Lee, Won Joon Yun, Soyi Jung, and Joongheon Kim. Coordinated multi-agent reinforcement learning for unmanned aerial vehicle swarms in autonomous mobile access applications.arXiv preprint arXiv:2304.08493, 2022

    work page Pith review arXiv 2022

  4. [4]

    Cooperative multiagent deep reinforcement learning for reliable surveillance via autonomous multi-uav control.IEEE Transactions on Industrial Informatics, 18(10):7086–7096, 2022

    Won Joon Yun, Soohyun Park, Joongheon Kim, MyungJae Shin, Soyi Jung, David A Mohaisen, and Jae-Hyun Kim. Cooperative multiagent deep reinforcement learning for reliable surveillance via autonomous multi-uav control.IEEE Transactions on Industrial Informatics, 18(10):7086–7096, 2022

    work page 2022

  5. [5]

    MyungJae Shin, Dae-Hyun Choi, and Joongheon Kim. Cooperative management for pv/ess-enabled electric vehicle charging stations: A multiagent deep reinforcement learning approach.IEEE Transactions on Industrial Informatics, 16(5):3493–3503, 2019

    work page 2019

  6. [6]

    Autogen: Enabling next-gen llm applications via multi-agent conversation.arXiv preprint arXiv:2308.08155, 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation.arXiv preprint arXiv:2308.08155, 2023

    arXiv 2023

  7. [7]

    Camel: Communicative agents for" mind" exploration of large language model society.Advances in neural information processing systems, 36:51991–52008, 2023

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society.Advances in neural information processing systems, 36:51991–52008, 2023

    2023

  8. [8]

    Robust llm-based multi-agent system with action negotiation and sharing redundancy enhancement

    Xujia Li, Beirong Cui, Junyu Huang, and Lei Chen. Robust llm-based multi-agent system with action negotiation and sharing redundancy enhancement. InProceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V . 1, pages 736–747, 2026

    work page 2026

Show all 55 references
  1. [9]

    Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection, 2023

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection, 2023

    2023

  2. [10]

    The byzantine generals problem

    Leslie Lamport, Robert Shostak, and Marshall Pease. The byzantine generals problem. InConcurrency: the works of leslie lamport, pages 203–226. 2019

    2019

  3. [11]

    Learning from history for byzantine robust optimization

    Sai Praneeth Karimireddy, Lie He, and Martin Jaggi. Learning from history for byzantine robust optimization. In International conference on machine learning, pages 5311–5319. PMLR, 2021

    2021

  4. [12]

    Adversarial policies: Attacking deep reinforcement learning.arXiv preprint arXiv:1905.10615, 2019

    Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning.arXiv preprint arXiv:1905.10615, 2019

    1905 arXiv

  5. [13]

    On the robustness of cooperative multi-agent reinforcement learning

    Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcia, and Nicolas Papernot. On the robustness of cooperative multi-agent reinforcement learning. In2020 IEEE Security and Privacy Workshops (SPW), pages 62–68. IEEE, 2020. 10

    2020

  6. [14]

    Attacking cooperative multi-agent reinforcement learning by adversarial minority influence

    Simin Li, Jun Guo, Jingqiao Xiu, Yuwei Zheng, Pu Feng, Xin Yu, Aishan Liu, Yaodong Yang, Bo An, Wenjun Wu, and Xianglong Liu. Attacking cooperative multi-agent reinforcement learning by adversarial minority influence. Neural Networks, 191:107747, 2025

    2025

  7. [15]

    Empirical study on robustness and resilience in cooperative multi-agent reinforcement learning.Advances in Neural Information Processing Systems, 38:44009–44052, 2026

    Simin Li, Zihao Mao, Hanxiao Li, Zonglei Jing, Jun Guo, Li Wang, Zhuoran Han, Ruixiao Xu, Xin Yu, Chengdong Ma, et al. Empirical study on robustness and resilience in cooperative multi-agent reinforcement learning.Advances in Neural Information Processing Systems, 38:44009–44052, 2026

    2026

  8. [16]

    Robust dynamic programming.Mathematics of Operations Research, 30(2):257–280, 2005

    Garud N Iyengar. Robust dynamic programming.Mathematics of Operations Research, 30(2):257–280, 2005

    2005

  9. [17]

    The statistical complexity of interactive decision making.arXiv preprint arXiv:2112.13487, 2021

    Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making.arXiv preprint arXiv:2112.13487, 2021

    2021 arXiv

  10. [18]

    Regret bounds for robust online decision making.arXiv preprint arXiv:2504.06820, 2025

    Alexander Appel and Vanessa Kosoy. Regret bounds for robust online decision making.arXiv preprint arXiv:2504.06820, 2025

    2025 arXiv

  11. [19]

    Action robust reinforcement learning and applications in continuous control

    Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. InProc. International Conference on Machine Learning (ICML), pages 6215–6224. PMLR, 2019

    2019

  12. [20]

    Resilient multi-agent reinforcement learning with adversarial value decomposition

    Thomy Phan, Lenz Belzner, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, and Claudia Linnhoff-Popien. Resilient multi-agent reinforcement learning with adversarial value decomposition. InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11308–11316, 2021

    2021

  13. [21]

    Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attackers

    Lei Yuan, Ziqian Zhang, Ke Xue, Hao Yin, Feng Chen, Cong Guan, Lihe Li, Chao Qian, and Yang Yu. Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attackers. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 117...

    2023

  14. [22]

    Alexander Bukharin, Yan Li, Yue Yu, Qingru Zhang, Zhehui Chen, Simiao Zuo, Chao Zhang, Songan Zhang, and Tuo Zhao. Robust multi-agent reinforcement learning via adversarial regularization: Theoretical foundation and stable algorithms.Advances in Neural Information Processing S...

    2023

  15. [23]

    Wolfpack adversarial attack for robust multi-agent reinforcement learning.arXiv preprint arXiv:2502.02844, 2025

    Sunwoo Lee, Jaebak Hwang, Yonghyeon Jo, and Seungyul Han. Wolfpack adversarial attack for robust multi-agent reinforcement learning.arXiv preprint arXiv:2502.02844, 2025

    2025 arXiv

  16. [24]

    Roping in uncertainty: Robustness and regularization in markov games

    Jeremy Mcmahan, Giovanni Artiglio, and Qiaomin Xie. Roping in uncertainty: Robustness and regularization in markov games. InInternational Conference on Machine Learning, pages 35267–35295. PMLR, 2024

    2024

  17. [25]

    Byzantine robust cooperative multi-agent reinforcement learning as a bayesian game

    Simin Li, Jun Guo, Jingqiao Xiu, Ruixiao Xu, Xin Yu, Jiakai Wang, Aishan Liu, Yaodong Yang, and Xianglong Liu. Byzantine robust cooperative multi-agent reinforcement learning as a bayesian game. InInternational Conference on Learning Representations, volume 2024, pages 31771–3...

    2024

  18. [26]

    Defending against unknown corrupted agents: Reinforcement learning of adversarially robust nash equilibria.Transactions on Machine Learning Research, 2024

    Andi Nika, Jonathan Nöther, Adish Singla, and Goran Radanovic. Defending against unknown corrupted agents: Reinforcement learning of adversarially robust nash equilibria.Transactions on Machine Learning Research, 2024

    2024

  19. [27]

    Byzantine-robust online and offline distributed reinforcement learning

    Yiding Chen, Xuezhou Zhang, Kaiqing Zhang, Mengdi Wang, and Xiaojin Zhu. Byzantine-robust online and offline distributed reinforcement learning. InInternational Conference on Artificial Intelligence and Statistics, pages 3230–3269. PMLR, 2023

    2023

  20. [28]

    Byzantine tolerant algorithms for federated learning.IEEE Transactions on Network Science and Engineering, 10(6):3172–3183, 2023

    Qi Xia, Zeyi Tao, Qun Li, and Songqing Chen. Byzantine tolerant algorithms for federated learning.IEEE Transactions on Network Science and Engineering, 10(6):3172–3183, 2023

    2023

  21. [29]

    Provably robust federated reinforcement learning

    Minghong Fang, Xilong Wang, and Neil Zhenqiang Gong. Provably robust federated reinforcement learning. In Proceedings of the ACM on Web Conference 2025, pages 896–909, 2025

    2025

  22. [30]

    Br- defedrl: Byzantine-robust decentralized federated reinforcement learning with fast convergence and communication efficiency

    Jing Qiao, Zuyuan Zhang, Sheng Yue, Yuan Yuan, Zhipeng Cai, Xiao Zhang, Ju Ren, and Dongxiao Yu. Br- defedrl: Byzantine-robust decentralized federated reinforcement learning with fast convergence and communication efficiency. InIEEE infocom 2024-IEEE conference on computer com...

    2024

  23. [31]

    Byzantine-robust federated reinforcement learning via critical parameter analysis.International Journal of Machine Learning and Cybernetics, 16(12):10607–10620, 2025

    Wenzheng Jiang, Ji Wang, Weidong Bao, and Yaohong Zhang. Byzantine-robust federated reinforcement learning via critical parameter analysis.International Journal of Machine Learning and Cybernetics, 16(12):10607–10620, 2025

    2025

  24. [32]

    Byzantine-robust federated deep deterministic policy gradient

    Qifeng Lin and Qing Ling. Byzantine-robust federated deep deterministic policy gradient. InICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4013–4017. IEEE, 2022

    2022

  25. [33]

    Byzantine-robust federated learning with optimal statistical rates

    Banghua Zhu, Lun Wang, Qi Pang, Shuai Wang, Jiantao Jiao, Dawn Song, and Michael I Jordan. Byzantine-robust federated learning with optimal statistical rates. InInternational Conference on Artificial Intelligence and Statistics, pages 3151–3178. PMLR, 2023. 11

    2023

  26. [34]

    Robust control of markov decision processes with uncertain transition matrices.Operations Research, 53(5):780–798, 2005

    Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices.Operations Research, 53(5):780–798, 2005

    2005

  27. [35]

    Robust markov decision processes.Mathematics of Operations Research, 38(1):153–183, 2013

    Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes.Mathematics of Operations Research, 38(1):153–183, 2013

    2013

  28. [36]

    Sample complexity of robust reinforcement learning with a generative model

    Kishan Panaganti and Dileep Kalathil. Sample complexity of robust reinforcement learning with a generative model. InProc. International Conference on Artifical Intelligence and Statistics (AISTATS), pages 9582–9602. PMLR, 2022

    2022

  29. [37]

    Toward theoretical understandings of robust markov decision processes: Sample complexity and asymptotics.The Annals of Statistics, 50(6):3223–3248, 2022

    Wenhao Yang, Liangyu Zhang, and Zhihua Zhang. Toward theoretical understandings of robust markov decision processes: Sample complexity and asymptotics.The Annals of Statistics, 50(6):3223–3248, 2022

    2022

  30. [38]

    The curious price of distributional robustness in reinforcement learning with a generative model.arXiv preprint arXiv:2305.16589, 2023

    Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, Matthieu Geist, and Yuejie Chi. The curious price of distributional robustness in reinforcement learning with a generative model.arXiv preprint arXiv:2305.16589, 2023

    2023 arXiv

  31. [39]

    Online robust reinforcement learning with model uncertainty

    Yue Wang and Shaofeng Zou. Online robust reinforcement learning with model uncertainty. InProc. Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 7193–7206, 2021

    2021

  32. [40]

    Miao Lu, Han Zhong, Tong Zhang, and Jose Blanchet. Distributionally robust reinforcement learning with interactive data collection: Fundamental hardness and near-optimal algorithms.Advances in Neural Information Processing Systems, 37:12528–12580, 2024

    2024

  33. [41]

    Sample complexity of distributionally robust off-dynamics reinforcement learning with online interaction

    Yiting He, Zhishuai Liu, Weixin Wang, and Pan Xu. Sample complexity of distributionally robust off-dynamics reinforcement learning with online interaction. InProc. International Conference on Machine Learning (ICML), 2025

    2025

  34. [42]

    Corruption-robust offline reinforcement learning with general function approximation.Advances in Neural Information Processing Systems, 36:36208–36221, 2023

    Chenlu Ye, Rui Yang, Quanquan Gu, and Tong Zhang. Corruption-robust offline reinforcement learning with general function approximation.Advances in Neural Information Processing Systems, 36:36208–36221, 2023

    2023

  35. [43]

    Corruption-robust offline reinforcement learning.arXiv preprint arXiv:2106.06630, 2021

    Xuezhou Zhang, Yiding Chen, Jerry Zhu, and Wen Sun. Corruption-robust offline reinforcement learning.arXiv preprint arXiv:2106.06630, 2021

    2021 arXiv

  36. [44]

    Corruption-robust exploration in episodic reinforcement learning

    Thodoris Lykouris, Max Simchowitz, Alex Slivkins, and Wen Sun. Corruption-robust exploration in episodic reinforcement learning. InProc. Annual Conference on Learning Theory (CoLT), pages 3242–3245. PMLR, 2021

    2021

  37. [45]

    Online learning in unknown markov games

    Yi Tian, Yuanhao Wang, Tiancheng Yu, and Suvrit Sra. Online learning in unknown markov games. InInternational conference on machine learning, pages 10279–10288. PMLR, 2021

    2021

  38. [46]

    Online learning for uninformed markov games: Empirical nash-value regret and non-stationarity adaptation.arXiv preprint arXiv:2602.07205, 2026

    Junyan Liu, Haipeng Luo, Zihan Zhang, and Lillian J Ratliff. Online learning for uninformed markov games: Empirical nash-value regret and non-stationarity adaptation.arXiv preprint arXiv:2602.07205, 2026

    2026

  39. [47]

    Learning markov games with adversarial opponents: Efficient algorithms and fundamental limits

    Qinghua Liu, Yuanhao Wang, and Chi Jin. Learning markov games with adversarial opponents: Efficient algorithms and fundamental limits. InInternational Conference on Machine Learning, pages 14036–14053. PMLR, 2022

    2022

  40. [48]

    Thanh Nguyen-Tang and Raman Arora. Learning in markov games with adaptive adversaries: Policy regret, fundamental barriers, and efficient algorithms.Advances in Neural Information Processing Systems, 37:56268– 56303, 2024

    2024

  41. [49]

    The maxmin value of stochastic games with imperfect monitoring.International journal of game theory, 32(1):133–150, 2003

    Dinah Rosenberg, Eilon Solan, and Nicolas Vieille. The maxmin value of stochastic games with imperfect monitoring.International journal of game theory, 32(1):133–150, 2003

    2003

  42. [50]

    Sample-efficient reinforcement learning of partially observable markov games.Advances in Neural Information Processing Systems, 35:18296–18308, 2022

    Qinghua Liu, Csaba Szepesvári, and Chi Jin. Sample-efficient reinforcement learning of partially observable markov games.Advances in Neural Information Processing Systems, 35:18296–18308, 2022

    2022

  43. [51]

    Fictitious play in markov games with single controller

    Muhammed O Sayin, Kaiqing Zhang, and Asuman Ozdaglar. Fictitious play in markov games with single controller. InProceedings of the 23rd ACM Conference on Economics and Computation, pages 919–936, 2022

    2022

  44. [52]

    A finite-sample analysis of payoff-based independent learning in zero-sum stochastic games.Advances in Neural Information Processing Systems, 36:75826–75883, 2023

    Zaiwei Chen, Kaiqing Zhang, Eric Mazumdar, Asuman Ozdaglar, and Adam Wierman. A finite-sample analysis of payoff-based independent learning in zero-sum stochastic games.Advances in Neural Information Processing Systems, 36:75826–75883, 2023

    2023

  45. [53]

    Learning in zero-sum markov games: Relaxing strong reachability and mixing time assumptions

    Reda Ouhamma and Maryam Kamgarpour. Learning in zero-sum markov games: Relaxing strong reachability and mixing time assumptions. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 17170–17178, 2026

    2026

  46. [54]

    Regret minimization and convergence to equilibria in general-sum markov games

    Liad Erez, Tal Lancewicki, Uri Sherman, Tomer Koren, and Yishay Mansour. Regret minimization and convergence to equilibria in general-sum markov games. InInternational Conference on Machine Learning, pages 9343–9373. PMLR, 2023

    2023

  47. [55]

    HX h=1 Rh # ≤E ϖ,u⋆

    Shinji Ito, Haipeng Luo, Arnab Maiti, Taira Tsuchiya, and Yue Wu. Adversarial learning in games with bandit feedback: Logarithmic pure-strategy maximin regret.arXiv preprint arXiv:2602.06348, 2026. 12 A Preliminaries of Robust MDPs Throughout this appendix, fix a finite state ...

    2026

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.