Pith. sign in

REVIEW 3 major objections 4 minor 140 references

Solving Zero-Sum Convex Markov Games

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Decentralized policy gradients provably find equilibria in zero-sum convex Markov games.

desk verdict A serious theory paper with a genuinely useful min-max toolbox, but its central claim overreaches because the definition of zero-sum cMGs is ambiguous and the hidden-convexity step at the foundation is unproved under the stated assumptions. read the letter →

arxiv 2506.16120 v1 pith:M5FLNCSK submitted 2025-06-19 cs.GT cs.LGcs.MAmath.OC

Fivos Kalogiannis , Emmanouil-Vasileios Vlatakis-Gkaragkounis , Ian Gemp , Georgios Piliouras This is my paper
classification cs.GTcs.LGcs.MAmath.OC MSC 91A1591A0590C47
keywords convexMarkovgameszero-sumNashequilibriumpolicygradienthiddenconvexityPolyak-Lojasiewiczconditionmin-maxoptimizationmulti-agentreinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes that two players in a zero-sum convex Markov game can reach an approximate Nash equilibrium by running simple, decentralized policy gradient updates, despite the game lacking the Bellman consistency that classical dynamic programming relies on. It provides the first global convergence guarantees for this setting, with iteration and sample counts that scale polynomially in the desired accuracy and in the sizes of the state and action spaces. The argument works by exploiting a hidden convexity in the players' utilities, adding a carefully chosen regularization, and then showing the regularized objective satisfies a proximal Polyak-Lojasiewicz condition that stabilizes gradient-based play.

What carries the argument

Hidden convexity of each player's utility through the state-action occupancy measure: although the payoff is nonconvex in the policy, it is concave in a latent variable (the occupancy measure), whose inverse map is Lipschitz continuous. Adding a small negative quadratic term in the maximizing player's occupancy makes that hidden concave function hidden strongly concave, which in turn yields the proximal Polyak-Lojasiewicz (pPL) condition in policy space. The pPL condition and the resulting quadratic growth property are what allow the best-response map to be Lipschitz, and they drive the convergence proofs for the nested and alternating gradient updates.

What would settle it

Construct a two-player zero-sum convex Markov game whose payoff F satisfies the paper's stated concavity condition on each occupancy measure but is not hidden convex in the minimizing player's own policy for a fixed opponent policy, e.g. by making the opponent's occupancy depend nonlinearly on the player's occupancy while keeping F concave; if running Algorithm 1 or 2 on such a game fails to reduce exploitability below a positive constant across many seeds and step-size choices, the central claim's hidden-convexity premise is violated.

Watch

Extended reading notes

Core claim

There exist decentralized policy gradient methods that compute an epsilon-approximate Nash equilibrium in any two-player zero-sum convex Markov game for any epsilon, using poly(1/epsilon, |S|, |A|+|B|, 1/(1-gamma)) iterations and samples. This is achieved by regularizing the min-max utility so that each player's best response becomes Lipschitz continuous in the opponent's policy, then showing the regularized objective fits a nonconvex-proximal Polyak-Lojasiewicz (NC-pPL) framework. Under that framework, both a nested gradient scheme and an alternating gradient descent-ascent scheme converge globally to saddle points, and hence to Nash equilibria, even when gradients are estimated from imperfect samples.

Load-bearing premise

The proof requires that, when the opponent's policy is held fixed, each player's utility is convex or concave in a hidden variable (the occupancy measure) and that the opponent's occupancy measure is a linear image of the player's own occupancy measure; if that cross-convexity fails, the gradient-dominance inequality at the heart of the argument does not hold.

Editorial extensions

If this is right

  • Equilibrium computation in zero-sum convex Markov games is tractable by simple first-order methods, contrary to what the game's nonconvexity and lack of Bellman consistency might suggest.
  • The two algorithms run with each player estimating gradients independently, so the results hold under a decentralized, inexact, stochastic gradient oracle satisfying the paper's Model.
  • A Lipschitz continuous best-response map holds for hidden-convex (and more generally NC-pPL) min-max objectives, going beyond the previously known Holder-type continuity for constrained problems.
  • The convergence rates are polynomial in 1/epsilon for general concave utilities and logarithmic in 1/epsilon when the utilities are hidden strongly concave, with the latter also giving last-iterate convergence for alternating updates.
  • The constrained min-max convergence theorems for NC-pPL and two-sided pPL objectives stand on their own as results about alternating and nested gradient methods, applicable beyond Markov games.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The results suggest that the failure of Bellman consistency by itself is not an obstacle to provable equilibrium computation; what matters is the hidden convex structure of the utility in the occupancy measure, so other multi-agent settings with such structure may be tractable by similar first-order methods.
  • The paper's Definition 1 states only that F1 and F2 are concave with F2 = -F1, which is too weak to imply the hidden convexity used in the proofs; a testable strengthening (a cross-convexity condition relating the two occupancy measures when the opponent's policy is fixed) would make the theorem's scope precise.
  • One could test the robustness of the method by constructing a two-player zero-sum game whose payoff is concave in each occupancy measure but fails the hidden-convexity condition with respect to the player's own policy; such a game should break the gradient-dominance lemma and likely cause the algorithms to cycle.
  • The regularizer's coefficient micro-regulates a bias-accuracy trade-off, suggesting a practical tuning rule: pick micro proportional to the desired epsilon and shrink it as the iterate approaches equilibrium, which the numerical experiments on iterated rock-paper-scissors-dummy illustrate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper addresses equilibrium computation in two-player zero-sum convex Markov games via independent policy gradient methods. It proposes a regularization of the utility by subtracting (μ/2)||λ2||², claims hidden convexity and hidden strong concavity for the two players, reduces the problem to constrained nonconvex-proximal Polyak–Łojasiewicz min-max optimization, and provides convergence rates for stochastic nested and alternating algorithms (Theorems 3.1–3.4 and 4.3–4.6). The advertised main result is the first poly(1/ε, |S|, |A|+|B|, 1/(1−γ)) guarantee for computing approximate Nash equilibria in any two-player zero-sum convex Markov game.

Significance. If correct, this would be a substantial advance: it would give the first global convergence guarantee for policy gradient methods in zero-sum convex Markov games despite the failure of Bellman consistency, and the constrained nonconvex-pPL analysis in Appendix D would be of independent interest. The paper is unusually complete in supplying explicit rate dependencies, stochastic gradient estimators, and a lengthy proof appendix. However, the correctness of the cMG application rests on an unproved hidden-convexity property for both players, and the formal Definition 1 as written trivializes the model. The significance is therefore conditional on a substantive repair of the modeling assumptions and of the proof of the key gradient-dominance lemma.

major comments (3)
  1. [Section 2.1, Definition 1] The line '-F1 = F2 =: F' together with 'there exist concave F1, F2' forces F to be both concave and convex, hence affine. As written, the definition therefore reduces zero-sum convex Markov games to linear-reward Markov games and contradicts the intended generality. The formal definition must state the cross-convexity assumption used later: for a fixed opponent policy, the player's own payoff should be convex or concave in the player's own marginal occupancy along the fiber λ2 = D_y λ1. Without such a statement, Claim 2.3 and Lemma 2.5 have no valid premises.
  2. [Section 2.3, Claim 2.3] The proof of Claim 2.3 is a single sentence citing Definitions 1 and 4 and Lemma 2.1. It omits the essential facts: (i) for fixed y, λ2(s,b) = y(b|s) Σ_a λ1(s,a), so λ2 is a linear image of λ1; (ii) the set {λ1(x,y) : x ∈ X} is convex; and (iii) concavity of F in each argument does not imply that H_y(λ1) = F(λ1, D_y λ1) is convex. For example, on a two-state instance with F(λ1,λ2) = ||λ1||² − ||λ2||², the function F is concave in λ2 but H_y is nonconvex when D_y has operator norm up to √|A|. Thus hidden convexity of U(·,y) for the minimizing player is not established for general zero-sum cMGs, and Proposition 2.4 cannot be invoked.
  3. [Section 2.3, Lemma 2.5, and Section 4] The gradient-dominance lemma, and therefore the pPL reduction and Theorems 4.3 and 4.5, depends on hidden convexity of U(·,y) and hidden concavity of U(x,·). Because Claim 2.3 does not establish these properties for the class stated in Definition 1, the central claim of Section 1.2 ('any two-player zero-sum convex Markov game') is unsupported. In addition, the regularized objective U^μ subtracts (μ/2)||λ2(x,y)||², which for fixed y is a convex function of the hidden variable λ1; the paper gives no argument that U^μ retains the hidden convexity in x needed for the x-updates in Algorithms 1 and 2. This is a load-bearing gap, not a presentation issue.
minor comments (4)
  1. [Throughout] The manuscript contains many typographical errors, including 'equilibirum' in Section 1.2, 'desried accuracy' in Theorem 4.3, 'Fucntions' in Appendix E.1, and 'T uning' in Appendix D.4. A careful proofreading pass is needed.
  2. [Section 2.1, Definition 1] The roles of F1 and F2 are unclear: F1, F2 are said to be concave, but then U is defined via F only, without explaining whether F is the maximizing player's utility or the minimizing player's utility. The sign convention should be stated explicitly.
  3. [Appendix E, Theorems E.10–E.13] The exact dependence of the rates on the hidden-strong-concavity moduli is inconsistent across statements: Theorem E.10 uses μ_x and μ_y, while Theorem E.11 introduces μ1, μ2 and then uses μ_x in the displayed formulas. These quantities should be defined once and used consistently.
  4. [Lemma E.4] The bias bound in Lemma E.4 has a constant 256 L_F that appears dimensionally inconsistent (L_F appears without a square, while adjacent bounds are quadratic in L_F). The constant and the derivation should be checked.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the cMG-to-pPL reduction is a derivation from stated structural assumptions, not a fitted prediction or self-citation chain.

full rationale

I walked the paper's derivation chain from the cMG definition through Claim 2.3 and Lemma 2.5 to the pPL-based convergence theorems. The central reduction is not circular: the regularizer coefficient mu is an algorithmic free parameter tuned as Theta(epsilon), not fitted to observed data or to a target Nash value, and the pPL/quadratic-growth properties are derived from hidden strong concavity of the regularized occupancy objective rather than assumed as the theorem's conclusion. The convergence theorems do not rename the cMG conclusion as an input; they condition on the structural property that, with the opponent fixed, the utility is hidden convex/concave in the player's own policy, which is part of the cMG model itself. Self-citations do appear: Lemma 2.1's proof is deferred to (Kalogiannis et al., 2024, Lemmata C.2 & C.3), and the existence of Nash equilibria is imported from (Gemp et al., 2024), which shares an author. But these are technical occupancy-continuity facts and background existence results; they do not carry the convergence-rate argument, which is built on the paper's own NC-pPL and alternating-GDA analyses. The skeptic's concern that Definition 1 literally forces F affine and that Claim 2.3's hidden convexity is not established for general cMGs is a correctness/assumption risk, not a circularity: an ambiguous or underproved premise does not make the derived claim identical to its input. No fitted input is renamed as a prediction, no uniqueness theorem is imported to forbid alternatives, and no known result is merely renamed as organization. Hence the honest finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard assumptions about occupancy maps, concavity and linearity properties of utilities, oracle access to gradients of F, and the algorithmic regularizer mu. No new physical or mathematical entities are postulated.

free parameters (2)
  • Regularization coefficient mu_reg = Theta((1-gamma) min_s rho(s) epsilon / L_F) for concave-utility case
    Hand-tuned to balance induced strong concavity and pPL modulus against bias in the fixed point; not fitted to any data set.
  • Exploration parameters epsilon_x, epsilon_y = Theta(epsilon * poly(min_s rho(s), (1-gamma), |S|, |A|+|B|)^-1)
    Hand-chosen to control policy-gradient estimator variance; they introduce an O(epsilon) bias to the game value and enter the convergence rates.
assumptions (5)
  • domain assumption The state-action occupancy measure is L_lambda-Lipschitz and ell_lambda-smooth in policies, and its inverse is L_lambda_inv-Lipschitz with L_lambda_inv = 2/(min_s rho(s)(1-gamma)).
    Used in Lemma 2.1 to establish hidden convexity and gradient dominance; standard MDP analysis, cited from Kalogiannis et al. (2024).
  • ad hoc to paper The zero-sum utility U is convex in the minimizing player's occupancy marginal and concave in the maximizing player's marginal, and for a fixed opponent policy the opponent's marginal occupancy is linear in the player's own marginal.
    This is implied by the intended definition of zero-sum cMG but not stated; Claim 2.3 depends on it and the paper does not prove the linearity explicitly.
  • domain assumption Both players have access to stochastic estimators of their gradients with bounded second moments and bounded bias, and to the gradient of the utility F with respect to the occupancy measure.
    This is the stochastic inexact first-order oracle model in Section 3; Remark 4 admits access to nabla_lambda F.
  • domain assumption Nash equilibria exist in convex Markov games.
    Needed to define epsilon-NE and the target value Phi*; inherited from Gemp et al. (2024).
  • standard math Standard equivalences among proximal Polyak-Lojasiewicz, Kurdyka-Lojasiewicz, quadratic growth, and error-bound conditions for constrained smooth functions.
    Used in Appendix C to convert hidden strong convexity to pPL and QG; cited to Karimi et al. (2016) and related works.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Solving Zero-Sum Convex Markov Games." pith.science (2026). https://pith.science/paper/M5FLNCSK

@misc{pith2026250616120,
  author       = {Pith},
  title        = {Pith review of: Solving Zero-Sum Convex Markov Games},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M5FLNCSK}},
  note         = {Machine review of arXiv:2506.16120}
}
read the original abstract

We contribute the first provable guarantees of global convergence to Nash equilibria (NE) in two-player zero-sum convex Markov games (cMGs) by using independent policy gradient methods. Convex Markov games, recently defined by Gemp et al. (2024), extend Markov decision processes to multi-agent settings with preferences that are convex over occupancy measures, offering a broad framework for modeling generic strategic interactions. However, even the fundamental min-max case of cMGs presents significant challenges, including inherent nonconvexity, the absence of Bellman consistency, and the complexity of the infinite horizon. We follow a two-step approach. First, leveraging properties of hidden-convex--hidden-concave functions, we show that a simple nonconvex regularization transforms the min-max optimization problem into a nonconvex-proximal Polyak-Lojasiewicz (NC-pPL) objective. Crucially, this regularization can stabilize the iterates of independent policy gradient methods and ultimately lead them to converge to equilibria. Second, building on this reduction, we address the general constrained min-max problems under NC-pPL and two-sided pPL conditions, providing the first global convergence guarantees for stochastic nested and alternating gradient descent-ascent methods, which we believe may be of independent interest.

Figures

Figures reproduced from arXiv: 2506.16120 by the authors.

Figure 1
Figure 1. Exploitability decays towards a small, but positive value corresponding to the bias introduced [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

140 extracted references · 48 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Ho, Michael Littman, Doina Precup, and Satinder Singh

    David Abel, Will Dabney, Anna Harutyunyan, Mark K. Ho, Michael Littman, Doina Precup, and Satinder Singh. On the expressivity of markov reward. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  3. [3]

    Theory of maxima and the method of lagrange

    SN Afriat. Theory of maxima and the method of lagrange. SIAM Journal on Applied Mathematics, 20 0 (3): 0 343--357, 1971

    1971

  4. [4]

    The rate of convergence of bregman proximal methods: Local geometry versus regularity versus sharpness

    Wa\" ss Azizian, Franck Iutzeler, J\' e r\^ o me Malick, and Panayotis Mertikopoulos. The rate of convergence of bregman proximal methods: Local geometry versus regularity versus sharpness. SIAM Journal on Optimization, 34 0 (3): 0 2440--2471, 2024. doi:10.1137/23M1580218. URL https://doi.org/10.1137/23M1580218

    work page doi:10.1137/23m1580218 2024

  5. [5]

    Provable self-play algorithms for competitive reinforcement learning

    Yu Bai and Chi Jin. Provable self-play algorithms for competitive reinforcement learning. In International conference on machine learning, pages 551--560. PMLR, 2020

    2020

  6. [6]

    Multiplicative weights update in zero-sum games

    James P Bailey and Georgios Piliouras. Multiplicative weights update in zero-sum games. In Proceedings of the 2018 ACM Conference on Economics and Computation, pages 321--338, 2018

    2018

  7. [7]

    Finite regret and cycles with fixed step-size via alternating gradient descent-ascent

    James P Bailey, Gauthier Gidel, and Georgios Piliouras. Finite regret and cycles with fixed step-size via alternating gradient descent-ascent. In Conference on Learning Theory, pages 391--407. PMLR, 2020

    2020

  8. [8]

    Mastering the game of no-press diplomacy via human-regularized reinforcement learning and planning

    Anton Bakhtin, David J Wu, Adam Lerer, Jonathan Gray, Athul Paul Jacob, Gabriele Farina, Alexander H Miller, and Noam Brown. Mastering the game of no-press diplomacy via human-regularized reinforcement learning and planning. arXiv preprint arXiv:2210.05492, 2022

    arXiv

Show all 140 references
  1. [9]

    Reinforcement learning with general utilities: Simpler variance reduction and large state-action space

    Anas Barakat, Ilyas Fatkhullin, and Niao He. Reinforcement learning with general utilities: Simpler variance reduction and large state-action space. In International Conference on Machine Learning, pages 1753--1800. PMLR, 2023

    2023

  2. [10]

    Bauschke and Patrick L

    Heinz H. Bauschke and Patrick L. Combettes. Convex analysis and monotone operator theory in Hilbert spaces. Springer Science & Business Media, 2011

    2011

  3. [11]

    Dynamic programming and stochastic control processes

    Richard Bellman. Dynamic programming and stochastic control processes. Information and Control, 1 0 (3): 0 228--239, 1958. ISSN 0019-9958. doi:https://doi.org/10.1016/S0019-9958(58)80003-0. URL https://www.sciencedirect.com/science/article/pii/S0019995858800030

    1958 doi

  4. [12]

    Hidden convexity in some nonconvex quadratically constrained quadratic programming

    Aharon Ben-Tal and Marc Teboulle. Hidden convexity in some nonconvex quadratically constrained quadratic programming. Mathematical Programming, 72 0 (1): 0 51--63, 1996

    1996

  5. [13]

    On the role of constraints in the complexity of min-max optimization

    Martino Bernasconi, Matteo Castiglioni, Andrea Celli, and Gabriele Farina. On the role of constraints in the complexity of min-max optimization. arXiv preprint arXiv:2411.03248, 2024

    2024 arXiv

  6. [14]

    Dota 2 with large scale deep reinforcement learning

    Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemys aw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019

    1912 arXiv

  7. [15]

    Lyapunov theory for discrete time systems

    Nicoletta Bof, Ruggero Carli, and Luca Schenato. Lyapunov theory for discrete time systems. arXiv preprint arXiv:1809.05289, 2018

    2018 arXiv

  8. [16]

    Characterizations of ojasiewicz inequalities: subgradient flows, talweg, convexity

    J \'e r \^o me Bolte, Aris Daniilidis, Olivier Ley, and Laurent Mazet. Characterizations of ojasiewicz inequalities: subgradient flows, talweg, convexity. Transactions of the American Mathematical Society, 362 0 (6): 0 3319--3363, 2010

    2010

  9. [17]

    Collaborative multi-robot exploration

    Wolfram Burgard, Mark Moors, Dieter Fox, Reid Simmons, and Sebastian Thrun. Collaborative multi-robot exploration. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 1, pa...

    2000

  10. [18]

    Uncoupled and convergent learning in two-player zero-sum markov games with bandit feedback

    Yang Cai, Haipeng Luo, Chen-Yu Wei, and Weiqiang Zheng. Uncoupled and convergent learning in two-player zero-sum markov games with bandit feedback. Advances in Neural Information Processing Systems, 36: 0 36364--36406, 2023

    2023

  11. [19]

    Near-optimal policy optimization for correlated equilibrium in general-sum markov games

    Yang Cai, Haipeng Luo, Chen-Yu Wei, and Weiqiang Zheng. Near-optimal policy optimization for correlated equilibrium in general-sum markov games. In International Conference on Artificial Intelligence and Statistics, pages 3889--3897. PMLR, 2024

    2024

  12. [20]

    Faster last-iterate convergence of policy optimization in zero-sum markov games

    Shicong Cen, Yuejie Chi, Simon S Du, and Lin Xiao. Faster last-iterate convergence of policy optimization in zero-sum markov games. arXiv preprint arXiv:2210.01050, 2022

    2022 arXiv

  13. [21]

    Alternation makes the adversary weaker in two-player games

    Volkan Cevher, Ashok Cutkosky, Ali Kavis, Georgios Piliouras, Stratis Skoulakis, and Luca Viano. Alternation makes the adversary weaker in two-player games. Advances in Neural Information Processing Systems, 36: 0 18263--18290, 2023

    2023

  14. [22]

    Chambolle and Thomas Pock

    A. Chambolle and Thomas Pock. A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision, 40: 0 120--145, 2011. URL https://api.semanticscholar.org/CorpusID:261281173

    2011

  15. [23]

    On the theory of reinforcement learning with once-per-episode feedback

    Niladri Chatterji, Aldo Pacchiano, Peter Bartlett, and Michael Jordan. On the theory of reinforcement learning with once-per-episode feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  16. [24]

    Reducing noise in gan training with variance reduced extragradient

    Tatjana Chavdarova, Gauthier Gidel, Fran c ois Fleuret, and Simon Lacoste-Julien. Reducing noise in gan training with variance reduced extragradient. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alch\' e -Buc, E. Fox, and R. Garnett, editors, Advances in Neural Informati...

    2019

  17. [25]

    Stich, Francois Fleuret, and Martin Jaggi

    Tatjana Chavdarova, Matteo Pagliardini, Sebastian U. Stich, Francois Fleuret, and Martin Jaggi. Taming gans with lookahead-minmax, 2021. URL https://arxiv.org/abs/2006.14567

    2021 arXiv

  18. [26]

    Efficient algorithms for a class of stochastic hidden convex optimization and its applications in network revenue management

    Xin Chen, Niao He, Yifan Hu, and Zikun Ye. Efficient algorithms for a class of stochastic hidden convex optimization and its applications in network revenue management. Operations Research, 2024

    2024

  19. [27]

    Exploration-exploitation trade-off in reinforcement learning on online markov decision processes with global concave rewards

    Wang Chi Cheung. Exploration-exploitation trade-off in reinforcement learning on online markov decision processes with global concave rewards. arXiv preprint arXiv:1905.06466, 2019 a

    1905 arXiv

  20. [28]

    Regret minimization for reinforcement learning with vectorial feedback and complex objectives

    Wang Chi Cheung. Regret minimization for reinforcement learning with vectorial feedback and complex objectives. In Advances in Neural Information Processing Systems (NeurIPS), 2019 b

    2019

  21. [29]

    When are offline two-player zero-sum markov games solvable? Advances in Neural Information Processing Systems, 35: 0 25779--25791, 2022

    Qiwen Cui and Simon S Du. When are offline two-player zero-sum markov games solvable? Advances in Neural Information Processing Systems, 35: 0 25779--25791, 2022

    2022

  22. [30]

    Breaking the curse of multiagents in a large state space: Rl in markov games with independent linear function approximation

    Qiwen Cui, Kaiqing Zhang, and Simon Du. Breaking the curse of multiagents in a large state space: Rl in markov games with independent linear function approximation. In The Thirty Sixth Annual Conference on Learning Theory, pages 2651--2652. PMLR, 2023

    2023

  23. [31]

    Independent policy gradient methods for competitive reinforcement learning

    Constantinos Daskalakis, Dylan J Foster, and Noah Golowich. Independent policy gradient methods for competitive reinforcement learning. Advances in neural information processing systems, 33: 0 5527--5540, 2020

    2020

  24. [32]

    The complexity of constrained min-max optimization

    Constantinos Daskalakis, Stratis Skoulakis, and Manolis Zampetakis. The complexity of constrained min-max optimization. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 1466--1478, 2021

    2021

  25. [33]

    The complexity of markov equilibrium in stochastic games

    Constantinos Daskalakis, Noah Golowich, and Kaiqing Zhang. The complexity of markov equilibrium in stochastic games. In The Thirty Sixth Annual Conference on Learning Theory, pages 4180--4234. PMLR, 2023

    2023

  26. [34]

    On the complexity of computing markov perfect equilibrium in general-sum stochastic games

    Xiaotie Deng, Ningyuan Li, David Mguni, Jun Wang, and Yaodong Yang. On the complexity of computing markov perfect equilibrium in general-sum stochastic games. National Science Review, 10 0 (1): 0 nwac256, 2023

    2023

  27. [35]

    First-order methods of smooth convex optimization with inexact oracle

    Olivier Devolder, Fran c ois Glineur, and Yurii Nesterov. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming, 146: 0 37--75, 2014

    2014

  28. [36]

    Efficient methods for structured nonconvex-nonconcave min-max optimization

    Jelena Diakonikolas, Constantinos Daskalakis, and Michael I Jordan. Efficient methods for structured nonconvex-nonconcave min-max optimization. In International Conference on Artificial Intelligence and Statistics, pages 2746--2754. PMLR, 2021

    2021

  29. [37]

    Independent policy gradient for large-scale markov potential games: Sharper rates, function approximation, and game-agnostic convergence

    Dongsheng Ding, Chen-Yu Wei, Kaiqing Zhang, and Mihailo Jovanovic. Independent policy gradient for large-scale markov potential games: Sharper rates, function approximation, and game-agnostic convergence. In International Conference on Machine Learning, pages 5166--5220. PMLR, 2022

    2022

  30. [38]

    Error bounds, quadratic growth, and linear convergence of proximal methods

    Dmitriy Drusvyatskiy and Adrian S Lewis. Error bounds, quadratic growth, and linear convergence of proximal methods. Mathematics of Operations Research, 43 0 (3): 0 919--948, 2018

    2018

  31. [39]

    Efficiency of minimizing compositions of convex functions and smooth maps

    Dmitriy Drusvyatskiy and Courtney Paquette. Efficiency of minimizing compositions of convex functions and smooth maps. Mathematical Programming, 178 0 (1): 0 503--558, 2019

    2019

  32. [40]

    Reinforcement learning with trajectory feedback

    Yonathan Efroni, Nadav Merlis, and Shie Mannor. Reinforcement learning with trajectory feedback. In AAAI Conference on Artificial Intelligence, 2021

    2021

  33. [41]

    Regret minimization and convergence to equilibria in general-sum markov games

    Liad Erez, Tal Lancewicki, Uri Sherman, Tomer Koren, and Yishay Mansour. Regret minimization and convergence to equilibria in general-sum markov games. In International Conference on Machine Learning, pages 9343--9373. PMLR, 2023

    2023

  34. [42]

    Finite-dimensional variational inequalities and complementarity problems

    Francisco Facchinei and Jong-Shi Pang. Finite-dimensional variational inequalities and complementarity problems. Springer, 2003

    2003

  35. [43]

    Finite-dimensional variational inequalities and complementarity problems

    Francisco Facchinei and Jong-Shi Pang. Finite-dimensional variational inequalities and complementarity problems. Springer Science & Business Media, 2007

    2007

  36. [44]

    Stochastic optimization under hidden convexity

    Ilyas Fatkhullin, Niao He, and Yifan Hu. Stochastic optimization under hidden convexity. arXiv preprint arXiv:2401.00108, 2023

    2023 arXiv

  37. [45]

    Supply and demand functions in inventory models

    Qi Feng and J George Shanthikumar. Supply and demand functions in inventory models. Operations Research, 66 0 (1): 0 77--91, 2018

    2018

  38. [46]

    Equilibrium in a stochastic n -person game

    Arlington M Fink. Equilibrium in a stochastic n -person game. Journal of science of the hiroshima university, series ai (mathematics), 28 0 (1): 0 89--93, 1964

    1964

  39. [47]

    Concave utility reinforcement learning: The mean-field game viewpoint

    Matthieu Geist, Julien P \'e rolat, Mathieu Lauri \`e re, Romuald Elie, Sarah Perrin, Oliver Bachem, R \'e mi Munos, and Olivier Pietquin. Concave utility reinforcement learning: The mean-field game viewpoint. In International Conference on Autonomous Agents and Multiagent Sys...

    2022

  40. [48]

    Convex markov games: A framework for fairness, imitation, and creativity in multi-agent learning

    Ian Gemp, Andreas Haupt, Luke Marris, Siqi Liu, and Georgios Piliouras. Convex markov games: A framework for fairness, imitation, and creativity in multi-agent learning. arXiv preprint arXiv:2410.16600, 2024

    2024 arXiv

  41. [49]

    Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization

    Saeed Ghadimi, Guanghui Lan, and Hongchao Zhang. Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming, 155 0 (1): 0 267--305, 2016

    2016

  42. [50]

    Negative momentum for improved game dynamics

    Gauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, R \'e mi Le Priol, Gabriel Huang, Simon Lacoste-Julien, and Ioannis Mitliagkas. Negative momentum for improved game dynamics. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1802--1...

    2019

  43. [51]

    Generative adversarial nets

    Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014

    2014

  44. [52]

    Multi-agent deep reinforcement learning: a survey

    Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review, 55 0 (2): 0 895--943, 2022

    2022

  45. [53]

    Provably efficient maximum entropy exploration

    Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning (ICML), 2019

    2019

  46. [54]

    Inequity aversion improves cooperation in intertemporal social dilemmas

    Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar Due \ n ez-Guzman, Antonio Garc \' a Casta \ n eda, Iain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, Heather Roff, and Thore Graepel. Inequity aversion improves cooperation in intertemporal social dilemmas. A...

    2018

  47. [55]

    Proximal stochastic methods for nonsmooth nonconvex finite-sum optimization

    Sashank J Reddi, Suvrit Sra, Barnabas Poczos, and Alexander J Smola. Proximal stochastic methods for nonsmooth nonconvex finite-sum optimization. Advances in neural information processing systems, 29, 2016

    2016

  48. [56]

    What is local optimality in nonconvex-nonconcave minimax optimization? In International conference on machine learning, pages 4880--4889

    Chi Jin, Praneeth Netrapalli, and Michael Jordan. What is local optimality in nonconvex-nonconcave minimax optimization? In International conference on machine learning, pages 4880--4889. PMLR, 2020

    2020

  49. [57]

    V-learning--a simple, efficient, decentralized algorithm for multiagent rl

    Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-learning--a simple, efficient, decentralized algorithm for multiagent rl. arXiv preprint arXiv:2110.14555, 2021

    2021 arXiv

  50. [58]

    The complexity of infinite-horizon general-sum stochastic games

    Yujia Jin, Vidya Muthukumar, and Aaron Sidford. The complexity of infinite-horizon general-sum stochastic games. arXiv preprint arXiv:2204.04186, 2022

    2022 arXiv

  51. [59]

    Zero-sum polymatrix markov games: Equilibrium collapse and efficient computation of nash equilibria

    Fivos Kalogiannis and Ioannis Panageas. Zero-sum polymatrix markov games: Equilibrium collapse and efficient computation of nash equilibria. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volu...

    2023

  52. [60]

    Efficiently computing nash equilibria in adversarial team markov games

    Fivos Kalogiannis, Ioannis Anagnostides, Ioannis Panageas, Emmanouil-Vasileios Vlatakis-Gkaragkounis, Vaggos Chatziafratis, and Stelios Stavroulakis. Efficiently computing nash equilibria in adversarial team markov games. arXiv preprint arXiv:2208.02204, 2022

    2022 arXiv

  53. [61]

    Learning equilibria in adversarial team markov games: A nonconvex-hidden-concave min-max optimization problem

    Fivos Kalogiannis, Jingming Yan, and Ioannis Panageas. Learning equilibria in adversarial team markov games: A nonconvex-hidden-concave min-max optimization problem. arXiv preprint arXiv:2410.05673, 2024

    2024 arXiv

  54. [62]

    Linear convergence of gradient and proximal-gradient methods under the polyak- ojasiewicz condition

    Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradient methods under the polyak- ojasiewicz condition. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19...

    2016

  55. [63]

    An introduction to variational inequalities and their applications

    David Kinderlehrer and Guido Stampacchia. An introduction to variational inequalities and their applications. SIAM, 2000

    2000

  56. [64]

    Normalizing flows: An introduction and review of current methods

    Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43 0 (11): 0 3964--3979, 2020

    2020

  57. [65]

    G. M. Korpelevich. The extragradient method for finding saddle points and other problems. Ekonomika i Matematicheskie Metody, 12: 0 747--756, 1976

    1976

  58. [66]

    Openspiel: A framework for reinforcement learning in games

    Marc Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien P \'e rolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, et al. Openspiel: A framework for reinforcement learning in games. arXiv preprint arXiv:1908.09453, 2019

    1908 arXiv

  59. [67]

    Fundamental benefit of alternating updates in minimax optimization

    Jaewook Lee, Hanseul Cho, and Chulhee Yun. Fundamental benefit of alternating updates in minimax optimization. arXiv preprint arXiv:2402.10475, 2024

    2024 arXiv

  60. [68]

    Global convergence of multi-agent policy gradient in markov potential games

    Stefanos Leonardos, Will Overman, Ioannis Panageas, and Georgios Piliouras. Global convergence of multi-agent policy gradient in markov potential games. arXiv preprint arXiv:2106.01969, 2021

    2021

  61. [69]

    Hidden convex minimization

    Duan Li, Zhi-You Wu, Heung-Wing Joseph Lee, Xin-Min Yang, and Lian-Sheng Zhang. Hidden convex minimization. Journal of Global Optimization, 31: 0 211--233, 2005

    2005

  62. [70]

    Calculus of the exponent of kurdyka-- ojasiewicz inequality and its applications to linear convergence of first-order methods

    Guoyin Li and Ting Kei Pong. Calculus of the exponent of kurdyka-- ojasiewicz inequality and its applications to linear convergence of first-order methods. Foundations of computational mathematics, 18 0 (5): 0 1199--1232, 2018

    2018

  63. [71]

    H \"o lder continuity of perturbed solution set for convex optimization problems

    Xiao-Bing Li and SJ Li. H \"o lder continuity of perturbed solution set for convex optimization problems. Applied Mathematics and Computation, 232: 0 908--918, 2014

    2014

  64. [72]

    Error bounds, pl condition, and quadratic growth for weakly convex functions, and linear convergences of proximal point methods

    Feng-Yi Liao, Lijun Ding, and Yang Zheng. Error bounds, pl condition, and quadratic growth for weakly convex functions, and linear convergences of proximal point methods. In 6th Annual Learning for Dynamics & Control Conference, pages 993--1005. PMLR, 2024

    2024

  65. [73]

    On gradient descent ascent for nonconvex-concave minimax problems

    Tianyi Lin, Chi Jin, and Michael Jordan. On gradient descent ascent for nonconvex-concave minimax problems. In International conference on machine learning, pages 6083--6093. PMLR, 2020

    2020

  66. [74]

    Markov games as a framework for multi-agent reinforcement learning

    Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157--163. Elsevier, 1994

    1994

  67. [75]

    First-order convergence theory for weakly-convex-weakly-concave min-max problems

    Mingrui Liu, Hassan Rafique, Qihang Lin, and Tianbao Yang. First-order convergence theory for weakly-convex-weakly-concave min-max problems. Journal of Machine Learning Research, 22 0 (169): 0 1--34, 2021

    2021

  68. [76]

    Alternating gradient descent ascent for nonconvex min-max problems in robust learning and gans

    Songtao Lu, Rahul Singh, Xiangyi Chen, Yongxin Chen, and Mingyi Hong. Alternating gradient descent ascent for nonconvex min-max problems in robust learning and gans. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers, pages 680--684. IEEE, 2019

    2019

  69. [77]

    Error bounds and convergence analysis of feasible descent methods: a general approach

    Zhi-Quan Luo and Paul Tseng. Error bounds and convergence analysis of feasible descent methods: a general approach. Annals of Operations Research, 46 0 (1): 0 157--178, 1993

    1993

  70. [78]

    Towards deep learning models resistant to adversarial attacks

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017

    2017 arXiv

  71. [79]

    Martinet

    B. Martinet. Régularisation d’inéquations variationnelles par approximations successives. ESAIM: Mathematical Modelling and Numerical Analysis - Modélisation Mathématique et Analyse Numérique, 1970

    1970

  72. [80]

    Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile

    Panayotis Mertikopoulos, Bruno Lecouat, Houssam Zenati, Chuan-Sheng Foo, Vijay Chandrasekhar, and Georgios Piliouras. Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile. arXiv preprint arXiv:1807.02629, 2018

    2018 arXiv

  73. [81]

    Generalized natural gradient flows in hidden convex-concave games and gans

    Andjela Mladenovic, Iosif Sakos, Gauthier Gidel, and Georgios Piliouras. Generalized natural gradient flows in hidden convex-concave games and gans. In International Conference on Learning Representations, 2021

    2021

  74. [82]

    Playing atari with deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013

    2013 arXiv

  75. [83]

    Dynamic regret bounds for constrained online nonconvex optimization based on polyak--lojasiewicz regions

    Julie Mulvaney-Kemp, SangWoo Park, Ming Jin, and Javad Lavaei. Dynamic regret bounds for constrained online nonconvex optimization based on polyak--lojasiewicz regions. IEEE Transactions on Control of Network Systems, 10 0 (2): 0 599--611, 2022

    2022

  76. [84]

    Algaedice: Policy gradient from arbitrary experience

    Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074, 2019

    1912 arXiv

  77. [85]

    Prox-method with rate of convergence o(1/t) for variational inequalities with lipschitz continuous monotone operators

    Arkadi Nemirovski. Prox-method with rate of convergence o(1/t) for variational inequalities with lipschitz continuous monotone operators. SIAM Journal on Optimization, 15 0 (1): 0 229--251, 2004

    2004

  78. [86]

    Robust stochastic approximation approach to stochastic programming

    Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19 0 (4): 0 1574--1609, 2009

    2009

  79. [87]

    Smooth minimization of non-smooth functions

    Yu Nesterov. Smooth minimization of non-smooth functions. Mathematical programming, 103: 0 127--152, 2005

    2005

  80. [88]

    A unified view of entropy-regularized markov decision processes

    Gergely Neu, Anders Jonsson, and Vicen c G \'o mez. A unified view of entropy-regularized markov decision processes. arXiv preprint arXiv:1705.07798, 2017

    2017 arXiv

  81. [89]

    Solving a class of non-convex min-max games using iterative first order methods

    Maher Nouiehed, Maziar Sanjabi, Tianjian Huang, Jason D Lee, and Meisam Razaviyayn. Solving a class of non-convex min-max games using iterative first order methods. Advances in Neural Information Processing Systems, 32, 2019

    2019

  82. [90]

    Forward-backward splitting under the light of generalized convexity

    Konstantinos Oikonomidis, Emanuel Laude, and Panagiotis Patrinos. Forward-backward splitting under the light of generalized convexity. arXiv preprint arXiv:2503.18098, 2025

    2025 arXiv

  83. [91]

    The computational complexity of multi-player concave games and kakutani fixed points

    Christos Papadimitriou, Emmanouil-Vasileios Vlatakis-Gkaragkounis, and Manolis Zampetakis. The computational complexity of multi-player concave games and kakutani fixed points. In Proceedings of the 24th ACM Conference on Economics and Computation, 2023

    2023

  84. [92]

    Multi-player zero-sum markov games with networked separable interactions

    Chanwoo Park, Kaiqing Zhang, and Asuman Ozdaglar. Multi-player zero-sum markov games with networked separable interactions. Advances in Neural Information Processing Systems, 36: 0 37354--37369, 2023

    2023

  85. [93]

    Relative entropy policy search

    Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1607--1612, 2010

    2010

  86. [94]

    Polyak and Anatoli B

    Boris T. Polyak and Anatoli B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30 0 (4): 0 838--855, 1992

    1992

  87. [95]

    Markov decision processes: discrete stochastic dynamic programming

    Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  88. [96]

    Maximum margin planning

    Nathan D Ratliff, J Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In Proceedings of the 23rd international conference on Machine learning, pages 729--736, 2006

    2006

  89. [97]

    Fast convergence to non-isolated minima: four equivalent conditions for c 2 functions

    Quentin Rebjock and Nicolas Boumal. Fast convergence to non-isolated minima: four equivalent conditions for c 2 functions. Mathematical Programming, pages 1--49, 2024

    2024

  90. [98]

    Tyrrell Rockafellar

    R. Tyrrell Rockafellar. Monotone operators and the proximal point algorithm. SIAM Journal on Control and Optimization, 14 0 (5): 0 877--898, 1976

    1976

  91. [99]

    Variational analysis, volume 317

    R Tyrrell Rockafellar and Roger J-B Wets. Variational analysis, volume 317. Springer Science & Business Media, 2009

    2009

  92. [100]

    Coordination strategies for multi-robot exploration and mapping

    John G Rogers, Carlos Nieto-Granda, and Henrik I Christensen. Coordination strategies for multi-robot exploration and mapping. In Experimental Robotics: The 13th International Symposium on Experimental Robotics, pages 231--243. Springer, 2013

    2013

  93. [101]

    Efficient estimations from a slowly convergent robbins-monro process

    David Ruppert. Efficient estimations from a slowly convergent robbins-monro process. Technical Report, Cornell University, 1988

    1988

  94. [102]

    Exploiting hidden structures in non-convex games for convergence to nash equilibrium

    Iosif Sakos, Emmanouil-Vasileios Vlatakis-Gkaragkounis, Panayotis Mertikopoulos, and Georgios Piliouras. Exploiting hidden structures in non-convex games for convergence to nash equilibrium. Advances in Neural Information Processing Systems, 36, 2024

    2024

  95. [103]

    Decentralized q-learning in zero-sum markov games

    Muhammed Sayin, Kaiqing Zhang, David Leslie, Tamer Basar, and Asuman Ozdaglar. Decentralized q-learning in zero-sum markov games. Advances in Neural Information Processing Systems, 34: 0 18320--18334, 2021

    2021

  96. [104]

    Schneider and D

    S. Schneider and D. H. Wagner. Error detection in redundant systems. In Papers Presented at the February 26-28, 1957, Western Joint Computer Conference: Techniques for Reliability, IRE-AIEE-ACM '57 (Western), page 115–121, New York, NY, USA, 1957. Association for Computing Mac...

    1957

  97. [105]

    Safe, multi-agent, reinforcement learning for autonomous driving

    Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295, 2016

    2016 arXiv

  98. [106]

    Stochastic games

    Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39 0 (10): 0 1095--1100, 1953

    1953

  99. [107]

    Mastering the game of go without human knowledge

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550 0 (7676): 0 354--359, 2017

    2017

  100. [108]

    Reinforcement learning in robotic applications: a comprehensive survey

    Bharat Singh, Rajesh Kumar, and Vinay Pratap Singh. Reinforcement learning in robotic applications: a comprehensive survey. Artificial Intelligence Review, 55 0 (2): 0 945--990, 2022

    2022

  101. [109]

    When can we learn general-sum markov games with a large number of players sample-efficiently? arXiv preprint arXiv:2110.04184, 2021

    Ziang Song, Song Mei, and Yu Bai. When can we learn general-sum markov games with a large number of players sample-efficiently? arXiv preprint arXiv:2110.04184, 2021

    2021 arXiv

  102. [110]

    Forward--backward quasi-newton methods for nonsmooth optimization problems

    Lorenzo Stella, Andreas Themelis, and Panagiotis Patrinos. Forward--backward quasi-newton methods for nonsmooth optimization problems. Computational Optimization and Applications, 67 0 (3): 0 443--487, 2017

    2017

  103. [111]

    A game-theoretic approach to apprenticeship learning

    Umar Syed and Robert E Schapire. A game-theoretic approach to apprenticeship learning. Advances in neural information processing systems, 20, 2007

    2007

  104. [112]

    Solving heads-up limit texas hold'em

    Oskari Tammelin, Neil Burch, Michael Johanson, and Michael Bowling. Solving heads-up limit texas hold'em. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI'15, page 645–652. AAAI Press, 2015. ISBN 9781577357384

    2015

  105. [113]

    Deep reinforcement learning for decentralized multi-robot exploration with macro actions

    Aaron Hao Tan, Federico Pizarro Bejarano, Yuhan Zhu, Richard Ren, and Goldie Nejat. Deep reinforcement learning for decentralized multi-robot exploration with macro actions. IEEE Robotics and Automation Letters, 8 0 (1): 0 272--279, 2022

    2022

  106. [114]

    J v. Neumann. Zur theorie der gesellschaftsspiele. Mathematische annalen, 100 0 (1): 0 295--320, 1928

    1928

  107. [115]

    Solving min-max optimization with hidden structure via gradient descent ascent

    Emmanouil-Vasileios Vlatakis-Gkaragkounis, Lampros Flokas, and Georgios Piliouras. Solving min-max optimization with hidden structure via gradient descent ascent. Advances in Neural Information Processing Systems, 34: 0 2373--2386, 2021

    2021

  108. [116]

    Team-maxmin equilibria

    Bernhard Von Stengel and Daphne Koller. Team-maxmin equilibria. Games and Economic Behavior, 21 0 (1-2): 0 309--321, 1997

    1997

  109. [117]

    The hidden convex optimization landscape of regularized two-layer relu networks: an exact characterization of optimal solutions

    Yifei Wang, Jonathan Lacotte, and Mert Pilanci. The hidden convex optimization landscape of regularized two-layer relu networks: an exact characterization of optimal solutions. In International Conference on Learning Representations, 2021

    2021

  110. [118]

    Breaking the curse of multiagency: Provably efficient decentralized multi-agent rl with function approximation

    Yuanhao Wang, Qinghua Liu, Yu Bai, and Chi Jin. Breaking the curse of multiagency: Provably efficient decentralized multi-agent rl with function approximation. In The Thirty Sixth Annual Conference on Learning Theory, pages 2793--2848. PMLR, 2023

    2023

  111. [119]

    Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive markov games

    Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive markov games. In Conference on learning theory, pages 4259--4299. PMLR, 2021

    2021

  112. [120]

    Alternating mirror descent for constrained min-max games

    Andre Wibisono, Molei Tao, and Georgios Piliouras. Alternating mirror descent for constrained min-max games. Advances in Neural Information Processing Systems, 35: 0 35201--35212, 2022

    2022

  113. [121]

    Simple statistical gradient-following algorithms for connectionist reinforcement learning

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8: 0 229--256, 1992

    1992

  114. [122]

    Multi-step alignment as markov games: An optimistic online gradient descent approach with convergence guarantees

    Yongtao Wu, Luca Viano, Yihang Chen, Zhenyu Zhu, Kimon Antonakopoulos, Quanquan Gu, and Volkan Cevher. Multi-step alignment as markov games: An optimistic online gradient descent approach with convergence guarantees. arXiv preprint arXiv:2502.12678, 2025

    2025 arXiv

  115. [123]

    Peeling off a nonconvex cover of an actual convex problem: hidden convexity

    Zhi-You Wu, Duan Li, Lian-Sheng Zhang, and XM Yang. Peeling off a nonconvex cover of an actual convex problem: hidden convexity. SIAM Journal on Optimization, 18 0 (2): 0 507--536, 2007

    2007

  116. [124]

    A survey of hidden convex optimization

    Yong Xia. A survey of hidden convex optimization. Journal of the Operations Research Society of China, 8 0 (1): 0 1--28, 2020

    2020

  117. [125]

    Global convergence and variance reduction for a class of nonconvex-nonconcave minimax problems

    Junchi Yang, Negar Kiyavash, and Niao He. Global convergence and variance reduction for a class of nonconvex-nonconcave minimax problems. Advances in Neural Information Processing Systems, 33: 0 1153--1165, 2020

    2020

  118. [126]

    Faster single-loop algorithms for minimax optimization without strong concavity

    Junchi Yang, Antonio Orvieto, Aurelien Lucchi, and Niao He. Faster single-loop algorithms for minimax optimization without strong concavity. In International Conference on Artificial Intelligence and Statistics, pages 5485--5517. PMLR, 2022

    2022

  119. [127]

    O(t-1) convergence of optimistic-follow-the-regularized-leader in two-player zero-sum markov games

    Yuepeng Yang and Cong Ma. O(t-1) convergence of optimistic-follow-the-regularized-leader in two-player zero-sum markov games. arXiv preprint arXiv:2209.12430, 2022

    2022 arXiv

  120. [128]

    Policy-based primal-dual methods for convex constrained markov decision processes

    Donghao Ying, Mengzi Amy Guo, Yuhao Ding, Javad Lavaei, and Zuo-Jun Shen. Policy-based primal-dual methods for convex constrained markov decision processes. Proceedings of the AAAI Conference on Artificial Intelligence, 37: 0 10963--10971, 2023

    2023

  121. [129]

    Reward is enough for convex mdps

    Tom Zahavy, Brendan O’Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex mdps. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  122. [130]

    Discovering policies with domino: Diversity optimization maintaining near optimality

    Tom Zahavy, Yannick Schroecker, Feryal Behbahani, Kate Baumli, Sebastian Flennerhag, Shaobo Hou, and Satinder Singh. Discovering policies with domino: Diversity optimization maintaining near optimality. arXiv preprint arXiv:2205.13521, 2022

    2022 arXiv

  123. [131]

    Diversifying ai: Towards creative chess with alphazero

    Tom Zahavy, Vivek Veeriah, Shaobo Hou, Kevin Waugh, Matthew Lai, Edouard Leurent, Nenad Tomasev, Lisa Schut, Demis Hassabis, and Satinder Singh. Diversifying ai: Towards creative chess with alphazero. arXiv preprint arXiv:2308.09175, 2023

    2023 arXiv

  124. [132]

    Regularized gradient descent ascent for two-player zero-sum markov games

    Sihan Zeng, Thinh Doan, and Justin Romberg. Regularized gradient descent ascent for two-player zero-sum markov games. Advances in Neural Information Processing Systems, 35: 0 34546--34558, 2022

    2022

  125. [133]

    Convergence of gradient methods on bilinear zero-sum games

    Guojun Zhang and Yaoliang Yu. Convergence of gradient methods on bilinear zero-sum games. In International Conference on Learning Representations, 2019. URL https://api.semanticscholar.org/CorpusID:211069144

    2019

  126. [134]

    Variational policy gradient method for reinforcement learning with general utilities

    Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvari, and Mengdi Wang. Variational policy gradient method for reinforcement learning with general utilities. Advances in Neural Information Processing Systems, 33: 0 4572--4583, 2020

    2020

  127. [135]

    On the convergence and sample efficiency of variance-reduced policy gradient method

    Junyu Zhang, Chengzhuo Ni, Zheng Yu, Csaba Szepesvari, and Mengdi Wang. On the convergence and sample efficiency of variance-reduced policy gradient method. Advances in Neural Information Processing Systems, 34: 0 2228--2240, 2021

    2021

  128. [136]

    Policy optimization for markov games: Unified framework and faster convergence

    Runyu Zhang, Qinghua Liu, Huan Wang, Caiming Xiong, Na Li, and Yu Bai. Policy optimization for markov games: Unified framework and faster convergence. Advances in Neural Information Processing Systems, 35: 0 21886--21899, 2022 a

    2022

  129. [137]

    On the global convergence rates of decentralized softmax gradient play in markov potential games

    Runyu Zhang, Jincheng Mei, Bo Dai, Dale Schuurmans, and Na Li. On the global convergence rates of decentralized softmax gradient play in markov potential games. Advances in Neural Information Processing Systems, 35: 0 1923--1935, 2022 b

    1923

  130. [138]

    Universal gradient descent ascent method for nonconvex-nonconcave minimax optimization

    Taoli Zheng, Linglingzhi Zhu, Anthony Man-Cho So, Jos \'e Blanchet, and Jiajin Li. Universal gradient descent ascent method for nonconvex-nonconcave minimax optimization. Advances in Neural Information Processing Systems, 36: 0 54075--54110, 2023

    2023

  131. [139]

    Maximum entropy inverse reinforcement learning

    Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433--1438. Chicago, IL, USA, 2008

    2008

  132. [140]

    Online learning in episodic markovian decision processes by relative entropy policy search

    Alexander Zimin and Gergely Neu. Online learning in episodic markovian decision processes by relative entropy policy search. Advances in neural information processing systems, 26, 2013

    2013

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.