Pith. sign in

REVIEW 4 major objections 4 minor 57 references

Continuous-Time Analysis of Heavy Ball Momentum in Min-Max Games

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In min-max games with strong player interaction, smaller heavy-ball momentum widens the local-convergence step-size range, while alternating updates add an O(h) acceleration.

desk verdict The continuous-time ODE models are a real contribution, but the paper's headline claim about discrete Sim-HB is falsified by a simple quadratic counterexample, so the local convergence story needs major revision before it can be trusted. read the letter →

arxiv 2505.19537 v1 pith:7G7URVBU submitted 2025-05-26 cs.GT cs.LG

Yi Feng , Kaito Fujii , Stratis Skoulakis , Xiao Wang , Volkan Cevher This is my paper · ORCID
classification cs.GTcs.LG MSC 91A1065K1034D20
keywords heavyballmomentummin-maxgamescontinuous-timeanalysislocalconvergenceimplicitgradientregularizationalternatingupdatesnegativeGANtrainingdynamics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to explain how heavy-ball momentum behaves in two-player zero-sum (min-max) games, where theory has lagged far behind practice. It claims that when player interaction dominates the local dynamics, smaller momentum — including negative values — expands the range of step sizes for which simultaneous heavy-ball updates converge locally, while positive momentum is optimal only for very small step sizes. Alternating updates add an O(h) acceleration over simultaneous ones under the same conditions. Globally, smaller momentum steers trajectories toward shallower-slope regions of the min-max landscape, and alternating updates amplify this effect. These conclusions reverse the familiar minimization picture, and the paper supports them with continuous-time models, eigenvalue analysis, and experiments including GAN training.

What carries the argument

The central objects are first-order continuous-time ODEs built from a modified loss $F(x,y) = \frac{1}{1-\beta} f(x,y) + \frac{h(1+\beta)}{4(1-\beta)^3}\left(\|\nabla_x f\|^2 - \|\nabla_y f\|^2\right)$. These ODEs are first-order rather than second-order so that momentum and step size can vary independently and negative momentum is allowed. The Jacobians at equilibrium reduce to $J_S = \left(\frac{1}{1-\beta}I - \frac{h(1+\beta)}{2(1-\beta)^3}J\right)J$ for simultaneous updates and $J_A = J_S - \frac{h}{(1-\beta)^2}P$ for alternating updates, where $J = S + A$ decomposes into a potential part and a Hamiltonian part; the assumption $A \gg S$ makes the eigenvalues of $J$ have imaginary parts larger than their real parts. This polynomial-and-perturbation structure turns local convergence into a one-dimensional eigenvalue inequality and yields the alternating-update acceleration through first-order matrix perturbation theory. Theorem 3.1 supplies the $O(h^3)$ finite-horizon fidelity that connects these ODEs to the discrete Sim-HB and Alt-HB algorithms.

What would settle it

Run discrete Sim-HB and Alt-HB on a quadratic game f(x,y) = (1/2)x^T Q x + x^T A y - (1/2)y^T R y with Q and R positive semidefinite having many zero eigenvalues and A full rank with distinct singular values, using $\beta$ = -0.9 and a step size h just below the bound in Theorem 4.6. If Sim-HB diverges while the continuous-time model predicts convergence, or if for very small h the alternating update converges more slowly than the simultaneous one, the transfer from ODE to algorithm, and with it the paper's asymptotic claims, fails.

Watch

Extended reading notes

Core claim

The central claim is that momentum plays an opposite role in min-max games with dominant player interactions from the role it plays in minimization. Concretely, the paper proves local convergence of the continuous-time heavy-ball model when the step size satisfies the inequality in Theorem 4.6, and since the factor $2(1-\beta)^2/(1+\beta)$ decreases as $\beta$ decreases, more negative momentum enlarges the admissible step-size set. For sufficiently small step sizes, however, the optimal momentum parameter is positive. The alternating-update Jacobian is a perturbation of the simultaneous one, and its eigenvalue with largest real part is smaller by $h/(1-\beta)^2 |\sigma|^2$ plus $O(h^2 + h\alpha)$, giving an exponential acceleration. Globally, the paper argues and experimentally supports the thesis that smaller momentum and alternating updates direct trajectories toward regions of lower gradient norm, an implicit regularization effect that also appears to correlate with better GAN training outcomes.

Load-bearing premise

The asymptotic conclusions assume the continuous-time ODEs remain faithful to the discrete Sim-HB and Alt-HB iterations for all time, although the approximation theorem only proves O($h^{3}$) closeness over finite horizons after a warm-up of O(log h / log|$\beta$|) steps.

Editorial extensions

If this is right

  • Negative momentum should not be seen as a failure mode: under the paper's interaction-dominance assumption, it is the parameter that keeps simultaneous heavy-ball updates stable at large step sizes.
  • For very small step sizes, users should switch to positive momentum to obtain the fastest local convergence rate; the optimal momentum crosses from negative to positive as the step size shrinks.
  • When the interaction matrix is full rank with distinct singular values and both the step size and the potential part are small, alternating updates converge exponentially faster than simultaneous updates for the same step size and momentum.
  • The implicit-gradient-regularization thesis implies that in interaction-dominated games, smaller momentum and alternating updates systematically reduce the average gradient norm along trajectories, including on limit cycles.
  • In GAN training with Adam-style heavy-ball momentum, trajectories with lower average slope also achieve better FID and inception scores, suggesting that negative momentum improves training stability through implicit regularization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the finite-horizon ODE fidelity extends to unbounded time, the local threshold in Corollary 4.7 translates directly into a tuning rule for Adam-style algorithms: choose negative beta-one when the interaction term dominates, and switch to positive beta-one only after shrinking the step size.
  • The alternating-update eigenvalue shift predicts a regret or cycle-amplitude separation between alternating and simultaneous momentum methods in interaction-dominated games; the paper's experiments suggest the separation, but a regret-rate proof is an unproved extension.
  • The average-slope and FID link observed in the GAN experiments could be used as a testable training signal: monitor cumulative average slope and reduce momentum when it rises, treating slope as a stability indicator rather than a convergence metric.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper derives first-order continuous-time models (Continuous Sim-HB and Continuous Alt-HB) for simultaneous and alternating heavy-ball updates in smooth min-max games, using a modified loss function F(x,y). Theorem 3.1 states that these ODEs track the discrete algorithms locally with O(h^3) error on finite horizons after a warm-up phase. The local analysis then studies the Jacobians of these ODEs at local Nash equilibria and claims: (Corollary 4.7) smaller momentum enlarges the step-size range for local convergence, (Theorem 4.8) positive momentum is optimal at sufficiently small step sizes, and (Theorem 4.9) alternating updates are locally faster when player interaction dominates. Section 5 argues, via coefficient inspection and experiments, that smaller momentum and alternating updates drive trajectories to lower gradient-norm regions, and GAN experiments are presented as supporting evidence. The paper's headline is that all these effects are opposite to the known role of momentum in minimization.

Significance. If the local results were established for the original discrete algorithms, the paper would offer a clean and somewhat surprising explanation of negative momentum in min-max games, complementing Gidel et al. (2019) and extending it beyond bilinear games. The Jacobian computations in Section 4.2 and the matrix-perturbation strategy in Appendix H are elegant, and Proposition 3.2 correctly recovers earlier bilinear-game behavior. The O(h^3) local-error construction is a genuine modeling contribution. However, the paper's central claims about algorithmic stability are currently only proved for the continuous-time models, and the discrete transfer is not justified; a concrete quadratic example even contradicts the abstract's unqualified 'smaller momentum' claim for Sim-HB. Since these issues affect the main message, the paper needs major revision, but the continuous-time core appears salvageable.

major comments (4)
  1. [§3, Theorem 3.1 and §4] Theorem 3.1 only establishes O(h^3) closeness on finite time horizons, after a warm-up of 4 log h / log|β| steps, and it does not provide any uniform-in-time or asymptotic error bound. Section 4 nevertheless applies Proposition 2.1 to the continuous Jacobians JS and JA and interprets their stability as algorithmic stability, and Section 5 interprets long-time trajectory statistics from the ODEs. This transfer is load-bearing: without it, the local convergence, divergence, and implicit-regularization claims are statements about the ODEs only. The warm-up formula is also undefined for β = 0, which is the GDA case used in comparisons. Please either prove a transfer result (for example, a Lyapunov function shared by the discrete and continuous dynamics under the stated assumptions) or explicitly re-scope all asymptotic claims to the continuous-time models.
  2. [Corollary 4.7 / Abstract] The abstract and Section 4.2 present Corollary 4.7 as a statement about algorithmic stability, but as a statement about discrete Sim-HB it is false. Consider f(x,y) = 0.5 x^2 - 0.5 y^2 + 2xy. At (0,0), J = [[-1,-2],[2,-1]] has eigenvalues -1 ± 2i, so Assumptions 4.4 and 4.5 hold. The Sim-HB linearization has characteristic polynomial μ^2 - (1+β+hλ)μ + β = 0, and the Schur-Cohn threshold is h_D(β) = 2(1-β)^2(1+β)/((1-β)^2 + 4(1+β)^2), giving h_D(0)=0.400 and h_D(-0.9)=0.198. Thus at h=0.2, Sim-HB with β=0 converges while Sim-HB with β=-0.9 diverges, even though the continuous-time thresholds satisfy the opposite ordering and both continuous ODEs converge at this h. The unqualified claim that smaller momentum enables convergence across a wider range of step sizes is therefore not merely missing a proof; it is contradicted by the discrete dynamics within the paper's own assumption class.
  3. [§G, proof of Theorem 4.8] The proof of Theorem 4.8 minimizes the per-eigenvalue function g_λ(β) individually, but the actual convergence rate of Continuous Sim-HB is max_{λ∈Sp(J)} g_λ(β), as stated in Eq. (66). Minimizing each term separately does not minimize the maximum over the spectrum unless additional structure is supplied; different eigenvalues can have different optimal β values, and the max may be minimized at a point that is not optimal for any single eigenvalue. The theorem should either prove the statement for the spectral maximum (for example, under a uniform ratio condition on the eigenvalues) or be re-stated as a per-eigenmode result.
  4. [Appendix H, Eq. (4) and Theorem 4.9] The first-order correction in Theorem 4.9 appears to be off by a factor of 2. With the stated eigenvectors and the reciprocal normalization W*V = I, the diagonal entry of W*PV for P = [[0,0],[∇yx∇²_x, ∇yx∇xy]] is (1/2)|σ_j|^2 + O(α), not |σ_j|^2. A decisive check is the bilinear case β=0 in Proposition 3.2: the exact Jacobian (52) has eigenvalues with zero real part, whereas Eq. (4) would predict -h|σ|^2/(1-β)^2 + O(h^2). The displayed calculation in Appendix H appears to insert the Hamiltonian matrix instead of P, and the formula for the eigenvalue real part in Lemma C.2 also has a denominator that does not match the trace computation. The qualitative O(h) acceleration conclusion is unaffected, but the quantitative statement in Eq. (4) needs correction.
minor comments (4)
  1. [§3, Theorem 3.1] The warm-up condition 4 log h / log|β| requires β ≠ 0; please state this assumption explicitly and discuss β = 0 separately, since the GDA case is used in comparisons.
  2. [§4.1, Assumption 4.4] Assumption 4.4 is informal: 'A ≫ S' is not defined as a norm inequality, and the implication that every eigenvalue of J satisfies |Im(λ)| > |Re(λ)| does not follow from A dominating S without further conditions. Consider stating the spectral condition directly as the assumption.
  3. [§1.2 and §5.2] There are several typos and awkward phrases, including 'boarder range', 'purposed', 'Future details', and 'ar presented in Appendix A'. These do not affect the mathematics but should be cleaned up.
  4. [§5.2, AvgSlope definition] The average-slope definition integrates over trajectory length |S(β)|, but the experimental measure in Figure 4 is a cumulative average over iterations. Please clarify the relationship between the two quantities in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the heavy-ball min-max results are genuine consequences of the paper's explicitly constructed ODE models, and the finite-horizon approximation transfer is a rigor gap rather than a circular reduction.

full rationale

The derivation chain is self-contained once the continuous-time models are accepted. The modified loss F and the Continuous Sim-HB/Alt-HB equations are constructed by matching the discrete Sim-HB/Alt-HB updates to O(h^3) local error (Theorem 3.1 and Appendix B), and the Jacobians JS and JA in Propositions 4.2-4.3 are then computed from those models. Theorems 4.6-4.9 are genuine spectral consequences of those Jacobians under Assumptions 4.4 and 4.5; Corollary 4.7 follows algebraically from the monotonicity of the threshold 2(1-beta)^2/(1+beta), and Theorem 4.8 from the derivative calculation in Appendix G. Nothing is fitted to the conclusion. The implicit-regularization thesis in Section 5.1 also reads off the signs and beta-dependence of coefficients that were derived from the Taylor matching, not imposed ad hoc to produce the desired direction. Assumption 4.5's spectral consequence is imported from Wang & Chizat (2024), an external result, and the other central citations (Gidel et al., Ghosh et al., Muehlebach & Jordan) are not self-citations. The main weakness is a transfer/fidelity gap, not circularity: Theorem 3.1 guarantees only finite-horizon O(h^3) closeness after a warm-up of 4 log h / log|beta| steps, while local convergence is concluded from Proposition 2.1 on unbounded horizons; the skeptical 2D quadratic counterexample shows discrete Sim-HB can behave differently from the ODE at moderate h. That is an unproved uniformity assumption in the discrete-time claim, not an equivalence-by-construction of the paper's own equations.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

The central claims rest on explicit domain assumptions (dominant interaction, generic kernel condition, bounded derivatives) and on an informal ODE-as-proxy assumption for asymptotic behavior. There are no fitted parameters: h and beta are algorithm inputs, and the alpha scaling in Theorem 4.9 is a bookkeeping device.

assumptions (5)
  • domain assumption The payoff function f(x,y) is smooth with bounded derivatives up to the fourth order.
    Required by Theorem 3.1 for the O(h^3) finite-time approximation; stated in Theorem 3.1.
  • domain assumption Assumption 4.4: A >> S in the decomposition J = S + A, formalized as |Im(lambda)| > |Re(lambda)| for all lambda in Sp(J).
    Central to Theorem 4.6, Corollary 4.7, and the global thesis; restricts to games with dominant player interaction. The implication from operator inequality to eigenvalue inequality is asserted without proof.
  • domain assumption Assumption 4.5: EigVec(A) intersect Ker(S) = {0}.
    Imported from Wang and Chizat (2024); guarantees Re(lambda) < 0 for Gradient Flow, used in Theorems 4.6 and 4.8.
  • domain assumption In Theorem 4.9, m = n and the cross-Hessian nabla_xy f has full rank with distinct singular values.
    Technical conditions for the matrix perturbation calculation and the SVD-based eigenvector construction.
  • ad hoc to paper The ODE models remain faithful proxies for the discrete algorithms over infinite time horizons (stability, limit cycles, slope statistics).
    Theorem 3.1 only proves finite-time O(h^3) closeness after a warm-up; the paper draws asymptotic conclusions from the ODE phase portrait without a transfer theorem. This is implicitly assumed in Sections 4 and 5.
invented entities (1)
  • Modified loss function F(x,y) = f(x,y)/(1-beta) + h(1+beta)/(4(1-beta)^3) * (||grad_x f||^2 - ||grad_y f||^2) independent evidence
    purpose: Defines the continuous-time models as gradient flow on F; encodes the O(h^3) correction of discrete heavy ball updates.
    The model is validated by Theorem 3.1 (O(h^3) approximation) and by numerical experiments (Figures 5 and 6); for beta = 0 it reduces to the model of Rosca et al. (2021).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continuous-Time Analysis of Heavy Ball Momentum in Min-Max Games." pith.science (2026). https://pith.science/paper/7G7URVBU

@misc{pith2026250519537,
  author       = {Pith},
  title        = {Pith review of: Continuous-Time Analysis of Heavy Ball Momentum in Min-Max Games},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7G7URVBU}},
  note         = {Machine review of arXiv:2505.19537}
}
read the original abstract

Since Polyak's pioneering work, heavy ball (HB) momentum has been widely studied in minimization. However, its role in min-max games remains largely unexplored. As a key component of practical min-max algorithms like Adam, this gap limits their effectiveness. In this paper, we present a continuous-time analysis for HB with simultaneous and alternating update schemes in min-max games. Locally, we prove smaller momentum enhances algorithmic stability by enabling local convergence across a wider range of step sizes, with alternating updates generally converging faster. Globally, we study the implicit regularization of HB, and find smaller momentum guides algorithms trajectories towards shallower slope regions of the loss landscapes, with alternating updates amplifying this effect. Surprisingly, all these phenomena differ from those observed in minimization, where larger momentum yields similar effects. Our results reveal fundamental differences between HB in min-max games and minimization, and numerical experiments further validate our theoretical results.

Figures

Figures reproduced from arXiv: 2505.19537 by the authors.

Figure 1
Figure 1. Comparison of O(h 3 ) and O(h 2 )-local error models with payoff function f(x, y) = xy. any given finite time horizon, the solution trajectories (x(t), y(t)) of Continuous Sim-HB (resp. Continuous Alt￾HB) is locally O(h 3 )-close to the trajectories of Sim-HB (resp. Alt-HB) after 4 log h/ log|β| steps. To highlight the advantages of our O(h 3 ) local error models, we compare them with existing continuous-time approx… view at source ↗
Figure 2
Figure 2. Distribution on the eigenvalues’ maximal real part of JS, which governs the local behaviors according to Proposition 2.1. The black region indicates divergence for the corresponding parameters. Smaller momentum expands the range of step sizes for convergence, supporting Corollary 4.7. For small step sizes, the optimal momentum is positive, consistent with Theorem 4.8. In minimization, it is well known that larger mo… view at source ↗
Figure 3
Figure 3. (a) and 3(c) show that smaller β guides the sample trajectories toward regions with lower slopes [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Experimental results for GANs training dynamics. Smaller momentum and alternating updates lead the trajectories to lower average slopes. Trajectories with lower average slopes also have lower FID, indicating the better GANs training outcome. 6. Conclusions and Future D…
Figure 5
Figure 5. Figure 5: The test function is f(x, y) = x(y − 0.45) + ϕ(x) − ϕ(y), ϕ(z) = 1 4 z 2 − 1 2 z 4 + 1 6 z 6 . This function is also used by top left of [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: The test function is f(x, y) = xy + ϕ(x) − ϕ(y), ϕ(z) = 1 2 z 2 − 1 4 z 4 + 1 6 z 6 − 1 8 z 8 . This function is also used by bottom left of [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Experimental results for Proposition F.1. G. Proof of Theorem 4.8 Recall from Proposition 2.1 and Proposition 4.2, the local convergence rate of Continuous Sim-HB is determined by max λ∈Sp(J ) 1 1 − β ℜ(λ) + h(1 + β) 2(1 − β) 3 [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]
Figure 8
Figure 8. Figure 8: Results for experiments on problems with dimension 20 (first row) and dimension 200 (second row). In each row, from left to right, the step sizes increase from small to large [PITH_FULL_IMAGE:figures/full_fig_p035_8.png]
Figure 9
Figure 9. Figure 9: Results for experiments on problems with dimension 20 (first row) and dimension 200 (second row). In each row, from left to right, the step sizes increase from small to large. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_9.png]
Figure 10
Figure 10. Figure 10: Experimental results on problems with dimension 100 for Theorem 4.9 [PITH_FULL_IMAGE:figures/full_fig_p036_10.png]
Figure 11
Figure 11. Figure 11: Experimental results on problems with dimension 1000 for Theorem 4.9. (a) Evolution of distances from equilibrium (b) Evolution of trajectories [PITH_FULL_IMAGE:figures/full_fig_p036_11.png]
Figure 12
Figure 12. Figure 12: Illustration for Example I.1, simultaneous updates converge faster than alternating updates. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_12.png]
Figure 13
Figure 13. Figure 13: Inception Scores of Wasserstain GANs training. Smaller momentum leads to a larger inception score, indicating that they have better performance. Moreover, alternating updates lead to a better inception score compared to simultaneous updates. 38 [PITH_FULL_IMAGE:figur…
Figure 14
Figure 14. Figure 14: (b), both the simultaneous updates and alternating updates use a momentum parameter β = −0.5. From [PITH_FULL_IMAGE:figures/full_fig_p039_14.png]
Figure 15
Figure 15. Figure 15: Sampled images of Wasserstein GANs with CIFAR-10 dataset, trained by alternating updates [PITH_FULL_IMAGE:figures/full_fig_p040_15.png]
Figure 16
Figure 16. Figure 16: Sampled images of Wasserstein GANs with CIFAR-10 dataset, trained by simultaneous updates. 40 [PITH_FULL_IMAGE:figures/full_fig_p040_16.png]
Figure 17
Figure 17. Figure 17: Sampled images of Vanilla GANs with MNIST dataset, trained by alternating updates [PITH_FULL_IMAGE:figures/full_fig_p041_17.png]
Figure 18
Figure 18. Figure 18: Sampled images of Vanilla GANs with MNIST dataset, trained by simultaneous updates. 41 [PITH_FULL_IMAGE:figures/full_fig_p041_18.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 45 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Accelerating smooth games by manipulating spectral shapes

    Azizian, W., Scieur, D., Mitliagkas, I., Lacoste-Julien, S., and Gidel, G. Accelerating smooth games by manipulating spectral shapes. In International Conference on Artificial Intelligence and Statistics, pp.\ 1705--1715. PMLR, 2020

    work page 2020

  3. [3]

    P., Gidel, G., and Piliouras, G

    Bailey, J. P., Gidel, G., and Piliouras, G. Finite regret and cycles with fixed step-size via alternating gradient descent-ascent. In Conference on Learning Theory, pp.\ 391--407. PMLR, 2020

    work page 2020

  4. [4]

    A tutorial on matrix perturbation theory (using compact matrix notation)

    Bamieh, B. A tutorial on matrix perturbation theory (using compact matrix notation). arXiv preprint arXiv:2002.05001, 2020

    arXiv 2002

  5. [5]

    Barrett, D. G. and Dherin, B. Implicit gradient regularization. ICLR, 2021

    work page 2021

  6. [6]

    M., Liu, T., Islamov, R., Proske, F

    Compagnoni, E. M., Liu, T., Islamov, R., Proske, F. N., Orvieto, A., and Lucchi, A. Adaptive methods through the lens of sdes: Theoretical insights on the role of noise. arXiv preprint arXiv:2411.15958, 2024 a

    arXiv 2024

  7. [7]

    M., Orvieto, A., Kersting, H., Proske, F., and Lucchi, A

    Compagnoni, E. M., Orvieto, A., Kersting, H., Proske, F., and Lucchi, A. Sdes for minimax optimization. In International Conference on Artificial Intelligence and Statistics, pp.\ 4834--4842. PMLR, 2024 b

    work page 2024

  8. [8]

    Training GAN s with optimism

    Daskalakis, C., Ilyas, A., Syrgkanis, V., and Zeng, H. Training GAN s with optimism. arXiv preprint arXiv:1711.00141, 2017

    arXiv 2017

Show all 57 references
  1. [9]

    H., Nguyen, D., Nguyen, D., Harikumar, H., Tran, T., Rana, S., and Venkatesh, S

    Do, K., Le, T. H., Nguyen, D., Nguyen, D., Harikumar, H., Tran, T., Rana, S., and Venkatesh, S. Momentum adversarial distillation: Handling large distribution shifts in data-free knowledge distillation. Advances in Neural Information Processing Systems, 35: 0 10055--10067, 2022

    2022

  2. [10]

    Rapid learning in constrained minimax games with negative momentum

    Fang, Z., Liu, Z., Yu, C., and Hu, C. Rapid learning in constrained minimax games with negative momentum. AAAI, 2025

    2025

  3. [11]

    Prediction accuracy of learning in games: Follow-the-regularized-leader meets heisenberg

    Feng, Y., Piliouras, G., and Wang, X. Prediction accuracy of learning in games: Follow-the-regularized-leader meets heisenberg. ICML, 2024

    2024

  4. [12]

    Sharpness-aware minimization for efficiently improving generalization

    Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. ICLR, 2021

    2021

  5. [13]

    Implicit regularization in heavy-ball momentum accelerated stochastic gradient descent

    Ghosh, A., Lyu, H., Zhang, X., and Wang, R. Implicit regularization in heavy-ball momentum accelerated stochastic gradient descent. ICLR, 2023

    2023

  6. [14]

    A., Pezeshki, M., Le Priol, R., Huang, G., Lacoste-Julien, S., and Mitliagkas, I

    Gidel, G., Hemmat, R. A., Pezeshki, M., Le Priol, R., Huang, G., Lacoste-Julien, S., and Mitliagkas, I. Negative momentum for improved game dynamics. In The 22nd International Conference on Artificial Intelligence and Statistics, 2019

    2019

  7. [15]

    Why momentum really works

    Goh, G. Why momentum really works. Distill, 2017. doi:10.23915/distill.00006. URL http://distill.pub/2017/momentum

    2017 doi

  8. [16]

    J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y

    Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, pp.\ 2672–2680, Ca...

    2014

  9. [17]

    Matrix Theory and Applications for Scientists and Engineers

    Graham, A. Matrix Theory and Applications for Scientists and Engineers. Courier Dover Publications, 2018

    2018

  10. [18]

    Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. Improved training of W asserstein GAN s. Advances in neural information processing systems, 30, 2017

    2017

  11. [19]

    Alternating regret for online convex optimization

    Hait, S., Li, P., Luo, H., and Zhang, M. Alternating regret for online convex optimization. arXiv preprint arXiv:2502.12529, 2025

    2025 arXiv

  12. [20]

    GAN s trained by a two time-scale update rule converge to a local N ash equilibrium

    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GAN s trained by a two time-scale update rule converge to a local N ash equilibrium. Advances in Neural Information Processing Systems, 30, 2017

    2017

  13. [21]

    The limits of min-max optimization algorithms: Convergence to spurious non-critical sets

    Hsieh, Y.-P., Mertikopoulos, P., and Cevher, V. The limits of min-max optimization algorithms: Convergence to spurious non-critical sets. In International Conference on Machine Learning, pp.\ 4337--4348. PMLR, 2021

    2021

  14. [22]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  15. [23]

    Korpelevich, G. M. The extragradient method for finding saddle points and other problems. Matecon, 12: 0 747--756, 1976

    1976

  16. [24]

    Kovachki, N. B. and Stuart, A. M. Continuous time analysis of momentum methods. Journal of Machine Learning Research, 22 0 (17): 0 1--40, 2021

    2021

  17. [25]

    Fundamental benefit of alternating updates in minimax optimization

    Lee, J., Cho, H., and Yun, C. Fundamental benefit of alternating updates in minimax optimization. In Proceedings of the 41st International Conference on Machine Learning, 2024

    2024

  18. [26]

    D., Panageas, I., Piliouras, G., Simchowitz, M., Jordan, M

    Lee, J. D., Panageas, I., Piliouras, G., Simchowitz, M., Jordan, M. I., and Recht, B. First-order methods almost always avoid strict saddle points. Mathematical programming, 176: 0 311--337, 2019

    2019

  19. [27]

    Analysis and design of optimization algorithms via integral quadratic constraints

    Lessard, L., Recht, B., and Packard, A. Analysis and design of optimization algorithms via integral quadratic constraints. SIAM Journal on Optimization, 26 0 (1): 0 57--95, 2016

    2016

  20. [28]

    Differentiable game mechanics

    Letcher, A., Balduzzi, D., Racani \`e re, S., Martens, J., Foerster, J., Tuyls, K., and Graepel, T. Differentiable game mechanics. Journal of Machine Learning Research, 20 0 (84): 0 1--40, 2019

    2019

  21. [29]

    Stochastic modified equations and adaptive stochastic gradient algorithms

    Li, Q., Tai, C., et al. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp.\ 2101--2110. PMLR, 2017

    2017

  22. [30]

    Accelerated regularized learning in finite n-person games

    Lotidis, K., Giannou, A., Mertikopoulos, P., and Bambos, N. Accelerated regularized learning in finite n-person games. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  23. [31]

    An o (sr)-resolution ode framework for understanding discrete-time algorithms and applications to the linear convergence of minimax problems

    Lu, H. An o (sr)-resolution ode framework for understanding discrete-time algorithms and applications to the linear convergence of minimax problems. Mathematical Programming, 194 0 (1): 0 1061--1112, 2022

    2022

  24. [32]

    Towards deep learning models resistant to adversarial attacks

    Madry, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017

    2017 arXiv

  25. [33]

    Which training methods for GAN s do actually converge? In International Conference on Machine Learning, pp.\ 3481--3490

    Mescheder, L., Geiger, A., and Nowozin, S. Which training methods for GAN s do actually converge? In International Conference on Machine Learning, pp.\ 3481--3490. PMLR, 2018

    2018

  26. [34]

    and Jordan, M

    Muehlebach, M. and Jordan, M. A dynamical systems perspective on N esterov acceleration. In International Conference on Machine Learning, pp.\ 4656--4662. PMLR, 2019

    2019

  27. [35]

    and Jordan, M

    Muehlebach, M. and Jordan, M. I. Optimization with momentum: Dynamical, control-theoretic, and symplectic perspectives. Journal of Machine Learning Research, 22 0 (73): 0 1--50, 2021

    2021

  28. [36]

    Local convergence of the heavy-ball method and i P iano for non-convex optimization

    Ochs, P. Local convergence of the heavy-ball method and i P iano for non-convex optimization. Journal of Optimization Theory and Applications, 177 0 (1): 0 153--180, 2018

    2018

  29. [37]

    and Candes, E

    O’Donoghue, B. and Candes, E. Adaptive restart for accelerated gradient schemes. Foundations of Computational Mathematics, 15: 0 715--732, 2015

    2015

  30. [38]

    Stable nonconvex-nonconcave training via linear interpolation

    Pethick, T., Xie, W., and Cevher, V. Stable nonconvex-nonconcave training via linear interpolation. Advances in Neural Information Processing Systems, 36: 0 49830--49841, 2023

    2023

  31. [39]

    Polyak, B. T. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4 0 (5): 0 1--17, 1964

    1964

  32. [40]

    On the momentum term in gradient descent learning algorithms

    Qian, N. On the momentum term in gradient descent learning algorithms. Neural networks, 12 0 (1): 0 145--151, 1999

    1999

  33. [41]

    J., Burden, S

    Ratliff, L. J., Burden, S. A., and Sastry, S. S. On the characterization of local N ash equilibria in continuous games. IEEE Transactions on Automatic Control, 61 0 (8): 0 2301--2307, 2016

    2016

  34. [42]

    C., Wu, Y., Dherin, B., and Barrett, D

    Rosca, M. C., Wu, Y., Dherin, B., and Barrett, D. Discretization drift in two-player games. In International Conference on Machine Learning, pp.\ 9064--9074. PMLR, 2021

    2021

  35. [43]

    Improved techniques for training gans

    Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016

    2016

  36. [44]

    S., Jordan, M

    Shi, B., Du, S. S., Jordan, M. I., and Su, W. J. Understanding the acceleration phenomenon via high-resolution differential equations. Mathematical Programming, pp.\ 1--70, 2022

    2022

  37. [45]

    Su, W., Boyd, S., and Cand \`e s, E. J. A differential equation for modeling nesterov's accelerated gradient method: Theory and insights. Journal of Machine Learning Research, 17 0 (153): 0 1--43, 2016

    2016

  38. [46]

    On the importance of initialization and momentum in deep learning

    Sutskever, I., Martens, J., Dahl, G., and Hinton, G. On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning, Proceedings of Machine Learning Research, pp.\ 1139--1147. PMLR, 17--19 Jun 2013

    2013

  39. [47]

    Improving generalization and stability of generative adversarial networks

    Thanh-Tung, H., Tran, T., and Venkatesh, S. Improving generalization and stability of generative adversarial networks. ICLR, 2019

    2019

  40. [48]

    and Chizat, L

    Wang, G. and Chizat, L. Local convergence of gradient methods for min-max games: partial curvature generically suffices. Advances in Neural Information Processing Systems, 36, 2024

    2024

  41. [49]

    Provable acceleration of heavy ball beyond quadratics for a class of P olyak- L ojasiewicz functions when the non-convexity is averaged-out

    Wang, J.-K., Lin, C.-H., Wibisono, A., and Hu, B. Provable acceleration of heavy ball beyond quadratics for a class of P olyak- L ojasiewicz functions when the non-convexity is averaged-out. In International Conference on Machine Learning, pp.\ 22839--22864. PMLR, 2022

    2022

  42. [50]

    On the convergence and robustness of adversarial training

    Wang, Y., Ma, X., Bailey, J., Yi, J., Zhou, B., and Gu, Q. On the convergence and robustness of adversarial training. In International Conference on Machine Learning, pp.\ 6586--6595. PMLR, 2019

    2019

  43. [51]

    C., and Jordan, M

    Wibisono, A., Wilson, A. C., and Jordan, M. I. A variational perspective on accelerated methods in optimization. proceedings of the National Academy of Sciences, 113 0 (47): 0 E7351--E7358, 2016

    2016

  44. [52]

    Alternating mirror descent for constrained min-max games

    Wibisono, A., Tao, M., and Piliouras, G. Alternating mirror descent for constrained min-max games. Advances in Neural Information Processing Systems, 35: 0 35201--35212, 2022

    2022

  45. [53]

    Gradient normalization for generative adversarial networks

    Wu, Y.-L., Shuai, H.-H., Tam, Z.-R., and Chiu, H.-Y. Gradient normalization for generative adversarial networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 6373--6382, 2021

    2021

  46. [54]

    Faster single-loop algorithms for minimax optimization without strong concavity

    Yang, J., Orvieto, A., Lucchi, A., and He, N. Faster single-loop algorithms for minimax optimization without strong concavity. In International Conference on Artificial Intelligence and Statistics, pp.\ 5485--5517. PMLR, 2022

    2022

  47. [55]

    and Wang, Y

    Zhang, G. and Wang, Y. On the suboptimality of negative momentum for minimax optimization. In International Conference on Artificial Intelligence and Statistics, pp.\ 2098--2106. PMLR, 2021

    2021

  48. [56]

    A unified analysis of first-order methods for smooth games via integral quadratic constraints

    Zhang, G., Bao, X., Lessard, L., and Grosse, R. A unified analysis of first-order methods for smooth games via integral quadratic constraints. Journal of Machine Learning Research, 22 0 (103): 0 1--39, 2021

    2021

  49. [57]

    Zhang, G., Wang, Y., Lessard, L., and Grosse, R. B. Near-optimal local convergence of alternating gradient descent-ascent for minimax optimization. In International Conference on Artificial Intelligence and Statistics, pp.\ 7659--7679. PMLR, 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.