Pith. sign in

REVIEW 1 major objections 3 minor 101 references

Layer-wise Quantization for Quantized Optimistic Dual Averaging

T0 review · 1 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Per-layer adaptive quantization can solve distributed monotone variational inequalities at the same asymptotic rates as uncompressed methods while dropping a common boundedness assumption.

desk verdict The layer-wise quantization theory is solid and worth engaging, but the headline QODA convergence theorem leans on an unstated bounded-iterates assumption, so the claimed removal of a.s. boundedness does not go through as written. read the letter →

arxiv 2505.14371 v1 pith:2RZTPCR2 submitted 2025-05-20 cs.LG math.OC

Anh Duc Nguyen , Ilia Markov , Frank Zhengqing Wu , Ali Ramezani-Kebrya , Kimon Antonakopoulos , Dan Alistarh , Volkan Cevher This is my paper · ORCID
classification cs.LGmath.OC
keywords layer-wisequantizationoptimisticdualaveragingvariationalinequalitiesdistributedoptimizationadaptivecommunication-efficienttrainingGANconvergencerates
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that communication-efficient training can be made layer-aware: instead of quantizing all coordinates with one global level sequence, it assigns each layer its own adaptive quantization sequence chosen to minimize expected quantization variance, and it proves bounds on the induced variance and transmitted bits. It embeds this layer-wise compressor in an optimistic dual averaging loop for distributed monotone variational inequalities, the equilibrium formulation behind minimax problems such as GAN training. The central claim is that the resulting algorithm, QODA, converges in the restricted gap function at rate $O(1/\sqrt{T})$ under absolute noise and $O(1/T)$ under relative noise, without the almost sure boundedness of stochastic dual vectors that earlier distributed VI solvers required. If correct, distributed training of adversarial models can quantize per layer and still match uncompressed convergence, with faster wall-clock time in bandwidth-limited settings.

What carries the argument

The load-bearing object is the layer-wise quantizer $Q_{L_M}(v)$, which normalizes a vector by its $L_q$ norm, maps each coordinate to one of $M$ type-specific sequences of quantization levels, and chooses those sequences by minimizing the expected quantization variance (MQV) from estimated marginal distributions. A companion coding protocol encodes the norm, signs, and level indices with per-type prefix codes, giving expected code length bounded by a weighted entropy expression. The quantizer feeds an optimistic dual averaging update, $X_{t+1/2}=X_t-\gamma_t \frac{1}{K}\sum_k \hat V_{k,t-1/2}$ and $Y_{t+1}=Y_t-\frac{1}{K}\sum_k \hat V_{k,t+1/2}$, where the same compressed vector is stored and reused for the next extrapolation; this is what halves the number of transmitted vectors relative to extra-gradient methods. The adaptive step sizes $\eta_t=\gamma_t=(1+\sum_{s<t}\frac{1}{K^2}\sum_k\|\hat V_{k,s+1/2}-\hat V_{k,s-1/2}\|_*^2)^{-1/2}$ are what let the proof telescope the error and control cumulative quantization noise through $\varepsilon_Q$.

What would settle it

Run QODA on a monotone bilinear game with an unbounded domain and check whether $\mathbb{E}\|X_{t+1/2}-X_{t-1/2}\|^2$ stays bounded by $D^2$; if it grows with $T$ while the empirical gap still decays, the proof's hidden boundedness premise fails even if the advertised rate is observed.

Watch

Extended reading notes

Core claim

The core discovery, stated on the paper's own terms, is that per-layer adaptive quantization does not cost asymptotic convergence. For a monotone $L$-Lipschitz operator, $K$ workers, and a compact test domain, QODA is claimed to achieve restricted-gap error $O(((LD+\|A(X_1)\|+\sigma)c_{\varepsilon_Q}+\sigma)D^2 L^2/\sqrt{T K})$ under absolute noise and $O((\sigma_R\varepsilon_Q+\varepsilon_Q+\sigma_R)D^2/(T K))$ under relative noise, where $\varepsilon_Q$ is the average per-layer quantization variance bound. The absolute-noise rate needs no co-coercivity and matches the known lower bound for distributed stochastic convex optimization; the relative-noise rate requires co-coercivity in the main theorem. Both rates match the global-quantization baseline Q-GenX while removing its almost sure boundedness assumption, and the optimistic update reuses the previous compressed dual vector so only one quantized vector is broadcast per iteration.

Load-bearing premise

The load-bearing premise is that the squared distance between successive extrapolated update points never exceeds the squared diameter $D^2$ of a fixed compact test set, even though the iterates are never projected back into that set.

Editorial extensions

If this is right

  • Layer-wise quantization never increases the expected variance relative to the best single global level sequence, so adapting levels per layer can only improve the accuracy of compressed communication.
  • Because the optimistic update reuses the previous compressed dual vector, QODA transmits one quantized vector per iteration instead of the two required by extra-gradient VI solvers, halving communication load.
  • Under absolute noise the restricted gap converges at $O(1/\sqrt{T K})$ with $K$ workers, matching the known distributed convex-optimization lower bound; under relative noise it converges at $O(1/T)$, matching the best global-quantization baseline.
  • These rates are obtained without the almost sure boundedness of stochastic dual vectors, the assumption that previous distributed VI analyses (including Q-GenX) required.
  • Empirically, layer-wise QODA shortens end-to-end WGAN training time by up to 150% on 12+ GPUs, and layer-wise quantization improves Transformer-XL compression rates over global quantization at matched perplexity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The variance-minimization principle suggests that optimal level sequences should mirror the drift of each layer's marginal distribution of normalized coordinates; measuring that drift would give a principled replacement for the fixed 10k-step level-update cadence used in the experiments.
  • The same layer-wise compressor should transfer to weakly Minty or non-monotone operators, and to adjacent communication bottlenecks such as adversarial training and robust reinforcement learning, though the paper only proves monotone guarantees.
  • The reported speedups (1.28x-2.5x depending on bandwidth and node count) were measured on a 5 Gbps inter-node link; as bandwidth rises toward memory speed, the wall-clock benefit of quantized communication should shrink, making the method most attractive for bandwidth-limited clusters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 3 minor

Summary. The paper has two components. First, it proposes a general layer-wise unbiased quantization framework with adaptive sequences of quantization levels per layer, and derives variance and code-length bounds (Theorems 5.1 and 5.3) that generalize global quantizers such as QSGD, NUQSGD, and Q-GenX, with claims of matching lower bounds in special cases. Second, it applies this compression framework to distributed monotone variational inequalities via QODA, an optimistic dual averaging algorithm with adaptive stepsizes, claiming O(1/sqrt(T)) restricted-gap convergence under absolute noise (Theorem 5.5), O(1/T) under relative noise with co-coercivity (Theorem 5.7), and O(1/T) under relative noise without co-coercivity but with an almost-sure boundedness assumption (Theorem 6.2). The paper also reports wall-clock speedups up to 150% for WGAN training and layer-wise compression gains on Transformer-XL.

Significance. The layer-wise quantization analysis in Section 5.1 is self-contained: the variance bound is proved by optimizing quantization level spacings and the code-length bound via entropy coding, with explicit comparisons to prior global-quantization bounds. These parts appear sound and are a useful contribution. If the VI results were valid, QODA would be a meaningful advance as the first distributed quantized VI solver combining optimism with layer-wise compression and avoiding the almost-sure boundedness assumption on stochastic dual vectors. However, the central convergence proof of Theorem 5.5 contains an unstated bounded-iterates assumption, and this invalidates the advertised removal of the almost-sure boundedness assumption. The main theoretical claim of the paper is therefore not established as stated.

major comments (1)
  1. [Appendix E.2, Eq. (11), proof of Theorem 5.5] The proof bounds E[||\hat V_{k,t+1/2} - \hat V_{k,t-1/2}||^2_*] by 2L^2 E[||X_{t+1/2} - X_{t-1/2}||^2] + 4\sigma^2 and then replaces E[||X_{t+1/2} - X_{t-1/2}||^2] by D^2, where D^2 := sup_{p \in \mathcal X} ||X_1 - p||^2. This substitution is not justified anywhere: Algorithm 1's optimistic dual averaging iterates are unprojected, and Assumptions 2.1-2.4 impose no confinement of {X_{t+1/2}, X_{t-1/2}} to \mathcal X. Consequently the bounds on the terms S1 and S2 in the proof of Theorem 5.5 are unsupported, and the claimed O(1/sqrt(T)) restricted-gap rate is not established. Theorem 5.7 inherits the same gap because its proof uses the S2 bound from Theorem 5.5, and Remark 5.8's claim that the almost-sure boundedness assumption is removed is therefore not justified. Repairing this would require either an explicit bounded-iterates or containment assumption, which would weaken the stated contribution, or a genuinely new argument controlling ||X_{t+1/2} - X_{t-1/2}||.
minor comments (3)
  1. [Appendix E.2, S3 term in Theorem 5.5] The proof applies Lemma E.3 to E[sup_{p \in \mathcal X} \langle \sum_{t,k} U_{k,t+1/2}, p \rangle], but Lemma E.3 as stated is a pointwise bound for a fixed x in C. A short justification for controlling the supremum, for example via sup_{p \in \mathcal X} \langle S, p \rangle \leq \sup_{p \in \mathcal X} ||p|| \cdot ||S||_*, would make this step complete.
  2. [Section 5.2, Theorems 5.5 and 5.7] The notation X_{t+1/2} is reused for the averaged iterate \bar X_{T+1/2} in the theorem statements; the average should be defined explicitly before its first use to avoid confusion with the half-step iterates in Algorithm 1.
  3. [Theorem 5.5 statement] The restricted gap function (GAP) and Lemma E.3 require convexity of the test domain, but Theorems 5.5 and 5.7 only state that \mathcal X is compact. The authors should either state explicitly that \mathcal X is convex and compact or adjust the proof accordingly.

Circularity Check

0 steps flagged · score 2.0 of 10

There is no significant circularity: QODA's convergence rates are derived from stated assumptions, and the flagged Eq. (11) boundedness step is a proof gap rather than a circular reduction.

full rationale

The central derivation chain is self-contained for the unquantized optimistic dual averaging analysis: Appendix E.2 proves Theorem 5.5 by combining the template inequality (Proposition E.1) with Lemmas E.2 and E.3, bounding the terms S1, S2, and S3 directly from Assumptions 2.1 through 2.4, and obtaining O(D^2 L^2 / sqrt(T K)). The final layer-wise compression extension is stated as 'by applying compression bound Lemma D.9 along the ideas of (Faghri et al., 2020, Theorem 4) and (Ramezani-Kebrya et al., 2023, Theorem 3),' which is a cited technique, not a renamed conclusion: Lemma D.9 itself is proved for the paper's quantization variance bound, and the remaining algebra is a standard variance-inflation argument. Similarly, Theorem 5.7 and Theorem 6.2 are proved from the template inequality and cited general lemmas from prior work, whose stated assumptions do not include the target rates. Self-citations to Q-GenX and L-GreCo appear mainly as baselines for comparison and as an experimental implementation choice, not as the load-bearing proof of the main theorems. The variance and code-length bounds in Theorems 5.1 and 5.3 are proved from first principles; recovering Q-GenX's bound as the M=1 special case is a consistency check rather than a circular dependence. The paper contains no fitted parameter renamed as a prediction and no uniqueness theorem imported from the authors' prior work. The most serious flagged issue is correctness-related, not circular: Eq. (11) in Appendix E.2 asserts the bound '2KL^2D^2 + 4Kσ^2' after citing L-Lipschitzness and absolute noise, but the unprojected iterates X_{t+1/2} need not lie in X, so the bounding of E||X_{t+1/2} - X_{t-1/2}||^2 by D^2 is not justified by the stated assumptions. This undermines the claimed removal of the almost sure boundedness assumption, but it is an unsupported inequality, not an equivalence-by-construction between an input and a prediction. Accordingly, the circularity burden is low.

Assumptions & free parameters 0 free parameters · 8 assumptions · 0 invented entities

The central convergence claims rest on standard monotone VI assumptions and noise models. The main hidden assumption is not listed: the iterates are implicitly assumed to remain in a compact set when bounding iterate differences by D in the proof of Theorem 5.5. No free parameters were fitted to data; the quantization levels are adaptively chosen and the learning rates are adaptive schedules. The paper introduces no new physical or mathematical entities.

assumptions (8)
  • domain assumption Monotonicity of the operator (Assumption 2.1)
    Standard assumption for VI convergence proofs; used throughout the analysis.
  • domain assumption Solution existence (Assumption 2.2)
    Needed to define the GAP and ensure a nonempty solution set.
  • domain assumption L-Lipschitz operator (Assumption 2.3)
    Used to bound differences of operator values in terms of iterate differences.
  • domain assumption Absolute noise model (Assumption 2.4)
    Stochastic oracle has bounded variance; used in Theorem 5.5 and related bounds.
  • domain assumption Relative noise model (Assumption 2.5)
    Noise variance proportional to ||A(x)||^2; used for fast O(1/T) rates and in Section 6.
  • domain assumption Co-coercivity for fast relative noise rate (Assumption 5.6)
    Required for the O(1/T) rate in Theorem 5.7; not needed for absolute noise or for Theorem 6.2.
  • domain assumption Almost sure boundedness for Section 6 (Assumption 6.1)
    ||g(x;omega)||_* <= J almost surely; used to remove co-coercivity but reintroduces a boundedness assumption.
  • standard math Lemma C.2 from Ramezani-Kebrya et al. (2021)
    External lemma used without proof in the variance bound derivation; is a published result from prior work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Layer-wise Quantization for Quantized Optimistic Dual Averaging." pith.science (2026). https://pith.science/paper/2RZTPCR2

@misc{pith2026250514371,
  author       = {Pith},
  title        = {Pith review of: Layer-wise Quantization for Quantized Optimistic Dual Averaging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2RZTPCR2}},
  note         = {Machine review of arXiv:2505.14371}
}
abstract

Modern deep neural networks exhibit heterogeneity across numerous layers of various types such as residuals, multi-head attention, etc., due to varying structures (dimensions, activation functions, etc.), distinct representation characteristics, which impact predictions. We develop a general layer-wise quantization framework with tight variance and code-length bounds, adapting to the heterogeneities over the course of training. We then apply a new layer-wise quantization technique within distributed variational inequalities (VIs), proposing a novel Quantized Optimistic Dual Averaging (QODA) algorithm with adaptive learning rates, which achieves competitive convergence rates for monotone VIs. We empirically show that QODA achieves up to a $150\%$ speedup over the baselines in end-to-end training time for training Wasserstein GAN on $12+$ GPUs.

Figures

Figures reproduced from arXiv: 2505.14371 by the authors.

Figure 1
Figure 1. A Visualization for Layer-wise vs Global Quantization on CIFAR-100. Our goal is hence to provide a general formulation incorporating the statistical heterogeneity across layers and establish tight theoretical guarantees for layer-wise quantization with tailored coding schemes. In [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 4
Figure 4. FID evolution during training. We compare basic Adam optimization against QODA-based extension of Adam with global (Q-GenX (Ramezani-Kebrya et al., 2023)) and layer-wise (L-GreCo) quantizations. larger extrapolation step in the first line of (ODA), so the noise is an order of magnitude smaller than the expected variation of utilities (Hsieh et al., 2022). We now provide the convergence of Algorithm 1 under relative … view at source ↗
Figure 5
Figure 5. Ablation Study for Transformer-XL PowerSGD with varying quantization levels (ranks). Each setup was repeated four times with different seeds, and the results are shown in [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

101 extracted references · 63 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    https://developer.nvidia.com/nccl, 2023

    NVIDIA Collective Communication Library . https://developer.nvidia.com/nccl, 2023

    2023

  3. [3]

    https://www.open-mpi.org/, 2023

    Open MPI: Open Source High Performance Computing . https://www.open-mpi.org/, 2023

    2023

  4. [4]

    Adaptive gradient communication via critical learning regime identification

    Agarwal, S., Wang, H., Lee, K., Venkataraman, S., and Papailiopoulos, D. Adaptive gradient communication via critical learning regime identification. In Conference on Machine Learning and Systems (MLSys), 2021

    2021

  5. [5]

    QSGD : Communication-efficient SGD via gradient quantization and encoding

    Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. QSGD : Communication-efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  6. [6]

    An adaptive mirror-prox method for variational inequalities with singular operators

    Antonakopoulos, K., Belmega, V., and Mertikopoulos, P. An adaptive mirror-prox method for variational inequalities with singular operators. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019

    2019

  7. [7]

    Sifting through the noise: Universal first-order methods for stochastic variational inequalities

    Antonakopoulos, K., Pethick, T., Kavis, A., Mertikopoulos, P., and Cevher, V. Sifting through the noise: Universal first-order methods for stochastic variational inequalities. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp.\ 13099--13111, 2021

    2021

  8. [8]

    Wasserstein generative adversarial networks

    Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein generative adversarial networks. In International Conference on Machine Learning (ICML), pp.\ 214--223. PMLR, 2017

    2017

Show all 101 references
  1. [9]

    Efqat: An efficient framework for quantization-aware training

    Ashkboos, S., Verhoef, B., Hoefler, T., Eleftheriou, E., and Dazzi, M. Efqat: An efficient framework for quantization-aware training. arXiv preprint arXiv:2411.11038, 2024

    2024 arXiv

  2. [10]

    Adaptive and self-confident on-line learning algorithms

    Auer, P., Cesa-Bianchi, N., and Gentile, C. Adaptive and self-confident on-line learning algorithms. Journal of Computer and System Sciences, 64 0 (1): 0 48--75, 2002

    2002

  3. [11]

    and Levy, K

    Bach, F. and Levy, K. Y. A universal algorithm for variational inequalities adaptive to smoothness and noise. In Conference on learning theory, pp.\ 164--194. PMLR, 2019

    2019

  4. [12]

    Bauschke, H. H. and Combettes, P. L. Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, 2017

    2017

  5. [13]

    Distributed methods with compressed communication for solving variational inequalities, with theoretical guarantees

    Beznosikov, A., Richt \'a rik, P., Diskin, M., Ryabinin, M., and Gasnikov, A. Distributed methods with compressed communication for solving variational inequalities, with theoretical guarantees. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp.\ 14...

    2022

  6. [14]

    Bregman proximal method for efficient communications under similarity, 2023 a

    Beznosikov, A., Dvinskikh, D., Semenov, A., and Gasnikov, A. Bregman proximal method for efficient communications under similarity, 2023 a

    2023

  7. [15]

    Stochastic gradient descent-ascent: Unified theory and new efficient methods

    Beznosikov, A., Gorbunov, E., Berard, H., and Loizou, N. Stochastic gradient descent-ascent: Unified theory and new efficient methods. In International Conference on Artificial Intelligence and Statistics, pp.\ 172--235. PMLR, 2023 b

    2023

  8. [16]

    and Pock, T

    Chambolle, A. and Pock, T. A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision, 40 0 (1): 0 120--145, May 2011

    2011

  9. [17]

    Reducing noise in gan training with variance reduced extragradient

    Chavdarova, T., Gidel, G., Fleuret, F., and Lacoste-Julien, S. Reducing noise in gan training with variance reduced extragradient. Advances in Neural Information Processing Systems, 32, 2019

    2019

  10. [18]

    Online optimization with gradual variations

    Chiang, C.-K., Yang, T., Lee, C.-J., Mahdavi, M., Lu, C.-J., Jin, R., and Zhu, S. Online optimization with gradual variations. In COLT '12: Proceedings of the 25th Annual Conference on Learning Theory, 2012

    2012

  11. [19]

    Cover, T. M. and Thomas, J. A. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006. ISBN 0471241954

    2006

  12. [20]

    and Shanbhag, U

    Cui, S. and Shanbhag, U. V. On the analysis of reflected gradient and splitting methods for monotone stochastic variational inequality problems. In CDC '16: Proceedings of the 57th IEEE Annual Conference on Decision and Control, 2016

    2016

  13. [21]

    Training gans with optimism

    Daskalakis, C., Ilyas, A., Syrgkanis, V., and Zeng, H. Training gans with optimism. arXiv preprint arXiv:1711.00141, 2017

    2017 arXiv

  14. [22]

    Training GANs with optimism

    Daskalakis, C., Ilyas, A., Syrgkanis, V., and Zeng, H. Training GANs with optimism. In ICLR '18: Proceedings of the 2018 International Conference on Learning Representations, 2018

    2018

  15. [23]

    New bounds for distributed mean estimation and variance reduction

    Davies, P., Gurunanthan, V., Moshrefi, N., Ashkboos, S., and Alistarh, D. New bounds for distributed mean estimation and variance reduction. In International Conference on Learning Representations (ICLR), 2021

    2021

  16. [24]

    S., Monga, R., Chen, K., Devin, M., Mao, M

    Dean, J., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M. Z., Ranzato, M., Senior, A., Tucker, P., Yang, K., Le, Q., and Ng, A. Y. Large scale distributed deep networks. In Advances in Neural Information Processing Systems (NeurIPS), 2012

    2012

  17. [25]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representation...

    2021

  18. [26]

    Adaptive subgradient methods for online learning and stochastic optimization

    Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research (JMLR), 12 0 (7), 2011

    2011

  19. [27]

    H., Abdelmoniem, A

    Dutta, A., Bergou, E. H., Abdelmoniem, A. M., Ho, C.-Y., Sahu, A. N., Canini, M., and Kalnis, P. On the discrepancy between the theoretical analysis and practical implementations of compressed communication for distributed deep learning. In Proceedings of the AAAI Conference o...

    2020

  20. [28]

    Universal codeword sets and representations of the integers

    Elias, P. Universal codeword sets and representations of the integers. IEEE Transactions on Information Theory, 21 0 (2): 0 194--203, 1975. doi:10.1109/TIT.1975.1055349

    1975

  21. [29]

    and Le Nguyen, H

    Ene, A. and Le Nguyen, H. Adaptive and universal algorithms for variational inequalities with optimal convergence. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 6559--6567, 2022

    2022

  22. [30]

    and Pang, J.-S

    Facchinei, F. and Pang, J.-S. Finite-dimensional variational inequalities and complementarity problems. Springer, 2003

    2003

  23. [31]

    M., and Ramezani-Kebrya, A

    Faghri, F., Tabrizian, I., Markov, I., Alistarh, D., Roy, D. M., and Ramezani-Kebrya, A. Adaptive gradient quantization for data-parallel SGD . In Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  24. [32]

    OPTQ : Accurate quantization for generative pre-trained transformers

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. OPTQ : Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=tcbBPnfwxS

    2023

  25. [33]

    A variational inequality perspective on generative adversarial networks

    Gidel, G., Berard, H., Vignoud, G., Vincent, P., and Lacoste-Julien, S. A variational inequality perspective on generative adversarial networks. arXiv preprint arXiv:1802.10551, 2018

    2018 arXiv

  26. [34]

    A variational inequality perspective on generative adversarial networks

    Gidel, G., Berard, H., Vignoud, G., Vincent, P., and Lacoste-Julien, S. A variational inequality perspective on generative adversarial networks. In ICLR '19: Proceedings of the 2019 International Conference on Learning Representations, 2019

    2019

  27. [35]

    Generative adversarial nets

    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), volume 27, 2014

    2014

  28. [36]

    Accelerating distributed deep learning by adaptive gradient quantization

    Guo, J., Liu, W., Wang, W., Han, J., Li, R., Lu, Y., and Hu, S. Accelerating distributed deep learning by adaptive gradient quantization. In IEEE International Conference on Acoustics, Speech and Signal Processing , 2020

    2020

  29. [37]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  30. [38]

    Stochastic distributed learning with gradient quantization and double-variance reduction

    Horv \'a th, S., Kovalev, D., Mishchenko, K., Richt \'a rik, P., and Stich, S. Stochastic distributed learning with gradient quantization and double-variance reduction. Optimization Methods and Software, 38 0 (1): 0 91--106, 2023

    2023

  31. [39]

    Explore aggressively, update conservatively: Stochastic extragradient methods with variable stepsize scaling

    Hsieh, Y.-G., Iutzeler, F., Malick, J., and Mertikopoulos, P. Explore aggressively, update conservatively: Stochastic extragradient methods with variable stepsize scaling. In Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  32. [40]

    Adaptive learning in continuous games: Optimal regret bounds and convergence to nash equilibrium

    Hsieh, Y.-G., Antonakopoulos, K., and Mertikopoulos, P. Adaptive learning in continuous games: Optimal regret bounds and convergence to nash equilibrium. In Conference on Learning Theory, pp.\ 2388--2422. PMLR, 2021

    2021

  33. [41]

    No-regret learning in games with noisy feedback: Faster rates and adaptivity via learning rate separation

    Hsieh, Y.-G., Antonakopoulos, K., Cevher, V., and Mertikopoulos, P. No-regret learning in games with noisy feedback: Faster rates and adaptivity via learning rate separation. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  34. [42]

    Huffman, D. A. A method for the construction of minimum-redundancy codes. Proceedings of the IRE, 40 0 (9): 0 1098--1101, 1952. doi:10.1109/JRPROC.1952.273898

    1952

  35. [43]

    N., Jofr \'e , A., Oliveira, R

    Iusem, A. N., Jofr \'e , A., Oliveira, R. I., and Thompson, P. Extragradient method with variance reduction for stochastic variational inequalities. SIAM Journal on Optimization, 27 0 (2): 0 686--724, 2017

    2017

  36. [44]

    and Sidford, A

    Jin, Y. and Sidford, A. Efficiently solving mdps with stochastic mirror descent. In International Conference on Machine Learning, pp.\ 4890--4900. PMLR, 2020

    2020

  37. [45]

    Solving variational inequalities with stochastic mirror-prox algorithm

    Juditsky, A., Nemirovski, A., and Tauvel, C. Solving variational inequalities with stochastic mirror-prox algorithm. Stochastic Systems, 1 0 (1): 0 17--58, 2011

    2011

  38. [46]

    B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A

    Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., D'Oliveira, R. G. L., Rouayheb, S. E., Evans, D., Gardner, J., Garrett, Z., Gasc\' o n, A., Ghazi, B., Gibbons, P. B., Gruteser, M., Harchaoui,...

    2021

  39. [47]

    Robust reinforcement learning via adversarial training with langevin dynamics

    Kamalaruban, P., Huang, Y.-T., Hsieh, Y.-P., Rolland, P., Shi, C., and Cevher, V. Robust reinforcement learning via adversarial training with langevin dynamics. Advances in Neural Information Processing Systems (NeurIPS), 33: 0 8127--8138, 2020

    2020

  40. [48]

    and Shanbhag, U

    Kannan, A. and Shanbhag, U. V. Optimal stochastic extragradient schemes for pseudomonotone stochastic variational inequality problems and their variants. Computational Optimization and Applications, 74 0 (3): 0 779--820, 2019

    2019

  41. [49]

    Korhonen, J. H. and Alistarh, D. Towards tight communication lower bounds for distributed optimisation. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  42. [50]

    Korpelevich, G. M. The extragradient method for finding saddle points and other problems. Matecon, 12: 0 747--756, 1976

    1976

  43. [51]

    Optimal algorithms for decentralized stochastic variational inequalities

    Kovalev, D., Beznosikov, A., Sadiev, A., Persiianov, M., Richt \'a rik, P., and Gasnikov, A. Optimal algorithms for decentralized stochastic variational inequalities. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp.\ 31073--31088, 2022

    2022

  44. [52]

    Learning multiple layers of features from tiny images

    Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  45. [53]

    Y., Yurtsever, A., and Cevher, V

    Levy, K. Y., Yurtsever, A., and Cevher, V. Online adaptive methods, universality and acceleration. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018

    2018

  46. [54]

    Det-cgd: Compressed gradient descent with matrix stepsizes for non-convex optimization

    Li, H., Karagulyan, A., and Richt \'a rik, P. Det-cgd: Compressed gradient descent with matrix stepsizes for non-convex optimization. In The Twelfth International Conference on Learning Representations, 2024

    2024

  47. [55]

    K., Talwalkar, A., and Smith, V

    Li, T., Sahu, A. K., Talwalkar, A., and Smith, V. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37 0 (3): 0 50--60, 2020

    2020

  48. [56]

    Finite-time last-iterate convergence for multi-agent learning in games

    Lin, T., Zhou, Z., Mertikopoulos, P., and Jordan, M. Finite-time last-iterate convergence for multi-agent learning in games. In ICML, pp.\ 6161--6171. PMLR, 2020

    2020

  49. [57]

    Adaptive compression for communication-efficient distributed training

    Makarenko, M., Gasanov, E., Islamov, R., Sadiev, A., and Richt \'a rik, P. Adaptive compression for communication-efficient distributed training. arXiv preprint arXiv:2211.00188, 2022

    2022 arXiv

  50. [58]

    Projected reflected gradient methods for monotone variational inequalities

    Malitsky, Y. Projected reflected gradient methods for monotone variational inequalities. SIAM Journal on Optimization, 25 0 (1): 0 502--520, 2015

    2015

  51. [59]

    Golden ratio algorithms for variational inequalities

    Malitsky, Y. Golden ratio algorithms for variational inequalities. Mathematical Programming, 2019

    2019

  52. [60]

    Cgx: adaptive system support for communication-efficient deep learning

    Markov, I., Ramezanikebrya, H., and Alistarh, D. Cgx: adaptive system support for communication-efficient deep learning. In Proceedings of the 23rd ACM/IFIP International Middleware Conference, pp.\ 241--254, 2022

    2022

  53. [61]

    L-greco: Layerwise-adaptive gradient compression for efficient data-parallel deep learning

    Markov, I., Alim, K., Frantar, E., and Alistarh, D. L-greco: Layerwise-adaptive gradient compression for efficient data-parallel deep learning. Proceedings of Machine Learning and Systems, 6: 0 312--324, 2024

    2024

  54. [62]

    McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.\ 1273--1282. PMLR, 2017

    2017

  55. [63]

    McMahan, H. B. and Streeter, M. Adaptive bound optimization for online convex optimization. arXiv preprint arXiv:1002.4908, 2010

    2010 arXiv

  56. [64]

    Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile

    Mertikopoulos, P., Lecouat, B., Zenati, H., Foo, C.-S., Chandrasekhar, V., and Piliouras, G. Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile. arXiv preprint arXiv:1807.02629, 2018

    2018 arXiv

  57. [65]

    Intsgd: Adaptive floatless compression of stochastic gradients

    Mishchenko, K., Wang, B., Kovalev, D., and Richt \'a rik, P. Intsgd: Adaptive floatless compression of stochastic gradients. In International Conference on Learning Representations, 2021. URL https://api.semanticscholar.org/CorpusID:247594371

    2021

  58. [66]

    Distributed learning with compressed gradient differences

    Mishchenko, K., Gorbunov, E., Takáč, M., and Richt \'a rik, P. Distributed learning with compressed gradient differences. Optimization Methods and Software, pp.\ 1--16, 2024

    2024

  59. [67]

    A unified analysis of extra-gradient and optimistic gradient methods for saddle point problems: proximal point approach

    Mokhtari, A., Ozdaglar, A., and Pattathil, S. A unified analysis of extra-gradient and optimistic gradient methods for saddle point problems: proximal point approach. https://arxiv.org/abs/1901.08511v2, 2019 a

    1901 arXiv

  60. [68]

    Convergence rate of O (1/k) for optimistic gradient and extra-gradient methods in smooth convex-concave saddle point problems

    Mokhtari, A., Ozdaglar, A., and Pattathil, S. Convergence rate of O (1/k) for optimistic gradient and extra-gradient methods in smooth convex-concave saddle point problems. https://arxiv.org/pdf/1906.01115.pdf, 2019 b

    1906 arXiv

  61. [69]

    Prox-method with rate of convergence o(1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems

    Nemirovski, A. Prox-method with rate of convergence o(1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization , 15 0 (1): 0 229--251, 2004

    2004

  62. [70]

    Robust stochastic approximation approach to stochastic programming

    Nemirovski, A., Juditsky, A., Lan, G., and Shapiro, A. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization, 19 0 (4): 0 1574--1609, 2009

    2009

  63. [71]

    Introductory Lectures on Convex Optimization: A Basic Course

    Nesterov, Y. Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic Publishers, 2004

    2004

  64. [72]

    Dual extrapolation and its applications to solving variational inequalities and related problems

    Nesterov, Y. Dual extrapolation and its applications to solving variational inequalities and related problems. Math. Program., 109 0 (2–3): 0 319–344, mar 2007. ISSN 0025-5610

    2007

  65. [73]

    Primal-dual subgradient methods for convex problems

    Nesterov, Y. Primal-dual subgradient methods for convex problems. Mathematical programming, 120 0 (1): 0 221--259, 2009

    2009

  66. [74]

    P., and Vian, J

    Omidshafiei, S., Pazis, J., Amato, C., How, J. P., and Vian, J. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In International Conference on Machine Learning, pp.\ 2681--2690. PMLR, 2017

    2017

  67. [75]

    and Soltanolkotabi, M

    Oymak, S. and Soltanolkotabi, M. Toward moderate overparameterization: Global convergence guarantees for training shallow neural networks. IEEE Journal on Selected Areas in Information Theory, 1 0 (1): 0 84--105, 2020. doi:10.1109/JSAIT.2020.2991332

    2020

  68. [76]

    Training GANs with centripetal acceleration

    Peng, W., Dai, Y.-H., Zhang, H., and Cheng, L. Training GANs with centripetal acceleration. https://arxiv.org/abs/1902.08949, 2019

    1902 arXiv

  69. [77]

    Polyak, B. T. Introduction to optimization, 1987

    1987

  70. [78]

    Popov, L. D. A modification of the Arrow Hurwicz method for search of saddle points. Mathematical Notes of the Academy of Sciences of the USSR, 28 0 (5): 0 845--848, 1980

    1980

  71. [79]

    Ramezani-Kebrya, A., Faghri, F., Markov, I., Aksenov, V., Alistarh, D., and Roy, D. M. NUQSGD : Provably communication-efficient data-parallel SGD via nonuniform quantization. Journal of Machine Learning Research (JMLR), 22 0 (114): 0 1--43, 2021

    2021

  72. [80]

    Mixtailor: Mixed gradient aggregation for robust learning against tailored attacks

    Ramezani-Kebrya, A., Tabrizian, I., Faghri, F., and Popovski, P. Mixtailor: Mixed gradient aggregation for robust learning against tailored attacks. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=tqDhrbKJLS

    2022

  73. [81]

    Distributed extra-gradient with optimal complexity and communication guarantees

    Ramezani-Kebrya, A., Antonakopoulos, K., Krawczuk, I., Deschenaux, J., and Cevher, V. Distributed extra-gradient with optimal complexity and communication guarantees. In International Conference on Learning Representations (ICLR), 2023

    2023

  74. [82]

    Adversarially robust generalization requires more data

    Schmidt, L., Santurkar, S., Tsipras, D., Talwar, K., and Madry, A. Adversarially robust generalization requires more data. In Advances in Neural Information Processing Systems (NeurIPS), 2018

    2018

  75. [83]

    Scalable distributed DNN training using commodity GPU cloud computing

    Strom, N. Scalable distributed DNN training using commodity GPU cloud computing. In INTERSPEECH, 2015

    2015

  76. [84]

    Syrgkanis, V., Agarwal, A., Luo, H., and Schapire, R. E. Fast convergence of regularized learning in games. In Advances in Neural Information Processing Systems (NeurIPS), 2015

    2015

  77. [85]

    Tsianos, K. I. and Rabbat, M. G. Distributed dual averaging for convex optimization under communication delays. In 2012 American Control Conference (ACC), pp.\ 1067--1072. IEEE, 2012

    2012

  78. [86]

    Tsitsiklis, J. N. and Luo, Z.-Q. Communication complexity of convex optimization. Journal of Complexity, 3 0 (3): 0 231--243, 1987

    1987

  79. [87]

    J., Wu, Y., Takac, M., Nandakumar, K., Horv \'a th, S., and Gorbunov, E

    Tupitsa, N., Almansoori, A. J., Wu, Y., Takac, M., Nandakumar, K., Horv \'a th, S., and Gorbunov, E. Byzantine-tolerant methods for distributed variational inequalities. Advances in Neural Information Processing Systems, 36, 2024

    2024

  80. [88]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  81. [89]

    Delay jitter control for real-time communication in a packet switching network

    Verma, D., Zhang, H., and Ferrari, D. Delay jitter control for real-time communication in a packet switching network. In Proceedings of TRICOMM `91: IEEE Conference on Communications Software: Communications for Distributed Applications and Systems, pp.\ 35--43, 1991. doi:10.1...

    1991

  82. [90]

    Theoretically better and numerically faster distributed optimization with smoothness-aware quantization techniques

    Wang, B., Safaryan, M., and Richt \'a rik, P. Theoretically better and numerically faster distributed optimization with smoothness-aware quantization techniques. Advances in Neural Information Processing Systems, 35: 0 9841--9852, 2022

    2022

  83. [91]

    Atomo: Communication-efficient learning via atomic sparsification

    Wang, H., Sievert, S., Liu, S., Charles, Z., Papailiopoulos, D., and Wright, S. Atomo: Communication-efficient learning via atomic sparsification. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018

    2018

  84. [92]

    TernGrad : Ternary gradients to reduce communication in distributed deep learning

    Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y., and Li, H. TernGrad : Ternary gradients to reduce communication in distributed deep learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  85. [93]

    E., Bullins, B., Shamir, O., and Srebro, N

    Woodworth, B. E., Bullins, B., Shamir, O., and Srebro, N. The min-max complexity of distributed stochastic convex optimization with intermittent communication. In Annual Conference Computational Learning Theory, 2021. URL https://api.semanticscholar.org/CorpusID:231749558

    2021

  86. [94]

    Mixed nash for robust federated learning

    Xie, W., Pethick, T., Ramezani-Kebrya, A., and Cevher, V. Mixed nash for robust federated learning. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=mqMzerrVOB

    2024

  87. [95]

    Kimad: Adaptive gradient compression with bandwidth awareness

    Xin, J., Ilin, I., Zhang, S., Canini, M., and Richt\' a rik, P. Kimad: Adaptive gradient compression with bandwidth awareness. In Proceedings of the 4th International Workshop on Distributed Machine Learning, DistributedML '23, pp.\ 35–48, New York, NY, USA, 2023. Association ...

    2023

  88. [96]

    W., Huang, L., Lin, Q., Salakhutdinov, R., and Carbonell, J

    Yu, A. W., Huang, L., Lin, Q., Salakhutdinov, R., and Carbonell, J. Block-normalized gradient method: An empirical study for training deep neural network. arXiv preprint arXiv:1707.04822, 2017

    2017 arXiv

  89. [97]

    Distributed dual averaging method for multi-agent optimization with quantized communication

    Yuan, D., Xu, S., Zhao, H., and Rong, L. Distributed dual averaging method for multi-agent optimization with quantized communication. Systems & Control Letters, 61 0 (11): 0 1053--1061, 2012

    2012

  90. [98]

    Zeiler, M. D. and Fergus, R. Visualizing and understanding convolutional networks. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pp.\ 818--833. Springer, 2014

    2014

  91. [99]

    Understanding deep learning (still) requires rethinking generalization

    Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64 0 (3): 0 107–115, feb 2021. ISSN 0001-0782. doi:10.1145/3446776. URL https://doi.org/10.1145/3446776

    2021 doi

  92. [100]

    ZipML : Training linear models with end-to-end low precision, and a little bit of deep learning

    Zhang, H., Li, J., Kara, K., Alistarh, D., Liu, J., and Zhang, C. ZipML : Training linear models with end-to-end low precision, and a little bit of deep learning. In International Conference on Machine Learning (ICML), 2017

    2017

  93. [101]

    Layer-wise learning based stochastic gradient descent method for the optimization of deep convolutional neural network

    Zheng, Q., Tian, X., Jiang, N., and Yang, M. Layer-wise learning based stochastic gradient descent method for the optimization of deep convolutional neural network. Journal of Intelligent & Fuzzy Systems, 37 0 (4): 0 5641--5654, 2019

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.