Pith. sign in

REVIEW 3 major objections 5 minor 76 references

Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Distributed deep learning can tolerate partial gradient collectives: faster workers need not wait for stragglers, and training still converges.

desk verdict A useful systems idea with a real proof gap: the convergence theorem's bounded-staleness assumption is asserted, not established, for either implemented collective. read the letter →

arxiv 1908.04207 v5 pith:OF6CYKPW submitted 2019-08-12 cs.DC cs.LG

Shigang Li , Tal Ben-Nun , Salvatore Di Girolamo , Dan Alistarh , Torsten Hoefler This is my paper · ORCID
classification cs.DCcs.LG
keywords stochasticgradientdescentdistributeddeeplearningeager-SGDworkloadimbalancecollectiveoperationspartialcollectivesallreduceconvergenceanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Distributed training of deep networks normally forces every worker to wait for the slowest process at each gradient-reduction step, which wastes time whenever workloads are imbalanced. This paper argues that this synchronization can be relaxed: if the blocking allreduce is replaced by a partial collective that completes once a quorum of gradients has arrived, the training can still converge, and the faster workers can run ahead. The proposed eager-SGD uses two partial collectives, solo allreduce (one initiator, wait-free) and majority allreduce (a randomly designated initiator, on average half the processes), and proves convergence for smooth non-convex objectives under bounded staleness. Experiments on CIFAR-10, ImageNet, and UCF101 report up to 1.29x speedup over synchronous decentralized SGD on ImageNet and a 1.27x speedup on an inherently imbalanced video task, with accuracy approximately equivalent in the majority case.

What carries the argument

The load-bearing mechanism is the partial collective operation, in particular the two allreduce variants it introduces. Solo allreduce is a wait-free schedule that a single fast process can trigger, using an activation broadcast followed by the reduction over whatever data is present; majority allreduce waits for a randomly designated initiator, giving an expected active set of half the processes. These are implemented as persistent, consumable schedules in a communication library, so multiple asynchronous rounds can be executed without application intervention. The theoretical machinery is an auxiliary-iterate analysis: the ideal iterate $\Lambda_t$ tracks a clean SGD path, and the partial collective's quorum and staleness bounds place a quantitative ceiling on how far any local view $w_t^i$ can drift from it, which is exactly the term that the convergence proof has to control.

What would settle it

Instrument the partial allreduce to log, per process, the longest run of consecutive rejected gradients. Under solo allreduce with a permanently slowest process, if that run grows without bound as training proceeds, then the staleness parameter $\tau$ is not finite and the theorem's drift bound $\alpha^2 \tau M^2(P-Q)/P^2$ does not apply; a run where every process's rejection run stays bounded would confirm the assumption in practice.

Watch

Extended reading notes

Core claim

At the core of the paper is the claim that synchronous allreduce is not necessary for correct distributed SGD. Eager-SGD replaces each round's global gradient accumulation with an asynchronous distributed sum: each process proposes its local gradient, but the collective completes when a designated initiator triggers it — the fastest process in solo mode, or a randomly chosen process in majority mode, which on average splits the participants in half. Gradients that arrive late are carried into the next round or contributed as null values, so the slowest process never blocks the others. The proof introduces an auxiliary iterate $\Lambda_t$ that would follow a clean full-participation SGD; the partial-collective guarantees of quorum size $Q \ge 1$ and bounded staleness $\tau$ imply $\mathbb{E}[\|\Lambda_t - w_t^i\|^2] \le \alpha^2 \tau M^2(P-Q)/P^2$, and this bound controls the error terms in the $L$-smooth descent inequality. Consequently, with a constant learning rate below a threshold and $T = \Theta\bigl((f(w_0)-m)\sqrt{\tau(P-Q)}\,/(P\epsilon^{3/2})\bigr)$ rounds, some iterate satisfies $\mathbb{E}\|\nabla f(w_{t^\star})\|^2 \le \epsilon$.

Load-bearing premise

The proof assumes that no gradient can be excluded from the collective for more than a fixed number of consecutive rounds, and that at least one process's gradient is accepted every round; if a persistently slow process could have its updates delayed without bound, the drift bound that anchors the convergence proof would no longer hold.

Editorial extensions

If this is right

  • Replacing blocking allreduce with solo allreduce under light load imbalance yields up to 1.29x speedup on ImageNet while keeping top-1 test accuracy within roughly half a point of synchronous SGD.
  • Under severe imbalance, majority allreduce matches synchronous SGD accuracy on the UCF101 LSTM workload (average top-1 69.7% vs 69.6%) and reduces training time by 1.27x; solo allreduce is faster but loses accuracy in this regime.
  • The convergence bound $T = \Theta((f(w_0)-m)\sqrt{\tau(P-Q)}/(P\epsilon^{3/2}))$ means staleness $\tau$ and missed-gradients-per-round $P-Q$ control how many extra rounds are needed, giving a tunable trade-off between synchronization cost and convergence speed.
  • Because eager-SGD propagates each update to all processes in one communication step, it outperforms asynchronous parameter-server SGD by 2.64x and gossip-based decentralized SGD on ImageNet throughput, without a central server.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the proof's assumptions (quorum $Q\ge1$, finite staleness $\tau$) are general enough that the same convergence argument should cover any partial collective that waits for a fixed number $Q$ of participants, not only the two variants tested; the paper mentions this spectrum as future work rather than exploring it.
  • Editorial extension: a workload-adaptive rule that observes recent rejection counts and switches between solo and majority modes could combine solo's speed in light imbalance with majority's accuracy under heavy skew; the paper evaluates the two modes separately but not such a rule.
  • Editorial extension: the bound $\mathbb{E}[\|\Lambda_t - w_t^i\|^2] \le \alpha^2 \tau M^2(P-Q)/P^2$ could serve as an online drift monitor if the auxiliary iterate is reconstructed from accepted gradients, turning a proof device into a practical diagnostic the paper does not describe.
  • Editorial extension: since the mechanism only uses quorum, staleness, and stochastic gradients, partial collectives could reduce synchronization stalls in other parallel iterative algorithms such as federated averaging or consensus; the paper notes this possibility without testing it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes eager-SGD, a decentralized asynchronous variant of SGD that replaces the global blocking allreduce with two partial collectives, solo and majority allreduce, implemented on top of the fflib2 library. The authors prove a convergence result for non-convex objectives under assumptions of quorum size and bounded staleness, and report experiments on CIFAR-10, ImageNet, and UCF101 showing speedups of 1.27x–2.64x over synchronous and asynchronous baselines. An artifact appendix provides code and scripts to reproduce the experiments.

Significance. Partial collectives are a compelling middle ground between synchronous and fully asynchronous distributed training, and the paper's microbenchmarks and large-scale experiments address realistic load-imbalance scenarios. The artifact appendix and the detailed description of the collective schedules are concrete strengths that improve reproducibility. However, the convergence theorem is built on a skipped proof of bounded staleness that the implemented algorithms do not actually provide, and the abstract's 'without losing accuracy' claim is contradicted by the paper's own reported accuracy numbers. If the theory is repaired or reframed, the contribution could be significant; in its current form the central convergence claim is not established.

major comments (3)
  1. [Section 6.1, Lemma 6.1] Lemma 6.1 asserts four properties of each ADS object, including the Quorum Size and the Staleness Bound, but its proof is skipped with a one-sentence justification. The Staleness Bound (item 4) is load-bearing: Lemma 6.2 and Theorem 6.3 both require a finite uniform tau over all rounds. For solo allreduce, Section 1 explicitly states that it 'does not guarantee bounded error, as necessary in SGD', and Section 5 describes a slow process's send buffer being overwritten indefinitely, which implies unbounded rejection streaks. For majority allreduce, Section 4.2 provides only the statistical statement that the randomly chosen initiator is on average the P/2-th process; because the number of consecutive rejections of a given process is geometric, it is unbounded with positive probability. The claimed iteration complexity T = Theta((f(w0)-m) sqrt(tau(P-Q))/(P epsilon^{3/2})) is therefore unsupported for both implemented variants.
  2. [Section 6.2, Lemma 6.2] Even granting Lemma 6.1, the proof of Lemma 6.2 is not rigorous as written. Equation (1) writes the norm of the distance between Lambda_t and w_i^t as an infinite sum over an index also called t, and the truncation to tau terms in Eq. (2) is not justified in the notation of the proof because the outer training round t is the same symbol. The bound in Eq. (5) has the correct order only after a careful accounting of the per-round number of missing gradients and the algebraic factor from the 1/P scaling; the current derivation does not pin down the measurability of the indicators delta_j^t with respect to the natural filtration. A clean indexation of rounds and a fully stated proof are needed before the theorem's precondition can be considered established.
  3. [Section 7.2.2 and Section 7.3] The abstract claims a speedup 'without losing accuracy,' but the paper's own data contradict this for solo allreduce. In Fig. 11c, eager-SGD with solo allreduce reaches 75.2% top-1 test accuracy on ImageNet versus 75.7–75.8% for the synchronous baselines; in Fig. 13b, solo allreduce on UCF101 reaches 60.6% average top-1 test accuracy versus 69.6% for Horovod. The body text adequately qualifies these cases, but the abstract and the opening of Section 7.2.2 ('without losing accuracy for deep neural networks in light load imbalance environment') should either restrict the claim to majority allreduce and the tested conditions or report the accuracy difference explicitly.
minor comments (5)
  1. [Abstract and Section 7] The abstract reports 1.27x speedup over synchronous SGD, while the full text also reports 1.29x for ResNet-50 on ImageNet; the headline number should be stated consistently with the experimental section.
  2. [Section 4.2] The text says the expectation of the randomly specified initiator is the P/2-th process; a uniform random rank among P sorted processes has expectation (P+1)/2, not exactly P/2. The statement 'on average half' is approximately correct but the precise wording is off by half a process.
  3. [Section 6.2, Eq. (1)–(3)] The notation reuses t for the training round and for the summation index in Eq. (1); using a different symbol (e.g., s) for the times of missing gradients would improve readability and avoid the apparent infinite sum over the current round.
  4. [Section 7.3, Fig. 13] The text reports average and maximum accuracies for the UCF101 experiments, but the standard-deviation bands in the figure are not translated into numeric confidence intervals in the text; reporting mean ± std for the key comparisons would make the 'equivalent accuracy' claim easier to assess.
  5. [Section 7.2.2, Fig. 11c] The ImageNet runs are repeated three times per configuration, but the paper does not report per-run variation or a significance test; given the 0.5 percentage point gap between eager-SGD and the baselines, the claim of 'without losing accuracy' needs error bars or a statement about run-to-run variance.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the convergence proof is a conditional derivation from explicitly stated assumptions; the only flagged weakness is an unproved bounded-staleness lemma, which is a soundness gap rather than a circular reduction.

full rationale

The paper's central convergence claim (Theorem 6.3) is derived, not assumed: the proof introduces the auxiliary iterate Lambda_t, applies L-smoothness, unbiasedness, the second-moment bound, and the drift bound of Lemma 6.2, which in turn follows algebraically from Lemma 6.1's quorum-size and bounded-staleness properties. No fitted constant is renamed as a prediction, and no equation in the theorem is identical to an input by construction. The weak point is Lemma 6.1: its proof is explicitly skipped ('The proof of the above properties follows directly from the structure of the partial-allreduce algorithm, and is therefore skipped'), and the algorithm section only establishes an average quorum for majority allreduce while admitting that solo allreduce 'does not guarantee bounded error'. Thus finite tau may fail for both proposed variants, making the theorem conditional on an unverified premise. That is a correctness/soundness concern, not circularity: the theorem remains a valid conditional statement under Lemma 6.1. The self-citations, including [17] for solo collectives, [9] for Deep500, and [24] for sPIN, overlap with the author list, but the convergence proof does not rest on them; Lemma 6.1 is claimed to follow from the algorithm structure itself. The empirical evaluation compares against external baselines (Horovod, TensorFlow asynchronous parameter server, D-PSGD, SGP) and is accompanied by an artifact appendix for reproduction, so no fitted input is being relabeled as a prediction. The score of 2 reflects only the presence of minor, non-load-bearing self-citation; no circular step is exhibited.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central proof rests on standard smooth non-convex optimization assumptions plus the asserted properties of the partial allreduce ADS objects (Lemma 6.1). The latter are not proven for the implemented collectives, and the majority-collective statistical guarantee is an average-case property, not the lower-bound quorum used in the theorem. No free parameters are fitted to data.

assumptions (5)
  • domain assumption The loss function is bounded below and L-smooth (Assumption 1).
    Standard assumptions for non-convex SGD analysis; used to bound the decrease of the objective in Theorem 6.3.
  • domain assumption Stochastic gradients are unbiased and have bounded second moment (Assumption 2).
    Standard in stochastic optimization; used for the second-moment bound and in Lemma 6.2.
  • ad hoc to paper The ADS objects satisfy Lemma 6.1: liveness, safety, quorum size Q>=1, and bounded staleness tau.
    These properties are asserted and the proof is skipped; the convergence proof relies on them, especially the bounded staleness tau which is not proven for the implemented solo/majority collectives.
  • domain assumption Each process eventually submits an update to each round's ADS, and there is a global serialization into rounds.
    Used in the analytic view to define Lambda_t and the 'missed gradients' bound in Lemma 6.2.
  • domain assumption For majority allreduce, the workload distribution has one mode and a tail so that the random initiator yields on average at least half of the processes participating.
    Used in Section 4.2 to argue the statistical guarantee of majority collectives; it is not used in the formal proof but motivates the quorum claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations." pith.science (2026). https://pith.science/paper/OF6CYKPW

@misc{pith2026190804207,
  author       = {Pith},
  title        = {Pith review of: Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OF6CYKPW}},
  note         = {Machine review of arXiv:1908.04207}
}
read the original abstract

Load imbalance pervasively exists in distributed deep learning training systems, either caused by the inherent imbalance in learned tasks or by the system itself. Traditional synchronous Stochastic Gradient Descent (SGD) achieves good accuracy for a wide variety of tasks, but relies on global synchronization to accumulate the gradients at every training step. In this paper, we propose eager-SGD, which relaxes the global synchronization for decentralized accumulation. To implement eager-SGD, we propose to use two partial collectives: solo and majority. With solo allreduce, the faster processes contribute their gradients eagerly without waiting for the slower processes, whereas with majority allreduce, at least half of the participants must contribute gradients before continuing, all without using a central parameter server. We theoretically prove the convergence of the algorithms and describe the partial collectives in detail. Experimental results on load-imbalanced environments (CIFAR-10, ImageNet, and UCF101 datasets) show that eager-SGD achieves 1.27x speedup over the state-of-the-art synchronous SGD, without losing accuracy.

Figures

Figures reproduced from arXiv: 1908.04207 by the authors.

Figure 1
Figure 1. Synch-SGD vs eager-SGD under load imbalance. w(t) are the weights in training step t. easily be avoided. Natural language processing tasks have sentences of highly varying length while video processing tasks have videos with different number of frames. For ex￾ample, the training dataset of UCF101 [53] contains videos that range from 29 to 1,776 frames. Several researchers have shown that the training process itself … view at source ↗
Figure 2
Figure 2. Load imbalance in the training of an LSTM model on UCF101 [53]. 0 1000 2000 3000 4000 500 1000 1500 2000 2500 3000 3500 Runtime (milliseconds) Number of batches [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Runtime distribution on a P100 GPU (batch size = 64), using a Transformer model on WMT16. 0 5000 10000 15000 20000 400 600 800 1000 1200 1400 1600 1800 Runtime (milliseconds) Number of batches [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Runtime distribution on Google Cloud with 2xV100 GPUs (batch size=256, ResNet-50 on ImageNet). distributed from 179 ms to 3,482 ms with a mean of 475 ms and standard deviation of 144 ms, which shows the inherent load imbalance in language model training. 2.3 Training i…
Figure 6
Figure 6. Figure 6: Solo collective activation (left) and process sched￾ule (right). Operations are represented by circles: blue = send, green = receive, orange = computation, white = NOP. A dashed border means the operation can be fired as soon as one of its dependencies are satisfied. m…
Figure 7
Figure 7. Figure 7: presents an example of how eager-SGD works with partial collectives, in which wp t and Gp t represent the weights and the gradients calculated on process p at training step t, respectively, and U (G,w) represents the update rule. In step t, suppose process P1 is faster…
Figure 9
Figure 9. Figure 9: Average latency comparison between MPI_Allreduce and partial allreduce running on 32 processes by 64 iterations. Processes are linearly skewed by injecting load imbalance from 1 ms to 32 ms. For the partial collective operations, we refer to the initia￾tor together wit…
Figure 8
Figure 8. Figure 8 [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 10
Figure 10. Figure 10: Comparison between synch-SGD and eager-SGD for hyperplane regression using 8 processes. "synch/eager￾SGD-200/300/400" represent 200/300/400 ms load imbalance injection, respectively. Each point is at the boundary of one epoch. a0x0 +a1x1 +...+a8191x8191 +noise, where …
Figure 11
Figure 11. Figure 11: Comparisons between synch-SGD and eager￾SGD for ResNet-50 on ImageNet using 64 processes. "synch/eager-SGD-300/460" represent 300/460 ms load im￾balance injection, respectively [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Top-1 test accuracy of synch-SGD (Horovod) and eager-SGD for ResNet-32 on Cifar-10 using 8 processes. Each point is at the boundary of every 10 epochs. the lower accuracy problem caused by solo allreduce, which achieves approximately equivalent accuracy to synch-SGD w…
Figure 13
Figure 13. Figure 13: Training results for LSTM on UCF101 using 8 processes. Each point is at the boundary of one epoch. on average 86.1% top-1 train accuracy and 96.6% top-5 train accuracy, while eager-SGD using majority allreduce achieves on average 86.7% top-1 train accuracy and 96.1% t…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 50 canonical work pages

  1. [1]

    Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murra...

    2015

  2. [2]

    Dan Alistarh, Torsten Hoefler, Mikael Johansson, Sarit Khirirat, Nikola Konstantinov, and Cedric Renggli. 2018. The Convergence of Sparsi- fied Gradient Methods. In Advances in Neural Information Processing Systems 31. Curran Associates, Inc

    work page 2018

  3. [3]

    Dario Amodei and Danny Hernandez. 2018. AI and Compute. https://openai.com/blog/ai-and-compute/

    work page 2018

  4. [4]

    Mahmoud Assran, Nicolas Loizou, Nicolas Ballas, and Michael Rabbat

    work page

  5. [5]

    A. Awan, K. Hamidouche, J. Hashmi, and D. Panda. 2017. S-Caffe: Co-designing MPI Runtimes and Caffe for Scalable Deep Learning on Modern GPU Clusters

    work page 2017

  6. [6]

    Jimmy Ba and Brendan Frey. 2013. Adaptive dropout for training deep neural networks. In Advances in Neural Information Processing Systems. 3084–3092

    work page 2013

  7. [7]

    Nicolas Ballas, Li Yao, Chris Pal, and Aaron Courville. 2015. Delving Deeper into Convolutional Networks for Learning Video Representa- tions. arXiv e-prints (2015). arXiv:1511.06432

    arXiv 2015

  8. [8]

    Brian W Barrett, Ron Brightwell, , E Ryan Grant, Scott Hemmert, Kevin Pedretti, Kyle Wheeler, Keith D Underwood, R Reisen, Torsten Hoefler, Arthur B Maccabe, and Trammell Hudson. 2018. The Portals 4.2 network programming interface. Sandia National Laboratories, November 2018, Technical Report SAND2017-3825 (2018)

    work page 2018

Show all 76 references
  1. [9]

    Ben-Nun, M

    T. Ben-Nun, M. Besta, S. Huber, A. N. Ziogas, D. Peter, and T. Hoefler

  2. [10]

    Ben-Nun and T

    T. Ben-Nun and T. Hoefler. 2018. Demystifying Parallel and Dis- tributed Deep Learning: An In-Depth Concurrency Analysis. CoRR 13 abs/1802.09941 (Feb. 2018)

    2018 arXiv

  3. [11]

    Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalya- naraman. 2014. Project Adam: Building an Efficient and Scalable Deep Learning Training System. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14) . USENIX Association, Broomfiel...

    2014

  4. [12]

    Jeff Daily, Abhinav Vishnu, Charles Siegel, Thomas Warfel, and Vinay Amatya. 2018. GossipGraD: Scalable Deep Learning using Gos- sip Communication based Asynchronous Gradient Descent. CoRR abs/1803.05880 (2018). arXiv:1803.05880 http://arxiv.org/abs/1803. 05880

    2018 arXiv

  5. [13]

    Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, and Andrew Y. Ng. 2012. Large scale distributed deep networks. InAdvances in neural information processing systems . 1223–1231

    2012

  6. [14]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei

  7. [15]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

  8. [16]

    Terrance Devries and Graham W. Taylor. 2017. Improved Regu- larization of Convolutional Neural Networks with Cutout. CoRR abs/1708.04552 (2017). arXiv:1708.04552 http://arxiv.org/abs/1708. 04552

    2017 arXiv

  9. [17]

    Salvatore Di Girolamo, Pierre Jolivet, Keith D Underwood, and Torsten Hoefler. 2015. Exploiting offload enabled network interfaces. In 2015 IEEE 23rd Annual Symposium on High-Performance Interconnects . IEEE, 26–33

    2015

  10. [18]

    Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell

  11. [19]

    CoRR abs/1810.04805 (2018)

    BERT: Pre-training of Deep Bidirectional Transformers for Lan- guage Understanding. CoRR abs/1810.04805 (2018). arXiv:1810.04805 http://arxiv.org/abs/1810.04805

    2018 arXiv

  12. [20]

    Suyog Gupta, Wei Zhang, and Fei Wang. 2015. Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study. arXiv e-prints (Sep 2015). arXiv:1509.04210

    2015 arXiv

  13. [21]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 770–778

    2016

  14. [22]

    Gibbons, Garth A

    Qirong Ho, James Cipar, Henggang Cui, Jin Kyu Kim, Seunghak Lee, Phillip B. Gibbons, Garth A. Gibson, Gregory R. Ganger, and Eric P. Xing. 2013. More Effective Distributed ML via a Stale Syn- chronous Parallel Parameter Server. In Proceedings of the 26th Inter- national Confer...

    2013

  15. [23]

    Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780

    1997

  16. [24]

    Andrew Gibiansky. 2017. Bringing HPC techniques to deep learn- ing.(2017). URL http://research. baidu. com/bringing-hpc-techniques- deep-learning (2017)

    2017

  17. [25]

    Torsten Hoefler, Andrew Lumsdaine, and Wolfgang Rehm. 2007. Im- plementation and performance analysis of non-blocking collective operations for MPI. In Proceedings of the 2007 ACM/IEEE conference on Supercomputing. ACM, 52

    2007

  18. [26]

    Hoefler and D

    T. Hoefler and D. Moor. 2014. Energy, Memory, and Runtime Tradeoffs for Implementing Collective Communication Operations. Journal of Supercomputing Frontiers and Innovations 1, 2 (Oct. 2014), 58–75

    2014

  19. [27]

    Hoefler, T

    T. Hoefler, T. Schneider, and A. Lumsdaine. 2009. The Effect of Network Noise on Large-Scale Collective Communications. Parallel Processing Letters (PPL) 19, 4 (Aug. 2009), 573–593

    2009

  20. [28]

    Hoefler, T

    T. Hoefler, T. Schneider, and A. Lumsdaine. 2010. Characterizing the In- fluence of System Noise on Large-Scale Applications by Simulation. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC’10)

    2010

  21. [29]

    Torsten Hoefler, Salvatore Di Girolamo, Konstantin Taranov, Ryan E Grant, and Ron Brightwell. 2017. sPIN: High-performance streaming Processing in the Network. In Proceedings of the International Confer- ence for High Performance Computing, Networking, Storage and Analysis. ACM, 59

    2017

  22. [30]

    Huang, Z

    G. Huang, Z. Liu, L. v. d. Maaten, and K. Q. Weinberger. 2017. Densely Connected Convolutional Networks. In 2017 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR) . 2261–2269. https://doi.org/10.1109/CVPR.2017.243

    2017 doi

  23. [31]

    Alexandru Iosup, Nezih Yigitbasi, and Dick Epema. 2011. On the perfor- mance variability of production cloud services. In 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing . IEEE, 104–113

    2011

  24. [32]

    Keith R Jackson, Lavanya Ramakrishnan, Krishna Muriki, Shane Canon, Shreyas Cholia, John Shalf, Harvey J Wasserman, and Nicholas J Wright. 2010. Performance analysis of high performance computing applications on the amazon web services cloud. In2nd IEEE international conferenc...

    2010

  25. [33]

    Anand Jayarajan, Jinliang Wei, Garth Gibson, Alexandra Fedorova, and Gennady Pekhimenko. 2019. Priority-based parameter propagation for distributed DNN training. In Proceedings of the 2nd SysML Conference

    2019

  26. [34]

    Ganger, Phillip B

    Kevin Hsieh, Aaron Harlap, Nandita Vijaykumar, Dimitris Konomis, Gregory R. Ganger, Phillip B. Gibbons, and Onur Mutlu. 2017. Gaia: Geo-distributed Machine Learning Approaching LAN Speeds. In Pro- ceedings of the 14th USENIX Conference on Networked Systems Design and Implement...

    2017

  27. [35]

    Thorsten Kurth, Sean Treichler, Joshua Romero, Mayur Mudigonda, Nathan Luehr, Everett Phillips, Ankur Mahesh, Michael Matheson, Jack Deslippe, Massimiliano Fatica, Prabhat, and Michael Houston. 2018. Exascale Deep Learning for Climate Analytics. In Proceedings of the Internati...

    2018

  28. [36]

    LeCun, Y

    Y. LeCun, Y. Bengio, and G. Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436–444

    2015

  29. [37]

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324

    1998

  30. [38]

    Andersen, Jun Woo Park, Alexander J

    Mu Li, David G. Andersen, Jun Woo Park, Alexander J. Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J. Shekita, and Bor-Yiing Su. 2014. Scaling Distributed Machine Learning with the Parameter Server. In Proceedings of the 11th USENIX Conference on Operating Sys- tems D...

    2014

  31. [39]

    Jin, Qiaochu Yuan, Forrest N

    Peter H. Jin, Qiaochu Yuan, Forrest N. Iandola, and Kurt Keutzer. 2016. How to scale distributed deep learning? CoRR abs/1611.04581 (2016). arXiv:1611.04581 http://arxiv.org/abs/1611.04581

    2016 arXiv

  32. [40]

    Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. 2018. Asynchronous Decentralized Parallel Stochastic Gradient Descent. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of 14 Machine Learning Research), Jennifer Dy and Andreas Krause (Eds.), V...

    2018

  33. [41]

    Pennycook, Kristyn Maschhoff, Jason Sewall, Nalini Kumar, Shirley Ho, Michael F

    Amrita Mathuriya, Deborah Bard, Peter Mendygral, Lawrence Mead- ows, James Arnemann, Lei Shao, Siyu He, Tuomas Kärnä, Diana Moise, Simon J. Pennycook, Kristyn Maschhoff, Jason Sewall, Nalini Kumar, Shirley Ho, Michael F. Ringenburg, Prabhat, and Victor Lee. 2018. Cos- moFlow: ...

    2018

  34. [42]

    Message Passing Interface Forum. 2015. MPI: A Message-Passing Interface Standard Version 3.1

    2015

  35. [43]

    Pitch Patarasuk and Xin Yuan. 2009. Bandwidth Optimal All-reduce Algorithms for Clusters of Workstations. J. Parallel Distrib. Comput. 69, 2 (Feb. 2009), 117–124. https://doi.org/10.1016/j.jpdc.2008.09.002

    2009 doi

  36. [44]

    Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. 2017. Can Decentralized Algorithms Outperform Centralized Al- gorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent. In Proceedings of the 31st International Conference on Neural ...

    2017

  37. [45]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language Models are Unsupervised Mul- titask Learners. (2018). https://d4mucfpksywv.cloudfront.net/ better-language-models/language-models.pdf

    2018

  38. [46]

    Recht, C

    B. Recht, C. Re, S. Wright, and F. Niu. 2011. Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent. In Advances in Neural Information Processing Systems 24 . 693–701

    2011

  39. [47]

    Cédric Renggli, Dan Alistarh, and Torsten Hoefler. 2018. SparCML: High-Performance Sparse Communication for Machine Learning. CoRR abs/1802.08021 (2018). arXiv:1802.08021 http://arxiv.org/abs/ 1802.08021

    2018 arXiv

  40. [48]

    Herbert Robbins and Sutton Monro. 1951. A Stochastic Approximation Method. The Annals of Mathematical Statistics (1951)

    1951

  41. [49]

    Rolf Rabenseifner. 2004. Optimization of collective reduction opera- tions. In International Conference on Computational Science . Springer, 1–9

    2004

  42. [50]

    Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 2014. 1- Bit Stochastic Gradient Descent and its Application to Data-Parallel Distributed Training of Speech DNNs. In Fifteenth Annual Conference of the International Speech Communication Association

    2014

  43. [51]

    Alexander Sergeev and Mike Del Balso. 2018. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.05799 (2018)

    2018 arXiv

  44. [52]

    Karen Simonyan and Andrew Zisserman. 2014. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

    2014 arXiv

  45. [53]

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)

    2012 arXiv

  46. [54]

    Jörg Schad, Jens Dittrich, and Jorge-Arnulfo Quiané-Ruiz. 2010. Run- time measurements in the cloud: observing, analyzing, and reducing variance. Proceedings of the VLDB Endowment 3, 1-2 (2010), 460–471

    2010

  47. [55]

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition . 2818–2826

    2016

  48. [56]

    Rajeev Thakur, Rolf Rabenseifner, and William Gropp. 2005. Opti- mization of collective communication operations in MPICH. The International Journal of High Performance Computing Applications 19, 1 (2005), 49–66

    2005

  49. [57]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. At- tention is all you need. In Advances in Neural Information Processing Systems. 5998–6008

    2017

  50. [58]

    Pengtao Xie, Jin Kyu Kim, Yi Zhou, Qirong Ho, Abhimanu Kumar, Yaoliang Yu, and Eric Xing. 2016. Lighter-communication Distributed Machine Learning via Sufficient Factor Broadcasting. In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence (UAI’...

    2016

  51. [59]

    Nikko Strom. 2015. Scalable distributed DNN training using com- modity GPU cloud computing. In Sixteenth Annual Conference of the International Speech Communication Association

    2015

  52. [60]

    Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. 2015. Beyond Short Snippets: Deep Networks for Video Classification. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2015

  53. [61]

    Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. 2015. Beyond short snippets: Deep networks for video classification. In Proceedings of the IEEE conference on computer vision and pattern recognition . 4694–4702

    2015

  54. [62]

    Wei Zhang, Suyog Gupta, Xiangru Lian, and Ji Liu. 2015. Staleness- aware async-sgd for distributed deep learning. arXiv preprint arXiv:1511.05950 (2015). 15 A Artifact Appendix A.1 Abstract We provide source code of eager-SGD and scripts to run experiments from the paper. This...

    2015 arXiv

  55. [64]

    Yang You, Zhao Zhang, Cho-Jui Hsieh, James Demmel, and Kurt Keutzer. 2018. Imagenet training in minutes. In Proceedings of the 47th International Conference on Parallel Processing . ACM, 1

    2018

  56. [68]

    Extract the artifact using unzip

    Download the artifact and move it to your personal $WORK direc- tory. Extract the artifact using unzip

  57. [69]

    > cd $WORK/eager-SGD-artifact/eager-SGD > pip install -r requirements.txt

    Install the dependent Python modules. > cd $WORK/eager-SGD-artifact/eager-SGD > pip install -r requirements.txt

  58. [70]

    > cd $WORK/eager-SGD-artifact/eager-SGD/fflib2/lib > cmake

    Compile fflib2 and set the environment variable. > cd $WORK/eager-SGD-artifact/eager-SGD/fflib2/lib > cmake .. && make > export LD_LIBRARY_PATH=$WORK/eager-SGD-artifact/ eager-SGD/fflib2/lib:$LD_LIBRARY_PATH

  59. [71]

    Configure a CMakelist file which will be used for compiling the customized Tensorflow operators. > vim $WORK/eager-SGD-artifact/eager-SGD/deep500/ deep500/frameworks/tensorflow/custom_operators/ CMakeLists.txt Update include_directories and link_directories by the path where f...

  60. [72]

    <" means

    Evaluate solo and majority allreduce and generate Fig. 9. > cd $WORK/eager-SGD-artifact/test-scripts/allreduce -scripts Submit the jobs. > ./sbatch_jobs.sh It may take about 10 minutes to finish the jobs, and then outputs majority.txt, solo.txt, and mpi.txt. Next, run the file...

  61. [73]

    Train hyperplane regression and generate Fig. 10. > cd ./$WORK/eager-SGD-artifact/test-scripts/ hyperplane-scripts 16 Submit the jobs. > ./sbatch_jobs.sh It may take about 20 minutes to finish the jobs, and then outputs solo200.txt, solo300.txt, solo400.txt, dfive200.txt, dfiv...

  62. [74]

    Train ResNet-50 on ImageNet and generate Fig. 11. Generate the TensorFlow data format for ImageNet, which may take several hours. > python $WORK/eager-SGD-artifact/test-models/tf- models-r1.11/official/data/build_imagenet_data.py > cd $WORK/eager-SGD-artifact/test-scripts/imag...

  63. [75]

    <" means

    Train ResNet-32 on CIFAR-10 and generate Fig. 12. > cd $WORK/eager-SGD-artifact/test-scripts/cifar10 -scripts Copy synchm.sh to the checkpoint directory. Submit the jobs. > ./sbatch_jobs.sh It may take about several hours to finish the jobs, and then outputs hvd.txt, major.txt...

  64. [76]

    <" means

    Train LSTM on UCF101 and generate Fig. 13. Extracting features from the raw data. It may take several hours. > python $WORK/eager-SGD-artifact/test-models/lstm- video-classification/extract_features.py Submit the jobs. > cd $WORK/eager-SGD-artifact/test-scripts/lstm- scripts >...

  65. [2009]

    In Pro- ceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    Imagenet: A large-scale hierarchical image database. In Pro- ceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 248–255

    2009

  66. [2014]

    CoRR abs/1411.4389 (2014)

    Long-term Recurrent Convolutional Networks for Visual Recog- nition and Description. CoRR abs/1411.4389 (2014). arXiv:1411.4389 http://arxiv.org/abs/1411.4389

    2014 arXiv

  67. [2018]

    arXiv preprint arXiv:1811.10792 (2018)

    Stochastic gradient push for distributed deep learning. arXiv preprint arXiv:1811.10792 (2018)

    2018 arXiv

  68. [2019]

    In 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

    A Modular Benchmarking Infrastructure for High-Performance and Reproducible Deep Learning. In 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS) . 66–77. https://doi.org/ 10.1109/IPDPS.2019.00018

    2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.