Pith. sign in

REVIEW 3 major objections 6 minor 35 references

MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MPipeMoE claims that adaptive pipeline parallelism with buffer sharing trains Mixture-of-Experts models up to 2.8x faster while cutting GPU memory by up to 47% compared to FasterMoE.

desk verdict A legitimate MoE training-system contribution with plausible adaptive pipelining and memory-reuse ideas, but the headline speedup and memory numbers come from isolated MoE-layer microbenchmarks, not end-to-end pretraining. read the letter →

arxiv 2506.22175 v1 pith:AZ4FQGD5 submitted 2025-06-27 cs.DC

Zheng Zhang , Donglin Yang , Yaqi Xia , Liang Ding , Dacheng Tao , Xiaobo Zhou , Dazhao Cheng This is my paper · ORCID
classification cs.DC
keywords Mixture-of-ExpertspipelineparallelismdistributedtrainingmemoryefficiencyAll-to-AllcommunicationadaptivegranularityGPUCPUoffloading
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Training large Mixture-of-Experts (MoE) models is bottlenecked by All-to-All communication and by GPU memory used for activations and temporary buffers. MPipeMoE attacks both at once: it splits each token batch into micro-batches, pipelines communication against expert computation, and chooses the number of pipeline partitions adaptively at runtime. It then reuses a single memory buffer across partitions, recovering overwritten tensors through recomputation, re-communication, or CPU offloading, and uses a performance model to pick the cheapest recovery strategy. On an 8-node A100 cluster, the paper reports up to 2.8x speedup and up to 47% lower memory footprint than the state-of-the-art FasterMoE. If the result holds, MoE training can run larger batches on the same hardware and spend less time waiting on the network.

What carries the argument

The load-bearing object is a micro-batch pipeline over the MoE layer, where the token batch of size $B$ is cut into $n$ partitions and the first All-to-All, expert computation, and second All-to-All of each partition are overlapped across CUDA streams. Algorithm 1 searches for the best $n$ online, relying on a monotonicity hypothesis: the optimal $n$ is non-decreasing as $B$ increases, so previously searched intervals can be cached. The memory-saving side rests on buffer sharing across partitions and on four recovery strategies (S1: offload both, S2: re-communicate $T_{DI}$ and offload $T_M$, S3: offload $T_{DI}$ and recompute $T_M$, S4: re-communicate and recompute), with a cost model that picks the fastest strategy from measured slowdown factors for computation, communication, and memory-copy streams.

What would settle it

Run an exhaustive grid search over pipeline partition counts $n$ on a real MoE layer with token batch sizes extending beyond the tested 4k-32k range and with non-uniform expert routing; if any interval shows the best $n$ decreasing as $B$ increases, Algorithm 1's monotonicity hypothesis fails, and its cached $n$ would be wrong.

Watch

Extended reading notes

Core claim

The paper's central claim is that the MoE training data flow $T_I \rightarrow T_{DI} \rightarrow T_M \rightarrow T_{DO} \rightarrow T_O$ can be pipelined and compressed at the same time. Unlike FasterMoE, which splits the token batch along the device dimension, MPipeMoE splits along the batch dimension, so each All-to-All remains a single collective call. The paper then observes that activations and temporary buffers dominate memory, and that pipeline partitions are activated sequentially, leaving memory bubbles that can be collapsed into one shared buffer, reducing the storage of $T_{DI}$, $T_M$, and $T_{DO}$ from $m$ to $m/n$. Because sharing overwrites tensors needed in the backward pass, it restores $T_{DI}$ by re-communication or CPU offloading and $T_M$ by recomputation or CPU offloading, choosing among four strategies with a performance model. The empirical upshot is an average memory reduction of about 27% and up to 47% versus FasterMoE, plus up to 2.8x training speedup.

Load-bearing premise

The load-bearing premise is that the optimal number of pipeline partitions grows monotonically as the token batch size grows; if that ordering ever fails, the online search can cache a partition count that is not optimal, and the reported speedups shrink.

Editorial extensions

If this is right

  • Larger token batches can fit on the same GPUs, allowing better GPU utilization and higher throughput in MoE training without out-of-memory failures.
  • The online granularity search removes manual tuning of pipeline depth and tracks the best fixed $n$ across the tested batch-size range.
  • The four recovery strategies cover the tradeoff spectrum from I/O-bound to compute-bound, so no single strategy is expected to dominate under all hardware settings.
  • MPipeMoE's achieved memory saving is close to its theoretical bound, indicating the memory model is predictive rather than optimistic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The monotonicity assumption behind the online search is untested outside the reported batch range; a workload where the best $n$ drops as $B$ grows would need cache invalidation or rollback, which the paper does not describe.
  • Buffer sharing could be pushed further: the gating network's routing tensors, which the paper excludes as small, and activations of surrounding dense layers could also be reused, extending memory savings end-to-end.
  • Because MPipeMoE's speedup is measured against a fixed-granularity pipeline, combining its adaptive granularity with congestion-avoiding expert selection or hierarchical All-to-All could compound the gains, though that combination is not implemented here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents MPipeMoE, a PyTorch-based MoE training library that combines adaptive pipeline parallelism with memory-reuse strategies. The MoE layer is decomposed into sub-stages (dispatch All-to-All, expert computation, gather All-to-All), and a batch of tokens is partitioned into micro-batches so that communication and computation can overlap. An online algorithm (Algorithm 1) searches for the number of pipeline partitions, and a performance model selects among four strategies (S1-S4) that recover overwritten activations via CPU offloading, re-communication, or re-computation. The authors evaluate on an 8-node DGX A100 cluster using three MoE layer configurations (GPT-S, GPT-XL, BERT-L) with dummy random-token inputs, reporting up to 2.8x speedup and up to 47% memory-footprint reduction compared with FasterMoE, and a theoretical memory-saving bound that is claimed to be met at about 95%.

Significance. If the claims hold, MPipeMoE would be a practically useful contribution to MoE training: it addresses two real bottlenecks (All-to-All communication and memory footprint) and ships as a library with a runtime strategy selector. The paper also contains genuinely positive elements: the system is compared against external baselines (FastMoE and FasterMoE), the performance model uses measured interference factors rather than fitted speedups, and the implementation is described at a level that should be reproducible. However, the headline result in the abstract and conclusion is stated for 'training large models,' whereas the experiments cover only isolated MoE layers with dummy tokens, and the memory-equation derivation in Section III.D is internally inconsistent. These issues are central to the paper's main contribution, so the current version requires substantive revision.

major comments (3)
  1. [V.A.2 and V.C] The abstract and conclusion claim that MPipeMoE 'achieves up to 2.8x speedup and reduces memory footprint by up to 47% in training large models,' but the evaluation is restricted to a single MoE layer with dummy random tokens (Section V.A.2). An isolated MoE-layer microbenchmark does not support an end-to-end training claim: the MoE layer is only a fraction of total step time, and the measured speedup from overlapping All-to-All with expert computation can be diluted by attention, embedding, optimizer, and data-parallel gradient communication. Similarly, a 47% reduction in MoE activation/temporary-buffer memory does not imply a 47% reduction in total memory footprint, since model states, attention activations, and optimizer states may dominate in a full pretraining run. The authors should either run end-to-end training of representative MoE transformer models or explicitly narrow the claims to per-MoE-layer speedup and memory savings.
  2. [III.D, Eqs. (4)-(6)] The memory-reuse derivation is not internally consistent. Equation (3) defines Mbuf = BM + BH, but Equation (4) asserts Mpipe_buf = Mpipe_act = 4BM + BH, making the temporary-buffer peak identical to the activation peak despite the different tensor counts. Equation (5) as typeset is dimensionally incorrect: with GPT-XL values (M=2048, H=8192, B=16k, n=4), the right-hand side B*(2M*n - 2/n + H*n - 1/n) is roughly 805 million elements, larger than the original activation memory of 268 million elements, so it cannot be a reduction. This is load-bearing because Equation (6) defines the theoretical memory-saving ratio and Figure 10 reports that MPipeMoE achieves about 95% of that bound. Please rewrite Equations (4)-(6) with a clear derivation of which tensors are shared among partitions and what the actual per-partition peak memory is.
  3. [III.C, Algorithm 1] The adaptive granularity search relies on the hypothesis that the optimal number of partitions n is monotonically non-decreasing in the token batch size B. This assumption is load-bearing: if it is violated on a real workload, the range-merging logic in Algorithm 1 can cache a suboptimal n, and the claimed speedups would not materialize. The only supporting evidence is Figure 12 for a single model (GPT-XL); the monotonicity should be justified analytically or tested across models, hardware configurations, and batch-size sequences. If a rigorous justification is not possible, the paper should provide a fallback that detects and corrects a stale n when the monotonicity assumption fails.
minor comments (6)
  1. [V.C and Figure 9] The speedup reporting should be made consistent across the text and figures: Section V.D reports a 3.1x speedup against FastMoE and a 2.8x speedup against FasterMoE, while Figure 8 shows PipeMoE reaching up to 3.4x against FasterMoE; please clarify which configuration and which comparison each number refers to.
  2. [V.A and Figures 8-13] No error bars, confidence intervals, or number of repeated runs are reported for any speedup or memory measurement; please state the number of trials and report mean and variance or at least the range across runs.
  3. [III.C, Algorithm 1] The range variables Blower_n and Bupper_n are used in lines 11-14 but their update rule is described only informally; please define the range-merge operation and the cache-invalidation condition precisely.
  4. [Figure 2] The left y-axis is labeled 'memory proportion' (0-1) and the right y-axis 'GPU utilization' (0-100%), but the figure does not clearly indicate which curves belong to which axis; please add a legend or explicit curve labels.
  5. [III.E, Eq. (10)] The symbol C is used both for the end-to-end execution time in Equation (10) and for the 'cost' of a strategy in Section III.E; please use distinct notation and define the time unit, since the right-hand side has units of inverse throughput.
  6. [Throughout] There are typographical inconsistencies such as 'state-of-art' vs 'state-of-the-art' and 'MpipeMoE' vs 'MPipeMoE'; please unify spelling throughout the manuscript.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MPipeMoE's speedup and memory claims rest on external baselines and measured runtime behavior, not on inputs that define the outputs.

full rationale

The central empirical claims are measured comparisons against FastMoE and FasterMoE (Figures 8 and 9), so the reported 2.8x speedup and up-to-47% memory reduction are external benchmark results rather than derived outputs. The performance model in Section III.E (Equation 10) is built from microbenchmarked slowdown factors and hardware speeds (alpha, beta) and is then used to choose among strategies S1-S4; Section V.G validates the chosen strategy against all alternatives, so the selection is not forced by construction. The theoretical memory-saving ratio in Section III.D (Equation 6) is an accounting identity for buffer reuse, but Figure 10 compares it with measured memory footprint, and the headline memory reduction is measured against FasterMoE, not taken from the formula. Algorithm 1's monotonicity assumption is explicitly labeled an 'intuitive hypothesis' in Section III.C and is tested in Figure 12, rather than being imported as an external theorem. The only self-citation of note, reference [23] by co-author Xiaobo Zhou, is cited for general batching of communication and computation; it is not load-bearing and does not supply any uniqueness theorem or forbidden alternative. The evaluation is limited to isolated MoE layers with a dummy random-token dataset, which is a generality concern about the abstract's 'in training large models' phrasing, but this is a correctness/scope issue, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on measured hardware constants (alpha, beta, mu, eta) and the monotonicity hypothesis for the granularity search. No numbers were fitted to the target speedup or memory reduction; the performance model is a heuristic based on microbenchmark measurements. No new physical entities are introduced.

assumptions (4)
  • ad hoc to paper The optimal pipeline granularity n is monotonically non-decreasing in batch size B.
    The adaptive search relies on this hypothesis to prune the search space. If false, the algorithm may return suboptimal n. Stated in Section III.C.
  • domain assumption Computation speed is unaffected by concurrent communication and memory streams, so sigma = 1.
    The performance model in Equation 10 sets sigma to 1 based on microbenchmarks, ignoring possible slowdown of GeMM under stream interference.
  • domain assumption Microbenchmark interference factors mu and eta measured in the cluster generalize to real training workloads.
    Used to select memory reusing strategies in Equation 10. If these factors differ under real load, strategy selection may be suboptimal.
  • domain assumption Overwriting activations and restoring them via offload or recompute yields gradients identical to storing them.
    Memory reusing relies on this equivalence, a standard assumption in activation checkpointing, but it is not explicitly proven for MoE's All-to-All tensors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism." pith.science (2026). https://pith.science/paper/AZ4FQGD5

@misc{pith2026250622175,
  author       = {Pith},
  title        = {Pith review of: MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AZ4FQGD5}},
  note         = {Machine review of arXiv:2506.22175}
}
read the original abstract

Recently, Mixture-of-Experts (MoE) has become one of the most popular techniques to scale pre-trained models to extraordinarily large sizes. Dynamic activation of experts allows for conditional computation, increasing the number of parameters of neural networks, which is critical for absorbing the vast amounts of knowledge available in many deep learning areas. However, despite the existing system and algorithm optimizations, there are significant challenges to be tackled when it comes to the inefficiencies of communication and memory consumption. In this paper, we present the design and implementation of MPipeMoE, a high-performance library that accelerates MoE training with adaptive and memory-efficient pipeline parallelism. Inspired by that the MoE training procedure can be divided into multiple independent sub-stages, we design adaptive pipeline parallelism with an online algorithm to configure the granularity of the pipelining. Further, we analyze the memory footprint breakdown of MoE training and identify that activations and temporary buffers are the primary contributors to the overall memory footprint. Toward memory efficiency, we propose memory reusing strategies to reduce memory requirements by eliminating memory redundancies, and develop an adaptive selection component to determine the optimal strategy that considers both hardware capacities and model characteristics at runtime. We implement MPipeMoE upon PyTorch and evaluate it with common MoE models in a physical cluster consisting of 8 NVIDIA DGX A100 servers. Compared with the state-of-art approach, MPipeMoE achieves up to 2.8x speedup and reduces memory footprint by up to 47% in training large models.

Figures

Figures reproduced from arXiv: 2506.22175 by the authors.

Figure 1
Figure 1. The illustration of expert parallelism of MoE and its data flow. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Breakdown of memory footprint ratio within model states, activations, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The interference between different operations. The values in the grid [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The illustration of GPipe and micro-batch pipeline parallelism in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Comparison between FasterMoE and our methods. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: The illustration of memory reusing. The top figure demonstrates [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 8
Figure 8. Figure 8: The speedup of different methods in MoE training with the same model [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: The memory footprint reduction by MPipeMoE. The bars and the [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: The MPipeMoE achieved memory reduction ratios compared to [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 11
Figure 11. Figure 11: Overall performance breakdown of MPipeMoE on GPT-XL model. [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 23 canonical work pages

  1. [1]

    On the optimization of deep networks: Implicit acceleration by overparameterization,

    S. Arora, N. Cohen, and E. Hazan, “On the optimization of deep networks: Implicit acceleration by overparameterization,” in Proc. of ICML, 2018

    work page 2018

  2. [2]

    Exploring the limits of weakly supervised pretraining,

    D. Mahajan, R. Girshick, V . Ramanathan, K. He, M. Paluri, Y . Li, A. Bharambe, and L. Van Der Maaten, “Exploring the limits of weakly supervised pretraining,” in Proc. of ECCV , 2018, pp. 181–196

    work page 2018

  3. [3]

    Antman: Dynamic scaling on gpu clusters for deep learning,

    W. Xiao, S. Ren, Y . Li, Y . Zhang, P. Hou, Z. Li, Y . Feng, W. Lin, and Y . Jia, “Antman: Dynamic scaling on gpu clusters for deep learning,” in Proc. of USENIX OSDI , 2020, pp. 533–548

    work page 2020

  4. [4]

    Whale: Efficient giant model training over heterogeneous gpus,

    X. Jia, L. Jiang, A. Wang, W. Xiao, Z. Shi, J. Zhang, X. Li, L. Chen, Y . Li, Z. Zheng et al. , “Whale: Efficient giant model training over heterogeneous gpus,” in USENIX Annual Technical Conference , 2022, pp. 673–688

    work page 2022

  5. [5]

    Axonn: An asynchronous, message-driven parallel framework for extreme-scale deep learning,

    S. Singh and A. Bhatele, “Axonn: An asynchronous, message-driven parallel framework for extreme-scale deep learning,” in Proc. of IEEE IPDPS, 2022, pp. 606–616

    work page 2022

  6. [6]

    An efficient and non-intrusive gpu schedul- ing framework for deep learning training systems,

    S. Wang, O. J. Gonzalez, X. Zhou, T. Williams, B. D. Friedman, M. Havemann, and T. Woo, “An efficient and non-intrusive gpu schedul- ing framework for deep learning training systems,” in Proc. IEEE/ACM SC, 2020

    work page 2020

  7. [7]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of NAACL-HLT, 2019, pp. 4171–4186

    work page 2019

  8. [8]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019

    arXiv 1907

Show all 35 references
  1. [9]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, P. J. Liuet al., “Exploring the limits of transfer learning with a unified text-to-text transformer.” J. Mach. Learn. Res. , vol. 21, no. 140, pp. 1–67, 2020

    2020

  2. [10]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  3. [11]

    Bliss: Robust sequence-to-sequence learning via self-supervised input repre- sentation,

    Z. Zhang, L. Ding, D. Cheng, X. Liu, M. Zhang, and D. Tao, “Bliss: Robust sequence-to-sequence learning via self-supervised input repre- sentation,” arXiv preprint arXiv:2204.07837 , 2022

    2022 arXiv

  4. [12]

    E2s2: Encoding- enhanced sequence-to-sequence pretraining for language understanding and generation,

    Q. Zhong, L. Ding, J. Liu, B. Du, and D. Tao, “E2s2: Encoding- enhanced sequence-to-sequence pretraining for language understanding and generation,” arXiv preprint arXiv:2205.14912 , 2022

    2022 arXiv

  5. [13]

    Unsu- pervised cross-lingual representation learning at scale,

    A. Conneau, K. Khandelwal, N. Goyal, V . Chaudhary, G. Wenzek, F. Guzm´an, E. Grave, M. Ott, L. Zettlemoyer, and V . Stoyanov, “Unsu- pervised cross-lingual representation learning at scale,” in Proc. of ACL, 2020

    2020

  6. [14]

    Toward efficient language model pretraining and downstream adaptation via self-evolution: A case study on superglue,

    Q. Zhong, L. Ding, Y . Zhan, Y . Qiao, Y . Wen, L. Shen, J. Liu, B. Yu, B. Du, Y . Chenet al., “Toward efficient language model pretraining and downstream adaptation via self-evolution: A case study on superglue,” arXiv preprint arXiv:2212.01853 , 2022

    2022 arXiv

  7. [15]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv preprint arXiv:1701.06538 , 2017

    2017 arXiv

  8. [16]

    Gshard: Scaling giant models with conditional computation and automatic sharding,

    D. Lepikhin, H. Lee, Y . Xu, D. Chen, O. Firat, Y . Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” arXiv preprint arXiv:2006.16668, 2020

    2006 arXiv

  9. [17]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” 2021

    2021

  10. [18]

    Cherry hypothesis: Identifying the cherry on the cake for dynamic networks,

    S. He, L. Ding, D. Dong, B. Liu, F. Yu, and D. Tao, “Cherry hypothesis: Identifying the cherry on the cake for dynamic networks,” arXiv preprint arXiv:2211.05528, 2022

    2022 arXiv

  11. [19]

    Base layers: Simplifying training of large, sparse models,

    M. Lewis, S. Bhosale, T. Dettmers, N. Goyal, and L. Zettlemoyer, “Base layers: Simplifying training of large, sparse models,” in Proc. of ICML. PMLR, 2021, pp. 6265–6274

    2021

  12. [20]

    Gating dropout: Communication-efficient regularization for sparsely activated transform- ers,

    R. Liu, Y . J. Kim, A. Muzio, and H. Hassan, “Gating dropout: Communication-efficient regularization for sparsely activated transform- ers,” in Proc. of ICML . PMLR, 2022, pp. 13 782–13 792

    2022

  13. [21]

    Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale,

    S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y . Aminabadi, A. A. Awan, J. Rasley, and Y . He, “Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale,” in Proc. of ICML, vol. 162, 2022, pp. 18 332–18 346

    2022

  14. [22]

    Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models,

    J. He, J. Zhai, T. Antunes, H. Wang, F. Luo, S. Shi, and Q. Li, “Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models,” in Proc. of ACM PPoPP , 2022, pp. 120–134

    2022

  15. [23]

    Scalable distributed dl training: Batching communication and computation,

    S. Wang, A. Pi, and X. Zhou, “Scalable distributed dl training: Batching communication and computation,” in Proceedings of the AAAI Confer- ence on Artificial Intelligence , 2019

    2019

  16. [24]

    Zero: Memory optimizations toward training trillion parameter models,

    S. Rajbhandari, J. Rasley, O. Ruwase, and Y . He, “Zero: Memory optimizations toward training trillion parameter models,” in Proc. of IEEE/ACM SC, 2020, pp. 1–16

    2020

  17. [25]

    Scalable and effi- cient moe training for multitask multilingual models,

    Y . J. Kim, A. A. Awan, A. Muzio, A. F. C. Salinas, L. Lu, A. Hendy, S. Rajbhandari, Y . He, and H. H. Awadalla, “Scalable and effi- cient moe training for multitask multilingual models,” arXiv preprint arXiv:2109.10465, 2021

    2021 arXiv

  18. [26]

    Training deep nets with sublinear memory cost,

    T. Chen, B. Xu, C. Zhang, and C. Guestrin, “Training deep nets with sublinear memory cost,” arXiv preprint arXiv:1604.06174 , 2016

    2016 arXiv

  19. [27]

    vdnn: Virtualized deep neural networks for scalable, memory-efficient neural network design,

    M. Rhu, N. Gimelshein, J. Clemons, A. Zulfiqar, and S. W. Keckler, “vdnn: Virtualized deep neural networks for scalable, memory-efficient neural network design,” in Proc. of IEEE MICRO , 2016, pp. 1–13

    2016

  20. [28]

    Buddy compression: Enabling larger memory for deep learning and hpc workloads on gpus,

    E. Choukse, M. B. Sullivan, M. O’Connor, M. Erez, J. Pool, D. Nellans, and S. W. Keckler, “Buddy compression: Enabling larger memory for deep learning and hpc workloads on gpus,” in Proc. of ACM/IEEE ISCA, 2020, pp. 926–939

    2020

  21. [29]

    Efficient large-scale language model training on gpu clusters using megatron-lm,

    D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V . Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro et al. , “Efficient large-scale language model training on gpu clusters using megatron-lm,” in Proc. IEEE/ACM SC, 2021, pp. 1–15

    2021

  22. [30]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. of ICLR , 2015

    2015

  23. [31]

    Gpipe: Efficient training of giant neu- ral networks using pipeline parallelism,

    Y . Huang, Y . Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V . Le, Y . Wuet al., “Gpipe: Efficient training of giant neu- ral networks using pipeline parallelism,” Advances in neural information processing systems, vol. 32, 2019

    2019

  24. [32]

    Tutel: Adaptive mixture-of-experts at scale,

    C. Hwang, W. Cui, Y . Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ramet al., “Tutel: Adaptive mixture-of-experts at scale,”arXiv preprint arXiv:2206.03382, 2022

    2022 arXiv

  25. [33]

    Mesh-tensorflow: Deep learning for supercomputers,

    N. Shazeer, Y . Cheng, N. Parmar, D. Tran, A. Vaswani, P. Koanantakool, P. Hawkins, H. Lee, M. Hong, C. Young et al. , “Mesh-tensorflow: Deep learning for supercomputers,” Advances in neural information processing systems, vol. 31, 2018

    2018

  26. [34]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catan- zaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053 , 2019

    1909 arXiv

  27. [35]

    Pipedream: Fast and efficient pipeline parallel dnn training,

    A. Harlap, D. Narayanan, A. Phanishayee, V . Seshadri, N. Devanur, G. Ganger, and P. Gibbons, “Pipedream: Fast and efficient pipeline parallel dnn training,” arXiv preprint arXiv:1806.03377 , 2018

    2018 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.