Pith. sign in

REVIEW 4 major objections 5 minor 69 references

Zorse: Optimizing LLM Training Efficiency on Heterogeneous GPU Clusters

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Combining pipeline and data parallelism with aggressive offloading lets mixed-GPU clusters train LLMs up to 3x faster.

desk verdict Zorse combines interleaved pipeline parallelism with ZeRO-2 and offloading in a way that is genuinely new for heterogeneous training, but the headline speedups rest on baseline comparisons that are not yet fully documented. read the letter →

arxiv 2507.10392 v1 pith:LOHGU5AX submitted 2025-07-14 cs.DC

Runsheng Benson Guo , Utkarsh Anand , Khuzaima Daudjee , Rathijit Sen This is my paper · ORCID
classification cs.DC
keywords heterogeneousGPUclusterspipelineparallelismdataZeRO-2interleavedpipeliningautomaticplannerCPUoffloadingLLMtraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Zorse claims that efficient LLM training on heterogeneous GPU clusters requires a specific combination: pipeline parallelism integrated with ZeRO-2 data parallelism, with each pipeline stage split into ministages whose parameters and activations are offloaded to CPU memory, plus the flexibility to give different pipeline stages different numbers and types of GPUs. The paper argues this is the first system to unify all these capabilities with an automatic planner. On three clusters spanning 20 to 128 GPUs and models up to 65B parameters, Zorse reports up to 3x higher training throughput than existing state-of-the-art systems, with memory use approaching that of ZeRO-3 while avoiding its communication overhead. If correct, this would let organizations pool whatever GPUs they have and still train large models efficiently.

What carries the argument

The central mechanism is interleaved ministages under a GPipe-style schedule: each pipeline stage is split into multiple ministages, microbatches are processed through one ministage fully before moving on, and parameters of all other ministages plus layer-boundary activations are offloaded to CPU. This keeps only two ministages' parameters in GPU memory at once, drops memory use by up to 40% versus standard PP+ZeRO-2 with minimal throughput loss, and enables interleaved optimizer updates that free gradients early and overlap gradient averaging with computation. Supporting it are heterogeneous pipeline parallelism (stages with different GPU counts and types, connected by a load-balancing many-to-many communication plan) and a planner that partitions the cluster with an approximate min-k cut and selects configuration via lightweight latency and memory models.

What would settle it

Run Llama 65B on Cluster A with TorchTitan-Het after thorough hyperparameter tuning to avoid the reported out-of-memory crash and compare throughput; if a tuned baseline closes the gap or beats Zorse, the central speedup claim fails. A simpler check: reproduce Zorse's Table 5 with HexiScale using the exact configuration from its original paper rather than the one chosen here.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the memory-efficiency of ZeRO-3 and the communication-efficiency of ZeRO-2 need not be mutually exclusive when combined with pipeline parallelism. Zorse uses a GPipe-style interleaved schedule in which each pipeline stage is subdivided into ministages; only the current and next ministage's parameters are materialized, and all others are offloaded to CPU memory, while activations are also offloaded and prefetched. Because microbatches are processed sequentially per ministage, parameter gathering happens once per layer rather than once per microbatch, preserving ZeRO-2's low communication while cutting memory to near ZeRO-3 levels. Zorse also allows pipeline stages to be asymmetric in GPU count and type, with an optimized communication plan that redistributes microbatches across stages and balances compute within stages. A two-phase planner uses approximate min-k cut clustering and latency/memory models to choose the configuration automatically. The paper reports consistently higher throughput than three strong baselines—TorchTitan-Het, HexiScale, and Cephalo—across clusters that mix low-, mid-, and high-end GPUs.

Load-bearing premise

The reported speedups assume the comparison systems were configured to their best achievable performance, but the paper does not describe a systematic tuning process for any baseline, and TorchTitan-Het is an author-adapted variant.

Editorial extensions

If this is right

  • Organizations with leftover or mixed-generation GPUs could train large language models nearly as efficiently as on homogeneous clusters, since Zorse reports hardware utilization comparable to homogeneous training on each GPU type.
  • Memory-constrained GPUs such as T4s can be included in training runs for 7B–33B models without running out of memory, because offloading and asymmetric stage sizes adapt per device.
  • Adding heterogeneous GPUs to a training cluster increases throughput while maintaining or improving utilization, suggesting the approach scales by pooling rather than by replacing hardware.
  • The planner's roughly three-minute planning time makes reconfiguration cheap enough to repeat periodically as hardware performance drifts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If confirmed across more clusters, the result suggests that ZeRO-2 plus aggressive offloading could replace ZeRO-3 for most heterogeneous training, removing the need for costly parameter gathering.
  • The cross-stage communication plan for many-to-many microbatch redistribution could be reused by other pipeline-parallel systems that allow asymmetric stages, opening a path to load balancing without uniform stage sizes.
  • A testable extension would be to apply Zorse's offloading and interleaving to clusters spread over more than two regions or datacenters; the planner's min-k cut formulation may need modification for wide-area links with very high latency.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents Zorse, a system for training LLMs on heterogeneous GPU clusters. Zorse integrates pipeline parallelism (PP) with ZeRO-2 data parallelism using a GPipe-style interleaved schedule, offloads parameters and activations to CPU memory, supports heterogeneous pipeline stages with variable GPU counts and mixed GPU types within a stage, and includes a two-phase planner that partitions the cluster and selects batch sizes, ministage counts, and layer assignments. The evaluation compares Zorse against TorchTitan-Het, HexiScale, and Cephalo on three clusters of up to 128 GPUs and models up to 65B parameters, reporting training throughput in TFlops and HFU. The paper claims that Zorse achieves up to 3x higher training throughput than state-of-the-art heterogeneous training systems, with Table 5 showing speedups up to about 4x in one configuration.

Significance. If the reported results hold, Zorse addresses a real and growing problem: efficiently training LLMs on clusters assembled from GPUs of different generations, memory capacities, and network interconnects. The system combines several individually known techniques (interleaved pipelining, CPU offloading, heterogeneous PP, automatic planning) in a way that goes beyond existing systems, and the paper provides ablations isolating the contribution of activation offloading, interleaved pipelining, and heterogeneous PP. The headline throughput numbers are measured rather than derived from a fitted model, which avoids circularity in the reported speedups. The main value of the paper is therefore empirical: it demonstrates that a carefully engineered combination of PP and ZeRO-2 can outperform existing heterogeneous-training systems in several realistic cloud settings. That claim is plausible but, in its current form, is not yet fully verifiable because the baseline configurations and experimental variance are not documented to the level required by the size of the speedups.

major comments (4)
  1. [§6.1, Table 5] The baseline comparison is not documented to the level needed to support the headline speedups. TorchTitan-Het is an author-adapted variant of TorchTitan, but Section 6.1 does not report the search grid, the number of configurations tried, or the per-cell settings for any baseline; no tuning procedure is described for HexiScale or Cephalo, and no run-to-run variance is reported. Because Table 5's largest gaps sit on exactly these cells (e.g., Cluster B Llama 33B: Zorse 2658.29 vs HexiScale 657.16 TFlops, and Cluster A Llama 33B: Zorse 5281.64 vs TorchTitan-Het 944.47 TFlops), the central 'up to 3x/4x' claim cannot currently be verified. Please provide the full searched configuration space, the selection rule, the per-cell chosen configurations, and ideally multiple runs with error bars for the key cells.
  2. [§6.2] The Cluster B discussion states that 'FlashFlex divides the LLama 33B model into over 16 stages,' but FlashFlex is not defined in the Section 6.1 baseline list, in the references, or anywhere else in the paper. This makes it impossible to determine which system produced the corresponding Table 5 row. Please identify the intended system or remove the reference and re-derive the discussion.
  3. [Table 2, §5.1] The memory and communication comparison is ambiguous about ZeRO-2 semantics. Table 2 lists PP+ZeRO-2 as having 0 sharded parameters while reporting 2×L AllGathers per iteration; standard ZeRO-2 shards optimizer states and gradients but keeps parameters replicated, so the AllGather pattern is not explained. Zorse's row shows the same 2×L AllGathers, yet Section 5.1 says Zorse 're-shard[s] and offload[s] ministage parameters to CPU post-forward pass,' which is a different parameter-lifecycle model. Please clarify what tensors are being gathered, when, and why the counts are equal to PP+ZeRO-2; this is load-bearing for the paper's central memory/communication-efficiency argument.
  4. [§6.1, §6.2] The throughput numbers are single-point measurements with no indication of the number of runs or variance. The paper itself motivates the problem through cloud variability and discusses hardware performance variability in Section 8, yet the evaluation reports TFlops without error bars. Given the large reported speedups, a single run could be unrepresentative. Please report at least three repetitions for the key cells of Table 5, or justify why variance is negligible, and state the hardware and software versions used for each baseline.
minor comments (5)
  1. [§6.1] The sentence 'More details on the training configurations' is a fragment; it appears to introduce the per-cluster descriptions but is not followed by a colon, table, or list.
  2. [§4.3.2] The stated complexity of exact min-k cut, O(N k^2), is internally inconsistent: for k=2 it gives O(N), while the next sentence assumes a min 2-cut costs O(N^3). Please correct the complexity statement or the citation.
  3. [§4.3.4, Eq. (1)] The latency model does not show the dependence on the number of microbatches or the pipeline fill/drain phase; the standard GPipe latency formula includes a term proportional to (number of microbatches + number of stages - 1). Please define all terms and explain why L_forward and L_backward alone are sufficient.
  4. [Abstract, §6.2, Conclusion] The abstract and conclusion state 'up to 3x' speedup, while Section 6.2 reports and Table 5 shows a 4.0x speedup (Cluster B Llama 33B). Please make the headline number consistent throughout.
  5. [Figure 1] The caption does not describe the model, batch size, or communication settings used to compute HFU, which makes the figure difficult to interpret.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the headline speedups are measured results from Table 5, and the only self-citation (Cephalo) is a baseline comparison that is not load-bearing. Baseline tuning and the undefined 'FlashFlex' mention are verifiability concerns, not circularity.

full rationale

The paper's central claim is an empirical performance comparison: Table 5 reports measured training throughput (TFlops) and utilization (HFU) for Zorse and three baselines. These numbers are not produced by the planner's latency model; Equations (1) and (2) are used only to select a configuration from the search space, and the reported throughput is measured after that selection. There is no fitted parameter renamed as a prediction, and no equation defines a claimed result in terms of its own inputs. The only self-citation is Cephalo [9], a prior system by three of the four authors, used as one of the comparison baselines. The paper's statement that Cephalo 'cannot efficiently utilize network resources in highly heterogeneous networks' is a description of the baseline's architecture, not a load-bearing premise that forces Zorse's design or its measured speedups. No uniqueness theorem from the authors' own prior work is invoked, and no ansatz is smuggled in via self-citation. The main risks are experimental and reproducibility-related rather than circular: Section 6.1 does not document tuning for HexiScale or Cephalo, TorchTitan-Het is an author-adapted baseline, and Section 6.2 refers to an undefined 'FlashFlex' system in the Cluster B discussion. These concerns affect how much weight the head-to-head speedups can carry, but they do not make the derivation circular. Therefore the score is 2, reflecting one minor, non-load-bearing self-citation while the central derivation remains self-contained.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claims rest on measured system design assumptions rather than a mathematical derivation. Free parameters are limited to profiled runtime models used by the planner. The main assumptions concern the unsuitability of tensor parallelism, the communication bottleneck of ZeRO-3 with pipeline parallelism, and the representativeness of the baselines.

free parameters (2)
  • Per-GPU layer runtime linear model coefficients = not reported
    Profiled layer runtimes at small batch sizes are fit to a linear model to predict runtimes at larger batch sizes (Section 4.3.1); the planner's configuration choice depends on this model.
  • Planner latency model terms (L_forward, L_backward) = derived from profile statistics
    Equation (1) combines profiled compute and communication latencies; its accuracy is not validated against exhaustive search, so planning quality relies on these fitted inputs.
assumptions (5)
  • domain assumption GPU layer runtime scales linearly with batch size.
    Section 4.3.1 states layer runtimes on each GPU are profiled for small batch sizes and fit with a linear model to predict unseen batch sizes; if the linearity assumption is wrong, planner choices may be suboptimal.
  • standard math Min-k cut approximation via SPLIT provides a 2 - 2/k approximation to optimal.
    Section 4.3.2 uses the Saran-Vazirani SPLIT algorithm; the guarantee is standard, but the planner only needs a heuristic, not exact optimality.
  • domain assumption Tensor parallelism is unsuitable in heterogeneous clusters due to communication overhead.
    Takeaway #1 (Section 2.1) motivates excluding TP from Zorse; this is an empirical and architectural assumption based on Figure 1 measurements.
  • domain assumption ZeRO-3 with pipeline parallelism creates a communication bottleneck because parameter gathering scales with microbatches.
    Section 2.3 argues this and motivates Zorse's ZeRO-2 design; the claim is supported by Figure 3 measurements.
  • domain assumption GLOO P2P non-blocking communication avoids deadlocks that NCCL blocking P2P would cause in cyclic heterogeneous pipeline parallelism.
    Section 5.2; the deadlock claim is not empirically demonstrated, only cited from NCCL documentation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zorse: Optimizing LLM Training Efficiency on Heterogeneous GPU Clusters." pith.science (2026). https://pith.science/paper/LOHGU5AX

@misc{pith2026250710392,
  author       = {Pith},
  title        = {Pith review of: Zorse: Optimizing LLM Training Efficiency on Heterogeneous GPU Clusters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LOHGU5AX}},
  note         = {Machine review of arXiv:2507.10392}
}
read the original abstract

Large language models (LLMs) require vast amounts of GPU compute to train, but limited availability and high costs of GPUs make homogeneous clusters impractical for many organizations. Instead, assembling heterogeneous clusters by pooling together GPUs of different generations allows them to achieve higher aggregate compute and make use of all available GPUs. However, training on heterogeneous clusters presents several challenges, including load balancing across GPUs, optimizing memory usage to accommodate varying memory capacities, and ensuring communication-efficient training over diverse network interconnects potentially spanning multiple datacenters. In this paper, we make the case that efficient training on heterogeneous clusters requires (1) the integration of pipeline parallelism and data parallelism in a manner that is both communication- and memory-efficient, and (2) a more adaptable configuration of pipeline and data parallelism, which includes the capability to flexibly partition GPUs into asymmetric pipeline parallel stages and to incorporate heterogeneous GPUs within the same data parallelism group. We propose Zorse, the first system to unify all these capabilities while incorporating a planner that automatically configures training strategies for a given workload. Our evaluation shows that Zorse significantly outperforms state-of-the-art systems in heterogeneous training scenarios.

Figures

Figures reproduced from arXiv: 2507.10392 by the authors.

Figure 1
Figure 1. Hardware Flops Utilization (HFU [3]) of tensor parallelism (TP) vs data parallelism (DP) with ZeRO-3 on 8 GPU AWS VMs. 3x higher training throughput on three representative clusters for models scaling up to 65 billion parameters. 2 Challenges in Heterogeneous Training In this section, we provide insights and establish key chal￾lenges to overcome for achieving efficient training on het￾erogeneous GPU clusters. 2.1 Te… view at source ↗
Figure 3
Figure 3. Comparison of pipeline parallelism (PP) with ZeRO-2 vs ZeRO-3 across different Llama[49, 58] model sizes on 8 V100s + 8 T4s. OOM indicates Out-of-Memory. 2.3 PP + DP in Heterogeneous Clusters ZeRO-3 [38, 53] (or FSDP [62] in PyTorch) is a popular vari￾ant of DP designed to avoid redundant storage of training states across GPUs. While this method significantly reduces the memory requirement for storing training state… view at source ↗
Figure 2
Figure 2. Unidirectional bandwidth between GPUs on AWS. 2.2 Data Parallelism in Heterogeneous Networks In homogeneous clusters with high-bandwidth intercon￾nects, DP has been shown to scale efficiently to large clusters of over 1000 GPUs [38, 53]. However, the scalability of DP is limited in heterogeneous clusters due to substantial network variability caused by differences in (1) intra-node networking, (2) inter-node network… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Interleaved pipelining in Zorse. The diagram shows a training iteration for a model with 20 layers (𝐿𝑖 ) and 3 microbatches. There are 3 GPU groups, each with 4 ministages. Due to different computational speeds, groups have different numbers of layers per ministage. Ea…
Figure 5
Figure 5. Figure 5: Interleaved optimizer updates in Zorse vs. tradi￾tional pipeline parallelism (4 ministages, 3 microbatches). Interleaved optimizer updates free gradient memory earlier and better overlap with computation. Cluster GPU GPU GPU GPU GPU Region 1 Region 2 Bandwidth High Med…
Figure 6
Figure 6. Figure 6: Heterogeneous pipeline parallelism in Zorse com￾pared to traditional pipeline parallelism. for only the current microbatch being computed and for the next microbatch being prefetched. In Section 5.4, we describe how Zorse efficiently overlaps offloading (and loading) o…
Figure 7
Figure 7. Figure 7: Architecture of Zorse’s planner: ○1 profile workload and cluster, ○2 partition cluster into GPU groups, ○3 partition model and data across GPU groups, and ○4 select the best plan considered. each GPU group. The planner considers the best way to partition the cluster in…
Figure 8
Figure 8. Figure 8: PFlops (1k TFlops) and HFU scaling as heteroge￾neous GPUs are added to the clusters. the training performance of the largest model that can be trained on the slowest GPUs. We then incrementally add faster GPUs to the training group, gradually increasing the total numbe…
Figure 9
Figure 9. Figure 9: GPU Utilization on heterogeneous clusters vs homogeneous subgroups of GPUs within the clusters. 6.4 Comparison to Homogeneous Training We assess the training efficiency of Zorse on heterogeneous versus homogeneous clusters in [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: TFlops and memory utilization for varying min￾istages per GPU. Values are normalized to 1 microstage. 6.5 Interleaved Pipelining Analysis In Section 2.3, we demonstrated how PP + ZeRO-3 is memory efficient but can lead to very low throughput due to high communication …
Figure 12
Figure 12. Figure 12: Planner runtime breakdown per cluster. 6.6 Ablation Study We conduct an ablation study to evaluate the impact of Zorse’s key components on training throughput and memory utilization. Starting with a baseline PP + ZeRO-2 implemen￾tation in TorchTitan, we incrementally …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 38 canonical work pages

  1. [1]

    AMD. 2025. PyTorch on ROCm. https://rocm.docs.amd.com/projects/ install-on-linux/en/latest/install/3rd-party/pytorch-install.html Ac- cessed: 2025-02-24

    work page 2025

  2. [2]

    Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ram- jee, and Nipun Kwatra. 2022. Varuna: scalable, low-cost training of massive deep learning models. In Proceedings of the Seventeenth Euro- pean Conference on Computer Systems (Rennes, France) (EuroSys ’22). Association for Computing Machinery, New York, NY, USA, 472–487. https://doi.org/10....

    arXiv 2022

  3. [3]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bra...

    work page 2023

  4. [4]

    Yushi Ding, Noam Botzer, and Tim Weninger. 2021. HetSeq: Distributed GPU Training on Heterogeneous Infrastructure. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 35. AAAI Press, 15432– 15438. https://doi.org/10.1609/aaai.v35i17.17813

    work page doi:10.1609/aaai.v35i17.17813 2021

  5. [5]

    Jiangfei Duan, Ziang Song, Xupeng Miao, Xiaoli Xi, Dahua Lin, Harry Xu, Minjia Zhang, and Zhihao Jia. 2024. Parcae: proactive, liveput- optimized DNN training on preemptible instances. InProceedings of the 21st USENIX Symposium on Networked Systems Design and Implemen- tation (Santa Clara, CA, USA) (NSDI’24). USENIX Association, USA, Article 62, 19 pages

    work page 2024

  6. [8]

    Hochbaum

    Oded Goldschmidt and Dorit S. Hochbaum. 1988. A Polynomial Al- gorithm for the k-Cut Problem for Fixed k. In Proceedings of the 29th Annual Symposium on Foundations of Computer Science (FOCS) . IEEE Computer Society, 444–451

    work page 1988

  7. [9]

    Runsheng Benson Guo, Utkarsh Anand, Arthur Chen, and Khuzaima Daudjee. 2024. Cephalo: Harnessing Heterogeneous GPU Clusters for Training Transformer Models. arXiv:2411.01075 [cs.DC] https: //arxiv.org/abs/2411.01075

    arXiv 2024

  8. [10]

    Ronghang Hu, Vaibhav Singh, Jack Cao, Milad Mohammadi, Yeounoh Chung, Shauheen Zahirazami, and Ross Girshick. 2022. Scaling Py- Torch Models on Cloud TPUs with FSDP. https://pytorch.org/blog/ scaling-pytorch-models-on-cloud-tpus-with-fsdp/ Accessed: 2025- 02-24

    work page 2022

Show all 69 references
  1. [11]

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems 32 (2019)

    2019

  2. [12]

    Sunyeol Hwang, Eungyeong Lee, Hongseok Oh, and Youngmin Yi

  3. [13]

    Alexandru Iosup, Nezih Yigitbasi, and Dick Epema. 2011. On the Performance Variability of Production Cloud Services. In 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Com- puting. 104–113. https://doi.org/10.1109/CCGrid.2011.22

    2011 doi

  4. [14]

    Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. 2020. Check- mate: Breaking the Memory Wall with Optimal Tensor Rematerializa- tion. In Proceedings of Machine Learning and Systems 2020 . 497–511

    2020

  5. [15]

    Xianyan Jia, Le Jiang, Ang Wang, Wencong Xiao, Ziji Shi, Jie Zhang, Xinyuan Li, Langshi Chen, Yong Li, Zhen Zheng, Xiaoyong Liu, and Wei Lin. 2022. Whale: Efficient Giant Model Training over Heterogeneous GPUs. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). USENIX...

    2022

  6. [16]

    Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Guoliang He, Xupeng Miao, Ana Klimovic, Bin Cui, Binhang Yuan, and Eiko Yoneki. 2025. De- mystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs. arXiv:2502.00722 [cs.DC] https://arxiv.org/abs/2502.00722

    2025 arXiv

  7. [17]

    Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi...

    2024

  8. [18]

    Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Clifford Young, Xiang Zhou, Zongwei Zhou, and David A Patterson. 2023. TPU v4: An Optically Reconfigurable Supercomputer for Machine Learn...

    2023

  9. [19]

    Kyeonglok Kim, Hyeonsu Lee, Seungmin Oh, and Euiseong Seo. 2022. Scale-Train: A Scalable DNN Training Framework for a Heterogeneous GPU Cloud. IEEE Access 10 (2022), 68468–68481

    2022

  10. [20]

    Joel Lamy-Poirier. 2021. Layered gradient accumulation and modu- lar pipeline parallelism: fast and efficient training of large language models. arXiv preprint arXiv:2106.02679 (2021)

    2021 arXiv

  11. [21]

    Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. 2023. Sequence Parallelism: Long Sequence Training from Sys- tem Perspective. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Anna Rogers...

    2023 doi

  12. [22]

    Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, Sanket Purandare, Gokul Nadathur, and Stratos Idreos. 2024. TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training....

    2024 arXiv

  13. [23]

    Liang Luo, Peter West, Pratyush Patel, Arvind Krishnamurthy, and Luis Ceze. 2022. Srifty: Swift and Thrifty Distributed Neural Network Training on the Cloud. InProceedings of Machine Learning and Systems (MLSys), Vol. 4. 833–847. https://proceedings.mlsys.org/paper_files/ pape...

    2022

  14. [24]

    MarketsandMarkets. 2023. Nvidia’s Dominance in the AI Chip Market. https://www.marketsandmarkets.com/blog/SE/nvidia- dominance-in-the-ai-chip-market

    2023

  15. [25]

    Xupeng Miao, Yining Shi, Zhi Yang, Bin Cui, and Zhihao Jia. 2023. Sdpipe: A semi-decentralized framework for heterogeneity-aware pipeline-parallel training. Proceedings of the VLDB Endowment 16, 9 (2023), 2354–2363

    2023

  16. [26]

    Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. 2022. Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism.Proc. VLDB Endow. 16, 3 (nov 2022), 470–479. https://doi.org/10.14778/3570690. 3570697

    2022 doi

  17. [27]

    Sergio Moreno-Alvarez, Juan M Haut, Mercedes E Paoletti, Juan A Rico-Gallego, Juan C Diaz-Martin, and Javier Plaza. 2020. Training deep neural networks: a static load balancing approach. The Journal of Supercomputing 76 (2020), 9739–9754

    2020

  18. [28]

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. 2019. PipeDream: Generalized pipeline parallelism for DNN training. In Proceedings of the 27th ACM Symposium on Operating Systems Princip...

    2019

  19. [29]

    Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. 2021. Memory-efficient pipeline-parallel dnn training. In International Conference on Machine Learning . PMLR, 7937–7947

    2021

  20. [30]

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGres- ley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021. Efficient large-scale language model training on gpu clusters using megatron- lm. In ...

    2021

  21. [31]

    Chengyi Nie, Jessica Maghakian, and Zhenhua Liu. 2024. Cannikin: Optimal Adaptive Distributed DNN Training over Heterogeneous Clusters. In Proceedings of the 25th International Middleware Conference (Hong Kong, Hong Kong)(Middleware ’24). Association for Computing Machinery, N...

    2024

  22. [32]

    NVIDIA. 2024. NCCL: NVIDIA Collective Communications Library. https://developer.nvidia.com/nccl

    2024

  23. [33]

    NVIDIA Corporation. 2020. Point To Point Communication Functions. NVIDIA NCCL User Guide API documentation, version 2.26.2. https:// docs.nvidia.com/deeplearning/nccl/user-guide/docs/api/p2p.html Ac- cessed: 2025-04-17

    2020

  24. [34]

    2020.{HetPipe}: Enabling large{DNN} training on (whimpy) heterogeneous{GPU} clusters through integration of pipelined model parallelism and data parallelism

    Jay H Park, Gyeongchan Yun, M Yi Chang, Nguyen T Nguyen, Seung- min Lee, Jaesik Choi, Sam H Noh, and Young-ri Choi. 2020.{HetPipe}: Enabling large{DNN} training on (whimpy) heterogeneous{GPU} clusters through integration of pipelined model parallelism and data parallelism. In ...

    2020

  25. [35]

    Xuan Peng, Xuanhua Shi, Hulin Dai, Hai Jin, Weiliang Ma, Qian Xiong, Fan Yang, and Xuehai Qian. 2020. Capuchin: Tensor-based GPU Mem- ory Management for Deep Learning. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Lan- gua...

    2020

  26. [36]

    PyTorch. 2023. Training a 1 Trillion Parameter Model with PyTorch Fully Sharded Data Parallel on AWS. https://shorturl.at/6Y4LT. Ac- cessed: 2024-01-30

    2023

  27. [37]

    Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin. 2024. Zero bubble (almost) pipeline parallelism. In The Twelfth International Con- ference on Learning Representations

    2024

  28. [38]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He

  29. [39]

    Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021. ZeRO-infinity: breaking the GPU memory wall for extreme scale deep learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (S...

    2021

  30. [40]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He

  31. [41]

    Suhail Rehman and Majd F

    M. Suhail Rehman and Majd F. Sakr. 2010. Initial Findings for Pro- visioning Variation in Cloud Computing. In Proceedings of the 2010 IEEE Second International Conference on Cloud Computing Technology and Science (CLOUDCOM ’10) . IEEE Computer Society, USA, 473–479. https://do...

    2010 doi

  32. [42]

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021. ZeRO-Offload: Democratizing Billion-Scale Model Train- ing. In 2021 USENIX Annual Technical Conference (USENIX ATC 21) . USENIX Association, 551–5...

    2021

  33. [43]

    In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 3505–3506

  34. [44]

    Vazirani

    Huzur Saran and Vijay V. Vazirani. 1995. Finding k Cuts within Twice the Optimal. SIAM J. Comput. 24, 1 (1995), 101–108. https://doi.org/10.1137/S0097539792251730 arXiv:https://doi.org/10.1137/S0097539792251730

    1995 doi

  35. [45]

    Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al . 2018. Mesh-tensorflow: Deep learning for supercomputers.Advances in neural information processing systems 31 (2018)

    2018

  36. [46]

    Robbins and S

    H. Robbins and S. Monro. 1951. A stochastic approximation method. The Annals of Mathematical Statistics 22, 3 (1951), 400–407. 14

    1951

  37. [47]

    Mechthild Stoer and Frank Wagner. 1997. A Simple Min-Cut Algorithm. J. ACM 44, 4 (1997), 585–591. https://doi.org/10.1145/263867.263872

    1997

  38. [48]

    Foteini Strati, Paul Elvinger, Tolga Kerimoglu, and Ana Klimovic. 2024. ML Training with Cloud GPU Shortages: Is Cross-Region the Answer?. In Proceedings of the 4th Workshop on Machine Learning and Systems (Athens, Greece) (EuroMLSys ’24). Association for Computing Machin- ery...

    2024 doi

  39. [49]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. CoRR abs/1909.08053 (2019). http://arxiv.org/abs/1909.08053

    2019 arXiv

  40. [50]

    Taegeon Um, Byungsoo Oh, Minyoung Kang, Woo-Yeon Lee, Goeun Kim, Dongseob Kim, Youngtaek Kim, Mohd Muzzammil, and Myeong- jae Jeon. 2024. Metis: Fast Automatic Distributed Training on Het- erogeneous{GPUs}. In 2024 USENIX Annual Technical Conference (USENIX ATC 24). 563–578

    2024

  41. [51]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. At- tention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  42. [52]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  43. [53]

    Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Xiaoxia Wu, Connor Holmes, Zhewei Yao, Samyam Rajbhandari, Olatunji Ruwase, Feng Yang, Lei Yang, and Yuxiong He. 2024. ZeRO++: Extremely Efficient Collective Communication for Large Model Training. In ICLR

    2024

  44. [54]

    Yifu Wang, Horace He, Less Wright, Luca Wehrstedt, Tianyu Liu, and Wanchao Liang. 2024. Distributed w/ TorchTi- tan: Introducing Async Tensor Parallelism in PyTorch. https://discuss.pytorch.org/t/distributed-w-torchtitan-introducing- async-tensor-parallelism-in-pytorch/209487 ...

    2024

  45. [55]

    Marcel Wagenländer, Guo Li, Bo Zhao, Luo Mai, and Peter Pietzuch

  46. [56]

    In Proceedings of the ACM SIGOPS 30th Sym- posium on Operating Systems Principles (Austin, TX, USA) (SOSP ’24)

    Tenplex: Dynamic Parallelism for Deep Learning using Paralleliz- able Tensor Collections. In Proceedings of the ACM SIGOPS 30th Sym- posium on Operating Systems Principles (Austin, TX, USA) (SOSP ’24). Association for Computing Machinery, New York, NY, USA, 195–210. https://do...

  47. [57]

    Jinghui Zhang, Geng Niu, Qiangsheng Dai, Haorui Li, Zhihua Wu, Fang Dong, and Zhiang Wu. 2023. PipePar: Enabling fast DNN pipeline parallel training in heterogeneous GPU clusters. Neurocomput. 555, C (Oct. 2023), 12 pages. https://doi.org/10.1016/j.neucom.2023.126661

    2023

  48. [58]

    https://www.microsoft.com/en-us/research/publication/zero- extremely-efficient-collective-communication-for-large-model- training/

  49. [59]

    Shiwei Zhang, Lansong Diao, Chuan Wu, Zongyan Cao, Siyu Wang, and Wei Lin. 2024. HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis. In Proceedings of the European Conference on Computer Systems (EuroSys ’24) (Athens, Greece, April 22–25). AC...

    2024

  50. [60]

    Ran Yan, Youhe Jiang, Xiaonan Nie, Fangcheng Fu, Bin Cui, and Bin- hang Yuan. 2025. HexiScale: Accommodating Large Language Model Training over Heterogeneous Environment. arXiv:2409.01143 [cs.DC] https://arxiv.org/abs/2409.01143

    2025 arXiv

  51. [61]

    Fei Yang, Shuang Peng, Ning Sun, Fangyu Wang, Yuanyuan Wang, Fu Wu, Jiezhong Qiu, and Aimin Pan. 2024. Holmes: Towards Distributed Training Across Clusters with Heterogeneous NIC Environment. In Proceedings of the 53rd International Conference on Parallel Processing (Gotland, ...

    2024

  52. [62]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. 2023. Py- Torch FSDP: Experienc...

    2023

  53. [63]

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu

  54. [64]

    arXiv:2401.02385 [cs.CL]

    TinyLlama: An Open-Source Small Language Model. arXiv:2401.02385 [cs.CL]

    arXiv

  55. [66]

    WenZheng Zhang, Yang Hu, Jing Shi, and Xiaoying Bai. 2025. Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 22587–22595

    2025

  56. [67]

    Zhen Zhang, Shuai Zheng, Yida Wang, Justin Chiu, George Karypis, Trishul Chilimbi, Mu Li, and Xin Jin. 2022. MiCS: near-linear scaling for training gigantic model on public cloud. Proc. VLDB Endow. 16, 1 (Sept. 2022), 37–50. https://doi.org/10.14778/3561261.3561265

    2022

  57. [69]

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. 2022. Alpa: Automating inter-and{Intra-Operator} parallelism for distributed deep learning. In 16th USENIX Symposium on Operating Syste...

    2022

  58. [70]

    Zhanda Zhu, Christina Giannoula, Muralidhar Andoorveedu, Qidong Su, Karttikeya Mangalam, Bojian Zheng, and Gennady Pekhimenko

  59. [2020]

    In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis

    Zero: Memory optimizations toward training trillion param- eter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis . IEEE, 1–16

  60. [2024]

    In Pro- ceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing (Pisa, Italy) (HPDC ’24)

    FASOP: Fast yet Accurate Automated Search for Optimal Par- allelization of Transformers on Heterogeneous GPU Clusters. In Pro- ceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing (Pisa, Italy) (HPDC ’24) . Asso- ciation for Compu...

  61. [2025]

    In Proceedings of the Twentieth European Conference on Computer Systems

    Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization. In Proceedings of the Twentieth European Conference on Computer Systems . 1298–1316. 15

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.