Pith. sign in

REVIEW 4 major objections 5 minor 69 references

CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CrossPipe claims that cross-datacenter LLM training slowdown from network latency and bandwidth can be largely eliminated by generating pipeline schedules that treat communication as first-class, cutting training time by up to 33.6% under…

desk verdict CrossPipe is a solid scheduling paper whose headline speedup is measured inside an emulator built from the same model the scheduler optimizes — a real caveat, but not a fatal one. read the letter →

arxiv 2507.00217 v1 pith:NHIWZAZS submitted 2025-06-30 cs.DC

Tiancheng Chen , Ales Kubicek , Langwen Huang , Torsten Hoefler This is my paper · ORCID
classification cs.DC
keywords cross-datacentertrainingpipelineparallelismschedulingdataoverlaplatencyandbandwidthmodelingconstraintoptimizationgreedylargelanguagemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to make training large language models across geographically separated datacenters practical. Its central claim is that static pipeline schedules, designed for near-zero communication cost inside one datacenter, accumulate delay on every cross-datacenter message, and that reordering pipeline blocks with explicit latency and bandwidth in mind removes most of that waste. CrossPipe models each pipeline message as a first-class operation, solves the schedule with a constraint-optimization solver or a fast greedy algorithm, and decouples scheduling from communication orchestration. Measured on emulated cross-datacenter links, it reduces iteration time by up to 33.6% relative to traditional schedules at the same memory limit, and approaches the efficiency of a delay-free schedule when memory is relaxed. The paper also argues that, except at very high link bandwidth, pipeline parallelism across datacenters beats data parallelism across datacenters.

What carries the argument

The central object is the acyclic pipeline dependency graph with split backward blocks: each forward block F, input-gradient block D, and weight-gradient block W, together with communication operations as first-class vertices carrying a latency term and a bandwidth term. Block start times are computed as the maximum of the predecessor's completion on the same stage and the dependent block's completion plus communication delay, and this topological-order evaluation is the performance model. The constraint-optimization formulation adds start-time and ordering variables with per-device and per-link non-overlap constraints and memory-capacity constraints, minimizing makespan. The greedy scheduler approximates the same object by splitting blocks into sub-blocks and using a range-based bandwidth occupancy model to capture queuing.

What would settle it

Measure CrossPipe against the best static schedule on a real cross-region wide-area link, or on a network emulator that adds packet loss and bandwidth fluctuation, for a 70B-scale model at a bandwidth-delay ratio of twice the per-stage forward time; if the observed speedup falls well below the 33.6% measured with delay injection, or if the performance model's predicted iteration time deviates from measurement beyond the validation error reported in the paper, the central claim fails.

Watch

Extended reading notes

Core claim

CrossPipe's discovery is that the pipeline bubble caused by cross-datacenter communication is not fixed: it grows with every hop because static schedules contain critical paths with O(n_mb) cross-datacenter sends, and it can be shrunk by reordering computation blocks so that latency and bandwidth appear on fewer critical-path edges. The paper shows the schedule that achieves this is the solution of a constraint-optimization problem over a dependency graph whose vertices are forward, input-gradient, and weight-gradient computation blocks plus communication operations, subject to device memory limits; a greedy sub-block version matches it in most delay regimes. Under identical memory constraints, measured reductions reach 33.6%. The same model shows cross-datacenter pipeline parallelism dominates data parallelism until link bandwidth exceeds roughly a terabyte per second.

Load-bearing premise

The whole evaluation rests on the Alpha-Beta communication model: a fixed latency plus message size divided by bandwidth, with queuing, accurately describes real cross-datacenter links, so the speedups measured with injected delays would transfer to production wide-area connections.

Editorial extensions

If this is right

  • If the central claim holds, training a frontier LLM across two or four datacenters need not pay the full wide-area-network penalty: at relaxed memory and larger global batch size, CrossPipe's schedules approach the iteration time of a delay-free single-site schedule.
  • The pipeline-over-data-parallelism conclusion gives a concrete design rule for cross-datacenter clusters: keep data parallelism inside datacenters and put pipeline stage boundaries on the wide-area link, especially for mixture-of-experts models where data-parallel communication volume is larger.
  • The greedy schedule generation opens the door to runtime adaptation: because generating a near-optimal schedule is cheap, a training run can re-profile the network and hot-swap schedules as conditions change.
  • The accuracy of the performance model means simulation can be used to pick among schedules before deployment, avoiding trial runs that consume expensive wide-area bandwidth.
  • Under bandwidth-bound delays, the paper's results imply that extra device memory and a larger global batch size are the main levers for hiding communication, while latency-bound delays are largely absorbed once the per-stage forward time is long enough.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is whether the speedup survives real wide-area-network behavior such as packet loss, congestion-driven bandwidth variation, and transport-protocol effects; the paper's delay-injection validation suggests optimism, but those effects sit outside the Alpha-Beta model.
  • The model implies a practical capacity-planning rule: link bandwidth becomes the binding constraint once bandwidth delay per message approaches per-stage forward time, and additional memory and global batch size are the primary mitigation; latency below roughly half of forward time is nearly free.
  • The schedule-generation approach could plausibly combine with communication compression or asynchronous gradient updates across datacenters to push efficiency further, though the paper does not test that combination.
  • The F/D/W backward split means the method applies most directly to Transformer-style models whose backward pass separates input-gradient and weight-gradient phases; other architectures may need a different block decomposition.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. CrossPipe is a framework for generating and executing pipeline-parallel schedules for LLM training across datacenters. It formulates schedule generation as a constraint optimization problem that incorporates Alpha-Beta latency and bandwidth costs, and it introduces a greedy sub-block scheduling heuristic for scalability. A two-layer execution engine decouples schedule generation from communication orchestration, using dedicated streams and delay-aware placement of NCCL receives. The paper reports up to 33.6% reduction in iteration time versus static schedules on emulated cross-DC conditions, with further gains when global batch size and memory budgets are relaxed. The evaluation runs on an Alps GH200 cluster with injected latency and bandwidth delays, and includes comparisons with 1F1B, ZBH1, ZBV, and interleaved 1F1B schedules.

Significance. The paper's core contribution is a practical scheduling framework with a clean two-layer abstraction, open-source code, and a thorough evaluation: the greedy scheduler closely matches solver-based schedules across delay regimes, the execution engine integrates with Megatron-LM, and the memory/GBS trade-offs in Sections 7.2 and 7.4 are informative and clearly presented. The main limitation is that the emulated network is constructed from and validated against the same Alpha-Beta model that the scheduler optimizes (Appendix F.2), so the measured speedups are internal-consistency results rather than validation against real cross-datacenter links. The paper overstates the external validity of its central claim in the Abstract and Conclusion, but the schedule comparison itself is fair because all schedules run under identical emulated conditions.

major comments (4)
  1. [Section 6.4 / Appendix F.2] The evaluation in Section 7 uses a delay-injection mechanism whose validation (Appendix F.2, Figure 17) checks only that the injected delays match the Alpha-Beta model of Section 2.2. Because the CrossPipe schedules are generated from the same model, Figure 11 demonstrates self-consistency of the model and the emulator, not accuracy on real cross-datacenter links. The Abstract's claim to 'reduce training time by up to 33.6%' and Section 10's 'validated pipeline performance model' therefore overstate what is measured. Please either qualify these claims to emulated Alpha-Beta conditions or add a real cross-region experiment; packet loss, TCP dynamics, jitter, and shared bandwidth are acknowledged in Section 8.2 but not tested. I consider this the main load-bearing issue because the central quantitative claim depends on it.
  2. [Section 4.1 / Appendix D.1] The paper describes solver-based schedules as 'optimal' in the Abstract, Section 1, and Section 4.1. Appendix D.1 reports that CPLEX runs with a 7200-second time limit and early termination at a 1% relative gap, and the 32/64-stage traces appear to terminate well before a tight bound is reached. Please state for which configurations optimality is proven, or change the terminology to 'best-found' or 'optimal up to a 1% gap' for the schedules used in Section 7. Without this, the comparison of greedy near-optimality to an 'optimal' baseline is not well-defined.
  3. [Section 4.2.1 / Algorithm 1] The greedy algorithm's sub-block count nsub is a central free parameter, yet the paper never reports the nsub values used for the CrossUDSub results in Figures 8, 11-12 or Tables 5-6, nor is there a sensitivity analysis. Without these values the greedy results are not reproducible, and the claim that CrossUDSub matches solver-based CrossUD in most delay regimes cannot be assessed for robustness. Please report nsub and an ablation or at least a statement of how it was chosen.
  4. [Section 3.5 / Section 7.1, Figure 11] The performance-model predictions plotted in Figure 11 are generated with the same Alpha-Beta model that the injection mechanism implements, so their agreement with measurements is expected by construction. This should be stated explicitly near Figure 11 and in Section 3.5; the claim that 'this model accurately predicts the runtime of tested schedules in most configurations' should be scoped to the assumed delay model rather than presented as general validation. This does not undermine the schedule comparison, which is fair because all schedules run under the same emulated conditions, but it prevents over-reading the prediction accuracy.
minor comments (5)
  1. [Section 5.1, Table 3] The text says the static schedules are 'summarized in Table 8,' but the referenced table is numbered Table 3; please fix the cross-reference.
  2. [Table 1] Table 1 contains a typo: 'ration mb/nPP' should read 'ratio nmb/nPP'; please also clarify the definition of epsilon by stating that it is the ratio used in Section 7.4.
  3. [Section 7.1] The mapping of injected delay ratios to realistic cross-region parameters at the end of Section 7.1 assumes nDP=16, while the experiments in that section use nDP=1; please clarify whether the real-world grounding is intended as a scenario analysis rather than a direct interpretation of the measured settings.
  4. [Section 6.3] The description of the four dedicated streams ({Send, Recv} × {Next, Prev}) is clear, but Figure 10 is dense and hard to parse at printed size; please enlarge the figure or use higher-resolution annotations.
  5. [Appendix F.2] The validation uses 4 GB messages on a single node; latency tests with such large messages are not representative of the small activation/gradient messages used in the evaluated models, and a short justification of this choice would help the reader assess the injection accuracy.

Circularity Check

1 steps flagged · score 6.0 of 10

The 'validated' performance model is checked against an emulator calibrated to the same Alpha-Beta model, so Figure 11's agreement and the 33.6% speedup are partly self-consistent by construction.

  1. fitted input called prediction [Section 6.4, Appendix F.2, Section 7.1, Figure 11]
    "Results in Figure 17 demonstrate that our injection methods accurately reflect the communication model described in Section 2.2. ... This model accurately predicts the runtime of tested schedules in most configurations."

    The performance model (Section 3.5) computes block start times using the Alpha-Beta communication delays of Section 2.2. The emulator (Section 6.4) manufactures exactly those delays by spinning kernels and then Appendix F.2 validates the injection only by checking that it matches the expected Alpha-Beta delay: 'our injection methods accurately reflect the communication model described in Section 2.2.' Therefore the measured iteration times in Figure 11 are generated from the same model that the scheduler optimizes and that the model predicts. The statement that 'This model accurately predicts the runtime of tested schedules' is thus a self-consistency check, not an independent empirical validation.

full rationale

The scheduling contributions themselves are not circular: the CO and greedy formulations in Section 4 solve a well-defined makespan-minimization problem with stated inputs, and comparing CrossPipe against static schedules in the same emulated environment is a fair test of the optimizer's internal behavior. No constants are fit to make CrossPipe win, and there is no load-bearing self-citation chain. However, the paper's claim of a 'validated pipeline performance model' is circular. The emulator used in Section 7.1 is calibrated and validated against the same Alpha-Beta model (Section 2.2) that the performance model uses, so Figure 11's agreement between prediction and measurement is partly by construction. Consequently, the central 33.6% speedup is established only within the emulated abstraction, not on real cross-datacenter links. This is a partial circularity in the evaluation rather than in the scheduling derivation, and it warrants a moderate score.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the Alpha-Beta network model and on the fidelity of the emulated delays; the CO/greedy schedulers optimize makespan under this model. No entirely new physical entities are introduced. The practical GPU throughput in the PP vs DP analysis is a hand-set assumption.

free parameters (2)
  • nsub (number of sub-blocks in greedy scheduling)
    User-chosen granularity in the greedy algorithm (Section 4.2) that trades scheduling overhead against bubble reduction; the value used in experiments is not reported.
  • PGPU (practical GPU throughput) = 500 TFLOPs
    Assumed in Section 5.2 and Appendix E for the cross-DC PP vs DP comparison; directly affects estimated per-layer time and the conclusion that PP is preferable at low bandwidth.
assumptions (5)
  • domain assumption Alpha-Beta network model: communication cost is additive latency plus size/bandwidth, with queuing when multiple messages share a link.
    Invoked in Section 2.2 and used throughout the scheduler, the performance model, and the delay injection.
  • domain assumption PP and DP are the only viable cross-DC parallelisms; TP, SP, and EP are excluded due to high-frequency or alltoall communication.
    Section 2.1 argues that other parallelisms are unsuitable for cross-DC links, which frames the entire analysis.
  • domain assumption Scheduling and communication orchestration can be decoupled, with synchronization overhead excluded from the performance model.
    Section 3.5 states this assumption; Section 6.3 partially compensates with delay-aware orchestration, but the model still ignores some synchronization costs.
  • domain assumption Injected latency and bandwidth delays faithfully emulate real cross-DC network behavior.
    The evaluation (Section 7) relies on this; Appendix F validates the injection only against the same Alpha-Beta model, not against real WAN links.
  • domain assumption Pipeline runtime is defined from the first block on stage 0 to the last block on any rank, assuming stage 0 finishes last.
    Appendix A defines the runtime metric; this matches frameworks with global gradient-norm synchronization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training." pith.science (2026). https://pith.science/paper/NHIWZAZS

@misc{pith2026250700217,
  author       = {Pith},
  title        = {Pith review of: CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NHIWZAZS}},
  note         = {Machine review of arXiv:2507.00217}
}
read the original abstract

Training large language models (LLMs) now requires resources that exceed a single datacenter, making cross-datacenter strategies increasingly crucial. We present CrossPipe, a framework designed to optimize model training across geographically distributed datacenters by explicitly modeling and mitigating the impact of network latency and limited bandwidth. It enables unified analysis and optimization incorporating both pipeline parallelism (PP) and opportunities for overlapping data parallelism (DP) communication. CrossPipe generates optimized pipeline schedules using either solver-based optimal or fast near-optimal greedy algorithms, built upon a flexible execution engine that separates scheduling logic from communication details. Our evaluation shows that CrossPipe reduces training time by up to 33.6\% compared to traditional pipeline schedules under identical memory constraints. When memory constraints are relaxed, CrossPipe maintains strong performance despite communication delays, approaching the efficiency of idealized schedules without delays. CrossPipe offers improved scalability and resource utilization, particularly in environments with high network latency or limited bandwidth.

Figures

Figures reproduced from arXiv: 2507.00217 by the authors.

Figure 1
Figure 1. Alpha-Beta communication model sending a single message . When multiple pending messages are to be sent, the latter messages need to wait for the previous ones to be placed on the transmission link first. This results in an extra bandwidth delay . In this work, we assume a small number of DCs (e.g., nDC ≤ 4). For modeling the communication time Tcomm, we adopt the Alpha-Beta model, accounting for both latency (α) an… view at source ↗
Figure 2
Figure 2. Cross-DC infrastructure setup types and their impact [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Typical LLM architecture (left). Cross-DC DP: Each DC maintains independent model copies. Collective operations (Allgather and Reduce-Scatter, or Allreduce) synchronize gradients and update parameters (ZeRO stage 0 or 1). Cross-DC PP: The model is partitioned among DCs at layer boundaries. Each DC holds one or more pipeline stages. DP communication happens internally within each DC to synchronize gradients of stages… view at source ↗
Figures from the paper (12 more)
Figure 5
Figure 5. Figure 5: Traversal patterns for a single microbatch (two [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: depicts a critical path in a 1F1B schedule across 2 DCs involving 8 cross-DC PP communications. Since the path consists solely of true and schedule dependencies (Sec￾tion 3.3), its length imposes a lower bound on overall run￾time. For a 1F1B schedule of nmb microbatche…
Figure 7
Figure 7. Figure 7: The data dependency in a Wave schedule with [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 9
Figure 9. Figure 9: Simulation results comparing cross-DC PP and DP [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Components of the CrossPipe module. An overview of our implementation is shown in Fig￾ure 10 . The module begins by collecting system parame￾ters via lightweight benchmarks (Section 6.1). It then gen￾erates dynamic pipeline schedules (defining the order and timing of …
Figure 11
Figure 11. Figure 11: Evaluation of static and CrossPipe schedules under various emulated latency and bandwidth delay ratios. The runs [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 12
Figure 12. Figure 12: Trade-off between PP and DP (nPP ×nDP is fixed) in a 2-DC training. Using M70 with 16 nodes (nT P = 4 fixed) and a fixed GBS. Each subplot shows iteration time vs. nPP. Labels indicate delay ratios (Tlat/TF, Tbw/TF) and static mem￾ory usage. Memory in percentage shows…
Figure 13
Figure 13. Figure 13: Separating input gradient computation (DGrad, or D) and weight gradient computation (WGrad, or W) in a linear layer [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Scalability of MILP and CP solvers finding optimal [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Illustration of bubble strides in various static pipeline schedules (1F1B, IV1F1B, ZBH1, and ZBV). Setup: 16 stages, [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 17
Figure 17. Figure 17: Validation of latency and bandwidth delay injection [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 16
Figure 16. Figure 16: Mechanism for latency and bandwidth delay injec [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 52 canonical work pages

  1. [1]

    Microsoft Azure CTO: US data cen- ters will soon hit size limits.Semafor, October 2024

    Reed Albergotti. Microsoft Azure CTO: US data cen- ters will soon hit size limits.Semafor, October 2024. Technology

    work page 2024

  2. [2]

    Performance analysis of the selective repeat arq protocol

    Miltiades Anagnostou and Emmanuel Protonotarios. Performance analysis of the selective repeat arq protocol. IEEE Transactions on Communications, 34(2):127–135, 2003

    work page 2003

  3. [3]

    Varuna: Scal- able, low-cost training of massive deep learning models, 2021

    Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ra- machandran Ramjee, and Nipun Kwatra. Varuna: Scal- able, low-cost training of massive deep learning models, 2021

    work page 2021

  4. [4]

    Demystifying parallel and distributed deep learning: An in-depth concurrency analysis, 2018

    Tal Ben-Nun and Torsten Hoefler. Demystifying parallel and distributed deep learning: An in-depth concurrency analysis, 2018

    work page 2018

  5. [5]

    Tech firms are asking energy giant nextera for enough electricity to power miami, 2024

    Bloomberg. Tech firms are asking energy giant nextera for enough electricity to power miami, 2024

    work page 2024

  6. [6]

    Training deep nets with sublinear memory cost, 2016

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost, 2016

    2016

  7. [7]

    Google turns to nuclear to power AI data centres.BBC News, October 2024

    João da Silva. Google turns to nuclear to power AI data centres.BBC News, October 2024. Business

    work page 2024

  8. [8]

    An in-depth analysis of the slingshot interconnect

    Daniele De Sensi, Salvatore Di Girolamo, Kim H McMa- hon, Duncan Roweth, and Torsten Hoefler. An in-depth analysis of the slingshot interconnect. InSC20: Inter- national Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–14. IEEE, 2020

    work page 2020

Show all 69 references
  1. [9]

    Zhang, Han Bao, Han- wei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingx- uan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei...

    2025

  2. [10]

    Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen

    Arthur Douillard, Qixuan Feng, Andrei A. Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low-communication training of lan- guage models, 2024

    2024

  3. [11]

    Efficient training of large language models on distributed infras- tructures: A survey.arXiv preprint arXiv:2407.20018, 2024

    Jiangfei Duan, Shuo Zhang, Zerui Wang, Lijuan Jiang, Wenwen Qu, Qinghao Hu, Guoteng Wang, Qizhen Weng, Hang Yan, Xingcheng Zhang, et al. Efficient training of large language models on distributed infras- tructures: A survey.arXiv preprint arXiv:2407.20018, 2024

    2024 arXiv

  4. [12]

    The llama 3 herd of models, 2024

    Abhimanyu Dubey et al. The llama 3 herd of models, 2024

    2024

  5. [13]

    Dapple: A pipelined data parallel approach for training large models, 2020

    Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, Lansong Diao, Xiaoyong Liu, and Wei Lin. Dapple: A pipelined data parallel approach for training large models, 2020

    2020

  6. [14]

    R. W. Hamming. Error detecting and error correcting codes.The Bell System Technical Journal, 29(2):147– 160, 1950

    1950

  7. [15]

    Parameter-efficient fine-tuning for large models: A com- 14 prehensive survey.arXiv preprint arXiv:2403.14608, 2024

    Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A com- 14 prehensive survey.arXiv preprint arXiv:2403.14608, 2024

    2024 arXiv

  8. [16]

    Trends in machine learning hardware, 2023

    Marius Hobbhahn, Lennart Heim, and Gökçe Aydos. Trends in machine learning hardware, 2023. Accessed: 2024-10-15

    2023

  9. [17]

    The impact of network noise at large-scale com- munication performance

    Torsten Hoefler, Timo Schneider, and Andrew Lums- daine. The impact of network noise at large-scale com- munication performance. In2009 IEEE International Symposium on Parallel & Distributed Processing, pages 1–8, 2009

    2009

  10. [18]

    DISTMM: Accelerating distributed multi- modal model training

    Jun Huang, Zhen Zhang, Shuai Zheng, Feng Qin, and Yida Wang. DISTMM: Accelerating distributed multi- modal model training. In21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 1157–1171, Santa Clara, CA, April 2024. USENIX Association

    2024

  11. [19]

    Gpipe: Effi- cient training of giant neural networks using pipeline parallelism.Advances in neural information processing systems, 32, 2019

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Effi- cient training of giant neural networks using pipeline parallelism.Advances in neural information processing systems, 32, 2019

    2019

  12. [20]

    Adaptive mixtures of local experts.Neural computation, 3(1):79–87, 1991

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts.Neural computation, 3(1):79–87, 1991

    1991

  13. [21]

    Deepspeed ulysses: System optimizations for enabling training of extreme long se- quence transformer models, 2023

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajb- handari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long se- quence transformer models, 2023

    2023

  14. [22]

    Oobleck: Resilient distributed training of large models using pipeline templates

    Insu Jang, Zhenning Yang, Zhen Zhang, Xin Jin, and Mosharaf Chowdhury. Oobleck: Resilient distributed training of large models using pipeline templates. InPro- ceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 382–395. ACM, October 2023

    2023

  15. [23]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Tho...

    2023

  16. [24]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-An...

    2024

  17. [25]

    Technology-driven, highly-scalable dragonfly topology

    John Kim, Wiliam J Dally, Steve Scott, and Dennis Abts. Technology-driven, highly-scalable dragonfly topology. ACM SIGARCH Computer Architecture News, 36(3):77– 88, 2008

    2008

  18. [26]

    BPipe: Memory-balanced pipeline parallelism for training large language models

    Taebum Kim, Hyoungjoo Kim, Gyeong-In Yu, and Byung-Gon Chun. BPipe: Memory-balanced pipeline parallelism for training large language models. In An- dreas Krause, Emma Brunskill, Kyunghyun Cho, Bar- bara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Proceedings of th...

    2023

  19. [27]

    Reducing activation recomputation in large transformer models, 2022

    Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models, 2022

    2022

  20. [28]

    Breadth-first pipeline parallelism, 2023

    Joel Lamy-Poirier. Breadth-first pipeline parallelism, 2023

    2023

  21. [29]

    Pre-trained language models for text generation: A survey.ACM Computing Surveys, 56(9):1– 39, 2024

    Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Pre-trained language models for text generation: A survey.ACM Computing Surveys, 56(9):1– 39, 2024

    2024

  22. [30]

    Chimera: efficiently training large-scale neural networks with bidirectional pipelines

    Shigang Li and Torsten Hoefler. Chimera: efficiently training large-scale neural networks with bidirectional pipelines. InProceedings of the International Con- ference for High Performance Computing, Networking, Storage and Analysis, pages 1–14, 2021

    2021

  23. [31]

    Personal llm agents: Insights and survey about the capability, efficiency and security

    Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459, 2024

    2024 arXiv

  24. [32]

    Terapipe: Token-level pipeline parallelism for training large-scale language models

    Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. Terapipe: Token-level pipeline parallelism for training large-scale language models. In Marina Meila and Tong Zhang, ed- itors,Proceedings of the 38th International Conference on Machin...

    2021

  25. [33]

    Ring at- tention with blockwise transformers for near-infinite context, 2023

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring at- tention with blockwise transformers for near-infinite context, 2023. 15

    2023

  26. [34]

    Hanayo: Harnessing wave-like pipeline parallelism for enhanced large model training efficiency

    Ziming Liu, Shenggan Cheng, Haotian Zhou, and Yang You. Hanayo: Harnessing wave-like pipeline parallelism for enhanced large model training efficiency. InProceed- ings of the International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, pages 1–13, 2023

    2023

  27. [35]

    {CheckFreq}: Frequent, {Fine- Grained}{DNN} checkpointing

    Jayashree Mohan, Amar Phanishayee, and Vijay Chidambaram. {CheckFreq}: Frequent, {Fine- Grained}{DNN} checkpointing. In19th USENIX Conference on File and Storage Technologies (F AST 21), pages 203–216, 2021

    2021

  28. [36]

    Amazon vies for nuclear-powered data center: The deal has become a flash point over energy fairness.IEEE Spectrum, August 2024

    Andrew Moseman. Amazon vies for nuclear-powered data center: The deal has become a flash point over energy fairness.IEEE Spectrum, August 2024

    2024

  29. [37]

    MPI Forum.MPI: A Message-Passing Interface Stan- dard V ersion 3.1, 2015

    2015

  30. [38]

    Pipedream: Gen- eralized pipeline parallelism for dnn training

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Gen- eralized pipeline parallelism for dnn training. InPro- ceedings of the 27th ACM symposium on operating sys- tems principl...

    2019

  31. [39]

    Efficient large-scale language model training on gpu clusters using megatron-lm

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedin...

    2021

  32. [40]

    Nvidia dgx-1 with tesla v100 system architec- ture, 2017

    NVIDIA. Nvidia dgx-1 with tesla v100 system architec- ture, 2017

    2017

  33. [41]

    Nvidia grace hopper superchip architecture

    NVIDIA. Nvidia grace hopper superchip architecture. Whitepaper, NVIDIA Corporation, 2024

    2024

  34. [42]

    Out-of-order backprop: An effective scheduling technique for deep learning

    Hyungjun Oh, Junyeol Lee, Hyeongju Kim, and Jiwon Seo. Out-of-order backprop: An effective scheduling technique for deep learning. InProceedings of the Sev- enteenth European Conference on Computer Systems, pages 435–452, 2022

    2022

  35. [43]

    Zero bubble pipeline parallelism, 2023

    Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin. Zero bubble pipeline parallelism, 2023

    2023

  36. [44]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Day- iheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Ke- qin Bao, Kexin Ya...

    2025

  37. [45]

    Zero: Memory optimizations toward train- ing trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward train- ing trillion parameter models. InSC20: International Conference for High Performance Computing, Network- ing, Storage and Analysis. IEEE, November 2020

    2020

  38. [46]

    Microsoft deal signals booming demand from data centers to power AI.Reuters, September 2024

    Reuters. Microsoft deal signals booming demand from data centers to power AI.Reuters, September 2024. Energy, Grid & Infrastructure, Nuclear

    2024

  39. [47]

    Swarm parallelism: Training large models can be surprisingly communication-efficient, 2023

    Max Ryabinin, Tim Dettmers, Michael Diskin, and Alexander Borzunov. Swarm parallelism: Training large models can be surprisingly communication-efficient, 2023

    2023

  40. [48]

    Training compute of frontier ai models grows by 4-5x per year, 2024

    Jaime Sevilla and Edu Roldán. Training compute of frontier ai models grows by 4-5x per year, 2024. Ac- cessed: 2024-10-15

    2024

  41. [49]

    High performance mpi over the sling- shot interconnect: Early experiences

    Kawthar Shafie Khorassani, Chen Chun Chen, Bharath Ramesh, Aamir Shafi, Hari Subramoni, and Dha- baleswar Panda. High performance mpi over the sling- shot interconnect: Early experiences. InPractice and Experience in Advanced Research Computing, pages 1–7. 2022

    2022

  42. [50]

    Flashattention- 3: Fast and accurate attention with asynchrony and low- precision, 2024

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention- 3: Fast and accurate attention with asynchrony and low- precision, 2024

    2024

  43. [51]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017

    2017

  44. [52]

    Llamp: Assessing net- work latency tolerance of hpc applications with linear programming, 2024

    Siyuan Shen, Langwen Huang, Marcin Chrapek, Timo Schneider, Jai Dayal, Manisha Gajbe, Robert Wis- niewski, and Torsten Hoefler. Llamp: Assessing net- work latency tolerance of hpc applications with linear programming, 2024

    2024

  45. [53]

    Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020

    2020

  46. [54]

    Foteini Strati, Paul Elvinger, Tolga Kerimoglu, and Ana Klimovic. Ml training with cloud gpu shortages: Is cross- region the answer? InProceedings of the 4th Workshop on Machine Learning and Systems, EuroMLSys ’24, page 107–116, New York, NY , USA, 2024. Association for Comput...

    2024

  47. [55]

    Ml training with cloud gpu shortages: Is cross-region the answer? InProceedings of the 4th Workshop on Machine Learning and Systems, pages 107– 116, 2024

    Foteini Strati, Paul Elvinger, Tolga Kerimoglu, and Ana Klimovic. Ml training with cloud gpu shortages: Is cross-region the answer? InProceedings of the 4th Workshop on Machine Learning and Systems, pages 107– 116, 2024

    2024

  48. [56]

    Seq1f1b: Efficient sequence-level pipeline parallelism for large language model training, 2024

    Ao Sun, Weilin Zhao, Xu Han, Cheng Yang, Xinrong Zhang, Zhiyuan Liu, Chuan Shi, and Maosong Sun. Seq1f1b: Efficient sequence-level pipeline parallelism for large language model training, 2024

    2024

  49. [57]

    Adapipe: Optimizing pipeline parallelism with adaptive recomputation and partitioning

    Zhenbo Sun, Huanqi Cao, Yuanwei Wang, Guanyu Feng, Shengqi Chen, Haojie Wang, and Wenguang Chen. Adapipe: Optimizing pipeline parallelism with adaptive recomputation and partitioning. InProceedings of the 29th ACM International Conference on Architectural Support for Programmi...

    2024

  50. [58]

    Fusionllm: A decentral- ized llm training system on geo-distributed gpus with adaptive compression, 2024

    Zhenheng Tang, Xueze Kang, Yiming Yin, Xinglin Pan, Yuxin Wang, Xin He, Qiang Wang, Rongfei Zeng, Kaiy- ong Zhao, Shaohuai Shi, Amelie Chi Zhou, Bo Li, Bing- sheng He, and Xiaowen Chu. Fusionllm: A decentral- ized llm training system on geo-distributed gpus with adaptive compr...

    2024

  51. [59]

    Large language models in medicine.Nature medicine, 29(8):1930–1940, 2023

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine.Nature medicine, 29(8):1930–1940, 2023

    1930

  52. [60]

    Bamboo: Making preemptible in- stances resilient for affordable training of large dnns, 2022

    John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yi- fan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, and Guoqing Harry Xu. Bamboo: Making preemptible in- stances resilient for affordable training of large dnns, 2022

    2022

  53. [61]

    Attention is all you need.Advances in Neural Information Processing Systems, 2017

    A Vaswani. Attention is all you need.Advances in Neural Information Processing Systems, 2017

    2017

  54. [62]

    Hiding communication cost in distributed llm training via micro- batch co-execution, 2024

    Haiquan Wang, Chaoyi Ruan, Jia He, Jiaqi Ruan, Chengjie Tang, Xiaosong Ma, and Cheng Li. Hiding communication cost in distributed llm training via micro- batch co-execution, 2024

    2024

  55. [63]

    CocktailSGD: Fine-tuning foundation mod- els over 500Mbps networks

    Jue Wang, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang, Christopher De Sa, Christopher Re, and Ce Zhang. CocktailSGD: Fine-tuning foundation mod- els over 500Mbps networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Sc...

    2023

  56. [64]

    Overlap communication with dependent computation via decomposition in large deep learning models

    Shibo Wang, Jinliang Wei, Amit Sabne, Andy Davis, Berkin Ilbeyi, Blake Hechtman, Dehao Chen, Karthik Srinivasa Murthy, Marcello Maggioni, Qiao Zhang, Sameer Kumar, Tongfei Guo, Yuanzhong Xu, and Zongwei Zhou. Overlap communication with dependent computation via decomposition i...

    2023

  57. [65]

    Rail-only: A low-cost high- performance network for training llms with trillion pa- rameters, 2024

    Weiyang Wang, Manya Ghobadi, Kayvon Shakeri, Ying Zhang, and Naader Hasani. Rail-only: A low-cost high- performance network for training llms with trillion pa- rameters, 2024

    2024

  58. [66]

    Aligning large language models with human: A survey.arXiv preprint arXiv:2307.12966, 2023

    Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xing- shan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey.arXiv preprint arXiv:2307.12966, 2023

    2023 arXiv

  59. [67]

    Zhuang Wang, Zhen Jia, Shuai Zheng, Zhen Zhang, Xin- wei Fu, T. S. Eugene Ng, and Yida Wang. Gemini: Fast failure recovery in distributed training with in-memory checkpoints. InProceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 364–381, New York,...

    2023

  60. [68]

    Decentralized training of foundation models in heterogeneous environments, 2023

    Binhang Yuan, Yongjun He, Jared Quincy Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy Liang, Christopher Re, and Ce Zhang. Decentralized training of foundation models in heterogeneous environments, 2023

    2023

  61. [69]

    ready to send

    Quan Zhou, Haiquan Wang, Xiaoyan Yu, Cheng Li, Youhui Bai, Feng Yan, and Yinlong Xu. Mpress: Democratizing billion-scale model training on multi- gpu servers via memory-saving inter-operator paral- lelism. In2023 IEEE International Symposium on High- Performance Computer Archi...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.