Pith. sign in

REVIEW 3 major objections 5 minor 55 references

Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ADAPTRA claims that communication stragglers in pipeline-parallel training can be neutralized by adapting the pipeline schedule's slackness, with CPU-delegated communication removing kernel-level blocking, cutting iteration time by…

desk verdict ADAPTRA has a genuinely new slackness-threshold idea and credible measured speedups; the unproven Eq. (1) is a real but non-fatal soft spot. read the letter →

arxiv 2504.19232 v1 pith:D7B52YYH submitted 2025-04-27 cs.DC

Tianyuan Wu , Lunxi Cao , Hanfeng Lu , Xiaoxiao Jiang , Yinghao Yu , Siran Yang , Guodong Yang , Jiamang Wang
show 3 more authors
Lin Qu Liping Zhang Wei Wang
This is my paper · ORCID
classification cs.DC
keywords pipelineparallelismstragglercommunicationdelayslacknessZeroBubblehead-of-lineblockingCPU-sideRDMAhybrid-paralleltraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ADAPTRA is a training system that targets a common failure mode: a single slow network link between two pipeline stages can stall an entire large-model training run. The paper identifies two distinct damage mechanisms — dependency bubbles that cascade across stages once a delay exceeds a threshold, and head-of-line blocking where a slow send prevents the GPU from launching later compute kernels. Its central claim is that both can be eliminated by adjusting the pipeline schedule itself: giving adjacent stages more slackness (extra warm-up forwards) lets the schedule absorb delays up to a provable threshold, and moving communication off the GPU to CPU-side RDMA delegates removes the kernel blocking. The system-level claim is a 1.2–3.5× reduction in iteration time across 7B–140B models and sustained throughput above 20 samples/s on a 128-GPU trace with injected faults and an RNIC failure.

What carries the argument

The load-bearing object is the slackness $\Delta_i = x_i - x_{i+1}$, the difference between the number of forward micro-batches a stage executes during warm-up and the number its downstream neighbour executes. In the ZeroBubble steady state, each adjacent pair of stages has a feasible interval of length $2\Delta_i t$ between the corresponding backward and forward operations; Theorem 1 shows this interval absorbs a communication delay up to $(\Delta_i-1)t$ with linear cost, and that beyond it the delay contributes $\Theta(N c_i/(\Delta_i+1))$. Eq. (1) generalizes the absorption condition to non-uniform $F/B/W$ times, and Algorithm 2 uses it to pick new slackness values from profiled times and measured delays. The full schedule is then built by a discrete-time simulation with a $B>F>W$ operator-selection policy. The second mechanism is the decoupled data plane: on straggler detection, GPU-direct RDMA is replaced by pinned host-memory buffers and CPU delegate processes using multi-queue RDMA, so slow sends never occupy the GPU's kernel stream.

What would settle it

Run the scheduler from Section 4.3 on a pipeline with non-uniform $F/B/W$ times chosen so that Eq. (1) holds while the uniform threshold $(\Delta_i-1)t$ is violated, inject the corresponding communication delay $c_i$, and measure the accumulated pipeline delay; a growth in $c_i$ that is not linear (beyond $\Theta(c_i)$) would falsify the generalized no-bubble condition.

Watch

Extended reading notes

Core claim

The paper's central discovery is a threshold law for pipeline resilience. For a ZeroBubble pipeline (a schedule that eliminates bubbles by splitting the backward pass into backward-input B and backward-weight W operators), let $\Delta_i$ be the slackness between adjacent stages $S_i$ and $S_{i+1}$: the difference in their warm-up forward counts. A communication delay $c_i$ up to $(\Delta_i - 1)t$, where $t$ is the per-operation time, is absorbed into the schedule's natural $2\Delta_i t$ feasible interval and costs only linear slowdown; once $c_i$ exceeds that threshold, the delay amplifies to $\Theta(N c_i/(\Delta_i + 1))$, where $N$ is the number of microbatches. The paper extends the no-bubble condition to non-uniform forward and backward times via the inequality $t_i^F + t_i^B + 2c_i \le \Delta_i(t_{i+1}^F + t_{i+1}^B)$. ADAPTRA operationalizes this by maximizing minimum slackness at startup, recomputing per-stage warm-up counts when a straggler is detected, and generating the full schedule with a discrete-time simulator that yields schedules within 1% of the MILP optimum in under 100 ms.

Load-bearing premise

The rule that chooses how much slackness to add is a formula derived from an idealized schedule in which every operation takes the same time; if that formula does not describe the schedule the discrete-time heuristic actually produces when forward and backward times differ, the adapted pipeline can still cascade bubbles under the very delays it was tuned for.

Editorial extensions

If this is right

  • With slackness $\Delta_i$ on a link, delays up to $(\Delta_i - 1)t$ cost only linearly in $c_i$, so the practical way to stay fast under a straggler is to widen slackness on the affected link rather than to re-route traffic.
  • In the 14B sensitivity test, adapted schedules hold iteration-time growth to 1.13× under a 60 ms injected delay, versus 2.15–4.24× for 1F1B, ZeroBubble, and the CPU-delegation-only variant, and multiple simultaneous slow links are handled with more than a halving of iteration time relative to the non-adapting baselines.
  • A DP-group straggler can first be reassigned to a PP link and then absorbed: ADAPTRA delivers 1.96× further speedup over Falcon after that reassignment.
  • The CPU-delegation data plane costs under 5% overhead without stragglers (up to 30B models, 17% at 60B), so it can wait on standby, and the 128-GPU trace shows ADAPTRA sustaining throughput above 20 samples/s and finishing 1.37–1.41× faster than the non-adapting baselines.
  • RNIC failures can be handled by rerouting through healthy host RNICs, which avoids checkpoint-restart entirely.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The $\Delta_i$ threshold suggests a general control rule for any pipeline-parallel scheduler that exposes warm-up counts: monitor per-link delay and raise slackness only when a link approaches $(\Delta_i-1)t$, turning straggler handling into a simple per-link feedback loop.
  • Eq. (1) could support preemptive adaptation: predict from a delay trend when the threshold will be crossed and reconfigure before bubbles form, an extension beyond the paper's reactive detection.
  • The same feasible-interval idea may absorb variance in data-parallel all-reduce tails or other synchronous collectives, not just PP links, because any synchronized dependency has an analogous slack budget.
  • The slackness knob is orthogonal to elastic checkpointing and precomputed fail-stop plans, so combining a dynamic slackening controller with those templates could simultaneously cover slow-down and crash failures.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents ADAPTRA, a system for mitigating communication stragglers in hybrid-parallel DNN training. It makes two contributions: (i) an analytical model relating inter-stage communication delay to pipeline 'slackness' Δ_i, with a threshold (Δ_i−1)t below which delays only add linearly and above which delays amplify, together with a dynamic scheduling algorithm (Algorithm 2) that reconfigures ZeroBubble warm-up counts to increase slackness on straggling links; and (ii) a CPU-delegation data plane that offloads PP communication to host memory and CPU-side RDMA, eliminating head-of-line blocking and providing RNIC fault tolerance. The system is implemented on Megatron-LM/ZeroBubble and evaluated with GPT-2 models from 7B to 140B parameters on up to 128 H800 GPUs, reporting 1.2–3.5× average iteration-time improvements over baselines under injected delays and a 1.41× throughput improvement over 1F1B/ZB on a 1,200-iteration trace with stragglers and an RNIC failure.

Significance. If the delay-resilience analysis is correct, the paper identifies a practical and general design principle—slackness between adjacent pipeline stages is the key structural parameter for absorbing communication delays—and demonstrates an effective end-to-end system combining schedule adaptation with CPU-delegated communication. The evaluation is unusually thorough for a systems paper: it covers single- and multi-link stragglers, DP-to-PP migration, a large-scale 128-GPU trace with injected faults, and overhead measurements showing <5% overhead for the delegated path up to 30B parameters and 17% at 60B. The near-optimality of the heuristic scheduler against MILP (within 1% gap) is also a concrete strength. However, the paper's theoretical claims are more formal than the proofs support, and one of the two key algorithms (Algorithm 2) lacks a feasibility guarantee; these issues need to be addressed before the central mechanism can be relied on beyond the specific configurations evaluated.

major comments (3)
  1. [Section 4.1, Eq. (1)] The 'if and only if' non-uniform extension in Eq. (1) is unproven. The proof of Theorem 1 itself does not rigorously derive the amplification Θ(N c_i/(Δ_i+1)); the statement 'each group of Δ_i+1 operations contributes c_i to the total' is asserted without accounting for exact schedule positions or the finite number N of microbatches. More importantly, Eq. (1) presumes that in schedules generated by the Algorithm 3/4 heuristic (B>F>W priority, discretization δ, non-uniform tF_i, tB_i) there is a regular steady-state window of length Δ_i(tF_{i+1}+tB_{i+1}) between the end of B_{i+1,a} and the start of F_{i+1,b}. The paper does not prove this, and Figure 8 validates the threshold only for uniform idealized schedules, not for the actual heuristic output. Since Algorithm 2 (line 7) chooses Δ_i based on Eq. (1), this gap directly affects the robustness of the claimed 1.2–3.5× speedup. I request either a proof of Eq. (1) under explicit assumptions on the generated schedule, or an additional simulation study that verifies, for random non-uniform profiles and delay values, that the schedule generated by §4.3 actually absorbs delays up to the threshold predicted by Eq. (1).
  2. [Section 4.2, Algorithm 2] The clamping min(N−2S, ...) does not guarantee that the computed warm-up counts are feasible. With x_{S−1}=1, the recursion x_i = x_{i+1}+Δ_i yields x_0 = 1 + Σ_{i=0}^{S−2} Δ_i. Even if each Δ_i ≤ N−2S, for S=8 and N=32 setting Δ_i=6 for all i gives x_0=43 > N, which is impossible because a stage cannot process more than N warm-up forwards when there are only N microbatches. The algorithm therefore needs an explicit feasibility constraint (e.g., x_i ≤ N, or a dependency-aware bound) or a proof that the recursion is always feasible for the intended input ranges; otherwise the dynamic adaptation may request a schedule that cannot be built.
  3. [Section 4.1 and Appendix] The proof of Lemma 1 relies on a specific ordering assumption (F_{i,x_i} ≺ B_{i,1} ≺ F_{i,x_i+1}) that is not established for general ZeroBubble schedules, where backward-input and backward-weight operators are interleaved. This ordering is also used in the proof of Theorem 1. As the lemma underpins the definition of slackness Δ_i and the sign of Δ_i, the paper should either state the exact scheduling disciplines under which the ordering holds or provide a proof that every schedule generated by Algorithm 3/4 satisfies it.
minor comments (5)
  1. [Section 7] The evaluation questions are numbered (1), (3), (2) in the text; renumber them consistently as (1), (2), (3).
  2. [Figure 21] The throughput values printed on the plot are ambiguous; label the curves directly or provide a clearer legend.
  3. [General] The paper does not report variance or repeated runs for the timing measurements; at least for the headline speedup numbers, a small number of repetitions or min/max ranges would strengthen the claims.
  4. [Algorithm 4, line 15] The condition uses an undefined symbol 's'; it should be 'i' (i.e., 'if o.type = B and i ≠ 0').
  5. [Section 4.3] The complexity bound O(NS^2⌈to/δ⌉) is stated without a derivation of the S^2 factor despite the claim of constant-time policy evaluation; please clarify how the quadratic term arises.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the analytical threshold and Eq. (1) guide schedule adaptation rather than being fitted to the reported speedups, and the build on Falcon is implementation reuse, not load-bearing.

full rationale

The paper's central derivation is Theorem 1 in Section 4.1, which relates the defined slackness Delta_i = x_i - x_{i+1} to a communication-delay tolerance threshold (Delta_i - 1)t. The proof uses an interval argument on an idealized ZeroBubble schedule; it is an analytic claim, not a restatement of measured iteration times. Algorithm 2 chooses Delta_i to satisfy Eq. (1), an explicit but unproven extension of the uniform-time condition to non-uniform F/B/W times; this is a control action (make slackness large enough to absorb the measured c_i), and the reported 1.2-3.5x improvements are end-to-end measurements, not numerical outputs forced by the construction of Delta_i. Figure 8 independently validates the threshold in simulation, and Section 7.4 checks scheduler optimality against an MILP formulation. The reuse of Falcon [48] (overlapping authors) for straggler detection and DP-to-PP link migration is implementation inheritance and background motivation; it does not carry the load of the new analytical model or the CPU-delegation design. No equation is shown to be equivalent to its own input by definition, and no fitted parameter is relabeled as a prediction. The unproved bridge is Eq. (1)'s 'if and only if' extension and its assumed steady-state window; this is a correctness and robustness risk, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No data-fitted free parameters were identified. The analytical model depends on uniform operation times in Theorem 1, a constant additive communication delay, no HOL blocking during the adaptation analysis, and the heuristic scheduler producing schedules that match the model's slackness. No new physical entities are proposed.

assumptions (4)
  • domain assumption All operations F, B, and W have uniform execution time t in the analytical model.
    Stated in Section 4.1. The paper claims an extension to heterogeneous times via Eq. (1), but that extension is not proven.
  • domain assumption The communication delay c_i can be represented as a constant additive latency per inter-stage transfer and is measurable at runtime.
    Used throughout Theorem 1 and Algorithm 2. Real congestion is time-varying and may violate the steady-state threshold.
  • ad hoc to paper The full-pipeline scheduler in Section 4.3 produces a schedule whose feasible intervals and slackness match the idealized ZeroBubble model in Theorem 1.
    Algorithm 2 chooses Δ_i from the model, but the discrete-time heuristic in Algorithm 4 may not realize exact intervals; no proof is given that the generated schedule preserves the model's absorption guarantees.
  • domain assumption No head-of-line blocking stalls occur during the pipeline adaptation analysis.
    Section 4 explicitly assumes this, relying on the delegation design in Section 5. If delegation has overhead or does not fully remove blocking, the analytical guarantee does not transfer to the system.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation." pith.science (2026). https://pith.science/paper/D7B52YYH

@misc{pith2026250419232,
  author       = {Pith},
  title        = {Pith review of: Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D7B52YYH}},
  note         = {Machine review of arXiv:2504.19232}
}
read the original abstract

Training large Deep Neural Network (DNN) models at scale often encounters straggler issues, mostly in communications due to network congestion, RNIC/switch defects, or topological asymmetry. Under advanced pipeline parallelism, even minor communication delays can induce significant training slowdowns. This occurs because (1) slow communication disrupts the pipeline schedule, creating cascading "bubbles" in a domino effect, and (2) current GPU kernel scheduling is susceptible to head-of-line blocking, where slow communication blocks subsequent computations, further adding to these bubbles. To address these challenges, we present ADAPTRA, a straggler-resilient training system with two key optimizations. First, it optimally adapts the pipeline schedule in the presence of stragglers to absorb communication delays without inducing cascading bubbles, using a simple yet effective algorithm guided by an analytical model. Second, upon detecting slow communication, ADAPTRA offloads communication operations from GPU to host memory and utilizes CPU-side RDMA for data transfer. This eliminates head-of-line blocking as subsequent computation kernels can be scheduled immediately on GPUs. Together, these optimizations effectively reduce pipeline stalls in the presence of communication stragglers, improving the training iteration time by 1.2-3.5x in our experiments under various settings.

Figures

Figures reproduced from arXiv: 2504.19232 by the authors.

Figure 1
Figure 1. GPT-2 14B training performance on 8 nodes (one [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Head-of-line blocking due to sequential kernel [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Iteration time growth under different per packet [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: An ideal straggler-free ZeroBubble [33] pipeline with 4 stages and 12 microbatches, completing in 390 ms. 3.1 Domino Effect of Cascading Bubbles Pipeline parallelism orchestrates stage execution with strict data dependencies. In ZB scheduling, these dependencies manife…
Figure 6
Figure 6. Figure 6: Slow communication (SF2) induces HOL blocking stalls (F4) due to sequential GPU kernel scheduling. 3.3 Which Layer to Optimize? Our analysis reveals that communication stragglers degrade pipeline performance through two mechanisms: dependency bubbles and HOL blocking s…
Figure 8
Figure 8. Figure 8: Simulated delay and bubble rate using different [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Naively adopting NCCL-based opportunistic com [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 11
Figure 11. Figure 11: Optimized data transfer for sending data. [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 15
Figure 15. Figure 15: Sensitivity analysis of ADAPTRA and baselines using a 14B model under various delay values. of each operator within an iteration using cudaEvent, with the execution timeline shown in [PITH_FULL_IMAGE:figures/full_fig_p010_15.png]
Figure 14
Figure 14. Figure 14: The actual execution of the schedule using [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]
Figure 16
Figure 16. Figure 16: Evaluation on single inter-PP communication degradation under various model settings and delay locations. [PITH_FULL_IMAGE:figures/full_fig_p011_16.png]
Figure 17
Figure 17. Figure 17: Iteration times of a 14B model under multiple [PITH_FULL_IMAGE:figures/full_fig_p011_17.png]
Figure 19
Figure 19. Figure 19: The relative error to optimal solution and solving [PITH_FULL_IMAGE:figures/full_fig_p012_19.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 29 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Challenging the need for packet spraying in large-scale distributed training

    Vamsi Addanki, Prateesh Goyal, and Ilias Marinos. Challenging the need for packet spraying in large-scale distributed training. arXiv preprint arXiv:2407.00550, 2024

    arXiv 2024

  3. [3]

    Conga: Distributed congestion-aware load balancing for datacenters

    Mohammad Alizadeh, Tom Edsall, Sarang Dharma- purikar, Ramanan Vaidyanathan, Kevin Chu, Andy Fin- gerhut, Vinh The Lam, Francis Matus, Rong Pan, Navin- dra Yadav, et al. Conga: Distributed congestion-aware load balancing for datacenters. In Proceedings of the 2014 ACM conference on SIGCOMM, pages 503–514, 2014

    work page 2014

  4. [4]

    Varuna: scal- able, low-cost training of massive deep learning models

    Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ra- machandran Ramjee, and Nipun Kwatra. Varuna: scal- able, low-cost training of massive deep learning models. In Proceedings of the Seventeenth European Conference on Computer Systems, pages 472–487, 2022

    2022

  5. [5]

    Crux: Gpu-efficient communication scheduling for deep learning training

    Jiamin Cao, Yu Guan, Kun Qian, Jiaqi Gao, Wencong Xiao, Jianbo Dong, Binzhang Fu, Dennis Cai, and Ennan Zhai. Crux: Gpu-efficient communication scheduling for deep learning training. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 1–15, 2024

    work page 2024

  6. [6]

    Revisiting distributed syn- chronous sgd

    Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Ben- gio, and Rafal Jozefowicz. Revisiting distributed syn- chronous sgd. arXiv preprint arXiv:1604.00981, 2016

    arXiv 2016

  7. [7]

    Mscclang: Microsoft collective communication language

    Meghan Cowan, Saeed Maleki, Madanlal Musuvathi, Olli Saarikivi, and Yifan Xiong. Mscclang: Microsoft collective communication language. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 502–514, 2023

    2023

  8. [8]

    Xputimer: Anomaly diagnostics for divergent llm training in gpu clusters of thousand-plus scale

    Weihao Cui, Ji Zhang, Han Zhao, Chao Liu, Wenhao Zhang, Jian Sha, Quan Chen, Bingsheng He, and Minyi Guo. Xputimer: Anomaly diagnostics for divergent llm training in gpu clusters of thousand-plus scale. arXiv preprint arXiv:2502.05413, 2025

    arXiv 2025

Show all 55 references
  1. [9]

    On the impact of packet spraying in data center networks

    Advait Dixit, Pawan Prakash, Y Charlie Hu, and Ra- mana Rao Kompella. On the impact of packet spraying in data center networks. In 2013 Proceedings IEEE INFOCOM, pages 2130–2138. IEEE, 2013

    2013

  2. [10]

    ACCL: Architecting highly scalable distributed training systems with highly efficient collective communication library

    Jianbo Dong, Shaochuang Wang, Fei Feng, Zheng Cao, Heng Pan, Lingbo Tang, Pengcheng Li, Hao Li, Qianyuan Ran, Yiqun Guo, et al. ACCL: Architecting highly scalable distributed training systems with highly efficient collective communication library. IEEE micro, 41(5):85–92, 2021

    2021

  3. [11]

    The Llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  4. [12]

    Recycle: Resilient training of large dnns using pipeline adaptation

    Swapnil Gandhi, Mark Zhao, Athinagoras Skiadopoulos, and Christos Kozyrakis. Recycle: Resilient training of large dnns using pipeline adaptation. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, pages 211–228, 2024

    2024

  5. [13]

    Rdma over ethernet for distributed training at meta scale

    Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mohammad Riftadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, et al. Rdma over ethernet for distributed training at meta scale. In Proceedings of the ACM SIGCOMM 2024 Conference, pa...

    2024

  6. [14]

    Deepseek-r1: Incentiviz- ing reasoning capability in LLMs via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentiviz- ing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  7. [15]

    Addressing the straggler problem for iterative convergent parallel ml

    Aaron Harlap, Henggang Cui, Wei Dai, Jinliang Wei, Gregory R Ganger, Phillip B Gibbons, Garth A Gibson, and Eric P Xing. Addressing the straggler problem for iterative convergent parallel ml. In Proceedings of the seventh ACM symposium on cloud computing , pages 98–111, 2016

    2016

  8. [16]

    Characterization of large language model development in the datacenter

    Qinghao Hu, Zhisheng Ye, Zerui Wang, Guoteng Wang, Meng Zhang, Qiaoling Chen, Peng Sun, Dahua Lin, Xi- aolin Wang, Yingwei Luo, et al. Characterization of large language model development in the datacenter. In 21st USENIX Symposium on Networked Systems Design and Implementatio...

    2024

  9. [17]

    Gpipe: Effi- cient training of giant neural networks using pipeline parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Effi- cient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019

    2019

  10. [18]

    Gloo, 2025

    Facebook Incubator. Gloo, 2025. Accessed: 2025-03- 26

    2025

  11. [19]

    Oobleck: Resilient distributed training of large models using pipeline templates

    Insu Jang, Zhenning Yang, Zhen Zhang, Xin Jin, and Mosharaf Chowdhury. Oobleck: Resilient distributed training of large models using pipeline templates. In 14 Proceedings of the 29th Symposium on Operating Sys- tems Principles, pages 382–395, 2023

    2023

  12. [20]

    Analysis of{Large-Scale}{Multi-Tenant}{GPU} clus- ters for{DNN} training workloads

    Myeongjae Jeon, Shivaram Venkataraman, Amar Phan- ishayee, Junjie Qian, Wencong Xiao, and Fan Yang. Analysis of{Large-Scale}{Multi-Tenant}{GPU} clus- ters for{DNN} training workloads. In 2019 USENIX Annual Technical Conference (USENIX ATC 19), pages 947–960, 2019

    2019

  13. [21]

    MegaScale: Scaling large language model training to more than 10,000{GPUs}

    Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, et al. MegaScale: Scaling large language model training to more than 10,000{GPUs}. In 21st USENIX Symposium on Networked Systems De- sign and Implementati...

    2024

  14. [22]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scal- ing laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  15. [23]

    Strack: A reliable multipath trans- port for ai/ml clusters

    Yanfang Le, Rong Pan, Peter Newman, Jeremias Blendin, Abdul Kabbani, Vipin Jain, Raghava Sivaramu, and Francis Matus. Strack: A reliable multipath trans- port for ai/ml clusters. arXiv preprint arXiv:2407.15266, 2024

    2024 arXiv

  16. [24]

    Malleus: Straggler-resilient hybrid parallel training of large-scale models via mal- leable data and model parallelization

    Haoyang Li, Fangcheng Fu, Hao Ge, Sheng Lin, Xu- anyu Wang, Jiawen Niu, Yujie Wang, Hailin Zhang, Xi- aonan Nie, and Bin Cui. Malleus: Straggler-resilient hybrid parallel training of large-scale models via mal- leable data and model parallelization. arXiv preprint arXiv:2410.1...

    arXiv

  17. [25]

    Torchtitan: One-stop pytorch native solution for production ready llm pre-training

    Wanchao Liang, Tianyu Liu, Less Wright, Will Consta- ble, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, et al. Torchtitan: One-stop pytorch native solution for production ready llm pre-training. arXiv preprint arXiv:2410.06511 , 2024

    arXiv

  18. [26]

    Deepseek-v3 techni- cal report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 techni- cal report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  19. [27]

    Sdpipe: A semi-decentralized framework for heterogeneity-aware pipeline-parallel training

    Xupeng Miao, Yining Shi, Zhi Yang, Bin Cui, and Zhi- hao Jia. Sdpipe: A semi-decentralized framework for heterogeneity-aware pipeline-parallel training. Proceed- ings of the VLDB Endowment, 16(9):2354–2363, 2023

    2023

  20. [28]

    {CheckFreq}: Frequent, {Fine- Grained}{DNN} checkpointing

    Jayashree Mohan, Amar Phanishayee, and Vijay Chidambaram. {CheckFreq}: Frequent, {Fine- Grained}{DNN} checkpointing. In 19th USENIX Conference on File and Storage Technologies (FAST 21), pages 203–216, 2021

    2021

  21. [29]

    Pipedream: Gen- eralized pipeline parallelism for dnn training

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Gen- eralized pipeline parallelism for dnn training. In Pro- ceedings of the 27th ACM symposium on operating sys- tems princip...

    2019

  22. [30]

    Efficient large-scale language model training on gpu clusters using megatron-lm

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedin...

    2021

  23. [31]

    Nvidia nsight systems, 2025

    NVIDIA Cooperation. Nvidia nsight systems, 2025. Accessed: 2025-03-24

    2025

  24. [32]

    Openai sora, 2024

    OpenAI. Openai sora, 2024. Accessed: 2024-09-13

    2024

  25. [33]

    Zero bubble pipeline parallelism

    Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin. Zero bubble pipeline parallelism. arXiv preprint arXiv:2401.10241, 2023

    2023 arXiv

  26. [34]

    Alibaba hpn: A data center network for large language model training

    Kun Qian, Yongqing Xi, Jiamin Cao, Jiaqi Gao, Yichi Xu, Yu Guan, Binzhang Fu, Xuemei Shi, Fangbo Zhu, Rui Miao, et al. Alibaba hpn: A data center network for large language model training. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 691–706, 2024

    2024

  27. [35]

    In 21st USENIX Sympo- sium on Networked Systems Design and Implementation (NSDI 24), pages 1403–1420, 2024

    Sudarsanan Rajasekaran, Manya Ghobadi, and Aditya Akella.{CASSINI}:{Network-Aware} job scheduling in machine learning clusters. In 21st USENIX Sympo- sium on Networked Systems Design and Implementation (NSDI 24), pages 1403–1420, 2024

    2024

  28. [36]

    Zero: Memory optimizations toward train- ing trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward train- ing trillion parameter models. In IEEE/ACM SC, 2020

    2020

  29. [37]

    Redis - the real-time data platform,

    Salvatore Sanfilippo. Redis - the real-time data platform,

  30. [38]

    BLOOM: A 176b- parameter open-access multilingual language model

    Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ili ´c, et al. BLOOM: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2023

    2023 arXiv

  31. [39]

    15 {TACCL}: Guiding collective algorithm synthesis us- ing communication sketches

    Aashaka Shah, Vijay Chidambaram, Meghan Cowan, Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Jacob Nelson, Olli Saarikivi, and Rachee Singh. 15 {TACCL}: Guiding collective algorithm synthesis us- ing communication sketches. In 20th USENIX Sympo- sium on Networked Systems Desi...

    2023

  32. [40]

    Megatron-LM: Training multi-billion parameter lan- guage models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter lan- guage models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  33. [41]

    Effective multi-gpu communication using multiple cuda streams and threads

    Mohammed Sourouri, Tor Gillberg, Scott B Baden, and Xing Cai. Effective multi-gpu communication using multiple cuda streams and threads. In 2014 20th IEEE International Conference on Parallel and Distributed Systems (ICPADS), pages 981–986. IEEE, 2014

    2014

  34. [42]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  35. [43]

    Multipath issues in unicast and multicast next-hop selection

    Dave Thaler and C Hopps. Multipath issues in unicast and multicast next-hop selection. Technical report, 2000

    2000

  36. [44]

    Bamboo: Making preemptible in- stances resilient for affordable training of large{DNNs}

    John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yi- fan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, and Guoqing Harry Xu. Bamboo: Making preemptible in- stances resilient for affordable training of large{DNNs}. In 20th USENIX Symposium on Networked Systems De- sign and Impl...

    2023

  37. [45]

    Jeffrey D. Ullman. Np-complete scheduling problems. Journal of Computer and System sciences, 10(3):384– 393, 1975

    1975

  38. [46]

    Ma- chine learning model sizes and the parameter gap

    Pablo Villalobos, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Anson Ho, and Marius Hobbhahn. Ma- chine learning model sizes and the parameter gap. arXiv preprint arXiv:2207.02852, 2022

    2022 arXiv

  39. [47]

    Zhuang Wang, Zhen Jia, Shuai Zheng, Zhen Zhang, Xin- wei Fu, T. S. Eugene Ng, and Yida Wang. Gemini: Fast failure recovery in distributed training with in-memory checkpoints. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 364–381, New York...

    2023

  40. [48]

    FALCON: Pinpointing and mitigating stragglers for large-scale hybrid-parallel train- ing

    Tianyuan Wu, Wei Wang, Yinghao Yu, Siran Yang, Wen- chao Wu, Qinkai Duan, Guodong Yang, Jiamang Wang, Lin Qu, and Liping Zhang. FALCON: Pinpointing and mitigating stragglers for large-scale hybrid-parallel train- ing. arXiv preprint arXiv:2410.12588, 2024

    arXiv

  41. [49]

    SuperBench: Improving cloud AI infrastructure reliability with proactive valida- tion

    Yifan Xiong, Yuting Jiang, Ziyue Yang, Lei Qu, Gu- oshuai Zhao, Shuguang Liu, Dong Zhong, Boris Pinzur, Jie Zhang, Yang Wang, et al. SuperBench: Improving cloud AI infrastructure reliability with proactive valida- tion. In 2024 USENIX Annual Technical Conference (ATC’24), page...

    2024

  42. [50]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  43. [51]

    Deepep: an efficient expert- parallel communication library

    Chenggang Zhao, Shangyan Zhou, Liyue Zhang, Chengqi Deng, Zhean Xu, Yuxuan Liu, Kuai Yu, Ji- ashi Li, and Liang Zhao. Deepep: an efficient expert- parallel communication library. https://github.com/ deepseek-ai/DeepEP, 2025

    2025

  44. [52]

    Alpa: Automating inter-and{Intra-Operator} parallelism for distributed deep learning

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. Alpa: Automating inter-and{Intra-Operator} parallelism for distributed deep learning. In 16th USENIX Sympo- sium on Operating Systems D...

    2022

  45. [53]

    Rlhfuse: Efficient rlhf training for large language models with inter-and intra- stage fusion

    Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, et al. Rlhfuse: Efficient rlhf training for large language models with inter-and intra- stage fusion. arXiv preprint arXiv:2409.13221, 2024

    2024 arXiv

  46. [54]

    Falcon: Addressing strag- glers in heterogeneous parameter server via multiple par- allelism

    Qihua Zhou, Song Guo, Haodong Lu, Li Li, Minyi Guo, Yanfei Sun, and Kun Wang. Falcon: Addressing strag- glers in heterogeneous parameter server via multiple par- allelism. IEEE Transactions on Computers, 70(1):139– 155, 2020. 16 Appendix Proof of the Lemma in § 4.1 Assume for ...

    2020

  47. [2009]

    Accessed: 2024-09-08

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.