Pith. sign in

REVIEW 4 major objections 4 minor 4 cited by

The paper claims that on increasingly heterogeneous LLM architectures, pipeline bubbles shrink—and throughput rises—only when model partition, stage placement, and micro-batch scheduling are co-optimized, not tuned one at a time.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 14:37 UTC pith:E5Y3OYWA

load-bearing objection A real co-optimization idea buried under an identity mismatch and thin performance-model validation; worth reviewing but not in this state. the 4 major comments →

arxiv 2509.23722 v2 pith:E5Y3OYWA submitted 2025-09-28 cs.DC cs.AI

OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Scheduling

Jihu Guo , Tenghui Ma , Wei Gao , Peng Sun , Xun Chen , Jiaxing Li , Zhisheng Ye , Yuyang Jin
show 1 more author
Dahua Lin
This is my paper
classification cs.DC cs.AI
keywords pipeline parallelismlarge language model trainingmodel heterogeneitymodel partitionmodel placementworkload schedulingbubble reductiondistributed training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The submission, titled OctoPipe but naming its system AdaPtis throughout the body, sets out to show that pipeline-parallel training of heterogeneous LLMs is best treated as one joint optimization problem rather than three separate ones. The target is the idle GPU time (pipeline bubbles) that grows when architectures mix oversized vocabularies, MoE layers, and hybrid attention/state-space blocks. The paper builds a simulator-based performance model, an iterative bubble-aware tuner that co-optimizes model partition, model placement, and workload scheduling, and an executor that runs irregular schedules without deadlock while overlapping communication with computation. On Gemma, DeepSeek, and Nemotron-H, the system reports 1.15–1.44× throughput gains over state-of-the-art pipeline baselines, with the largest gains on the most heterogeneous models. The appeal of the result is practical: if correct, it converts architectural heterogeneity from a pipeline-parallel liability into a tuneable scheduling problem.

Core claim

The paper's central claim is that the runtime of each device in a pipeline can be modeled as T_d = C_d + BubbleTime(d) − OverlapTime(d), and that this decomposition makes it possible to search the joint space of partition, placement, and schedule. Using profiled per-layer costs, the model aggregates stages into devices, simulates where bubbles appear, and detects where communication could hide under computation. A generator then tunes the bottleneck phase repeatedly—moving layers between stages with the largest and smallest bubbles, permuting stage-to-device mappings for finer granularity, and reordering forward/input-gradient/parameter-gradient work within memory constraints—while rolling b

What carries the argument

Three pieces carry the argument. The Pipeline Performance Model is a graph-based simulator whose core identity is T_d = C_d + BubbleTime(d) − OverlapTime(d): profiled per-layer compute and memory costs are summed into stages, stages into devices, and the device's wall time is estimated by subtracting communication that can overlap computation from the raw compute plus bubbles. The Pipeline Generator is a heuristic search that starts from known baseline pipelines and each iteration tunes only the current bottleneck phase, rolling back any change that degrades the simulated objective, which keeps a combinatorial search tractable. The Pipeline Executor turns schedules into computation and commu

Load-bearing premise

The load-bearing premise is that a device's true wall-clock time is well approximated by summing profiled per-layer compute costs and then subtracting cleanly overlapped communication from compute-plus-bubble time (T_d = C_d + BubbleTime(d) − OverlapTime(d)); fidelity is verified only on Nemotron-H at 4K sequence length, so the model's accuracy on Gemma, DeepSeek, and other sequence lengths is assumed.

What would settle it

A concrete check: profile Gemma and DeepSeek at 1K, 16K, and 32K sequence lengths, generate OctoPipe/AdaPtis schedules from the simulator, then run them. If real throughput deviates from the simulated throughput by much more than the measured 2.12% average error (or if the gap to baselines disappears), the additive performance model—and the speedups built on it—would fail.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Throughput on heterogeneous LLM families improves by 1.15–1.44× over prior pipeline-parallel methods, with the gap growing on the most heterogeneous architectures (up to 2.14× over the interleaved-1F1B baseline).
  • Single-phase tuning is not just weaker but can backfire: the ablation shows placement-only tuning slowing Nemotron-H by 16%, evidence that the three phases interact.
  • Pipeline generation remains practical: the iterative search completes large configurations—large models, many stages, 256 micro-batches—in about 100 seconds, where ILP-based search is impractical.
  • The performance model is accurate enough to guide search: average throughput prediction error is 2.12% on the Nemotron-H family at 4K sequence length, with simulated traces closely matching real execution traces.
  • The system scales: from 8 to 128 GPUs it reports 534% strong-scaling and 519% weak-scaling throughput growth, ahead of all compared methods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same co-optimization loop should transfer to other axes of heterogeneity—variable sequence lengths across micro-batches, mixed expert parallelism, or mixed GPU types—because the model only needs per-layer profiled costs, but the paper only validates architectural heterogeneity.
  • Editorial inference: since the simulator exposes bubble and overlap breakdowns, it could serve as an online re-tuner when batch size, sequence length, or cluster shape changes during a long run; the paper presents the tuner as offline generation only.
  • Editorial inference: the instruction-based, deadlock-checked executor is not pipeline-specific in principle, so the scheduling machinery might be reusable for sequence-parallel or disaggregated inference workloads; this is not attempted in the paper.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript body, titled 'AdaPtis: Reducing Pipeline Bubbles with Adaptive Pipeline Parallelism on Heterogeneous Models', proposes a pipeline-parallelism system that co-optimizes model partition, model placement, and workload scheduling for heterogeneous LLMs. It introduces a Pipeline Performance Model that estimates per-device runtime from profiled layer costs and simulated bubble/overlap times; a Pipeline Generator that iteratively tunes the three phases using this model; and a unified Pipeline Executor that emits computation/communication instructions with deadlock elimination and communication-computation overlap. Experiments on Gemma, DeepSeek, and Nemotron-H report average speedups of 1.34x, 1.42x, 1.34x, and 1.20x over S-1F1B, I-1F1B, ZB, and Mist respectively, with up to 2.14x over I-1F1B. However, the arXiv-level abstract provided for this submission describes a different system, 'OctoPipe', with different speedup numbers (1.15-1.44x), and no part of the body connects OctoPipe to AdaPtis. The central technical claim is the co-optimization benefit, but the performance-model underpinning is only partially specified and validated in a narrow regime.

Significance. If the reported results are reproducible, the paper's contribution is meaningful for the systems community: it addresses a real gap by co-optimizing three pipeline phases rather than one, and the instruction-based executor is a reasonable design for supporting irregular schedules. The evaluation covers multiple heterogeneous model families and GPU scales, and the core claim is falsifiable. However, the paper ships no code, no raw data, and no error bars, which limits verification. The correctness of the tuning mechanism depends on a performance model whose formal specification and validation coverage are incomplete. The circularity concern raised in the stress-test note does not land: the model uses profiled per-layer costs and simulation, not fitted speedups, so it is not circular in the sense of defining the output as a function of the measured result. The decisive issues are identity inconsistency, under-specified model internals, and insufficient fidelity evidence outside the validated configuration.

major comments (4)
  1. [Abstract / Title] The manuscript is internally inconsistent about what system it presents. The title of the provided submission and the opening abstract describe 'OctoPipe' with claimed speedups of 1.15-1.44x over state-of-the-art pipeline approaches, while the body title, body abstract, all sections, and all experiments describe 'AdaPtis' with average speedups of 1.34x/1.42x/1.34x/1.20x over S-1F1B/I-1F1B/ZB/Mist and up to 2.14x over I-1F1B. No sentence connects the two names. A reader cannot tell whether the paper is evaluating the system announced in the abstract. This must be reconciled before the paper can be assessed.
  2. [Section 4.2, Algorithm 1] The core performance equation is under-specified. Algorithm 1 Step 3 computes T_d = C_d + BubbleTime(d) - OverlapTime(d), but BubbleTime(d) and OverlapTime(d) are never defined by formulas, pseudocode, or a precise simulation procedure; Section 4.2 only says the model 'simulates the execution behavior' and 'identifies when and where device idle time occurs.' Since the Pipeline Generator makes tuning decisions based on differences in these terms, the method is not reproducible as written. Please provide a formal definition of the simulation, including how per-device execution order, communication events, and overlap are computed.
  3. [Section 5.5, Figure 12] The fidelity validation of the Pipeline Performance Model is too narrow to anchor the E2E claims. Figure 12 and the accompanying text validate only Nemotron-H models at SeqLen=4K, report normalized (relative to S-1F1B) throughput rather than absolute predictions, and give no error bars or per-configuration variance. Yet Figures 8-9 claim speedups for Gemma, DeepSeek, and Nemotron-H at SeqLen=2K/4K and up to 32K. The surrogate's accuracy in those regimes is assumed, not shown. A tuner guided by a model that is inaccurate by more than the difference between candidate pipelines could select suboptimal configurations; please add fidelity measurements, or a sensitivity analysis showing that the reported speedups are robust to model error, for the other model families and sequence lengths.
  4. [Section 5.7, Figures 14-15] The scaling claims are mislabeled or miscalculated. The text says AdaPtis achieves a 'scaling efficiency of 534%' when scaling from 8 to 128 GPUs. Efficiency cannot exceed 100%: with 16x more GPUs, a 5.34x throughput increase corresponds to 33% strong-scaling efficiency, not 534%. If the intended quantity is a speedup factor relative to AdaPtis on 8 GPUs, it should be stated as speedup, not efficiency. The same issue applies to the weak-scaling claim of 519%. Please correct the terminology and provide the actual throughput values or a clear derivation.
minor comments (4)
  1. [Section 5.5] The fidelity numbers are reported inconsistently: the text states an average prediction error of 2.12% and then lists maximum errors that include '2.12% for ZB'. The same value appears twice, and no maximum is given for S-1F1B. Please clarify which number is which and report errors for all baselines.
  2. [Figures 8-15] All throughput plots are normalized and single measurements, with no absolute throughput values, standard deviations, or run counts. This makes it difficult to assess whether the reported speedups exceed run-to-run variance. Please include absolute numbers or an artifact with raw data for verification.
  3. [Figure 1] The bubble-ratio measurements in Figure 1 have no methodology description. Please state how bubble ratio is computed from traces, and whether the values are from real executions or from the simulator.
  4. [Figures 14-15] The model name is misspelled as 'Nemontron-H' in the captions of Figures 14 and 15. It should be 'Nemotron-H'.

Circularity Check

0 steps flagged

No significant circularity; the reported speedups are measured real throughput differences, and the performance model is validated independently rather than fitted to the target results.

full rationale

The claimed derivation chain is: profile per-layer computation/memory costs; aggregate them per stage/device; simulate each workload-scheduling policy to obtain BubbleTime(d) and OverlapTime(d); compute T_d = C_d + BubbleTime(d) - OverlapTime(d); and minimize max_d T_d. The output of this chain is a configuration, not the reported speedup. The end-to-end speedups in Figures 8-9 are measured real throughput ratios (normalized to S-1F1B) on H800 GPUs, so the final claim is not defined in terms of the model's outputs. The model is calibrated to profiled hardware costs and validated in Section 5.5 against actual throughput for Nemotron-H at SeqLen=4K (2.12% average error), which is an independent check rather than a fit to the reported speedups. There is no equation in which a predicted quantity is the fitted value of an input, and no load-bearing self-citation: reference [4] (SPPO) appears only in Related Works as a pointer and is not used to justify the co-optimization claim. The OctoPipe/AdaPtis naming mismatch is a presentational/attribution inconsistency, not a circular step. Under-specification of BubbleTime/OverlapTime in Algorithm 1 is a correctness/validation concern, not circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests on a profiled-cost simulator that assumes linear additivity of per-layer costs and clean subtractability of communication overlap. The tuner's stopping thresholds are hand-set but unspecified. No new entities are postulated.

free parameters (4)
  • Bubble-difference stopping threshold
    Section 4.3 Model Partition Tuning stops when the difference in BubbleTime(d) is smaller than the maximum C_s; the exact threshold is not quantified.
  • Grouped permutation size
    Section 4.3 uses grouped permutations that permute all layers on a stage at once; the grouping size is a design choice not specified.
  • Memory margin for OOM avoidance
    Section 4.3 advances B and W to free memory when potential OOM time is identified; the safety margin is not specified.
  • Profiled per-layer computation and memory costs
    Inputs to the performance model (Algorithm 1); measured on the testbed, but the profile stability across runs and the exact measurement protocol are not given.
axioms (4)
  • domain assumption Per-layer profiled costs are representative of steady-state execution and unchanged by pipeline placement
    Pipeline Performance Model sums profiled layer costs (Algorithm 1, Step 1) and never accounts for dynamic effects such as L2 locality, kernel launch order, or interference between stages on the same device.
  • domain assumption Communication time is additive with computed overlap
    Algorithm 1 computes T_d = C_d + BubbleTime(d) - OverlapTime(d) assuming overlap is subtractable without loss; no correction for partial overlap or contention.
  • domain assumption The simulated event ordering (Step 3) captures real GPU execution order
    Used for BubbleTime(d) and OverlapTime(d); validated on Nemotron-H only (Section 5.5), not across all models.
  • ad hoc to paper The baseline set {S-1F1B, I-1F1B, Hanayo, ZB, Mist} is sufficient seed for the optimum
    Section 4.3 narrows the search to these baseline pipelines; the optimum of the joint space may lie outside this seed set.

pith-pipeline@v1.3.0-alltime-deepseek · 19105 in / 12303 out tokens · 271561 ms · 2026-08-04T14:37:48.788661+00:00 · methodology

0 comments
read the original abstract

Pipeline parallelism is widely used to train large language models (LLMs). However, increasing heterogeneity in model architectures exacerbates pipeline bubbles, thereby reducing training efficiency. Prior approaches typically optimize a single phase of the pipeline schedule (i.e., partitioning, placement, or scheduling), leaving substantial pipeline bubbles. While promising, co-optimization poses three key challenges: (1) complex performance modeling, (2) a combinatorial search space, and (3) irregular execution orders. To address these challenges, we propose OctoPipe, a pipeline parallelism system to jointly optimize partitioning, placement, and scheduling. First, we build a graph-based pipeline simulator to model heterogeneous pipeline execution for co-optimization. Second, on top of the simulator, we develop an iterative bubble-aware tuner to efficiently explore the combinatorial search space. Third, we implement a unified pipeline executor that dynamically orchestrates computation and communication to support irregular execution orders without deadlocks while maximizing communication-computation overlap. Experiments show that OctoPipe achieves 1.15--1.44x throughput improvement over the state-of-the-art pipeline parallelism approaches across various models and GPU cluster scales.

Figures

Figures reproduced from arXiv: 2509.23722 by Dahua Lin, Jiaxing Li, Jihu Guo, Peng Sun, Tenghui Ma, Wei Gao, Xun Chen, Yuyang Jin, Zhisheng Ye.

Figure 1
Figure 1. Figure 1: Bubble ratios of pipeline parallelism methods on different models. The model and training configurations are set to 𝐿 = 32, 𝑃 = 4,𝑇 = 2,𝐺 = 16, 𝑛𝑚𝑏 = 16 on 8 GPUs. 32K). Beyond vocabulary size, DeepSeek [30] integrates both Feed-Forward Network (FFN) and Mixture-of-Experts (MoE) layers [20], while Nemotron-H [2] combines Self-Attention (SA) [55] with Mamba [10], leading to even greater archi￾tectural heter… view at source ↗
Figure 2
Figure 2. Figure 2: Illustrations of Model Partition, Model Placement, and Workload Scheduling in Pipeline Parallelism. in a pipelined manner across all stages to improve device utilization. We next introduce these three phases of PP. 2.2 Model Partition At the beginning of pipeline parallelism, the model is parti￾tioned into multiple stages. A common strategy is to evenly allocate transformer layers across stages, with the i… view at source ↗
Figure 4
Figure 4. Figure 4: [Motivation]. The vast search space calls for an efficient pipeline generation method. to tune the model partition and mitigate the computational imbalance among devices, achieving a 1.49× speedup over the baseline. Finally, we refine model placement to make each computation finer-grained and apply workload sched￾uling optimization on the new model partition and model placement. These optimizations substan… view at source ↗
Figure 3
Figure 3. Figure 3: [Motivation]. Illustration of co-optimizing work￾load scheduling, model partition, and model placement for ac￾celerating training on a heterogeneous model with a large vo￾cabulary size (e.g. Gemma [52]) with 𝐿 = 32, 𝑃 = 4, 𝑛𝑚𝑏 = 4. the necessity of adaptive pipeline parallelism to mitigate bubbles and improve training efficiency. 3.2 Demand for Adaptive Pipeline Parallelism Reducing pipeline bubbles in het… view at source ↗
Figure 5
Figure 5. Figure 5: AdaPtis system design overview [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Illustrations of repeat tuning (a) model partition, (b) model placement, and (c) workload scheduling. devices, placing adjacent layers on different devices. This per￾mutation increases the effective number of pipeline stages, resulting in more fine-grained computation tasks. Second, Pipeline Generator performs workload scheduling on the permuted model placement while keeping the model par￾tition fixed. If … view at source ↗
Figure 7
Figure 7. Figure 7: Illustration of pipeline execution instruction gen￾eration and communication optimizations. scheduled immediately, communication and computation are forced to proceed sequentially. In contrast, by first executing other independent computations and delaying the dependent ones, communication can be overlapped with computation. This overlap-aware policy not only increases OverlapTime(𝑑) but also reduces Bubbl… view at source ↗
Figure 8
Figure 8. Figure 8: End-to-end training throughput of different PP methods on various model types and model sizes with input sequence length = (2K, 4K). The numbers above the bars indicate the normalized speedup over S-1F1B [47]. 1K 2K 4K 8K 16K 32K Sequence Length 2 4 6 8 10 Throughput (Normalized) S-1F1B I-1F1B ZB Mist AdaPtis [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Throughput of AdaPtis, Mist, ZB, I-1F1B, and S-1F1B on Nemotron-H (Large) with 𝑃 = 8,𝑇 = 4,𝐺 = 64, 𝑛𝑚𝑏 = 64 across various input sequence lengths. Baseline +1 +2 +3 +1+2 +1+3 +2+3+1+2+3 0.6 0.7 0.8 0.9 1.0 1.1 Throughput (Normalized) 76% 72% 78% 88% 92% 95% 96% 100% 73% 75% 74% 84% 89% 95% 91% 100% 75% 63% 76% 84% 70% 92% 92% 100% Gemma DeepSeek Nemotron-H [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Ablation study of pipeline co-optimization with ① adaptive model placement, ② adaptive workload sched￾uling, and ③ adaptive model partition across models. Moreover, [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
Figure 14
Figure 14. Figure 14: Strong scaling experimental results of training Nemontron-H (Large) with 𝑆𝑒𝑞𝑙𝑒𝑛 = 4K on 128 GPUs. 8 GPUs 16 GPUs 32 GPUs 64 GPUs 128 GPUs 0 2 4 6 8 10 Throughput (Normalized) 1.00x 1.00x 1.00x 1.00x 1.00x 0.89x 0.79x 0.70x 0.75x 0.74x 1.01x 1.03x 0.93x 1.05x 1.03x 1.06x 1.09x 1.12x 1.17x 1.14x 1.22x 1.33x 1.33x 1.40x S-1F1B 1.26x I-1F1B ZB Mist AdaPtis [PITH_FULL_IMAGE:figures/full_fig_p010_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Weak scaling experimental results of training Nemontron-H (Large) with 𝑆𝑒𝑞𝑙𝑒𝑛 = 4K on 128 GPUs. 5.7 Scalability Experimental Results Strong Scaling. As shown in [PITH_FULL_IMAGE:figures/full_fig_p010_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining

    cs.DC 2026-04 unverdicted novelty 7.0

    NestPipe achieves up to 3.06x speedup and 94.07% scaling efficiency on 1,536 workers via dual-buffer inter-batch and frozen-window intra-batch pipelining that overlaps communication with computation.

  2. A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability

    cs.DC 2026-05 unverdicted novelty 6.0

    RRFP introduces a readiness-driven runtime for pipeline parallelism that uses schedules as hints and ready-set arbitration to improve utilization under runtime variability, reporting up to 2.77x speedup on multimodal ...

  3. ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism

    cs.DC 2026-05 unverdicted novelty 5.0

    ResiHP improves LLM training throughput by 1.04-4.39x under hardware failures by using a workload-aware execution time predictor to avoid false failure detections and a scheduler that dynamically changes parallelism g...

  4. ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism

    cs.DC 2026-05 unverdicted novelty 4.0

    ResiHP introduces a workload-aware failure detector and dynamic scheduler for hybrid-parallel LLM training that achieves 1.04-4.39x higher throughput than prior resilient systems under failures on a 256-GPU cluster.

Reference graph

Works this paper leans on

68 extracted references · 30 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, et al. 2025. Llama-nemotron: Efficient reasoning models. arXiv preprint arXiv:2505.00949(2025)

    arXiv 2025

  2. [2]

    Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Ren- duchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabalesh- warkar, et al. 2025. Nemotron-h: A family of accurate and efficient hybrid mamba-transformer models.arXiv preprint arXiv:2504.03624 (2025)

    Pith/arXiv arXiv 2025

  3. [3]

    Tom B Brown. 2020. Language models are few-shot learners.arXiv preprint arXiv:2005.14165(2020)

    Pith/arXiv arXiv 2020

  4. [4]

    Qiaoling Chen, Shenggui Li, Wei Gao, Peng Sun, Yonggang Wen, and Tianwei Zhang. 2025. SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloading. arXiv:2503.10377 [cs.DC]https://arxiv.org/abs/2503.10377

    Pith/arXiv arXiv 2025

  5. [5]

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost.arXiv preprint arXiv:1604.06174(2016)

    Pith/arXiv arXiv 2016

  6. [6]

    Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An efficient SMT solver. InInternational conference on Tools and Algorithms for the Con- struction and Analysis of Systems. Springer, 337–340

    2008

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Ka- dian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

    Pith/arXiv arXiv 2024

  8. [8]

    Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, et al . 2021. DAPPLE: A pipelined data parallel approach for training large models. InProceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 431–445

    2021

  9. [9]

    Hao Ge, Junda Feng, Qi Huang, Fangcheng Fu, Xiaonan Nie, Lei Zuo, Haibin Lin, Bin Cui, and Xin Liu. 2025. ByteScale: Communication- Efficient Scaling of LLM Training with a 2048K Context Length on 16384 GPUs. InProceedings of the ACM SIGCOMM 2025 Conference. 963–978

    2025

  10. [10]

    Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752(2023)

    Pith/arXiv arXiv 2023

  11. [11]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforce- ment learning.arXiv preprint arXiv:2501.12948(2025)

    Pith/arXiv arXiv 2025

  12. [12]

    Gurobi Optimization, LLC. 2024. Gurobi Optimizer Reference Manual. https://www.gurobi.com

    2024

  13. [13]

    Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. 2018. Pipedream: Fast and efficient pipeline parallel dnn training.arXiv preprint arXiv:1806.03377(2018)

    Pith/arXiv arXiv 2018

  14. [14]

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism.Advances in neural information processing systems32 (2019)

    2019

  15. [15]

    Chenyu Jiang, Zhen Jia, Shuai Zheng, Yida Wang, and Chuan Wu. 2024. DynaPipe: Optimizing multi-task training through dynamic pipelines. InProceedings of the Nineteenth European Conference on Computer Systems. 542–559

    2024

  16. [16]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Ben- jamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361(2020)

    Pith/arXiv arXiv 2020

  17. [17]

    Taebum Kim, Hyoungjoo Kim, Gyeong-In Yu, and Byung-Gon Chun

  18. [18]

    Zhiquan Lai, Shengwei Li, Xudong Tang, Keshi Ge, Weijie Liu, Yabo Duan, Linbo Qiao, and Dongsheng Li. 2023. Merak: An efficient dis- tributed dnn training framework with automated 3d parallelism for giant foundation models.IEEE Transactions on Parallel and Distributed Systems34, 5 (2023), 1466–1478

    2023

  19. [19]

    Joel Lamy-Poirier. 2023. Breadth-first pipeline parallelism.Proceedings of Machine Learning and Systems5 (2023), 48–67

    2023

  20. [20]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668 (2020)

    Pith/arXiv arXiv 2020

  21. [21]

    Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. 2025. Minimax-01: Scaling foundation models with lightning attention.arXiv 11 preprint arXiv:2501.08313(2025)

    Pith/arXiv arXiv 2025

  22. [22]

    Dacheng Li, Hongyi Wang, Eric Xing, and Hao Zhang. 2022. Amp: Automatically finding model parallel strategies with heterogeneity awareness.Advances in Neural Information Processing Systems35 (2022), 6630–6639

    2022

  23. [23]

    Shigang Li and Torsten Hoefler. 2021. Chimera: efficiently training large-scale neural networks with bidirectional pipelines. InProceed- ings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1–14

    2021

  24. [24]

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. 2020. Pytorch distributed: Experiences on accelerating data parallel training.arXiv preprint arXiv:2006.15704(2020)

    Pith/arXiv arXiv 2020

  25. [25]

    Zhouyang Li, Yuliang Liu, Wei Zhang, Tailing Yuan, Bin Chen, Chengru Song, and Di Zhang. 2025. SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training.arXiv preprint arXiv:2504.14519(2025)

    Pith/arXiv arXiv 2025

  26. [26]

    Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. 2021. Terapipe: Token-level pipeline parallelism for training large-scale language models. InInternational Conference on Machine Learning. PMLR, 6543–6552

    2021

  27. [27]

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avshalom Manevich, Nir Ratner, Noam Rozen, Erez Schwartz, Mor Zusman, and Yoav Shoham. 2024. Jamba: A Hybrid Transformer-Mamba ...

    Pith/arXiv arXiv 2024

  28. [28]

    Zhiqi Lin, Youshan Miao, Guanbin Xu, Cheng Li, Olli Saarikivi, Saeed Maleki, and Fan Yang. 2024. Tessel: Boosting distributed execution of large dnn models via flexible schedule search. In2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 803–816

    2024

  29. [29]

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434(2024)

    Pith/arXiv arXiv 2024

  30. [30]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al

  31. [31]

    Guodong Liu, Youshan Miao, Zhiqi Lin, Xiaoxiang Shi, Saeed Maleki, Fan Yang, Yungang Bao, and Sa Wang. 2024. Aceso: Efficient Parallel DNN Training through Iterative Bottleneck Alleviation. InProceedings of the Nineteenth European Conference on Computer Systems. 163–181

    2024

  32. [32]

    Weijian Liu, Mingzhen Li, Guangming Tan, and Weile Jia. 2025. Mario: Near Zero-cost Activation Checkpointing in Pipeline Parallelism. In Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming. 197–211

    2025

  33. [33]

    Ziming Liu, Shenggan Cheng, Haotian Zhou, and Yang You. 2023. Hanayo: Harnessing wave-like pipeline parallelism for enhanced large model training efficiency. InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1–13

    2023

  34. [34]

    Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. 2022. Galvatron: Efficient transformer training over multiple gpus using automatic parallelism.arXiv preprint arXiv:2211.13878(2022)

    Pith/arXiv arXiv 2022

  35. [35]

    MiniMax, :, Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, Chengjun Xiao, Chengyu Du, Chi Zhang, Chu Qiao, Chunhao Zhang, Chunhui Du, Congchao Guo, Da Chen, Deming Ding, Dianjun Sun, Dong Li, Enwei Jiao, Haigang Zhou, Haimo Zhang, Han Ding, Haohai Sun, Haoyu Feng, Huaiguang Cai, Haichao Z...

    Pith/arXiv arXiv 2025

  36. [36]

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGres- ley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021. Efficient large-scale language model training on gpu clusters using megatron- lm. InProceedings of the International Conference for High Performance Computing, Netwo...

    2021

  37. [37]

    Hyungjun Oh, Junyeol Lee, Hyeongju Kim, and Jiwon Seo. 2022. Out- of-order backprop: An effective scheduling technique for deep learning. InProceedings of the Seventeenth European Conference on Computer Systems. 435–452

    2022

  38. [38]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al . 2019. Pytorch: An imperative style, high-performance deep learning library.Advances in neural informa- tion processing systems32 (2019)

    2019

  39. [39]

    Penghui Qi, Xinyi Wan, Nyamdavaa Amar, and Min Lin. 2024. Pipeline Parallelism with Controllable Memory.arXiv preprint arXiv:2405.15362 (2024)

    Pith/arXiv arXiv 2024

  40. [40]

    Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin. 2024. Zero Bubble (Almost) Pipeline Parallelism. InThe Twelfth International Con- ference on Learning Representations

    2024

  41. [41]

    Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. 2024. Various lengths, constant speed: Efficient language modeling with lightning attention.arXiv preprint arXiv:2405.17381 (2024)

    Pith/arXiv arXiv 2024

  42. [42]

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yux- iong He. 2022. Deepspeed-moe: Advancing mixture-of-experts infer- ence and training to power next-generation ai scale. InInternational conference on machine learning. PMLR, 18332–18346

    2022

  43. [43]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He

  44. [44]

    Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. InProceedings of the international conference for high performance computing, networking, storage and analysis. 1–14

    2021

  45. [45]

    2021.{Zero-offload}: Democratizing{billion-scale} model training

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021.{Zero-offload}: Democratizing{billion-scale} model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). 551–564. 12

    2021

  46. [46]

    Alexander Sergeev and Mike Del Balso. 2018. Horovod: fast and easy distributed deep learning in TensorFlow.arXiv preprint arXiv:1802.05799(2018)

    Pith/arXiv arXiv 2018

  47. [47]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi- billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053(2019)

    Pith/arXiv arXiv 2019

  48. [48]

    Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2020. En- ergy and policy considerations for modern deep learning research. InProceedings of the AAAI conference on artificial intelligence, Vol. 34. 13693–13696

    2020

  49. [49]

    Ao Sun, Weilin Zhao, Xu Han, Cheng Yang, Xinrong Zhang, Zhiyuan Liu, Chuan Shi, and Maosong Sun. 2024. Seq1f1b: Efficient sequence- level pipeline parallelism for large language model training.arXiv preprint arXiv:2406.03488(2024)

    Pith/arXiv arXiv 2024

  50. [50]

    Zhenbo Sun, Huanqi Cao, Yuanwei Wang, Guanyu Feng, Shengqi Chen, Haojie Wang, and Wenguang Chen. 2024. AdaPipe: Optimizing Pipeline Parallelism with Adaptive Recomputation and Partitioning. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. 86–100

    2024

  51. [51]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530 (2024)

    Pith/arXiv arXiv 2024

  52. [52]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexan- dre Ramé, Morgane Rivière, et al . 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786(2025)

    Pith/arXiv arXiv 2025

  53. [53]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Alma- hairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288(2023)

    Pith/arXiv arXiv 2023

  54. [54]

    Taegeon Um, Byungsoo Oh, Minyoung Kang, Woo-Yeon Lee, Goeun Kim, Dongseob Kim, Youngtaek Kim, Mohd Muzzammil, and Myeong- jae Jeon. 2024. Metis: Fast Automatic Distributed Training on Het- erogeneous{GPUs}. In2024 USENIX Annual Technical Conference (USENIX ATC 24). 563–578

    2024

  55. [55]

    A Vaswani. 2017. Attention is all you need.Advances in Neural Information Processing Systems(2017)

    2017

  56. [56]

    Yujie Wang, Shiju Wang, Shenhan Zhu, Fangcheng Fu, Xinyi Liu, Xue- feng Xiao, Huixia Li, Jiashi Li, Faming Wu, and Bin Cui. 2025. Flexsp: Accelerating large language model training via flexible sequence par- allelism. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Sys- tems, Volume 2...

    2025

  57. [57]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebas- tian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models.arXiv preprint arXiv:2206.07682(2022)

    Pith/arXiv arXiv 2022

  58. [58]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    Pith/arXiv arXiv 2025

  59. [59]

    Man Tsung Yeung, Penghui Qi, Min Lin, and Xinyi Wan. 2024. Balanc- ing Pipeline Parallelism with Vocabulary Parallelism.arXiv preprint arXiv:2411.05288(2024)

    Pith/arXiv arXiv 2024

  60. [60]

    Zili Zhang, Yinmin Zhong, Ranchen Ming, Hanpeng Hu, Jianjian Sun, Zheng Ge, Yibo Zhu, and Xin Jin. 2024. Disttrain: Addressing model and data heterogeneity with disaggregated training for multimodal large language models.arXiv preprint arXiv:2408.04275(2024)

    arXiv 2024

  61. [61]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al

  62. [62]

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. 2022. Alpa: Automating inter-and{Intra-Operator} parallelism for distributed deep learning. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 559–578

    2022

  63. [63]

    Zhanda Zhu, Christina Giannoula, Muralidhar Andoorveedu, Qidong Su, Karttikeya Mangalam, Bojian Zheng, and Gennady Pekhimenko

  64. [65]

    arXiv preprint arXiv:2304.11277(2023)

    Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277(2023)

    Pith/arXiv arXiv 2023

  65. [2020]

    InSC20: International Conference for High Performance Computing, Networking, Storage and Analysis

    Zero: Memory optimizations toward training trillion param- eter models. InSC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–16

  66. [2023]

    InInternational Conference on Machine Learning

    BPIPE: memory-balanced pipeline parallelism for training large language models. InInternational Conference on Machine Learning. PMLR, 16639–16653

  67. [2024]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437 (2024)

    Pith/arXiv arXiv 2024

  68. [2025]

    InProceedings of the Twentieth European Conference on Computer Systems

    Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization. InProceedings of the Twentieth European Conference on Computer Systems. 1298–1316. 13 Jihu Guo, Tenghui Ma, Wei Gao, Peng Sun, Jiaxing Li, Xun Chen, Yuyang Jin, Dahua Lin„