REVIEW 4 major objections 6 minor 35 references
A schedule algebra yields a static transpose that hides MLLM encoder work in LLM pipeline warmup bubbles, cutting step time without changing the loss.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-12 03:58 UTC pith:BKECUPVC
load-bearing objection Clean static idea (encoder into warmup bubbles) plus a small schedule algebra; the 2.70× is believable, the 1.21× production claim is under-controlled because the DistTrain baseline is reconstructed. the 4 major comments →
HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper shows that a compact schedule algebra is enough to derive a static heterogeneous schedule, called transpose, that relocates modality-encoder computation into the otherwise idle warmup bubbles of an LLM pipeline, removing encoder variance from the critical path without runtime rescheduling or loss of training correctness.
What carries the argument
The schedule algebra: a valid cut and list of skeletons (1f1b, gpipe, transpose) maps, via a single derive function, to placement, collectives, and a dependency order; from it the authors instantiate transpose (Replicated encoder + Sharded backbone) and prove a backward-footprint lemma that keeps the schedule correct whether the encoder is frozen or trained.
Load-bearing premise
The speedup rests on encoder work largely fitting inside each rank’s warmup slack; when it does not, residual spill stays on the critical path and the static owner map cannot remove it.
What would settle it
Measure end-to-end step time and training loss for the same MLLM under the paper’s transpose schedule versus a conventional pipelined-encoder baseline while deliberately increasing image resolution or tile count until per-rank encoder work exceeds measured warmup slack; if step-time gains vanish or loss diverges, the central claim fails.
If this is right
- MLLM training can keep a single static schedule that is invariant to modality mix and adds no per-iteration scheduling cost.
- Encoder-stage variance no longer needs to be chased by runtime search or data-dependent rebalancing when it fits inside pipeline bubbles.
- The same algebra can express classical 1F1B, GPipe, and interleaved VPP as points in one space, so new heterogeneous schedules become derivable rather than hand-crafted.
- On large NPU clusters the reported step-time reductions (up to 2.70× small-scale, 1.21× at 512 cards) raise effective MFU for encoder-heavy MLLM jobs without changing the loss.
Where Pith is reading between the lines
- If residual spill remains the dominant limit, a loss-preserving metadata-guided reordering of microbatches could close most of the remaining gap without abandoning static schedules.
- The same algebra could be extended to sink-side generators by adding a trailing sharded region rather than forcing them into warmup bubbles they cannot use.
- Cost-model predictions of spill versus scale could guide automatic choice between transpose and conventional placement before a large run starts.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Mpipe, a schedule algebra that maps a compact specification (a cut of the model into depth regions plus a list of per-region skeletons) to concrete runtime placement, collectives, and execution order. From this algebra it derives transpose: a Replicated encoder skeleton that runs modality-encoder work inside LLM pipeline warmup bubbles and gathers outputs into the first backbone stage via a Replicated→Sharded seam. A backward-footprint lemma and schedule-invariance corollary state that backward events are gated by trainability, not by the schedule, so one transpose schedule covers frozen or trained encoders. On Ascend 910C, the authors report 2.70× end-to-end step-time reduction on an 8-device Qwen3.5 MLLM workload versus a Megatron-style baseline, and 1.21× on a 512-card ViT+DeepSeek workload versus a reconstructed DistTrain-like baseline, with the claim that training loss is unchanged.
Significance. If the algebra and transpose schedule hold as stated, the work offers a clean, static alternative to per-iteration profiling or runtime load balancing for the source-side encoder variance that depresses MLLM MFU. The formalization (layouts, seam table, derive map, backward-footprint lemma, schedule-invariance corollary) is a genuine contribution relative to ad-hoc pipeline schedule descriptions, and the cost model that predicts spill as max(0, E_r − w_r) is falsifiable and useful. Demonstrated wall-clock gains on both small and 512-card Ascend clusters, without runtime scheduling overhead, would matter for production MLLM training. Strengths include an explicit composable algebra rather than a one-off schedule, a clear trainability-gated correctness argument, and end-to-end measurements rather than microbenchmarks alone.
major comments (4)
- §5 Experiment A / Table 3: The production-scale 1.21× claim is the load-bearing industrial result, yet the baseline is explicitly a reconstructed “DistTrain-like” system (one encoder pipeline stage with a customized parallel strategy; backbone under conventional 5D) because DistTrain is unreleased. Without validation that this reconstruction matches DistTrain’s resource split, placement, or published numbers, and without an ablation that isolates the transpose gather/ByMicrobatch owner from that baseline, the 1.21× cannot be securely attributed to the algebra-derived schedule. A stronger evaluation would report (i) the exact baseline configuration, (ii) a pure 1F1B/Megatron-style PP baseline on the same 512-card setup, and (iii) an ablation turning transpose on/off under fixed cut and hardware.
- §1, §5 Analysis, and abstract: The paper repeatedly asserts “no change to the training loss” / “loss-preserving,” but neither loss curves, final loss values, nor any numerical comparison of baseline vs Mpipe loss appear in the manuscript. For a systems paper that relocates computation and inserts Gather/Scatter seams, loss invariance is a central correctness claim and should be shown (even a short training-loss overlay or a fixed-step loss table would suffice).
- §4.3 Cost Model and §5 Analysis: The cost model predicts that transpose hides min(E_r, w_r) and exposes max_r spill_r = max(0, E_r − w_r), and the authors use this to explain why Exp B (2.70×) exceeds Exp A (1.21×). However, the manuscript never reports measured E_r, w_r, spill, bubble fractions, or predicted vs measured step times. Without that check, the quantitative story remains unvalidated, and the weaker large-scale gain could equally be baseline quality or residual spill. Adding a short predicted-vs-measured table (or per-rank bubble/encoder timing) would make the central mechanism falsifiable rather than post-hoc.
- §5 overall: Empirical support is limited to two average step-time tables with no error bars, no multi-run variance, no MFU numbers matching Figure 1’s framing, and no sensitivity to modality mix (the invariance claim). At minimum, report step-time stddev over several steps/runs and one controlled sweep of encoder load (e.g., image resolution or image fraction) to show that the static schedule absorbs dynamic encoder work as claimed in §2–§4.
minor comments (6)
- Figure 2 is hard to parse: microbatch indices and stage bars are dense; a clearer legend distinguishing DataLoad / Encoder / LLM and marking the Gather seam would help.
- Table 1 units and setup (visual length fixed to 4× LLM sequence length of 8K) are useful but the FLOPs magnitudes (~10^18) look like full-run aggregates rather than per-layer; clarify the aggregation scope.
- §4.1 notation mixes skeleton names (1f1b, gpipe, transpose) with layout names (Sharded, Replicated); a short glossary box would reduce cognitive load.
- Related work cites Optimus, DIP, DistTrain, OrchMLLM, MegaScale-Omni appropriately; a one-row comparison table (static vs runtime, encoder placement, overhead) would sharpen the positioning in §3/§6.
- Typos / polish: “derivetranspose” spacing in the abstract/intro; “1f1b” vs “1F1B” capitalization is inconsistent; arXiv IDs and access dates for DualPipe/DualPipeV are fine but GitHub “accessed 2026-06-24” will age oddly in print.
- §4.1 Applicability mentions extensions (Hanayo fold, DualPipeV event split) left to future work; a single sentence on what is implemented in Hyper-Parallel today vs only formal would set expectations.
Circularity Check
No load-bearing circularity: algebra formalizes schedules and experiments measure wall-clock step times against baselines; minor self-reference to authors' prior SGL/Hyper-Parallel tooling is background, not a closed loop forcing the speedups.
full rationale
This is an empirical systems paper. The schedule algebra (§4.1) defines skeletons, layouts, seams, and a derive map from (cut, schedule, model) to placement/collectives/order; transpose is simply the Replicated skeleton placed on the encoder region so that Fwd(enc) runs inside warmup bubbles. That is definitional formalization of a static placement idea, not a prediction that reduces to its own inputs. The cost model (§4.3) weights the same order graph with roofline and α-β (SGL) costs and reads longest-path makespan; it is used only to explain when spill = max(0, E_r − w_r) is small, and is checked against measured step times rather than fitted and re-presented as prediction. End-to-end claims (Tables 3–4) are measured average step times on Ascend 910C (2.70× vs Megatron-style baseline; 1.21× vs a reconstructed DistTrain-like baseline) with asserted loss invariance. Self-citations (Hyper-Parallel implementation vehicle; Li & Hains 2012 SGL for the bridging cost model and scatter-gather language) supply tooling and a standard communication model; they do not supply a uniqueness theorem, fitted constant, or ansatz that forces the measured speedups. No self-definitional loop, no fitted-input-called-prediction, no uniqueness imported from authors, and no renaming of a known empirical pattern as a first-principles derivation. Residual concerns (reconstructed baseline fidelity, E_r ≤ w_r assumption) are correctness/attribution risks, not circularity. Score 1 only for the minor non-load-bearing self-reference to SGL/Hyper-Parallel.
Axiom & Free-Parameter Ledger
free parameters (3)
- ByMicrobatch owner assignment (encoder microbatch→rank map)
- Pipeline cut and skeleton list σ (e.g. ⟨transpose,1f1b⟩) =
⟨transpose,1f1b⟩ for reported runs
- Roofline peak rate F and HBM bandwidth B; collective α,β
axioms (5)
- ad hoc to paper Pipeline schedules can be factored into per-depth-region skeletons with layouts Replicated or Sharded, and adjacent layouts determine seam collectives (Table 2).
- domain assumption Backward footprint is gated only by region trainability and cut, not by schedule skeleton (Lemma 1 / Corollary 1).
- domain assumption Step makespan equals longest path in the weighted order graph under roofline compute and α-β collective costs; pipeline overlap is captured by path max rather than stage sum.
- domain assumption Dominant MLLM training inefficiency addressed here is source-side encoder workload variance exposable as a pipeline stage; sequence-length variance and output generators are out of scope for transpose.
- standard math Standard pipeline, tensor, and data parallelism semantics (1F1B/GPipe/VPP, gather/scatter, DP reduce) as in Megatron-style systems.
invented entities (2)
-
Schedule algebra (skeleton list + derive → placement × collectives × order)
no independent evidence
-
Transpose skeleton (Replicated encoder in warmup/cooldown bubbles with gather into first LLM stage)
no independent evidence
read the original abstract
Modern AI applications have expanded beyond text-only interaction into a wide range of multimodal scenarios, making multimodal large language models (MLLMs) crucial for both research and industry. However, compared with traditional decoder-only LLM training, large-scale MLLM training often shows much lower MFU. We analyze the key pain points in MLLM training and introduce Mpipe, which uses a schedule algebra to derive concrete runtime behavior from a compact schedule specification. From this algebra, Mpipe derives transpose, a multimodal-aware heterogeneous parallel schedule that remaps modality-encoder computation into otherwise idle pipeline regions. On Ascend 910C NPU clusters, Mpipe achieves 2.70x speedup in a small-scale setting and 1.21x speedup in a 512-card large-scale setting.
Figures
Reference graph
Works this paper leans on
-
[1]
21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) , pages =
Jun Huang and Zhen Zhang and Shuai Zheng and Feng Qin and Yida Wang , title =. 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) , pages =. 2024 , publisher =
2024
-
[2]
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages =
Qiying Yu and Quan Sun and Xiaosong Zhang and Yufeng Cui and Fan Zhang and Yue Cao and Xinlong Wang and Jingjing Liu , title =. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages =. 2024 , address =
2024
-
[3]
Gomez and Lukasz Kaiser and Illia Polosukhin , title =
Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin , title =. Advances in Neural Information Processing Systems , volume =. 2017 , publisher =
2017
-
[4]
International Conference on Learning Representations , year =
Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby , title =. International Conference on Learning Representations , year =
-
[5]
Medical Image Computing and Computer-Assisted Intervention , pages =
Olaf Ronneberger and Philipp Fischer and Thomas Brox , title =. Medical Image Computing and Computer-Assisted Intervention , pages =. 2015 , publisher =
2015
-
[6]
International Conference on Computer Vision , pages =
William Peebles and Saining Xie , title =. International Conference on Computer Vision , pages =
-
[7]
Corrado and Rajat Monga and Kai Chen and Matthieu Devin and Mark Mao and Marc'Aurelio Ranzato and Andrew Senior and Paul Tucker and Ke Yang and Quoc V
Jeffrey Dean and Greg S. Corrado and Rajat Monga and Kai Chen and Matthieu Devin and Mark Mao and Marc'Aurelio Ranzato and Andrew Senior and Paul Tucker and Ke Yang and Quoc V. Le and Andrew Y. Ng , title =. Advances in Neural Information Processing Systems , volume =. 2012 , publisher =
2012
-
[8]
arXiv preprint arXiv:2307.09288 , year =
Hugo Touvron and Louis Martin and Kevin Stone and Peter Albert and Amjad Almahairi and Yasmine Babaei and Nikolay Bashlykov and Soumya Batra and Prajjwal Bhargava and Shruti Bhosale and others , title =. arXiv preprint arXiv:2307.09288 , year =
-
[9]
Journal of Machine Learning Research , volume =
William Fedus and Barret Zoph and Noam Shazeer , title =. Journal of Machine Learning Research , volume =. 2022 , url =
2022
-
[10]
Proceedings of the VLDB Endowment , volume =
Shen Li and Yanli Zhao and Rohan Varma and Omkar Salpekar and Pieter Noordhuis and Teng Li and Adam Paszke and Jeff Smith and Brian Vaughan and Pritam Damania and Soumith Chintala , title =. Proceedings of the VLDB Endowment , volume =. 2020 , doi =
2020
-
[11]
Proceedings of Machine Learning and Systems , volume =
Vijay Anand Korthikanti and Jared Casper and Sangkug Lym and Lawrence McAfee and Michael Andersch and Mohammad Shoeybi and Bryan Catanzaro , title =. Proceedings of Machine Learning and Systems , volume =
-
[12]
International Conference on Learning Representations , year =
Hao Liu and Matei Zaharia and Pieter Abbeel , title =. International Conference on Learning Representations , year =
-
[13]
Valiant , title =
Leslie G. Valiant , title =. Communications of the ACM , volume =. 1990 , doi =
1990
-
[14]
Proceedings of the 41st International Conference on Machine Learning , articleno =
Shengqiong Wu and Hao Fei and Leigang Qu and Wei Ji and Tat-Seng Chua , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , series =
2024
-
[15]
Proceedings of the ACM SIGCOMM 2025 Conference , pages =
Zili Zhang and Yinmin Zhong and Yimin Jiang and Hanpeng Hu and Jianjian Sun and Zheng Ge and Yibo Zhu and Daxin Jiang and Xin Jin , title =. Proceedings of the ACM SIGCOMM 2025 Conference , pages =. 2025 , publisher =. doi:10.1145/3718958.3750472 , url =
-
[16]
Proceedings of the 2025 USENIX Conference on USENIX Annual Technical Conference , articleno =
Weiqi Feng and Yangrui Chen and Shaoyu Wang and Yanghua Peng and Haibin Lin and Minlan Yu , title =. Proceedings of the 2025 USENIX Conference on USENIX Annual Technical Conference , articleno =. 2025 , series =
2025
-
[17]
Zhenliang Xue and Hanpeng Hu and Xing Chen and Yimin Jiang and Yixin Song and Zeyu Mi and Yibo Zhu and Daxin Jiang and Yubin Xia and Haibo Chen , title =. Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , pages =. 2026 , series =. doi:10.1145/3779212.3790154 , url =
-
[18]
Proceedings of the International Conference on Machine Learning , pages =
Tianqi Chen and Bing Xu and Chiyuan Zhang and Carlos Guestrin , title =. Proceedings of the International Conference on Machine Learning , pages =. 2016 , publisher =
2016
-
[19]
Le and Yonghui Wu and Zhifeng Chen , title =
Yanping Huang and Youlong Cheng and Ankur Bapna and Orhan Firat and Mia Xu Chen and Dehao Chen and HyoukJoong Lee and Jiquan Ngiam and Quoc V. Le and Yonghui Wu and Zhifeng Chen , title =. Proceedings of the 33rd International Conference on Neural Information Processing Systems , articleno =. 2019 , publisher =
2019
-
[20]
International Journal of High Performance Computing and Networking (IJHPCN) , volume=
Li, Chong and Hains, Ga. International Journal of High Performance Computing and Networking (IJHPCN) , volume=
-
[21]
Bulk Synchronous Parallel
Loulergue, Fr. Bulk Synchronous Parallel. International Conference on Computational Science (ICCS) , series =. 2005 , publisher =
2005
-
[22]
Advanced Parallel Processing Technologies (APPT) , series =
Javed, Noman and Loulergue, Fr. Advanced Parallel Processing Technologies (APPT) , series =. 2009 , publisher =
2009
-
[23]
, title =
Cole, Murray I. , title =
-
[24]
Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '23) , year =
Liu, Ziming and Cheng, Shenggan and Zhou, Haotian and You, Yang , title =. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '23) , year =
-
[25]
Advances in Neural Information Processing Systems (NeurIPS) , year =
Qi, Penghui and Wan, Xinyi and Amar, Nyamdavaa and Lin, Min , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[26]
ACM Transactions on Programming Languages and Systems (TOPLAS) , volume =
Gorlatch, Sergei , title =. ACM Transactions on Programming Languages and Systems (TOPLAS) , volume =. 2004 , publisher =
2004
-
[27]
Proceedings of the 21st European Conference on Computer Systems (EuroSys '26) , year =
Xue, Chunyu and Chen, Yangrui and Jiang, Jianyu and Zheng, Ningxin and Feng, Junda and Chen, Jingji and Zhao, Shixiong and Yan, Shen and Lin, Yi and Shi, Lei and Wang, Zanbo and Luo, Lishu and Wu, Faming and Lin, Haibin and Liu, Xin and Peng, Yanghua and Chen, Quan , title =. Proceedings of the 21st European Conference on Computer Systems (EuroSys '26) , year =
-
[28]
arXiv preprint arXiv:2407.21783 , year =
Aaron Grattafiori and Abhimanyu Dubey and Abhinav Jauhri and Abhinav Pandey and Abhishek Kadian and others , title =. arXiv preprint arXiv:2407.21783 , year =
-
[29]
arXiv preprint arXiv:2506.12708 , year =
Pengfei Zuo and Huimin Lin and Junbo Deng and Nan Zou and Xingkun Yang and Yingyu Diao and Weifeng Gao and Ke Xu and Zhangyu Chen and Shirui Lu and others , title =. arXiv preprint arXiv:2506.12708 , year =
-
[30]
arXiv preprint arXiv:2409.12191 , year =
Peng Wang and Shuai Bai and Sinan Tan and Shijie Wang and Zhihao Fan and Jinze Bai and Keqin Chen and Xuejing Liu and Jialin Wang and Wenbin Ge and Yang Fan and Kai Dang and Mengfei Du and Xuancheng Ren and Rui Men and Dayiheng Liu and Chang Zhou and Jingren Zhou and Junyang Lin , title =. arXiv preprint arXiv:2409.12191 , year =
-
[31]
arXiv preprint arXiv:2604.15804 , year =
-
[32]
arXiv preprint arXiv:2412.19437 , year =
-
[33]
arXiv preprint arXiv:2503.23830 , year =
Yijie Zheng and Bangjun Xiao and Lei Shi and Xiaoyang Li and Faming Wu and Tianyu Li and Xuefeng Xiao and Yang Zhang and Yuxuan Wang and Shouda Liu , title =. arXiv preprint arXiv:2503.23830 , year =
-
[34]
arXiv preprint arXiv:1909.08053 , year =
Mohammad Shoeybi and Mostofa Patwary and Raul Puri and Patrick LeGresley and Jared Casper and Bryan Catanzaro , title =. arXiv preprint arXiv:1909.08053 , year =
Pith/arXiv arXiv 1909
-
[35]
Synchronization and Linearity: An Algebra for Discrete Event Systems , publisher =
Baccelli, Fran. Synchronization and Linearity: An Algebra for Discrete Event Systems , publisher =
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.