Pith. sign in

REVIEW 2 major objections 5 minor 60 references

On production superpods, reworking MoE All-to-All without global barriers cuts dispatch latency up to 52.4% and end-to-end TPOT up to 11.1%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 01:05 UTC pith:BRM2KPJO

load-bearing objection Solid production EPCL redesign for multi-tier superpods: real latency/TPOT gains on CM384, with the main soft spot being 512B-atomic portability rather than any internal flaw. the 2 major comments →

arxiv 2607.06202 v2 pith:BRM2KPJO submitted 2026-07-07 cs.DC cs.AIcs.NI

UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods

Yipeng Liu , Chang Liu , Si Shen , Jiaqi Zheng , Mingfan Li , Yuyang Yang , Guanhua Li , Yuquan Zhang
show 11 more authors
Yimeng Xu Zhongzhe Hu Zhiyuan Huang Qihang Duan Junsong Wang Wenkai Ling Baochuan Yang Xianzhi Yu Han Bao Yijie Chen Guihai Chen
This is my paper
classification cs.DC cs.AIcs.NI
keywords Mixture-of-ExpertsExpert parallelismAll-to-All communicationCommunication librarySuperpodData center networksBulk Synchronous ParallelToken scheduling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Mixture-of-Experts models move irregular tokens across many accelerators. On modern superpods the fabric is so fast that old Bulk Synchronous Parallel libraries waste that speed: they serialize interdependent phases behind global barriers, pay a large synchronization tax, and schedule tokens without regard to hop distance, so stragglers dominate. UBEP is a production communication library that decomposes the All-to-All into fine-grained tasks scheduled by data availability, embeds flags inside atomic data blocks so control messages disappear, and assigns tokens with a hop-aware hierarchical mapper. On a production CloudMatrix384 superpod with up to 256 NPU dies it cuts All-to-All latency by as much as 52.4% versus the stock library and improves Time Per Output Token by as much as 11.1% for models at DeepSeek-R1 scale. A sympathetic reader cares because the paper shows that, once raw bandwidth is no longer the limit, the remaining bottlenecks are software orchestration, and they can be removed without changing the model.

Core claim

The paper claims that three superpod-specific bottlenecks—BSP-style serialization of interdependent MoE phases, a synchronization tax that grows relative to high-bandwidth links, and topology-agnostic token scheduling—explain why existing expert-parallel libraries under-utilize modern fabrics, and that replacing them with dependency-driven kernel decomposition, Data-as-Flag atomic synchronization, and hierarchical hop-aware scheduling recovers most of that lost performance, yielding up to 52.4% lower All-to-All latency and 11.1% better end-to-end TPOT on a production multi-tier superpod.

What carries the argument

Data-as-Flag: a 512-byte atomic remote store that embeds the completion flag (or uses the payload itself as a checksum or sentinel) so receivers observe payload and readiness as one transaction, eliminating separate control messages and global barriers.

Load-bearing premise

The near-zero-overhead claim for Data-as-Flag rests on the fabric guaranteeing that a full 512-byte remote write is observed as a single atomic transaction; if atomicity is weaker or smaller, the design falls back to fences or lower-efficiency blocks and the synchronization tax returns.

What would settle it

On the same CM384 hardware and models, replace the 512-byte atomic path with ordinary stores plus explicit fences or acks and measure whether dispatch latency and TPOT still improve by the reported margins over the baseline library; if the gains vanish, the atomicity assumption is load-bearing.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • MoE inference stacks on multi-tier superpods can drop BSP barriers inside the All-to-All path without changing model semantics.
  • Token schedulers that ignore hop distance leave measurable straggler latency on hierarchical fabrics; hop-aware mapping removes it.
  • The same dependency-driven decomposition can be applied to the combine phase and, in principle, to other collective kernels that currently use global barriers.
  • End-to-end TPOT gains of roughly 10% are available from communication-library changes alone once the fabric is no longer bandwidth-bound.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If next-generation fabrics expose smaller atomic units, the same design would trade payload efficiency for correctness, suggesting hardware vendors should keep large atomic write sizes for sparse MoE traffic.
  • Attention-FFN disaggregation turns All-to-All into bipartite Many-to-Many traffic; the same data-availability scheduling should still hide latency without topology-specific rewrites.
  • The residual gap between operator-level and end-to-end gains points to framework and kernel-launch overheads as the next place to apply the same fine-grained dependency machinery.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents UBEP, a production EPCL for multi-tier superpods (evaluated on Huawei CM384 up to 256 NPU dies). It identifies three bottlenecks of BSP-style MoE All-to-All on low-latency UGAS fabrics—phase serialization, synchronization tax, and topology-agnostic token scheduling—and addresses them with (1) kernel decomposition that partitions AIVs into token-send vs. TokenCnt/CalCumSum groups and replaces SyncAll with point-to-point data signals (§3.2), (2) a hierarchical hop-aware token-level scheduler with a SIMD quaternary-search mapper that homogenizes one-hop/two-hop load (§3.3, Alg. 1), and (3) Data-as-Flag (TFF/DC/SP) that embeds or elides control via 512B atomic remote stores (§3.4). On CM384, UBEP reports up to 52.4% lower dispatch latency vs. CANN EP and up to 11.1% better end-to-end TPOT on Qwen3-30B, GLM-4.7, DeepSeek-R1, and DeepSeek-V3.2, with ablations, sensitivity sweeps, and appendices on AIV allocation modeling and NP-hardness of the scheduling ILP.

Significance. If the reported gains hold, the work is a solid systems contribution for multi-tier superpods (NVL72/576, CM384). It shows that once link latency falls into the hundreds of nanoseconds, BSP barriers and topology-oblivious token placement become first-order, and that fine-grained dependency-driven orchestration plus hop-aware scheduling can reclaim substantial bandwidth. Strengths include production-scale evaluation (256 dies), multi-model TPOT, operator phase breakdowns (Fig. 8), ablations of mapping and of TFF/DC/SP vs. stop-and-wait (Figs. 9–10), sensitivity to BS/ranks/experts/Top-k (Fig. 7, App. C), an explicit portability checklist (Table 5), and a formal NP-hardness reduction for the scheduling ILP (App. B). The result is engineering rather than theoretical, but it is concrete, measured against an independent baseline on the same fabric, and relevant to network systems for AI.

major comments (2)
  1. §3.4 and Table 5: The near-zero-overhead claim for Data-as-Flag (and thus much of the synchronization-tax elimination) rests on UB Fabric guaranteeing 512B atomic remote stores observed as a single transaction, plus (for SP) a reserved sentinel never appearing in real activations. The paper states the degradation path correctly, but the main results (Table 4, Fig. 8, 52.4% peak) are reported only under full atomicity. A short quantitative sensitivity—e.g., TFF with 64B/128B blocks, or forced DC/fence fallback—would show how much of the 52.4%/11.1% survives when the assumption is weakened, and would strengthen the portability argument already sketched in Table 5.
  2. §5.5 / Fig. 11: End-to-end TPOT gains (up to 11.1%) are substantially smaller than operator-level dispatch gains, and the text notes that MoE communication is ~50% of per-token latency but only ~20% of hardware execution time. The manuscript would be stronger if it quantified, for each model, the fraction of the TPOT reduction attributable to the three UBEP mechanisms versus residual framework/kernel-launch/memory-movement overhead. Without that attribution, the causal link from the three designs to the headline 11.1% remains only partially established.
minor comments (5)
  1. Table 1: Hybrid-EP is listed as ASP/Warp for 1-tier superpods; a one-sentence clarification of how UBEP’s core-level ASP differs from Hybrid-EP’s warp-level ASP (beyond hierarchy) would help readers place the contribution.
  2. Fig. 3(b) and §2.3 Insight 1: The claim that 24 AIVs already saturate bandwidth is central to the decomposition rationale; stating the measured saturation bandwidth and the token size used would make the figure self-contained.
  3. Algorithm 1: The quaternary-search description is clear, but a brief note on worst-case iteration count for the expert counts used in the evaluation (128–256) would help readers judge the 1 µs budget claim.
  4. §6 / Table 5: The portability checklist is valuable; adding a short sentence on whether NVLink/NVSHMEM or UALink currently expose comparable atomic remote-store granularity would make the discussion more actionable for non-Ascend readers.
  5. Typos / polish: artifacts in Fig. 2 caption text; “EV ALUATION” spacing in §5 heading; occasional “stop-and-wait” vs. “Stop-and-Wait” inconsistency. None affect substance.

Circularity Check

0 steps flagged

No circularity: measured engineering gains vs independent baseline; analytical allocation model is validated by sweeps, not definitional of the result.

full rationale

UBEP is an empirical systems paper. The central claims (up to 52.4% All-to-All latency reduction and 11.1% TPOT vs CANN EP on CM384) are operator- and end-to-end measurements against an independent baseline on the same hardware (Tables 4, Figs. 7–11, App. C), not quantities derived from fitted constants or self-defined identities. Kernel decomposition (§3.2), hierarchical hop-aware scheduling (§3.3, Alg. 1), and Data-as-Flag (§3.4) are design choices whose benefits are shown by latency breakdowns and ablations; none reduces by construction to its own inputs. The Appendix A cost model yields a proportionality for AIV allocation that is then checked by resource-allocation sweeps (App. C.3), not used as a tautological prediction. The NP-hardness reduction (App. B) is a standard Subset-Sum argument. Related-work citations (DeepEP, Hybrid-EP, CANN EP, etc.) position the system; they do not force the measured speedups. Hardware atomicity assumptions for Data-as-Flag are portability limits, not circular reasoning. No self-definitional step, fitted-input-as-prediction, load-bearing self-citation uniqueness claim, or renaming of a known result appears in the derivation chain.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 2 invented entities

The central performance claims rest on hardware and workload assumptions of multi-tier UGAS superpods (especially CM384), not on free-fitted physical constants. Free parameters are engineering knobs (AIV split, sentinel pattern). Axioms are domain facts about the fabric and MoE traffic. Invented entities are named software mechanisms, not new physical objects; they have independent handles only via the reported measurements.

free parameters (2)
  • n_count (AIV cores for TokenCnt/CalCumSum vs token send) = workload-dependent; e.g. more CalCumSum cores as ranks/experts grow, fewer as BS grows
    Optimal split is derived from a linear cost model then tuned by grid search over ranks and batch size (Appendix A, C.3 heatmaps). Headline latency depends on this allocation.
  • Sentinel bit-pattern width/value for SP = 32B reserved unused BF16/FP16 pattern
    Chosen so collision probability is claimed ~2^-256 under uniform assumption; verified at init that weights/activations never emit it. Affects SP safety.
axioms (5)
  • domain assumption UB fabric provides unified global address space with remote load/store and 512B atomic write atomicity observed as one transaction across scale-up and NoC.
    Load-bearing for Data-as-Flag happens-before arguments and barrier elimination (§3.4, Table 5).
  • domain assumption A subset of AIVs (roughly half) saturates available dispatch bandwidth on CM384; extra cores mostly wait at BSP barriers.
    Justifies kernel decomposition and non-uniform AIV roles (Insight 1, Fig. 3b).
  • domain assumption Two-hop NPU-to-NPU latency is substantially higher than one-hop (reported ~11.5× vs intra-NPU; Table 2), so hop-agnostic load balance creates stragglers.
    Justifies hierarchical hop-homogenizing scheduler (Insight 3, §3.3).
  • domain assumption Tokens have roughly uniform size and per-token transfer time is small relative to propagation RTT on the superpod.
    Used to reduce the NP-hard objective to hop-count homogenization plus cardinality balance (§3.3).
  • standard math Program (1) token-to-AIV assignment minimizing max completion time under load balance is NP-hard (Subset-Sum reduction).
    Appendix B; motivates the quaternary-search heuristic rather than exact ILP.
invented entities (2)
  • Data-as-Flag (TFF / DC / SP variants) no independent evidence
    purpose: Embed or replace explicit flags so receivers detect completion without global SyncAll or separate control messages.
    Software protocol built on claimed 512B atomics; evidence is the paper's own latency ablations, not an external independent measurement of the protocol.
  • Hierarchical token-level scheduler with quaternary SIMD search mapper no independent evidence
    purpose: Assign tokens to AIVs balancing both count and hop-class mix within ~1 µs.
    New scheduling construction for multi-tier EP; validated only inside this evaluation.

pith-pipeline@v1.1.0-grok45 · 35702 in / 3594 out tokens · 42376 ms · 2026-07-11T01:05:46.519452+00:00 · methodology

0 comments
read the original abstract

The deployment of Mixture-of-Experts (MoE) models on production high-bandwidth superpods, such as NVIDIA's NVL72/576 and Huawei's CloudMatrix384, introduces critical challenges beyond raw interconnect bandwidth. While these systems provide unified global address spaces and high-bandwidth fabrics, their full potential for sparse MoE communication is hindered by three fundamental bottlenecks: (1) Strict execution serialization imposed by coarse-grained Bulk Synchronous Parallel (BSP) orchestration of interdependent communication phases; (2) Prohibitive synchronization overhead that fails to scale alongside high interconnect bandwidth; and (3) Severe load imbalance resulting from distance-agnostic scheduling of irregular token traffic. To eliminate these bottlenecks, we introduce UBEP (Unified-Bus Expert Parallelism), a production-ready communication library that rethinks MoE's All-to-All primitives for modern superpod architectures. Through large scale experiments, UBEP reduces All-to-All latency by up to 52.4% and MoE inference Time Per Output Token (TPOT) by up to 11.1%.

Figures

Figures reproduced from arXiv: 2607.06202 by Baochuan Yang, Chang Liu, Guanhua Li, Guihai Chen, Han Bao, Jiaqi Zheng, Junsong Wang, Mingfan Li, Qihang Duan, Si Shen, Wenkai Ling, Xianzhi Yu, Yijie Chen, Yimeng Xu, Yipeng Liu, Yuquan Zhang, Yuyang Yang, Zhiyuan Huang, Zhongzhe Hu.

Figure 2
Figure 2. Figure 2: MoE Dispatch workflow. Left: logical view of se [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance analysis of All-to-All communication. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the UBEP architecture. 3 UBEP 3.1 Overview UBEP (Unified-Bus Expert Parallelism) is a communication library for low-latency MoE inference. We build UBEP on three key designs ( [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of BSP-based multi-phase execution and kernel decomposition for token dispatch. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of three Data-as-Flag–based synchronization mechanisms across NPUs. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Latency composition of the MoE dispatch operator across representative execution configurations. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Task-wise latency breakdown of the dispatch operator across varying batch sizes. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: AIV-Level token dispatch latency under different [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Dispatch latency of Data-as-Flag variants versus [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Normalized TPOT performance of CANN EP and [PITH_FULL_IMAGE:figures/full_fig_p011_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Supplementary latency analysis of the MoE Dis [PITH_FULL_IMAGE:figures/full_fig_p017_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Performance scalability of different Top- [PITH_FULL_IMAGE:figures/full_fig_p018_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Heatmaps illustrating the relationship between the [PITH_FULL_IMAGE:figures/full_fig_p018_14.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

60 extracted references · 60 canonical work pages · 13 internal anchors

  1. [1]

    Osayamen Jonathan Aimuyo, Byungsoo Oh, and Rachee Singh. 2025. FlashDMoE: Fast Distributed MoE in a Single Kernel. (2025). arXiv:cs.DC/2506.04667 https: //arxiv.org/abs/2506.04667v2

    work page arXiv 2025

  2. [2]

    Amazon Web Services. 2025. Trainium2 Architecture — AWS Neuron Docu- mentation. (2025). https://awsdocs-neuron.readthedocs-hosted.com/en/latest/ about-neuron/arch/neuron-hardware/trainium2.html Official AWS documenta- tion. Trainium2 includes 16 CC-Cores for collective communication. Accessed June 2026

    work page 2025

  3. [3]

    AMD. 2021. Introducing AMD CDNA 2 Architecture. (2021). https: //www.amd.com/content/dam/amd/en/documents/instinct-business-docs/ white-papers/amd-cdna2-white-paper.pdf

    work page 2021

  4. [4]

    2024.Scale-Up Ethernet Framework (SUE)

    Broadcom Inc. 2024.Scale-Up Ethernet Framework (SUE). Technical Report. Broadcom Inc. https://docs.broadcom.com/doc/scale-up-ethernet-framework White Paper. 12 UBEP SIGCOMM ’26, August 17–21, 2026, Denver, CO, USA

    work page 2024

  5. [5]

    DeepSeek-AI. 2025. An early research stage expert-parallel load balancer for MoE models based on linear programming. https://github.com/deepseek-ai/LPLB. (2025)

    work page 2025

  6. [6]

    DeepSeek-AI. 2025. Expert Parallelism Load Balancer. https://github.com/ deepseek-ai/eplb. (2025)

    work page 2025

  7. [7]

    DeepSeek-V3 Technical Report

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huaj...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  8. [8]

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and C...

    work page 2022

  9. [9]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.Journal of Machine Learning Research23, 120 (2022), 1–39. http://jmlr.org/papers/v23/ 21-0998.html

    work page 2022

  10. [10]

    Google Cloud. 2025. TPU7x (Ironwood). (2025). https://docs.cloud.google.com/ tpu/docs/tpu7x Google Cloud official documentation. Accessed June 2026

    work page 2025

  11. [11]

    Yu Han, Lehan Pan, Jie Peng, Ziyang Tao, Wuyang Zhang, and Yanyong Zhang

    work page

  12. [12]

    GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference.arXiv preprint arXiv:2509.25041(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [13]

    Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. 2022. FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models. InProceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP ’22). Association for Computing Machinery, New York, NY, USA, 120–134...

    work page arXiv 2022

  14. [14]

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, and zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. InAdvances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and...

    work page 2019

  15. [15]

    Huawei. 2025. unifiedbus 2.0 Specification. (2025). https://www.unifiedbus.com/

    work page 2025

  16. [16]

    Huawei Technologies Co., Ltd. 2025. CANN: Compute Architecture for Neural Networks. (2025). https://www.hiascend.com/en/software/cann

    work page 2025

  17. [17]

    Changho Hwang, Peng Cheng, Roshan Dathathri, Abhinav Jangda, Saeed Maleki, Madan Musuvathi, Olli Saarikivi, Aashaka Shah, Ziyue Yang, Binyang Li, Caio Rocha, Qinghua Zhou, Mahdieh Ghazimirsaeed, Sreevatsa Anantharamu, and Jithin Jose. 2026. MSCCL++: Rethinking GPU Communication Abstractions for AI Inference. InProceedings of the 31st ACM International Con...

    work page doi:10.1145/3779212.3790188 2026

  18. [18]

    Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, HoYuen Chau, Peng Cheng, Fan Yang, Mao Yang, and Yongqiang Xiong. 2023. Tutel: Adaptive Mixture-of-Experts at Scale. InProceedings of Machine Learning and Systems, D. Song, M. Carbin, and T. Chen (Eds.), Vol. 5. Curan, 269–287. https://pro...

    work page 2023

  19. [19]

    Alexander Ishii and Ryan Wells. 2022. The NVLink-Network Switch: NVIDIA’s Switch Chip for High Communication-Bandwidth Superpods. In2022 IEEE Hot Chips 34 Symposium (HCS). https://doi.org/10.1109/HCS55958.2022.9895480

    work page doi:10.1109/hcs55958.2022.9895480 2022

  20. [20]

    Chenyu Jiang, Ye Tian, Zhen Jia, Shuai Zheng, Chuan Wu, and Yida Wang. 2024. Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping. InProceedings of Ma- chine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. 74–86. https://proceedings.mlsys.org/paper_files/paper/2024/file/ 339ca...

    work page 2024

  21. [21]

    Chao Jin, Ziheng Jiang, Zhihao Bai, Zheng Zhong, Juncai Liu, Xiang Li, Ningxin Zheng, Xi Wang, Cong Xie, Qi Huang, Wen Heng, Yiyuan Ma, Wenlei Bao, Size Zheng, Yanghua Peng, Haibin Lin, Xuanzhe Liu, Xin Jin, and Xin Liu. 2025. MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of- Experts Models in Production. (2025). arXiv:cs.LG/2505....

    work page arXiv 2025

  22. [22]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles. 611–626

    work page 2023

  23. [23]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668(2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  24. [24]

    Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. 2023. Accelerating Distributed MoE Training and Inference with Lina. In2023 USENIX Annual Technical Conference (USENIX ATC 23). USENIX Association, Boston, MA, 945–

    work page 2023

  25. [25]

    https://www.usenix.org/conference/atc23/presentation/li-jiamin

    work page

  26. [26]

    Yan Li, Pengfei Zheng, Shuang Chen, Zewei Xu, Yuanhao Lai, Yunfei Du, and Zhengang Wang. 2025. Speculative MoE: Communication Efficient Parallel MoE Inference with Speculative Token and Expert Pre-scheduling. (2025). arXiv:cs.LG/2503.04398 https://arxiv.org/abs/2503.04398

    work page arXiv 2025

  27. [27]

    Heng Liao, Bingyang Liu, Xianping Chen, Zhigang Guo, Chuanning Cheng, Jianbing Wang, Xiangyu Chen, Peng Dong, Rui Meng, Wenjie Liu, Zhe Zhou, Ziyang Zhang, Yuhang Gai, Cunle Qian, Yi Xiong, Zhongwu Cheng, Jing Xia, Yuli Ma, Xi Chen, Wenhua Du, Shizhong Xiao, Chungang Li, Yong Qin, Liudong Xiong, Zhou Yu, Lv Chen, Lei Chen, Buyun Wang, Pei Wu, Junen Gao, X...

    work page doi:10.1109/mm.2025.3592688 2025

  28. [28]

    Wenxiang Lin, Xinglin Pan, Lin Zhang, Shaohuai Shi, Xuan Wang, and Xi- aowen Chu. 2025. HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap. (2025). arXiv:cs.DC/2508.09591 https: //arxiv.org/abs/2508.09591

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  29. [29]

    Xinyi Liu, Yujie Wang, Fangcheng Fu, Xupeng Miao, Shenhan Zhu, Xiao- nan Nie, and Bin Cui. 2025. NetMoE: Accelerating MoE Training through Dynamic Sample Placement. InThe Thirteenth International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2025/file/ e0c256700465c158de71081b4cf5e8c3-Paper-Conference.pdf

    work page 2025

  30. [30]

    Ziming Mao, Yihan Zhang, Chihan Cui, Kaichao You, Zhongjie Chen, Zhiying Xu, Scott Shenker, Costin Raiciu, Yang Zhou, and Ion Stoica. 2025. UCCL-EP: Portable Expert-Parallel Communication. (2025). arXiv:cs.DC/2512.19849 https: //arxiv.org/abs/2512.19849

    work page arXiv 2025

  31. [31]

    Devanur, Gregory R

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. 2019. PipeDream: generalized pipeline parallelism for DNN training. InProceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP ’19). Association for Computing Machinery, New York, NY, USA, 1–15. ...

    work page arXiv 2019

  32. [32]

    Xiaonan Nie, Qibin Liu, Fangcheng Fu, Shenhan Zhu, Xupeng Miao, Xiaoyang Li, Yang Zhang, Shouda Liu, and Bin Cui. 2024. LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing. InAdvances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran ...

    work page doi:10.52202/079017-1716 2024

  33. [33]

    NVIDIA. 2024. NVIDIA GB200 NVL72. (2024). https://www.nvidia.com/en-us/ data-center/gb200-nvl72/

    work page 2024

  34. [34]

    NVIDIA. 2025. Hybrid-EP. https://github.com/deepseek-ai/DeepEP/tree/ hybrid-ep. (2025). 13 SIGCOMM ’26, August 17–21, 2026, Denver, CO, USA Y. Liu, C. Liu, S. Shen, et al

    work page 2025

  35. [35]

    NVIDIA. 2026. NVSHMEM Memory Ordering. https://docs.nvidia.com/ nvshmem/api/gen/api/ordering.html. (2026)

    work page 2026

  36. [36]

    NVIDIA Corporation. 2024. NVIDIA NVLink. (2024). https://www.nvidia.com/ en-us/products/workstations/nvlink-bridges/

    work page 2024

  37. [37]

    Yanghua Peng, Yibo Zhu, Yangrui Chen, Yixin Bao, Bairen Yi, Chang Lan, Chuan Wu, and Chuanxiong Guo. 2019. A generic communication scheduler for dis- tributed DNN training acceleration. InProceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP ’19). Association for Computing Machinery, New York, NY, USA, 16–29. https://doi.org/10.1145/...

    work page doi:10.1145/3341301.3359642 2019

  38. [38]

    Hilfer fractional advection-diffusion equations with power-law initial condition; a Numerical study using variational iteration method

    Kishore Punniyamurthy, Khaled Hamidouche, and Bradford M. Beckmann. 2024. Optimizing Distributed ML Communication with Fused Computation-Collective Operations. InSC24: International Conference for High Performance Computing, Networking, Storage and Analysis. 1–17. https://doi.org/10.1109/SC41406.2024. 00094

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1109/sc41406.2024 2024

  39. [39]

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, {Reza Yazdani} Aminabadi, {Ammar Ahmad} Awan, Jeff Rasley, and Yuxiong He. 2022. DeepSpeed- MoE: Advancing Mixture-of-Experts Inference and Training to Power Next- Generation AI Scale.Proceedings of Machine Learning Research162 (2022), 18332– 18346. Publisher Copyright: Copyright©2022 by the autho...

    work page 2022

  40. [40]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. InInternational Conference on Learning Representations. https://arxiv.org/abs/1701.06538

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  41. [41]

    Shaohuai Shi, Xinglin Pan, Xiaowen Chu, and Bo Li. 2023. PipeMoE: Acceler- ating Mixture-of-Experts through Adaptive Pipelining. InIEEE INFOCOM 2023 - IEEE Conference on Computer Communications. 1–10. https://doi.org/10.1109/ INFOCOM53939.2023.10228874

    work page arXiv 2023

  42. [42]

    Shaohuai Shi, Xinglin Pan, Qiang Wang, Chengjian Liu, Xiaozhe Ren, Zhongzhe Hu, Yu Yang, Bo Li, and Xiaowen Chu. 2024. ScheMoE: An Extensible Mixture-of- Experts Distributed Training System with Tasks Scheduling. InProceedings of the Nineteenth European Conference on Computer Systems (EuroSys ’24). Association for Computing Machinery, New York, NY, USA, 2...

    work page arXiv 2024

  43. [43]

    Xiaoyong Song, Danyuan Zhou, Kai Li, Jiayuan Chen, Hao Zhang, Xiaoguang Zhang, and Xuxia Zhong. 2025. Survey of Intra-Node GPU Interconnection in Scale-Up Network: Challenges, Status, Insights, and Future Directions.Future Internet17, 12 (2025). https://doi.org/10.3390/fi17120537

    work page doi:10.3390/fi17120537 2025

  44. [44]

    StepFun, Bin Wang, Bojun Wang, Changyi Wan, Guanzhe Huang, Hanpeng Hu, Haonan Jia, Hao Nie, Mingliang Li, Nuo Chen, Siyu Chen, Song Yuan, Wuxun Xie, Xiaoniu Song, Xing Chen, Xingping Yang, Xuelin Zhang, Yanbo Yu, Yaoyu Wang, Yibo Zhu, Yimin Jiang, Yu Zhou, Yuanwei Lu, Houyi Li, Jingcheng Hu, Ka Man Lo, Ailin Huang, Binxing Jiao, Bo Li, Boyu Chen, Changxin...

    work page

  45. [45]

    Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding. (2025). arXiv:cs.LG/2507.19427 https://arxiv.org/abs/2507.19427

    work page arXiv 2025

  46. [46]

    GLM Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengx- ing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang, Zilin Zhu, Bohan Zhang, Bosi Wen, Bowen Wu, ...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  47. [47]

    2025.Introducing UALink 200G 1.0 Specification

    ULTRA ACCELERATOR LINK CONSORTIUM. 2025.Introducing UALink 200G 1.0 Specification. Technical Report. ULTRA ACCELERATOR LINK CONSOR- TIUM. https://ualinkconsortium.org/wp-content/uploads/2025/04/UALink-1. 0-White_Paper_FINAL.pdf White Paper

    work page 2025

  48. [48]

    vllm-project. 2025. Community maintained hardware plugin for vLLM on Ascend. https://github.com/vllm-project/vllm-ascend. (2025)

    work page 2025

  49. [49]

    Hulin Wang, Yaqi Xia, Donglin Yang, Xiaobo Zhou, and Dazhao Cheng. 2025. Harnessing Inter-GPU Shared Memory for Seamless MoE Communication- Computation Fusion. InProceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, PPoPP 2025, Las Vegas, NV, USA, March 1-5, 2025. ACM, 170–182. https://doi.org/10.1145/371...

    work page doi:10.1145/3710848.3710868 2025

  50. [50]

    Tian Wu, Liming Wang, Zijian Wen, Xiaoxi Zhang, Jingpu Duan, Xianwei Zhang, and Jinhang Zuo. 2025. Accelerating Edge Inference for Distributed MoE Mod- els with Latency-Optimized Expert Placement.arXiv preprint arXiv:2508.12851 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  51. [51]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  52. [52]

    Shulai Zhang, Ningxin Zheng, Haibin Lin, Ziheng Jiang, Wenlei Bao, Chengquan Jiang, Qi Hou, Weihao Cui, Size Zheng, Li-Wen Chang, Quan Chen, and Xin Liu. 2025. Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts. (2025). arXiv:cs.DC/2502.19811 https://arxiv.org/abs/2502. 19811

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  53. [53]

    Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Huazuo Gao, Jiashi Li, Liyue Zhang, Panpan Huang, Shangyan Zhou, Shirong Ma, Wenfeng Liang, Ying He, Yuqing Wang, Yuxuan Liu, and Y.X. Wei. 2025. Insights into DeepSeek- V3: Scaling Challenges and Reflections on Hardware for AI Architectures. In Proceedings of the 52nd Annual International Symposium on ...

    work page doi:10.1145/3695053.3731412 2025

  54. [54]

    Chenggang Zhao, Shangyan Zhou, Liyue Zhang, Chengqi Deng, Zhean Xu, Yuxuan Liu, Kuai Yu, Jiashi Li, and Liang Zhao. 2025. DeepEP: an efficient expert-parallel communication library. https://github.com/deepseek-ai/DeepEP. (2025)

    work page 2025

  55. [55]

    Xing, Joseph E

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yan- ping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. 2022. Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning. In16th USENIX Symposium on Op- erating Systems Design and Implementation (OSDI 22). USENIX ...

    work page 2022

  56. [56]

    Size Zheng, Xuegui Zheng, Li wen Chang, and Jidong Zhai. 2026. UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Training. (2026). arXiv:cs.DC/2604.19241 https://arxiv.org/abs/2604.19241

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  57. [57]

    Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Y. Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon. 2022. Mixture-of-experts 14 UBEP SIGCOMM ’26, August 17–21, 2026, Denver, CO, USA with expert choice routing. InProceedings of the 36th International Conference on Neural Information Processing Systems (NIPS ’22). Curran Associate...

    work page 2022

  58. [58]

    Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, Jianzhe Xiao, Xinyi Zhang, Lingjun Liu, Haibin Lin, Li-Wen Chang, Jianxi Ye, Xiao Yu, Xuanzhe Liu, Xin Jin, and Xin Liu. 2025. MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism. (2025)....

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  59. [59]

    Zhuoran Zhu, Chunyang Zhu, Hao Lin, Xu Fu, Yiming Zhou, Quanlu Zhang, Zhenhua Li, Feng Qian, Chao Yu, Boxun Li, Guohao Dai, and Yu Wang. 2025. FUSCO: High-Performance Distributed Data Shuffling via Transformation- Communication Fusion.arXiv preprint arXiv:2512.22036(2025)

    work page arXiv 2025

  60. [60]

    Pengfei Zuo, Huimin Lin, Junbo Deng, Nan Zou, Xingkun Yang, Yingyu Diao, Weifeng Gao, Ke Xu, Zhangyu Chen, Shirui Lu, Zhao Qiu, Peiyang Li, Xianyu Chang, Zhengzhong Yu, Fangzheng Miao, Jia Zheng, Ying Li, Yuan Feng, Bei Wang, Zaijian Zong, Mosong Zhou, Wenli Zhou, Houjiang Chen, Xingyu Liao, Yipeng Li, Wenxiao Zhang, Ping Zhu, Yinggang Wang, Chuanjie Xiao...

    work page internal anchor Pith review Pith/arXiv arXiv 2025