Pith. sign in

REVIEW 5 major objections 6 minor 52 references

FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FSMoE reports 1.18x–3.01x speedups over DeepSpeed-MoE and Tutel by co-scheduling intra-node and inter-node communication with expert computation.

desk verdict Real speedups and a clean modular design, but the near-optimal scheduling claim lacks oracle validation and the abstract overstates the baseline comparison; still a strong systems paper worth refereeing. read the letter →

arxiv 2501.10714 v1 pith:UKMKYUKR submitted 2025-01-18 cs.LG

Xinglin Pan , Wenxiang Lin , Lin Zhang , Shaohuai Shi , Zhenheng Tang , Rui Wang , Bo Li , Xiaowen Chu This is my paper · ORCID
classification cs.LG
keywords mixture-of-expertsdistributedtrainingtaskschedulingcommunicationoverlapgradientpartitioningexpertparallelismpipelinedegreeMoEroutingfunctions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the communication overhead that dominates sparse mixture-of-experts (MoE) training is largely schedulable rather than unavoidable. FSMoE modularizes the MoE layer, profiles each module online, and computes a near-optimal pipeline that overlaps intra-node collectives, inter-node AlltoAll, expert computation, and gradient aggregation. A sympathetic reader would care because the optimization targets the common configuration where model-parallel and expert-sharding groups equal the number of GPUs per node, and it promises to make new routing functions faster without manual per-configuration tuning. The reported results are 1.18x–1.22x average speedups over Tutel on 1458 configured MoE layers and 1.19x–3.01x average speedups over DeepSpeed-MoE and Tutel on real GPT-2- and Mixtral-based MoE models.

What carries the argument

The load-bearing machinery is a four-case scheduling model that selects an optimal pipeline degree $r$. Each communication and computation task is fit to a linear model of the form $t = \alpha + (n/r)\beta$ for inputs split into $r$ chunks. Seven inequalities, Q1–Q7, compare the chunked durations of AlltoAll, AllGather, ReduceScatter, expert GEMMs, and Gradient-AllReduce, and classify the schedule into one of four cases: inter-node dominated, expert dominated, AlltoAll dominated, or intra-node dominated. Each case has a closed-form time formula, and an SLSQP solver picks the $r$ that minimizes the relevant formula, with forward and backward phases solved separately. A second stage partitions the gradient across layers using an inverse AllReduce model and treats the remaining gradient assigned to each MoE layer as variables in a differential-evolution optimization, so Gradient-AllReduce overlaps with MoE work rather than only with dense parts.

What would settle it

Run the same 1458 configured MoE layers on a cluster where NVLink and InfiniBand traffic share bandwidth or where small chunk sizes violate the linear models, and compare the measured optimal pipeline degree with the one Algorithm 1 predicts; a systematic mismatch would refute the near-optimal scheduling claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that the expensive AlltoAll communication of sparse MoE training can be largely hidden, rather than merely reduced, when the model-parallel and expert-sharding groups are aligned with the GPUs inside a node. FSMoE pipelines the intra-node ESP-AllGather and ESP-ReduceScatter with the inter-node AlltoAll dispatch/combine and with expert computation, and then adaptively partitions gradients so the gradient AllReduce overlaps with MoE layers instead of only dense parts. The scheduler chooses the pipeline degree separately for forward and backward passes using linear performance models. In the paper's experiments this yields average speedups of 1.18x–1.22x over Tutel (with PipeMoE) on 1458 configured MoE layers and 1.19x–3.01x over DeepSpeed-MoE and Tutel on real GPT-2- and Mixtral-based MoE models, plus up to 1.42x over DeepSpeed-MoE across the four routing functions the system supports.

Load-bearing premise

The schedule assumes each communication and computation time scales linearly with the chunked input size and that intra-node and inter-node traffic do not steal bandwidth from each other; if either fails, the chosen pipeline degree is not near-optimal.

Editorial extensions

If this is right

  • Users get per-phase pipeline degrees: the same model can use one chunk count in forward propagation and another in backward propagation, which the paper finds is optimal in 912 of 1458 configured cases.
  • The gradient-partitioning step makes Gradient-AllReduce overlap with MoE layers, which is the main source of the real-model gains over schedules that overlap gradient aggregation only with dense parts.
  • Because the scheduler only needs coefficients fit once from microbenchmarks, taking under 100 seconds before training, bringing FSMoE to a new cluster requires no per-configuration tuning.
  • The modular gate, order, dispatch, combine, and expert abstractions mean new routing functions can be added and receive the same overlap scheduling; the paper demonstrates this on GShard, Sigmoid, X-MoE, and SoftMoE gates.
  • The reported speedups hold in the DP+MP+EP+ESP setting where the MP and ESP groups match the number of GPUs per node; outside that alignment, the inter-node and intra-node overlap the system relies on does not apply.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested extension: if a routing function produces highly imbalanced token counts per expert, the per-chunk workload is no longer uniform and the linear model in Eq. (1) may degrade; re-fitting with a load-balance term would show whether the four-case schedule still picks near-optimal degrees.
  • The cost-geometry argument is not obviously MoE-specific: the same pattern of overlapping intra-node collectives with inter-node collectives could benefit other hybrid-parallel workloads whose parallel groups align with node boundaries, though the paper only evaluates MoE layers.
  • On clusters with shared or oversubscribed inter-node links, the assumption that AlltoAll and Gradient-AllReduce do not contend for bandwidth could fail, and a contention term would need to be added to the performance model.
  • The optimization runs once before training; an online re-fitting variant would be needed to track thermal or multi-tenant bandwidth drift, which the paper does not address.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper presents FSMoE, a distributed training system for sparse mixture-of-experts (MoE) models. The system modularizes MoE layers into Gate, Order, I-Order, Dispatch, Combine, and Expert components, fits linear performance models to communication and GEMM microbenchmarks, and uses a scheduling algorithm (Section 4, Algorithm 1) that selects a pipeline degree r by minimizing one of four makespan expressions f1-f4, which represent different dominance cases among inter-node AlltoAll/Gradient-AllReduce, intra-node AllGather/ReduceScatter, and expert computation. It additionally proposes an adaptive gradient partitioning method (Section 5) to overlap Gradient-AllReduce with MoE-layer computation. The evaluation reports average speedups of 1.18x-1.22x over Tutel (with PipeMoE) across 1458 configured MoE-layer cases and 1.19x-3.01x over Tutel/DeepSpeed-MoE on GPT-2 and Mixtral-based models on two GPU clusters, with ablations (Tutel-Improved and FSMoE-No-IIO) that isolate the contributions of gradient overlap and inter/intra-node communication co-scheduling.

Significance. If the claimed speedups hold, FSMoE's co-scheduling of intra-node and inter-node communication with expert computation, together with adaptive gradient partitioning, is a practically useful contribution for training large MoE models under DP+MP+EP+ESP. The paper is unusually thorough in its ablation structure: Tutel-Improved and FSMoE-No-IIO isolate the effect of the proposed overlap, the evaluation covers 1,458 configurations and two cluster types with different GPUs and interconnects, and the source code is publicly available. The measured end-to-end speedups are independent of the cost-model parameters, so the comparison with Tutel and DeepSpeed-MoE is meaningful even if the scheduling model is imperfect. The main weakness is that the 'near-optimal' scheduling claim is not directly validated, and the no-contention assumption for concurrent intra/inter-node collectives is asserted rather than measured; these issues are addressable in revision.

major comments (5)
  1. [Section 4.2, Algorithm 1, Eq. (1)] The central claim of 'near-optimal task scheduling' is not validated. The paper never compares the pipeline degree r selected by Algorithm 1 with the empirically optimal r from an exhaustive sweep, nor does it report the prediction error of Eq. (1) on the combined MoE-layer makespan. The R^2 values in Section 6.2 are for individual communication and GEMM operations in isolation, not for the pipeline schedule itself. Please add an oracle comparison (e.g., a sweep over r for a representative subset of the 1458 configurations) and report the discrepancy between predicted and measured MoE-layer execution time; otherwise the 'near-optimal' phrasing is unsupported.
  2. [Section 4, Fig. 3, makespan formulas f1-f4] The schedule assumes that concurrently running intra-node collectives (ESP-AllGather, ESP-ReduceScatter) and inter-node collectives (AlltoAll, Gradient-AllReduce) on separate CUDA streams do not contend for NVLink, PCIe, or InfiniBand bandwidth. This assumption is asserted, not measured. If contention occurs, the makespan expressions f1-f4 overestimate the overlap and the selected r may be suboptimal on other clusters. Please provide a contention microbenchmark (e.g., simultaneous intra-node and inter-node collective execution) and, if contention is observed, extend the cost model accordingly.
  3. [Section 6.2, abstract, Section 3.2] The abstract and Section 3.2 claim 'online profiling' of MoE modules, but Section 6.2 describes offline microbenchmarking performed once before training; no runtime adaptation is demonstrated. This terminological mismatch should be corrected, or the paper should show evidence of online model updating. Relatedly, the validation in Section 6.2 covers only the linear fits of individual operations; it does not validate the predicted total MoE-layer time or the selected pipeline degree against measured end-to-end time.
  4. [Abstract and Section 6.4, Fig. 6] The headline 'outperforms DeepSpeed-MoE and Tutel by 1.19x-3.01x' conflates two different baselines: the 1.19x figure is the average speedup over Tutel, while the 3.01x figure is the speedup over DeepSpeed-MoE. These should be reported separately and the text should say which baseline corresponds to which range. In addition, the DeepSpeed-MoE baseline configuration (e.g., whether PipeMoE-style overlap or other optimizations were enabled) is not described, making the cross-system comparison difficult to interpret.
  5. [Figures 6-8] The speedup plots contain no error bars, confidence intervals, or per-run statistics. Table 6 provides mean and standard deviation for iteration times, but the speedup figures do not. Given that the incremental gains over the closest baselines are modest (e.g., FSMoE vs FSMoE-No-IIO is about 1.07x on real models; Table 5 shows 1.18x vs 1.12x), variance information is necessary to establish that the improvements are not noise. Please add error bars or report per-run statistics for the speedup measurements.
minor comments (6)
  1. [Section 4.4] The phrase 'the forward phrase' should be 'the forward phase'.
  2. [Section 5.1] The word 'starup' should be 'startup'.
  3. [Section 5.3, Eq. (5)] The constraint '0 < i < n_l' should likely be '1 <= i <= n_l', and the upper limit of the summation 'j = i-1' is unclear because x_j^gar is not defined; please clarify the indexing.
  4. [Section 6.3] The text says the comparison is with 'PipeMoE [42] in the structure illustrated in Fig. 2', but PipeMoE is a scheduling algorithm rather than a standalone system; please clarify how the PipeMoE baseline was assembled on top of Tutel.
  5. [Section 3.1 and Table 6] The paper states that FSMoE pre-implements GShard, Sigmoid, X-MoE, and SoftMoE routing, but Table 6 evaluates GShard, X-MoE, Sigmoid, and EC routing; please clarify which four gating functions are actually supported and evaluated.
  6. [Abstract] The phrase '1458 MoE layers' should be '1458 MoE-layer configurations' to avoid implying that the system was tested on 1458 distinct layers in a single model.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: FSMoE's end-to-end speedups are measured against external baselines, and the self-cited linear model and PipeMoE baseline are independently validated or used as comparisons.

full rationale

The central claims are empirical speedups over Tutel and DeepSpeed-MoE, measured as wall-clock iteration times on two clusters (Table 5 and Figs. 6-8). These measurements are not derived from the fitted performance models: the linear models in Eq. 1 are fit to microbenchmarks of individual GEMM/NCCL operations (Section 6.2, r^2 values reported), and Algorithm 1 uses them only to choose a pipeline degree r and a gradient-partitioning plan. The final end-to-end times are then measured with the chosen schedule, so the speedup numbers are independent of the model parameters. The gradient-partitioning scheme in Section 5 is a genuine optimization over the AllReduce model, not a fitted quantity renamed as a prediction. The paper cites PipeMoE [42], ScheMoE [43], and Parm [33], which share authors, but these are used as a stronger baseline (Tutel w/ PipeMoE) or as related work; the linear-model assumption taken from [42] is re-validated in Section 6.2 on this paper's own testbeds, so the citation is not load-bearing. The 'near-optimal' pipeline-degree claim is not checked against an exhaustive sweep, and the no-contention assumption for overlapping NVLink/InfiniBand collectives is asserted rather than measured; those are correctness/robustness concerns, not circularity. No claim in the paper reduces by construction to its own inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on calibrated performance-model coefficients and on the validity of the linear, no-contention scheduling model. No new physical entities are introduced; the system's abstractions and scheduler are software components.

free parameters (2)
  • Testbed-A performance model coefficients (alpha_*, beta_*) = alpha_a2a=2.87e-1, beta_a2a=2.21e-7, alpha_ag=3.37e-1, beta_ag=2.32e-6, alpha_rs=3.95e-1, beta_rs=2.34e-7…
    Fitted to NCCL and GEMM microbenchmarks on Testbed-A (Fig. 5a, 5b); used in Algorithm 1 to select pipeline degree and in Section 5 to partition gradients.
  • Testbed-B performance model coefficients (alpha_*, beta_*) = alpha_a2a=1.75e-1, beta_a2a=3.06e-7, alpha_ag=3.20e-2, beta_ag=1.68e-7, alpha_rs=3.91e-2, beta_rs=1.67e-7…
    Fitted to NCCL and GEMM microbenchmarks on Testbed-B (Fig. 5c, 5d); used for pipeline-degree and gradient-partition decisions.
assumptions (4)
  • domain assumption Execution time of chunked AlltoAll, AllGather, ReduceScatter, AllReduce, and GEMM scales linearly with message size or workload (Eq. 1).
    The paper fits intercepts and slopes to microbenchmarks and reports R^2 > 0.998 for tested ranges, but linearity is assumed for untested sizes and configurations.
  • domain assumption Overlapping intra-node and inter-node communication tasks does not cause bandwidth or resource contention that changes their execution times.
    The schedules in Fig. 3c/3d and the time formulas in Section 4.2 rely on concurrent execution of ESP-AllGather/ReduceScatter and AlltoAll without interference.
  • ad hoc to paper The four cases in Section 4.2 cover all possible relative timings, and the objective functions f1-f4 correctly represent the makespan of the MoE layer for each case.
    The paper states that multiple factors can be categorized into one of the cases but does not prove exhaustiveness or validate against brute-force search over the pipeline degree.
  • domain assumption NCCL collective operations are the correct primitives and the per-byte cost is independent of the chunking pattern used in the pipeline.
    Used in Sections 4.1 and 5.1; if chunking changes the linear-model parameters (e.g., at very small sizes), the optimal degree would differ.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models." pith.science (2026). https://pith.science/paper/UKMKYUKR

@misc{pith2026250110714,
  author       = {Pith},
  title        = {Pith review of: FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UKMKYUKR}},
  note         = {Machine review of arXiv:2501.10714}
}
abstract

Recent large language models (LLMs) have tended to leverage sparsity to reduce computations, employing the sparsely activated mixture-of-experts (MoE) technique. MoE introduces four modules, including token routing, token communication, expert computation, and expert parallelism, that impact model quality and training efficiency. To enable versatile usage of MoE models, we introduce FSMoE, a flexible training system optimizing task scheduling with three novel techniques: 1) Unified abstraction and online profiling of MoE modules for task scheduling across various MoE implementations. 2) Co-scheduling intra-node and inter-node communications with computations to minimize communication overheads. 3) To support near-optimal task scheduling, we design an adaptive gradient partitioning method for gradient aggregation and a schedule to adaptively pipeline communications and computations. We conduct extensive experiments with configured MoE layers and real-world MoE models on two GPU clusters. Experimental results show that 1) our FSMoE supports four popular types of MoE routing functions and is more efficient than existing implementations (with up to a 1.42$\times$ speedup), and 2) FSMoE outperforms the state-of-the-art MoE training systems (DeepSpeed-MoE and Tutel) by 1.18$\times$-1.22$\times$ on 1458 MoE layers and 1.19$\times$-3.01$\times$ on real-world MoE models based on GPT-2 and Mixtral using a popular routing function.

Figures

Figures reproduced from arXiv: 2501.10714 by the authors.

Figure 1
Figure 1. A typical MoE structure with 𝐸 experts. 2.1 Mixture-of-Experts Layer In modern MoE models, which are typically built atop the Transformer [45] architecture, an MoE layer is used to re￾place the ffn layer. As shown in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An example of 𝑁DP = 𝑁MP = 𝑁EP = 𝑁ESP = 2. The attention is partitioned into two parts across MP groups, and the two experts are distributed to the two EP groups (GPU1 and GPU3, as well as GPU2 and GPU4) in EP, and each expert is further partitioned into two shards across the ESP group. The blue and green rectangles indicate the data tensors. Optimizing Network Communication. As shown in Fig. 3a, various parallel par… view at source ↗
Figure 4
Figure 4. Four cases when scheduling the pipelining of ESP-AllGather/ESP-ReduceScatter, AlltoAll Dispatch/Com￾bine, expert computations and Gradient-AllReduce with the pipeline degree 𝑟 = 2. (a) Case1: The AlltoAll communica￾tions are slower than intra-node communication and expert computations, but the inter-node communications (AlltoAll and Gradient-AllReduce) are not slower than intra-node com￾munication and expert computa… view at source ↗
Figures from the paper (5 more)
Figure 3
Figure 3. Figure 3: Backpropagation of four schedules in DP+MP+EP+ESP with the pipeline degree 𝑟 = 4 in￾cluding (a) the default schedule, (b) an improved Tutel version (Tutel-Improved) where Gradient-AllReduce is overlapped with other dense operations using PipeMoE, (c) our proposed sched…
Figure 5
Figure 5. Figure 5: Performance models. Markers are measured values and lines are predicted values with estimated pa￾rameters. (a) 𝛼𝑔𝑒𝑚𝑚=4.26e-2, 𝛽𝑔𝑒𝑚𝑚=2.29e-11 on Testbed-A. (b) 𝛼𝑎2𝑎=2.87e-1, 𝛽𝑎2𝑎=2.21e-7, 𝛼𝑎𝑔=3.37e-1, 𝛽𝑎𝑔=2.32e-06, 𝛼𝑟𝑠=3.95e-1, 𝛽𝑟𝑠=2.34e-7, 𝛼𝑎𝑟=5.11e-1, 𝛽𝑎𝑟=4.95e-6 on T…
Figure 8
Figure 8. Figure 8: Speedups of five schedules over DS-MoE on Testbed-A when PP is enabled. Testbed-A and Testbed-B, respectively. 𝐿 is set to 256 on Testbed-B and to 1024 on Testbed-A. Ensuring the models to be held on Testbed-B (32x 2080Ti 11GB), we set the number of layers for Mixtral-…
Figure 6
Figure 6. Figure 6: Speedups of FSMoE, FSMoE-No-IIO, Tutel, Tutel￾Improved, PipeMoE+Lina (PipeMoE with the additional schedule introduced by Lina [24] that partitions the gra￾dient into fixed chunk size) over DeepSpeed-MoE (DS-MoE) on MoE models (GPT2-XL, Mixtral-7B and Mixtral-22B) [PIT…
Figure 7
Figure 7. Figure 7: Speedups of five schedules over DS-MoE on Testbed-A with different configurations. over Tutel in [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 34 canonical work pages

  1. [1]

    https://developer.nvidia.com/blog/doubling-all2all- performance-with-nvidia-collective-communication-library-2-12/

    Doubling all2all performance with nvidia collective communica- tion library 2.12. https://developer.nvidia.com/blog/doubling-all2all- performance-with-nvidia-collective-communication-library-2-12/. Ac- cessed: 2022-07-13

    work page 2022

  2. [2]

    Deepspeed-inference: enabling efficient infer- ence of transformer models at unprecedented scale

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed-inference: enabling efficient infer- ence of transformer models at unprecedented scale. In International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–...

    work page 2022

  3. [3]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    1901

  4. [4]

    FLUX: fast software-based communication overlap on gpus through kernel fusion

    Li-Wen Chang, Wenlei Bao, Qi Hou, Chengquan Jiang, Ningxin Zheng, Yinmin Zhong, Xuanrun Zhang, Zuquan Song, Ziheng Jiang, Haibin Lin, Xin Jin, and Xin Liu. FLUX: fast software-based communication overlap on gpus through kernel fusion. CoRR, abs/2406.06858, 2024

    arXiv 2024

  5. [5]

    Centauri: Enabling efficient sched- uling for communication-computation overlap in large model train- ing via communication partitioning

    Chang Chen, Xiuhong Li, Qianchao Zhu, Jiangfei Duan, Peng Sun, Xingcheng Zhang, and Chao Yang. Centauri: Enabling efficient sched- uling for communication-computation overlap in large model train- ing via communication partitioning. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Sys...

    work page 2024

  6. [6]

    On the representation collapse of sparse mixture of experts

    Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, et al. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems , 35:34600–34613, 2022

    work page 2022

  7. [7]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1–113, 2023

    2023

  8. [8]

    Stablemoe: Stable routing strategy for mixture of experts

    Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. Stablemoe: Stable routing strategy for mixture of experts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7085–7095, 2022

    work page 2022

Show all 52 references
  1. [9]

    Large scale distributed deep networks

    Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc’aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks. Advances in neural information processing systems, 25, 2012

    2012

  2. [10]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024

    DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024

    2024

  3. [11]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  4. [12]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research , 23(1):5232–5270, 2022

    2022

  5. [13]

    Fastmoe: A fast mixture-of-expert training system

    Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. Fastmoe: A fast mixture-of-expert training system. arXiv preprint arXiv:2103.13262, 2021

    2021 arXiv

  6. [14]

    FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models

    Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, ...

    2022

  7. [15]

    Gpipe: Efficient training of giant neural networks using pipeline parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019

    2019

  8. [16]

    Experts weights averaging: A new general training scheme for vision transformers

    Yongqi Huang, Peng Ye, Xiaoshui Huang, Sheng Li, Tao Chen, and Wanli Ouyang. Experts weights averaging: A new general training scheme for vision transformers. arXiv preprint arXiv:2308.06093, 2023

    2023 arXiv

  9. [17]

    Tutel: Adaptive mixture-of-experts at scale

    Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. Tutel: Adaptive mixture-of-experts at scale. Proceedings of Machine Learning and Systems, 5, 2023

    2023

  10. [18]

    Breaking the computation and communication abstraction barrier in distributed machine learning workloads

    Abhinav Jangda, Jun Huang, Guodong Liu, Amir Hossein Nodehi Sabet, Saeed Maleki, Youshan Miao, Madanlal Musuvathi, Todd Mytkowicz, and Olli Saarikivi. Breaking the computation and communication abstraction barrier in distributed machine learning workloads. In ASPLOS, pages 402...

    2022

  11. [19]

    Highly scalable deep learning training system with mixed-precision: Training ImageNet in four minutes

    Xianyan Jia, Shutao Song, Shaohuai Shi, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, Tiegang Chen, Guangxiao Hu, and Xiaowen Chu. Highly scalable deep learning training system with mixed-precision: Training ImageNet in fou...

    2018

  12. [20]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  13. [21]

    Lancet: Accelerating mixture-of-experts training by over- lapping weight gradient computation and all-to-all communication

    Chenyu Jiang, Ye Tian, Zhen Jia, Chuan Wu, Yida Wang, and Shuai Zheng. Lancet: Accelerating mixture-of-experts training by over- lapping weight gradient computation and all-to-all communication. Proceedings of Machine Learning and Systems , 6:74–86, 2024

    2024

  14. [22]

    Gshard: Scaling giant models with conditional compu- tation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional compu- tation and automatic sharding. In International Conference on Learning Representations, 2020

    2020

  15. [23]

    BASE layers: Simplifying training of large, sparse models

    Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. BASE layers: Simplifying training of large, sparse models. In International Conference on Machine Learning , pages 6265–6274. PMLR, 2021

    2021

  16. [24]

    Acceler- ating distributed{MoE} training and inference with lina

    Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. Acceler- ating distributed{MoE} training and inference with lina. In USENIX Annual Technical Conference, pages 945–959, 2023

    2023

  17. [25]

    Janus: A unified dis- tributed training framework for sparse mixture-of-experts models

    Juncai Liu, Jessie Hui Wang, and Yimin Jiang. Janus: A unified dis- tributed training framework for sparse mixture-of-experts models. In Proceedings of the ACM SIGCOMM 2023 Conference , pages 486–498, 2023

    2023

  18. [26]

    Gating dropout: Communication-efficient regularization for sparsely activated transformers

    Rui Liu, Young Jin Kim, Alexandre Muzio, and Hany Hassan. Gating dropout: Communication-efficient regularization for sparsely activated transformers. In International Conference on Machine Learning , pages 13782–13792. PMLR, 2022

    2022

  19. [27]

    Modeling task relationships in multi-task learning with multi- gate mixture-of-experts

    Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi- gate mixture-of-experts. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 1930–1939, 2018

    1930

  20. [28]

    Bagualu: targeting brain scale pretrained models with over 37 million cores

    Zixuan Ma, Jiaao He, Jiezhong Qiu, Huanqi Cao, Yuanwei Wang, Zhenbo Sun, Liyan Zheng, Haojie Wang, Shizhi Tang, Tianyu Zheng, et al. Bagualu: targeting brain scale pretrained models with over 37 million cores. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and ...

    2022

  21. [29]

    Efficient large- scale language model training on GPU clusters using Megatron-LM

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGres- ley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large- scale language model training on GPU clusters using Megatron-LM. In Procee...

    2021

  22. [30]

    Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement

    Xiaonan Nie, Xupeng Miao, Zilong Wang, Zichao Yang, Jilong Xue, Lingxiao Ma, Gang Cao, and Bin Cui. Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement. Pro- ceedings of the ACM on Management of Data , 1(1):1–19, 2023

    2023

  23. [31]

    Het- umoe: An efficient trillion-scale mixture-of-expert distributed training system

    Xiaonan Nie, Pinxue Zhao, Xupeng Miao, Tong Zhao, and Bin Cui. Het- umoe: An efficient trillion-scale mixture-of-expert distributed training system. arXiv preprint arXiv:2203.14685, 2022

    2022 arXiv

  24. [32]

    Springer, 1999

    Jorge Nocedal and Stephen J Wright.Numerical optimization. Springer, 1999

    1999

  25. [33]

    Parm: Efficient training of large sparsely-activated models with dedicated schedules

    Xinglin Pan, Wenxiang Lin, Shaohuai Shi, Xiaowen Chu, Weinong Sun, and Bo Li. Parm: Efficient training of large sparsely-activated models with dedicated schedules. In IEEE INFOCOM 2024-IEEE Conference on Computer Communications, 2024

    2024

  26. [34]

    Sinclair

    Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena, and Matthew D. Sinclair. T3: transparent tracking & triggering for fine- grained overlap of compute & collectives. In ASPLOS (2), pages 1146–

  27. [35]

    Differential evolution

    Kenneth V Price. Differential evolution. In Handbook of optimization: From classical to modern approach , pages 187–214. Springer, 2013

    2013

  28. [36]

    From sparse to soft mixtures of experts.arXiv preprint arXiv:2308.00951, 2023

    Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts.arXiv preprint arXiv:2308.00951, 2023

    2023 arXiv

  29. [37]

    Beckmann

    Kishore Punniyamurthy, Khaled Hamidouche, and Bradford M. Beckmann. Optimizing distributed ml communication with fused computation-collective operations, 2024

    2024

  30. [38]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  31. [39]

    Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yux- iong He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. In International Conference on Machine Lear...

    2022

  32. [40]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InInternational Conference on Learning Representations , 2016

    2016

  33. [41]

    Exploiting simultaneous communications to accelerate data parallel distributed deep learning

    Shaohuai Shi, Xiaowen Chu, and Bo Li. Exploiting simultaneous communications to accelerate data parallel distributed deep learning. In IEEE INFOCOM 2021-IEEE Conference on Computer Communications , pages 1–10. IEEE, 2021

    2021

  34. [42]

    PipeMoE: Ac- celerating mixture-of-experts through adaptive pipelining

    Shaohuai Shi, Xinglin Pan, Xiaowen Chu, and Bo Li. PipeMoE: Ac- celerating mixture-of-experts through adaptive pipelining. In IEEE INFOCOM 2023-IEEE Conference on Computer Communications , 2023

    2023

  35. [43]

    Schemoe: An ex- tensible mixture-of-experts distributed training system with tasks scheduling

    Shaohuai Shi, Xinglin Pan, Qiang Wang, Chengjian Liu, Xiaozhe Ren, Zhongzhe Hu, Yu Yang, Bo Li, and Xiaowen Chu. Schemoe: An ex- tensible mixture-of-experts distributed training system with tasks scheduling. In Proceedings of the Nineteenth European Conference on Computer Syst...

    2024

  36. [44]

    A hybrid tensor-expert- data parallelism approach to optimize mixture-of-experts training

    Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Ra- jbhandari, Yuxiong He, and Abhinav Bhatele. A hybrid tensor-expert- data parallelism approach to optimize mixture-of-experts training. In Proceedings of the 37th International Conference on Supercomputing , pages 2...

    2023

  37. [45]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017

    2017

  38. [46]

    Overlap communication with dependent compu- tation via decomposition in large deep learning models

    Shibo Wang, Jinliang Wei, Amit Sabne, Andy Davis, Berkin Ilbeyi, Blake Hechtman, Dehao Chen, Karthik Srinivasa Murthy, Marcello Maggioni, Qiao Zhang, Sameer Kumar, Tongfei Guo, Yuanzhong Xu, and Zongwei Zhou. Overlap communication with dependent compu- tation via decomposition...

    2023

  39. [47]

    Large batch optimization for deep learning: Training BERT in 76 minutes

    Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Sri- nadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In International Conference on Learning Represen- tations, 2020

    2020

  40. [48]

    Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts

    Zhao You, Shulin Feng, Dan Su, and Dong Yu. Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts. arXiv preprint arXiv:2105.03036, 2021

    2021 arXiv

  41. [49]

    SmartMoE: Efficiently training Sparsely-Activated mod- els through combining offline and online parallelization

    Mingshu Zhai, Jiaao He, Zixuan Ma, Zan Zong, Runqing Zhang, and Jidong Zhai. SmartMoE: Efficiently training Sparsely-Activated mod- els through combining offline and online parallelization. In USENIX Annual Technical Conference, pages 961–975, 2023

    2023

  42. [50]

    Pit: Optimization of dynamic sparse deep learning models via permutation invariant transformation

    Ningxin Zheng, Huiqiang Jiang, Quanlu Zhang, Zhenhua Han, Lingx- iao Ma, Yuqing Yang, Fan Yang, Chengruidong Zhang, Lili Qiu, Mao Yang, et al. Pit: Optimization of dynamic sparse deep learning models via permutation invariant transformation. In Proceedings of the 29th Symposiu...

    2023

  43. [51]

    Mixture-of-experts with expert choice routing

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vin- cent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon. Mixture-of-experts with expert choice routing. arXiv preprint arXiv:2202.09368, 2022

    2022 arXiv

  44. [52]

    Taming sparsely activated transformer with stochastic experts

    Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. Taming sparsely activated transformer with stochastic experts. In International Conference on Learning Representations, 2021

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.