Pith. sign in

REVIEW 8 cited by

Tutel: Adaptive Mixture-of-Experts at Scale

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2206.03382 v2 pith:QIBR76MS submitted 2022-06-07 cs.DC cs.CLcs.CV

Changho Hwang , Wei Cui , Yifan Xiong , Ziyue Yang , Ze Liu , Han Hu , Zilong Wang , Rafael Salas
show 7 more authors
Jithin Jose Prabhat Ram Joe Chau Peng Cheng Fan Yang Mao Yang Yongqiang Xiong
This is my paper · ORCID
classification cs.DCcs.CLcs.CV
keywords flexmodelparallelismpipeliningadaptivescalespeedupswinv2-moe
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Sparsely-gated mixture-of-experts (MoE) has been widely adopted to scale deep learning models to trillion-plus parameters with fixed computational cost. The algorithmic performance of MoE relies on its token routing mechanism that forwards each input token to the right sub-models or experts. While token routing dynamically determines the amount of expert workload at runtime, existing systems suffer inefficient computation due to their static execution, namely static parallelism and pipelining, which does not adapt to the dynamic workload. We present Flex, a highly scalable stack design and implementation for MoE with dynamically adaptive parallelism and pipelining. Flex designs an identical layout for distributing MoE model parameters and input data, which can be leveraged by all possible parallelism or pipelining methods without any mathematical inequivalence or tensor migration overhead. This enables adaptive parallelism/pipelining optimization at zero cost during runtime. Based on this key design, Flex also implements various MoE acceleration techniques. Aggregating all techniques, Flex finally delivers huge speedup at any scale -- 4.96x and 5.75x speedup of a single MoE layer over 16 and 2,048 A100 GPUs, respectively, over the previous state-of-the-art. Our evaluation shows that Flex efficiently and effectively runs a real-world MoE-based model named SwinV2-MoE, built upon Swin Transformer V2, a state-of-the-art computer vision architecture. On efficiency, Flex accelerates SwinV2-MoE, achieving up to 1.55x and 2.11x speedup in training and inference over Fairseq, respectively. On effectiveness, the SwinV2-MoE model achieves superior accuracy in both pre-training and down-stream computer vision tasks such as COCO object detection than the counterpart dense model, indicating the readiness of Flex for end-to-end real-world model training and inference.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 31 citations worldwide. Full citation record

  1. Relax Within, Balance Across: Geometry-Guided Load Balancing for Vision-Language Mixture-of-Experts

    cs.CV 2026-08 accept novelty 6.0 of 10

    The standard mixed image-text load-balancing loss lets modality errors cancel at one token mix; ReBA balances each modality separately and per image, reducing load across resolutions and tiles with comparable accuracy.

  2. A Training-Memory Regression in MLA Sequence Parallelism: Why Megatron-Core Forbids Absorption, and LAGA -- a Communication-Efficient Fix

    cs.DC 2026-07 conditional novelty 6.0 of 10

    LAGA replaces MLA training's per-head K/V all-to-all with a latent all-gather and local up-projection, matching explicit-form memory while cutting collective communication ~1.98x.

  3. Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference

    cs.DC 2026-07 conditional novelty 6.0 of 10

    Communication-aware expert placement plus device-level pruning yields 1.23–1.86× MoE inference throughput and better accuracy at equal speedup than load-balance or sequential baselines.

  4. Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory

    cs.DC 2026-05 unverdicted novelty 6.0 of 10

    Expert-parallel scaling leaves per-expert routing imbalance flat; mock-token benchmarks overestimate real-text imbalance and fake a batch-size trend; architectures split into data-resilient (MHA, Mamba-2) and persiste...

  5. Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers

    cs.CL 2025-12 conditional novelty 6.0 of 10

    Canon layers—residual 1-d causal convolutions over adjacent tokens—boost synthetic reasoning depth 2-4x, lift NoPE to RoPE level, and bring GLA up to Mamba2/GDN, with qualitative real-world confirmation.

  6. MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism

    cs.DC 2025-06 conditional novelty 6.0 of 10

    MPipeMoE speeds up MoE training by adaptively pipelining token batches and reusing memory buffers across partitions, achieving up to 2.8x speedup and 47% memory reduction over FasterMoE.

  7. Maximum Score Routing For Mixture-of-Experts

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    MaxScore casts MoE routing as min-cost max-flow with SoftTopk and claims better loss and eval scores at equal FLOPs; unverified because the full text is unreadable.

  8. Expert Merging in Sparse Mixture of Experts with Nash Bargaining

    cs.LG 2025-10 conditional novelty 4.0 of 10

    Expert merging in MoE models can be improved by setting per-expert weights with the Nash bargaining solution instead of simple averaging.

Pith tools