Pith. sign in

REVIEW 11 cited by

FastMoE: A Fast Mixture-of-Expert Training System

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2103.13262 v1 pith:HCIUEKOC submitted 2021-03-24 cs.LG cs.CLcs.DC

Jiaao He , Jiezhong Qiu , Aohan Zeng , Zhilin Yang , Jidong Zhai , Jie Tang This is my paper · ORCID
classification cs.LGcs.CLcs.DC
keywords systemfastmoetrainingdifferentpytorchavailabledistributedenlarging
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Mixture-of-Expert (MoE) presents a strong potential in enlarging the size of language model to trillions of parameters. However, training trillion-scale MoE requires algorithm and system co-design for a well-tuned high performance distributed training system. Unfortunately, the only existing platform that meets the requirements strongly depends on Google's hardware (TPU) and software (Mesh Tensorflow) stack, and is not open and available to the public, especially GPU and PyTorch communities. In this paper, we present FastMoE, a distributed MoE training system based on PyTorch with common accelerators. The system provides a hierarchical interface for both flexible model design and easy adaption to different applications, such as Transformer-XL and Megatron-LM. Different from direct implementation of MoE models using PyTorch, the training speed is highly optimized in FastMoE by sophisticated high-performance acceleration skills. The system supports placing different experts on multiple GPUs across multiple nodes, enabling enlarging the number of experts linearly against the number of GPUs. The source of FastMoE is available at https://github.com/laekov/fastmoe under Apache-2 license.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BrainFIBRE: A Foundation Model via Information Decomposition for Brain Microstructure

    cs.CV 2026-07 unverdicted novelty 7.0 of 10

    BrainFIBRE pretrains a five-expert Mixture-of-Experts model on NODDI-derived microstructural maps and outperforms prior deep models on age, sex, cerebrovascular, neurodegenerative, and cognitive prediction.

  2. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    cs.DC 2026-06 accept novelty 6.5 of 10

    KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.

  3. Relax Within, Balance Across: Geometry-Guided Load Balancing for Vision-Language Mixture-of-Experts

    cs.CV 2026-08 accept novelty 6.0 of 10

    The standard mixed image-text load-balancing loss lets modality errors cancel at one token mix; ReBA balances each modality separately and per image, reducing load across resolutions and tiles with comparable accuracy.

  4. On the Design of Mixture-of-Experts for Dynamic Gaussian Splatting

    cs.CV 2026-07 accept novelty 6.0 of 10

    Two MoE integration strategies (joint canonical MoDE vs. independent-then-route MoE-GS) improve dynamic Gaussian Splatting by composing complementary deformation priors.

  5. BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A ReLU-routed MoE with chunk-level sparsity training objectives and custom kernels combining activation sparsity with speculative decoding achieves over 70% 8-token chunk sparsity and up to 3.67x end-side speedup.

  6. HarMoEny: Efficient Multi-GPU Inference of MoE Models

    cs.DC 2025-06 conditional novelty 5.0 of 10

    HarMoEny uses dynamic token redistribution and asynchronous expert prefetching to achieve near-perfect GPU load balance in multi-GPU MoE inference.

  7. Two Is Better Than One: Rotations Scale LoRAs

    cs.LG 2025-05 conditional novelty 5.0 of 10

    RadarGate rotates LoRA outputs before mixing them, expanding the set of reachable outputs beyond plain weighted sums.

  8. MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing

    cs.LG 2025-02 conditional novelty 5.0 of 10

    MoETuner formulates MoE expert placement as two ILPs to balance token loads and minimize inter-GPU communication, reporting 9.3% and 17.5% end-to-end inference speedups on Mixtral-8x7B.

  9. UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning

    cs.LG 2025-08 conditional novelty 4.0 of 10

    A redesigned memory-layer architecture with five engineering improvements reaches performance parity with 8-expert MoE at similar compute, with lower memory access and stronger long-context memorization.

  10. Towards Accurate and Efficient 3D Object Detection for Autonomous Driving: A Mixture of Experts Computing System on Edge

    cs.CV 2025-07 reject novelty 4.0 of 10

    EMC2 routes each driving scene to one of three expert 3D detectors using distance and confidence thresholds, reporting higher accuracy and 2.6x faster inference on Jetson, though thresholds are tuned on the validation set.

  11. MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing

    cs.LG 2025-06 conditional novelty 4.0 of 10

    MoE-GPS shows that predicting only the aggregate token-to-expert distribution, instead of exact token routing, can improve MoE inference latency by more than 23% in simulation, with guidelines for when each strategy wins.

Pith tools