Pith. sign in

REVIEW 5 major objections 6 minor 4 cited by

MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing

T0 review · 5 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-stage integer linear program that places MoE experts to balance token loads and routing traffic delivers 9.3% and 17.5% end-to-end speedups on Mixtral-8x7B inference.

desk verdict The routing-dependency idea is real and the ILP design is plausible, but the printed ILP1 doesn't enforce a valid expert partition, and the evaluation is too narrow and in-sample to support the strength of the claims. read the letter →

arxiv 2502.06643 v1 pith:IWFB3USP submitted 2025-02-10 cs.LG cs.DC

Seokjin Go , Divya Mahajan This is my paper · ORCID
classification cs.LGcs.DC
keywords Mixture-of-ExpertsexpertparallelismILPoptimizationtokenroutingloadbalancingtaillatencyLLMinferenceall-to-allcommunication
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Expert-parallel serving of Mixture-of-Experts models suffers from two intertwined problems: some GPUs process far more tokens than others because expert activation is skewed, and some GPU pairs exchange far more tokens than others during all-to-all routing. MoETuner's claim is that both problems have a common cause, the way experts are assigned to GPUs, and a common fix, an integer linear program that balances token-processing load while minimizing the slowest inter-GPU communication path in each layer. On Mixtral-8x7B the paper reports that this placement reduces end-to-end inference time by 9.3% in a single node and 17.5% across two nodes, with the largest gains coming from lower tail latency in token processing and all-to-all communication. If the claim holds, expert placement becomes a one-time offline optimization that a serving stack can adopt without changing the model or its routing policy.

What carries the argument

The load-bearing object is a two-stage integer linear program built from a token-routing history table. The table records, for each pair of neighboring layers, how many tokens traveled from each expert in layer $l$ to each expert in layer $l+1$; because routing is stable across batches, a small profile approximates it. ILP 1, Load-Balanced Expert Clustering, assigns each expert in each layer to one of $G$ clusters so that cluster loads $T_{c,l}$ deviate as little as possible from the per-layer average $\bar{T}_l$. ILP 2, Cluster-to-GPU Assignment, maps the clusters to physical GPUs with binary variables $y_{c,g,l}$, minimizing the sum over layers of the maximum communication cost across all GPU pairs, where the cost of sending $C_{c_1,c_2,l}$ tokens between clusters is divided by the GPU-pair bandwidth $B_{g_1,g_2}$. The two-stage split keeps the problem tractable while encoding both load balance and communication skew as hard constraints.

What would settle it

Take the placement MoETuner computes after profiling one dataset, say WikiText-103, and benchmark it on a different dataset or on held-out inputs from the same task. If token-processing or all-to-all tail latency is no better than the baseline's, or if the distribution of tokens per expert shifts enough to unbalance the GPUs again, the profiling-stability assumption fails. A direct measurement: compare the per-expert token counts from the profiling subset against a fresh batch from the live workload; a large divergence in those counts would mean the ILP solution is stale.

Watch

Extended reading notes

Core claim

The paper's central claim is that the two bottlenecks of expert-parallel MoE inference, skewed token-processing load across GPUs and skewed inter-GPU token traffic, can be removed by choosing the expert-to-GPU mapping itself, rather than by overlapping communication or re-routing tokens at runtime. MoETuner profiles token routing on a small sampled subset of the task data, then solves two integer linear programs: the first clusters experts within each layer so that each cluster carries nearly the same token-processing load, and the second assigns those clusters to GPUs to minimize the maximum per-layer communication cost between GPU pairs, weighted by available bandwidth. The result is a placement that keeps every GPU's parameter count equal while redistributing token work and remote dispatch volume. On Mixtral-8x7B, the authors report 9.3% and 17.5% end-to-end speedups in single-node and multi-node inference, with 27–36% reductions in token-processing tail latency and 30–36% reductions in all-to-all communication tail latency.

Load-bearing premise

The whole scheme rests on the assumption that the token routing patterns seen in a small profiled subset of a task's data are stable enough to represent the full dataset and the live serving workload, so an expert-to-GPU mapping optimized once stays near-optimal at inference time.

Editorial extensions

If this is right

  • A placement computed once offline remains valid for repeated inference on the same task, so the optimization cost is amortized over the serving lifetime.
  • Models with more severe routing skew stand to gain more, because the method targets the tail of the token-processing and communication distributions rather than their averages.
  • Deployments with slower inter-node links benefit more: the reported multi-node speedup of 17.5% exceeds the single-node speedup of 9.3%, consistent with reduced inter-node traffic.
  • MoETuner preserves the equal-per-GPU parameter balance of standard expert parallelism, so it does not trade memory balance for load balance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The profiling-stability assumption is the main unstated risk: if the live workload's routing distribution drifts, the ILP solution becomes stale. A natural extension is periodic re-profiling or an online trigger that re-runs the ILP when routing statistics shift.
  • The paper itself notes that some layer-level latency fluctuations in the multi-node results are likely due to short profiling iterations; this self-acknowledged sensitivity makes the profiling-stability assumption the first thing to test before trusting the reported gains.
  • The same two-stage formulation should transfer to MoE models with more than eight experts per layer and to training-time expert parallelism, where routing patterns also stabilize; the paper only evaluates inference on Mixtral-8x7B.
  • A lighter-weight test of the core claim would compare the ILP placement against a random placement with the same load-balanced constraint: if random placements achieve similar gains, the communication-minimizing objective, not load balance, is responsible for the speedups.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper proposes MoETuner, a two-stage ILP-based expert placement strategy for distributed Mixture-of-Experts inference. The first ILP clusters experts within each layer using token routing statistics to balance token-processing load; the second ILP maps those clusters to GPUs to minimize inter-GPU communication cost. The method is evaluated on Mixtral-8x7B in single-node (8xH100) and multi-node (16xH200) configurations against a Megatron-LM baseline, reporting 9.3% and 17.5% end-to-end speedups, 27-36% reductions in token-processing tail latency, and 24-36% reductions in all-to-all latency. The paper claims that the resulting assignment is optimal with respect to both load balance and communication cost.

Significance. If the claims hold, MoETuner addresses a real systems bottleneck: expert-parallel MoE inference is dominated by load imbalance and communication skew, and prior work often targets only total communication volume or compute-communication overlap. The idea of exploiting inter-layer routing dependencies in an ILP formulation is reasonable, and the reported measurements on a real model are potentially useful. The authors also provide unusually clear experimental conditions, including the ILP solver, parallelism configuration, and hardware details. However, the significance is conditional: the formal notion of optimality is not established by the published formulation because ILP1 lacks an expert partition constraint, the two-stage decomposition is not shown to preserve a joint optimum, and the evaluation is in-sample because the placement is fitted to routing statistics profiled on the same datasets on which it is then measured. No code or artifact is provided, which limits reproducibility.

major comments (5)
  1. [Section 4.2.1 (Eq. 7)] The ILP1 formulation does not enforce that each expert belongs to exactly one cluster. The only placement constraint is sum_e x_{c,e,l} >= 1, which requires each cluster to contain at least one expert but does not prevent an expert from being assigned to several clusters or to none. Because T_{c,l} in Eq. (6) sums P_{e,l} over every cluster containing expert e, duplicating a hot expert across clusters artificially reduces the load imbalance that the objective in Eq. (1) penalizes, so the solver may return a solution with no valid expert-to-GPU mapping. ILP2's Eq. (13) counts experts through x_{c,e,l} * y_{c,g,l} and therefore counts a duplicated expert once per cluster, which does not restore feasibility. Please add the partition constraint sum_c x_{c,e,l} = 1 for every e and l, and rerun the ILP and the experiments.
  2. [Sections 4.1 and 5] The evaluation is in-sample with respect to the profiling data. Token routing statistics are collected on a small sampled subset of each evaluation dataset, the ILP is solved for that subset, and the resulting placement is then benchmarked on the same dataset. Figure 8 only shows that routing patterns are consistent across batches of the same task; it does not show stability across tasks or input distributions. The reported 9.3% and 17.5% end-to-end speedups and the tail-latency reductions are therefore fitting-to-data results rather than evidence of generalization. Please add an out-of-sample experiment (for example, optimize on one dataset or workload and evaluate on another) and report the profiling subset size and number of iterations used.
  3. [Section 4.2 (Abstract and Introduction)] The paper claims an 'optimal expert-to-GPU assignment' that jointly minimizes communication and balances load, but the proposed method is a sequential two-stage decomposition: ILP1 minimizes load imbalance without considering routing dependencies, and ILP2 then minimizes communication for the fixed clusters produced by ILP1. No argument is given that this decomposition preserves the joint optimum of the two objectives. A clustering that is optimal for load balance can be poor for communication affinity, and ILP2 cannot repair it. Please either prove a separation property that makes the decomposition exact, or soften the optimality claim to a two-stage heuristic and measure the gap against a joint ILP formulation.
  4. [Section 5.2 (Tail-latency metric)] The metric called 'tail latency' is defined as the average over iterations of the maximum GPU execution time per layer, which is a mean-of-max statistic rather than a tail quantile such as p95 or p99. The abstract and Section 5.3 report reductions of 27-36% in tail latency, but the measured quantity does not establish a reduction in the tail of the latency distribution. Please report proper tail percentiles (p95/p99) or rename the metric to 'mean maximum latency' and qualify the claims accordingly.
  5. [Section 4.2.2 (Eqs. 8 and 11)] The objective function in Eqs. (8) and (11) is underspecified: the max operator has no index or argument, and the expression inside already sums over all clusters and all GPU pairs, so as written it maximizes a fully summed scalar per layer. This does not define the advertised 'maximum communication cost across GPU pairs.' Please write the max over GPU pairs explicitly (for example, max_{g1,g2} of the per-pair communication cost) and verify that the ILP2 objective is linearizable.
minor comments (6)
  1. [Section 4.2.1 (Eq. 2)] Equation (2) includes a sum over t from 0 to T-1, but P_{e,l} does not depend on t; this sum is unexplained and should be removed or T should be defined.
  2. [Section 4.2.1] The text says ILP1 groups experts 'based on routing dependencies,' but the objective in Eqs. (1)-(7) only balances token loads; dependencies first appear in ILP2. Please align the wording with the actual formulation.
  3. [Section 4.2.1 (objective)] The absolute-value objective in Eq. (1) is not an ILP as displayed; the standard epigraph reformulation with auxiliary variables should be stated so that the ILP claim is formally correct.
  4. [Section 5.1] Single-node experiments use H100 GPUs while multi-node experiments use H200 GPUs; the paper should state explicitly that the two speedup numbers are not directly comparable across hardware generations.
  5. [Section 5.2 and Related Work] The evaluation compares only with Megatron-LM's contiguous expert placement. Since ExFlow and Lina are discussed as dependency-aware or communication-optimized baselines, an empirical comparison with at least one of them would make the claimed improvements more convincing.
  6. [Figure 13 caption] The caption for Figure 13(b) says 'tail latency of token processing time' but the figure shows all-to-all communication; the caption should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the ILP placement is derived from profiled routing statistics, but the measured latency and throughput outcomes are empirical, not equal to the optimization objective by construction.

full rationale

MoETuner's derivation chain is: profile token routing on a sampled subset, solve ILP1 to balance per-cluster loads (Eqs. 1-7), solve ILP2 to minimize the maximum inter-GPU communication cost (Eqs. 8-15), then instantiate the placement and measure end-to-end time, token processing latency, and all-to-all latency. No step reduces to its input by definition. The ILP objectives are proxies (load deviation and max communication cost), not the reported end-to-end speedups; a placement that optimizes these proxies could still fail to improve wall-clock time, so the speedups are empirical rather than forced. There are no load-bearing self-citations: the reference list contains no works by the present authors, and the cited prior work on inter-layer expert affinity (ExFlow) is external. There is also no imported uniqueness theorem or ansatz smuggled in via citation. The closest concern is evaluation in-sample: profiling and measurement use the same task datasets, and the claim that 'the routing patterns of the sampled dataset reliably approximate the overall routing behavior across the full dataset for a given task' (Section 4.1) is asserted with only a consistency plot rather than tested on held-out tasks or distributions. That is a generalization and validity limitation, not circularity, because the reported speedups are not a fitted parameter renamed as a prediction. Separately, the formulation-soundness issue that ILP1's only placement constraint (Eq. 7) is sum_e x_{c,e,l} >= 1, without an exclusivity constraint requiring each expert to be assigned to exactly one cluster, is a correctness concern about the optimization problem as written, not a circular-derivation step.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method depends on measured routing statistics as input. The main unstated costs are the stability of those statistics, the linear models for compute and communication time, and the unproven optimality of decomposing the placement problem into two sequential ILPs. No new physical entities are introduced.

free parameters (3)
  • Profiling subset size and number of profiling iterations
    Section 4.1 says a small subset is used over a predefined number of iterations, but neither is specified; the placement and speedups depend on this hand-chosen amount.
  • ILP optimality gap tolerance = 0.025
    Section 5.1 sets the Gurobi tolerance to 0.025; the solver may stop with a 2.5% gap, which can change the placement and measured speedup, with no sensitivity analysis.
  • Warmup and measurement step counts = 100 warmup, 100 measurement steps for end-to-end; 100 warmup, 10 profiled steps for latency
    Section 5.2 fixes these counts by hand without studying their effect on the reported averages.
assumptions (4)
  • domain assumption Routing statistics from a sampled subset generalize to the full dataset and to the serving workload
    Section 4.1 relies on this to justify profiling only a small subset; if routing drifts, the optimized placement is stale.
  • domain assumption Token processing load is proportional to the profiled token count Pe,l
    Equation 2 defines cluster load as the sum of Pe,l over assigned experts, ignoring variable token lengths, padding, and other compute overheads.
  • domain assumption All-to-all latency is proportional to the maximum over GPU pairs of token volume divided by bandwidth
    Equation 8 uses this linear model for communication cost, which abstracts away NCCL algorithms, message sizes, and topology effects.
  • ad hoc to paper The two-stage ILP decomposition preserves the claimed joint optimum
    The paper claims an optimal placement strategy, but ILP1 fixes cluster membership before ILP2 optimizes communication; no proof is given that this decomposition achieves the joint optimum of load and communication.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing." pith.science (2026). https://pith.science/paper/IWFB3USP

@misc{pith2026250206643,
  author       = {Pith},
  title        = {Pith review of: MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IWFB3USP}},
  note         = {Machine review of arXiv:2502.06643}
}
read the original abstract

Mixture-of-Experts (MoE) model architecture has emerged as a promising solution for scaling transformer models efficiently, offering sparse activation that reduces computational costs while increasing model capacity. However, as MoE models scale, they need to be distributed across GPU devices, thus face critical performance bottlenecks due to their large memory footprint. Expert parallelism distributes experts across GPUs, however, faces key challenges including an unbalanced token routing and expert activation, resulting in communication tail latency and processing inefficiencies. While existing solutions address some of these issues, they fail to resolve the dual challenges of load imbalance and communication skew. The imbalance in token processing load across experts causes uneven processing times on different GPUs, while communication skew between GPUs leads to unbalanced inter-GPU data transfers. These factors degrade the performance of MoE models by increasing tail latency and reducing overall throughput. To address these limitations, we propose an Integer Linear Programming (ILP) formulation to optimize expert placement by jointly considering token load, communication, and computation costs. We exploit the property that there is a token routing dependency across layers, where tokens routed to a specific expert in one layer are likely to be routed to a limited set of experts in the subsequent layer. Our solution, MoETuner, offers an optimal expert-to-GPU assignment that minimizes inter-GPU token routing costs and balances token processing across devices, thereby reducing tail latency and end-to-end execution time. Experimental results demonstrate 9.3% and 17.5% of end-to-end speedups for single-node and multi-node inference respectively, showcasing the potential of our ILP-based optimization for offering expert parallel solutions for next-generation MoEs.

Figures

Figures reproduced from arXiv: 2502.06643 by the authors.

Figure 1
Figure 1. Token routing statistics for Mixtral-8x7B. Each [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An example of Mixture-of-Experts (MoE) model [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Time distribution of representative operations dur [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Expert activation frequency of Mixtral-8x7B, high [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Dependency table of Mixtral-8x7B illustrating inter [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Overview of the MOETUNER framework [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Token routing statistics to different experts during [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: End-to-end inference performance, normalized to [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Token routing statistics for Mixtral-8x7B, with [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 12
Figure 12. Figure 12: Distribution of tokens processed by a single GPU [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 14
Figure 14. Figure 14: Distribution of total token dispatching between [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]
Figure 13
Figure 13. Figure 13: Comparison of average and tail latency of all-to-all [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement

    cs.LG 2026-06 conditional novelty 6.5 of 10

    Prediction-driven online expert placement with a (1+ε)-approx polynomial optimizer and compute-overlapped migration reduces MoE serving latency 11-55% versus offline and reactive baselines.

  2. Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference

    cs.DC 2026-07 conditional novelty 6.0 of 10

    Communication-aware expert placement plus device-level pruning yields 1.23–1.86× MoE inference throughput and better accuracy at equal speedup than load-balance or sequential baselines.

  3. Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory

    cs.DC 2026-05 unverdicted novelty 6.0 of 10

    Expert-parallel scaling leaves per-expert routing imbalance flat; mock-token benchmarks overestimate real-text imbalance and fake a batch-size trend; architectures split into data-resilient (MHA, Mamba-2) and persiste...

  4. HD-MoE: Hybrid and Dynamic Parallelism for Mixture-of-Expert LLMs with 3D Near-Memory Processing

    cs.PF 2025-09 conditional novelty 6.0 of 10

    HD-MoE combines an offline linear-programming placement search with online expert pre-broadcast, cutting simulated MoE inference latency on 3D near-memory processors by 1.1-1.8x over tensor parallelism.

Reference graph

Works this paper leans on

46 extracted references · 21 canonical work pages · cited by 4 Pith papers

  1. [1]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christo- pher Hesse, Mark Chen, Eric Sigler, Mateusz Lit...

    work page 2020

  2. [2]

    Shortcut-connected expert par- allelism for accelerating mixture-of-experts

    Weilin Cai, Juyong Jiang, Le Qin, Junwei Cui, Sunghun Kim, and Jiayi Huang. Shortcut-connected expert par- allelism for accelerating mixture-of-experts. arXiv preprint arXiv:2404.05019, 2024

    arXiv 2024

  3. [3]

    A survey on mixture of experts

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint arXiv:2407.06204, 2024

    arXiv 2024

  4. [4]

    Ta-moe: Topology-aware large scale mixture- of-expert training

    Chang Chen, Min Li, Zhihua Wu, Dianhai Yu, and Chao Yang. Ta-moe: Topology-aware large scale mixture- of-expert training. Advances in Neural Information Processing Systems, 35:22173–22186, 2022

    work page 2022

  5. [5]

    Switch transformers: scaling to trillion parameter models with simple and efficient sparsity.J

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity.J. Mach. Learn. Res., 23(1), January 2022

    2022

  6. [6]

    Gurobi Optimizer Refer- ence Manual, 2024

    Gurobi Optimization, LLC. Gurobi Optimizer Refer- ence Manual, 2024. https://www.gurobi.com

    work page 2024

  7. [7]

    Fastmoe: A fast mixture-of-expert training system

    Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. Fastmoe: A fast mixture-of-expert training system. arXiv preprint arXiv:2103.13262, 2021

    arXiv 2021

  8. [8]

    Fastermoe: modeling and optimizing training of large-scale dy- namic pre-trained models

    Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. Fastermoe: modeling and optimizing training of large-scale dy- namic pre-trained models. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 120–134, 2022

    work page 2022

Show all 46 references
  1. [9]

    Towards moe de- ployment: Mitigating inefficiencies in mixture-of-expert (moe) inference

    Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Hsien-Hsin S Lee, Anjali Sridhar, Shruti Bhosale, Carole-Jean Wu, and Benjamin Lee. Towards moe de- ployment: Mitigating inefficiencies in mixture-of-expert (moe) inference. arXiv preprint arXiv:2303.06182 , 2023

    2023 arXiv

  2. [10]

    Gpipe: Effi- cient training of giant neural networks using pipeline parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Effi- cient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019

    2019

  3. [11]

    Huggingface Hub documentation, 2024

    Huggingface. Huggingface Hub documentation, 2024. https://huggingface.co/docs/hub/en/index

    2024

  4. [12]

    The human knowledge compression contest, 2006

    Marcus Hutter. The human knowledge compression contest, 2006. http://prize.hutter1.net

    2006

  5. [13]

    Tutel: Adaptive mixture-of-experts at scale

    Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. Tutel: Adaptive mixture-of-experts at scale. Proceedings of Machine Learning and Systems, 5:269–287, 2023

    2023

  6. [14]

    Pre- gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference

    Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang. Pre- gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), page...

    2024

  7. [15]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-An...

    2024 arXiv

  8. [16]

    Lancet: Acceler- ating mixture-of-experts training via whole graph computation-communication overlapping

    Chenyu Jiang, Ye Tian, Zhen Jia, Shuai Zheng, Chuan Wu, and Yida Wang. Lancet: Acceler- ating mixture-of-experts training via whole graph computation-communication overlapping. arXiv preprint arXiv:2404.19429, 2024

    2024 arXiv

  9. [17]

    The minipile challenge for data-efficient language models

    Jean Kaddour. The minipile challenge for data-efficient language models. arXiv preprint arXiv:2304.08442 , 2023

    2023 arXiv

  10. [18]

    Scalable and efficient moe train- ing for multitask multilingual models

    Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andres Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan Awadalla. Scalable and efficient moe train- ing for multitask multilingual models. arXiv preprint arXiv:2109.10465, 2021

    2021 arXiv

  11. [19]

    Gonza- lez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonza- lez, Hao Zhang, and Ion Stoica. Efficient memory man- 13 agement for large language model serving with page- dattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operatin...

    2023

  12. [20]

    Gshard: Scaling gi- ant models with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, De- hao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling gi- ant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020

    2006 arXiv

  13. [21]

    Accelerating distributed {MoE} training and inference with lina

    Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. Accelerating distributed {MoE} training and inference with lina. In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pages 945–959, 2023

    2023

  14. [22]

    Merge, then compress: Demystify efficient smoe with hints from its routing policy

    Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient smoe with hints from its routing policy. arXiv preprint arXiv:2310.01334 , 2023

    2023 arXiv

  15. [23]

    Pytorch dis- tributed: Experiences on accelerating data parallel train- ing

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch dis- tributed: Experiences on accelerating data parallel train- ing. arXiv preprint arXiv:2006.15704, 2020

    2006 arXiv

  16. [24]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

    2016

  17. [25]

    Pipedream: Gen- eralized pipeline parallelism for dnn training

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Gen- eralized pipeline parallelism for dnn training. In Pro- ceedings of the 27th ACM symposium on operating sys- tems princip...

    2019

  18. [26]

    Memory-efficient pipeline- parallel dnn training

    Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. Memory-efficient pipeline- parallel dnn training. In International Conference on Machine Learning, pages 7937–7947. PMLR, 2021

    2021

  19. [27]

    NVIDIA H100 Tensor Core GPU, 2023

    NVIDIA. NVIDIA H100 Tensor Core GPU, 2023. https://resources. nvidia.com/en-us-tensor-core/ nvidia-tensor-core-gpu-datasheet

    2023

  20. [28]

    CUDA Toolkit, 2024

    NVIDIA. CUDA Toolkit, 2024. https://developer. nvidia.com/cuda-toolkit

    2024

  21. [29]

    NVIDIA Collective Communications Li- brary (NCCL), 2024

    NVIDIA. NVIDIA Collective Communications Li- brary (NCCL), 2024. https://developer.nvidia. com/nccl

    2024

  22. [30]

    NVIDIA H200 Tensor Core GPU, 2024

    NVIDIA. NVIDIA H200 Tensor Core GPU, 2024. https://resources.nvidia. com/en-us-data-center-overview-mc/ en-us-data-center-overview/ hpc-datasheet-sc23-h200

    2024

  23. [31]

    The lambada dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germán Kruszewski, Angeliki Lazari- dou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016

    2016 arXiv

  24. [32]

    Py- torch: An imperative style, high-performance deep learn- ing library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Py- torch: An imperative style, high-performance deep learn- ing library. Advances in neural information processing systems,...

    2019

  25. [33]

    PyTorch Profiler, February 2023

    PyTorch. PyTorch Profiler, February 2023. https://pytorch.org/tutorials/recipes/ recipes/profiler_recipe.html

    2023

  26. [34]

    Language mod- els are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language mod- els are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  27. [35]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  28. [36]

    Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Min- jia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. In International confer- ence on machine le...

    2022

  29. [37]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion pa- rameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion pa- rameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages...

    2020

  30. [38]

    Red Hat Enterprise Linux 9, 2024

    Red Hat, Inc. Red Hat Enterprise Linux 9, 2024. https://docs.redhat.com/en/documentation/ red_hat_enterprise_linux/9

    2024

  31. [39]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  32. [40]

    14 Megatron-lm: Training multi-billion parameter lan- guage models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 14 Megatron-lm: Training multi-billion parameter lan- guage models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  33. [41]

    A hybrid tensor-expert-data parallelism approach to optimize mixture-of-experts training

    Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, and Abhi- nav Bhatele. A hybrid tensor-expert-data parallelism approach to optimize mixture-of-experts training. In Proceedings of the 37th International Conference on Supercomputing, pages 203...

    2023

  34. [42]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Lau- rent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Al...

    2024 arXiv

  35. [43]

    Llama: Open and efficient foun- dation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap- tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foun- dation lang...

    2023 arXiv

  36. [44]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  37. [45]

    Exploiting inter- layer expert affinity for accelerating mixture-of-experts model inference

    Jinghan Yao, Quentin Anthony, Aamir Shafi, Hari Sub- ramoni, and Dhabaleswar K DK Panda. Exploiting inter- layer expert affinity for accelerating mixture-of-experts model inference. In 2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS) , pages 915–92...

    2024

  38. [46]

    Pytorch fsdp: experiences on scaling fully sharded data parallel

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Sho- janazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023. 15

    2023 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.