Pith. sign in

REVIEW 4 major objections 5 minor 52 references

Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A system called Luffy claims that the all-to-all token traffic that dominates distributed Mixture-of-Experts training can be largely removed by migrating whole sequences toward their tokens and condensing near-duplicate tokens, achieving…

desk verdict The per-iteration 2.73x speedup is internally consistent, but the paper's central time-to-accuracy claim is not actually measured, and token condensation changes the training objective in a way that the evaluation does not rule out. read the letter →

arxiv 2411.15419 v1 pith:SVGALYVO submitted 2024-11-23 cs.DC

Fahao Chen , Peng Li , Zicong Hong , Zhou Su , Song Guo This is my paper · ORCID
classification cs.DC
keywords Mixture-of-Expertsdistributedtrainingexpertparallelismall-to-allcommunicationtokencondensationsequencemigration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that distributed training of Mixture-of-Experts (MoE) models can be made much faster without sacrificing expert parallelism by reducing the number of tokens that travel between GPUs. It proposes Luffy, a system that relocates whole sequences to the GPU that already holds most of their tokens, and that identifies similar tokens heading to the same expert and sends only one representative. On a 16-GPU testbed, Luffy speeds up batch training by up to 2.73x over a standard expert-parallel baseline while keeping final model quality nearly unchanged. The implication is that the all-to-all token traffic that dominates expert-parallel training time is largely redundant and can be removed.

What carries the argument

The two load-bearing mechanisms are the sequence migration algorithm and the token condensation module. Sequence migration uses a cost model for attention time, $T_{att}(B,L)=(3BLd^2+2BL^2d)/P$, to decide which GPU should re-assemble each sequence, balancing token-pull traffic against attention workload balance and length-similar packing that reduces padding. Token condensation builds a token-graph whose edges encode similarity, uses expert activation and historical similarity to skip most pair-wise comparisons, and applies an adaptive threshold $h_t = 1/(1+\exp(l_{norm}))$ that lets the fraction of condensed tokens grow as training stabilizes. The receiver reuses one expert output for a group of similar tokens, which cuts both dispatch traffic and expert computation.

What would settle it

Train a MoE model on a dataset where tokens routed to the same expert are deliberately made dissimilar (e.g., by shuffling sentences so each sequence contains heterogeneous topics) and measure the loss curve; if the speedup from condensation vanishes or accuracy drops substantially at the same adaptive thresholds, the similarity assumption fails. Concretely, run Luffy with condensation disabled versus enabled on a controlled corpus with low within-expert token similarity, and compare final task metrics.

Watch

Extended reading notes

Core claim

The paper's central claim is that the all-to-all token dispatch and combine traffic in expert-parallel MoE training can be reduced without moving experts between GPUs, by migrating sequences in the combine phase and condensing similar tokens in the dispatch phase. Sequence migration replaces the costly pull of scattered tokens back to their origin GPU with a cheaper re-location of the whole sequence, and token condensation exploits the observation that many tokens routed to the same expert are near-duplicates, so transmitting one and reusing its expert output is enough. Together these techniques cut both communication and expert computation, giving up to 2.73x batch speedup over the vanilla expert-parallel baseline with final accuracy close to full training (e.g., WikiText-103 perplexity 25.28 vs 25.13, SQuAD F1 89.17 vs 90.82).

Load-bearing premise

Token condensation is safe: replacing many similar tokens with one representative token and reusing its expert output does not harm training convergence, an assumption validated only empirically on three model/dataset pairs.

Editorial extensions

If this is right

  • If correct, expert-parallel MoE training can scale to more experts without communication dominating the iteration time.
  • Token condensation offers a compression axis orthogonal to existing all-to-all scheduling techniques, so it can be combined with overlap-based methods.
  • Sequence migration turns the combine phase from a token-pull into a sequence-relocation decision, opening the door to attention-aware placement.
  • The speedup grows with expert count, suggesting the benefit increases precisely where vanilla expert parallelism degrades most.
  • Accuracy preservation with the adaptive threshold indicates a tunable trade-off between training speed and model quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit consequence is that token condensation amounts to a form of lossy activation compression defined by token similarity; its safety should depend on how redundant the routing distribution is, so workloads with highly uniform expert routing may see smaller gains.
  • The historical-similarity heuristic could be stress-tested by adversarially permuting token order across blocks; if similarity relationships are an artifact of sentence structure, migration benefits may not transfer to other modalities.
  • The adaptive threshold ties condensation rate to loss decrease; a testable extension is to tie it instead to per-expert validation metrics or to a target communication budget, which would make the trade-off more predictable.
  • Since the paper reports results on PCIe-connected GPUs, an inference is that on higher-bandwidth interconnects the relative speedup from traffic reduction shrinks but the expert-computation savings from condensation remains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents LUFFY, a distributed MoE training system that reduces all-to-all traffic without moving experts, through two mechanisms: sequence migration (reconstructing sequences at the GPU that hosts most of their tokens) and token condensation (identifying similar tokens and sending only one representative to the expert). It introduces heuristics for migration decisions based on an attention-cost model, a fast similarity measurement exploiting expert activation and historical similarity, and an adaptive condensation threshold controlled by the training loss. On a 16-GPU V100 testbed with three MoE models, LUFFY reports up to 2.73x per-iteration speedup over a DeepSpeed-style Vanilla baseline and final accuracy close to that of the baseline (e.g., WikiText-103 PPL 25.28 vs. 25.13; SQuAD F1 89.17 vs. 90.82).

Significance. If the reported results hold, LUFFY offers a credible alternative to expert-transfer-based approaches: it reduces the dominant all-to-all communication in both dispatch and combine phases while preserving expert-level parallelism, and the ablation study (Fig. 9) shows that both proposed components contribute. The paper ships a real PyTorch implementation, a calibrated attention-cost model with about 5% error, and a convergence check showing that the adaptive condensation policy avoids the large accuracy drop of a fixed low threshold. The work is relevant to systems venues for distributed MoE training. However, the headline claim is 'improving time-to-accuracy,' and the current evidence supports per-iteration speedup and final quality at an unspecified training budget, not wall-clock time-to-accuracy.

major comments (4)
  1. [§III, §VII-B, §VII-E, Eq. (2)] The paper states in §III that LUFFY's goal is improving time-to-accuracy, but Fig. 8 reports only normalized per-iteration batch time, not wall-clock time to a target accuracy. This gap is consequential because token condensation changes the training objective: for a condensed token j, the forward pass uses the representative expert output while the loss at position j is still evaluated against j's own label, and the gradient with respect to the condensed token is zero in that path. Eq. (2) makes the procedure most aggressive late in training, when the normalized loss decrease is largest and gradient magnitudes are small, so the approximation bias need not vanish. Table IV reports final PPL/F1/ROUGE at a single, unspecified training budget, with no number of iterations, no wall-clock time, and no learning curves comparing LUFFY with Vanilla; Fig. 10(d) shows loss-versus-time only for similarity-threshold configurations, not for the headline comparison. As written, the evaluation supports 'faster batch' and 'similar final quality at the same budget,' but not the claimed time-to-accuracy improvement.
  2. [§IV-A, Algorithm 1] Algorithm 1 contains an apparent sign/objective contradiction. Line 5 defines s_i,j = T_att(B_{j←i}, L_{j←i}) − T_att(B_j, L_j), i.e., the increase in attention cost if sequence i is migrated to GPU j. The surrounding text says the algorithm selects the GPU with the minimum cost growth, but line 6 says 'Migrate sequence i to the GPU j* with maximum s_i,j.' Since the cost model is monotone in B and L, maximizing s_i,j would select the worst candidate. Please correct the pseudocode (minimum s_i,j) or clarify the intended sign convention.
  3. [§VII-A, Abstract] The abstract and §VII claim superiority over 'state-of-the-art MoE training systems,' but the implemented baselines are only Vanilla, EXT, and HYT. Tutel [8] and Lina [11], both discussed at length in §VIII and both directly targeting the all-to-all bottleneck, are not measured. Direct experiments against at least one of these systems would be needed to support the stated competitive claim; otherwise, the claim should be restricted to the three implemented baselines.
  4. [§VII-B–VII-E, Table IV] All speedup and accuracy results are reported as single numbers, with no error bars, no repeated-seed statistics, and no variance information. This is especially problematic for Table IV: the PPL difference 25.28 vs. 25.13 is smaller than typical run-to-run variation for models of this size, and the F1 drop of 1.65 points has no confidence interval. Repeated runs with different random seeds for at least the main configurations are needed to establish that adaptive condensation preserves accuracy rather than shifting within noise.
minor comments (5)
  1. [§V-B] The text says 'We use an exponential function here,' but Eq. (2) is a logistic/sigmoid function; please align the wording with the equation.
  2. [§II-B] The 'normalized cosine similarity' is never formally defined; please specify how raw cosine similarity is normalized to the [0,1] range used in Fig. 5.
  3. [Throughout] The system name is spelled inconsistently ('LUFFY', 'Luffy', 'L UFFY'); use one form consistently.
  4. [§VII-E, Table IV] The column headings mix 'Luffy (h=0.3)', 'Luffy (h=0.8)', and 'LUFFY'; clarify that the last column is the adaptive policy and report the number of training iterations or steps used for each model.
  5. [§VII-A] The description 'Vanilla: the MoE implementation with expert parallelism, adopted by DeepSpeed [7]' is potentially misleading, since DeepSpeed-MoE includes additional communication optimizations beyond naive expert parallelism; state precisely which components of DeepSpeed are used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: speedups are measured, the attention cost model is validated against independent profiling, and the adaptive condensation threshold is feedback control, not a fitted prediction.

full rationale

The paper's central claims are supported by direct measurement rather than by self-referential derivation. Sequence migration is driven by a cost model T_att(B,L) = (3BLd^2 + 2BL^2d)/P, where P is profiled from hardware and the model is validated against real attention costs with about 5% average error (Fig. 10b); it is not fitted to reproduce the reported speedups. Token condensation uses empirical cosine-similarity statistics and an adaptive threshold h_t = 1/(1+exp(l_norm)) based on the previous iteration's loss decrease. This is feedback control: the loss influences how aggressively tokens are condensed, and the resulting training loss then influences the next threshold. It is not a case of defining a quantity in terms of the result it is supposed to predict. The headline 2.73x speedup is obtained by normalizing measured average per-iteration batch times against the Vanilla baseline (Fig. 8), and Table IV reports independently evaluated final accuracy metrics. I found no load-bearing self-citation chain, no imported uniqueness theorem, and no fitted parameter renamed as a prediction. The skeptical concern that per-iteration speedup may not translate to wall-clock time-to-accuracy is a validation gap, not a circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central contributions are empirical system optimizations. The main assumptions are domain assumptions about similarity preservation and approximating attention cost; the migration heuristic has user-set parameters, and the cost model has one profiled hardware speed parameter.

free parameters (3)
  • q (candidate GPU set size in Algorithm 1) = 2-5 in sensitivity analysis; default not stated
    Controls the trade-off between combine-phase traffic and attention batching efficiency; user-selected and not learned from data.
  • S1 and S2 (historical similarity thresholds) = e.g., S1=0.8, S2=0.2
    Determines which token pairs are classified as similar or dissimilar without cosine computation; sensitive to measurement cost and convergence.
  • GPU speed P in attention cost model = profiled per GPU
    Calibrated by running attention layers several times; used in Eq. (1) to estimate attention computation time.
assumptions (4)
  • domain assumption Condensing similar tokens to one representative does not materially change training outcomes
    Empirical observation in Figure 5 and Table IV; no proof; load-bearing for the token condensation technique.
  • domain assumption Token similarity is preserved after expert execution
    Figure 5(b) supports this; needed so condensed tokens can reuse the expert output of the representative token.
  • domain assumption Historical similarity persists across consecutive blocks
    Figure 7 supports this; basis of the fast similarity measurement in Section V-A.
  • domain assumption Attention cost model T_att(B,L) captures real relative cost
    Linear projection and dot-product FLOPs divided by profiled speed P; validated with about 5% error in Figure 10(b) but assumes compute-bound attention and ignores softmax.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation." pith.science (2026). https://pith.science/paper/SVGALYVO

@misc{pith2026241115419,
  author       = {Pith},
  title        = {Pith review of: Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SVGALYVO}},
  note         = {Machine review of arXiv:2411.15419}
}
read the original abstract

Mixture-of-Experts (MoE) is an emerging technique for scaling large models with sparse activation. MoE models are typically trained in a distributed manner with an expert parallelism scheme, where experts in each MoE layer are distributed across multiple GPUs. However, the default expert parallelism suffers from the heavy network burden due to the all-to-all intermediate data exchange among GPUs before and after the expert run. Some existing works have proposed to reduce intermediate data exchanges by transferring experts to reduce the network loads, however, which would decrease parallelism level of expert execution and make computation inefficient. The weaknesses of existing works motivate us to explore whether it is possible to reduce inter-GPU traffic while maintaining a high degree of expert parallelism. This paper gives a positive response by presenting Luffy, a communication-efficient distributed MoE training system with two new techniques. First, Luffy migrates sequences among GPUs to hide heavy token pulling paths within GPUs and avoid copying experts over GPUs. Second, we propose token condensation that identifies similar tokens and then eliminates redundant transmissions. We implement Luffy based on PyTorch and evaluate its performance on a testbed of 16 V100 GPUs. Luffy system can achieve a speedup of up to 2.73x compared to state-of-the-art MoE training systems.

Figures

Figures reproduced from arXiv: 2411.15419 by the authors.

Figure 1
Figure 1. Comparison between existing works and our ideas. We assume that each GPU holds one expert (e.g., GPU [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An illustration of MoE. Model Expert=4, Batch = 8 Expert=4, Batch = 16 Expert=8, Batch = 8 S (GB) C (ms) R (%) S (GB) C (ms) R (%) S (GB) C (ms) R (%) MoE-TransformerXL 3.19 327 18.1 6.15 507 14.8 3.98 381 30.5 MoE-BERT-Large 6.73 439 36.6 13.07 859 40.3 7.92 477 47.5 MoE-GPT2 6.53 411 34.6 12.13 707 35.9 7.52 452 45.9 TABLE I: Communication bottleneck for distributed MoE training. The total number of experts is set… view at source ↗
Figure 5
Figure 5. Token similarity and the change after the expert [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Batch time on one GPU with different number of [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: Architecture overview of LUFFY. little consideration of expert or attention computation. Third, LUFFY should not compromise the training convergence, and thus preserve the quality of the final MoE model. LUFFY can allow a certain level of computational approximation du…
Figure 7
Figure 7. Figure 7: Similarity change across consecutive blocks for the [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Batch training time speedup of different MoE training systems. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Performance improvements of the optimizations separately. We use Vanilla as the baseline and show speedups of [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Sensitivity analysis on migration algorithm and fast similarity measurement. [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 14 canonical work pages

  1. [8]

    Tutel: Adaptive mixture-of-experts at scale,

    C. Hwang, W. Cui, Y . Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram et al. , “Tutel: Adaptive mixture-of-experts at scale,” Proceedings of Machine Learning and Systems , vol. 5, 2023

    2023

  2. [11]

    Accelerating distributed {MoE} training and inference with lina,

    J. Li, Y . Jiang, Y . Zhu, C. Wang, and H. Xu, “Accelerating distributed {MoE} training and inference with lina,” in 2023 USENIX Annual Technical Conference (USENIX ATC 23) , 2023, pp. 945–959

    2023

  3. [1]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  4. [2]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  5. [3]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al. , “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

    2019

  6. [4]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361 , 2020

    arXiv 2001

  7. [5]

    Gshard: Scaling giant models with conditional computation and automatic sharding,

    D. Lepikhin, H. Lee, Y . Xu, D. Chen, O. Firat, Y . Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” arXiv preprint arXiv:2006.16668 , 2020

    arXiv 2006

  8. [6]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,”The Journal of Machine Learning Research , vol. 23, no. 1, pp. 5232–5270, 2022

    2022

Show all 52 references
  1. [7]

    Deepspeed-moe: Advancing mixture-of- experts inference and training to power next-generation ai scale,

    S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y . Aminabadi, A. A. Awan, J. Rasley, and Y . He, “Deepspeed-moe: Advancing mixture-of- experts inference and training to power next-generation ai scale,” in International Conference on Machine Learning . PMLR, 2022, pp. 18 332–18 346

    2022

  2. [9]

    Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement,

    X. Nie, X. Miao, Z. Wang, Z. Yang, J. Xue, L. Ma, G. Cao, and B. Cui, “Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement,” Proceedings of the ACM on Management of Data , vol. 1, no. 1, pp. 1–19, 2023

    2023

  3. [10]

    Janus: A unified distributed training framework for sparse mixture-of-experts models,

    J. Liu, J. H. Wang, and Y . Jiang, “Janus: A unified distributed training framework for sparse mixture-of-experts models,” in Proceedings of the ACM SIGCOMM 2023 Conference , 2023, pp. 486–498

    2023

  4. [12]

    Gating dropout: Communication-efficient regularization for sparsely activated transform- ers,

    R. Liu, Y . J. Kim, A. Muzio, and H. Hassan, “Gating dropout: Communication-efficient regularization for sparsely activated transform- ers,” in International Conference on Machine Learning . PMLR, 2022, pp. 13 782–13 792

    2022

  5. [13]

    Faster- moe: modeling and optimizing training of large-scale dynamic pre- trained models,

    J. He, J. Zhai, T. Antunes, H. Wang, F. Luo, S. Shi, and Q. Li, “Faster- moe: modeling and optimizing training of large-scale dynamic pre- trained models,” in Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 2022, pp. 120–134

    2022

  6. [14]

    Dota: detect and omit weak attentions for scalable transformer acceleration,

    Z. Qu, L. Liu, F. Tu, Z. Chen, Y . Ding, and Y . Xie, “Dota: detect and omit weak attentions for scalable transformer acceleration,” in Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems , 2022, pp. 14–26

    2022

  7. [15]

    Vitcod: Vision transformer acceleration via dedicated algorithm and accelerator co-design,

    H. You, Z. Sun, H. Shi, Z. Yu, Y . Zhao, Y . Zhang, C. Li, B. Li, and Y . Lin, “Vitcod: Vision transformer acceleration via dedicated algorithm and accelerator co-design,” in 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA) . IEEE, 2023, pp. 273–286

    2023

  8. [16]

    Scaling vision with sparse mixture of experts,

    C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby, “Scaling vision with sparse mixture of experts,” Advances in Neural Information Processing Systems, vol. 34, pp. 8583–8595, 2021

    2021

  9. [17]

    Designing effective sparse expert models,

    B. Zoph, I. Bello, S. Kumar, N. Du, Y . Huang, J. Dean, N. Shazeer, and W. Fedus, “Designing effective sparse expert models,” arXiv preprint arXiv:2202.08906, vol. 2, 2022

    2022 arXiv

  10. [18]

    Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale,

    R. Y . Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley et al., “Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale,” in SC22: International Conference for High Performance Com- ...

    2022

  11. [19]

    Bagualu: targeting brain scale pretrained models with over 37 million cores,

    Z. Ma, J. He, J. Qiu, H. Cao, Y . Wang, Z. Sun, L. Zheng, H. Wang, S. Tang, T. Zheng et al. , “Bagualu: targeting brain scale pretrained models with over 37 million cores,” in Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Program- ming, 2...

    2022

  12. [20]

    Fastmoe: A fast mixture-of-expert training system,

    J. He, J. Qiu, A. Zeng, Z. Yang, J. Zhai, and J. Tang, “Fastmoe: A fast mixture-of-expert training system,” arXiv preprint arXiv:2103.13262 , 2021

    2021 arXiv

  13. [21]

    {SmartMoE}: Efficiently training {Sparsely-Activated} models through combining offline and online parallelization,

    M. Zhai, J. He, Z. Ma, Z. Zong, R. Zhang, and J. Zhai, “ {SmartMoE}: Efficiently training {Sparsely-Activated} models through combining offline and online parallelization,” in 2023 USENIX Annual Technical Conference (USENIX ATC 23) , 2023, pp. 961–975

    2023

  14. [22]

    Transformer-xl: Attentive language models beyond a fixed-length context,

    Z. Dai, Z. Yang, Y . Yang, J. Carbonell, Q. V . Le, and R. Salakhutdi- nov, “Transformer-xl: Attentive language models beyond a fixed-length context,” arXiv preprint arXiv:1901.02860 , 2019

    1901 arXiv

  15. [23]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand et al. , “Mixtral of experts,” arXiv preprint arXiv:2401.04088 , 2024

    2024 arXiv

  16. [24]

    Evaluating the stability of embedding- based word similarities,

    M. Antoniak and D. Mimno, “Evaluating the stability of embedding- based word similarities,” Transactions of the Association for Computa- tional Linguistics , vol. 6, pp. 107–119, 2018

    2018

  17. [25]

    Sentiment classification using docu- ment embeddings trained with cosine similarity,

    T. Thongtan and T. Phienthrakul, “Sentiment classification using docu- ment embeddings trained with cosine similarity,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop , 2019, pp. 407–414

    2019

  18. [26]

    Problems with cosine as a measure of embedding similarity for high frequency words,

    K. Zhou, K. Ethayarajh, D. Card, and D. Jurafsky, “Problems with cosine as a measure of embedding similarity for high frequency words,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 2: Short Papers) , 2022, pp. 401– 423

    2022

  19. [27]

    Pangu-π: Enhancing language model architec- tures via nonlinearity compensation,

    Y . Wang, H. Chen, Y . Tang, T. Guo, K. Han, Y . Nie, X. Wang, H. Hu, Z. Bai, Y . Wang et al. , “Pangu-π: Enhancing language model architec- tures via nonlinearity compensation,” arXiv preprint arXiv:2312.17276 , 2023

    2023 arXiv

  20. [28]

    Turbotransformers: an efficient gpu serving system for transformer models,

    J. Fang, Y . Yu, C. Zhao, and J. Zhou, “Turbotransformers: an efficient gpu serving system for transformer models,” in Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 2021, pp. 389–402

    2021

  21. [29]

    Flashattention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,”Advances in Neural Information Processing Systems , vol. 35, pp. 16 344–16 359, 2022

    2022

  22. [30]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al. , “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems , vol. 32, 2019

    2019

  23. [31]

    Deep graph library: Towards efficient and scalable deep learning on graphs,

    M. Y . Wang, “Deep graph library: Towards efficient and scalable deep learning on graphs,” in ICLR workshop on representation learning on graphs and manifolds , 2019

    2019

  24. [32]

    Pointer sentinel mix- ture models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mix- ture models,” in International Conference on Learning Representations , 2022

    2022

  25. [33]

    Squad: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, “Squad: 100,000+ questions for machine comprehension of text,” arXiv preprint arXiv:1606.05250 , 2016

    2016 arXiv

  26. [34]

    Samsum corpus: A human-annotated dialogue dataset for abstractive summarization,

    B. Gliwa, I. Mochol, M. Biesek, and A. Wawer, “Samsum corpus: A human-annotated dialogue dataset for abstractive summarization,” in Proceedings of the 2nd Workshop on New Frontiers in Summarization , 2019, pp. 70–79

    2019

  27. [35]

    Language models with image descriptors are strong few-shot video-language learners,

    Z. Wang, M. Li, R. Xu, L. Zhou, J. Lei, X. Lin, S. Wang, Z. Yang, C. Zhu, D. Hoiem et al. , “Language models with image descriptors are strong few-shot video-language learners,” Advances in Neural Informa- tion Processing Systems , vol. 35, pp. 8483–8497, 2022

    2022

  28. [36]

    Multitask mixture of sequential experts for user activity streams,

    Z. Qin, Y . Cheng, Z. Zhao, Z. Chen, D. Metzler, and J. Qin, “Multitask mixture of sequential experts for user activity streams,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2020, pp. 3083–3091

    2020

  29. [37]

    Taming sparsely activated transformer with stochastic experts,

    S. Zuo, X. Liu, J. Jiao, Y . J. Kim, H. Hassan, R. Zhang, T. Zhao, and J. Gao, “Taming sparsely activated transformer with stochastic experts,” arXiv preprint arXiv:2110.04260 , 2021

    2021 arXiv

  30. [38]

    Generalizable person re- identification with relevance-aware mixture of experts,

    Y . Dai, X. Li, J. Liu, Z. Tong, and L.-Y . Duan, “Generalizable person re- identification with relevance-aware mixture of experts,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2021, pp. 16 145–16 154. 12

    2021

  31. [39]

    Vlmo: Unified vision-language pre-training with mixture-of-modality-experts,

    H. Bao, W. Wang, L. Dong, Q. Liu, O. K. Mohammed, K. Aggarwal, S. Som, S. Piao, and F. Wei, “Vlmo: Unified vision-language pre-training with mixture-of-modality-experts,” Advances in Neural Information Pro- cessing Systems , vol. 35, pp. 32 897–32 912, 2022

    2022

  32. [40]

    Palm: Scal- ing language modeling with pathways,

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al. , “Palm: Scal- ing language modeling with pathways,” Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023

    2023

  33. [41]

    Glam: Efficient scaling of language models with mixture-of-experts,

    N. Du, Y . Huang, A. M. Dai, S. Tong, D. Lepikhin, Y . Xu, M. Krikun, Y . Zhou, A. W. Yu, O. Firat et al. , “Glam: Efficient scaling of language models with mixture-of-experts,” in International Conference on Machine Learning . PMLR, 2022, pp. 5547–5569

    2022

  34. [42]

    Llama-moe: Building mixture-of-experts from llama with continual pre-training,

    L.-M. Team, “Llama-moe: Building mixture-of-experts from llama with continual pre-training,” 2023

    2023

  35. [43]

    Open- moe: An early effort on open mixture-of-experts language models,

    F. Xue, Z. Zheng, Y . Fu, J. Ni, Z. Zheng, W. Zhou, and Y . You, “Open- moe: An early effort on open mixture-of-experts language models,” arXiv preprint arXiv:2402.01739 , 2024

    2024 arXiv

  36. [44]

    Deepseekmoe: Towards ultimate expert spe- cialization in mixture-of-experts language models,

    D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y . Wu et al. , “Deepseekmoe: Towards ultimate expert spe- cialization in mixture-of-experts language models,” arXiv preprint arXiv:2401.06066, 2024

    2024 arXiv

  37. [45]

    Gspmd: general and scalable parallelization for ml computation graphs,

    Y . Xu, H. Lee, D. Chen, B. Hechtman, Y . Huang, R. Joshi, M. Krikun, D. Lepikhin, A. Ly, M. Maggioni et al. , “Gspmd: general and scalable parallelization for ml computation graphs,” arXiv preprint arXiv:2105.04663, 2021

    2021 arXiv

  38. [46]

    Base layers: Simplifying training of large, sparse models,

    M. Lewis, S. Bhosale, T. Dettmers, N. Goyal, and L. Zettlemoyer, “Base layers: Simplifying training of large, sparse models,” in International Conference on Machine Learning . PMLR, 2021, pp. 6265–6274

    2021

  39. [47]

    fairseq: A fast, extensible toolkit for sequence modeling,

    M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, “fairseq: A fast, extensible toolkit for sequence modeling,” arXiv preprint arXiv:1904.01038 , 2019

    1904 arXiv

  40. [48]

    Pipemoe: Accelerating mixture- of-experts through adaptive pipelining,

    S. Shi, X. Pan, X. Chu, and B. Li, “Pipemoe: Accelerating mixture- of-experts through adaptive pipelining,” in IEEE INFOCOM 2023-IEEE Conference on Computer Communications . IEEE, 2023, pp. 1–10

    2023

  41. [49]

    Mpipemoe: Memory efficient moe for pre-trained models with adaptive pipeline parallelism,

    Z. Zhang, D. Yang, Y . Xia, L. Ding, D. Tao, X. Zhou, and D. Cheng, “Mpipemoe: Memory efficient moe for pre-trained models with adaptive pipeline parallelism,” in 2023 IEEE International Parallel and Dis- tributed Processing Symposium (IPDPS) . IEEE, 2023, pp. 167–177

    2023

  42. [50]

    Se-moe: A scalable and efficient mixture- of-experts distributed training and inference system,

    L. Shen, Z. Wu, W. Gong, H. Hao, Y . Bai, H. Wu, X. Wu, J. Bian, H. Xiong, D. Yu et al. , “Se-moe: A scalable and efficient mixture- of-experts distributed training and inference system,” arXiv preprint arXiv:2205.10034, 2022

    2022 arXiv

  43. [51]

    Alpa: Automating inter-and {Intra- Operator} parallelism for distributed deep learning,

    L. Zheng, Z. Li, H. Zhang, Y . Zhuang, Z. Chen, Y . Huang, Y . Wang, Y . Xu, D. Zhuo, E. P. Xing et al. , “Alpa: Automating inter-and {Intra- Operator} parallelism for distributed deep learning,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)...

    2022

  44. [52]

    Schemoe: An extensible mixture-of-experts distributed training system with tasks scheduling,

    S. Shi, X. Pan, Q. Wang, C. Liu, X. Ren, Z. Hu, Y . Yang, B. Li, and X. Chu, “Schemoe: An extensible mixture-of-experts distributed training system with tasks scheduling,” in Proceedings of the Nineteenth European Conference on Computer Systems , 2024, pp. 236–249

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.