Pith. sign in

REVIEW 5 cited by

Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.06563 v1 pith:RW5XONPR submitted 2024-06-03 cs.CL cs.AI

Tianwen Wei , Bo Zhu , Liang Zhao , Cheng Cheng , Biye Li , Weiwei Lü , Peng Cheng , Jianhao Zhang
show 8 more authors
Xiaoyu Zhang Liang Zeng Xiaokun Wang Yutuan Ma Rui Hu Shuicheng Yan Han Fang Yahui Zhou
This is my paper · ORCID
classification cs.CLcs.AI
keywords trainingmodelskywork-moetechniquesauxiliarycheckpointscoefficientsdense
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this technical report, we introduce the training methodologies implemented in the development of Skywork-MoE, a high-performance mixture-of-experts (MoE) large language model (LLM) with 146 billion parameters and 16 experts. It is initialized from the pre-existing dense checkpoints of our Skywork-13B model. We explore the comparative effectiveness of upcycling versus training from scratch initializations. Our findings suggest that the choice between these two approaches should consider both the performance of the existing dense checkpoints and the MoE training budget. We highlight two innovative techniques: gating logit normalization, which improves expert diversification, and adaptive auxiliary loss coefficients, allowing for layer-specific adjustment of auxiliary loss coefficients. Our experimental results validate the effectiveness of these methods. Leveraging these techniques and insights, we trained our upcycled Skywork-MoE on a condensed subset of our SkyPile corpus. The evaluation results demonstrate that our model delivers strong performance across a wide range of benchmarks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Universal Pansharpening Model

    cs.CV 2026-03 conditional novelty 6.0 of 10

    A single pansharpening model works across 4-, 7-, 8-, and 10-band satellite images by projecting arbitrary-band MS data into a fixed latent space and fusing with PAN via a latent diffusion bridge.

  2. DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training

    cs.AI 2025-12 conditional novelty 6.0 of 10

    A PI-controller-tuned, layerwise-normalized Top-p router trains sparse MoE models that beat Top-k at matched average activated-expert count.

  3. Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Chain-of-Experts replaces one parallel MoE routing step with several sequential expert steps inside a layer, reporting lower loss and memory use in small-scale experiments.

  4. DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts

    cs.CL 2025-06 conditional novelty 6.0 of 10

    DIVE builds diverse MoE experts by pruning the dense FFN with different domain-specific calibration sets, and reports better perplexity and accuracy than LLM-Pruner, FLAP, and LLaMA-MoE at equal active parameters on T...

  5. Scaling Laws for Upcycling Mixture-of-Experts Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Upcycled MoE language-model loss follows a multiplicative power law in dense and upcycled training tokens, with a log-D1 interaction that caps the benefit of upcycling at large compute budgets.

Pith tools