Pith. sign in

REVIEW 3 major objections 5 minor 73 references

SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SlimMoE compresses a 41.9B-parameter MoE into 7.6B and 3.8B models using multi-stage expert slimming and distillation, and the smaller models match larger baselines using only 400B tokens.

desk verdict A genuinely useful empirical MoE compression paper with a fair multi-stage comparison; the flagged 'degenerate KL gradient' issue does not hold up on inspection, and the main caveats are missing error bars and a thinly described corpus. read the letter →

arxiv 2506.18349 v1 pith:J6OGSFYR submitted 2025-06-23 cs.LG cs.CL

Zichong Li , Chen Liang , Zixuan Zhang , Ilgee Hong , Young Jin Kim , Weizhu Chen , Tuo Zhao This is my paper · ORCID
classification cs.LGcs.CL
keywords mixtureofexpertsstructuredpruningexpertslimmingknowledgedistillationmodelcompressionlargelanguagemodelsPhi-3.5-MoEmulti-stage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SlimMoE claims that a very large Mixture-of-Experts model can be shrunk to a fraction of its size without training from scratch: instead of deleting experts, the method removes the least important neurons inside every expert, then distills the original model through intermediate sizes. The paper compresses Phi-3.5-MoE (41.9B total, 6.6B activated parameters) into Phi-mini-MoE (7.6B total, 2.4B activated) and Phi-tiny-MoE (3.8B total, 1.1B activated) using 400B tokens, under 10% of the original training data. The compressed models outperform other models with similar activated-parameter counts and stay competitive with larger dense models, which matters because it offers a route to deploying and fine-tuning MoE capabilities on one GPU.

What carries the argument

The load-bearing mechanism is neuron-level expert slimming driven by a sensitivity score: for each parameter in an expert's GLU down-projection, the gradient of the top-8-logits distillation loss times the weight is aggregated by $\ell^2$ norm across the row, and the lowest-scoring neurons are removed from all sixteen experts uniformly. The second mechanism is a geometric multi-stage schedule that prunes to an intermediate size, distills until gains plateau, and repeats; this keeps every intermediate model close enough to the teacher that distillation recovers performance, whereas a one-shot cut to the target size collapses too far. Attention is pruned at the granularity of whole GQA groups, since those dominate cost at small scale.

What would settle it

Compress a MoE family with near-duplicate experts, such as Mixtral 8x7B, with the identical SlimMoE schedule: if one-shot pruning then distillation matches the multi-stage curve, or if removing whole experts performs as well as slimming neurons, the claim that the method succeeds by preserving heterogeneous specialized knowledge across all 16 experts would be directly contradicted.

Watch

Extended reading notes

Core claim

The central claim is that MoE knowledge survives high-ratio compression if all experts are kept but each one is uniformly slimmed, and if pruning is spread across stages rather than done once. SlimMoE reduces the expert dimension of Phi-3.5-MoE to 15% to form Phi-mini-MoE, and further to 7% plus a 50% cut of GQA attention groups to form Phi-tiny-MoE, pruning at each stage based on sensitivity of the top-8-logit distillation loss and then distilling from the full model. The multi-stage schedule avoids the MMLU collapse the paper reports for one-shot pruning, and final models match or exceed baselines such as Phi-3-mini and LLaMA 3.1 8B on several benchmarks with lower activated-parameter counts.

Load-bearing premise

The load-bearing premise is that uniformly cutting the least important neurons from every expert, with importance ranked by a distillation-loss sensitivity score computed on 16,000 calibration examples, keeps the specialized knowledge the large model spread across its sixteen experts.

Editorial extensions

If this is right

  • A 41.9B/6.6B MoE can be delivered as 7.6B/2.4B and 3.8B/1.1B models that beat similarly sized dense and MoE baselines on the reported benchmarks.
  • Phi-mini-MoE reaches similar or better scores than Phi-3-mini with two-thirds of the activated parameters, and MMLU scores comparable to LLaMA 3.1 8B at lower latency.
  • Under an equal token budget, the multi-stage prune-and-distill path beats one-shot pruning on every benchmark tested and reaches the baseline's final performance with roughly 0.74 times the compute for Phi-mini-MoE and 0.91 times for Phi-tiny-MoE.
  • Longer initialization stages improve final performance, so the paper recommends advancing to the next stage only when current-stage gains become minimal.
  • The compressed base models are fine-tunable on a single A100 for Phi-mini-MoE or a single A6000 for Phi-tiny-MoE, which makes them available for academic and resource-limited settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report this, but if the heterogeneous-expert explanation in its appendix is right, SlimMoE's advantage over expert pruning should shrink on MoE families like Mixtral whose experts are highly similar; applying the pipeline there would test whether expert slimming is universally needed or only when experts are specialized.
  • Because the same top-8-logit distillation loss is used for importance scoring and for student training, a natural extension is to decouple these and test whether task-specific or auxiliary losses give better pruning decisions at the most aggressive ratios.
  • The authors recommend extending earlier stages but allocate most tokens to the final stage, so a practical corollary is that token budgets could be tuned by a validation-loss plateau rule; their 90B-token experiment suggests the method degrades gracefully under resource constraints.
  • The method's architecture-agnostic claim can be probed on other MoE designs, for example routers that do not use SparseMixer-v2-style gradient routing, to see whether the staged recovery depends on that routing mechanism.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SlimMoE, a multi-stage structured-compression framework for MoE language models. Starting from Phi-3.5-MoE (41.9B total / 6.6B active), the authors prune expert dimensions and GQA attention groups over two or three stages, interleaving one-shot pruning with distillation from the original teacher, and produce two released models: Phi-mini-MoE (7.6B total / 2.4B active) and Phi-tiny-MoE (3.8B total / 1.1B active). The compressed models are evaluated on MMLU, MMLU-pro, BBH, ARC-C, HellaSwag, coding, math, and instruction-following benchmarks and are compared against similarly sized MoE and dense baselines. Additional experiments compare multi-stage against one-stage and iterative pruning under matched token budgets, ablate pruning criteria, and test the claim that MoEs are easier to prune than dense models. The paper's central empirical claim is that high-ratio structured MoE compression with staged distillation yields models that outperform same-size baselines and remain competitive with larger models.

Significance. If the empirical results hold, the paper makes a useful practical contribution: it demonstrates that high-ratio structured compression of a large MoE is feasible with 400B tokens and releases two compact models that are competitive with publicly available dense and MoE baselines. The released checkpoints are a concrete asset for resource-constrained deployment and downstream fine-tuning. The staged-pruning comparison is a meaningful addition to the pruning literature, and the ablation study distinguishing expert slimming from expert pruning is informative. The paper is honest about some limitations, including the proprietary training corpus and the expert-heterogeneity differences between Phi-3.5-MoE and Mixtral. However, the central comparison between multi-stage and one-stage pruning is weakened by the fact that stage budgets were selected using the same MMLU benchmark that later serves as the headline outcome, and several fine-grained claims rest on single-run scores without uncertainty estimates. These issues are addressable in revision.

major comments (3)
  1. [Sec. 4.3 / Fig. 2 / Table 3] The stage-stopping rule is a free hyperparameter, and the paper reports that the intermediate token budgets were chosen by inspecting MMLU curves during training. Because the same MMLU metric is used in the headline comparison of multi-stage versus one-stage pruning in Table 3 and Figure 1(b), the claimed advantage is partly a selection-on-evaluation artifact rather than a property of the method alone. Please provide a pre-specified stopping rule with a concrete plateau criterion, or validate the multi-stage advantage on a benchmark that was not used for schedule selection, or at minimum disclose the validation split used for choosing the budgets and report the sensitivity of the conclusions to the chosen budgets.
  2. [Tables 2, 3, and 6] All reported benchmark numbers are single-run scores with no error bars or seed information. Several conclusions rest on small differences: in Table 3, Phi-tiny multi-stage versus iterative on MMLU (60.08 vs 60.05) and HellaSwag (67.33 vs 67.52) is within a point, and in Table 6 some cross-model differences are comparably small. For claims of 'consistently outperforms' and 'matches or exceeds,' please report multiple runs with standard deviations for stochastic evaluation settings, or state explicitly that the non-generative tasks are deterministic under lm-evaluation-harness and explain why the single-run values are sufficient for the conclusions drawn.
  3. [Sec. 5 / Conclusion] The conclusion states that SlimMoE is 'architecturally agnostic' and broadly applicable to other MoE families, but only Phi-3.5-MoE is compressed in this work. Appendix A.8 itself argues that Phi-3.5-MoE differs from Mixtral in expert heterogeneity, so the transferability of expert slimming to other MoE architectures remains untested. Please soften the generality claim or provide at least one additional, qualitatively different MoE family as supporting evidence.
minor comments (5)
  1. [Sec. 3.2, Eq. (3)] The paper should clarify that p_W is the full student next-token distribution, while the teacher distribution is top-8 masked. With this reading, the KL term is not zero at Stage 1 even when W is initialized to the teacher, because KL(p_teacher_top8 || p_teacher_full) is nonzero and has a nonzero gradient. Without this clarification, readers may infer a zero-gradient degeneracy at initialization, which would be incorrect.
  2. [Table 4] The rows labeled 'Prune Ratio 50%' and 'Prune Ratio 25%' appear to be reversed or misnamed, since the 25% row gives uniformly lower scores than the 50% row. If the labels denote the remaining model size rather than the fraction pruned, please rename them accordingly so the ablation can be interpreted correctly.
  3. [Sec. 4.4 / Table 8] The text states that the 16% compression of Phi-3-medium gives approximately 2.3B parameters, while Table 8 lists 1.9B total parameters for the 16% Phi 3 Dense model. These numbers should be reconciled.
  4. [Data description] The 400B-token subset of the proprietary Phi-3.5-MoE pre-training corpus is not described in terms of data mixture, domain distribution, or deduplication. Since the released models are central artifacts, please document the data composition as fully as possible or provide a public proxy recipe that allows reproduction of the compression pipeline.
  5. [Throughout] The benchmark name 'Winograde' appears in several places, including the list of evaluation tasks and table captions; the correct name is WinoGrande.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SlimMoE's compression results are empirically validated on external benchmarks, and no claimed derivation reduces to a fitted input or to a self-citation.

full rationale

SlimMoE's central claims are empirical: the compressed checkpoints are measured on external benchmarks (MMLU, BBH, GSM8K, etc.) that are not used as fitting targets anywhere in the method. The pruning criterion in Eqs. (2)-(4) is defined from the KD/auxiliary loss and calibration data, and the final scores are direct measurements of released models, not quantities reconstructed from the criterion. The self-citations to GRIN-MoE (Liu et al., 2024b) and HomoDistil (Liang et al., 2023) supply the teacher model, data mixture, and a comparison baseline; they are inputs or baselines rather than evidence that forces the conclusion, and the released checkpoints are externally checkable. The Section 4.3 stage-length study selects a training schedule by monitoring MMLU trajectories, but the reported MMLU numbers are measured outcomes rather than predictions derived from that schedule, so this is at most validation-set selection and not circular reasoning. No equation in the paper is equivalent to its own output by construction, and no uniqueness theorem or prior-work result is invoked to rule out alternatives. The potential Stage-1 degeneracy of the KD sensitivity score, if it occurs under the paper's masking convention, would be an operational correctness issue, not a case of a prediction being equivalent to its input, so it does not affect this circularity verdict. Accordingly, no circular step is identified.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central claim is empirical and depends on design choices such as the 16K calibration set, top-8 logits, geometric multi-stage schedule, 400B token budget, and proprietary corpus, rather than on mathematical axioms. No new physical or model-level entities are introduced. The listed axioms are domain assumptions about transferability of importance scores, distillation supervision, and the fairness of baselines.

free parameters (7)
  • Calibration dataset size = 16K examples
    Used to compute sensitivity scores for pruning; no analysis of sensitivity to this number is provided.
  • Top-k logits for distillation loss = k=8
    Adopted from Peng et al. (2024); not tuned for this task.
  • Number of stages = T=2 for Phi-mini-MoE, T=3 for Phi-tiny-MoE
    Assigned based on target compression ratio; not derived from a criterion.
  • Inter-stage compression schedule = Geometric factor alpha^(1/T); intermediate expert dims: 2240 and 960 for mini, 2624, 1024, and 448 for tiny
    Chosen to balance pruning across stages; documented in Table 7.
  • Stage token budgets = 135B (mini initialization) and 130B (tiny initialization) out of 400B total; alternatives 25B and 50B tested
    Selected after observing MMLU curves in Figure 2; directly affects final performance.
  • Total training token budget = 400B tokens; 90B tokens in a reduced-budget experiment
    Budget choice; the paper shows robustness at 90B but uses 400B for the final models.
  • Aux load-balancing loss term
    Equation (3) adds Aux(W) without reporting its coefficient, so its effect on pruning scores is not reproducible.
assumptions (6)
  • domain assumption Teacher logits from Phi-3.5-MoE, restricted to top-8 tokens, provide sufficient distillation signal to recover performance after pruning.
    The entire SlimMoE pipeline optimizes this KL objective in Equation (3); no ablation varies the teacher or the value of k.
  • domain assumption A 400B-token subset of the original pretraining corpus is representative and sufficient for compression.
    Section 4 states the subset is used but the corpus is proprietary and not described in detail.
  • domain assumption Neuron importance estimates computed on 16K calibration examples generalize to the full training distribution.
    Section 3.2 samples 16K examples; no evidence is given that this sample size is sufficient.
  • domain assumption Uniform slimming across experts preserves expert-specialized knowledge better than removing whole experts.
    Central design choice in Section 3.1, empirically supported only for Phi-3.5-MoE in Section 4.5 and Appendix A.8; may not transfer to Mixtral-style MoEs with high inter-expert similarity.
  • domain assumption Phi-3-medium is a fair dense counterpart to Phi-3.5-MoE for the pruning-robustness comparison.
    Section 4.4 relies on comparable pre-compression performance and shared data sources, but the architectures and parameter counts differ.
  • domain assumption Evaluation settings are comparable across baseline models.
    Appendix A.6 specifies per-model few-shot and chat settings, but variances and prompt sensitivity are not reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation." pith.science (2026). https://pith.science/paper/J6OGSFYR

@misc{pith2026250618349,
  author       = {Pith},
  title        = {Pith review of: SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J6OGSFYR}},
  note         = {Machine review of arXiv:2506.18349}
}
read the original abstract

The Mixture of Experts (MoE) architecture has emerged as a powerful paradigm for scaling large language models (LLMs) while maintaining inference efficiency. However, their enormous memory requirements make them prohibitively expensive to fine-tune or deploy in resource-constrained environments. To address this challenge, we introduce SlimMoE, a multi-stage compression framework for transforming large MoE models into much smaller, efficient variants without incurring the prohibitive costs of training from scratch. Our method systematically reduces parameter counts by slimming experts and transferring knowledge through intermediate stages, effectively mitigating the performance degradation common in one-shot pruning approaches. Using this framework, we compress Phi 3.5-MoE (41.9B total/6.6B activated parameters) to create Phi-mini-MoE (7.6B total/2.4B activated parameters) and Phi-tiny-MoE (3.8B total/1.1B activated parameters) using only 400B tokens--less than 10% of the original model's training data. These compressed models can be fine-tuned on a single GPU (A100 for Phi-mini-MoE, A6000 for Phi-tiny-MoE), making them highly suitable for academic and resource-limited settings. Our experiments demonstrate that these compressed models outperform others of similar size and remain competitive with larger models. For instance, Phi-mini-MoE achieves similar or better performance to Phi-3-mini using only 2/3 of the activated parameters and yields comparable MMLU scores to Llama 3.1 8B despite having significantly lower latency. Our findings demonstrate that structured pruning combined with staged distillation offers an effective path to creating high-quality, compact MoE models, paving the way for broader adoption of MoE architectures. We make our models publicly available at https://huggingface.co/microsoft/Phi-mini-MoE-instruct and https://huggingface.co/microsoft/Phi-tiny-MoE-instruct .

Figures

Figures reproduced from arXiv: 2506.18349 by the authors.

Figure 1
Figure 1. (a) Comparison of MMLU scores with recent open-sourced MoE and dense LLMs. (b) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. MMLU performance analysis across compression stages. (a) Comparison between Slim [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Performance of pruned Phi-3.5-MoE versus pruned Phi-3-medium. Left: pruned ratio [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Latency versus throughput comparison across models under varying client loads. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Cosine similarity distribution of neurons in randomly sampled two experts [PITH_FULL_IMAGE:figures/full_fig_p027_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 31 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := ...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    , Aneja, J

    Abdin, M. , Aneja, J. , Awadalla, H. , Awadallah, A. , Awan, A. A. , Bach, N. , Bahree, A. , Bakhtiari, A. , Bao, J. , Behl, H. S. , Benhaim, A. , Bilenko, M. , Bjorck, J. , Bubeck, S. , Cai, M. , Cai, Q. , Chaudhary, V. , Chen, D. , Chen, D. , Chen, W. , Chen, Y.-C. , Chen, Y.-L. , Cheng, H. , Chopra, P. , Dai, X. , Dixon, M. , Eldan, R. , Fragoso, V. , ...

    arXiv 2024

  4. [4]

    , Lee - Thorp, J

    Ainslie, J. , Lee - Thorp, J. , de Jong, M. , Zemlyanskiy, Y. , Lebr \' o n, F. and Sanghai, S. (2023). GQA: training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 (H. Bouamor, J. Pino and K. Bali,...

    doi:10.18653/v1/2023.emnlp-main.298 2023

  5. [5]

    , Croci, M

    Ashkboos, S. , Croci, M. L. , Nascimento, M. G. D. , Hoefler, T. and Hensman, J. (2024). Slicegpt: Compress large language models by deleting rows and columns. In ICLR . OpenReview.net

    work page 2024

  6. [6]

    , Odena, A

    Austin, J. , Odena, A. , Nye, M. I. , Bosma, M. , Michalewski, H. , Dohan, D. , Jiang, E. , Cai, C. J. , Terry, M. , Le, Q. V. and Sutton, C. (2021). Program synthesis with large language models. CoRR, abs/2108.07732

    arXiv 2021

  7. [7]

    , Zellers, R

    Bisk, Y. , Zellers, R. , Bras, R. L. , Gao, J. and Choi, Y. (2020). PIQA: reasoning about physical commonsense in natural language. In AAAI . AAAI Press

    work page 2020

  8. [8]

    , Tworek, J

    Chen, M. , Tworek, J. , Jun, H. , Yuan, Q. , de Oliveira Pinto, H. P. , Kaplan, J. , Edwards, H. , Burda, Y. , Joseph, N. , Brockman, G. , Ray, A. , Puri, R. , Krueger, G. , Petrov, M. , Khlaaf, H. , Sastry, G. , Mishkin, P. , Chan, B. , Gray, S. , Ryder, N. , Pavlov, M. , Power, A. , Kaiser, L. , Bavarian, M. , Winter, C. , Tillet, P. , Such, F. P. , Cum...

    arXiv 2021

Show all 73 references
  1. [9]

    , Huang, S

    Chen, T. , Huang, S. , Xie, Y. , Jiao, B. , Jiang, D. , Zhou, H. , Li, J. and Wei, F. (2022). Task-specific expert pruning for sparse mixture-of-experts. CoRR, abs/2206.00277

    2022 arXiv

  2. [10]

    Cho, J. H. and Hariharan, B. (2019). On the efficacy of knowledge distillation. In ICCV . IEEE

    2019

  3. [11]

    Chowdhury, M. N. R. , Wang, M. , Maghraoui, K. E. , Wang, N. , Chen, P.-Y. and Carothers, C. (2024). A provably effective method for pruning experts in fine-tuned sparse mixture-of-experts. arXiv preprint arXiv:2405.16646

    2024 arXiv

  4. [12]

    , Lee, K

    Clark, C. , Lee, K. , Chang, M. , Kwiatkowski, T. , Collins, M. and Toutanova, K. (2019). Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL-HLT (1) . Association for Computational Linguistics

    2019

  5. [13]

    , Cowhey, I

    Clark, P. , Cowhey, I. , Etzioni, O. , Khot, T. , Sabharwal, A. , Schoenick, C. and Tafjord, O. (2018). Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457

    2018 arXiv

  6. [14]

    , Kosaraju, V

    Cobbe, K. , Kosaraju, V. , Bavarian, M. , Chen, M. , Jun, H. , Kaiser, L. , Plappert, M. , Tworek, J. , Hilton, J. , Nakano, R. , Hesse, C. and Schulman, J. (2021). Training verifiers to solve math word problems. CoRR, abs/2110.14168

    2021 arXiv

  7. [15]

    , Feng, B

    DeepSeek - AI , Liu, A. , Feng, B. , Wang, B. , Wang, B. , Liu, B. , Zhao, C. , Deng, C. , Ruan, C. , Dai, D. , Guo, D. , Yang, D. , Chen, D. , Ji, D. , Li, E. , Lin, F. , Luo, F. , Hao, G. , Chen, G. , Li, G. , Zhang, H. , Xu, H. , Yang, H. , Zhang, H. , Ding, H. , Xin, H. , ...

    2024 arXiv

  8. [16]

    , Feng, B

    DeepSeek - AI , Liu, A. , Feng, B. , Xue, B. , Wang, B. , Wu, B. , Lu, C. , Zhao, C. , Deng, C. , Zhang, C. , Ruan, C. , Dai, D. , Guo, D. , Yang, D. , Chen, D. , Ji, D. , Li, E. , Lin, F. , Dai, F. , Luo, F. , Hao, G. , Chen, G. , Li, G. , Zhang, H. , Bao, H. , Xu, H. , Wang,...

    2024 arXiv

  9. [17]

    , Lewis, M

    Dettmers, T. , Lewis, M. , Shleifer, S. and Zettlemoyer, L. (2021). 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861

    2021 arXiv

  10. [18]

    , Jauhri, A

    Dubey, A. , Jauhri, A. , Pandey, A. , Kadian, A. , Al - Dahle, A. , Letman, A. , Mathur, A. , Schelten, A. , Yang, A. , Fan, A. , Goyal, A. , Hartshorn, A. , Yang, A. , Mitra, A. , Sravankumar, A. , Korenev, A. , Hinsvark, A. , Rao, A. , Zhang, A. , Rodriguez, A. , Gregerson, ...

    2024 arXiv

  11. [19]

    , Yin, H

    Fang, G. , Yin, H. , Muralidharan, S. , Heinrich, G. , Pool, J. , Kautz, J. , Molchanov, P. and Wang, X. (2024). Maskllm: Learnable semi-structured sparsity for large language models. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Informat...

    2024

  12. [20]

    , Zoph, B

    Fedus, W. , Zoph, B. and Shazeer, N. (2022). Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 1--39

    2022

  13. [21]

    , Tow, J

    Gao, L. , Tow, J. , Abbasi, B. , Biderman, S. , Black, S. , DiPofi, A. , Foster, C. , Golding, L. , Hsu, J. , Le Noac'h, A. , Li, H. , McDonell, K. , Muennighoff, N. , Ociepa, C. , Phang, J. , Reynolds, L. , Schoelkopf, H. , Skowron, A. , Sutawika, L. , Tang, E. , Thite, A. , ...

    2024

  14. [22]

    Granite Team, I. (2024). Granite 3.0 language models

    2024

  15. [23]

    , Mao, H

    Han, S. , Mao, H. and Dally, W. J. (2015 a ). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149

    2015 arXiv

  16. [24]

    , Pool, J

    Han, S. , Pool, J. , Tran, J. and Dally, W. J. (2015 b ). Learning both weights and connections for efficient neural networks. arXiv preprint arXiv:1506.02626

    2015 arXiv

  17. [25]

    , Dong, D

    He, S. , Dong, D. , Ding, L. and Li, A. (2025). Towards efficient mixture of experts: A holistic study of compression techniques. ://arxiv.org/abs/2406.02500

    2025 arXiv

  18. [26]

    , Burns, C

    Hendrycks, D. , Burns, C. , Basart, S. , Zou, A. , Mazeika, M. , Song, D. and Steinhardt, J. (2021). Measuring massive multitask language understanding. In ICLR . OpenReview.net

    2021

  19. [27]

    , Vinyals, O

    Hinton, G. , Vinyals, O. and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    2015 arXiv

  20. [28]

    Jiang, A. Q. , Sablayrolles, A. , Roux, A. , Mensch, A. , Savary, B. , Bamford, C. , Chaplot, D. S. , de Las Casas, D. , Hanna, E. B. , Bressand, F. , Lengyel, G. , Bour, G. , Lample, G. , Lavaud, L. R. , Saulnier, L. , Lachaux, M. , Stock, P. , Subramanian, S. , Yang, S. , An...

    2024 arXiv

  21. [29]

    , Jin, Y

    Jordan, K. , Jin, Y. , Boza, V. , You, J. , Cesista, F. , Newhouse, L. and Bernstein, J. (2024). Muon: An optimizer for hidden layers in neural networks. ://kellerjordan.github.io/posts/muon/

    2024

  22. [30]

    Kim, Y. J. , Fahim, R. and Awadalla, H. H. (2023). Mixture of quantized experts (moqe): Complementary effect of low-bit quantization and robustness. arXiv preprint arXiv:2310.02410

    2023 arXiv

  23. [31]

    , Lee, H

    Lepikhin, D. , Lee, H. , Xu, Y. , Chen, D. , Firat, O. , Huang, Y. , Krikun, M. , Shazeer, N. and Chen, Z. (2020). Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668

    2020 arXiv

  24. [32]

    , Zhang, Z

    Li, P. , Zhang, Z. , Yadav, P. , Sung, Y. , Cheng, Y. , Bansal, M. and Chen, T. (2024). Merge, then compress: Demystify efficient smoe with hints from its routing policy. In ICLR . OpenReview.net

    2024

  25. [33]

    , Jiang, H

    Liang, C. , Jiang, H. , Li, Z. , Tang, X. , Yin, B. and Zhao, T. (2023). Homodistil: Homotopic task-agnostic distillation of pre-trained transformers. In ICLR . OpenReview.net

    2023

  26. [34]

    , Zhu, J

    Liu, E. , Zhu, J. , Lin, Z. , Ning, X. , Blaschko, M. B. , Yan, S. , Dai, G. , Yang, H. and Wang, Y. (2024 a ). Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs. CoRR, abs/2407.00945

    2024 arXiv

  27. [35]

    , Xia, C

    Liu, J. , Xia, C. S. , Wang, Y. and Zhang, L. (2023 a ). Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems. ://openreview.net/forum?id=1qvx610Cu7

    2023

  28. [36]

    , Dong, C

    Liu, L. , Dong, C. , Liu, X. , Yu, B. and Gao, J. (2023 b ). Bridging discrete and backpropagation: Straight-through and beyond. Advances in Neural Information Processing Systems, 36 12291--12311

    2023

  29. [37]

    , Gao, J

    Liu, L. , Gao, J. and Chen, W. (2023 c ). Sparse backpropagation for moe training. arXiv preprint arXiv:2310.00811

    2023 arXiv

  30. [38]

    , Kim, Y

    Liu, L. , Kim, Y. J. , Wang, S. , Liang, C. , Shen, Y. , Cheng, H. , Liu, X. , Tanaka, M. , Wu, X. , Hu, W. , Chaudhary, V. , Lin, Z. , Zhang, C. , Xue, J. , Awadalla, H. , Gao, J. and Chen, W. (2024 b ). GRIN: gradient-informed moe. CoRR, abs/2409.12136

    2024 arXiv

  31. [39]

    and Hutter, F

    Loshchilov, I. and Hutter, F. (2017). Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  32. [40]

    , Welling, M

    Louizos, C. , Welling, M. and Kingma, D. P. (2017). Learning sparse neural networks through l\_0 regularization. arXiv preprint arXiv:1712.01312

    2017 arXiv

  33. [41]

    , Liu, Q

    Lu, X. , Liu, Q. , Xu, Y. , Zhou, A. , Huang, S. , Zhang, B. , Yan, J. and Li, H. (2024). Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. In ACL (1) . Association for Computational Linguistics

    2024

  34. [42]

    , Fang, G

    Ma, X. , Fang, G. and Wang, X. (2023). Llm-pruner: On the structural pruning of large language models. In NeurIPS

    2023

  35. [43]

    Men, X. , Xu, M. , Zhang, Q. , Wang, B. , Lin, H. , Lu, Y. , Han, X. and Chen, W. (2024). Shortgpt: Layers in large language models are more redundant than you expect. CoRR, abs/2403.03853

    2024 arXiv

  36. [44]

    , Clark, P

    Mihaylov, T. , Clark, P. , Khot, T. and Sabharwal, A. (2018). Can a suit of armor conduct electricity? A new dataset for open book question answering. In EMNLP . Association for Computational Linguistics

    2018

  37. [45]

    , Farajtabar, M

    Mirzadeh, S. , Farajtabar, M. , Li, A. , Levine, N. , Matsukawa, A. and Ghasemzadeh, H. (2020). Improved knowledge distillation via teacher assistant. In AAAI . AAAI Press

    2020

  38. [46]

    , Mallya, A

    Molchanov, P. , Mallya, A. , Tyree, S. , Frosio, I. and Kautz, J. (2019). Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    2019

  39. [47]

    , Tyree, S

    Molchanov, P. , Tyree, S. , Karras, T. , Aila, T. and Kautz, J. (2016). Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440

    2016 arXiv

  40. [48]

    , Soldaini, L

    Muennighoff, N. , Soldaini, L. , Groeneveld, D. , Lo, K. , Morrison, J. , Min, S. , Shi, W. , Walsh, P. , Tafjord, O. , Lambert, N. et al. (2024). Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060

    2024 arXiv

  41. [49]

    , Sreenivas, S

    Muralidharan, S. , Sreenivas, S. T. , Joshi, R. , Chochowski, M. , Patwary, M. , Shoeybi, M. , Catanzaro, B. , Kautz, J. and Molchanov, P. (2024). Compact language models via pruning and knowledge distillation. In NeurIPS

    2024

  42. [50]

    , Sun, A

    Muzio, A. , Sun, A. and He, C. (2024). Seer-moe: Sparse expert efficiency through regularization for mixture-of-experts. CoRR, abs/2404.05089

    2024 arXiv

  43. [51]

    , Prabhumoye, S

    Parmar, J. , Prabhumoye, S. , Jennings, J. , Patwary, M. , Subramanian, S. , Su, D. , Zhu, C. , Narayanan, D. , Jhunjhunwala, A. , Dattagupta, A. , Jawa, V. , Liu, J. , Mahabaleshwarkar, A. , Nitski, O. , Brundyn, A. , Maki, J. , Martinez, M. , You, J. , Kamalu, J. , LeGresley...

    2024 arXiv

  44. [52]

    Peng, H. , Lv, X. , Bai, Y. , Yao, Z. , Zhang, J. , Hou, L. and Li, J. (2024). Pre-training distillation for large language models: A design space exploration. CoRR, abs/2410.16215. ://doi.org/10.48550/arXiv.2410.16215

    2024 doi arXiv

  45. [53]

    , Sharma, A

    Rafailov, R. , Sharma, A. , Mitchell, E. , Manning, C. D. , Ermon, S. and Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36 53728--53741

    2023

  46. [54]

    , Hou, B

    Rein, D. , Hou, B. L. , Stickland, A. C. , Petty, J. , Pang, R. Y. , Dirani, J. , Michael, J. and Bowman, S. R. (2023). GPQA: A graduate-level google-proof q & a benchmark. CoRR, abs/2311.12022

    2023 arXiv

  47. [55]

    , Ballas, N

    Romero, A. , Ballas, N. , Kahou, S. E. , Chassang, A. , Gatta, C. and Bengio, Y. (2014). Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550

    2014 arXiv

  48. [56]

    , Bras, R

    Sakaguchi, K. , Bras, R. L. , Bhagavatula, C. and Choi, Y. (2020). Winogrande: An adversarial winograd schema challenge at scale. In AAAI . AAAI Press

    2020

  49. [57]

    , Wolf, T

    Sanh, V. , Wolf, T. and Rush, A. (2020). Movement pruning: Adaptive sparsity by fine-tuning. Advances in neural information processing systems, 33 20378--20389

    2020

  50. [58]

    Shazeer, N. (2020). Glu variants improve transformer. arXiv preprint arXiv:2002.05202

    2020 arXiv

  51. [59]

    , Mirhoseini, A

    Shazeer, N. , Mirhoseini, A. , Maziarz, K. , Davis, A. , Le, Q. , Hinton, G. and Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538

    2017 arXiv

  52. [60]

    , Scales, N

    Suzgun, M. , Scales, N. , Sch \" a rli, N. , Gehrmann, S. , Tay, Y. , Chung, H. W. , Chowdhery, A. , Le, Q. V. , Chi, E. H. , Zhou, D. and Wei, J. (2023). Challenging big-bench tasks and whether chain-of-thought can solve them. In ACL (Findings) . Association for Computational...

    2023

  53. [61]

    , Kamath, A

    Team, G. , Kamath, A. , Ferret, J. , Pathak, S. , Vieillard, N. , Merhej, R. , Perrin, S. , Matejovicova, T. , Ram \'e , A. , Rivi \`e re, M. et al. (2025). Gemma 3 technical report. arXiv preprint arXiv:2503.19786

    2025 arXiv

  54. [62]

    Team, Q. (2024). Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters". ://qwenlm.github.io/blog/qwen-moe/

    2024

  55. [63]

    , Shazeer, N

    Vaswani, A. , Shazeer, N. , Parmar, N. , Uszkoreit, J. , Jones, L. , Gomez, A. N. , Kaiser, . and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30

    2017

  56. [64]

    Wang, Y. , Ma, X. , Zhang, G. , Ni, Y. , Chandra, A. , Guo, S. , Ren, W. , Arulraj, A. , He, X. , Jiang, Z. , Li, T. , Ku, M. , Wang, K. , Zhuang, A. , Fan, R. , Yue, X. and Chen, W. (2024). Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. I...

    2024

  57. [65]

    , Gao, T

    Xia, M. , Gao, T. , Zeng, Z. and Chen, D. (2024). Sheared llama: Accelerating language model pre-training via structured pruning. In ICLR . OpenReview.net

    2024

  58. [66]

    , Zhong, Z

    Xia, M. , Zhong, Z. and Chen, D. (2022). Structured pruning learns compact and accurate models. arXiv preprint arXiv:2204.00408

    2022 arXiv

  59. [67]

    , Zhang, Z

    Xie, Y. , Zhang, Z. , Zhou, D. , Xie, C. , Song, Z. , Liu, X. , Wang, Y. , Lin, X. and Xu, A. (2024). Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router. CoRR, abs/2410.12013

    2024 arXiv

  60. [68]

    , Yang, B

    Yang, A. , Yang, B. , Zhang, B. , Hui, B. , Zheng, B. , Yu, B. , Li, C. , Liu, D. , Huang, F. , Wei, H. , Lin, H. , Yang, J. , Tu, J. , Zhang, J. , Yang, J. , Yang, J. , Zhou, J. , Lin, J. , Dang, K. , Lu, K. , Bao, K. , Yang, K. , Yu, L. , Li, M. , Xue, M. , Zhang, P. , Zhu, ...

    2024 arXiv

  61. [69]

    , Sui, Y

    Yang, C. , Sui, Y. , Xiao, J. , Huang, L. , Gong, Y. , Duan, Y. , Jia, W. , Yin, M. , Cheng, Y. and Yuan, B. (2024 b ). Moe-i\( ^ 2 \): Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition. CoRR, abs/2411.01016

    2024 arXiv

  62. [70]

    , Cao, Z

    Yang, Y. , Cao, Z. and Zhao, H. (2024 c ). Laco: Large language model pruning via layer collapse. In EMNLP (Findings) . Association for Computational Linguistics

    2024

  63. [71]

    , Holtzman, A

    Zellers, R. , Holtzman, A. , Bisk, Y. , Farhadi, A. and Choi, Y. (2019). Hellaswag: Can a machine really finish your sentence? In ACL (1) . Association for Computational Linguistics

    2019

  64. [72]

    , Zuo, S

    Zhang, Q. , Zuo, S. , Liang, C. , Bukharin, A. , He, P. , Chen, W. and Zhao, T. (2022). Platon: Pruning large transformer models with upper confidence bound of weight importance. In International conference on machine learning. PMLR

    2022

  65. [73]

    , Chiang, W

    Zheng, L. , Chiang, W. , Sheng, Y. , Zhuang, S. , Wu, Z. , Zhuang, Y. , Lin, Z. , Li, Z. , Li, D. , Xing, E. P. , Zhang, H. , Gonzalez, J. E. and Stoica, I. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.