Pith. sign in

REVIEW 2 major objections 4 minor 33 references

Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models

T0 review · 2 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A brief router-only fine-tune can identify which experts to prune in Mixture-of-Experts models.

desk verdict The Mixtral core is a solid, honest empirical study of a cheap expert-pruning signal; the Qwen transfer claim is weakened by a circular pre-selection and should be re-scoped. read the letter →

arxiv 2608.07890 v1 pith:3FB4T352 submitted 2026-08-08 cs.LG cs.AI

Ali Janati , Kaoutar El Maghraoui , Xinyi Luo , Wenyuan Shen , Owen Zou , Yankai Mao This is my paper · ORCID
classification cs.LGcs.AI
keywords mixture-of-expertsexpertpruningroutersensitivityparameter-efficientfine-tuningLoRAone-shotMMLU-Promodelcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the signal needed to prune experts from a Mixture-of-Experts language model can be recovered from a brief, parameter-efficient fine-tuning run rather than from full fine-tuning. The authors' recipe is to attach a small adapter, read off how much each expert's router gate vector (or, when the router is frozen, its logits) moved, and delete the least-changed experts in one shot. On Mixtral-8×7B-Instruct, restricting LoRA to the router weights, training 0.002% of parameters, outperforms adapting all modules at the same rank once half the experts are removed (27.54% vs. 24.42% MMLU-Pro), and the ranking transfers to Qwen1.5-MoE fine-tuned on mathematics, where half the experts can be removed while retaining 49.7% mean accuracy over eleven math benchmarks. The practical stake is that expert pruning is the direct route to shrinking MoE memory and latency, and the paper shows the accuracy cost of compression is quasi-linear and predictable rather than a cliff.

What carries the argument

The load-bearing object is the router gate matrix $W_g \in \mathbb{R}^{E \times d}$, whose rows are the gate vectors $w_e$; the router is the only place where the decision of which experts serve a token is expressed as parameters. The pruning score is the per-expert $\ell_2$ norm change, $s_e = \left| \|w_e^{(T)}\|_2 - \|w_e^{(0)}\|_2 \right|$, computed between the pretrained and adapted checkpoints, with a logit-delta variant $s^{\text{logit}}_e = \frac{1}{\sqrt{N}}\|\Delta L_{:,e}\|_2$ for adaptations such as IA3 and Houlsby adapters that leave $W_g$ frozen. The argument is that this score measures how much the task revises the router's reliance on each expert, and that pruning the least-revised experts one shot, with no retraining or calibration, preserves accuracy in proportion to how faithfully the adapter was forced and disciplined in moving the router.

What would settle it

Apply router-only LoRA at rank 8 to a second coarse-grained Mixture-of-Experts model, prune half the experts one shot, and evaluate on MMLU-Pro; if accuracy falls below magnitude-based or random pruning, or if the decay curve shows a cliff beyond one expert, the router-sensitivity signal is not general.

Watch

Extended reading notes

Core claim

At the paper's center is a measurement: adapt briefly, then rank experts by the magnitude of the change adaptation induces in their router vectors, $s_e = \left|\|w_e^{(T)}\|_2 - \|w_e^{(0)}\|_2\right|$, and remove the experts with the smallest scores from every layer. The authors establish, on Mixtral-8×7B-Instruct, that this router-sensitivity ranking is strongest when adaptation is confined to the router, improves monotonically with adapter rank up to the largest rank tested, and survives when the router is left frozen and the score is instead read from $\ell_2$ shifts in router logits. They further show that unconstrained additive adapters degrade the signal, while multiplicative scaling matches direct router adaptation. With half the experts removed, MMLU-Pro accuracy under router-guided pruning is 27.54% at rank 8 and 28.76% at rank 16, roughly 1.8 times the accuracy of magnitude-based or random pruning at maximal compression, while memory falls 49% and per-token latency 37%. The criterion transfers to Qwen1.5-MoE fully fine-tuned on mathematics, retaining 49.7% mean accuracy over eleven benchmarks with half the experts removed where random pruning collapses to 4.4%.

Load-bearing premise

Expert utility for a task is expressed primarily in how adaptation moves the router's gate vectors or logits, and that movement is not obscured when adaptation also modifies attention and expert weights.

Editorial extensions

If this is right

  • Practitioners who already run a LoRA fine-tune before deployment can obtain an expert-pruning ranking as a byproduct, at no extra training cost.
  • Compression targets can be priced in advance: on the measured hardware each removed expert saves roughly 3GB of memory and 3.5 to 4.5ms per token, while MMLU-Pro accuracy decays quasi-linearly.
  • Adaptation configuration is a control surface: adapting on a corpus that exercises a capability protects the experts that capability relies on, so a practitioner who needs arithmetic preserved should adapt on arithmetic.
  • The signal does not require writing to the router; frozen-router methods such as multiplicative scaling can be scored through logit shifts, broadening the set of usable adapters.
  • Router-guided pruning is competitive with calibration-based criteria at moderate compression while using only a tiny fraction of their signal cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is right, the same ranking should improve further when the signed norm change is used instead of the magnitude, since contracting gate norms are observable in the sensitivity maps; the paper leaves that re-ranking as an open experiment.
  • The Qwen transfer is partly confounded because the 20-expert starting checkpoint was itself produced by router-sensitivity selection from the 60-expert model, so a fully independent cross-family test would begin from the original checkpoint.
  • The paper's view of adaptation as a measuring instrument inverts standard training guidance: broad LoRA attachment is best for task loss but worst for pruning signal, suggesting future adaptation recipes should be chosen by measurement legibility rather than by downstream accuracy.
  • If the adaptation corpus is truly a steering wheel, then mixing corpora with known capability labels before adaptation should yield an expert ranking that trades off protected capabilities controllably; this is a direct testable extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes using the magnitude of the per-expert change in router weights induced by a lightweight PEFT run (or, for frozen routers, the shift in router logits) as a one-shot expert-pruning score for MoE language models. On Mixtral-8x7B-Instruct, the authors report that router-only LoRA (0.002% of parameters) yields better MMLU-Pro accuracy after pruning than all-module LoRA at matched rank, that the signal improves monotonically with LoRA rank, that multiplicative IA3 roughly matches router LoRA, and that accuracy under pruning decays quasi-linearly. They also report a transfer test on Qwen1.5-MoE fine-tuned for mathematics, keeping 49.7% mean accuracy over eleven benchmarks with half the experts removed. The paper includes controlled sweeps, replication runs, deployment measurements, and released code and checkpoints.

Significance. The Mixtral results are internally consistent and supported by replication analysis; the central comparison between router-only and all-module LoRA exceeds the stated measurement resolution, and the controlled sweeps are a useful empirical mapping of when the criterion works. If the Qwen transfer claim could be independently validated, this would be a practically significant recipe for expert pruning in deployment settings that already include PEFT. However, the Qwen experiment is circular as a transfer test, and the paper's abstract and conclusion overstate the cross-family evidence. The paper is unusually transparent about its limitations, which is a strength, but the current framing does not fully reflect them.

major comments (2)
  1. [Section 4, Section 8.3, Table 7, Abstract] The Qwen transfer result is circular. Section 4 states that the original 60-expert Qwen model was first reduced to 20 experts per layer 'using a router-only LoRA adaptation to select the experts to remove.' Section 8.3 then reports the 49.7% retention after re-pruning this pre-selected 20-expert base with the same criterion, with random and magnitude baselines computed only at the second stage. This does not independently validate cross-model transfer, because the initial 60-to-20 selection may itself be responsible for the resilience of the surviving experts; the baselines do not control for that initial selection. Since this is the only cross-family, full-fine-tuning validation in the paper, the abstract's claim that 'the criterion also transfers to Qwen1.5-MoE' is not supported. Please either re-run the full experiment on the original 60-expert Qwen model (e.g., using the ZeRO-offloading setup already described), or explicitly recast Section 8.3 as a study of re-pruning a pre-selected base and remove or carefully qualify the transfer claims in the abstract and conclusion.
  2. [Section 3.2, Eq. (4), Section 11, Abstract] The pruning score used in the main results is the magnitude of the router-norm change s_e in Eq. (4), but the theoretical guarantee from Chowdhury et al. is stated for the signed norm change. Section 11 acknowledges that the sign question is open, yet the abstract and conclusion repeatedly describe the approach as 'provably motivated.' As written, this overstates the theoretical grounding for the exact score being evaluated. Please either report the signed-score variant in the main results (or at minimum show that the choice of sign does not affect the qualitative findings), or soften 'provably motivated' to 'empirically motivated' in the abstract and conclusion.
minor comments (4)
  1. [Appendix D] In the sensitivity scoring code, the line `router.__wrapped__(lb) if False else router(lb)` appears to be a leftover debugging construct; simplify it to `router(lb)`.
  2. [Section 7] The section title 'How Should Adaptation Perturb the Model?' suggests a causal conclusion, but the IA3 and Houlsby comparisons are run under unmatched budgets and the section itself notes this. Consider a more cautious title or an explicit caveat in the first paragraph.
  3. [Table 9] The note distinguishing the 4-bit memory measurement (24.2GB) from the 24.2B parameter count of the four-experts-pruned model is helpful, but the row labels '1 pruned' through '4 pruned' can be misread as the number of experts retained; consider labeling the column 'Experts retained' for clarity.
  4. [Section 8.2] The text states that MMLU-Pro loses 21.0 points end to end for selection D, but Table 6 does not include MMLU-Pro; adding a row or footnote with that number would make the cross-benchmark comparison easier to verify.

Circularity Check

2 steps flagged · score 6.0 of 10

The Qwen transfer experiment is circular because its 20-expert testbed was pre-selected by the same router-sensitivity criterion, so Section 8.3 cannot independently validate cross-model transfer; the Mixtral results are self-contained.

  1. fitted input called prediction [Section 4 (Qwen preprocessing) and Section 8.3, Table 7 (Across Models and Domains)]
    "Because full fine-tuning the original 60-expert model exceeded the memory capacity of our cluster, we first reduced it to 20 experts per layer, using a router-only LoRA adaptation to select the experts to remove. We then treated this reduced checkpoint as the starting point for the full fine-tuning experiments. This hardware-constrained preprocessing means that the Qwen study evaluates the criterion on a checkpoint whose initial expert subset was itself selected using router sensitivity, rather than directly on the original 60-expert model. ..."

    The Qwen transfer result is offered as independent validation that router sensitivity transfers across model families, but the 20-expert base on which it is measured was itself produced by applying that same router-sensitivity criterion to the original 60-expert model. The 49.7% figure therefore re-tests an expert population already filtered by the score under evaluation, and the magnitude/random baselines are computed only on that pre-selected base, so they do not control for the initial selection. The experiment cannot distinguish between genuine transfer and a checkpoint whose surviving experts were made resilient to further pruning by the initial 60-to-20 router-sensitivity selection. The paper's own disclosure in Section 4 confirms the confound.

  2. self definitional [Section 8.3, 'The location result recurs']
    "The 20-expert base model itself was chosen using preliminary Qwen adaptations in which router-only LoRA reached 29.31% MMLU-Pro against 28.13% for LoRA on all modules. The gap of 1.18 points sits within the measurement resolution; its direction repeats, on a second model family, the central finding of Section 5."

    This passage presents the router-only-over-all-modules ordering on Qwen as independent confirmation of the Section 5 module-location result. But the 20-expert Qwen base was selected using exactly the router-only LoRA preference that the passage is trying to confirm, so the comparison is not a fresh replicate: the testbed was constructed by a procedure that favors the very configuration being validated. Re-measuring the same criterion on its own selection output cannot serve as cross-model transfer evidence.

full rationale

The Mixtral-8x7B experiments are self-contained and not circular: Section 3.4 prunes the original pretrained checkpoint, discarding the adapter after ranking, and the router-only LoRA rankings are compared against magnitude and random baselines on the same unaffected checkpoint, with no fitted parameter reused as a prediction. The rank and module sweeps are independent empirical characterizations. The cited theory [3] is an external, peer-reviewed paper, so the shared author does not by itself make the criterion circular. The circularity is localized to the Qwen transfer claim: Section 4 explicitly states that the 20-expert Qwen base was constructed by the same router-sensitivity criterion, after which Section 8.3 reports the criterion's 49.7% retention and the 'location result recurs' on that very base. Because the transfer testbed is an output of the method under test, the cross-model generalization claim in the abstract and conclusion is partially circular; the Mixtral results remain valid. This yields a partial circularity score of 6, not 8 or 10, because the central Mixtral derivation does not reduce to its inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The pruning criterion itself has no fitted constants; experimental hyperparameters such as rank, steps, and learning rate are chosen by the protocol and swept, not fitted to the evaluation benchmark. The Qwen reduced-model size (20 experts) is a hardware-driven choice, not a fitted parameter. The paper introduces no new physical or architectural entities; the logit-delta score of Equation 5 is a new measurement quantity, not a postulated entity.

assumptions (4)
  • domain assumption Chowdhury et al.'s theorem: pruning experts with the smallest signed router-norm change during full fine-tuning preserves test accuracy on a simplified MoE for binary classification.
    The paper takes this as motivation and extends it empirically to PEFT and MoE LLMs; it does not re-prove it. Invoked in Sections 1, 2, 3.2, and 11.
  • domain assumption The router gate matrix, or its logits under frozen-router adaptation, is the primary observable locus of task-driven expert recruitment.
    This is the mechanistic premise behind the score, stated in Section 11 ('The gate matrix is the one place in the network where the decision of which experts serve a token is expressed as parameters') and implicitly throughout the paper.
  • domain assumption A generic instruction-following corpus induces router changes that are representative of expert recruitment for the evaluated benchmarks.
    The adaptation corpus for Mixtral is generic, yet the signal is evaluated on MMLU-Pro and other benchmarks. Section 8.2 shows task-dependence, so this assumption is only partially valid.
  • standard math Removing rows from W_g keeps top-k routing and softmax normalization over surviving experts well-formed, so the pruned model remains functional.
    Equation 2 is still well-defined over E-n logits; this is a mathematical fact stated in Section 3.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models." pith.science (2026). https://pith.science/paper/3FB4T352

@misc{pith2026260807890,
  author       = {Pith},
  title        = {Pith review of: Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3FB4T352}},
  note         = {Machine review of arXiv:2608.07890}
}
abstract

Mixture-of-Experts (MoE) models decouple total parameters from per-token compute, but deployment still requires storing every expert. Recent theory shows that pruning experts with the smallest router-norm changes during fine-tuning can preserve accuracy, but assumes full fine-tuning. We test whether lightweight adaptation can recover this signal. We briefly fine-tune with a parameter-efficient adapter, rank experts by the induced $\ell_2$ router change, and prune the least-changed experts in one shot. On Mixtral-8$\times$7B-Instruct (44.83% MMLU-Pro), router-only LoRA trains 0.002% of parameters and outperforms all-module LoRA at matched rank with half the experts removed (27.54% vs. 24.42%); signal quality declines as adaptation spreads to attention and expert weights. Accuracy improves monotonically with LoRA rank, reaching 28.76%. IA3, which leaves router weights frozen, matches direct router adaptation, whereas unconstrained additive adapters degrade the signal. Router-guided MMLU-Pro accuracy decays quasi-linearly rather than collapsing, remains nearly 1.8 times that of magnitude-based or random pruning at maximal compression, and reduces memory by 49% and per-token latency by 37%. At 25% compression, retention is competitive with methods using full activation statistics. The criterion also transfers to Qwen1.5-MoE fine-tuned for mathematics, retaining 49.7% mean accuracy over eleven benchmarks with half the experts removed while random pruning falls to single digits. Router sensitivity under lightweight fine-tuning therefore makes provably motivated expert pruning practical at scale.

Figures

Figures reproduced from arXiv: 2608.07890 by the authors.

Figure 1
Figure 1. Router sensitivity across Mixtral-8×7B-Instruct: the per-expert ℓ2 norm change 𝑠𝑒 of Equation 4, for all 32 layers and 8 experts, from an all-module LoRA adaptation at rank 16. The other adaptation runs in this study produce maps with the same scattered structure, differing mainly in overall scale. Sensitivity is not concentrated: most cells sit near their layer’s median while a few expert-layer pairs stand out by a… view at source ↗
Figure 2
Figure 2. MMLU-Pro accuracy as experts are removed from every layer of Mixtral-8 [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. MMLU-Pro accuracy after pruning for router-only LoRA at ranks [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: MMLU-Pro accuracy after pruning by adaptation geometry (Table 3). IA3, which never writes to [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Accuracy retained relative to each benchmark’s own unpruned score, for pruned models from the [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Qwen1.5-MoE fully fine-tuned on mathematical reasoning: mean accuracy over eleven mathematics [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 14 canonical work pages

  1. [1]

    DiEP: Adaptive mixture-of-experts compression through differentiable expert pruning.arXiv preprint arXiv:2509.16105, 2025

    Sikai Bai, Haoxi Li, Jie Zhang, Zicong Hong, and Song Guo. DiEP: Adaptive mixture-of-experts compression through differentiable expert pruning.arXiv preprint arXiv:2509.16105, 2025

    arXiv 2025

  2. [2]

    Task-specificexpertpruningforsparsemixture-of-experts.arXiv preprint arXiv:2206.00277, 2022

    Tianyu Chen, Shaohan Huang, Yuan Xie, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. Task-specificexpertpruningforsparsemixture-of-experts.arXiv preprint arXiv:2206.00277, 2022

    arXiv 2022

  3. [3]

    A provably effective method for pruning experts in fine-tuned sparse mixture-of-experts

    Mohammed Nowaz Rabbani Chowdhury, Meng Wang, Kaoutar El Maghraoui, Naigang Wang, Pin-Yu Chen, and Christopher Carothers. A provably effective method for pruning experts in fine-tuned sparse mixture-of-experts. InProceedings of the 41st International Conference on Machine Learning (ICML),

    work page

  4. [4]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 2924–2936, 2019

    work page 2019

  5. [5]

    Think you have solved question answering? try ARC, the AI2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  6. [6]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  7. [7]

    Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, et al. DeepSeekMoE: Towards ultimate expert specialization in mixture- of-experts language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. arXiv:2401.06066

    arXiv 2024

  8. [8]

    QLoRA: Efficient finetun- ing of quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetun- ing of quantized LLMs. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2305.14314

    arXiv 2023

Show all 33 references
  1. [9]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022. 21

    2022

  2. [10]

    LightEval: A lightweight framework for LLM evaluation.https://github.com/huggingface/lighteval, 2023

    Clémentine Fourrier, Nathan Habib, Thomas Wolf, and Lewis Tunstall. LightEval: A lightweight framework for LLM evaluation.https://github.com/huggingface/lighteval, 2023

    2023

  3. [11]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages ...

    2019

  4. [12]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations (ICLR), 2022. arXiv:2106.09685

    2022 arXiv

  5. [13]

    Whatgetsactivated: Uncovering domain and driver experts in MoE language models.arXiv preprint arXiv:2601.10159, 2026

    GuiminHu,MengLi,QiweiPeng,LijieHu,BoyanXu,andRuichuCai. Whatgetsactivated: Uncovering domain and driver experts in MoE language models.arXiv preprint arXiv:2601.10159, 2026

    2026

  6. [14]

    Isretraining-freeenough? thenecessityofroutercalibrationforefficient MoE compression.arXiv preprint arXiv:2603.02217, 2026

    SieunHyeonandJaeyoungDo. Isretraining-freeenough? thenecessityofroutercalibrationforefficient MoE compression.arXiv preprint arXiv:2603.02217, 2026

    2026

  7. [15]

    Finding fantastic experts in MoEs: A unified study for expert dropping strategies and observations.arXiv preprint arXiv:2504.05586, 2025

    Ajay Jaiswal, Jianyu Wang, Yixiao Li, Pingzhi Li, Tianlong Chen, Zhangyang Wang, Chong Wang, Ruoming Pang, and Xianzhi Du. Finding fantastic experts in MoEs: A unified study for expert dropping strategies and observations.arXiv preprint arXiv:2504.05586, 2025

    2025 arXiv

  8. [16]

    Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    AlbertQ.Jiang,AlexandreSablayrolles,AntoineRoux,ArthurMensch,BlancheSavary,ChrisBamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  9. [17]

    Memory-efficient NLLB-200: Language-specificexpertpruningofamassivelymultilingualmachinetranslationmodel

    Yeskendir Koishekenov, Alexandre Berard, and Vassilina Nikoulina. Memory-efficient NLLB-200: Language-specificexpertpruningofamassivelymultilingualmachinetranslationmodel. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), pages 35...

    2023

  10. [18]

    REAP the experts: Why pruning prevails for one-shot MoE compression

    Mike Lasby, Ivan Lazarevich, Nish Sinnadurai, Sean Lie, Yani Ioannou, and Vithursan Thangarasa. REAP the experts: Why pruning prevails for one-shot MoE compression. InInternational Conference on Learning Representations (ICLR), 2026. arXiv:2510.13999

    2026 arXiv

  11. [19]

    GShard: Scaling giant models with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. InInternational Conference on Learning Representations (ICLR),...

    2021 arXiv

  12. [20]

    Merge, then compress: Demystify efficient SMoE with hints from its routing policy

    Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient SMoE with hints from its routing policy. InInternational Conference on Learning Representations (ICLR), 2024. arXiv:2310.01334

    2024 arXiv

  13. [21]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4582–4597, 2021

    2021

  14. [22]

    Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang

    Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B. Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs.arXiv preprint arXiv:2407.00945, 2024. 22

    2024 arXiv

  15. [23]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. arXiv:2205.05638

    2022 arXiv

  16. [24]

    Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models

    Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. InProceedings of the 62nd Annual Meeting of the Association for Computa...

    2024

  17. [25]

    Ilyas Moutawwakil and Hugging Face. Optimum-benchmark: A framework for benchmarking the performance of transformers models with different hardwares, backends and optimizations.https: //github.com/huggingface/optimum-benchmark, 2023

    2023

  18. [26]

    SEER-MoE:Sparseexpertefficiencythroughregularization for mixture-of-experts.arXiv preprint arXiv:2404.05089, 2024

    AlexandreMuzio,AlexSun,andChuranHe. SEER-MoE:Sparseexpertefficiencythroughregularization for mixture-of-experts.arXiv preprint arXiv:2404.05089, 2024

    2024 arXiv

  19. [27]

    Qwen1.5-MoE: Matching 7B model performance with 1/3 activated parameters.https: //qwenlm.github.io/blog/qwen-moe/, 2024

    Qwen Team. Qwen1.5-MoE: Matching 7B model performance with 1/3 activated parameters.https: //qwenlm.github.io/blog/qwen-moe/, 2024. Accessed 2026

    2024

  20. [28]

    Outrageouslylargeneuralnetworks: Thesparsely-gatedmixture-of-expertslayer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageouslylargeneuralnetworks: Thesparsely-gatedmixture-of-expertslayer. InInternational Conference on Learning Representations (ICLR), 2017. arXiv:1701.06538

    2017 arXiv

  21. [29]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. InFindings of the Association for Comput...

    2023 arXiv

  22. [30]

    LoRA without regret

    Thinking Machines Lab. LoRA without regret. https://thinkingmachines.ai/blog/lora/,

  23. [31]

    MMLU-Pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. InAdvances in Neural Information Processing Systems (Ne...

    2024 arXiv

  24. [32]

    MoE-Pruner: Pruning mixture-of-experts large language model using the hints from its router.arXiv preprint arXiv:2410.12013, 2024

    YanyueXie,ZhiZhang,DingZhou,CongXie,ZiangSong,XinLiu,YanzhiWang,XueLin,andAnXu. MoE-Pruner: Pruning mixture-of-experts large language model using the hints from its router.arXiv preprint arXiv:2410.12013, 2024

    2024 arXiv

  25. [33]

    MoE pathfinder: Trajectory-driven expert pruning.arXiv preprint arXiv:2512.18425, 2025

    Xican Yang, Yuanhe Tian, and Yan Song. MoE pathfinder: Trajectory-driven expert pruning.arXiv preprint arXiv:2512.18425, 2025. A Artifacts and Full Hyperparameter Configurations Code for adaptation, sensitivity scoring, pruning and evaluation is available athttps://github.com/...

    2025

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.