{"id":"3ef57ae4-c370-4727-8746-36db7c58d924","arxiv_id":"2506.18349","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"SlimMoE uses multi-stage expert slimming and distillation to compress Phi-3.5-MoE into 7.6B and 3.8B parameter models that match or exceed similarly sized baselines while using about 10% of the original training data.","lead":"The authors prune a 42-billion-parameter Mixture-of-Experts language model down to 7.6B and 3.8B parameter versions by repeatedly cutting neurons inside experts and retraining with knowledge distillation on 400B tokens. The compressed models score competitively with dense models of similar size while using fewer active parameters, which could make MoE models cheaper to deploy and fine-tune.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stage-1 pruning scores are degenerate: Eq. (3) has zero KL gradient when student=teacher, so KD-based neuron selection cannot work as written at the first stage.","rationale":"I focused on whether the stage-1 pruning decisions can actually be made by the described algorithm. The reader's weakest assumption concerned expert knowledge preservation and transfer to other MoE families; I do not dispute that this is a limitation, but the more immediate problem is internal: at the first pruning step, the model to be pruned and the teacher are the same checkpoint, making the KL term in Eq. (3) exactly zero. Hence Eq. (2) cannot rank neurons by KD importance at stage 1; the reported 'expert slimming (KL)' superiority in Table 4 is either computed under a different, undocumented protocol or is not attributable to the KL signal. This is a correctness risk in the central method, not a matter of external consensus. I would keep the verdict conditional, but the condition must include a precise statement of the scoring protocol (e.g., warm-up steps or use of CLM/Aux scores at stage 1); without that, the method as written is not reproducible. I still credit the released checkpoints and the matched-token-budget comparisons as useful empirical evidence, which is why I do not move to reject outright.","tokens_in":21682,"tokens_out":10105,"duration_ms":108516,"concrete_test":"Implement Eq. (2)-(3) exactly as written: load Phi-3.5-MoE (or a smaller proxy MoE), set the student W to the teacher weights, sample 16K calibration examples, and compute the KL-gradient contribution to s_ij. If that contribution is machine-precision zero and the neuron ranking is unchanged when the KL term is removed (i.e., only Aux(W) remains), then stage-1 pruning did not use the claimed KD criterion; the authors must disclose the actual scoring procedure before the method can be reproduced. Ideally, also verify on the first-stage intermediate checkpoint (Appendix A.3) whether the kept neurons match a KL-based ranking after a warm-up.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.2, the one-shot pruning criterion is the sensitivity score of Eq. (2) built from L_KD in Eq. (3), with p_teacher_top8 as the teacher distribution and p_W as the student distribution. At the beginning of Stage 1, the model being pruned is the teacher itself (Phi-3.5-MoE). With W equal to the teacher, KL(p_teacher_top8 || p_W) = 0 and its gradient with respect to W is zero for every parameter; the only surviving signal is Aux(W). Therefore the described top-8-logit KD score cannot rank neurons at the first stage, and the central design choice 'expert slimming with KD loss' is not operational as written. The same degeneracy would affect the 'Expert Slimming (KL)' row of Table 4 if computed at initialization. If the authors in fact run a short distillation warm-up before scoring, or score the stage-k student against the original teacher only after some training, that step is missing from the paper. This is an internal-consistency concern: the method, as documented, cannot produce the reported pruning decisions.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SlimMoE, a multi-stage structured-compression framework for MoE language models. Starting from Phi-3.5-MoE (41.9B total / 6.6B active), the authors prune expert dimensions and GQA attention groups over two or three stages, interleaving one-shot pruning with distillation from the original teacher, and produce two released models: Phi-mini-MoE (7.6B total / 2.4B active) and Phi-tiny-MoE (3.8B total / 1.1B active). The compressed models are evaluated on MMLU, MMLU-pro, BBH, ARC-C, HellaSwag, coding, math, and instruction-following benchmarks and are compared against similarly sized MoE and dense baselines. Additional experiments compare multi-stage against one-stage and iterative pruning under matched token budgets, ablate pruning criteria, and test the claim that MoEs are easier to prune than dense models. The paper's central empirical claim is that high-ratio structured MoE compression with staged distillation yields models that outperform same-size baselines and remain competitive with larger models.","tokens_in":21879,"tokens_out":11105,"duration_ms":119514,"significance":"If the empirical results hold, the paper makes a useful practical contribution: it demonstrates that high-ratio structured compression of a large MoE is feasible with 400B tokens and releases two compact models that are competitive with publicly available dense and MoE baselines. The released checkpoints are a concrete asset for resource-constrained deployment and downstream fine-tuning. The staged-pruning comparison is a meaningful addition to the pruning literature, and the ablation study distinguishing expert slimming from expert pruning is informative. The paper is honest about some limitations, including the proprietary training corpus and the expert-heterogeneity differences between Phi-3.5-MoE and Mixtral. However, the central comparison between multi-stage and one-stage pruning is weakened by the fact that stage budgets were selected using the same MMLU benchmark that later serves as the headline outcome, and several fine-grained claims rest on single-run scores without uncertainty estimates. These issues are addressable in revision.","major_comments":[{"comment":"The stage-stopping rule is a free hyperparameter, and the paper reports that the intermediate token budgets were chosen by inspecting MMLU curves during training. Because the same MMLU metric is used in the headline comparison of multi-stage versus one-stage pruning in Table 3 and Figure 1(b), the claimed advantage is partly a selection-on-evaluation artifact rather than a property of the method alone. Please provide a pre-specified stopping rule with a concrete plateau criterion, or validate the multi-stage advantage on a benchmark that was not used for schedule selection, or at minimum disclose the validation split used for choosing the budgets and report the sensitivity of the conclusions to the chosen budgets.","section":"Sec. 4.3 / Fig. 2 / Table 3"},{"comment":"All reported benchmark numbers are single-run scores with no error bars or seed information. Several conclusions rest on small differences: in Table 3, Phi-tiny multi-stage versus iterative on MMLU (60.08 vs 60.05) and HellaSwag (67.33 vs 67.52) is within a point, and in Table 6 some cross-model differences are comparably small. For claims of 'consistently outperforms' and 'matches or exceeds,' please report multiple runs with standard deviations for stochastic evaluation settings, or state explicitly that the non-generative tasks are deterministic under lm-evaluation-harness and explain why the single-run values are sufficient for the conclusions drawn.","section":"Tables 2, 3, and 6"},{"comment":"The conclusion states that SlimMoE is 'architecturally agnostic' and broadly applicable to other MoE families, but only Phi-3.5-MoE is compressed in this work. Appendix A.8 itself argues that Phi-3.5-MoE differs from Mixtral in expert heterogeneity, so the transferability of expert slimming to other MoE architectures remains untested. Please soften the generality claim or provide at least one additional, qualitatively different MoE family as supporting evidence.","section":"Sec. 5 / Conclusion"}],"minor_comments":[{"comment":"The paper should clarify that p_W is the full student next-token distribution, while the teacher distribution is top-8 masked. With this reading, the KL term is not zero at Stage 1 even when W is initialized to the teacher, because KL(p_teacher_top8 || p_teacher_full) is nonzero and has a nonzero gradient. Without this clarification, readers may infer a zero-gradient degeneracy at initialization, which would be incorrect.","section":"Sec. 3.2, Eq. (3)"},{"comment":"The rows labeled 'Prune Ratio 50%' and 'Prune Ratio 25%' appear to be reversed or misnamed, since the 25% row gives uniformly lower scores than the 50% row. If the labels denote the remaining model size rather than the fraction pruned, please rename them accordingly so the ablation can be interpreted correctly.","section":"Table 4"},{"comment":"The text states that the 16% compression of Phi-3-medium gives approximately 2.3B parameters, while Table 8 lists 1.9B total parameters for the 16% Phi 3 Dense model. These numbers should be reconciled.","section":"Sec. 4.4 / Table 8"},{"comment":"The 400B-token subset of the proprietary Phi-3.5-MoE pre-training corpus is not described in terms of data mixture, domain distribution, or deduplication. Since the released models are central artifacts, please document the data composition as fully as possible or provide a public proxy recipe that allows reproduction of the compression pipeline.","section":"Data description"},{"comment":"The benchmark name 'Winograde' appears in several places, including the list of evaluation tasks and table captions; the correct name is WinoGrande.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The released models and the detailed staged-compression comparison are the main strengths of the paper. The strongest technical objection in the stress-test, namely that the sensitivity score is degenerate at Stage 1 because KL(p_teacher_top8 || p_W)=0 when W equals the teacher, does not land: the student distribution is the full distribution, so the KL term is nonzero at initialization. The more serious concerns are methodological: the stage-stopping rule was tuned on the same benchmark used for the central comparison, and single-run scores are used for small margins. These are fixable with additional analysis and disclosure, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is a solid empirical contribution: SlimMoE compresses Phi-3.5-MoE to two functional checkpoints (7.6B and 3.8B total) using a multi-stage prune-and-distill recipe, and it reports the cleanest matched-token-budget comparison I have seen between multi-stage, one-shot, and iterative pruning for MoE models. The released checkpoints are real assets, and the ablation showing expert slimming beats expert pruning is informative. Second, the stress-test concern about Eq. (3) is wrong. The KL value is indeed zero when the student equals the teacher, but the gradient is not. Because the teacher's top-8 distribution is masked without renormalization, the loss acts as a negative log-likelihood on those tokens, and the student's full softmax produces a nonzero gradient that pushes probability mass toward the top-8 set. The sensitivity scores are therefore well-defined. The paper's notation is sloppy here, but the method is not degenerate.\n\nThe genuine soft spots are moderate. All benchmark numbers are single-run, with no error bars or significance tests. The training corpus is proprietary and only vaguely described, though the authors do release the models and give fairly detailed hyperparameters. The stage token budgets were selected by watching MMLU curves during training, and MMLU is also a headline evaluation benchmark, so there is a mild circularity risk. The claim that MoE models are easier to prune than dense models is suggestive but confounded by architecture and ratio choices; the Phi-3-medium comparison is a reasonable attempt but not definitive. None of these are fatal.\n\nThis paper deserves a serious referee. The core comparison is novel, the checkpoints make the claims testable, and the empirical gap over one-shot pruning is consistent across benchmarks. The revision should add standard deviations or confidence intervals, clarify the top-k masking (renormalized or not), and provide more detail on the data mixture. For a compression audience, this is a useful data point. I would bring it to a reading group and cite it for the matched-budget comparison, and I would send it to review rather than desk reject.","headline":"A genuinely useful empirical MoE compression paper with a fair multi-stage comparison; the flagged 'degenerate KL gradient' issue does not hold up on inspection, and the main caveats are missing error bars and a thinly described corpus.","tokens_in":22466,"tokens_out":6248,"would_cite":true,"duration_ms":62760,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SlimMoE compresses a 41.9B-parameter MoE into 7.6B and 3.8B models using multi-stage expert slimming and distillation, and the smaller models match larger baselines using only 400B tokens.","keywords":["mixture of experts","structured pruning","expert slimming","knowledge distillation","model compression","large language models","Phi-3.5-MoE","multi-stage distillation"],"falsifier":"Compress a MoE family with near-duplicate experts, such as Mixtral 8x7B, with the identical SlimMoE schedule: if one-shot pruning then distillation matches the multi-stage curve, or if removing whole experts performs as well as slimming neurons, the claim that the method succeeds by preserving heterogeneous specialized knowledge across all 16 experts would be directly contradicted.","tokens_in":21454,"feed_emoji":"✂️","tokens_out":8801,"duration_ms":72995,"temperature":0.7,"pith_summary":"SlimMoE claims that a very large Mixture-of-Experts model can be shrunk to a fraction of its size without training from scratch: instead of deleting experts, the method removes the least important neurons inside every expert, then distills the original model through intermediate sizes. The paper compresses Phi-3.5-MoE (41.9B total, 6.6B activated parameters) into Phi-mini-MoE (7.6B total, 2.4B activated) and Phi-tiny-MoE (3.8B total, 1.1B activated) using 400B tokens, under 10% of the original training data. The compressed models outperform other models with similar activated-parameter counts and stay competitive with larger dense models, which matters because it offers a route to deploying and fine-tuning MoE capabilities on one GPU.","feed_headline":"Staged expert slimming cuts a 42B MoE to 7.6B and 3.8B models","feed_subtitle":"Multi-stage pruning keeps this 7.6B MoE competitive with Llama 3.1 8B at a third of the active parameters","key_machinery":"The load-bearing mechanism is neuron-level expert slimming driven by a sensitivity score: for each parameter in an expert's GLU down-projection, the gradient of the top-8-logits distillation loss times the weight is aggregated by $\\ell^2$ norm across the row, and the lowest-scoring neurons are removed from all sixteen experts uniformly. The second mechanism is a geometric multi-stage schedule that prunes to an intermediate size, distills until gains plateau, and repeats; this keeps every intermediate model close enough to the teacher that distillation recovers performance, whereas a one-shot cut to the target size collapses too far. Attention is pruned at the granularity of whole GQA groups, since those dominate cost at small scale.","core_discovery":"The central claim is that MoE knowledge survives high-ratio compression if all experts are kept but each one is uniformly slimmed, and if pruning is spread across stages rather than done once. SlimMoE reduces the expert dimension of Phi-3.5-MoE to 15% to form Phi-mini-MoE, and further to 7% plus a 50% cut of GQA attention groups to form Phi-tiny-MoE, pruning at each stage based on sensitivity of the top-8-logit distillation loss and then distilling from the full model. The multi-stage schedule avoids the MMLU collapse the paper reports for one-shot pruning, and final models match or exceed baselines such as Phi-3-mini and LLaMA 3.1 8B on several benchmarks with lower activated-parameter counts.","pith_inferences":["The paper does not report this, but if the heterogeneous-expert explanation in its appendix is right, SlimMoE's advantage over expert pruning should shrink on MoE families like Mixtral whose experts are highly similar; applying the pipeline there would test whether expert slimming is universally needed or only when experts are specialized.","Because the same top-8-logit distillation loss is used for importance scoring and for student training, a natural extension is to decouple these and test whether task-specific or auxiliary losses give better pruning decisions at the most aggressive ratios.","The authors recommend extending earlier stages but allocate most tokens to the final stage, so a practical corollary is that token budgets could be tuned by a validation-loss plateau rule; their 90B-token experiment suggests the method degrades gracefully under resource constraints.","The method's architecture-agnostic claim can be probed on other MoE designs, for example routers that do not use SparseMixer-v2-style gradient routing, to see whether the staged recovery depends on that routing mechanism."],"forward_implications":["A 41.9B/6.6B MoE can be delivered as 7.6B/2.4B and 3.8B/1.1B models that beat similarly sized dense and MoE baselines on the reported benchmarks.","Phi-mini-MoE reaches similar or better scores than Phi-3-mini with two-thirds of the activated parameters, and MMLU scores comparable to LLaMA 3.1 8B at lower latency.","Under an equal token budget, the multi-stage prune-and-distill path beats one-shot pruning on every benchmark tested and reaches the baseline's final performance with roughly 0.74 times the compute for Phi-mini-MoE and 0.91 times for Phi-tiny-MoE.","Longer initialization stages improve final performance, so the paper recommends advancing to the next stage only when current-stage gains become minimal.","The compressed base models are fine-tunable on a single A100 for Phi-mini-MoE or a single A6000 for Phi-tiny-MoE, which makes them available for academic and resource-limited settings."],"supporting_citations":[{"why":"Supplies the 41.9B Phi-3.5-MoE source model and the Phi-3 series dense baselines.","marker":"Abdin et al., 2024"},{"why":"Defines the Phi-3.5-MoE MoE layer, SparseMixer-v2 routing, auxiliary load-balancing loss, and the post-training corpus used for SFT and DPO.","marker":"Liu et al., 2024b"},{"why":"Provides the one-shot prune-and-distill baseline that SlimMoE compares against and extends into a multi-stage schedule.","marker":"Muralidharan et al., 2024"},{"why":"Provides the iterative pruning baseline that uses masks and full-model loading, which SlimMoE avoids.","marker":"Liang et al., 2023"},{"why":"Introduces the top-8 logits distillation objective that SlimMoE uses both for sensitivity scoring and for student training.","marker":"Peng et al., 2024"},{"why":"Supplies the activation-frequency expert pruning baseline that SlimMoE outperforms.","marker":"Muzio et al., 2024"},{"why":"Supplies the M-SMoE expert merging baseline that SlimMoE compares against.","marker":"Li et al., 2024"},{"why":"Supplies the Llama 3 series dense baselines for the claim that compressed models remain competitive with larger models.","marker":"Dubey et al., 2024"}],"fun_headline_variants":["SlimMoE: prune experts in stages, keep MoE performance","Keep every expert, slim them: MoE compression that works","42B MoE becomes 7.6B and 3.8B via staged slimming","SlimMoE: 7.6B MoE rivals Llama 3.1 8B on MMLU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that uniformly cutting the least important neurons from every expert, with importance ranked by a distillation-loss sensitivity score computed on 16,000 calibration examples, keeps the specialized knowledge the large model spread across its sixteen experts.","fun_headline_variants_meta":{"raw":{"variants":["SlimMoE: prune experts in stages, keep MoE performance","Keep every expert, slim them: MoE compression that works","42B MoE becomes 7.6B and 3.8B via staged slimming","SlimMoE: 7.6B MoE rivals Llama 3.1 8B on MMLU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0012,"raw_usage":{"total_tokens":5036,"prompt_tokens":1121,"completion_tokens":3915,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":737,"completion_tokens_details":{"reasoning_tokens":3820}},"tokens_in":737,"tokens_out":3915,"duration_ms":28020,"temperature":1.0,"reasoning_tokens":3820,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:50:50.464124+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compress a MoE family with near-duplicate experts, such as Mixtral 8x7B, with the identical SlimMoE schedule: if one-shot pruning then distillation matches the multi-stage curve, or if removing whole experts performs as well as slimming neurons, the claim that the method succeeds by preserving heterogeneous specialized knowledge across all 16 experts would be directly contradicted.","supporting_citations":[{"cited_title":", Sreenivas, S","cited_arxiv_id":null,"evidence_quote":"Provides the one-shot prune-and-distill baseline that SlimMoE compares against and extends into a multi-stage schedule."},{"cited_title":", Jiang, H","cited_arxiv_id":null,"evidence_quote":"Provides the iterative pruning baseline that uses masks and full-model loading, which SlimMoE avoids."}],"review_version":1}