Pith. sign in

REVIEW 4 major objections 6 minor 67 references

BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new MoE architecture, BlockFFN, uses ReLU+RMSNorm routing and CLS-aware training objectives to raise chunk-level sparsity above 70% and achieve up to 3.67x faster decoding on edge devices than dense models.

desk verdict BlockFFN is a real, honestly-reported step for end-side MoE acceleration, with the main caveat that its headline sparsity numbers are optimized-for rather than predicted, and the speculative-decoding speedup is demonstrated in one favorable configuration. read the letter →

arxiv 2507.08771 v2 pith:SVKE3I5I submitted 2025-07-11 cs.LG cs.CL

Chenyang Song , Weilin Zhao , Xu Han , Chaojun Xiao , Yingfa Chen , Yuxuan Li , Zhiyuan Liu , Maosong Sun This is my paper
classification cs.LGcs.CL
keywords mixture-of-expertsactivationsparsitychunk-levelspeculativedecodingend-sideinferenceReLUroutingRMSNormCLS-awaretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that for mixture-of-experts (MoE) language models to accelerate on end-side devices, token-level sparsity alone is not enough: what matters is chunk-level sparsity, meaning that a consecutive chunk of tokens activates a small union of experts. BlockFFN is a MoE variant whose ReLU+RMSNorm router is differentiable and lets each token choose its own number of experts, and whose two training losses—an activation locality loss and a chunk sparsification loss—push neighboring tokens to activate overlapping expert sets while keeping overall sparsity high. Under matched token budgets, BlockFFN matches or beats TopK, DeepSeekMoE, GRIN, and ReMoE baselines in perplexity and downstream accuracy while lifting 8-token chunk-level sparsity above 70%. The authors' CUDA kernels combine this chunk-level sparsity with EAGLE-2 speculative decoding, reaching up to 3.67x faster decoding on an NVIDIA Jetson Orin NX than a dense auto-regressive baseline.

What carries the argument

The load-bearing mechanism is the pair of CLS-aware training objectives: the activation locality loss $L_{\mathrm{al}}$, a binary cross-entropy between sharp sigmoid approximations of the router patterns of adjacent tokens, and the chunk sparsification loss $L_{\mathrm{cs}}$, the average over experts of the probability that an expert is activated by at least one token in a chunk. These make the ReLU router pattern $A_1(x)$ both sparse and locally stable, so the union of experts for a chunk is nearly the same as the set for one token. The second mechanism is the chunk-speculative kernel: given $n=32$ draft tokens from a speculative tree, compute only the experts in the union of activated experts, precompute all $n$ tokens for each such expert because $\mathrm{CLS}\approx\mathrm{TLS}$, then mask the few irrelevant token-expert entries; this is implemented by modifying GEMM kernel loops to scan only the activated-expert union, with tensor-core support.

What would settle it

Measure the actual union sparsity CLS_spec of EAGLE-2 draft trees on a held-out corpus of varied topics for a BlockFFN model trained only with consecutive-chunk objectives. If CLS_spec on trees with diverse token content falls well below the 8-token consecutive CLS (for example, below 60% while consecutive CLS is above 70%), the 32-token kernel's speedup will not transfer; this can be confirmed directly by timing the 32-token kernel versus the 1-token kernel on draft trees whose accepted tokens come from heterogeneous sources.

Watch

Extended reading notes

Core claim

The central discovery is that a MoE can be trained to align its expert-activation patterns across consecutive tokens without losing language-modeling quality, and that this alignment—not raw token-level sparsity—is what makes activation sparsity pay off when verifying speculative drafts. BlockFFN computes expert activation as $A(x)=\mathrm{RMSNorm}(\mathrm{ReLU}(W_{\mathrm{router}}^\top x))$, so routing is differentiable and each token activates a variable number of experts. The activation locality loss minimizes binary cross-entropy between sharpened activation patterns of adjacent tokens; the chunk sparsification loss minimizes, for each expert, the probability that the expert is activated by at least one token in a chunk of $L$ tokens. Across four scales, BlockFFN holds token-level sparsity above 80% while raising 8-token chunk-level sparsity to 71–76%, versus 42–66% for the baselines, at equal or better perplexity. Because chunk-level sparsity stays close to token-level sparsity, every expert in a draft-token union is used by almost every token in the chunk, so the acceleration kernel can precompute all 32 draft tokens for each union expert and then mask the irrelevant entries; the reported FFN timings reach the sparsity-imposed upper bound.

Load-bearing premise

The 32-token kernel's speedup assumes that the combined set of experts used by a speculative draft tree is small and that every expert in that set is used by most of the tree's tokens, so computing all tokens for every used expert wastes almost no work; the paper measures this combined-set size after the fact on Spec-Bench but never trains the model to reduce it for trees.

Editorial extensions

If this is right

  • BlockFFN shows that MoE models can be trained with over 70% 8-token chunk-level sparsity without penalizing perplexity, making activation sparsity compatible with speculative decoding and offloading on low-resource devices.
  • The 32-token kernel demonstrates the first combination of activation sparsity with speculative decoding and reaches the sparsity-imposed upper bound of FFN acceleration on Spec-Bench.
  • On NVIDIA Jetson Orin NX, the combined kernel achieves a 3.67x speedup over a well-tuned auto-regressive baseline and about 2.1x over EAGLE-2 alone, showing that chunk-level sparsity adds real value beyond speculative decoding.
  • The high activation locality also yields expert reuse ratios above 85% between consecutive tokens, which improves the feasibility of expert offloading for memory-constrained devices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The training objectives optimize consecutive-chunk CLS, but speculative draft trees are not single paths; the argument that each root-to-leaf path is consecutive leaves the tree-level union sparsity unoptimized, so CLS_spec should be measured on arbitrary draft trees as an evaluation metric, not just reported on Spec-Bench.
  • The precompute-all-tokens-then-mask strategy pays off only when each union expert is used by most tokens; on tasks with short acceptance lengths (the paper's own Translation case shows this), the 32-token kernel falls behind the 1-token kernel, so the kernel choice should adapt to draft-tree acceptance statistics.
  • BlockFFN's RMSNorm-separated router could be combined with other routing objectives—entropy, load balancing, or expert-utilization penalties—to trade chunk-level sparsity against distributed deployment, since the authors deliberately drop load balancing for the end-side setting.
  • The observed competition between router and expert sparsity for gated MLPs suggests a testable extension: applying the same CLS-aware losses to a gated-expert MoE to see whether that competition is a general property or specific to this architecture.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces BlockFFN, a Mixture-of-Experts architecture for end-side LLM inference. BlockFFN replaces TopK routing with a ReLU-activated linear router followed by RMSNorm, and adds two CLS-aware training losses: an activation locality loss (binary cross-entropy between neighboring tokens' soft activation patterns) and a chunk sparsification loss that minimizes the probability that an expert is activated at least once in a consecutive chunk. The authors train models from 0.1B to 2.8B parameters and report validation perplexity, downstream task accuracy, token/chunk-level sparsity, and decoding speed on an NVIDIA Jetson Orin NX. The central claims are that BlockFFN outperforms TopK MoE, DeepSeekMoE, GRIN, and ReMoE in language modeling at similar activated parameter counts, that it attains over 80% TLS and over 70% 8-token CLS, and that a 32-token speculative-decoding kernel reaches a 3.67x speedup over a strong auto-regressive baseline.

Significance. If the claims hold, BlockFFN is a useful contribution: chunk-level activation sparsity is a practical bottleneck for end-side MoE and for combining sparse models with speculative decoding, and the paper is, to my knowledge, the first to train for CLS explicitly and to couple it with draft-tree verification. The paper ships public code and checkpoints, reports kernel engineering details, and includes ablation and upper-bound analyses that make the efficiency story largely reproducible. Two qualifications temper the significance: the high CLS values are the direct target of the trained loss and thus a scheduled property rather than an emergent prediction, and the headline speedup depends on tree-union sparsity that is only measured post hoc under one draft configuration. These issues are fixable and do not negate the architectural or empirical contribution.

major comments (4)
  1. [Abstract; §4.1.1, Tables 2 and 3] The unqualified claim that BlockFFN shows 'superior performance ... over other MoE baselines' is contradicted by the Small setting. In Table 2, Small BlockFFN has validation PPL 14.88, worse than ReMoE's 14.60; in Table 3, its C.R. score (44.80) is also slightly below ReMoE (45.22). The conclusion repeats the unqualified superiority statement. Either restrict the performance claim to scales where it holds (e.g., Medium and above) or provide an analysis of why the Small setting behaves differently.
  2. [§3.2, Eqs. (4)-(5); Appendix G; Tables 6 and 11] The 3.67x speedup (Table 6) is realized by a kernel that precomputes all 32 draft-tree tokens for each expert in the union, and its efficiency is bounded by the union sparsity of the EAGLE-2 draft tree (CLS_spec approx. 69%, Table 11). Neither Eq. (4) nor Eq. (5) optimizes the union of experts across a tree: the activation locality loss couples adjacent tokens on a single sequence, and the chunk sparsification loss operates on consecutive chunks. Appendix G argues transfer by noting that each root-to-leaf path is consecutive, but the tree union is the union over many alternative paths, and cross-path activation overlap is not trained. CLS_spec is a post hoc measurement on Spec-Bench with one draft model and one tree size. This makes the headline speedup contingent on a favorable draft configuration that is not a training invariant. Please add a tree-union-aware objective or report robustness across draft models, tree depths, and domains; at minimum, the speedup claim should be qualified as configuration-dependent.
  3. [§3.2, Eq. (5)] The chunk sparsification loss computes P_i_act = 1 - exp(Sum_k ln(1 - p_ik)), which is exact only if the L activation events for expert i are independent across tokens. This independence assumption is not stated. If activations are positively correlated (as the activation locality loss is trying to make them), the true probability that expert i is activated at least once in the chunk is larger than this estimate, so the loss systematically underestimates the quantity it claims to minimize. The authors should state the assumption explicitly and, ideally, validate it by comparing the loss's estimate with the empirical co-activation frequency on held-out chunks.
  4. [§4.1.1, Table 2; §3.2; Appendix B] The 70%+ CLS8 results should be presented as the outcome of an explicit optimization target rather than as an independently discovered property. Eq. (5) is exactly the CLS surrogate, and Appendix B introduces an adaptive scheduler that adjusts lambda_cs to control L_cs; therefore the reported CLS values largely reflect the choice and tuning of the objective. This does not diminish the practical value of the method, but the paper's framing (e.g., Figure 1a and the abstract's 'achieving over 70% CLS') may overstate the architectural discovery. Please add one sentence in §4.1.1 clarifying that BlockFFN's CLS is directly driven by the CLS-aware losses and the scheduler.
minor comments (6)
  1. [Table 4] Table 4 reports reuse ratios only for BlockFFN; without baseline values, the claim of 'significant expert selection stability' cannot be evaluated. Please add the same metric for TopK, DSMoE, GRIN, and ReMoE.
  2. [§3.2 and Appendix B] The hyperparameters of the CLS-aware objectives and the adaptive scheduler (lambda_al, lambda0_cs, alpha, N_st, N_adj, gamma_min) have no sensitivity analysis; a short study (e.g., for the Small setting) would help establish that the 70%+ CLS is not fragile.
  3. [Figure 1a] In Figure 1a, BlockFFN-1.2B is compared with DeepSeekMoE-16B and ProSparse-LLaMA2-7B, which differ in scale and training setup; please label this as a cross-scale illustration or use comparable models.
  4. [Tables 2 and 3] No error bars or multiple random seeds are reported; given differences like the 44.80 vs 45.22 C.R. scores in Table 3, the reader cannot assess whether these gaps are significant.
  5. [Appendix H vs Section 4.3] Appendix H's finding that Ours (32-Tok) underperforms Ours (1-Tok) on Translation is an important caveat that appears only in an appendix; it should be mentioned in Section 4.3 where the 3.67x result is discussed.
  6. [Table 11] The notation in Table 11 is confusing: the TLS and CLS_spec columns show expressions such as '1 - 12.45%' rather than a clear sparsity percentage; please define the reported quantity explicitly.

Circularity Check

1 steps flagged · score 4.0 of 10

The headline 70%+ CLS is the chunk-sparsification loss itself, with an adaptive controller on λcs; the performance and speedup results remain externally benchmarked and are not circular.

  1. self definitional [Section 3.2, Eq. 5; Appendix B, Eq. 7; Section 4.1.1, Table 2]
    "we design the chunk sparsification loss, which directly minimizes the chunk-level sparsity of a chunk with L consecutive tokens. ... P^i_act = 1 − exp(∑_{k=1}^L ln(1 − p^i_k)), Lcs = 1/Ne ∑_{i=1}^{Ne} P^i_act ... We introduce an adaptive factor scheduler to adaptively determine λcs according to the dynamics of Lcs ... In experiments, we obtain average TLS values higher than 80% and 8-token CLS values higher than 70% (Table 2)."

    By the paper's own definitions, CLS_L is the ratio of experts contributing weakly to all tokens in a consecutive chunk, i.e., CLS_L = 1 − (1/Ne)∑_i Pr(expert i is activated at least once in the chunk). Eq. 5 defines Lcs as exactly that union-activation probability (1/Ne)∑_i P^i_act, so Lcs = 1 − CLS_L. The total loss includes λcsLcs, and the adaptive scheduler in Eq. 7 raises λcs whenever Lcs rises, actively steering Lcs downward. Thus the reported 70%+ 8-token CLS is not an independent emergent prediction; it is the direct minimization target, with the scheduler controlling the trade-off. The claim that experiments 'demonstrate' high CLS reduces by construction to the training objective.

full rationale

BlockFFN's central performance and deployment claims are evaluated against external comparisons: perplexity and downstream benchmarks against Dense, TopK, DeepSeekMoE, GRIN, and ReMoE, and wall-clock decoding speed on NVIDIA Jetson Orin NX against Huggingface, Baseline AR, and EAGLE-2. Those numbers are not built into the training loss, so they carry independent content. Appendix G also honestly states that the CLS-aware objectives do not directly optimize tree-level union sparsity, which is a limitation of external validity rather than a circular step. The one construction-reduced claim is the headline CLS/TLS sparsity: Eq. 5 defines Lcs as the complement of chunk-level sparsity, and the adaptive scheduler in Eq. 7 tunes λcs to control Lcs, so the resulting 70%+ CLS8 is the optimized variable itself rather than a discovered prediction. Self-citations to the authors' prior work (e.g., ProSparse, Sparsing Law) appear in background and baseline discussions but are not load-bearing for the main results, and no uniqueness theorem is imported from the authors. Overall the circularity is partial and localized to the sparsity metric; the architecture and speedup contributions remain externally validated.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rely on several free hyperparameters (λ_al, λ_cs, α, scheduler settings, sparsity threshold), on an independence assumption inside the chunk sparsification loss, and on the transfer assumption from consecutive chunks to draft-tree unions. No new physical or architectural entities are postulated.

free parameters (5)
  • λ_al = 1e-3 or 2e-3
    Weight of the activation locality loss in Eq. 4; set per model scale in Table 8.
  • λ_cs initial (λ0_cs) = 5e-2 (1e-1 for 2.8B)
    Initial weight of the chunk sparsification loss; adapted online by the scheduler in Appendix B.
  • α (sharpness in activation locality loss) = not reported
    Sharpness of the soft sigmoid approximation of activation patterns in Eq. 4; no value is given in the paper.
  • Scheduler parameters Nst, Nadj, γ_min = 1000, 100, 1.025
    Adaptive factor scheduler settings for λ_cs (Appendix B).
  • Weakly-contributed expert threshold = unspecified
    TLS/CLS numbers in Table 2 depend on the threshold separating weakly-contributed from activated experts; Section 2.1 says 'zero or a low value' but never operationalizes it.
assumptions (4)
  • ad hoc to paper The chunk sparsification loss treats token activation events as independent when computing P_i_act.
    Eq. 5 computes P_i_act = 1 - exp(sum_k ln(1-p_ik)), which is exact only under independence; actual activations within a chunk are correlated, so L_cs is a proxy for the union probability.
  • ad hoc to paper CLS-aware training on consecutive chunks transfers to EAGLE-2 draft-tree patterns.
    Appendix G asserts each root-to-leaf path is consecutive so training transfers; the union of a draft tree is not a single path and tree-level union sparsity is never optimized during training.
  • domain assumption RMSNorm after ReLU separates activation pattern from magnitude and reduces magnitude disturbance from regularization.
    Sections 3.1 and 4.1.4 use this to justify the router design; the supporting evidence is one Small-scale ablation, not a demonstrated mechanism.
  • domain assumption ReLU-based routing produces a bimodal expert allocation that maps to token difficulty and demonstrates routing flexibility.
    Section 4.1.3 interprets Figure 3's bimodal distribution as evidence of adaptive allocation; the link between bimodality and improved performance is asserted rather than causally tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity." pith.science (2026). https://pith.science/paper/SVKE3I5I

@misc{pith2026250708771,
  author       = {Pith},
  title        = {Pith review of: BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SVKE3I5I}},
  note         = {Machine review of arXiv:2507.08771}
}
abstract

To alleviate the computational burden of large language models (LLMs), architectures with activation sparsity, represented by mixture-of-experts (MoE), have attracted increasing attention. However, the non-differentiable and inflexible routing of vanilla MoE hurts model performance. Moreover, while each token activates only a few parameters, these sparsely-activated architectures exhibit low chunk-level sparsity, indicating that the union of multiple consecutive tokens activates a large ratio of parameters. Such a sparsity pattern is unfriendly for acceleration under low-resource conditions (e.g., end-side devices) and incompatible with mainstream acceleration techniques (e.g., speculative decoding). To address these challenges, we introduce a novel MoE architecture, BlockFFN, as well as its efficient training and deployment techniques. Specifically, we use a router integrating ReLU activation and RMSNorm for differentiable and flexible routing. Next, to promote both token-level sparsity (TLS) and chunk-level sparsity (CLS), CLS-aware training objectives are designed, making BlockFFN more acceleration-friendly. Finally, we implement efficient acceleration kernels, combining activation sparsity and speculative decoding for the first time. The experimental results demonstrate the superior performance of BlockFFN over other MoE baselines, achieving over 80% TLS and 70% 8-token CLS. Our kernels achieve up to 3.67$\times$ speedup on real end-side devices than dense models. All codes and checkpoints are available publicly (https://github.com/thunlp/BlockFFN).

Figures

Figures reproduced from arXiv: 2507.08771 by the authors.

Figure 1
Figure 1. (a) For models with high TLS (except BlockFFN-1.2B), CLS quickly collapses to a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of our acceleration kernels (the up projection part), which [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. The layer-wise distributions of av￾erage activation magnitudes on the “Small” settings. While BlockFFN uses CLS-aware objectives, ReMoE adopts L1 regularization. (2) Sparsity: Under close TLS values, BlockFFN always has considerably higher CLS values than other baselines. Attributed to CLS-oriented training objectives, this property makes BlockFFN more friendly for acceleration. 4.1.2 Expert Selection Stability Low-… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: The validation loss of BlockFFN Medium with different expert granularities. 10 20 40 80 160 320 Number of Experts (Ne) 65 70 75 80 85 Sparsity Value (%) TLS CLS8 [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 7
Figure 7. Figure 7: The detailed framework of our efficient acceleration kernels. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

67 extracted references · 24 canonical work pages

  1. [1]

    PIQA : Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA : Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020. URL https://ojs.aaai.org/index.php/AAAI/article/view/6239/6095

    work page 2020

  2. [2]

    Medusa: Simple LLM inference acceleration framework with multiple decoding heads

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Forty-first International Conference on Machine Learning, 2024. URL https://arxiv.org/pdf/2401.10774

    arXiv 2024

  3. [3]

    BoolQ : Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ : Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)...

    work page 2019

  4. [4]

    TyDi QA : A benchmark for information-seeking question answering in typologically diverse languages

    Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA : A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8: 0 454--470, 2020. URL https://aclanthology.org/2020.tacl-1.30.pdf

    work page 2020

  5. [5]

    DeepSeekMoE : Towards ultimate expert specialization in mixture-of-experts language models

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. DeepSeekMoE : Towards ultimate expert specialization in mixture-of-experts language models. CoRR, 2024. URL http://arxiv.org/pdf/2401.06066

    arXiv 2024

  6. [6]

    Language modeling with gated convolutional networks

    Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International Conference on Machine Learning, pp.\ 933--941. PMLR, 2017. URL https://proceedings.mlr.press/v70/dauphin17a/dauphin17a.pdf

    work page 2017

  7. [7]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023. URL https://arxiv.org/pdf/2305.14233.pdf

    arXiv 2023

  8. [8]

    The Llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/pdf/2407.21783

    arXiv 2024

Show all 67 references
  1. [9]

    Switch Transformers : Scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers : Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 0 (120): 0 1--39, 2022. URL https://www.jmlr.org/papers/volume23/21-0998/21-0998.pdf

    2022

  2. [10]

    SparseGPT : Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. SparseGPT : Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp.\ 10323--10337. PMLR, 2023. URL https://proceedings.mlr.press/v202/frantar23a/frantar23a.pdf

    2023

  3. [11]

    MegaBlocks : Efficient sparse training with mixture-of-experts

    Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. MegaBlocks : Efficient sparse training with mixture-of-experts. Proceedings of Machine Learning and Systems, 5: 0 288--304, 2023. URL https://proceedings.mlsys.org/paper_files/paper/2023/file/5a54f79333768effe7e892...

    2023

  4. [12]

    The P ile: An 800 GB dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The P ile: An 800 GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. URL https://arxiv.org/pdf/...

    2020 arXiv

  5. [13]

    Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543, 2023. URL https://arxiv.org/pdf/2306.08543.pdf

    2023 arXiv

  6. [14]

    FastMoE : A fast mixture-of-expert training system

    Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. FastMoE : A fast mixture-of-expert training system. arXiv preprint arXiv:2103.13262, 2021. URL https://arxiv.org/pdf/2103.13262

    2021 arXiv

  7. [15]

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2...

    2023 arXiv

  8. [16]

    MiniCPM : Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. MiniCPM : Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024

    2024 arXiv

  9. [17]

    Harder tasks need more experts: Dynamic routing in MoE models

    Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. Harder tasks need more experts: Dynamic routing in MoE models. arXiv preprint arXiv:2403.07652, 2024. URL https://arxiv.org/pdf/2403.07652

    2024

  10. [18]

    Tutel: Adaptive mixture-of-experts at scale

    Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. Tutel: Adaptive mixture-of-experts at scale. Proceedings of Machine Learning and Systems, 5: 0 269--287, 2023. URL https://proceedings.mlsys.org/paper_f...

    2023

  11. [19]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. URL https://arxiv.org/pdf/2401.04088

    2024 arXiv

  12. [20]

    Scaling laws for fine-grained mixture of experts

    Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pi \'o ro, Micha Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Kr \'o l, Tomasz Odrzyg \'o \'z d \'z , Piotr Sankowski, et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07871, 20...

    2024 arXiv

  13. [21]

    Fast inference from Transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from Transformers via speculative decoding. In International Conference on Machine Learning, pp.\ 19274--19286. PMLR, 2023. URL https://proceedings.mlr.press/v202/leviathan23a/leviathan23a.pdf

    2023

  14. [22]

    StarCoder : may the source be with you! arXiv preprint arXiv:2305.06161, 2023

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. StarCoder : may the source be with you! arXiv preprint arXiv:2305.06161, 2023. URL https://arxiv.org/pdf/2305.06161.pdf

    2023 arXiv

  15. [23]

    EAGLE : Speculative sampling requires rethinking feature uncertainty

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE : Speculative sampling requires rethinking feature uncertainty. In Forty-first International Conference on Machine Learning, 2024 a . URL https://arxiv.org/pdf/2401.15077

    2024 arXiv

  16. [24]

    EAGLE-2 : Faster inference of language models with dynamic draft trees

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2 : Faster inference of language models with dynamic draft trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 7421--7432, 2024 b . URL https://aclanthology.org/2024...

    2024

  17. [25]

    The lazy neuron phenomenon: On emergence of activation sparsity in Transformers

    Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, et al. The lazy neuron phenomenon: On emergence of activation sparsity in Transformers . In The Eleventh International Conference on Learning R...

    2022

  18. [26]

    DeepSeek-V2 : A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. DeepSeek-V2 : A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024 a . URL https://arxiv.org/pdf...

    2024 arXiv

  19. [27]

    DeepSeek-V3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024 b . URL https://arxiv.org/pdf/2412.19437

    2024 arXiv

  20. [28]

    GRIN : Gradient-informed MoE

    Liyuan Liu, Young Jin Kim, Shuohang Wang, Chen Liang, Yelong Shen, Hao Cheng, Xiaodong Liu, Masahiro Tanaka, Xiaoxia Wu, Wenxiang Hu, et al. GRIN : Gradient-informed MoE . arXiv preprint arXiv:2409.12136, 2024 c . URL https://arxiv.org/pdf/2409.12136

    2024 arXiv

  21. [29]

    Deja Vu : Contextual sparsity for efficient LLMs at inference time

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja Vu : Contextual sparsity for efficient LLMs at inference time. In International Conference on Machine Learning, pp.\ 22137--22176. ...

    2023

  22. [30]

    S parsing L aw: Towards large language models with greater activation sparsity

    Yuqi Luo, Chenyang Song, Xu Han, Yingfa Chen, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. S parsing L aw: Towards large language models with greater activation sparsity. arXiv preprint arXiv:2411.02335, 2024. URL https://arxiv.org/pdf/2411.02335

    2024 arXiv

  23. [31]

    LLM-Pruner : On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-Pruner : On the structural pruning of large language models. arXiv preprint arXiv:2305.11627, 2023. URL https://arxiv.org/pdf/2305.11627.pdf

    2023 arXiv

  24. [32]

    ReLU strikes back: Exploiting activation sparsity in large language models

    Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. ReLU strikes back: Exploiting activation sparsity in large language models. arXiv preprint arXiv:2310.04564, 2023. URL https://arxiv.org/pd...

    arXiv

  25. [33]

    Soft merging of experts with adaptive routing

    Mohammed Muqeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive routing. arXiv preprint arXiv:2306.03745, 2023. URL https://arxiv.org/pdf/2306.03745

    2023 arXiv

  26. [34]

    The LAMBADA dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germ \'a n Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern \'a ndez. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meet...

    2016

  27. [35]

    Exploring the limits of transfer learning with a unified text-to-text Transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text Transformer . Journal of machine learning research, 21 0 (140): 0 1--67, 2020. URL...

    2020

  28. [36]

    Improving dictionary learning with gated sparse autoencoders

    Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, J \'a nos Kram \'a r, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404.16014, 2024. URL https://arxiv.org/pdf/2404.16014

    2024 arXiv

  29. [37]

    Searching for activation functions

    Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017. URL https://arxiv.org/pdf/1710.05941

    2017 arXiv

  30. [38]

    SocialIQA : Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. SocialIQA : Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural ...

    2019

  31. [39]

    Omniquant: Omnidirectionally calibrated quantization for large language models

    Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In The Twelfth International Conference on Learning Representations, 2023...

    2023 arXiv

  32. [40]

    GLU variants improve Transformer

    Noam Shazeer. GLU variants improve Transformer . arXiv preprint arXiv:2002.05202, 2020. URL https://arxiv.org/pdf/2002.05202.pdf

    2002 arXiv

  33. [41]

    Dolma: An open corpus of three trillion tokens for language model pretraining research

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159...

    2024 arXiv

  34. [42]

    P ro S parse: Introducing and enhancing intrinsic activation sparsity within large language models

    Chenyang Song, Xu Han, Zhengyan Zhang, Shengding Hu, Xiyu Shi, Kuai Li, Chen Chen, Zhiyuan Liu, Guangli Li, Tao Yang, and Maosong Sun. P ro S parse: Introducing and enhancing intrinsic activation sparsity within large language models. In Proceedings of the 31st International C...

    2025

  35. [43]

    PowerInfer : Fast large language model serving with a consumer-grade GPU

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. PowerInfer : Fast large language model serving with a consumer-grade GPU . arXiv preprint arXiv:2312.12456, 2023. URL https://arxiv.org/pdf/2312.12456.pdf

    2023 arXiv

  36. [44]

    Turbo Sparse : Achieving LLM SOTA performance with minimal activated parameters

    Yixin Song, Haotong Xie, Zhengyan Zhang, Bo Wen, Li Ma, Zeyu Mi, and Haibo Chen. Turbo Sparse : Achieving LLM SOTA performance with minimal activated parameters. arXiv preprint arXiv:2406.05955, 2024. URL https://arxiv.org/pdf/2406.05955

    2024 arXiv

  37. [45]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023. URL https://arxiv.org/pdf/2306.11695.pdf

    2023 arXiv

  38. [46]

    CUTLASS , Jan 2023

    Vijay Thakkar, Pradeep Ramani, Cris Cecka, Aniket Shivam, Honghao Lu, Ethan Yan, Jack Kosaian, Mark Hoemmen, Haicheng Wu, Andrew Kerr, Matt Nicely, Duane Merrill, Dustyn Blasig, Fengqi Qiao, Piotr Majcher, Paul Springer, Markus Hohnerbach, Jin Wang, and Manish Gupta. CUTLASS ,...

    2023

  39. [47]

    Efficient large language models: A survey

    Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, et al. Efficient large language models: A survey. Transactions on Machine Learning Research, 2023. URL https://openreview.net/pdf?id=bsCCJHbO8A

    2023

  40. [48]

    Auxiliary-loss-free load balancing strategy for mixture-of-experts

    Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-loss-free load balancing strategy for mixture-of-experts. arXiv preprint arXiv:2408.15664, 2024 a . URL https://arxiv.org/pdf/2408.15664

    2024 arXiv

  41. [49]

    ReMoE : Fully differentiable mixture-of-experts with ReLU routing

    Ziteng Wang, Jianfei Chen, and Jun Zhu. ReMoE : Fully differentiable mixture-of-experts with ReLU routing. arXiv preprint arXiv:2412.14711, 2024 b . URL https://arxiv.org/pdf/2412.14711

    2024 arXiv

  42. [50]

    Magicoder: Empowering code generation with OSS-Instruct

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with OSS-Instruct . In Forty-first International Conference on Machine Learning, 2024. URL https://arxiv.org/pdf/2312.02120

    2024 arXiv

  43. [51]

    Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding

    Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In Findings of the Association for Computational Linguistics ACL 2...

    2024

  44. [52]

    Sheared LLaMA : Accelerating language model pre-training via structured pruning

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA : Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694, 2023. URL https://arxiv.org/pdf/2310.06694.pdf

    2023 arXiv

  45. [53]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp.\ 38087--38099. PMLR, 2023. URL https://proceedings.mlr.p...

    2023

  46. [54]

    WizardLM : Empowering large language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardLM : Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023. URL https://arxiv.org/pdf/2304.12244

    2023 arXiv

  47. [55]

    PowerInfer-2 : Fast large language model inference on a smartphone

    Zhenliang Xue, Yixin Song, Zeyu Mi, Le Chen, Yubin Xia, and Haibo Chen. PowerInfer-2 : Fast large language model inference on a smartphone. arXiv preprint arXiv:2406.06282, 2024. URL https://arxiv.org/pdf/2406.06282

    2024 arXiv

  48. [56]

    Tensor programs V : Tuning large neural networks via zero-shot hyperparameter transfer

    Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs V : Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022. URL https...

    2022 arXiv

  49. [57]

    A comprehensive study on post-training quantization for large language models

    Zhewei Yao, Cheng Li, Xiaoxia Wu, Stephen Youn, and Yuxiong He. A comprehensive study on post-training quantization for large language models. arXiv preprint arXiv:2303.08302, 2023. URL https://arxiv.org/pdf/2303.08302.pdf

    2023 arXiv

  50. [58]

    HellaSwag : Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4800, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag : Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4800, 2019. URL https://aclanthology.org/P19-1472.pdf

    2019

  51. [59]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019. URL https://proceedings.neurips.cc/paper/2019/file/1e8a19426224ca89e83cef47f1e7f53b-Paper.pdf

    2019

  52. [60]

    ReLU ^2 wins: Discovering efficient activation functions for sparse LLMs

    Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, and Maosong Sun. ReLU ^2 wins: Discovering efficient activation functions for sparse LLMs . arXiv preprint arXiv:2402.03804, 2024 a . URL https://arxiv.org/pdf/2402....

    2024 arXiv

  53. [61]

    Exploring the benefit of activation sparsity in pre-training

    Zhengyan Zhang, Chaojun Xiao, Qiujieli Qin, Yankai Lin, Zhiyuan Zeng, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, and Jie Zhou. Exploring the benefit of activation sparsity in pre-training. In Forty-first International Conference on Machine Learning, 2024 b . URL https://op...

    2024

  54. [62]

    Ouroboros: Generating longer drafts phrase by phrase for faster speculative decoding

    Weilin Zhao, Yuxiang Huang, Xu Han, Wang Xu, Chaojun Xiao, Xinrong Zhang, Yewei Fang, Kaihuo Zhang, Zhiyuan Liu, and Maosong Sun. Ouroboros: Generating longer drafts phrase by phrase for faster speculative decoding. In Proceedings of the 2024 Conference on Empirical Methods in...

    2024

  55. [63]

    FR-Spec : Accelerating large-vocabulary language models via frequency-ranked speculative sampling

    Weilin Zhao, Tengyu Pan, Xu Han, Yudi Zhang, Ao Sun, Yuxiang Huang, Kaihuo Zhang, Weilun Zhao, Yuxuan Li, Jianyong Wang, et al. FR-Spec : Accelerating large-vocabulary language models via frequency-ranked speculative sampling. arXiv preprint arXiv:2502.14856, 2025. URL https:/...

    2025 arXiv

  56. [64]

    Lory: Fully differentiable mixture-of-experts for autoregressive language model pre-training

    Zexuan Zhong, Mengzhou Xia, Danqi Chen, and Mike Lewis. Lory: Fully differentiable mixture-of-experts for autoregressive language model pre-training. arXiv preprint arXiv:2405.03133, 2024. URL https://arxiv.org/pdf/2405.03133

    2024 arXiv

  57. [65]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  58. [66]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  59. [67]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.