Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Vertical expert splitting cuts offloaded MoE inference latency by up to 65.7%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 20:45 UTC pith:JLXN3MUA

load-bearing objection A genuinely new expert-split caching mechanism with credible measured speedups, but the cache configurator's statistics are under-specified and the evaluation is narrow; send it to a careful referee. the 3 major comments →

arxiv 2509.08342 v1 pith:JLXN3MUA submitted 2025-09-10 cs.LG cs.AI

Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism

Jiaming Yan , Jianchun Liu , Hongli Xu , Liusheng Huang This is my paper
classification cs.LG cs.AI
keywords Mixture-of-Expertsexpert offloadinginference accelerationexpert prefetchingexpert cachingVRAM efficiencyadaptive cache configurationLLM inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that the usual remedies for offloaded Mixture-of-Experts inference—caching whole experts in VRAM or prefetching whole experts from RAM—leave latency on the table because a full expert is too large a unit to cache or fetch efficiently. MoEpic splits each expert vertically and caches only the top segment of hot experts while prefetching the remaining segment before the next layer, which the paper says raises cache hit rates, shortens loads, and widens the computation-transfer overlap. The same design needs a per-layer answer for how much VRAM and how large a top segment to give each layer, so the paper contributes a divide-and-conquer fixed-point algorithm for that configuration. On two popular open MoE models, MoEpic reports lowering per-token decode latency by 37.51% to 65.73% against existing offloading systems, and reaching a target latency with roughly half the VRAM footprint. If true, this makes single-GPU or edge deployment of large MoE models materially cheaper.

Core claim

The paper's central claim is that partial-expert caching is a better operating point than full-expert caching or prefetch-only when VRAM is constrained. MoEpic vertically splits each expert into a top segment and a bottom segment, keeps the top segments of frequently activated experts in VRAM, and prefetches the bottom segments (or full experts on a miss) during the previous layer's computation. This simultaneously increases the number of experts that can be cached under a fixed VRAM budget and reduces the amount of data that must be fetched for a cache-hit expert, improving both cache hit rate and transfer-computation overlap. On top of this mechanism, the adaptive cache configurator assign

What carries the argument

The central object is the expert split mechanism: each MoE expert is divided into a top and a bottom segment, with top segments of hot experts cached in VRAM and bottom segments (or whole experts, on a miss) prefetched ahead of the next layer. This shortens each prefetch and lets more distinct experts fit into the same VRAM budget. Supporting it are three components: a speculative prefetcher that feeds the current layer's intermediate activation into the next layer's router to guess activated experts; a cache-priority metric, LCP, that blends activation frequency with activation recency; and a divide-and-conquer fixed-point algorithm that solves for each layer's VRAM budget and split ratio.

Load-bearing premise

The optimizer assumes the routing statistics measured under the current cache configuration—cache hit rate, expert prediction accuracy, and prediction-cache hit rate—stay valid when it scores alternative cache sizes and split ratios; if expert routing shifts once the cache changes, the chosen configuration is no longer optimal.

What would settle it

Run MoEpic on a decoding workload whose router decisions are deliberately non-stationary, such as a long sequence that switches topic every few hundred tokens, and compare its measured TPOT and cache hit rate under the automatically chosen configuration against a configuration chosen fresh on that workload. If the adaptive algorithm's pre-measured statistics no longer predict the optimal cache size or split ratio, the exposed loading latency will grow toward the prefetch-only baseline, falsifying the optimizer's model of the workload.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Partial-expert caching becomes a better operating point than full-expert caching or prefetch-only when VRAM is tight, since the same budget caches more experts and each fetch is smaller.
  • The measured speedups translate directly into hardware cost: a target time-per-output-token can be reached with roughly half the VRAM, so fewer or smaller GPUs are needed for on-premises MoE deployment.
  • Per-layer cache configuration matters: the ablation attributes a 1.21x/1.46x slowdown in TTFT/TPOT to replacing the adaptive configurator with uniform budgets and a fixed 0.5 split ratio.
  • The approach works on existing MoE checkpoints without retraining, so it can be layered into current serving stacks that already offload experts.
  • The latency advantage holds across models with very different expert counts and activation patterns, suggesting the split addresses a general property of the offloading bottleneck rather than a quirk of one model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the paper's cost model is that the optimal top-segment ratio should shrink as the number of activated experts per token grows, because the computation window available for prefetching shrinks accordingly.
  • The configurator treats routing statistics as stationary between updates; on workloads with abrupt topic or task shifts, the optimizer's internal model can become stale, so re-profiling triggers or a non-stationary-aware optimizer would be a natural next step.
  • The same top/bottom split idea could transfer to offloading other large parameter blobs used per token, such as attention weights or KV-cache segments, where only part of the blob is needed early in the compute pass.
  • A further implication is that the split ratio should be tuned online against current PCIe contention and batch-size-driven computation windows, since both shift the balance the paper's fixed profile assumes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MoEpic, an expert-offloading inference system for MoE LLMs. Each expert is vertically split into a top and bottom segment; MoEpic caches top segments of hot experts in VRAM, prefetches bottom segments (or full experts) for the next layer, and uses a priority-based cache policy (LCP) combining activation frequency and recency. A divide-and-conquer, fixed-point-style algorithm adaptively allocates the VRAM budget and split ratio per layer. Experiments on Qwen1.5-MoE and Mixtral-8x7B over MMLU report TTFT/TPOT reductions and roughly half the VRAM footprint compared with Pre-gated MoE, Mixtral-offloading, AdapMoE, and MoE-Infinity, with an ablation attributing the largest latency contribution to the cache configuration algorithm (CCA).

Significance. If the reported results hold, the paper identifies a useful new operating point for MoE offloading: partial-expert caching plus speculative prefetching of the remaining segments is better than either full-expert caching or prefetch-only under constrained VRAM. The manuscript contains real end-to-end measurements, a direct ablation of the three system components, and a sensitivity analysis. These are strengths. However, the load-bearing adaptive optimizer is under-specified in ways that affect the validity of Eq. (11), and the empirical evidence is narrow (two models, one dataset, no error bars). The central idea is plausible and likely of interest to the systems/ML-systems community.

major comments (3)
  1. [§3.4.2, Algorithm 1, Eq. (11)] The cache configurator searches over all cache sizes C_i ∈ [N] using statistics H_i(C_i), P_i(y), and PH_i(y,C_i). Algorithm 1 initializes these as full tables (line 3) and says they are updated every token (line 7), but in an online system only the currently active C_i is directly observed. No rule is given for updating H_i(c) and PH_i(y,c) for alternative cache sizes c. Without such a rule (e.g., recomputation from per-expert activation traces and priority ranks, or an explicit offline profiling phase), Eq. (11) is scored with stale or undefined entries. This is load-bearing because Table 3 shows CCA is the largest performance component (1.46x TPOT degradation without it). Please specify the exact update rule for all C_i, or restrict the search to configurations whose statistics are actually observed or provably computable.
  2. [§3.4.2, Eq. (11)] The sub-problem is not well-posed as written. It says 'at most Y expert prefetch operations can be completed within the window T_wind', but Y is never defined. The objective sums A_pref(y,C_i) for y=1..Y while the constraint bounds the cumulative T_pref(y,C_i) for the same Y. Because T_pref(y,C_i) depends on C_i and θ_i = V_i/C_i, the largest feasible prefix length is itself a function of C_i. A fixed Y makes the constraint either redundant or inconsistent with the objective; a variable Y requires an explicit maximization over the feasible prefix. Please reformulate Eq. (11) as, e.g., Y(C_i) = max{n : Σ_{y=1}^n T_pref(y,C_i) ≤ T_wind} and maximize over C_i and the feasible prefix. This directly affects ExpertSplit in Algorithm 1, which solves Eq. (11).
  3. [§4.1-§4.4] The paper states that all results are averages of five independent runs, but no error bars, standard deviations, or per-run data are reported anywhere. The reader cannot assess whether the claimed 37.51%-65.73% TPOT reductions and the Table 3 ablation differences are significant relative to run-to-run variation. In addition, the evaluation is limited to two models, one of which is heavily quantized, and one dataset (MMLU); the headline 'save about half of the GPU cost' rests on a single target-TPOT comparison. Please add variance information and at least one additional workload/model, or temper the generalization claims accordingly.
minor comments (5)
  1. [§4.3] The text says 'target TTFT of 140ms' but the metric and figure caption refer to TPOT; this is a typo that should be corrected.
  2. [Figure 9 caption] The caption says 'Mixtral-7x8B'; the model name is Mixtral-8x7B.
  3. [§4.1] The buffer size is denoted U_b in Eq. (8), but §4.1 says 'buffer size U_e is set to...'; please use consistent notation.
  4. [Algorithm 1, lines 14-15] When V_i < ζV_e, the perturbed budget V_i - ζV_e can become negative, which is outside the feasible domain of Eq. (8). Add a clamping or skip condition for layers whose budget is already at the lower bound.
  5. [References] Reference [26] is cited for Qwen1.5-MoE, but the listed title is the Qwen2 Technical Report; please verify the correct source.

Circularity Check

0 steps flagged

No circular derivation chain; empirical latency/VRAM claims are benchmark-grounded; score 2 for a minor non-load-bearing self-citation.

full rationale

The paper's headline result is an end-to-end measured comparison: Figure 7 reports TTFT/TPOT for MoEpic against Pre-gated MoE, Mixtral-offloading, AdapMoE, and MoE-Infinity, and Figure 8 reports VRAM footprint at matched target TPOT. These comparisons do not depend on deriving the speedup from the optimizer's own equations, so the central claim is not circular. The cache-configuration subroutine (Eq. 11, Algorithm 1) uses measured statistics H_i(C_i), P_i(y), PH_i(y,C_i) as inputs to choose C_i and theta_i; although this is a feedback loop (the statistics are collected under the current configuration), the paper defines them as empirical counts for arbitrary cache sizes, not as outputs of the optimum being chosen, so there is no reduction of Eq. (11) to its own solution by construction. The only author self-citation ([16]) supports the hardware description in Section 4.1 and is not load-bearing; it is also likely a misplaced reference. A genuine omitted-support issue exists: Algorithm 1 line 7 claims to update H_i(C_i) and PH_i(a,C_i) for all C_i after each token without specifying how one observed token updates entries for non-current cache sizes; this is an observability/stationarity gap affecting the optimizer's validity, but it is a correctness risk rather than a circular argument. No uniqueness theorem or ansatz is imported from the authors' prior work. Score 2 is assigned for the minor self-citation, not for any circular step.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The system's central mechanism rests on empirical regularities (inter-layer activation similarity, long-tail and temporal locality of expert routing, stationarity of measured routing statistics) and on a simplified hardware model (linear PCIe transfer time, proportional expert computation time). These are pragmatic engineering assumptions rather than proven first principles; the paper's own sensitivity analysis partially mitigates the hyperparameter choices but does not validate statistics stationarity.

free parameters (4)
  • LCP balance parameter rho = 0.25
    Hyperparameter in cache priority P_i,j = mu_i,j * rho^(nu_i,j/omega); controls trade-off between activation frequency and recency. Set by default, not fitted to the benchmark; sensitivity shown in Figure 9a.
  • LCP observation window omega = 128
    Observation window for activation interval in the cache priority formula; default setting, minor sensitivity.
  • Allocation granularity zeta = 0.01
    Step size for transferring VRAM budget between layers in Algorithm 1; performance degrades with larger zeta (Figure 9b).
  • Cache update frequency tau = 5000
    Number of tokens between executions of the cache configuration algorithm.
axioms (4)
  • domain assumption Intermediate activations of adjacent Transformer layers are similar enough that feeding h_i to the next layer's router predicts the next layer's expert choices (Eq. (3), Section 3.2, Figure 6).
    Load-bearing for the speculative prefetcher; the paper itself notes low similarity in early layers, which it addresses by allocating more VRAM there.
  • domain assumption Expert activation patterns in each layer are stationary enough over time that historical cache hit rates, prediction accuracies, and prediction cache hit rates H_i, P_i, PH_i measured on past tokens predict future behavior at different cache sizes (Section 3.4.2).
    The optimization in Eq. (11) and Algorithm 1 reuses these measured statistics to score alternative configurations.
  • domain assumption Expert parameters can be vertically split so that processing the top segment is independent of the bottom segment and consumes a theta_i fraction of the expert computation time (Section 3.1, Eq. (5)).
    The transfer-computation overlap model assumes top-segment computation can proceed while bottom segments load, without synchronization overhead.
  • domain assumption PCIe transfer time for an expert is linearly proportional to its byte size, T_exp_load = U_e / B, with no bandwidth contention or overhead modeled (Section 3.4.1).
    Simplifies the optimization; real PCIe transfer has setup overheads and contention.

pith-pipeline@v1.3.0-alltime-deepseek · 20267 in / 19329 out tokens · 190574 ms · 2026-08-04T20:45:00.470183+00:00 · methodology

0 comments
read the original abstract

Mixture-of-Experts (MoE) has emerged as a promising architecture for modern large language models (LLMs). However, massive parameters impose heavy GPU memory (i.e., VRAM) demands, hindering the widespread adoption of MoE LLMs. Offloading the expert parameters to CPU RAM offers an effective way to alleviate the VRAM requirements for MoE inference. Existing approaches typically cache a small subset of experts in VRAM and dynamically prefetch experts from RAM during inference, leading to significant degradation in inference speed due to the poor cache hit rate and substantial expert loading latency. In this work, we propose MoEpic, an efficient MoE inference system with a novel expert split mechanism. Specifically, each expert is vertically divided into two segments: top and bottom. MoEpic caches the top segment of hot experts, so that more experts will be stored under the limited VRAM budget, thereby improving the cache hit rate. During each layer's inference, MoEpic predicts and prefetches the activated experts for the next layer. Since the top segments of cached experts are exempt from fetching, the loading time is reduced, which allows efficient transfer-computation overlap. Nevertheless, the performance of MoEpic critically depends on the cache configuration (i.e., each layer's VRAM budget and expert split ratio). To this end, we propose a divide-and-conquer algorithm based on fixed-point iteration for adaptive cache configuration. Extensive experiments on popular MoE LLMs demonstrate that MoEpic can save about half of the GPU cost, while lowering the inference latency by about 37.51%-65.73% compared to the baselines.

Figures

Figures reproduced from arXiv: 2509.08342 by Hongli Xu, Jiaming Yan, Jianchun Liu, Liusheng Huang.

Figure 1
Figure 1. Figure 1: Inference process of the MoE model. expert loading latency that cannot be effectively hidden by the computation time. Besides, since the VRAM budget al￾located to each layer varies, there is no universally optimal solution for all layers. The main contributions of this paper are summarized as follows: • We propose an efficient MoE inference system with expert offloading, named MoEpic. By the novel expert s… view at source ↗
Figure 2
Figure 2. Figure 2: The results of preliminary experiments using the Qwen1.5-MoE model on the MMLU dataset: (a) The VRAM footprint and average cache hit rate across all layers under different cache sizes (i.e., the number of cached experts per layer); (b) The inference latency (i.e., time per output token) with given cache hit rates; (c) The expert prediction accuracy of each layer; (d) The per-layer computation time and the … view at source ↗
Figure 3
Figure 3. Figure 3: The inference process of two consecutive layers un￾der different approaches. For clarity of exposition, we do not include loading latency for the first layer, without loss of cor￾rectness. "Top" denotes the top segments of cache-hit experts. "M.S." denotes the activated experts’ missing (top/bottom) segments that are neither cached nor prefetched. 𝐸𝑖,𝑗 (𝑗 ∈ [𝑁]), then activates the 𝐾 (< 𝑁) experts with the… view at source ↗
Figure 4
Figure 4. Figure 4: Motivations for MoEpic’s design. budget to efficiently hide the loading latency. 1) According to our experiments, the cache hit rate varies by up to 5.55% across different layers (58.12% vs. 63.67%). For layers with relatively low cache hit rates, additional VRAM budget is necessary to cache more experts and maintain adequate hit rate. 2) For layers with low expert prediction accuracy, e.g., those located … view at source ↗
Figure 5
Figure 5. Figure 5: System overview of MoEpic. in [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Layer-wise cosine similarity of two models. expert prediction accuracy, which is also validated in [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Inference latency (i.e., TTFT and TPOT) of different approaches with Qwen1.5-MoE and Mixtral-8x7B. 80 1 00 1 20 1 40 1 60 0 3 6 9 1 2 1 5 1 8 Target TPOT (ms) Mixtral-offloading AdapMoE MoE-Infinity MoEpic 80 1 00 1 20 1 40 1 60 0 3 6 9 1 2 1 5 1 8 21 VRAM Footprint (GB) Target TPOT (ms) (a) Footprint with Qwen1.5-MoE. 1 00 1 30 1 60 1 90 220 0 2 4 6 8 1 0 VRAM Footprint (GB) Target TPOT (ms) (b) Footprint… view at source ↗
Figure 8
Figure 8. Figure 8: VRAM footprint of different approaches under various target TPOT with Qwen1.5-MoE and Mixtral-8x7B. achieves the lowest TPOT among all baselines. For exam￾ple, given a VRAM budget of 160 experts in Mixtral-8x7B, MoEpic presents a TPOT of 85ms, while Pre-gated MoE, Mixtral-offloading, AdapMoE, and MoE-Infinity separately show a TPOT of 248ms, 162ms, 144ms, and 136ms. As a re￾sult, MoEpic can mitigate the TP… view at source ↗
Figure 9
Figure 9. Figure 9: Impact of different hyper-parameters settings on MoEpic’s performance with Mixtral-7x8B. in the prefill stage. These results indicate the necessity and importance of all three components, while CCA plays the most critical role in MoEpic ’s performance. 4.5 Sensitivity Analysis In this section, we conduct a set of experiments using Mixtral￾8x7B to measure MoEpic’s performance (i.e., TPOT) with different set… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models

    cs.AI 2026-06 conditional novelty 5.0

    SpecPrefetch trains lightweight adapters to prefetch next-layer experts during offloaded MoE inference while keeping the native router authoritative, improving decoding throughput by up to ~20% on a mobile device.

  2. CogniRoute: Learning to Route Social Evidence in Omni-Modal Models

    cs.CV 2026-06 unverdicted novelty 5.0

    CogniRoute adds a cognitive schema and route-aware RL to an omni-modal MoE, reaching 59.38% accuracy on a new 118K-example social video QA benchmark and beating prior baselines by 15-27 points.

Reference graph

Works this paper leans on

32 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    Hicham Badri and Appu Shaji. [n. d.]. Half-quadratic quantiza- tion of large machine learning models, November 2023. URL https://mobiusml. github. io/hqq_blog ([n. d.])

    work page 2023

  3. [3]

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2024. A survey on mixture of experts. arXiv preprint arXiv:2407.06204 (2024)

    Pith/arXiv arXiv 2024

  4. [4]

    Zhixu Du, Shiyu Li, Yuhao Wu, Xiangyu Jiang, Jingwei Sun, Qilin Zheng, Yongkai Wu, Ang Li, Hai Li, and Yiran Chen. 2024. Sida: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models. Proceedings of Machine Learning and Systems 6 (2024), 224–238

    work page 2024

  5. [5]

    Artyom Eliseev and Denis Mazur. 2023. Fast inference of mixture- of-experts language models with offloading. arXiv preprint arXiv:2312.17238 (2023)

    Pith/arXiv arXiv 2023

  6. [6]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv e-prints (2024), arXiv–2407

    work page 2024

  7. [7]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforce- ment learning. arXiv preprint arXiv:2501.12948 (2025)

    Pith/arXiv arXiv 2025

  8. [8]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778

    2016

  9. [9]

    Xin He, Shunkang Zhang, Yuxin Wang, Haiyan Yin, Zihao Zeng, Shao- huai Shi, Zhenheng Tang, Xiaowen Chu, Ivor Tsang, and Ong Yew Soon. 2024. Expertflow: Optimized expert activation and token al- location for efficient mixture-of-experts inference. arXiv preprint arXiv:2410.17954 (2024)

    arXiv 2024

  10. [10]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 (2020)

    Pith/arXiv arXiv 2020

  11. [11]

    Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and Xiaojuan Qi. 2024. Mixture Com- pressor for Mixture-of-Experts LLMs Gains More. arXiv preprint arXiv:2410.06270 (2024)

    Pith/arXiv arXiv 2024

  12. [12]

    Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang. 2024. Pre-gated moe: An algorithm- system co-design for fast and scalable mixture-of-expert inference. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 1018–1031

    work page 2024

  13. [13]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al . 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024)

    Pith/arXiv arXiv 2024

  14. [14]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Ben- jamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

    Pith/arXiv arXiv 2020

  15. [15]

    Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. [n. d.]. Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy. In The Twelfth International Conference on Learning Representations

    work page

  16. [16]

    Jianchun Liu, Jiaming Yan, Hongli Xu, Zhiyuan Wang, Jinyang Huang, and Yang Xu. 2023. Finch: Enhancing federated learning with hi- erarchical neural architecture search. IEEE Transactions on Mobile Computing (2023)

    work page 2023

  17. [17]

    Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. 2024. Not All Experts are Equal: Ef- ficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 6159–6172

    work page 2024

  18. [18]

    Jie Ren, Dong Xu, Shuangyan Yang, Jiacheng Zhao, Zhicheng Li, Christian Navasca, Chenxi Wang, Harry Xu, and Dong Li. 2024. En- abling large dynamic neural network training with learning-based memory management. In 2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 788–802

    work page 2024

  19. [19]

    Xiaoniu Song, Zihang Zhong, Rong Chen, and Haibo Chen. 2024. Promoe: Fast moe-based llm serving using proactive caching. arXiv preprint arXiv:2410.22134 (2024)

    Pith/arXiv arXiv 2024

  20. [20]

    Peng Tang, Jiacheng Liu, Xiaofeng Hou, Yifei Pu, Jing Wang, Pheng- Ann Heng, Chao Li, and Minyi Guo. 2024. Hobbit: A mixed preci- sion expert offloading system for fast moe inference. arXiv preprint arXiv:2411.01433 (2024)

    Pith/arXiv arXiv 2024

  21. [21]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  22. [22]

    Duo Wu, Xianda Wang, Yaqi Qiao, Zhi Wang, Junchen Jiang, Shuguang Cui, and Fangxin Wang. 2024. Netllm: Adapting large language models for networking. In Proceedings of the ACM SIGCOMM 2024 Conference. 661–678

    work page 2024

  23. [23]

    Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu

    work page

  24. [24]

    Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. 2024. MoE-Infinity: Offloading-Efficient MoE Model Serving. arXiv preprint arXiv:2401.14361 (2024)

    Pith/arXiv arXiv 2024

  25. [25]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    Pith/arXiv arXiv 2025

  26. [26]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guant- ing Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfen...

    Pith/arXiv arXiv 2024

  27. [27]

    Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. 2024. MoE- I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition. In Findings of the Association for Computational Linguistics: EMNLP 2024. 10456– 10466. 12 Accelerating Mixture-of-Expe...

    work page 2024

  28. [28]

    Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. 2021. Tokens-to- token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision. 558–567

    work page 2021

  29. [29]

    Yuping Yuan, Zhao You, Shulin Feng, Dan Su, Yanchun Liang, Xiaohu Shi, and Dong Yu. 2023. Compressed MoE ASR Model Based on Knowledge Distillation and Quantization. In Proc. Interspeech 2023. 3337–3341

    work page 2023

  30. [30]

    Shuzhang Zhong, Ling Liang, Yuan Wang, Runsheng Wang, Ru Huang, and Meng Li. 2024. Adapmoe: Adaptive sensitivity-based expert gating and management for efficient moe inference. In Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design. 1–9

    work page 2024

  31. [31]

    Yan Zhuang, Zhenzhe Zheng, Fan Wu, and Guihai Chen. 2024. Lite- MoE: Customizing On-device LLM Serving via Proxy Submodel Tuning. In Proceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems. 521–534. 13

    work page 2024

  32. [2020]

    In International conference on machine learning

    On layer normalization in the transformer architecture. In International conference on machine learning. PMLR, 10524–10533

    work page