Pith. sign in

REVIEW 4 major objections 4 minor 37 references

HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Routing a token's activated experts as a set, with a four-component cost model, cuts edge MoE latency by up to 59 percent.

desk verdict A coherent and well-argued framework for set-level MoE routing whose headline performance numbers rest on a simulator that uses the router's own cost function as ground truth. read the letter →

arxiv 2608.00577 v1 pith:LM2NOVBI submitted 2026-08-01 cs.NI cs.CL

Xin Yuan , Ning Li , Wenchao Xu , Athanasios V. Vasilakos , Song Guo , Haijun Zhang This is my paper
classification cs.NIcs.CL
keywords Mixture-of-ExpertsedgeinferencecollaborativeroutingGPU-CPUoffloadingmodelquantizationqueueingdelaydistributedLLMservingexpertreplication
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

When a token in a Mixture-of-Experts model activates its Top-k experts, those experts may live on different edge servers, in GPU or CPU memory, at different quantization precisions, behind links of different bandwidth, and on GPUs with different queues. HetRoute's central claim is that the right routing decision must weigh all four of these factors at once, and must choose the whole set of activated experts together rather than picking the cheapest server per expert. The paper argues that a local CPU-resident expert can be slower than a remote GPU-resident one, so local-first routing is wrong, and that per-expert greedy routing ignores the fan-out and fan-in coupling that makes a collaboration set expensive. If the claim is right, distributed edge MoE serving can get large latency, tail-latency, traffic, and throughput gains—up to 59 percent average latency, 58 percent P99 latency, 72 percent cross-server traffic, and 2.13x throughput—while staying inside a configured quality budget, because the cost model turns routing into a bottleneck-minimization problem with a quality-safe fallback.

What carries the argument

The load-bearing object is the unified per-assignment heterogeneous cost C_{i,l}(E,E',n), which sums four components: cross-server transmission of the hidden state (zero for local execution), GPU-CPU expert loading when the replica is CPU-resident, GPU computation plus instantaneous queueing delay Q_n(t)/F_n, and a quality penalty combining quantization loss and substitute-execution loss. The layer delay is the max over participating servers of fan-out transmission, compute-plus-loading, and fan-in transmission to the next aggregation server. The router minimizes this bottleneck cost over complete sets of assignments, and the offline deployment stages GPU residency and redundant replication

What would settle it

Run HetRoute and a per-expert greedy baseline on a physical 10-server heterogeneous edge testbed with real Internet links and PCIe transfers, measuring end-to-end latency for Mixtral-8x7B at 40 requests per second; if the measured relative gain over Prism or the P99-tail reduction is not close to the reported 28.1% and 58%, or if measured latency falls outside the model's predicted transmission-plus-queueing range, the central performance claim fails.

Watch

Extended reading notes

Core claim

The paper introduces HetRoute, a two-stage framework for distributed edge MoE inference. The offline stage fixes expert placement, GPU/CPU residency, and per-replica quantization by estimating how each decision changes the expected online routing cost under calibration traffic. The online stage treats each token-layer's Top-k activated experts as a set: it builds a candidate collaboration domain from exact and feasible substitute replicas, prunes candidates with a hard per-token quality guard and a normal per-server stability guard, and minimizes the bottleneck layer delay (the slowest fan-out, execute, and fan-in branch) by exact enumeration for small domains or beam search with 1-exchange

Load-bearing premise

The reported gains come from a trace-driven simulation whose delay formula is the same one the router minimizes; if real PCIe transfer, link contention, or queueing behaves differently, the absolute latency and throughput numbers may not transfer to a physical deployment.

Editorial extensions

If this is right

  • Distributed edge MoE inference should be treated as a set-level collaborative routing problem, not a per-expert or placement-only problem; the ablation attributes the largest latency increase (156 ms to 203 ms) to disabling set-level routing.
  • A remote GPU-resident hot expert can beat a local CPU-resident cold expert, so local-first and placement-only strategies are dominated whenever the cost model is accurate.
  • A full-precision exact replica per expert makes the system permanently quality-safe: emergency fallback pays latency but never violates the per-token quality budget.
  • Online routing cost depends on average replica count rather than number of servers, so the approach scales to larger edge deployments with modest candidate sets.
  • Relaxing the quality budget trades smoothly for latency (188 ms at 0.5% budget down to 142 ms at 3.0%), giving operators a tunable quality-latency knob.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same bottleneck-cost formulation could extend beyond MoE to other sparse-activation or conditional-computation models across heterogeneous devices, since the key interaction—fan-out/fan-in coupling versus parallel compute—is generic.
  • Editorial inference: the reported gains are trace-driven with the same delay model the router optimizes; a physical deployment would likely show smaller absolute gains unless the per-component costs are recalibrated online against measured PCIe contention and communication-computation overlap.
  • Editorial inference: the adaptive unit-cost feedback rule suggests a testable extension—learn the per-component costs online from observed delays instead of fixed coefficients—which could make the router robust to traffic and link-state drift.
  • Editorial inference: the exact-first substitute policy could be relaxed to deliberately select a cheaper high-quality substitute when the quality budget has slack; the paper does not explore this direction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes HetRoute, a two-stage framework for serving Mixture-of-Experts models over heterogeneous edge servers. An offline stage (Algorithm 1) decides expert placement, GPU/CPU residency, replication, and quantization precision by simulating the online router over calibration traffic; an online stage (Algorithm 2) routes each token's Top-k activated expert set layer by layer, minimizing a bottleneck layer cost that includes cross-server transmission, GPU-CPU offload, queueing, and quantization/substitution quality penalties. The paper proves several properties of this design: the emergency fallback is always quality-feasible, the participating server count is bounded by k, exact enumeration is per-layer optimal for small candidate domains, the 1-exchange refinement is locally optimal, and online complexity is independent of the number of servers. Trace-driven experiments on three MoE models report large latency, traffic, and throughput gains over four baselines, with controlled quality loss.

Significance. If the reported results are taken at face value, HetRoute makes a useful contribution to distributed edge MoE inference: it is one of the few works that jointly model expert placement, memory tier, quantization, and online routing under a single cost structure, and it correctly identifies the set-level trade-off between fan-out/fan-in parallelism and per-branch cost. The formal properties in Section V.C are argued cleanly from the stated constraints and give the design a solid algorithmic backbone. The main caveat is evidential: the quantitative claims (59% latency reduction, 2.13x throughput) are computed with the same bottleneck model that the router optimizes, so the gains are, as presented, properties of the model rather than validated end-to-end system behavior.

major comments (4)
  1. [§VI.B, Fig. 2; Eq. (10)] The central performance claims rest on a circular evaluation. Eq. (10) is at once the router's objective (P_{\tau,l} in §V.B), the offline replication benefit (Eq. 14), and the ground-truth latency generator in the 'testbed' of §VI.A. The simulated servers and links use the same linear backlog term Q_n(t)/F_n, non-overlapped offload w/\beta_n, and transmission d/link with no contention or overlap. Therefore Fig. 2 measures how well the router optimizes its own model, not whether the model predicts real edge behavior. The authors should either report measurements from a physical testbed, or at minimum calibrate each component of Eq. (10) against real PCIe/network/GPU data and add a sensitivity analysis showing that errors in the model (e.g., non-linear queueing, communication-computation overlap) do not erase the claimed gains.
  2. [§IV, Algorithm 1 lines 3–14; Eqs. (13)–(14)] There is an offline-online circularity that is structurally related to the evaluation issue. Stage 1 obtains p^{route} by simulating Algorithm 2 on calibration traffic, then uses p^{route} to compute residency benefits (Eq. 13) and replication benefits (Eq. 14), which in turn shape the candidate domain that Algorithm 2 later searches. This is a coherent co-design, but it means the placement is tuned to the same cost model that the evaluation uses as ground truth. The paper should state this limitation explicitly and test robustness to model mismatch (e.g., perturb the cost coefficients or inject a correlated error term) to show the design does not degrade catastrophically when the model is wrong.
  3. [§III.C, Eq. (8); §VI.D] The quality degradation model is presented as an additive surrogate and the text says its correlation with task metrics 'is validated empirically in Section VI.' However, Section VI only reports final perplexity/F1/accuracy numbers for the complete routing policy; it does not validate the additive form, nor the profiled per-replica quantization losses, nor the substitution losses. Please add a direct comparison of the additive surrogate against measured end-to-end quality on the calibration set (e.g., scatter plot or correlation coefficient) so that the quality budget constraint (c.9) and the latency-quality trade-off in Fig. 4(b) are supported.
  4. [§V.B, Eq. (9) references] Equation numbering is inconsistent in the algorithmic parts. The per-assignment cost is defined in Eq. (9), but §V.A and Algorithm 1 line 10 refer to it as 'C in (11)' and '(11')'. Similarly, §III.D calls Eq. (10) a layer delay while §V.B denotes the same objective D_{\tau,l}. Please renumber and unify all cross-references before resubmission to avoid ambiguity in the specification.
minor comments (4)
  1. [§VI.A] Baseline citations are placeholders: 'Prism [], EdgeShard [], MoE-Infinity [], Petals []' appear without reference numbers. Please fill in the actual bibliographic entries.
  2. [Property 4] The proof says the feasible set is finite '(at most R̄^q complete assignments)'. I believe this should be R̄^k (or the enumeration threshold R̄_q), because each of the k targets has at most R̄ candidates. Please correct the notation.
  3. [§VI, figures] Several captions (Figs. 2, 3, 6, 7, 8) lack axis units and legend descriptions. For example, Fig. 2(a) should state that average latency is in milliseconds, and Fig. 2(c) should state the traffic unit (GB per one thousand tokens). This is important for reproducibility and comparison.
  4. [§III.D, Eq. (10)] The fan-in phase uses d^{out}, which is described in prose but not defined until the sentence after Eq. (10). Please define it in the notation block near Eq. (9) to keep the model self-contained.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported latency/throughput gains are computed with the same bottleneck cost model (Eq. 10) that the online router minimizes, so the headline reductions are the router optimizing its own objective.

  1. self definitional [Sec. III-D Eq. (10); Sec. V-B P_{i,l}; Sec. VI-B Fig. 2(a)]
    "Because the 𝑘 activated experts run in parallel ..., the layer delay is determined by the slowest branch, i.e., it is a bottleneck (max) rather than a sum. Specifically, the layer-𝑙 delay ... 𝐷𝑖,𝑙=... (10). ... Consequently, the end-to-end inference delay ... is 𝐷𝑖,tot=∑𝐷𝑖,𝑙. ... The router therefore solves the per-layer subproblem: 𝑃𝑖,𝑙:min𝒛𝐷𝑖,𝑙(𝒛). ... As shown in Fig. 2(a), HetRoute attains the lowest average latency of 156 ms, which is 59.0% lower than EdgeShard."

    The evaluation reports as 'latency' the same expression D_{i,l} in Eq. (10) that Algorithm 2 explicitly minimizes in the per-layer subproblem P_{i,l}. The trace-driven testbed is parameterized with the same transmission, GPU-CPU offload, queueing (Q_n(t)/F_n), and fan-in/fan-out terms used in the cost model, and no independent end-to-end measurement on real hardware is described. Hence the claimed 59.0% average-latency reduction and 2.13x throughput improvement are statements about the router minimizing its own cost model: by construction, a method that optimizes a cost C will reduce C relative to methods that optimize a different objective, when C is used as the measurement. The result is real only to the extent Eq. (10) faithfully models a physical edge system, which the paper does not i

full rationale

The strongest circularity is in the evaluation, not in the algorithm design. In Sec. III-D the paper defines end-to-end latency as the sum of the bottleneck layer delays D_{i,l} in Eq. (10), with transmission, GPU-CPU loading, queueing Q_n(t)/F_n, and fan-in/fan-out terms. In Sec. V-B the online stage minimizes exactly this D_{i,l} as P_{i,l}. The evaluation section then reports 'average latency' and 'throughput' from a trace-driven testbed whose server/link parameters are the same as those in Eq. (10), and no independent end-to-end measurement on actual hardware is described. Consequently, the headline reductions are the router minimizing its own cost model; they are real only to the extent that Eq. (10) faithfully models a real edge system, which is not established. This is a self-definitional validation loop, not a mathematical circularity in the offline-online decomposition. The offline stage's use of p^route from simulating Algorithm 2 is a deliberate feedback design and would not by itself be circular; it becomes part of the problem only because the same model is used as ground truth in evaluation. The self-citation to [38] for substitute execution is a dependency, but the paper's own theoretical properties (fallback feasibility, bounded participation, enumeration optimality) are proven from constraints (c.4)-(c.9) and do not reduce to that citation, so it does not add circularity. Score 6 reflects that the headline performance claims reduce by construction to the objective being optimized.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The framework contributes a routing and placement algorithm plus a cost model, but the cost model's numerical predictions rest on the axioms above, and several weighting coefficients (lambda_q, mu, omega, alpha_j, rho_bar) are chosen by hand rather than fitted to independent data. The heaviest external burden is the dependency on the authors' own companion paper [38] for substitute execution.

free parameters (6)
  • lambda_q (quality coefficient in Eq. 9) = not specified
    Scales the quantization plus substitution quality penalty inside the per-assignment cost C_i,l; used in offline replication and candidate evaluation. Its value is not reported.
  • alpha_j(0) and rho_bar_j (unit costs and target shares, Eq. 21) = not specified
    Initial per-component unit costs and the target share vector that drives the projected feedback rule; these shape all online cost decisions and are not given in the paper.
  • mu (memory price, Eq. 14) = not specified
    Prices GPU/CPU memory in the replication benefit b_{E,n}; chosen by hand and not reported.
  • omega_d, omega_q (utility weights, Eq. 12) = not specified
    Tradeoff weights between normalized delay and normalized quality in P1; set by the operator; only the sum constraint is given.
  • Lambda_max (per-token quality budget) = 2% default
    The hard quality bound in guard (18); controls how much quantized and substitute execution is allowed and thus the reported quality-latency tradeoff.
  • beam width W, enumeration threshold R, local threshold tau_loc, replication cap K_rep, scheduling window Delta_t, memory = W=8, R=64, memory ratio=2.0; others unstated
    Algorithm knobs that directly determine routing behavior and reported performance; some values are given only in Section VI, others not at all.
assumptions (4)
  • domain assumption Additivity of quality degradation (Eq. 8)
    The per-token budget is the sum of per-assignment quantization and substitution losses across layers and experts. The paper calls it a first-order approximation and claims validation in Section VI, but no correlation analysis is shown.
  • domain assumption Bottleneck delay model (Eq. 10) with linear queueing term Q_n(t)/F_n
    Layer delay is the max of fan-out, parallel compute with additive queueing backlog, and fan-in; assumes known instantaneous backlogs and linear scaling, with no overlap or contention beyond the backlog term. This same model is the simulator's ground truth.
  • domain assumption Substitute-execution model imported from companion paper [38]
    Feasible substitute sets V(E,sim), substitution quality Q_sub, and the redundant-deployment principle come from the authors' own unpublished OrderMoE paper; its similarity model is unverifiable here.
  • domain assumption Full-precision exact replica invariant (c.4) is feasible within memory budgets
    The fallback guarantee assumes at least one full-precision replica of every expert fits in the system's aggregate memory and is deployed; feasibility under memory constraints (c.1)-(c.2) is asserted, not proven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference." pith.science (2026). https://pith.science/paper/LM2NOVBI

@misc{pith2026260800577,
  author       = {Pith},
  title        = {Pith review of: HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LM2NOVBI}},
  note         = {Machine review of arXiv:2608.00577}
}
read the original abstract

Mixture-of-Experts (MoE) models have become a dominant architecture for large-scale AI services, yet deploying them over geo-distributed heterogeneous edge servers remains challenging. When the Top-k activated experts of a token are spread across multiple servers, the optimal routing depends jointly on cross-server link bandwidth, heterogeneous GPU computing capability, GPU-CPU expert loading delay, instantaneous queueing backlog, and replica-level quantization quality loss. Existing distributed inference and MoE serving methods address these factors separately and do not provide a unified framework for online multi-server collaborative routing. In this paper, we propose HetRoute, a heterogeneous-cost-aware collaborative routing framework for distributed edge MoE inference. HetRoute introduces a unified per-assignment cost model that explicitly captures four cost components: cross-server transmission, GPU-CPU offloading, GPU computation with queueing, and quantization-induced quality penalty. Guided by this model, the offline stage determines expert server placement, GPU-CPU residency, and replica precision through a routing-cost-coupled deployment algorithm, while the online stage routes the Top-k activated expert set as a whole by minimizing the bottleneck layer cost via exact enumeration or beam search. Theoretical analysis establishes fallback feasibility, a bound on the number of participating servers, per-layer optimality for small candidate domains, and online computational complexity. Trace-driven evaluation on three MoE models over a heterogeneous 10-server edge testbed shows that HetRoute reduces average inference latency by up to 59.0% and P99 latency by up to 58.0%, cuts cross-server traffic by up to 72.1%, and achieves 2.13x throughput improvement compared with representative baselines, while keeping quality degradation within the configured budget.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 32 canonical work pages

  1. [38]

    OrderMoE: An expert similarity driven distributed edge MoE inference

    X. Yuan, N. Li, Q. Chen, W. Xu, A. V. Va s i l a k o s , S. Guo, H. Zhang, “OrderMoE: An expert similarity driven distributed edge MoE inference,” arXiv: 2607.17154, 2026, pp: 1-17

    work page Pith review arXiv 2026

  2. [1]

    A Survey of Large Language Models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, et al., "A Survey of Large Language Models," Frontiers of Computer Science, vol,20, no.12, 2026, pp: 1-40

    work page 2026

  3. [2]

    Llama 2: Open Foundation and Fine-Tuned Chat Models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models," arXiv:2307.09288, 2023, pp: 1-77

    arXiv 2023

  4. [3]

    GPT-4 Technical Report,

    OpenAI, "GPT-4 Technical Report," arXiv:2303.08774, 2023, pp: 1-100

    arXiv 2023

  5. [4]

    A Survey on Large Language Model Based Autonomous Agents,

    L. Wang, C. Ma, X. Feng, Z. Zhang, et al., "A Survey on Large Language Model Based Autonomous Agents," Frontiers of Computer Science, vol.18, no.6, 2024, pp: 1-26

    work page 2024

  6. [5]

    A Survey on Multimodal Large Language Models,

    S. Yang, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, E. Chen, "A Survey on Multimodal Large Language Models," National Science Review, vol.11, no.12, 2024, pp: 1-21

    work page 2024

  7. [7]

    EdgeShard: Efficient LLM Inference via Collaborative Edge Computing,

    M. Zhang, X. Shen, J. Cao, Z. Cui, S. Jiang, "EdgeShard: Efficient LLM Inference via Collaborative Edge Computing," IEEE Internet of Things Journal, vol.12, no.10, 2024, pp: 13119-13131

    work page 2024

  8. [8]

    FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU,

    Y. S h e n g , L. Zheng, B. Yuan, Z. Li, M. Ryabinin, et al., "FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU," ICML, 2023, Hawaii, USA, pp: 31094 - 31116

    work page 2023

Show all 37 references
  1. [9]

    Mobile Edge Intelligence for Large Language Models: A Contemporary Survey,

    G. Qu, W. Chen, W. Wei, Z. Lin, X. Chen, K. Huang, "Mobile Edge Intelligence for Large Language Models: A Contemporary Survey," IEEE Communications & Tutorials, vol.27, no.6, 2025, pp: 3820-3860

    2025

  2. [10]

    A Review on edge large language models: Design, Execution, and Applications,

    Y. Zheng, Y . Chen, B. Qian, X. Shi, Y . Shu, J. Chen, “A Review on edge large language models: Design, Execution, and Applications,” ACM Computing Surveys, vol.57, no.8, 2025, pp: 1-35

    2025

  3. [11]

    LLM-based edge intelligence: A Comprehensive survey on Architectures, Applications, Security and Trustworthiness,

    O. Friha, M.A. Ferrag, B. Kantarci, B. Cakmak, et al., “LLM-based edge intelligence: A Comprehensive survey on Architectures, Applications, Security and Trustworthiness,” IEEE Open Journal of the Communication Society, vol.5, 2024, pp: 5799-5856

    2024

  4. [12]

    Efficient Inference for edge large language models: A Survey,

    G. Cai, R. Tian, L. Yang, Y . Jia, L. Li, J. Wang, “Efficient Inference for edge large language models: A Survey,” Tsinghua Science and 15 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID Technology, vol.31, no.6, 2026, pp: 1365-1380

    2026

  5. [13]

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer,

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer," ICLR, 2017, Toulon, France, pp: 1-19

    2017

  6. [14]

    Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity," Journal of Machine Learning Research, vol. 23, no. 1, 2022, pp. 5232-5270

    2022

  7. [15]

    Mixtral of Experts,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, et al., "Mixtral of Experts," arXiv:2401.04088, 2024, pp: 1-13

    2024 arXiv

  8. [16]

    MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache,

    L. Xue, Y . Fu, Z. Lu, L. Mai, M. Marina, "MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache," arXiv:2401.14361, 2024, pp: 1-11

    2024 arXiv

  9. [17]

    Tutel: Adaptive Mixture-of-Experts at Scale,

    C. Hwang, W. Cui, Y . Xiong, Z. Yang, Z. Liu, H. Hu, et al., "Tutel: Adaptive Mixture-of-Experts at Scale," MLSys, 2023, Miami, USA, pp: 1-19

    2023

  10. [18]

    MegaBlocks: Efficient Sparse Training with Mixture-of-Experts,

    T. Gale, D. Narayanan, C. Young, and M. Zaharia, "MegaBlocks: Efficient Sparse Training with Mixture-of-Experts," MLSys, 2023, Miami, USA, pp: 1-17

    2023

  11. [19]

    MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs,

    Z. Jiang, H. Lin, Y . Zhong, Q. Huang, Y . Chen, Z. Zhang, et al., "MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs," USENIX NSDI, 2024, CA, USA, pp: 745-760

    2024

  12. [20]

    D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving,

    H. Wang, Q. Zhou, Z. Hong, S. Guo, " D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving," ACM MobiCom, 2025, Hong Kong, China, pp: 574-588

    2025

  13. [21]

    SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models,

    Z. Du, S. Li, Y . Wu, et al., "SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models," MLSys, 2024, pp: 1-15

    2024

  14. [22]

    Petals: Collaborative Inference and Fine-tuning of Large Models,

    A. Borzunov, M. Ryabinin, A. Chumachenko, D. Baranchuk, T. Dettmers, Y . Belkada, et al., "Petals: Collaborative Inference and Fine-tuning of Large Models," ACL, 2023, Toronto, Canda, pp: 558-568

    2023

  15. [23]

    Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement,

    T. Wu, L. Wang, Z. Wen, X. Zhang, X. Chen, J. Duan, X. Zhang, J. Zuo, "Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement," arXiv:2508.12851, 2025, pp: 1-11

    2025 arXiv

  16. [24]

    EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices,

    R. Yi, L. Guo, S. Wei, A. Zhou, S. Wang, M. Xu, "EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices," IEEE Transactions on Mobile Computing, vol.24, no.8, 2025, pp: 7059-7073

    2025

  17. [25]

    Mixture-of-Experts for Distributed Edge Computing with Channel-Aware Gating Function,

    Q. Song, S. Jing, S. Zhang, S. Zhang, C. Huang, " Mixture-of-Experts for Distributed Edge Computing with Channel-Aware Gating Function," IEEE ICC, 2025, Montreal, Canda, pp: 1-6

    2025

  18. [26]

    Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge,

    Y. K a n g , J . H a u s w a l d , C . G a o , A . R o v i n s k i , T. M u d g e , J . M a r s , a n d L . Tang, "Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge," ACM SIGARCH Computer Architecture News, vol.45, no.1, 2017, pp: 615-629

    2017

  19. [27]

    Distributed Deep Neural Networks over the Cloud, the Edge and End Devices,

    S. Teerapittayanon, B. McDanel, and H. T. Kung, "Distributed Deep Neural Networks over the Cloud, the Edge and End Devices," IEEE ICDCS, GA, USA, 2017, pp: 1-12

    2017

  20. [28]

    JointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Services,

    A. E. Eshratifar, M. S. Abrishami, and M. Pedram, "JointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Services," IEEE Transactions on Mobile Computing, vol. 20, no. 2, 2021, pp. 565-576

    2021

  21. [29]

    Mobility and Cost Aware Inference Accelerating Algorithm for Edge Intelligence,

    X. Yuan, N. Li, K. Wei, W. Xu, Q. Chen, H. Chen, S. Guo, “Mobility and Cost Aware Inference Accelerating Algorithm for Edge Intelligence,” IEEE Transactions on Mobile Computing, vol.24, no.3, 2025, pp: 1530 -

    2025

  22. [30]

    Joint Optimization of Model Partitioning and Resource Allocation for Edge Computing with Intermittently Operating Devices,

    Q. Wu, Y . Zhang, C. Yang, J. Sun, "Joint Optimization of Model Partitioning and Resource Allocation for Edge Computing with Intermittently Operating Devices," IEEE ICPADS, Ocean Flower Island, China, 2023, pp: 1-8

    2023

  23. [31]

    HAPI: Hardware-Aware Progressive Inference,

    S. Laskaridis, S. I. Venieris, H. Kim, and N. D. Lane, "HAPI: Hardware-Aware Progressive Inference," IEEE/ACM ICCAD, CA, USA, 2020, pp: 1-9

    2020

  24. [32]

    Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing,

    E. Li, L. Zeng, Z. Zhou, and X. Chen, "Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing," IEEE Transactions on Wireless Communications, vol. 19, no. 1, 2020, pp. 447-457

    2020

  25. [33]

    Large Scale Distributed Deep Networks,

    J. Dean, G. Corrado, R. Monga, et al., "Large Scale Distributed Deep Networks," NeurIPS, Lake Tahoe, 2012, pp: 1223-1231

    2012

  26. [34]

    QLoRA: Efficient Finetuning of Quantized LLMs,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, "QLoRA: Efficient Finetuning of Quantized LLMs," NeurIPS, New Orleans, USA, 2023, pp: 1-28

    2023

  27. [35]

    OPTQ: Accurate Quantization for Generative Pre-trained Transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, "OPTQ: Accurate Quantization for Generative Pre-trained Transformers," ICLR, Kigali Rwanda, 2023, pp: 1-16

    2023

  28. [36]

    SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models," ICML, Hawaii, USA, 2023, pp: 1-13

    2023

  29. [37]

    AWQ: Activation-Aware Weight Quantization for LLM Compression and Acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han, "AWQ: Activation-Aware Weight Quantization for LLM Compression and Acceleration," MLSys, Santa Clara, USA, 2024, pp: 1-15

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.