TrimMoE A communication aware and adaptive depth framework for distributed edge inference

Athanasios V. Vasilakos; Haijun Zhang; Ning Li; Shuting Bai; Song Guo; Wenchao Xu; Xin Yuan

REVIEW 4 major objections 4 minor 37 references

TrimMoE A communication aware and adaptive depth framework for distributed edge inference

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TrimMoE cuts average latency by up to 62.8% in distributed edge MoE inference by skipping low-importance layers and exiting early under a confidence gate, while keeping quality loss within a 2% budget.

desk verdict A solid joint skip/exit/server-selection system for edge MoE, with an overclaimed quality guarantee; referee it. read the letter →

arxiv 2608.00573 v1 pith:OUCESMLI submitted 2026-08-01 cs.NI cs.CL

Ning Li , Shuting Bai , Xin Yuan , Wenchao Xu , Athanasios V. Vasilakos , Song Guo , Haijun Zhang This is my paper · ORCID

classification cs.NIcs.CL

keywords edgeinferencemixture-of-expertsearlyexitlayerskippingdistributedcommunicationreductionLLMservingqualitybudget

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Serving Mixture-of-Experts LLMs across edge servers is bottlenecked by cross-server expert transmission. TrimMoE's central claim is that a large fraction of that traffic is unnecessary: individual tokens pass through layers that contribute little to them, and many tokens reach a stable representation before the final layer. The paper therefore proposes to decide online whether each layer, and the layers after it, should be executed at all, coupling layer skipping with confidence-gated early exit, substitute expert execution, and server-expert selection under a per-user quality budget. On a 10-server heterogeneous testbed with three MoE models, it reports up to 62.8% lower average latency, 77.2% lower cross-server traffic, a lower remote-execution ratio, and sustained throughput while keeping measured task-quality degradation within 2%. The hard guarantee proved in the paper bounds the calibrated proxy degradation by the configured budget; the quality result on real tasks rests on how faithfully those proxies track task quality.

What carries the argument

Key machinery: layer importance—the aggregate gating mass of a token's activated experts (Eq. 4), known before the expert executes and gating whether a layer may be skipped; the skip-degradation curve Λ = Δ_l(I) (Eq. 5, calibrated in Eq. 24), converting importance into a budgeted quality cost; per-layer linear exit heads trained to predict consistency with the full-depth output (Eqs. 22-23), forming the confidence gate for early exit; a look-ahead cost-to-go recursion over expert transition statistics (Eqs. 38-39), letting online selection anticipate token movement; and a skip/exit-aware redundancy benefit (Eq. 28) that places expert replicas where they actually cut transmission after depth

What would settle it

Run TrimMoE on live traffic drawn from a distribution different from the calibration set (e.g., a different domain, longer prompts, or a different task) and compare the internal cumulative proxy degradation against the measured task metric (perplexity, F1, accuracy): if measured degradation exceeds the configured 2% budget while the proxy reports compliance, the proxy-to-quality link is the failure point.

Watch

Extended reading notes

Core claim

The paper reframes distributed edge MoE serving: instead of asking how to reach a remote expert faster, ask whether that layer and all remaining layers need to be executed at all. It formulates a joint optimization over expert deployment, server-expert selection, layer skipping, and early exit, with a common probabilistic degradation budget for substitution and skipping. Early exit is admitted only when a per-layer exit head's calibrated confidence meets the user's requirement, and a skipped layer never triggers exit. Offline, the backbone is frozen while exit heads, importance thresholds, and skip-exit-aware replica placement are calibrated; online, a transition-aware look-ahead recursion s

Load-bearing premise

The proved 2% budget applies to estimated, calibration-derived proxy losses (substitution and skip degradation) and to the exit confidence gate; if those estimates stop tracking real task quality under distribution shift, the measured quality degradation can exceed 2% even while the algorithm never violates its own budget.

Editorial extensions

If this is right

Depth reduction can be a first-class lever for cutting communication in distributed MoE inference, not just a compute saver: skipping a layer whose experts are all remote removes that transmission entirely, and early exit cancels all remaining hops at once.
Skipping should be suppressed when a local execute candidate exists; the paper's transmission necessity score makes the skip decision communication-aware rather than purely importance-based.
A hard per-user quality budget can be enforced online by checking each action's proxy degradation before admission, with exact execution as an always-available fallback.
A small look-ahead horizon (H=3 in the experiments) captures most of the latency and traffic gains and saturates beyond it, keeping online overhead modest.
The latency advantage over baselines grows with model scale, from Switch-Base-8E to Mixtral-8x7B, because larger models have more transmission-prone layers for skipping and early exit to remove.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the proved 2% budget is a bound on calibrated proxy degradation (substitution loss plus skip-degradation estimates), so on a shifted live distribution the measured task quality could exceed 2% while the internal budget check still reports compliance. A natural test is to log the proxy budget alongside real task metrics under distribution shift.
Beyond the paper: the same skip/exit machinery could extend to attention-sharded deployments, where cross-server traffic is not purely expert-driven; the conclusion already flags attention sharding as future work.
Beyond the paper: the confidence gate could double as a scheduling signal—tokens that become confident early could be routed to cheaper servers, while tokens that never gain confidence could be reserved for full-depth exact execution.
Beyond the paper: the layer-importance thresholds and transition statistics could be updated online rather than fixed at calibration, making the look-ahead and skip decisions robust to evolving workloads and content mixes.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a real systems contribution, but the paper sells a proxy guarantee as a task-quality guarantee. The core idea is fresh—nobody has coupled MoD-style skipping, CALM-style early exit, and substitute execution into a single communication-aware edge MoE scheduler. The joint formulation and the skip/exit-aware redundancy benefit are the actual new pieces, and the testbed results on three MoE models are consistent with the story: latency down 62.8% vs EdgeShard, cross-server traffic down 77.2%, remote-execution ratio from 71% to 16%, throughput up.

The offline calibration (exit heads, importance thresholds, transition statistics) is standard but competently done, and the look-ahead recursion is a reasonable way to make depth-reduction transmission-aware. The ablation isolates what each component buys, which is more than most systems papers do.

Now the soft spots. Property 2 is not a guarantee about task quality; it is a check that the online algorithm never admits an action that would push the cumulative calibration proxy over the budget. Since Eq. (34) enforces exactly that inequality, the induction cannot fail. The abstract's "task-quality degradation within a 2% bound" is stronger than anything proved. What is proved is a bound on D_i, a sum of calibration-set prediction-change fractions; that sum is not a task metric and can double-count tokens. Early exit contributes zero to D_i, so Property 2 says nothing about exit-induced loss; Property 3 only says the exit head confidence passed a threshold on calibration. The measured perplexity/F1/GSM8K numbers are the real evidence, and they look fine, but they are not covered by the formal claim.

There is also a units problem in Eq. (33): dividing a 2% per-user budget by the token count gives a per-token budget that seems inconsistent with the reported 11% skip and 12% exit rates unless the per-event degradation is much smaller than the budget. That should be clarified.

Missing error bars and missing artifacts keep me from treating the headline numbers as established. But the paper is clear enough, the literature coverage is honest, and the conclusion actually draws the proxy/task distinction that the abstract blurs. I would send it to a serious referee, with instructions to push on the quality-budget claim and ask for code/data, and I would cite it if I worked on edge MoE inference.

Referee Report

4 major / 4 minor

Summary. TrimMoE addresses cross-server expert transmission in distributed edge MoE serving by reducing depth: it skips low-importance layers, exits early under a confidence gate, and selects server-expert pairs, all under a per-user quality budget. Offline Algorithm 1 trains per-layer exit heads, calibrates skip thresholds, and replicates experts based on a skip/exit-aware benefit; online Algorithm 2 evaluates per-token actions with a look-ahead cost-to-go. Experiments on a 10-server testbed with Switch-Base-8E, Qwen-MoE-A2.7B, and Mixtral-8x7B report up to 62.8% average latency reduction and reduced cross-server traffic while keeping task-quality degradation within a 2% bound. Formal properties assert deployment feasibility, a quality-budget guarantee, confidence-preserving exit, and polynomial complexity.

Significance. The core idea—eliminating transmissions by skipping layers and exiting early rather than accelerating remote-expert access—is well motivated for bandwidth-limited edge networks. The paper provides a detailed formulation, offline calibration with a frozen backbone, and a nontrivial online selection algorithm. The evaluation covers three model families, five baselines, ablations, and load variation, which is a solid experimental scope for a systems paper. If the empirical results are reproducible, the approach is a meaningful step beyond prior work. However, the formal guarantee in Property 2 applies to a calibration proxy, not to the task-quality metrics named in the abstract, and the proof is essentially a restatement of the admission rule. The empirical section lacks error bars and artifacts. The contribution is promising, but the central claims need re-scoping or additional validation.

major comments (4)

[Section V-F, Property 2 vs. Abstract] The abstract claims TrimMoE keeps 'task-quality degradation within a 2% bound,' but Property 2 proves only D_i <= D_i^max, where D_i in Eqs. (17)-(18) is the sum of calibration-set prediction-change fractions for substitution (Q_l) and skip (Lambda_l). These are not task metrics; summing them across tokens and layers double-counts tokens whose predictions change at multiple layers, and early exit contributes zero to D_i by design. The proof then shows that only budget-satisfying actions are admitted by (34), so the 'guarantee' is a restatement of the admission rule rather than a task-quality bound. The authors should either validate the proxy-to-task relationship on held-out data or explicitly limit the abstract's quality claim to the proxy.
[Eq. (33) and Section VI] The per-token budget D_bar_i = D_i^max/|Q_i| has undefined units. If D_i^max is the 2% budget used in Fig. 6(b) and |Q_i| is the number of tokens in a user request, the per-token budget becomes 0.002% for a 1000-token request, making any skip with Lambda>0 infeasible. Yet Section VI-C reports an 11% skip rate. The calibration tolerance in Eq. (25) inherits the same ambiguity and would push thresholds toward zero for long requests. Please define the unit of D_i^max and reconcile the budget with the reported skip/exit rates.
[Section IV-A, Property 3, and Eq. (37)] Early exit is gated by a confidence score trained to predict whether the layer-l representation gives the same next-token/full-depth prediction (Eqs. (22)-(23)). This does not imply task-level quality (e.g., SQuAD F1 or GSM8K accuracy), and a sigmoid head trained with binary cross-entropy is not automatically well-calibrated. The adaptive update (37) is described as adjusting to the 'false-exit rate,' but the equation moves p_task toward the average confidence of exits, which cannot correct miscalibration. Property 3's 'quality-preserving by construction' is therefore overstated; the paper should report calibration curves (e.g., reliability diagrams) and task-quality loss conditional on exit layer.
[Section VI, Figs. 4-11] The empirical comparisons report point estimates without error bars, confidence intervals, significance tests, or an artifact. Since the headline numbers (62.8% latency reduction, 2% quality bound) are central to the paper's claims, the absence of repeated-run statistics makes it impossible to assess variability across network conditions and loads. Please add repeated trials/error bars and release the testbed configuration and code, or clearly state why a single run is representative.

minor comments (4)

[Eqs. (8), (34), Algorithm 2] The exit condition is written as xi_{i,q,l} >= p in (8) but as xi_{i,q,l-1} >= p in (34); Algorithm 2 updates the state after execution. Please standardize the index for the layer after which exit is evaluated.
[Eq. (38)] The transition probability definition appears garbled; the denominator is missing the summation index and limits. Please restate it clearly.
[Section IV-F, Section V-E] Table I and Table II are algorithm pseudocode blocks, not tables. Label them as Algorithm 1 and Algorithm 2 rather than using table captions.
[Section VI-D, Fig. 6(b)] The claim that the measured degradation 'always stays strictly below the budget bound' is hard to evaluate because the y-axis unit is unspecified. Clarify whether the plotted quantity is the proxy D_i or a task metric.

Circularity Check

1 steps flagged · score 4.0 of 10

Quality-budget 'guarantee' restates the admission rule; the abstract's 2% task-quality bound is an input budget, while latency/traffic results remain independent.

self definitional [Section V-F (Property 2), Section V-B Eq. (34), and Abstract]
"The execute candidate is admitted into A_{d,q,l} only if D̂_{d,q,l}+Q_l(r_l(q),k*)≤D̄_i by the budget condition in (34). Therefore D̂_{d,q,l+1}=D̂_{d,q,l}+Q_l(r_l(q),k*)≤D̄_i."

Property 2's inductive step is exactly the membership test in Eq. (34): an action is in the feasible set only if the accumulated proxy loss stays within the per-token budget. Summing these token bounds via Eq. (33) makes D_i≤D_i^max follow by arithmetic. Thus the 'quality-budget guarantee' is an accounting identity over the algorithm's own admission filter, not an independently derived bound. The abstract then upgrades this to 'task-quality degradation within a 2% bound,' but D_i is a sum of calibration consistency statistics (Eqs. 17-18), early exit contributes zero to D_i by design, and the sigmoid exit head's calibration is assumed rather than proved. The 2% is a configured input budget, not a derived task-quality bound. This is a definitional/proxy circularity in the formal quality cla

full rationale

The main latency/communication story is not circular: the online algorithm minimizes a cost built from transmission and computation models, and the reported 62.8% latency reduction, traffic cuts, and throughput gains are testbed measurements against external baselines, not consequences of the budget proof. The one self-referential spot is the formal quality guarantee. Property 2 proves D_i≤D_i^max by induction over the feasible-action set in Eq. (34), but Eq. (34) defines feasibility as 'accumulated proxy loss ≤ per-token budget'; therefore the theorem restates the admission rule plus the arithmetic of Eq. (33). The abstract's 'task-quality degradation within a 2% bound' upgrades this proxy identity into a task-level claim that the proof does not support. The paper itself, in the conclusion, scopes the guarantee to 'substitution-and-skipping proxy degradation,' which is the manuscript's own limitation statement and mitigates the severity. The self-citation [37] supplies the substitution and deployment basis, but since Substitute-Only [37] is benchmarked as a baseline, that citation is not load-bearing in a circular way. Overall, the circularity is partial and confined to the quality-guarantee wording; the central latency claim has independent empirical content.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The list captures the quantities and assumptions the central claims depend on. The trained exit heads, calibrated importance thresholds, skip-degradation curves, and calibration statistics are all fitted to the calibration set; the quality budget operates on these fitted proxies. The main assumptions are that gating mass is a valid importance signal, that calibration transfers to runtime, and that substitution and skip losses are additive on a common scale.

free parameters (6)

exit head weights h_l = not listed (trained per layer)
Trained on calibration consistency labels (Eq. 22-23); the confidence gate and Property 3 depend entirely on these fitted heads.
layer importance thresholds theta_l = not listed (per-layer values)
Calibrated from the skip-degradation curve via Eq. (24)-(25); controls which layers become skip candidates.
skip degradation estimate Delta_l(I) = empirical per-layer curve
Used as Lambda in Eq. (5) and in the skip cost (Eq. 32); it is an average calibration prediction-change fraction, not a measured task-quality loss.
confidence requirement p_task and adaptive floor p = 0.9 default
User/operator chosen; sets the exit admission gate and strongly influences the quality-latency tradeoff.
transition statistics P_l(k to k') and token-similarity centroids = calibration-derived
Used by the look-ahead recursion (Eq. 38-39) and exit-point prediction (Eq. 36); these empirical statistics determine the predicted future costs.
redundancy benefit factors rho_{i,l}, average bandwidth, average latency = estimated from access and network data
Used in Eq. (28) for replica placement; these estimates shape the deployment that the online algorithm assumes.

assumptions (6)

domain assumption The gating mass I in Eq. (4) is a valid per-token importance signal, and low-importance layers can be skipped with bounded degradation.
Underlies skip admission (Eq. 5, Eq. 24, Algorithm 2 line 10); if gating mass does not track layer value, skipping can damage quality beyond the modeled loss.
domain assumption Calibration-derived statistics (consistency labels, Delta, transition counts, exit centroids) remain representative at runtime.
The confidence gate and skip degradation model transfer from calibration to live traffic; the adaptive rules (37) and (43) are only partial corrections.
ad hoc to paper Substitution loss and skip loss are additive on a common prediction-change scale.
Eq. (17) sums Q and Lambda into a single budget; this additive model is introduced to make the budget tractable, with no independent evidence for exact additivity.
domain assumption Attention sublayers, routers, and shared experts are replicated on every server; all cross-server traffic comes from routed-expert execution.
Section III-C assumes attention is always local; the authors acknowledge that sharded attention is a future extension, so the latency model breaks if attention crosses servers.
domain assumption The active probability, skip probability, and activation frequency are approximately independent in Eq. (27)-(28).
The paper states these are weakly positively correlated, which slightly overestimates deep-expert execution frequency; the placement benefit is therefore approximate.
domain assumption The baseline deployment and similarity grouping from [37] are available and satisfy memory and coverage constraints.
Algorithm 1 initializes from [37], and Property 1 assumes every expert can be stored at least once; the present paper does not re-derive the substitute model.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TrimMoE A communication aware and adaptive depth framework for distributed edge inference." pith.science (2026). https://pith.science/paper/OUCESMLI

@misc{pith2026260800573,
  author       = {Pith},
  title        = {Pith review of: TrimMoE A communication aware and adaptive depth framework for distributed edge inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OUCESMLI}},
  note         = {Machine review of arXiv:2608.00573}
}

read the original abstract

Serving Mixture-of-Experts (MoE) large language models across distributed edge servers is bottlenecked by the cross-server expert transmission. The existing approaches mainly focus on how to reach a remote expert faster. However, in this paper, we instead consider whether a given layer, and the layers after it, need to be executed at all. To this end, a communication-aware adaptive-depth framework is proposed in this paper, termed TrimMoE, which couples layer skipping and confidence-based early exit with substitute execution and server-expert selection under a unified quality budget. Specifically, in the offline stage, TrimMoE freezes the backbone, trains the lightweight per-layer exit heads, calibrates the per-layer importance thresholds, and allocates the expert replicas by a skip/exit-aware redundancy benefit. In the online stage, a transition-aware look-ahead anticipates the token movement, so that the depth reduction targets the costliest transmissions, and besides, two feedback rules adapt the delay-quality weights and the exit threshold. Moreover, we prove that the substitution-and-skipping proxy degradation never exceeds the configured budget, and that the early exit is admitted only under a calibrated confidence gate. On a heterogeneous 10-server testbed with Switch-Base-8E, Qwen-MoE-A2.7B, and Mixtral-8x7B, TrimMoE reduces the average latency by up to 62.8%, lowers the cross-server traffic and the remote-execution ratio, and sustains high throughput under load, while keeping the task-quality degradation within a 2% bound.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 30 canonical work pages

[1]

A survey of large language models,

W. Zhao, K. Zhou, et al., "A survey of large language models," Frontiers of Computer Science, vol. 20, 2026, pp: 1-144

work page 2026
[2]

A survey on efficient inference for large language models,

Z. Zhou, X. Ning, K. Hong, et al., "A survey on efficient inference for large language models," arXiv:2404.14294, 2024, pp: 1-36

arXiv 2024
[3]

A r e v i e w o n edge large language models: Design, execution, and applications,

Y. Z h e n g , Y. C h e n , B . Q i a n , X . S h i , Y. S h u , a n d J . C h e n , " A r e v i e w o n edge large language models: Design, execution, and applications," ACM Computing Surveys, vol.57, no.8, 2025, pp: 1-35

work page 2025
[4]

Mixtral of experts,

A.Q. Jiang, A. Sablayrolles, A. Roux, et al., "Mixtral of experts," arXiv:2401.04088, 2024, pp: 1-13

arXiv 2024
[5]

A survey on mixture of experts in large language models,

W. Cai, J. Jiang, F. Wang, J. Tang, S. Kim, and J. Huang, "A survey on mixture of experts in large language models," IEEE Transactions on Knowledge and Data Engineering, vol.37, no.7, 2025, pp: 3896-3915

work page 2025
[6]

GPTQ: Accurate post-training quantization for generative pre-trained transformers,

E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, "GPTQ: Accurate post-training quantization for generative pre-trained transformers," ICLR, 2023, Kigali, Rwanda, pp: 1-16

work page 2023
[7]

LLM-Pruner: On the structural pruning of large language models,

X. Ma, G. Fang, and X. Wang, "LLM-Pruner: On the structural pruning of large language models," NeurIPS, 2023, New Orleans, USA, pp: 1-20

work page 2023
[8]

Survey on knowledge distillation for large language models: Methods, Evaluation, and Application,

C. Yang, Y . Zhu, W. Lu, Y . Wang, Q. Chen, C. Gao, B. Yan, Y . Chen, "Survey on knowledge distillation for large language models: Methods, Evaluation, and Application," ACM Transactions on Intelligent Systems and Technology, vol.16, no.6, 2025, pp: 1-27

work page 2025

Show all 37 references

[9]

SVD-LLM: Truncation-aware singular value decomposition for large language model compression,

X. Wang, Y . Zheng, Z. Wan, and M. Zhang, "SVD-LLM: Truncation-aware singular value decomposition for large language model compression," ICLR, 2025, Singapore, pp: 1-21

2025
[10]

LoRA: Low-rank adaptation of large language models,

E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, et al., "LoRA: Low-rank adaptation of large language models," ICLR, 2022, Virtual, pp: 1-26

2022
[11]

LLM in a flash: Efficient large language model inference with limited memory,

K. Alizadeh et al., "LLM in a flash: Efficient large language model inference with limited memory," ACL, 2024, Bangkok, Thailand, pp. 12562–12584

2024
[12]

EdgeShard: Efficient large language model inference via collaborative edge computing,

M. Zhang, X. Shen, J. Cao, Z. Cui, and S. Jiang, "EdgeShard: Efficient large language model inference via collaborative edge computing," IEEE Internet of Things Journal, vol.12, no. 10, 2025, pp: 13119-13131

2025
[13]

AlpaServe: Statistical multiplexing with model parallelism for deep learning serving,

Z. Li，L. Zheng, Y . Zhong, V . Liu, et al., "AlpaServe: Statistical multiplexing with model parallelism for deep learning serving," USENIX OSDI, 2023, Boston, USA, pp. 663-679

2023
[14]

Orca: A distributed serving system for transformer-based generative models,

G.I. Yu, J. S. Jeong, G.W. Kim, S. Kim, and B.G. Chun, "Orca: A distributed serving system for transformer-based generative models," USENIX OSDI, 2022, Carlsbad, USA, pp. 521-538

2022
[15]

Megatron-LM: Training multi-billion parameter language models using model parallelism,

M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, "Megatron-LM: Training multi-billion parameter language models using model parallelism," arXiv:1909.08053, 2019, pp: 1-15

1909 arXiv
[16]

RDMA over commodity Ethernet at scale,

C. Guo et al., "RDMA over commodity Ethernet at scale," ACM 17 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID SIGCOMM, 2016, Florianopolis, Brazil, pp. 202-215

2016
[17]

An introduction to the InfiniBand architecture,

G. F. Pfister, T. Cortes, H. Jin, et al., "An introduction to the InfiniBand architecture," High Performance Mass Storage and Parallel I/O: Technologies and Applications, IEEE Press, 2002, pp. 617-632. DOI: 10.1109/9780470544839.ch42

2002 doi
[18]

A s u r v e y o n mobile edge computing: The communication perspective,

Y. M a o , C . Yo u , J . Z h a n g , K . H u a n g , a n d K . B . L e t a i e f , " A s u r v e y o n mobile edge computing: The communication perspective," IEEE Communications Surveys & Tutorials, vol. 19, no. 4, 2017, pp. 2322-2358

2017
[19]

Resource management in mobile edge computing: A comprehensive survey,

X. Zhang and S. Debroy, "Resource management in mobile edge computing: A comprehensive survey," ACM Computing Surveys, vol. 55, no. 13s, 2023, pp. 1–37

2023
[20]

ShortGPT: Layers in large language models are more redundant than you expect,

X. Men, J. Nabende, E. Shutova, M.T. Pilehvar, "ShortGPT: Layers in large language models are more redundant than you expect," ACL, Vienna, Austria 2025, pp: 20192-20204

2025
[21]

Mixture-of-Depths: Dynamically allocating compute in transformer-based language models,

D. Raposo, S. Ritter, B. Richards, T. Lillicrap, P. C. Humphreys, and A. Santoro, "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models," arXiv:2404.02258, 2024, pp: 1-14

2024 arXiv
[22]

Confident adaptive language modeling,

T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V . Tran, Y . Tay, D. Metzler, "Confident adaptive language modeling," NeurIPS, New Orleans, USA, 2022, pp:1-17

2022
[23]

Distributed inference acceleration with adaptive DNN partitioning and offloading,

T. Mohammed, C. Joe-Wong, R. Babbar, and M.D. Francesco, "Distributed inference acceleration with adaptive DNN partitioning and offloading," IEEE NFOCOM, 2020, Toronto, Canada, pp. 854-863

2020
[24]

PArtNNer: Platform-agnostic adaptive edge-cloud DNN partitioning for minimizing end-to-end latency,

S. K. Ghosh, S. K. Raghavan Unnithan, I. Diaconita, A. Raha, V . Raghunathan, and A. Raghunathan, "PArtNNer: Platform-agnostic adaptive edge-cloud DNN partitioning for minimizing end-to-end latency," ACM Transactions on Embedded Computing Systems, vol. 23, no. 1, 2024, pp. 1-38

2024
[25]

S p l i t c o m p u t i n g a n d e a r l y exiting for deep learning applications: Survey and research challenges,

Y. M a t s u b a r a , M . L e v o r a t o , a n d F. R e s t u c c i a , " S p l i t c o m p u t i n g a n d e a r l y exiting for deep learning applications: Survey and research challenges," ACM Computing Surveys, vol. 55, no. 5, 2023, pp. 1-30

2023
[26]

Edge-LLM: A collaborative framework for large language model serving in edge computing,

F. Cai, D. Yuan, Z. Yang, and L. Cui, "Edge-LLM: A collaborative framework for large language model serving in edge computing," IEEE ICWS, Shenzhen, China, 2024 pp: 1-11

2024
[27]

Communication-efficient distributed on-device LLM inference over wireless networks,

K. Zhang, Y . Mao, S. Leng, S. Maharjan, and Y . Zhang, "Communication-efficient distributed on-device LLM inference over wireless networks," IEEE Journal of Selected Topics in Signal Processing, vol.19, no.7, 2025, pp: 1301-1317

2025
[28]

Jupiter: Fast and resource-efficient collaborative inference of generative LLMs on edge devices,

S. Ye, B. Ouyang, L. Zeng, T. Qian, X. Chu, J. Tang, and X. Chen, "Jupiter: Fast and resource-efficient collaborative inference of generative LLMs on edge devices," IEEE ICC, Montreal, Canada, 2025, pp: 1-10

2025
[29]

AdapMoE: Adaptive sensitivity-based expert gating and management for efficient MoE inference,

S. Zhong, L. Liang, Y . Wang, R. Wang, R. Huang, and M. Li, "AdapMoE: Adaptive sensitivity-based expert gating and management for efficient MoE inference," IEEE/ACM ICCAD, NJ, USA, 2024, pp: 1-9

2024
[30]

SlimCaching: Edge caching of mixture-of-experts for distributed inference,

Q. Chen, X. Chen, and K. Huang, "SlimCaching: Edge caching of mixture-of-experts for distributed inference," IEEE Transactions on Mobile Computing, vol.25, no.7, 2026, pp: 10924-10938

2026
[31]

Diff-MoE: Efficient batched MoE inference with priority-driven differential expert caching,

K. Li, W. Huang, Q. Wang, L. Zheng, X. Liao, H. Jin, and J. Xue, "Diff-MoE: Efficient batched MoE inference with priority-driven differential expert caching," ACM SC, ST. LOUIS, USA, 2025, pp: 1951-1965

2025
[32]

PROBE: Co-balancing computation and communication in MoE inference via real-time predictive prefetching,

Q. Zhu, X. Ye, Y . Liu, H. Ouyang, and C. Song, "PROBE: Co-balancing computation and communication in MoE inference via real-time predictive prefetching," arXiv:2602.00509, 2026, pp: 1-14

2026
[33]

MegaScale-Infer: Efficient mixture-of-experts model serving with disaggregated expert parallelism,

R. Zhu, Z. Jiang, C. Jin, P. Wu, et al., "MegaScale-Infer: Efficient mixture-of-experts model serving with disaggregated expert parallelism," ACM SIGCOMM, Coimbra, Portugal, 2025, pp: 592-608

2025
[34]

WDMoE: Wireless distributed mixture of experts for large language models,

N. Xue, Y . Sun, Z. Chen, M. Tao, X. Xu, L. Qian, S. Cui, and P. Zhang, "WDMoE: Wireless distributed mixture of experts for large language models," IEEE GLOBECOM, Cape Town, South Africa, 2024, pp: 1-10

2024
[35]

MoE2: Optimizing collaborative inference for edge large language models,

L. Jin, Y . Zhang, Y . Li, S. Wang, H. H. Yang, J. Wu, and M. Zhang, "MoE2: Optimizing collaborative inference for edge large language models," IEEE Transactions on Networking, vol.34, 2026, pp: 4637-4651

2026
[36]

LayerSkip: Enabling early exit inference and self-speculative decoding,

M. Elhoushi et al., "LayerSkip: Enabling early exit inference and self-speculative decoding," ACL, Bangkok, Thailand, 2024, pp. 12622–12642

2024
[37]

OrderMoE: An Expert Similarity Driven Distributed Edge MoE Inference,

X. Yuan, N. Li, Q. Chen, W. Xu, A.V . Vasilakos, S. Guo, H. Zhang, “OrderMoE: An Expert Similarity Driven Distributed Edge MoE Inference,” arXiv: 2607.17154, 2026, pp: 1-17

2026 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

A survey of large language models,

W. Zhao, K. Zhou, et al., "A survey of large language models," Frontiers of Computer Science, vol. 20, 2026, pp: 1-144

work page 2026

[2] [2]

A survey on efficient inference for large language models,

Z. Zhou, X. Ning, K. Hong, et al., "A survey on efficient inference for large language models," arXiv:2404.14294, 2024, pp: 1-36

arXiv 2024

[3] [3]

A r e v i e w o n edge large language models: Design, execution, and applications,

Y. Z h e n g , Y. C h e n , B . Q i a n , X . S h i , Y. S h u , a n d J . C h e n , " A r e v i e w o n edge large language models: Design, execution, and applications," ACM Computing Surveys, vol.57, no.8, 2025, pp: 1-35

work page 2025

[4] [4]

Mixtral of experts,

A.Q. Jiang, A. Sablayrolles, A. Roux, et al., "Mixtral of experts," arXiv:2401.04088, 2024, pp: 1-13

arXiv 2024

[5] [5]

A survey on mixture of experts in large language models,

W. Cai, J. Jiang, F. Wang, J. Tang, S. Kim, and J. Huang, "A survey on mixture of experts in large language models," IEEE Transactions on Knowledge and Data Engineering, vol.37, no.7, 2025, pp: 3896-3915

work page 2025

[6] [6]

GPTQ: Accurate post-training quantization for generative pre-trained transformers,

E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, "GPTQ: Accurate post-training quantization for generative pre-trained transformers," ICLR, 2023, Kigali, Rwanda, pp: 1-16

work page 2023

[7] [7]

LLM-Pruner: On the structural pruning of large language models,

X. Ma, G. Fang, and X. Wang, "LLM-Pruner: On the structural pruning of large language models," NeurIPS, 2023, New Orleans, USA, pp: 1-20

work page 2023

[8] [8]

Survey on knowledge distillation for large language models: Methods, Evaluation, and Application,

C. Yang, Y . Zhu, W. Lu, Y . Wang, Q. Chen, C. Gao, B. Yan, Y . Chen, "Survey on knowledge distillation for large language models: Methods, Evaluation, and Application," ACM Transactions on Intelligent Systems and Technology, vol.16, no.6, 2025, pp: 1-27

work page 2025