Pith. sign in

REVIEW 4 major objections 6 minor 69 references

EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a software-hardware co-designed edge accelerator can reconcile mixture-of-experts and speculative decoding, cutting expert memory traffic enough to lower latency by up to 56.3% and energy by up to 44.1% while holding…

desk verdict A genuinely useful co-design that reframes MoE+speculative decoding around prompt- and depth-level structure, but the headline 56%/44% gains rest on simulator numbers and a bf16/A8W4 gap that needs closing before the claims are taken at face value. read the letter →

arxiv 2608.05303 v1 pith:FQDUY6JE submitted 2026-08-05 cs.AR cs.CLcs.LG

Sangwoo Ha , Hyunwoo Seo , Yurim Jo , Youngjin Moon , Hoi-Jun Yoo This is my paper · ORCID
classification cs.ARcs.CLcs.LG
keywords edgeLLMinferencemixture-of-expertsspeculativedecodingexternalmemoryaccessexpertreusechannelcoalescingacceleratorsoftware-hardwareco-design
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the two leading tricks for cheap edge inference, mixture-of-experts (MoE) and speculative decoding, fight each other: speculative decoding feeds several candidate tokens into the model at once, and each token can call a different expert, so the memory traffic of loading experts grows. The proposed device, EdgeXpert, turns that conflict into a saving in two steps. During prefill, a lightweight encoder marks important tokens, and only those tokens open new experts; the rest reuse a shared expert set. During decoding, candidate tokens at the same tree depth often want the same experts while only one of them will be kept, so EdgeXpert loads only a salient subset of each expert's channels and compensates with extra in-core computation. On a 28nm test chip at 800 MHz, the paper reports up to 56.3% latency reduction, 44.1% energy reduction, and near-baseline accuracy across four MoE models, meeting edge responsiveness targets of under 450 ms to first token and 50 ms per output token.

What carries the argument

Two named mechanisms carry the argument. Prompt-wise expert reuse reformulates MoE routing as prompt-level set formation rather than independent per-token expert selection: a lightweight bidirectional encoder scores tokens by summary-token attention, a threshold-based partitioning network splits them into major, minor, and trivial groups, and a budgeted routing policy builds a shared expert set from major tokens' top experts, then routes minor and trivial tokens with a reduced budget by preferring intersections with the shared set before falling back to remaining top candidates. Its operating point is chosen by a Pareto search over candidate and routed expert counts that minimizes MoE-layer output deviation at a target activated-expert count. Depth-aware expert coalescing converts channel-level sparsity into DRAM-load reduction: for each tree depth, the hardware computes a channel score from input magnitudes and attention weights across routed tokens, loads only channels whose salience score exceeds a selection fraction of the channel score, and then runs computational calibration, recomputing all routed tokens over the already-loaded channels to restore acceptance length. A dual reconfigurable adder tree executes the score accumulation for multiple depth groups, and an output-channel-first weight fetch keeps the selected channels contiguous in DRAM.

What would settle it

Measure the same-depth expert co-activation ratio (the paper reports up to 88.4% for three or more shared experts among eight) together with acceptance length and benchmark accuracy on a different MoE family and a different draft model; if the co-activation ratio falls materially or the acceptance length after computational calibration drops below baseline, the central EMA reduction claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the apparent incompatibility between MoE and speculative decoding on memory-limited devices is not fundamental; it is an artifact of loading expert weights without regard to what the candidate tokens share. The paper first documents the failure mode: attaching a self-speculative draft framework such as the one in [25] to a MoE model produces about 4.3 accepted tokens per stage, but the verification pass activates 52.1 experts per stage, inflating expert EMA and compute by 6.5x over autoregressive decoding. EdgeXpert then targets the two stages separately. For prefill, prompt-wise expert reuse treats routing as the construction of a shared expert set: a tiny encoder computes a summary-token attention vector, a threshold-based partitioning network labels tokens as major, minor, or trivial, major tokens open experts at the full routing budget, and minor and trivial tokens keep their top-1 expert and fill remaining slots from the shared set. The reported effect is 55.7% lower prefill expert EMA with negligible additional encoder traffic. For decode, depth-aware expert coalescing exploits two measured properties of the candidate token tree: same-depth candidate tokens co-activate experts (up to 88.4% of same-depth groups share three or more of eight experts) and at most one token per depth survives verification. Instead of loading the union of all channels required by a depth group, EdgeXpert loads channels whose salience score exceeds a fraction of the channel score and then applies computational calibration, letting every token recompute over the loaded channels to recover acceptance length without extra DRAM traffic. Synthesized at 800 MHz in 28nm with 8K MACs and 512 KB on-chip SRAM, the device reports 10.7-61.1% latency reduction and 9.5-56.6% energy reduction against its own baseline, and up to 56.3% latency and 44.1% energy reduction against the three prior baselines, with MT-Bench loss at most 0.1 and GSM8K degradation of 3-4% only in non-thinking mode.

Load-bearing premise

The argument hinges on the empirical regularity that same-depth candidate tokens largely activate the same experts, so loading a subset of channels and recomputing in-core loses little; when that overlap is absent or calibration cannot restore acceptance length, the reported latency and energy savings collapse.

Editorial extensions

If this is right

  • MoE and speculative decoding become a complementary pair on memory-bound edge hardware instead of an incompatible one: MoE shrinks the active weights per stage and speculative decoding shrinks the number of stages.
  • Edge deployments can hit the 450 ms time-to-first-token and 50 ms per-token targets on models up to 30B-A3B with 16 GB/s DRAM, where the comparison says EdgeXpert is the only design satisfying both across all evaluated benchmarks.
  • Channel sparsity from pruning translates into actual external-memory reduction only with hardware that fetches experts and channels selectively on the fly; software-only pruning mainly saves compute.
  • Larger experts benefit most: the paper finds the largest latency reduction on the model with the largest expert size, and expects finer-grained or larger MoE models to gain more from depth-aware coalescing.
  • Computational calibration after coalescing recovers acceptance length, so the extra in-core compute is a net win because external memory dominates energy on this class of devices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that prompt-wise expert reuse could carry over to multimodal inputs, since the paper uses summary-token attention as an importance signal and the same signal exists for image tokens, but the paper only demonstrates text MoE models.
  • I infer that the decode-stage benefit depends on draft-tree coherence; a draft model that samples more diverse hypotheses would reduce same-depth expert overlap and shrink the EMA gain, so a useful extension is to report the gain as a function of tree diversity.
  • I infer that the same channel-coalescing-plus-calibration idea could be applied to GPU inference where verification is batched by depth, since that setting would not require the specialized on-chip adder tree the paper implements.
  • I infer that the calibration data dependence means the method is not fully parameter-free: the routing budgets and channel selection ratio are set once per model using a validation corpus and a small prompt sample, so a different domain may require recalibration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes EdgeXpert, a software-hardware co-designed accelerator for on-device LLM inference that targets the external memory access (EMA) overhead arising when mixture-of-experts (MoE) and speculative decoding are combined. The core algorithmic contributions are a prefill-stage prompt-wise expert reuse policy, which builds a shared expert set from important tokens and routes less important tokens with reduced expert budgets, and a decode-stage depth-aware expert coalescing policy, which loads only salient expert channels for same-depth candidate tokens and uses computational calibration to recover acceptance length and accuracy. The hardware is synthesized in Samsung 28nm at 800 MHz, and an instruction-level simulator reports up to 56.3% latency reduction and 44.1% energy reduction compared to prior works, with TTFT below 450 ms and TPOT below 50 ms across four MoE models.

Significance. If the reported results are reproducible, the paper makes a valuable contribution: it demonstrates a concrete mechanism for turning the MoE-plus-speculative-decoding combination from an EMA-increasing configuration into an EMA-reducing one, which is a real obstacle for edge deployment. The hardware design is described in detail, the ablation structure is coherent, and the artifact is publicly released. The paper also selects hyperparameters via distribution-based metrics rather than downstream benchmark accuracy, which reduces overfitting concerns. However, the headline quantitative claims rest on an unvalidated instruction-level simulator and on a precision mismatch between the released artifact and the simulated configuration; both need to be addressed before the central claims can be accepted.

major comments (4)
  1. [Appendix H and Section V-B] The precision mismatch between the released artifact and the reported results is load-bearing. Appendix H states that the artifact runs in bf16, while the paper's results are simulated with A8W4 using GPTQ group size 32. The simulator computes EMA, latency, and energy from the A8W4 memory footprint, but the acceptance lengths and accuracy in Table IV, which set the number of decode stages through Eq. (4), appear to come from the bf16 artifact, since no quantized checkpoints or quantized accuracy runs are released. If A8W4 quantization lowers the acceptance length relative to bf16, the reported EMA reductions are inflated, because total EMA per accepted token scales roughly as P_total/NA. The paper should provide A8W4 accuracy and acceptance-length measurements, or at minimum a sensitivity analysis showing how the headline reductions degrade with a 0.5-token drop in NA.
  2. [Table IV and Section V-B] The claim of 'near-baseline accuracy' is not fully supported by the reported numbers. For example, OLMoE drops from 62.6% to 58.9% on GSM8K and from 51.4% to 49.1% on MMLU; Qwen3 drops from 72.8% to 69.4% on GSM8K. The text acknowledges a 3-4% GSM8K degradation but attributes it to the non-thinking setting and shows mitigation only for Qwen3 in thinking mode on one benchmark. These drops are larger than the stated '1.0-3.1%' range, and the paper should either qualify the headline accuracy claim with the per-benchmark magnitudes or provide additional evidence that the degradation is acceptable for the target use cases.
  3. [Section III-B and Section IV-C] The decode-stage gains rest on the empirical premise that same-depth candidate tokens co-activate experts, with the paper reporting 'up to 88.4%' co-activation for three or more of eight experts. This statistic is given as an upper bound without specifying the model, layer, or depth distribution. The depth-aware expert coalescing and the sr calibration based on 75% channel coverage both rely on this overlap being typical rather than exceptional. The paper should report the average and variance of the co-activation ratio across the four evaluated models and across layers and depths, and should show that the 88.4% figure is representative rather than a best case.
  4. [Section V-A] All latency and energy results are produced by an instruction-level custom simulator rather than by RTL simulation or silicon measurement. The simulator accumulates per-instruction costs obtained from post-synthesis analysis, but there is no validation that the simulator's cycle counts and memory-request timing match the synthesized RTL behavior. Given that the headline 56.3% latency and 44.1% energy reductions are the central claims, the paper should provide a validation experiment, for example comparing simulator-reported cycle counts against RTL simulation for representative MoE layer configurations, or clearly state the simulator's accuracy limitations and how they affect the reported speedups.
minor comments (6)
  1. [Section IV-B] The terminology 'CLS token' may be confusing for decoder-only MoE LLMs; clarify that the importance signal comes from the lightweight external encoder's [CLS] representation rather than from the target model.
  2. [Table III] The per-model hyperparameters (kmin, nmin, ktri, ntri, θ1, θ2, sr) are calibrated on WikiText-2 and MT-Bench prompts; it would strengthen the paper to report how sensitive the end-to-end results are to small perturbations of these values.
  3. [Table II] The baseline scaling procedure (SMoLPU from 200 MHz to 800 MHz, EdgeMoE mapped to dedicated INT MAC units) should be described in more detail, including whether the baseline pruning policies are reimplemented in the same instruction-level simulator and how their power and throughput figures are derived.
  4. [Section V-B] The phrase 'maximum degradation is only 1.0-3.1%' should be rephrased as percentage points and should be applied only to the PR configuration, since the ALL configuration shows larger drops on benchmarks such as GSM8K.
  5. [Appendix H] The artifact description should state whether the A8W4 quantization configuration used for the paper's simulations will be released, and whether run.py can reproduce the simulated setting or only the bf16 setting.
  6. [Figure 12(c)] The TTFT and TPOT thresholds (450 ms and 50 ms) are stated in the text but are not visually marked in the figure; adding threshold lines would make the claim easier to verify.

Circularity Check

1 steps flagged · score 2.0 of 10

Mostly self-contained hardware/MoE derivation; one mild benchmark-calibration overlap (s_r on MT-Bench) and a bf16/A8W4 validity caveat.

  1. fitted input called prediction [Section V-A (Experimental Setup), hyperparameter calibration paragraph; evaluation in Table IV and Fig. 12]
    "The calibration is performed only once per model:k min,tri andn min,tri are calibrated on WikiText-2 validation set [65], whilesris selected using a small calibration set sampled from MT-Bench prompts to preserve the original channel distribution, rather than to optimize benchmark accuracy."

    The decode-stage selection ratio s_r is load-bearing: it sets how aggressively depth-aware coalescing drops expert channels (Section IV-C), and the acceptance-length numbers in Table IV enter Eq. (4) and the simulator's latency/energy totals. Because s_r is calibrated on MT-Bench prompts and the reported accuracy/acceptance-length table and speedup figures include MT-Bench results (Table IV, Figure 12), the MT-Bench portion of the claimed reductions is not fully out-of-sample. The paper's stated intent not to tune benchmark accuracy weakens, but does not eliminate, this overlap.

full rationale

The derivation chain is largely self-contained: Eqs. (1)-(4) are standard EMA models; the prefill and decode policies are explicitly defined algorithms; and the reported latency/energy are accumulated by an instruction-level simulator from per-instruction costs, post-synthesis power numbers, and measured DRAM traffic rather than derived from the hyperparameters by construction. The hardware claims rest on synthesis and post-synthesis power analysis, and baselines are normalized to the same technology, frequency, and bandwidth. The only mild circularity is the s_r calibration/evaluation overlap on MT-Bench prompts: s_r is fitted to a sample of MT-Bench prompts, and MT-Bench accuracy and acceptance length are then reported and used to drive the simulator totals through Eq. (4). This is a benchmark-leak concern rather than a mathematical equivalence, so it is weighted lightly. Appendix H also asserts a validity caveat: the released artifact runs in bf16 while the paper's reported results were simulated with A8W4 GPTQ quantization; if quantized acceptance lengths differ, the headline reductions would change. That is a correctness risk, not a circularity, and does not by itself raise the circularity score. No load-bearing self-citation chain or imported uniqueness claim is present: the SMoLPU baseline [3] is the authors' own prior work, but it is used as an adopted baseline and the new contributions are evaluated independently against it.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper's central design relies on several empirically calibrated per-model numbers rather than a derivation. The main free parameters are the routing budgets, token-partition thresholds, channel selection ratio, and manually set coverage or MSE targets. The key axioms are the CLS-encoder importance proxy, the same-depth expert overlap and mutual-exclusivity property, the fidelity of the instruction-level simulator, and the assumption that A8W4 quantization preserves bf16-evaluated accuracy. No new physical entities are introduced.

free parameters (6)
  • Routing budgets (kmin, nmin, ktri, ntri) per model = Granite 8/7/10/6; OLMoE 10/6/12/4; DeepSeek 6/5/8/3; Qwen 10/6/12/4
    Selected by a Pareto search over Eq. (5) balancing MoE output MSE against activated expert count. The per-model values are fitted to validation data, not derived from first principles.
  • Partitioning thresholds theta1, theta2 = Granite 0.65/0.45; OLMoE 0.80/0.40; DeepSeek 0.55/0.30; Qwen 0.80/0.55
    Calibrated once per model using WikiText-2 perplexity to define major, minor, and trivial token groups. The choice controls the prefill quality and EMA trade-off.
  • Channel selection ratio sr = Granite 0.70; OLMoE 0.90; DeepSeek 0.90; Qwen 0.90
    Chosen to maintain at least 75% average channel coverage of the accepted token on a calibration set sampled from MT-Bench prompts. This directly sets the amount of channel coalescing and the resulting accuracy and EMA trade-off.
  • Accepted-token channel coverage target = 0.75
    Hand-set threshold in Section IV-C used to pick sr. No independent derivation is given for why 75% is the right coverage level.
  • Trivial-token MSE bound = 3x the minor-token elbow-point MSE
    In Figure 7(c), the search accepts the most aggressive reuse point whose MSE is within 3 times the minor-token elbow MSE. This multiplier is chosen by hand.
  • Candidate token budget and depth = 64 candidates, max depth 6
    Section II-B sets the optimal candidate count to 64 based on the authors' own energy analysis of EAGLE-3. The result is a design choice, not a fitted physical constant.
assumptions (4)
  • domain assumption A lightweight encoder's bidirectional CLS attention is a reliable estimator of which prompt tokens should get larger expert routing budgets.
    Invoked in Sections III-A and IV-B to justify using all-MiniLM-L6-v2 for token importance. The paper compares against target-model attention only qualitatively and provides no independent benchmark of importance accuracy.
  • domain assumption Same-depth speculative candidate tokens co-activate experts and are mutually exclusive in verification.
    Sections III-B and IV-C use this to justify channel coalescing. It is measured on the evaluated models with overlap up to 88.4% for three or more of eight experts, but assumed to generalize to other models, prompts, and draft depths.
  • domain assumption The instruction-level simulator with per-instruction costs from post-synthesis power analysis faithfully reproduces the hardware's latency and energy for dynamic expert and channel loading.
    Section V-A describes the simulator, but no silicon or RTL simulation is provided to validate it. The headline 56.3% and 44.1% numbers depend on this fidelity.
  • domain assumption GPTQ A8W4 quantization preserves the behavior measured in the bf16 software artifact and in the accuracy tables.
    Appendix H states the released artifact runs bf16 while the paper's results were simulated with A8W4. The accuracy and acceptance-length tables therefore assume quantization has no additional damaging interaction with coalescing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding." pith.science (2026). https://pith.science/paper/FQDUY6JE

@misc{pith2026260805303,
  author       = {Pith},
  title        = {Pith review of: EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FQDUY6JE}},
  note         = {Machine review of arXiv:2608.05303}
}
read the original abstract

On-device deployment of Large Language Models (LLMs) has become essential for personalized edge applications. A primary bottleneck is external memory access (EMA) in feed-forward network (FFN) layers. Speculative decoding and mixture-of-experts (MoE) are promising solutions. Speculative decoding reduces the number of decoding stages by generating multiple tokens per stage, and MoE minimizes per-stage cost through sparse expert activation. However, there is an incompatibility when combining these two techniques. We propose EdgeXpert, a software-hardware co-designed LLM accelerator that resolves this incompatibility. In the prefill stage, the prompt-wise expert reuse reformulates routing as prompt-level expert reuse rather than independent per-token expert selection. It identifies important tokens using a lightweight encoder, constructs a shared expert set from them, and routes less important tokens with a reduced expert budget to lower expert EMA. In the decode stage, depth-aware expert coalescing exploits the contextual similarity and mutual exclusivity of same-depth candidate tokens. Rather than loading the union of all required channels, EdgeXpert loads only salient channels and applies computational calibration to recover accuracy without additional memory access. Synthesized in Samsung 28nm technology at 800 MHz, EdgeXpert achieves up to 56.3% latency reduction and 44.1% energy reduction compared to prior works, while maintaining near-baseline accuracy.

Figures

Figures reproduced from arXiv: 2608.05303 by the authors.

Figure 1
Figure 1. Overall LLM system breakdown in edge devices. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) Incompatibility of speculative decoding and MoE. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Operational flow of MoE. this incompatibility is resolved, MoE and speculative decoding provide complementary benefits: MoE reduces the active FFN weights per decode stage, while speculative decoding reduces the number of decode stages. To address the above EMA overhead of the MoE layer, we propose EdgeXpert, which combines software and hardware optimizations for memory-efficient LLM inference in edge devices. Exper… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: (a) Operational flow of speculative decoding. (b) [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: Contextual similarity and expert overlap ratio. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Overall architecture. A. Overall Architecture [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: (a) Example of shared expert reuse. (b) Operation [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: (a) Detailed hardware of the partitioning network. (b) [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: (a) Limitation of pruning on EMA reduction. (b) [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: (a) Dual reconfigurable adder tree in coalescing unit. [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Comparison of speculative decoding frameworks. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: (a) Normalized end-to-end latency of EdgeXpert. (b) [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: (a) Normalized energy consumption of EdgeXpert. [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 30 canonical work pages

  1. [1]

    MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router,

    Y . Xie, Z. Zhang, D. Zhou, C. Xie, Z. Song, X. Liu, Y . Wang, X. Lin, and A. Xu, “MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router,”arXiv preprint arXiv:2410.12013, 2024

    arXiv 2024

  2. [2]

    EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices,

    R. Yi, L. Guo, S. Wei, A. Zhou, S. Wang, and M. Xu, “EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices,”IEEE Transactions on Mobile Computing, vol. 24, no. 8, pp. 7059–7073, 2025

    work page 2025

  3. [3]

    SMoLPU: 122.1µJ/Token Sparse MoE- Based Speculative Decoding Language Processing Unit with Adaptive- Offload NPU-CIM Core,

    S. Ha, J. Lee, Y . Moon, S. Whang, W. Jo, G. Park, S. Kim, S. Um, J. Ryu, Y . Jo, and H.-J. Yoo, “SMoLPU: 122.1µJ/Token Sparse MoE- Based Speculative Decoding Language Processing Unit with Adaptive- Offload NPU-CIM Core,” in2026 IEEE International Solid-State Cir- cuits Conference (ISSCC), vol. 69, 2026, pp. 312–314

    work page 2026

  4. [4]

    GPT-4 Technical Report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “GPT-4 Technical Report,”arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  5. [5]

    The Llama 3 Herd of Models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The Llama 3 Herd of Models,”arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  6. [6]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    G. Team, P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wanget al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,”arXiv preprint arXiv:2403.05530, 2024

    arXiv 2024

  7. [7]

    Squeezed atten- tion: Accelerating long context length llm inference,

    C. Hooper, S. Kim, H. Mohammadzadeh, M. Maheswaran, J. Paik, M. W. Mahoney, K. Keutzer, and A. Gholami, “Squeezed atten- tion: Accelerating long context length llm inference,”arXiv preprint arXiv:2411.09688, 2024

    arXiv 2024

  8. [8]

    ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching,

    Y . Zhao, D. Wu, and J. Wang, “ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching,” in2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024, pp. 1005–1017

    work page 2024

Show all 69 references
  1. [9]

    23.7 BROCA: A 52.4-to-559.2mW Mobile Social Agent System-on-Chip with Adaptive Bit-Truncate Unit and Acoustic-Cluster Bit Grouping,

    W. Jo, S. Hong, J. Choi, B. Kwon, H. Sang, D. Im, S. Kim, S. Kim, T. Lee, and H.-J. Yoo, “23.7 BROCA: A 52.4-to-559.2mW Mobile Social Agent System-on-Chip with Adaptive Bit-Truncate Unit and Acoustic-Cluster Bit Grouping,” in2025 IEEE International Solid-State Circuits Confere...

    2025

  2. [10]

    Fast on-device LLM inference with npus,

    D. Xu, H. Zhang, L. Yang, R. Liu, G. Huang, M. Xu, and X. Liu, “Fast on-device LLM inference with npus,” inProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, 2025, pp. 445–462

    2025

  3. [11]

    C-Transformer: An Energy-Efficient Homogeneous DNN- Transformer/SNN-Transformer Processor for Large Language Models,

    S. Kim, S. Kim, W. Jo, S. Kim, S. Hong, N. Lee, J. Lee, and H.-J. Yoo, “C-Transformer: An Energy-Efficient Homogeneous DNN- Transformer/SNN-Transformer Processor for Large Language Models,” IEEE Journal of Solid-State Circuits, 2025

    2025

  4. [12]

    MECLA: Memory-Compute-Efficient LLM Accelerator with Scaling Sub-matrix Partition,

    Y . Qin, Y . Wang, Z. Zhao, X. Yang, Y . Zhou, S. Wei, Y . Hu, and S. Yin, “MECLA: Memory-Compute-Efficient LLM Accelerator with Scaling Sub-matrix Partition,” in2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024, pp. 1032– 1047

    2024

  5. [13]

    Granite 3.0 Language Models,

    I. Granite Team, “Granite 3.0 Language Models,” October

  6. [14]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  7. [15]

    DeepSeekMoE: Towards ultimate expert spe- cialization in mixture-of-experts language models,

    D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y . Wuet al., “DeepSeekMoE: Towards ultimate expert spe- cialization in mixture-of-experts language models,”arXiv preprint arXiv:2401.06066, 2024

    2024 arXiv

  8. [16]

    GlaM: Efficient scaling of language models with mixture-of-experts,

    N. Du, Y . Huang, A. M. Dai, S. Tong, D. Lepikhin, Y . Xu, M. Krikun, Y . Zhou, A. W. Yu, O. Firatet al., “GlaM: Efficient scaling of language models with mixture-of-experts,” inInternational conference on machine learning. PMLR, 2022, pp. 5547–5569

    2022

  9. [17]

    Mixtral of Experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressandet al., “Mixtral of Experts,”arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  10. [18]

    LLaMA-MoE: Building mixture-of-experts from llama with continual pre-training,

    T. Zhu, X. Qu, D. Dong, J. Ruan, J. Tong, C. He, and Y . Cheng, “LLaMA-MoE: Building mixture-of-experts from llama with continual pre-training,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 15 913–15 923

    2024

  11. [19]

    Medusa: Simple llm inference acceleration framework with multiple decoding heads,

    T. Cai, Y . Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao, “Medusa: Simple llm inference acceleration framework with multiple decoding heads,”arXiv preprint arXiv:2401.10774, 2024

    2024 arXiv

  12. [20]

    Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation,

    H. Xia, T. Ge, P. Wang, S.-Q. Chen, F. Wei, and Z. Sui, “Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation,”arXiv preprint arXiv:2203.16487, 2022

    2022 arXiv

  13. [21]

    Fast inference from transform- ers via speculative decoding,

    Y . Leviathan, M. Kalman, and Y . Matias, “Fast inference from transform- ers via speculative decoding,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 19 274–19 286

    2023

  14. [22]

    Accelerating large language model decoding with speculative sam- pling,

    C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating large language model decoding with speculative sam- pling,”arXiv preprint arXiv:2302.01318, 2023

    2023 arXiv

  15. [23]

    LayerSkip: Enabling early exit inference and self-speculative decoding,

    M. Elhoushi, A. Shrivastava, D. Liskovich, B. Hosmer, B. Wasti, L. Lai, A. Mahmoud, B. Acun, S. Agarwal, A. Romanet al., “LayerSkip: Enabling early exit inference and self-speculative decoding,”arXiv preprint arXiv:2404.16710, 2024

    2024 arXiv

  16. [24]

    Speculative decoding with big little decoder,

    S. Kim, K. Mangalam, S. Moon, J. Malik, M. W. Mahoney, A. Gholami, and K. Keutzer, “Speculative decoding with big little decoder,”Advances in Neural Information Processing Systems, vol. 36, pp. 39 236–39 256, 2023

    2023

  17. [25]

    EAGLE-3: Scaling up inference acceleration of large language models via training-time test,

    Y . Li, F. Wei, C. Zhang, and H. Zhang, “EAGLE-3: Scaling up inference acceleration of large language models via training-time test,”arXiv preprint arXiv:2503.01840, 2025

    2025 arXiv

  18. [26]

    ML-SpecQD: Multi-level speculative decoding with quantized drafts,

    E. Georganas, D. Kalamkar, A. Kozlov, and A. Heinecke, “ML-SpecQD: Multi-level speculative decoding with quantized drafts,”arXiv preprint arXiv:2503.13565, 2025

    2025 arXiv

  19. [27]

    EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding,

    D. Xu, W. Yin, H. Zhang, X. Jin, Y . Zhang, S. Wei, M. Xu, and X. Liu, “EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding,”IEEE Transactions on Mobile Computing, vol. 24, no. 4, pp. 3256–3273, 2025

    2025

  20. [28]

    SpecMemo: Speculative Decoding is in Your Pocket,

    S. Yildirim and D. Chen, “SpecMemo: Speculative Decoding is in Your Pocket,”arXiv preprint arXiv:2506.01986, 2025

    2025 arXiv

  21. [29]

    DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Biet al., “DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning,”arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  22. [30]

    Eagle: Speculative sampling re- quires rethinking feature uncertainty,

    Y . Li, F. Wei, C. Zhang, and H. Zhang, “Eagle: Speculative sampling re- quires rethinking feature uncertainty,”arXiv preprint arXiv:2401.15077, 2024

    2024 arXiv

  23. [31]

    MoESD: Unveil Speculative Decoding’s Potential for Accelerating Sparse MoE,

    Z. Huang, L. Zhu, Z. Zhan, T. Hu, W. Mao, X. Yu, Y . Liu, and T. Zhang, “MoESD: Unveil Speculative Decoding’s Potential for Accelerating Sparse MoE,”arXiv preprint arXiv:2505.19645, 2025

    2025

  24. [32]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,

    A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Dengr, C. Ruan, D. Dai, D. Guoet al., “Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,”arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  25. [33]

    OLMoE: Open mixture- of-experts language models,

    N. Muennighoff, L. Soldaini, D. Groeneveld, K. Lo, J. Morrison, S. Min, W. Shi, P. Walsh, O. Tafjord, N. Lambertet al., “OLMoE: Open mixture- of-experts language models,”arXiv preprint arXiv:2409.02060, 2024

    2024 arXiv

  26. [34]

    Mixture of cache- conditional experts for efficient mobile device inference,

    A. Skliar, T. van Rozendaal, R. Lepert, T. Boinovski, M. Van Baalen, M. Nagel, P. Whatmough, and B. E. Bejnordi, “Mixture of cache- conditional experts for efficient mobile device inference,”arXiv preprint arXiv:2412.00099, 2024

    2024 arXiv

  27. [35]

    Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design,

    R. Cai, Y . Ro, G.-W. Kim, P. Wang, B. Ehteshami Bejnordi, A. Akella, Z. Wanget al., “Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design,”Advances in Neural Information Processing Systems, vol. 37, pp. 116 126–116 148, 2024

    2024

  28. [36]

    Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batching,

    S. Yun, K. Kyung, J. Cho, J. Choi, J. Kim, B. Kim, S. Lee, K. Sohn, and J. H. Ahn, “Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batching,” in 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO)...

    2024

  29. [37]

    20.8 Space-Mate: A 303.5mW Real-Time Sparse Mixture-of-Experts- Based NeRF-SLAM Processor for Mobile Spatial Computing,

    G. Park, S. Song, H. Sang, D. Im, D. Han, S. Kim, H. Lee, and H.-J. Yoo, “20.8 Space-Mate: A 303.5mW Real-Time Sparse Mixture-of-Experts- Based NeRF-SLAM Processor for Mobile Spatial Computing,” in2024 IEEE International Solid-State Circuits Conference (ISSCC), vol. 67, 2024, ...

    2024

  30. [38]

    SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification,

    X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, Z. Zhang, R. Y . Y . Wong, A. Zhu, L. Yang, X. Shi, C. Shi, Z. Chen, D. Arfeen, R. Abhyankar, and Z. Jia, “SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification,” ser. ASPLOS...

    2024

  31. [39]

    Specdec++: Boosting spec- ulative decoding via adaptive candidate lengths,

    K. Huang, X. Guo, and M. Wang, “Specdec++: Boosting spec- ulative decoding via adaptive candidate lengths,”arXiv preprint arXiv:2405.19715, 2024

    2024 arXiv

  32. [40]

    Fast best-of-n decoding via speculative rejection,

    H. Sun, M. Haider, R. Zhang, H. Yang, J. Qiu, M. Yin, M. Wang, P. Bartlett, and A. Zanette, “Fast best-of-n decoding via speculative rejection,”Advances in Neural Information Processing Systems, vol. 37, pp. 32 630–32 652, 2024

    2024

  33. [41]

    Sequoia: Scalable, robust, and hardware-aware speculative decoding,

    Z. Chen, A. May, R. Svirschevski, Y . Huang, M. Ryabinin, Z. Jia, and B. Chen, “Sequoia: Scalable, robust, and hardware-aware speculative decoding,”arXiv preprint arXiv:2402.12374, 2024

    2024 arXiv

  34. [42]

    Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models,

    X. Lu, Q. Liu, Y . Xu, A. Zhou, S. Huang, B. Zhang, J. Yan, and H. Li, “Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vo...

    2024

  35. [43]

    Moe-i2: Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decom- position,

    C. Yang, Y . Sui, J. Xiao, L. Huang, Y . Gong, Y . Duan, W. Jia, M. Yin, Y . Cheng, and B. Yuan, “Moe-i2: Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decom- position,” inFindings of the Association for Computational Linguistics:...

    2024

  36. [44]

    Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs,

    E. Liu, J. Zhu, Z. Lin, X. Ning, M. B. Blaschko, S. Yan, G. Dai, H. Yang, and Y . Wang, “Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs,” arXiv preprint arXiv:2407.00945, 2024

    2024 arXiv

  37. [45]

    Self-speculative decoding for on-device moe acceleration,

    P. Zheng, W. Xu, and H. Wang, “Self-speculative decoding for on-device moe acceleration,” ser. WWW ’26. New York, NY , USA: Association for Computing Machinery, 2026, p. 5155–5164. [Online]. Available: https://doi.org/10.1145/3774904.3792218

    2026

  38. [46]

    MoE-Spec: Ex- pert Budgeting for Efficient Speculative Decoding,

    B. McDanel, S. Li, S. Surineni, and H. Khaitan, “MoE-Spec: Ex- pert Budgeting for Efficient Speculative Decoding,”arXiv preprint arXiv:2602.16052, 2026

    2026

  39. [47]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inPro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolog...

    2019

  40. [48]

    Emerging properties in self-supervised vision transformers,

    M. Caron, H. Touvron, I. Misra, H. J ´egou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 9650–9660

    2021

  41. [49]

    Dense passage retrieval for open-domain question answering,

    V . Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, “Dense passage retrieval for open-domain question answering,” inProceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), 2020, pp. 6769–6781

    2020

  42. [50]

    [CLS] attention is all you need for training- free visual token pruning: Make vlm inference faster,

    Q. Zhang, A. Cheng, M. Lu, Z. Zhuo, M. Wang, J. Cao, S. Guo, Q. She, and S. Zhang, “[CLS] attention is all you need for training- free visual token pruning: Make vlm inference faster,”arXiv preprint arXiv:2412.01818v1, 2024

    2024 arXiv

  43. [51]

    HiPrune: Training-Free Visual Token Pruning via Hierarchical Attention in Vision-Language Models (Student Ab- stract),

    J. Liu, G. Zhu, and F. Du, “HiPrune: Training-Free Visual Token Pruning via Hierarchical Attention in Vision-Language Models (Student Ab- stract),” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 48, 2026, pp. 41 275–41 277

    2026

  44. [52]

    Atp-llava: Adaptive token pruning for large vision language models,

    X. Ye, Y . Gan, Y . Ge, X.-P. Zhang, and Y . Tang, “Atp-llava: Adaptive token pruning for large vision language models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 24 972–24 982

    2025

  45. [53]

    all-minilm-l6-v2: Sentence transformers model,

    N. Reimers and I. Gurevych, “all-minilm-l6-v2: Sentence transformers model,” https://huggingface.co/sentence-transformers/ all-MiniLM-L6-v2, 2021, hugging Face Model Card

    2021

  46. [54]

    Pre-Gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference,

    R. Hwang, J. Wei, S. Cao, C. Hwang, X. Tang, T. Cao, and M. Yang, “Pre-Gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference,” inProceedings of the 51st Annual International Symposium on Computer Architecture, ser. ISCA ’24. IEEE Press, 202...

    2025

  47. [55]

    Sigma: A sparse and irregular gemm ac- celerator with flexible interconnects for dnn training,

    E. Qin, A. Samajdar, H. Kwon, V . Nadella, S. Srinivasan, D. Das, B. Kaul, and T. Krishna, “Sigma: A sparse and irregular gemm ac- celerator with flexible interconnects for dnn training,” in2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEE...

    2020

  48. [56]

    2022, rev

    Micron Technology, Inc.,LPDDR4/LPDDR4X SDRAM: MT53E1536M32D4, MT53E768M64D4 Datasheet, Micron Tech- nology, Inc., Jun. 2022, rev. D, datasheet. [Online]. Available: https://www.mouser.com/datasheet/2/671/z3bm embedded lpddr4 lpddr4x-3193457.pdf

    2022

  49. [57]

    SCNN: An accelerator for compressed-sparse convolutional neural networks,

    A. Parashar, M. Rhu, A. Mukkara, A. Puglielli, R. Venkatesan, B. Khailany, J. Emer, S. W. Keckler, and W. J. Dally, “SCNN: An accelerator for compressed-sparse convolutional neural networks,”ACM SIGARCH computer architecture news, vol. 45, no. 2, pp. 27–40, 2017

    2017

  50. [58]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xinget al., “Judging llm-as-a-judge with mt-bench and chatbot arena,”Advances in neural information processing systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  51. [59]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakanoet al., “Training verifiers to solve math word problems,”arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  52. [60]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  53. [61]

    Hel- laswag: Can a machine really finish your sentence?

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hel- laswag: Can a machine really finish your sentence?”arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  54. [62]

    Think you have solved question answering? try arc, the ai2 reasoning challenge,

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,”arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  55. [63]

    Winogrande: An adversarial winograd schema challenge at scale,

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “Winogrande: An adversarial winograd schema challenge at scale,”Communications of the ACM, vol. 64, no. 9, pp. 99–106, 2021

    2021

  56. [64]

    Piqa: Reasoning about physical commonsense in natural language,

    Y . Bisk, R. Zellers, J. Gao, Y . Choiet al., “Piqa: Reasoning about physical commonsense in natural language,” inProceedings of the AAAI conference on artificial intelligence, vol. 34, no. 05, 2020, pp. 7432– 7439

    2020

  57. [65]

    Pointer Sentinel Mixture Models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer Sentinel Mixture Models,”arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  58. [66]

    MLPerf Inference interactive benchmark,

    MLCommons, “MLPerf Inference interactive benchmark,” https://github.com/mlcommons/inference/blob/master/language/ llama2-70b/README.md, 2024

    2024

  59. [67]

    Spatten: Efficient sparse attention architecture with cascade token and head pruning,

    H. Wang, Z. Zhang, and S. Han, “Spatten: Efficient sparse attention architecture with cascade token and head pruning,” in2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2021, pp. 97–110

    2021

  60. [68]

    FACT: FFN-Attention Co-optimized Transformer Architecture with Eager Correlation Prediction,

    Y . Qin, Y . Wang, D. Deng, Z. Zhao, X. Yang, L. Liu, S. Wei, Y . Hu, and S. Yin, “FACT: FFN-Attention Co-optimized Transformer Architecture with Eager Correlation Prediction,” inProceedings of the 50th Annual International Symposium on Computer Architecture, ser. ISCA ’23. Ne...

    2023

  61. [2024]

    Available: https://github.com/ibm-granite/granite-3

    [Online]. Available: https://github.com/ibm-granite/granite-3. 0-language-models/

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.