The reviewed record of science sign in
Pith

arxiv: 2605.27435 · v1 · pith:FEH2JYPB · submitted 2026-05-22 · cs.AR · cs.AI

When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference

Pu Li , Jiawen Qi , Qinyu Chen This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-30 14:59 UTCgrok-4.3pith:FEH2JYPBrecord.jsonopen to challenge →

classification cs.AR cs.AI
keywords mobile LLM inferenceNPU accelerationCPU vs NPUprefill stagedecode stageenergy consumptionheterogeneous SoCpipeline benchmarking
0
0 comments X

The pith

CPUs outperform NPUs in the prefill stage of mobile LLM inference while NPUs give only marginal decode gains and raise energy use.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper measures LLM inference performance on a mobile CPU-NPU system by breaking execution into prefill and decode stages. It finds that CPUs run up to 1.6 times faster than NPUs during the compute-heavy prefill stage, while NPUs deliver only 1.05 to 1.2 times speedup in the memory-bound decode stage. The work also shows that moving more work to the NPU increases energy consumption by up to 51 percent. These measurements matter because on-device LLM deployment increasingly assumes NPUs will provide reliable acceleration, yet the stage-level results indicate that assumption does not hold uniformly.

Core claim

CPUs outperform NPUs in the compute-intensive Prefill stage (up to 1.6x), while NPUs provide only limited acceleration in the memory-bound Decode stage (1.05-1.2x). Scheduling overhead and cross-backend fallback reduce the practical benefits of NPU offloading. Increasing NPU offloading leads to higher energy consumption (up to 51%).

What carries the argument

OPMASK-based controlled pipeline decomposition methodology that isolates communication, quantization, and computation overheads within the NPU execution path.

If this is right

  • NPU offloading decisions must be made stage by stage rather than applied uniformly across the inference pipeline.
  • Scheduling overhead and fallback paths must be reduced before NPU acceleration becomes practical for on-device LLMs.
  • NPU hardware designs for mobile LLM inference should prioritize improvements in both compute-intensive and memory-bound phases.
  • Energy budgets for mobile LLM deployment will be higher when NPU usage increases, requiring trade-off analysis at design time.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same stage-level reversal may appear in other heterogeneous mobile accelerators if their operator support matches the NPU pattern.
  • Applying the decomposition method to smaller or quantized models could test whether the prefill disadvantage shrinks with reduced compute load.
  • Future mobile SoC roadmaps might benefit from tighter CPU-NPU data sharing to cut the observed communication overhead.

Load-bearing premise

The OPMASK-based controlled pipeline decomposition isolates communication, quantization, and computation overheads without introducing measurement artifacts or bias from the decomposition itself.

What would settle it

Repeating the stage-level benchmarks on the same mobile SoC but without applying the OPMASK decomposition and checking whether the CPU-NPU performance reversal in prefill disappears.

Figures

Figures reproduced from arXiv: 2605.27435 by Jiawen Qi, Pu Li, Qinyu Chen.

Figure 1
Figure 1. Figure 1: Overview of the multi-level analysis framework. Stage-level analysis [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: System-level throughput under varying NPU offloading ratios ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Decode-stage per-operator latency comparison (op-usec vs. call-usec) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
read the original abstract

Deploying large language models (LLMs) on mobile devices increasingly relies on heterogeneous execution, yet no prior study has systematically characterized NPU effectiveness at the operator and pipeline level. We present the first stage-aware, multi-level benchmarking study of mobile LLM inference on a CPU-NPU heterogeneous SoC. We introduce an OPMASK-based controlled pipeline decomposition methodology that isolates communication, quantization, and computation overheads within the NPU execution path. Our results reveal a counter-intuitive stage-level performance reversal: CPUs outperform NPUs in the compute-intensive Prefill stage (up to 1.6x), while NPUs provide only limited acceleration in the memory-bound Decode stage (1.05-1.2x). We further show that scheduling overhead and cross-backend fallback reduce the practical benefits of NPU offloading. For the energy trend, increasing NPU offloading leads to higher energy consumption (up to 51%). Based on these findings, we derive design guidelines for NPU architects targeting on-device LLM inference.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper presents the first stage-aware benchmarking study of LLM inference on mobile CPU-NPU heterogeneous SoCs. It introduces an OPMASK-based controlled pipeline decomposition to isolate communication, quantization, and computation overheads, and reports a performance reversal: CPUs outperform NPUs by up to 1.6x in the compute-intensive prefill stage while NPUs yield only 1.05-1.2x in the memory-bound decode stage. It further finds that greater NPU offloading increases energy consumption by up to 51% and derives design guidelines for NPU architects.

Significance. If the OPMASK decomposition is shown to be free of differential measurement bias, the work supplies the first systematic operator- and stage-level characterization of NPU effectiveness for on-device LLMs. The empirical reversal and energy findings would directly inform heterogeneous scheduling policies and future NPU microarchitecture targeting LLM workloads.

major comments (2)
  1. [Methods (OPMASK)] Methods section (OPMASK decomposition): the central reversal claim (CPU 1.6x prefill advantage, limited NPU decode gain) rests on the assumption that the OPMASK masking and control logic isolates overheads without introducing scheduling or timing artifacts that differ between CPU and NPU paths. No explicit cross-check (end-to-end vs. decomposed latency on identical runs, or ablation of the mask itself) is described; this validation is required before the stage-level differences can be treated as hardware properties rather than measurement effects.
  2. [Results] Results section (performance and energy numbers): the abstract and results report concrete speedups (1.6x, 1.05-1.2x) and energy penalties (up to 51%) without error bars, model sizes, dataset details, number of runs, or raw measurement distributions. These omissions make it impossible to assess whether the reported reversals exceed measurement noise or post-hoc selection effects.
minor comments (2)
  1. [Methods] The paper should clarify the exact set of operators masked by OPMASK and whether fallback paths are identical across CPU-only and NPU-offloaded configurations.
  2. [Figures] Figure captions and axis labels for stage-level latency and energy plots should explicitly state the number of repetitions and any normalization applied.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. The comments highlight important aspects of methodological validation and experimental reporting that we will address in the revision.

read point-by-point responses

  1. Referee: Methods section (OPMASK decomposition): the central reversal claim (CPU 1.6x prefill advantage, limited NPU decode gain) rests on the assumption that the OPMASK masking and control logic isolates overheads without introducing scheduling or timing artifacts that differ between CPU and NPU paths. No explicit cross-check (end-to-end vs. decomposed latency on identical runs, or ablation of the mask itself) is described; this validation is required before the stage-level differences can be treated as hardware properties rather than measurement effects.

    Authors: We agree that explicit validation of the OPMASK decomposition is necessary to confirm the absence of differential artifacts. In the revised manuscript we will add a new subsection presenting end-to-end versus decomposed latency comparisons performed on identical runs, together with mask-ablation results. These additions will demonstrate that the reported stage-level differences arise from hardware properties rather than measurement effects. revision: yes

  2. Referee: Results section (performance and energy numbers): the abstract and results report concrete speedups (1.6x, 1.05-1.2x) and energy penalties (up to 51%) without error bars, model sizes, dataset details, number of runs, or raw measurement distributions. These omissions make it impossible to assess whether the reported reversals exceed measurement noise or post-hoc selection effects.

    Authors: We accept that the current reporting lacks sufficient statistical and experimental detail. The revised manuscript will include error bars (standard deviation across repeated runs), explicit model sizes and architectures, dataset specifications, the number of runs per configuration, and summary statistics of the raw measurement distributions. These changes will allow readers to evaluate whether the observed reversals exceed measurement variability. revision: yes

Circularity Check

0 steps flagged

Empirical benchmarking study with no derivation chain or self-referential reductions

full rationale

The paper is a measurement-driven benchmarking study that introduces an OPMASK-based decomposition method and reports observed speedups and energy trends from hardware runs. No equations, fitted parameters, or predictions are presented that reduce to the inputs by construction. The abstract and described content contain no self-citations used as load-bearing uniqueness theorems, no ansatzes smuggled via prior work, and no renaming of known results as new derivations. The central claims rest on direct empirical isolation of stages rather than any closed logical loop, making the work self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 1 invented entities

The paper is an empirical measurement study; it introduces one new method (OPMASK) but does not rely on fitted parameters, unproven axioms, or postulated physical entities.

invented entities (1)
  • OPMASK methodology no independent evidence
    purpose: Controlled pipeline decomposition to isolate communication, quantization, and computation overheads in NPU path
    New technique presented in the abstract as the core enabler of the stage-level analysis.

pith-pipeline@v0.9.1-grok · 5708 in / 1237 out tokens · 33828 ms · 2026-06-30T14:59:59.083207+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages · 2 internal anchors

  1. [1]

    A survey of LLM inference systems,

    J. Pan and G. Li, “A survey of LLM inference systems,”arXiv preprint arXiv:2506.21901, 2025

    work page arXiv 2025

  2. [2]

    Empowering edge intelligence: A comprehensive survey on on-device AI models,

    X. Wang, Z. Tang, J. Guo, T. Meng, C. Wang, T. Wang, and W. Jia, “Empowering edge intelligence: A comprehensive survey on on-device AI models,”ACM Computing Surveys, 2025

    work page 2025

  3. [3]

    LLaMA 3.2: Vision and edge-optimized models for multi- modal and mobile AI,

    Meta AI, “LLaMA 3.2: Vision and edge-optimized models for multi- modal and mobile AI,” Meta AI Blog, 2024

    work page 2024

  4. [4]

    Qwen3 Technical Report

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate post-training quantization for generative pre-trained transformers,”arXiv preprint arXiv:2210.17323, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  6. [6]

    AWQ: Activation-aware weight quantiza- tion for on-device LLM compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “AWQ: Activation-aware weight quantiza- tion for on-device LLM compression and acceleration,” inProceedings of Machine Learning and Systems, vol. 6, 2024, pp. 87–100

    work page 2024

  7. [7]

    SmoothQuant: Accurate and efficient post-training quantization for large language models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and efficient post-training quantization for large language models,” inInternational Conference on Machine Learn- ing. PMLR, 2023, pp. 38 087–38 099

    work page 2023

  8. [8]

    Scaling laws for precision,

    T. Kumar, Z. Ankner, B. F. Spector, B. Bordelon, N. Muennighoff, M. Paul, C. Pehlevan, C. R ´e, and A. Raghunathan, “Scaling laws for precision,”arXiv preprint arXiv:2411.04330, 2024

    work page arXiv 2024

  9. [9]

    Efficient memory management for large language model serving with PagedAttention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” inProceedings of the 29th Symposium on Operating Systems Principles (SOSP), 2023

    work page 2023

  10. [10]

    MLC-LLM: Universal llm deployment engine with ml com- pilation,

    MLC AI, “MLC-LLM: Universal llm deployment engine with ml com- pilation,” https://github.com/mlc-ai/mlc-llm, 2024, gitHub repository. Accessed: 2026-05-22

    work page 2024

  11. [11]

    Fast on-device LLM inference with NPUs,

    D. Xu, H. Zhang, L. Yang, R. Liu, G. Huang, M. Xu, and X. Liu, “Fast on-device LLM inference with NPUs,” inProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2025, pp. 445–462

    work page 2025

  12. [12]

    Powerinfer-2: Fast large language model inference on a smartphone,

    Z. Xue, Y . Song, Z. Mi, X. Zheng, Y . Xia, and H. Chen, “PowerInfer-2: Fast large language model inference on a smartphone,”arXiv preprint arXiv:2406.06282, 2024

    work page arXiv 2024

  13. [13]

    Flightllm: Efficient large language model inference with a complete mapping flow on FPGAs,

    S. Zeng, J. Liu, G. Dai, X. Yang, T. Fu, H. Wang, W. Ma, H. Sun, S. Li, Z. Huanget al., “Flightllm: Efficient large language model inference with a complete mapping flow on FPGAs,” inProceedings of the 2024 ACM/SIGDA International Symposium on Field Programmable Gate Arrays, 2024, pp. 223–234

    work page 2024

  14. [14]

    MLPerf mobile inference benchmark,

    V . J. Reddi, D. Kanter, P. Mattson, J. Duke, T. Nguyen, R. Chukka, K. Shiring, K.-S. Tan, M. Charlebois, W. Chouet al., “MLPerf mobile inference benchmark,”arXiv preprint arXiv:2012.02328, 2020

    work page arXiv 2012

  15. [15]

    Characterizing mobile SoC for accelerating heterogeneous LLM inference,

    L. Chen, D. Feng, E. Feng, Y . Wang, R. Zhao, Y . Xia, P. Xu, and H. Chen, “Characterizing mobile SoC for accelerating heterogeneous LLM inference,” inProceedings of the 31st ACM SIGOPS Symposium on Operating Systems Principles (SOSP), 2025

    work page 2025

  16. [16]

    Scaling LLM test-time compute with mobile NPU on smartphones,

    Z. Hao, J. Wei, T. Wang, M. Huang, H. Jiang, S. Jiang, T. Cao, and J. Ren, “Scaling LLM test-time compute with mobile NPU on smartphones,”arXiv preprint arXiv:2509.23324, 2025

    work page arXiv 2025

  17. [17]

    Challenging GPU dominance: When CPUs outperform for on-device LLM inference,

    H. Zhang and J. Huang, “Challenging GPU dominance: When CPUs outperform for on-device LLM inference,”arXiv preprint arXiv:2505.06461, 2025

    work page arXiv 2025

  18. [18]

    llama.cpp: Llm inference in c/c++,

    G. Gerganov and contributors, “llama.cpp: Llm inference in c/c++,” https://github.com/ggml-org/llama.cpp, 2023, accessed: 2026-05-22

    work page 2023